How to find download/upload progress for Uplink-C v1.0.2?

Hi!

I am able to successfully download/upload files from/to Storj using Uplink-C 1.0.2.

My code is as follows:

//export download_object_custom
// download_object_custom starts  download to the specified key.
func download_object_custom(project *C.Project, bucket_name, object_key, destFullFileName *C.char, options *C.DownloadOptions) C.DownloadResult { //nolint:golint
	if project == nil {
		return C.DownloadResult{
			error: mallocError(ErrNull.New("project")),
		}
	}
	if bucket_name == nil {
		return C.DownloadResult{
			error: mallocError(ErrNull.New("bucket_name")),
		}
	}
	if object_key == nil {
		return C.DownloadResult{
			error: mallocError(ErrNull.New("object_key")),
		}
	}

	proj, ok := universe.Get(project._handle).(*Project)
	if !ok {
		return C.DownloadResult{
			error: mallocError(ErrInvalidHandle.New("project")),
		}
	}
	scope := proj.scope.child()

	opts := &uplink.DownloadOptions{
		Offset: 0,
		Length: -1,
	}
	if options != nil {
		opts.Offset = int64(options.offset)
		opts.Length = int64(options.length)
	}

	download, err := proj.DownloadObject(scope.ctx, C.GoString(bucket_name), C.GoString(object_key), opts)
	if err != nil {
		return C.DownloadResult{
			error: mallocError(err),
		}
	}

	// Read everything from the download stream
	receivedContents, err := ioutil.ReadAll(download)
	if err != nil {
		return C.DownloadResult{
			error: mallocError(err),
		}
	}
	
	// Write the content to the desired file
	err = ioutil.WriteFile(C.GoString(destFullFileName), receivedContents, 0644)
	if err != nil {
		return C.DownloadResult{
			error: mallocError(err),
		}
	}
	
	return C.DownloadResult{
		download: (*C.Download)(mallocHandle(universe.Add(&Download{scope, download}))),
	}	
}

//export upload_object_custom
// upload_object_custom starts an upload to the specified key.
func upload_object_custom(project *C.Project, bucket_name, srcFullFileName, path *C.char, options *C.UploadOptions) C.UploadResult { //nolint:golint
	if project == nil {
		return C.UploadResult{
			error: mallocError(ErrNull.New("project")),
		}
	}
	if bucket_name == nil {
		return C.UploadResult{
			error: mallocError(ErrNull.New("bucket_name")),
		}
	}
	if path == nil {
		return C.UploadResult{
			error: mallocError(ErrNull.New("path")),
		}
	}

	proj, ok := universe.Get(project._handle).(*Project)
	if !ok {
		return C.UploadResult{
			error: mallocError(ErrInvalidHandle.New("project")),
		}
	}
	scope := proj.scope.child()

	opts := &uplink.UploadOptions{}
	if options != nil {
		if options.expires > 0 {
			opts.Expires = time.Unix(int64(options.expires), 0)
		}
	}

	upload, err := proj.UploadObject(scope.ctx, C.GoString(bucket_name), C.GoString(path), opts)
	if err != nil {
		return C.UploadResult{
			error: mallocError(err),
		}
	}

	dataToUpload, err := ioutil.ReadFile(C.GoString(srcFullFileName))
	if err != nil {
		return C.UploadResult{
			error: mallocError(err),
		}
	}

	// Copy the data to the upload.
	buf := bytes.NewBuffer(dataToUpload)
	_, err = io.Copy(upload, buf)
	if err != nil {
		_ = upload.Abort()
		//return fmt.Errorf("could not upload data: %v", err)
		return C.UploadResult{
			error: mallocError(err),
		}
	}

	// Commit the uploaded object.
	err = upload.Commit()
	if err != nil {
		//return fmt.Errorf("could not commit uploaded object: %v", err)
		return C.UploadResult{
			error: mallocError(err),
		}
	}

	return C.UploadResult{
		upload: (*C.Upload)(mallocHandle(universe.Add(&Upload{scope, upload}))),
	}
}

I am unable to find any method for download/upload progress such as:
(i) Download/upload progress(in %)
(ii) Estimated time left
etc.

In the deprecated libstorj library, there are some parameters like uv_async_t progress_handle (For example, https://github.com/storj/libstorj/blob/25218f03208e5cd0929cdfb38856f82a5a14504a/src/uploader.h)

But, I could not find any such functions in Uplink-C

Kindly help

For new uplink-c we have implemented the basic api for uploading and downloading.

You can take a look at: https://github.com/storj/uplink-c/blob/c01c92c84a7d3765e9b3606a62f4ae5638cb0d7b/testsuite/testdata/example_test.c#L73

As an example:

size_t buffer_size = 256 << 10; // downloading 256kib at a time
char *buffer = malloc(buffer_size);

Download *download = download_result.download;

// you can use `download_info(download)` to get information how much data is available, don't forget to call `free_object_result`

while (true) {
    ReadResult result = download_read(download, buffer, buffer_size);

    // now the "buffer" will contain result.bytes_read of data
    // here you could write that number of bytes to a file instead
    for (size_t p = 0; p < result.bytes_read; p++) {
        putchar(buffer[p]);
    }

    // handle any error in case something went wrong
    if (result.error) {
        if (result.error->code == EOF) {
            free_read_result(result);
            break;
        }
        fprintf(stderr, "download failed to read: %s\n", result.error->message);
        free_read_result(result);
        return;
    }
    free_read_result(result);
}

Error *close_error = close_download(download);
if (close_error) {
    fprintf(stderr, "download failed to close: %s\n", close_error->message);
    free_error(close_error);
}

free_download_result(download_result);

This isn’t as convenient for “just downloading a file to disk”, however it will work in many situations, e.g. where you might want to stream that content over network. We’ll probably add more convenience helpers along the way.

Implementing your own custom C-api is also a great way to tailor things to your liking. There you would need to do something similar for tracking the amount of data downloaded. You can implement a wrapper around either the writing or reading side (https://golangcode.com/download-a-file-with-progress/). If you wish to integrate it with C, then you would need to call the appropriate method from inside your progress tracking.

Estimating the expected time can be somewhat difficult, but the easiest would probably be, in pseudo-code: speed_of_writing = data_written / time_spent_writing; time_left = total_data_to_write / speed_of_writing.

Also beware ioutil.ReadAll, it will read the files into memory which could end up problematic with files larger than 8GB. Also, always ensure that you call download.Close and upload.Close in the failure paths, otherwise you will have a leaking connection in your system.

1 Like

I am now trying to implement the download & upload in C++ as follows:

extern "C" void fv_downloadObject(Project *project, std::string bucket, std::string id, std::string file)
{
       DownloadResult download_result = download_object(project, const_cast<char*>(bucket.c_str()), const_cast<char*>(id.c_str()), NULL);
        if (download_result.error) {
            printf("download starting failed: %s", download_result.error->message);
            free_download_result(download_result);
            return;
        }

//      size_t buffer_size = 1024;
		size_t buffer_size = 1000;
//		size_t buffer_size = 32768;
        char *buffer =static_cast<char*>(malloc(buffer_size));

		///////////////////////////////////////////////////////////////
		std::ofstream outfile (file,std::ofstream::binary);
		///////////////////////////////////////////////////////////////

        Download *download = download_result.download;

        ObjectResult object_result = download_info(download);
        require_noerror(object_result.error);
        require(object_result.object != NULL);
		
		Object *object = object_result.object;

		size_t downloaded_total = 0;
        while (true) {
					
            ReadResult result = download_read(download, buffer, buffer_size);

            downloaded_total += result.bytes_read;
			
	    outfile.write(buffer, buffer_size);


            if (result.error) {
                if (result.error->code == EOF) {
                    free_read_result(result);
                    break;
                }
                printf("download failed to read: %s", result.error->message);
                free_read_result(result);
                return;
            }
            free_read_result(result);
        }

        Error *close_error = close_download(download);
        if (close_error) {
            printf("download failed to close: %s", close_error->message);
            free_error(close_error);
        }
        free_download_result(download_result);
}

There are some issues with this code:
File is downloaded but the file size is more than actual file size on Storj.
Also, I am unable to open the downloaded file.

It seems I have missed some point somewhere.
Kindly correct

Here you should use outfile.write(buffer, result.bytes_read);. During downloading, it sometimes can return immediately from an internal buffer, however that may not always fill the buffer fully… so sometimes it may return less data than the full buffer size.

Thanks! It worked :smiley:

One more question:

I am facing similar issue while uploading an object.
The object is uploaded, but the file size on Storj is more than that of actual file size on local system.

My code for upload is:

extern "C" void fv_uploadObject(Project *project, std::string object_key, std::string file, std::string objectName)
{
	std::ifstream is (file, std::ifstream::binary);
  
    // get length of file:
    is.seekg (0, is.end);
	size_t length = is.tellg();
    is.seekg (0, is.beg);
	
	size_t buffer_size = 32768;
    char *buffer =static_cast<char*>(malloc(buffer_size));
    
    UploadResult upload_result = upload_object(project, const_cast<char*>(object_key.c_str()), const_cast<char*>(objectName.c_str()), NULL);
    
    require_noerror(upload_result.error);
    require(upload_result.upload->_handle != 0);

    Upload *upload = upload_result.upload;

    size_t uploaded_total = 0;

    while (uploaded_total < length) {

		is.read (buffer, buffer_size);
		is.close();

		WriteResult result = upload_write(upload, buffer, buffer_size);
		
               uploaded_total += result.bytes_written;
        
		require_noerror(result.error);
                require(result.bytes_written > 0);
                free_write_result(result);
    }

    Error *commit_err = upload_commit(upload);
    require_noerror(commit_err);

    free_upload_result(upload_result);
}

Kindly correct
Thanks

It’s the same issue I think. http://www.cplusplus.com/reference/istream/istream/read/ is not guaranteed to fill the buffer. You need to use is.gcount() to get the actual number of bytes you read.

Thanks once again! :smiley:
Just one last doubt!

Using the above functions which I had written, I am now able to successfully download as well as upload objects from/to Storj

However, there is 1 discrepancy:

I am able to successfully upload any object with whitespace in its name.
(For example, object name on my local system is E:\Downloads\A B C.txt)

But, while downloading the same object(A B C.txt), I get an error:

download starting failed: uplink: object not found("A")

So, my question is whether Storj allows whitespaces in object names.
If it does not allow, then why is such error displayed while download, but not at the time of upload?

1 Like

I rechecked that & found out that my code uploads a file with whitespace in its name to Storj, but that file doesn’t list up while listing the objects.

So, I would like to know whether Storj allows download/upload of file with whitespace in its name?

Yes, the Storj network definitely allows files with whitespace in the name. In fact, it allows basically any byte sequence for object names, because the object names are encrypted before they even leave the uplink.

How are you trying to list the objects? What results do you see and what did you expect?

@thepaul

I re-checked my download & upload code multiple times, so it was correct.

On further check, I realised that in case of objects with whitespace in the name, the object name was incorrectly trimmed after the first whitespace & only the objectname part upto first whitespace was getting passed as an argument to the download function.

Resolved that with some string functions.

Thanks everyone!
(I wish every answer here could be marked as the solution!)