I am creating a service for data collection from diverse stock/crypto martkets based on Storj.
The data structure is quite easy:
bucket = some specific asset (e.g. Bitcoin), which will include various 1 level folders (exchanges) and finally the minute ohlc charts.
To be more precise, this is an examplary structure:
myProject → bitcoin (bucket) → binance (folder) → bitcoin_1min.csv (file)
myProject → bitcoin (bucket) → kraken (folder) → bitcoin_1min.csv (file)
To be flexible in terms of scalability I have to create folders on demand (e.g. further exchange added) and do this programmatically from the application.
I found the CreateBucket() function which does exactly what I need to do one layer lower Is there any CreateFolder() function with a similar functionality? Or alternatively, how do I include a bucket in a bucket? The trick with multiple buckets aka “bitcoin_binance” and “bitcoin_kraken” can be ofc used, but I would like to avoid this dirty style
In principle, there are no folders in Storj. The concept is that you only have object keys delimited by slashes. It’s a regular key value store, with some helpers for handling hierarchical data.
In other words, there’s no need to create a folder. You can upload to “binance/bitcoin_1min.csv” directly without the so called directory.
Sorry for my late response, yesterday I was absolutely tired and wanted to try something, but insead I went to sleep)))
I was a bit confused by the “create folder” button in Storj web client. That’s why I started to look for the options, how to preserve the folders structure. But with keys, which is basically the trick with “kraken_bitcoin_1min.csv” I can achieve my goal.
However I ran into the other issue with the web client, which does not dosplay for some reasons the folders and keys, which I am creating programmatically. I see them in my GO client, but they dissapear on the web. I will clear now the cache with cookies (chrome) and give you my feedback. Otherwise it will be a bit blind development with lots of try and errors, as I simply don’t know yet how the Storj handles the data.
Nope, the issue with created folders is still present even after clearing the browser data.
Regardless this presentation issue, could you please explain me, how the keys, e.g. binance/bitcoin_1min.csv will be displayed in web version? I am using the UploadObject() function now →
project.UploadObject(ctx, “abc, “abc”+”/test", nil)
So my expectation for the web is, that I see there “abc” bucket and inside it “test” directory. Since I do not see this in web but only the new bucket which was created, I might have an undertanding problem
First issue with my trouble found → I was using a wrong UploadKey, firstly with the name of the bucket, afterwards with a slash in the beginning. So I am evolutioning)))
Ah, yes, in the object browser it’s harder to manage the hierarchical structure, so it’s less confusing to call it “folder” there. Which is somewhat unfortunate.
For that, that upload would be that you see a “folder” called abc and then an empty “file” test inside that folder. Or in other words, the code tells to start uploading an object with key abc/test.
I’m not sure whether your code is doing it, because it’s a small code snippet only, but the code should call Commit and check the errors etc. If the Commit is missing, then the server assumes you are still uploading the object.
Yes, it’s possible to use multiple encryption keys per bucket. This is partly because on the server side there’s no idea whether the code is using the same encryption key or different. Also there were few users who wanted to use different keys per prefix.
As a consequence, in the object browser you’ll only see the objects that have been encrypted with the same key.
As a test for whether the data is there, you can use the api to list the objects recursively instead of using the object browser.
I’m not sure whether your code is doing it, because it’s a small code snippet only, but the code should call Commit and check the errors etc. If the Commit is missing, then the server assumes you are still uploading the object.
Yeah, I figured yesterday out that part with commit))) Was fun to realize, that something is missing after my attempts to upload the files
So I slowly getting to the point with the whole structure in your project. Sorry multiple times for probably dumb questions!
Could you please supervise me on listing the uploaded files.
I have a 1 level key structure: bucket → key/filename1…x
How can I access the uploaded files there?
This is my attempt:
for buckets.Next() {
objects := project.ListObjects(ctx, buckets.Item().Name, nil)
for objects.Next() {
item := objects.Item()
fmt.Println(item.Key, item.IsPrefix, buckets.Item()) // here I try to get all files stored in "bucket/key/
}
}
Despite uploading multiple files, by printing I receive only a single entry, e.g.
I found the explanation for it → that’s the definition of Bucket struct in uplink lib So bucket class/struct is basically a dummy with only informational attributes. I was looking for it in a wrong uplink directory (the one in the main Storj repository) and wondering, where are all the attributes)))
@Egon@bre
I will close this thread, since the initial question got solved. My other questions related to listing the items within a bucket I’ll post in a new thread. TY
By default ListObjects lists non-recursively. Depending on your use-case you may want to do a recursive listing, which basically lists all objects in the bucket. For example:
That’s an ephemeral entry for the prefix. They aren’t actually stored anywhere. This “pseudo-entry” is needed to support hierarchical navigation of the objects.
When you use “Recursive: true” then those won’t be shown since they aren’t needed.
Forum is probably a good place to ask such questions. It means other people can find them. Discord and other chats are less discoverable.
This is just brilliant! Everything works like charm! So now it’s time for me to create a proper structure, deploy to cloud an not be worried anymore about the data storage, running scrips and so on! Yeah!
P.S. I created an explicit thread about iterating over folders → I will link now this answer there. TY again!
Is there any CreateFolder() function with a similar functionality? Or alternatively, how do I include a bucket in a bucket? The trick with multiple buckets aka “bitcoin_binance” and “bitcoin_kraken” can be ofc used, but I would like to avoid this dirty style 
