As I understand it, small files are stored on Storj servers, because they are to small to be spread into pieces across the public network.
Can’t they be gattered together in some bigger files, like the logs we are using in hashstore, and spread those in pieces across the network?
This way, the central servers are freed from them, and the safety of the data is assured in the same way as with big files.
Is it a possibility?
Look at it this way:
Small fiels are stored inline because overhead of fetching small amount of data from nodes is way too expensive; latency is much higher than time needed to transfer data.
You are suggesting to add time needed to download large log and unpack that small file to the process. it will be significantly worse.
I’d argue satellites are teh safest place to keep files. If Node dies – nothign happens. If satellite dies – everythign happens.
Satellites are redudand and distributed just like (if not better than) nodes.
3 Likes
Isn’t that suggestion what is in the roadmap?
2 Likes
Ouh, didn’t saw that… or maybe I did, and a brain elf reminded me somehow… 
They can. But depends on the customer. Our link sharing service allows to have access to a single file from the zip archive and it’s pretty fast.
So, the main suggestion for the customers with a lot of small files to pack them to a zip archive and upload, then use them in their website for example without any issues and struggling.
The linked GitHub roadmap ticket is to do so automagically.
1 Like
I was thinking only about the Storj side, not customer’s. Like… the software could pack small files from different customers, not only one, and process those bigger files.
But AR pointed out the problems…
1 Like
No, it would become much worse. The linked ticket suggest to pack such small files with their metadata. Not sure that it’s possible to implement, though.
Might be acceptable trade-off for some customers, so that they do not need to pay excessive per-segment fee.
This per-segment fee is introduced not only to cover costs of inline segments but also to prevent the abuse: you may store a million of small objects with almost zero size, where metadata will be still greater than the object, or upload a 1kB object with 1 byte chunks, etc.
2 Likes