Architecture of files

Hello Storj Team,

I imagine that a solution will be made to organize files in different maneer
i will try to do my best here in order to help you at the maxmum, dear Storj

  1. Tiered Architecture of files in storage nodes

Tiered architecture consist of 3 layers of way in the files level :

  • Most frequently accessed Files : SSD ONLY
  • Frequently Accessed Files : ALL OTHER STORAGE CLASS
  • Unfrequently Accessed Files : ALL OTHER STORAGE CLASS ZIPPED
  1. Architecture of files inside the Filesystem
  • Run a IOPS Test on container launching to determine performance of drives

  • Organize a blob called SSD on nodes with SSD and move files into (which are the most frequently accessed)

  • IOPS xor Bandwidth throughput may not be victim of a bottleneck

  • Zip files on your servers or on compute nodes in functions then upload large zip files

  • For the frequently accessed files on ALL OTHER STORAGE CLASSES keep your file structure and organization intact

Feel free to contact me in case of needs as volunteer.


1 Like

A QNAP could do this for you out of the box. But the diskspeed will not your problem if all lockings are optimized. The disk will be faster than your internetconnection.

Also your node stores encrypted blobs. That means it’s random noise that can’t be compressed.

My node is already using an SSD cache. But this hardly helps with the actual file transfers. It does seem to help with the db operations, but as @BlackDuck mentioned, those can be further optimized.

1 Like

Maybe largest files (Zip) would permit a better network bandwidth, i dont speak about compression of encrypted files but on packing themself, plus if you put ssd blob called ssd and you put only most frequently accessed files on it then restrict this usage on ssd, ssd would increase the network bandwidth, coupled to a 1Gb connection, maybe this blob will be faster than others to deliver content to users.

The feature i want the must is upload of largest files, (may increase the network troughput?)

The on-fly compression will reduce the speed. We want to be faster than Amazon S3, not slower


On the topic of faster storage: imagine a network situation where there are 100 nodes with spinning hard drives but which are in your immediate neighborhood. Then imagine another 100 nodes with SSDs on the exact opposite side of the world. If your data can reach you faster by way of the spinning hard drive nodes, wouldn’t you prefer to get it from them, even though the SSD machines have faster storage hardware?

This is why the V3 network starts to download from many more nodes than necessary, and stops after you get enough pieces from the fastest ones (whatever that means for you)! This, plus the planned feature of hot-file caching, should provide lots more benefit than an SSD-only storage class.

As for compressing things, that would of course need to happen in the uplink, before the pieces are sent out to the Storj network, so from the Storj network point of view, it’s exactly the same thing (so there is no need for a special storage class). It could just be implemented as some special attribute that the uplink recognizes when encoding or decoding. Or, it could be implemented a higher layer still. I agree that in a few cases, this would make things faster, and maybe even cost less- so it’s likely to be implemented at some point.

1 Like

This is not correct, on-fly compression will need more CPU resources but will save Network I/O. Most storage systems will have a benefit with compression, because in most cases CPU resources are not the limiting faktor. ZFS, LTO tapebackups, Arcserve Backup, Veeam or duplicity just to name some will gain performance with compression enabled.

In the case from storj the compression can only done on client/uplink side, the data on SN is already encrypted and therefore not compressable.

@Eioz compression does not work on SN side. Each file getting splitted in 64MB segments, every segment get splitted with redundancy in 130 pieces, the upload stops when 80 pieces are in the network. If you download a file <64 MB then you will do this from 29 out of 80 nodes. 29 streams are fast even if alle nodes are limited to 5Mbit in upload your download speed is 5 * 29.

The tiering you are requesting is a build in feature in QNAP Nas systems. They support auto-tiering out of the box. And this can be extended with ssd cache

1 Like

Don’t think on compress encrypted files, this is not possible to have a good compression factor due to the files encryption.
but, i see that the network throughtput actually is at 2MB/s, because pieces are 2MB weight and uploaded each second ? i have a 37,5MB/s upload link so, if we receive us SNO larger files it will speed up the upload. for me 37,5MB/file/seconds. please split files unused arround 20MB to begin… :wink:

Maybe with largest files the download speed on SNO (put option) may be more fast
But i recognize that larger files would be more difficult to reorganize.
So maybe devellop storage tiers to store pieces that are unused into piece larger than 2MB

audit get events
when file isnt “get” move it to “blob called glacier”
compute largest part files “zip” on satelitte (not compressing, just do single parts!!!)
then largest files are archived to glacier blob when user need satelitte decompress glacier archive and delete then put or move file from glacier to normal blob.

i suggest to use satelitte as compute node

also maybe it is more fast to move disk head on one sector location
not on all… try to maximize piece size and reduce number of request may fast the network) actually it’s like put speed is at 2MB/s max :wink:

Ahh I understand what you want to say. First the pieces are not fixed to 2MB, the piecesize depends on the minimum number ob pieces (29 per default) and the choosen segmentsize (64Mb per default). Right now uploads are done from devs with default settings, that is the reason that there right now only 2MB pieces, but everything else is already implemented and can be used from the customer.

Second, larger pieces will not result in more throughput, it will not speed up the upload for you, because the limit is on uplink side. As you said the upload (put) speed is 2MB/s the uplink has to do this 80 times simultan for one file. And the test nodes uploading multiple files at once. The line on uplink side is burning at the limit.

You will see more line usage as more customers join the network.

1 Like

Ok so you guys have the limit of 2MB/s x 80 so 160MB/s = 1280Mb uplink ?
if i understand, the limit is at the uploader side ? (storj devs?)

do you have thinked on stage files on multiple primary servers (satelittes) then upload them with 10GBe internet uplink ?

Thanks for the informations @BlackDuck

maybe those limits would increase at the customers venue, i understand links are costly :wink:

We are near. Yes the limit ist on uploader side. But the data did not pass the satellite. The satellite is only there for accounting. Every data is peer-to-peer between SN and uplink. There only a handfull of devs with an account (this will change over next week). The mass of data you are getting right now are send from storj admins to test performance and search for bottlenecks.

There is no need to stage data, it wents fast as possible from customer to SN. As more customer join and use storj, as more you will get saturated you line. There is no balance on both sides.

A handfull upload servers try to feed near two thousand SN, that is the reason why it is impossible to fill your line.


Ok, so without intermediary beetween SN and uploader link, i understand now that the bandwidth will increase soon, thanks for the answer @BlackDuck.

if pieces can be at 64MB max weight so the network bandwidth will automatically grow :wink:
if needed you can use my uplink too :wink:

Nice week-end to all storj forum.