I imagine that a solution will be made to organize files in different maneer
i will try to do my best here in order to help you at the maxmum, dear Storj
Tiered Architecture of files in storage nodes
Tiered architecture consist of 3 layers of way in the files level :
Most frequently accessed Files : SSD ONLY
Frequently Accessed Files : ALL OTHER STORAGE CLASS
Unfrequently Accessed Files : ALL OTHER STORAGE CLASS ZIPPED
Architecture of files inside the Filesystem
Run a IOPS Test on container launching to determine performance of drives
Organize a blob called SSD on nodes with SSD and move files into (which are the most frequently accessed)
IOPS xor Bandwidth throughput may not be victim of a bottleneck
Zip files on your servers or on compute nodes in functions then upload large zip files
For the frequently accessed files on ALL OTHER STORAGE CLASSES keep your file structure and organization intact
Feel free to contact me in case of needs as volunteer.
A QNAP could do this for you out of the box. But the diskspeed will not your problem if all lockings are optimized. The disk will be faster than your internetconnection.
Also your node stores encrypted blobs. That means it’s random noise that can’t be compressed.
My node is already using an SSD cache. But this hardly helps with the actual file transfers. It does seem to help with the db operations, but as @BlackDuck mentioned, those can be further optimized.
Maybe largest files (Zip) would permit a better network bandwidth, i dont speak about compression of encrypted files but on packing themself, plus if you put ssd blob called ssd and you put only most frequently accessed files on it then restrict this usage on ssd, ssd would increase the network bandwidth, coupled to a 1Gb connection, maybe this blob will be faster than others to deliver content to users.
The feature i want the must is upload of largest files, (may increase the network troughput?)
On the topic of faster storage: imagine a network situation where there are 100 nodes with spinning hard drives but which are in your immediate neighborhood. Then imagine another 100 nodes with SSDs on the exact opposite side of the world. If your data can reach you faster by way of the spinning hard drive nodes, wouldn’t you prefer to get it from them, even though the SSD machines have faster storage hardware?
This is why the V3 network starts to download from many more nodes than necessary, and stops after you get enough pieces from the fastest ones (whatever that means for you)! This, plus the planned feature of hot-file caching, should provide lots more benefit than an SSD-only storage class.
As for compressing things, that would of course need to happen in the uplink, before the pieces are sent out to the Storj network, so from the Storj network point of view, it’s exactly the same thing (so there is no need for a special storage class). It could just be implemented as some special attribute that the uplink recognizes when encoding or decoding. Or, it could be implemented a higher layer still. I agree that in a few cases, this would make things faster, and maybe even cost less- so it’s likely to be implemented at some point.
This is not correct, on-fly compression will need more CPU resources but will save Network I/O. Most storage systems will have a benefit with compression, because in most cases CPU resources are not the limiting faktor. ZFS, LTO tapebackups, Arcserve Backup, Veeam or duplicity just to name some will gain performance with compression enabled.
In the case from storj the compression can only done on client/uplink side, the data on SN is already encrypted and therefore not compressable.
@Eioz compression does not work on SN side. Each file getting splitted in 64MB segments, every segment get splitted with redundancy in 130 pieces, the upload stops when 80 pieces are in the network. If you download a file <64 MB then you will do this from 29 out of 80 nodes. 29 streams are fast even if alle nodes are limited to 5Mbit in upload your download speed is 5 * 29.
The tiering you are requesting is a build in feature in QNAP Nas systems. They support auto-tiering out of the box. And this can be extended with ssd cache
Don’t think on compress encrypted files, this is not possible to have a good compression factor due to the files encryption.
but, i see that the network throughtput actually is at 2MB/s, because pieces are 2MB weight and uploaded each second ? i have a 37,5MB/s upload link so, if we receive us SNO larger files it will speed up the upload. for me 37,5MB/file/seconds. please split files unused arround 20MB to begin…
Maybe with largest files the download speed on SNO (put option) may be more fast
But i recognize that larger files would be more difficult to reorganize.
So maybe devellop storage tiers to store pieces that are unused into piece larger than 2MB
audit get events
when file isnt “get” move it to “blob called glacier”
compute largest part files “zip” on satelitte (not compressing, just do single parts!!!)
then largest files are archived to glacier blob when user need satelitte decompress glacier archive and delete then put or move file from glacier to normal blob.
i suggest to use satelitte as compute node
also maybe it is more fast to move disk head on one sector location
not on all… try to maximize piece size and reduce number of request may fast the network) actually it’s like put speed is at 2MB/s max
Ahh I understand what you want to say. First the pieces are not fixed to 2MB, the piecesize depends on the minimum number ob pieces (29 per default) and the choosen segmentsize (64Mb per default). Right now uploads are done from devs with default settings, that is the reason that there right now only 2MB pieces, but everything else is already implemented and can be used from the customer.
Second, larger pieces will not result in more throughput, it will not speed up the upload for you, because the limit is on uplink side. As you said the upload (put) speed is 2MB/s the uplink has to do this 80 times simultan for one file. And the test nodes uploading multiple files at once. The line on uplink side is burning at the limit.
You will see more line usage as more customers join the network.
We are near. Yes the limit ist on uploader side. But the data did not pass the satellite. The satellite is only there for accounting. Every data is peer-to-peer between SN and uplink. There only a handfull of devs with an account (this will change over next week). The mass of data you are getting right now are send from storj admins to test performance and search for bottlenecks.
There is no need to stage data, it wents fast as possible from customer to SN. As more customer join and use storj, as more you will get saturated you line. There is no balance on both sides.
A handfull upload servers try to feed near two thousand SN, that is the reason why it is impossible to fill your line.
