Why is the storj segment size only 64 MB?

Even torrents have several segment sizes according to powers of 2. (e.g. from a few kb to a few MB).
File storage costs can tolerate an error of 10%, but not 100 times more.
64MB is too large for images, audio, html, text files, etc., and is not suitable for hosting most static websites and web apps.
The only suitable application in practical terms is video.
Why is the storj segment size only 64MB?

2 Likes

This is a recommended size for segments, it works best for Storj. However, you can use any segment size, the network will use less or equal 64MB segments under the hood.
But using smaller segments will work slowly and incur additional costs for number of segments, see Usage Limit Increases | Storj Docs

3 Likes

As a storage node, I would prefer to see a bigger segment size. We currently have about 14K nodes or so. Let’s say 15K. Some customers are uploading big files with a size of 100GB or more. With a segment size of 64MB, we are talking about 1600 segments. Each segment is stored on 80 nodes. That means the network is storing 128K pieces. So each storage node is storing around 10 pieces of the same file. At that point, we store 10 times the metadata and open 10 times more connections for no good reason. Ideally, each storage node gets just one big piece instead of 10 small pieces. That would be much healthier for basically everyone in the network and that means better performance for everyone in the network.

If we ever have time for it we might implement a dynamic segment size based on the file size. For now, the 64 MB segment size seems to be a good compromise. I don’t expect a lower segment size. For a 5 MB picture, there is no reason to create 15K pieces out of it to max out the available bandwidth of the entire network. The bookkeeping for that would take longer than just a single 5 MB segment. A segment size of 64 MB means every file that is smaller than 64 MB is just a single segment.

6 Likes

I understand the reason for the segment minimum.
I have an additional question, if it is less than 64MB, it is chunked, but is it actually stored as 64MB of data?

2 Likes

No, anything below 64MB is just stored as is. 1MB will just be a 1MB segment which is erasure coded and stored spread over up to 80 nodes.

3 Likes

so many small files…
so so many…

1 Like

You can see here that the recent average size of a segment is around 13 MB only. This is how customers upload their files, Storj doesn’t require any minimum size.

2 Likes

Understood.
But wouldn’t it be fairer to charge separately for storage space and number of requests? (Same as s3)

1 Like

Why don’t we expose the setting in uplink config or upload options?

We charge for storage, egress and number of segments (AWS charges for storage, egress, number of requests, for multizone and for storage class). Smaller segments would be more expensive for the network and thus for you.

1 Like

In fact, it is often the case that files are not referenced much after they are saved.
Is it reasonable to pay a monthly segment fee when they are not referenced?
If the requests are overloading the network, then we should charge by the number of requests.
Users are accustomed to s3 compliant billing.