HashBackup performance test: Storj, B2, Wasabi, iDrive, S3

[hashbackup]

It’s been about 5 years since I last ran performance tests on Storj. I’ve read about many changes, quite often performance related, and thought it would be interesting to run tests again.

These tests are run from my home (Midwest), with 100/100 cable service. My ISP could be throttling service, so I don’t put much faith in the absolute numbers, only the relative difference. For Storj, the most recent uplink binary is used without -p. The 3 S3 services use the boto Python library to transfer files, and B2 uses the native B2 API. All transfers are single-threaded, ie, multipart is disabled. If anyone else wants to do the tests, let me know and I’ll post the HB dest.conf file I used.

$ hb dest -c sj test -s 4k 64m
HashBackup #3357 Copyright 2009-2026 HashBackup, LLC
Using destinations in dest.conf
Warning: destination is disabled: sjs3
Warning: destination is disabled: b2s3
CPU count: 4

2026-08-18 15:07:05 --- Testing sj, type shell, 4 workers, 1 part ---

  4 KiB: 
    Round  1  Up 405.2ms   9.9 KiB/s   Down 470.4ms   8.5 KiB/s   Del 471.5ms   8.5 KiB/s  
    Round  2  Up 422.4ms   9.5 KiB/s   Down 349.2ms  11.5 KiB/s   Del 410.5ms   9.7 KiB/s  
    Round  3  Up 445.6ms   9.0 KiB/s   Down 380.4ms  10.5 KiB/s   Del 393.7ms  10.2 KiB/s  
  > Average   Up 424.4ms   9.4 KiB/s   Down 400.0ms  10.0 KiB/s   Del 425.2ms   9.4 KiB/s  

  64 MiB: 
    Round  1  Up  36.0s    1.8 MiB/s   Down   9.4s    6.8 MiB/s   Del 388.9ms 164.6 MiB/s  
    Round  2  Up  38.4s    1.7 MiB/s   Down   9.5s    6.7 MiB/s   Del 403.3ms 158.7 MiB/s  
    Round  3  Up  36.8s    1.7 MiB/s   Down   8.9s    7.2 MiB/s   Del 631.8ms 101.3 MiB/s  
  > Average   Up  37.1s    1.7 MiB/s   Down   9.3s    6.9 MiB/s   Del 474.7ms 134.8 MiB/s  


2026-08-18 15:09:30 --- Testing b2, type b2, 4 workers, 1 part ---

  4 KiB: 
    Round  1  Up   1.4s    2.8 KiB/s   Down 393.9ms  10.2 KiB/s   Del 209.0ms  19.1 KiB/s  
    Round  2  Up 919.9ms   4.3 KiB/s   Down 101.4ms  39.5 KiB/s   Del 207.4ms  19.3 KiB/s  
    Round  3  Up 899.2ms   4.4 KiB/s   Down 103.3ms  38.7 KiB/s   Del 197.9ms  20.2 KiB/s  
  > Average   Up   1.1s    3.7 KiB/s   Down 199.5ms  20.0 KiB/s   Del 204.8ms  19.5 KiB/s  

  64 MiB: 
    Round  1  Up  15.0s    4.3 MiB/s   Down   7.2s    8.9 MiB/s   Del 491.9ms 130.1 MiB/s  
    Round  2  Up  14.6s    4.4 MiB/s   Down   6.8s    9.4 MiB/s   Del 513.7ms 124.6 MiB/s  
    Round  3  Up  36.1s    1.8 MiB/s   Down   7.0s    9.2 MiB/s   Del 580.7ms 110.2 MiB/s  
  > Average   Up  21.9s    2.9 MiB/s   Down   7.0s    9.2 MiB/s   Del 528.8ms 121.0 MiB/s  


2026-08-18 15:11:04 --- Testing wasabi, type s3, 4 workers, s3.us-east-2.wasabisys.com, 1 part ---

  4 KiB: 
    Round  1  Up 309.5ms  12.9 KiB/s   Down  45.4ms  88.1 KiB/s   Del  55.9ms  71.5 KiB/s  
    Round  2  Up  70.8ms  56.5 KiB/s   Down  43.9ms  91.0 KiB/s   Del  60.2ms  66.5 KiB/s  
    Round  3  Up  66.3ms  60.3 KiB/s   Down  57.1ms  70.1 KiB/s   Del  62.5ms  64.0 KiB/s  
  > Average   Up 148.9ms  26.9 KiB/s   Down  48.8ms  82.0 KiB/s   Del  59.5ms  67.2 KiB/s  

  64 MiB: 
    Round  1  Up  13.4s    4.8 MiB/s   Down   8.4s    7.6 MiB/s   Del 221.5ms 288.9 MiB/s  
    Round  2  Up  14.1s    4.5 MiB/s   Down   7.6s    8.4 MiB/s   Del 237.2ms 269.8 MiB/s  
    Round  3  Up  13.2s    4.9 MiB/s   Down   8.2s    7.8 MiB/s   Del 226.9ms 282.1 MiB/s  
  > Average   Up  13.5s    4.7 MiB/s   Down   8.1s    7.9 MiB/s   Del 228.5ms 280.0 MiB/s  


2026-08-18 15:12:11 --- Testing ide2, type s3, 4 workers, z8i9.ch31.idrivee2-8.com, 1 part ---

  4 KiB: 
    Round  1  Up 402.1ms   9.9 KiB/s   Down  51.5ms  77.7 KiB/s   Del  46.9ms  85.3 KiB/s  
    Round  2  Up  55.6ms  72.0 KiB/s   Down  53.7ms  74.5 KiB/s   Del  48.3ms  82.8 KiB/s  
    Round  3  Up  52.1ms  76.8 KiB/s   Down  52.5ms  76.2 KiB/s   Del  46.5ms  86.1 KiB/s  
  > Average   Up 169.9ms  23.5 KiB/s   Down  52.6ms  76.1 KiB/s   Del  47.2ms  84.7 KiB/s  

  64 MiB: 
    Round  1  Up  12.9s    5.0 MiB/s   Down   8.4s    7.6 MiB/s   Del 218.6ms 292.8 MiB/s  
    Round  2  Up  13.4s    4.8 MiB/s   Down   8.5s    7.5 MiB/s   Del 213.0ms 300.5 MiB/s  
    Round  3  Up  13.6s    4.7 MiB/s   Down   8.7s    7.4 MiB/s   Del 211.7ms 302.3 MiB/s  
  > Average   Up  13.3s    4.8 MiB/s   Down   8.5s    7.5 MiB/s   Del 214.4ms 298.5 MiB/s  


2026-08-18 15:13:18 --- Testing s3, type s3, 4 workers, s3-us-east-2.amazonaws.com, 1 part ---

  4 KiB: 
    Round  1  Up 169.5ms  23.6 KiB/s   Down  52.4ms  76.4 KiB/s   Del  46.0ms  87.0 KiB/s  
    Round  2  Up  55.5ms  72.0 KiB/s   Down  57.9ms  69.1 KiB/s   Del  53.7ms  74.4 KiB/s  
    Round  3  Up  49.9ms  80.2 KiB/s   Down  58.6ms  68.3 KiB/s   Del  50.0ms  80.0 KiB/s  
  > Average   Up  91.7ms  43.6 KiB/s   Down  56.3ms  71.1 KiB/s   Del  49.9ms  80.2 KiB/s  

  64 MiB: 
    Round  1  Up  13.3s    4.8 MiB/s   Down   7.5s    8.5 MiB/s   Del 156.7ms 408.5 MiB/s  
    Round  2  Up  13.7s    4.7 MiB/s   Down   7.1s    9.0 MiB/s   Del 160.7ms 398.3 MiB/s  
    Round  3  Up  14.3s    4.5 MiB/s   Down   6.6s    9.7 MiB/s   Del 161.3ms 396.8 MiB/s  
  > Average   Up  13.7s    4.7 MiB/s   Down   7.1s    9.0 MiB/s   Del 159.5ms 401.2 MiB/s  

Test complete

I tried doing a 64MB uplink upload with -p 4 but it was a few seconds slower, but -p 4 might only work on a file with multiple 64MB segments.

I wanted to test Storj and Backblaze via S3, but HashBackup is having trouble connecting to them for some reason.

[Alexey]

For upload and limited internet channel (less than 1Gbps) it’s better to use S3, for downloads - native/S3 are on par. The difference become noticeable starting from 250mbps. And especially with multi thread. Single thread upload/download wouldn’t be so fast for neither of them.

See

Hotrodding Decentralized Storage

Preface Greetings everyone. We have been working on some exciting performance enhancements and wanted to share them with the community and solicit feedback. I hope you have as much fun with this as I have had! Before we commit this to our documentation we’d like to get your feedback and questions. -Dominick Storj DCS Performance Overview Performance Basics In decentralized storage, the concept of parallelism has a huge effect on performance. As you will read below we can enact parallel effort…

[DutchSNO]

I gave it a try on a new installed VM. Just not entirely sure how to compare both tests.
My dest file is very straight forward, after the obvious connection settings I had to add:

secure true

My results with a 1gbit up/down fiber connection:

hb dest -c hashbackup test -s 4k 64m
HashBackup #3336 Copyright 2009-2026 HashBackup, LLC
Using destinations in database
CPU count: 4

2026-08-19 16:21:14 --- Testing storj, type s3, 4 workers, gateway.storjshare.io, var partsize ---

  4 KiB: 1 part
    Round  1  Up  50.9ms  78.6 KiB/s   Down  26.8ms 149.3 KiB/s   Del  34.7ms 115.1 KiB/s
    Round  2  Up  46.4ms  86.2 KiB/s   Down  24.5ms 163.5 KiB/s   Del  27.6ms 144.8 KiB/s
    Round  3  Up  58.7ms  68.1 KiB/s   Down  26.3ms 152.0 KiB/s   Del  25.2ms 159.0 KiB/s
  > Average   Up  52.0ms  76.9 KiB/s   Down  25.9ms 154.7 KiB/s   Del  29.2ms 137.1 KiB/s

  64 MiB: 4 parts @ 16 MiB
    Round  1  Up   1.6s   41.1 MiB/s   Down   1.0s   63.1 MiB/s   Del  72.9ms 878.4 MiB/s
    Round  2  Up   1.5s   43.1 MiB/s   Down 925.4ms  69.2 MiB/s   Del  66.1ms 968.5 MiB/s
    Round  3  Up   1.3s   49.7 MiB/s   Down 929.6ms  68.8 MiB/s   Del  72.0ms 889.1 MiB/s
  > Average   Up   1.4s   44.4 MiB/s   Down 956.6ms  66.9 MiB/s   Del  70.3ms 910.3 MiB/s

Test complete

If you would like me to test something else, just let me know. I’ll run the test and post the results asap.

[hashbackup]

Hey, cool! Do you have S3 credentials to compare it? How does it compare with using uplink?

I tried again and did get Storj+S3 working here. I’m going to try to run some tests today on a VPS with higher bandwidth. I usually use Vultr for that, but they decided to add a Google captcha to my login page so I have to identify bikes, motorcycles, traffic lights, etc. Ain’t gonna happen, so I’m in the process of cancelling my account. I’ll try it on Linode; that’s where I host the HB website.

If you want to try bigger files, you can add -s to the test, like -s 1gb. There’s also a partsize option in dest.conf, so you can set the Storj segment size, like partsize 64m.

Jim

[DutchSNO]

Currently I don’t have other S3 credentials, just Storj.
For uplink I have to do some research how to get this to work. But that’s for an other time, maybe upcoming weekend (depends on the weather).

I did however run some additional tests

1. default config with 1gb test
2. 1gb with 64mb partsize
3. 1gb with 64mb partsize and multipart true

I don’t think multipart did anything, or it just doesn’t work as it think it would work

The results so far, and for today…

hb dest -c hashbackup test -s 4k 64m 1GB
HashBackup #3336 Copyright 2009-2026 HashBackup, LLC
Using destinations in database
CPU count: 4

2026-08-19 17:32:09 --- Testing storj, type s3, 4 workers, gateway.storjshare.io, var partsize ---

  4 KiB: 1 part
    Round  1  Up  46.9ms  85.2 KiB/s   Down  25.8ms 155.1 KiB/s   Del  27.5ms 145.7 KiB/s
    Round  2  Up  48.2ms  83.0 KiB/s   Down  26.4ms 151.7 KiB/s   Del  26.8ms 149.3 KiB/s
    Round  3  Up  47.9ms  83.5 KiB/s   Down  26.6ms 150.4 KiB/s   Del  27.9ms 143.4 KiB/s
  > Average   Up  47.7ms  83.9 KiB/s   Down  26.2ms 152.4 KiB/s   Del  27.4ms 146.1 KiB/s

  64 MiB: 4 parts @ 16 MiB
    Round  1  Up   1.2s   52.1 MiB/s   Down 945.5ms  67.7 MiB/s   Del  70.6ms 906.5 MiB/s
    Round  2  Up   1.3s   50.3 MiB/s   Down 885.4ms  72.3 MiB/s   Del  71.8ms 891.9 MiB/s
    Round  3  Up   1.4s   45.3 MiB/s   Down   1.0s   61.4 MiB/s   Del  71.1ms 900.5 MiB/s
  > Average   Up   1.3s   49.0 MiB/s   Down 957.8ms  66.8 MiB/s   Del  71.1ms 899.6 MiB/s

  1 GiB: 15 parts @ 68.266667 MiB
    Round  1  Up  12.6s   81.4 MiB/s   Down  12.0s   85.5 MiB/s   Del  74.2ms  13.5 GiB/s
    Round  2  Up  12.7s   80.6 MiB/s   Down  12.1s   84.8 MiB/s   Del  69.2ms  14.4 GiB/s
    Round  3  Up  12.4s   82.6 MiB/s   Down  12.4s   82.5 MiB/s   Del  79.1ms  12.6 GiB/s
  > Average   Up  12.6s   81.5 MiB/s   Down  12.2s   84.2 MiB/s   Del  74.2ms  13.5 GiB/s


2026-08-19 17:33:40 --- Testing storj-partsize_64m, type s3, 4 workers, gateway.storjshare.io, 64 MiB partsize ---

  4 KiB: 1 part
    Round  1  Up  87.8ms  45.6 KiB/s   Down  24.8ms 161.3 KiB/s   Del  27.6ms 144.7 KiB/s
    Round  2  Up  46.8ms  85.5 KiB/s   Down  25.9ms 154.6 KiB/s   Del  38.6ms 103.5 KiB/s
    Round  3  Up  45.7ms  87.6 KiB/s   Down  26.9ms 148.7 KiB/s   Del  25.4ms 157.8 KiB/s
  > Average   Up  60.1ms  66.6 KiB/s   Down  25.9ms 154.7 KiB/s   Del  30.5ms 131.0 KiB/s

  64 MiB: 1 part
    Round  1  Up   2.1s   30.7 MiB/s   Down   1.5s   43.1 MiB/s   Del  76.0ms 842.5 MiB/s
    Round  2  Up   1.4s   46.7 MiB/s   Down   1.6s   39.5 MiB/s   Del  78.7ms 813.7 MiB/s
    Round  3  Up   1.4s   44.2 MiB/s   Down   1.4s   46.6 MiB/s   Del  73.5ms 871.3 MiB/s
  > Average   Up   1.6s   39.2 MiB/s   Down   1.5s   42.9 MiB/s   Del  76.0ms 841.9 MiB/s

  1 GiB: 16 parts @ 64 MiB
    Round  1  Up  11.9s   86.3 MiB/s   Down  11.5s   89.1 MiB/s   Del  74.1ms  13.5 GiB/s
    Round  2  Up  11.7s   87.5 MiB/s   Down  11.5s   89.3 MiB/s   Del  69.3ms  14.4 GiB/s
    Round  3  Up  12.4s   82.2 MiB/s   Down  12.0s   85.5 MiB/s   Del  78.9ms  12.7 GiB/s
  > Average   Up  12.0s   85.3 MiB/s   Down  11.6s   88.0 MiB/s   Del  74.1ms  13.5 GiB/s


2026-08-19 17:35:11 --- Testing storj-partsize_64m-multipart_true, type s3, 4 workers, gateway.storjshare.io, 64 MiB partsize ---

  4 KiB: 1 part
    Round  1  Up  88.1ms  45.4 KiB/s   Down  26.1ms 153.5 KiB/s   Del  28.4ms 140.8 KiB/s
    Round  2  Up  60.0ms  66.7 KiB/s   Down  25.6ms 156.5 KiB/s   Del  26.8ms 149.1 KiB/s
    Round  3  Up  52.4ms  76.4 KiB/s   Down  25.6ms 156.5 KiB/s   Del  39.2ms 102.1 KiB/s
  > Average   Up  66.8ms  59.9 KiB/s   Down  25.7ms 155.5 KiB/s   Del  31.5ms 127.1 KiB/s

  64 MiB: 1 part
    Round  1  Up   1.3s   49.4 MiB/s   Down   1.5s   42.4 MiB/s   Del  73.8ms 867.3 MiB/s
    Round  2  Up   1.2s   52.9 MiB/s   Down   1.5s   42.2 MiB/s   Del  76.6ms 836.0 MiB/s
    Round  3  Up   2.6s   25.0 MiB/s   Down   1.5s   42.7 MiB/s   Del  71.7ms 893.2 MiB/s
  > Average   Up   1.7s   37.9 MiB/s   Down   1.5s   42.4 MiB/s   Del  74.0ms 864.9 MiB/s

  1 GiB: 16 parts @ 64 MiB
    Round  1  Up  12.2s   84.2 MiB/s   Down  12.1s   84.9 MiB/s   Del  64.8ms  15.4 GiB/s
    Round  2  Up  12.2s   84.1 MiB/s   Down  11.8s   87.0 MiB/s   Del 148.4ms   6.7 GiB/s
    Round  3  Up  12.5s   82.1 MiB/s   Down  11.7s   87.2 MiB/s   Del  69.7ms  14.4 GiB/s
  > Average   Up  12.3s   83.5 MiB/s   Down  11.9s   86.3 MiB/s   Del  94.3ms  10.6 GiB/s

Test complete

[DutchSNO]

Couldn’t let it go, I think that Storj S3 is multipart compatible according to this compatibility matrix. Did I do something wrong?
https://storj.dev/dcs/api/s3/s3-compatibility

[hashbackup]

multipart defaults to True if it isn’t in dest.conf; you have to use multipart false to disable it. Looks like all of your tests were with multipart true.

It’s recommended to use partsize 64m with Storj so that you are uploading full segments rather than letting HB figure out part sizes and doing “1 GiB: 15 parts @ 68.266667 MiB”, which is going to get you 30 segments on Storj, with the even segments getting charged extra because of the 50K minimum.

You could try adding “workers 8” up to 16 in dest.conf to see if more concurrency changes things. The default is 4 workers. Python isn’t great at multithreading, but the S3 destination uses multiprocessing, ie, multiple processes, not threads, and has better concurrency.

The way multipart works on the S3 end is that each part is a separate object. Amazon came up with this because objects are limited to 5GB. To store a file larger than that you have to use multipart uploads.

[DutchSNO]

Somehow I miss interpret the hashbackup documentation and thought multipart was default enabled for Google storage instead for S3 in general. Instead it is the other way around
I’m gonna use “it was a very long day” as a excuse

Later I’ll test more workers, I’m curious if this would max out mij ISP connection.

[DutchSNO]

Looks like something at Storj was freaking out?
Dont know if I hit an API limit or wanted to upload to fast. With 6 to 8 workers I max out my ISP.

Must say that I like how easy you can test S3 performance with Hashbackup.

2026-08-20 15:08:38 --- Testing storj-partsize_64m-workers_6, type s3, 6 workers, gateway.storjshare.io, 64 MiB partsize ---

  1 GiB: 16 parts @ 64 MiB
    Round  1  Up  10.8s   94.4 MiB/s   Down  10.2s  100.7 MiB/s   Del  74.0ms  13.5 GiB/s
    Round  2  Up  10.3s   99.6 MiB/s   Down  10.4s   98.7 MiB/s   Del  72.9ms  13.7 GiB/s
    Round  3  Up  10.4s   98.6 MiB/s   Down  10.5s   97.5 MiB/s   Del  74.3ms  13.5 GiB/s
  > Average   Up  10.5s   97.5 MiB/s   Down  10.3s   99.0 MiB/s   Del  73.7ms  13.6 GiB/s


2026-08-20 15:09:50 --- Testing storj-partsize_64m-workers_8, type s3, 8 workers, gateway.storjshare.io, 64 MiB partsize ---

  1 GiB: 16 parts @ 64 MiB
    Round  1  Up  10.7s   95.5 MiB/s   Down  10.3s   99.2 MiB/s   Del  71.9ms  13.9 GiB/s
    Round  2  Up  10.4s   98.2 MiB/s  Traceback (most recent call last):
  File "/s3dest.py", line 331, in _mpworker
  File "/s3dest.py", line 269, in getworker
  File "/opt/lib/python2.7/site-packages/boto/s3/key.py", line 1650, in get_contents_to_file
  File "/opt/lib/python2.7/site-packages/boto/s3/key.py", line 1482, in get_file
  File "/opt/lib/python2.7/site-packages/boto/s3/key.py", line 1514, in _get_file_internal
  File "/opt/lib/python2.7/site-packages/boto/s3/key.py", line 343, in open
  File "/opt/lib/python2.7/site-packages/boto/s3/key.py", line 303, in open_read
S3ResponseError: S3ResponseError: 429 Too Many Requests
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SlowDown</Code><Message>Please reduce your request rate.</Message><Key>storj-partsize_64m-workers_8/hbtest.tmp.2</Key><BucketName>hachbackup-bucket</BucketName><Resource>/storj-partsize_64m-workers_8/hbtest.tmp.2</Resource><RequestId>18CD8C10A4F70844</RequestId><HostId></HostId></Error>


Traceback (most recent call last):
  File "/hb.py", line 265, in <module>
  File "/destcmd.py", line 460, in main
  File "/destcmd.py", line 274, in dotest
  File "/s3dest.py", line 802, in getfile
  File "/s3dest.py", line 783, in _getmulti
  File "/s3dest.py", line 654, in getmpresult
S3ResponseError: S3ResponseError: 429 Too Many Requests
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SlowDown</Code><Message>Please reduce your request rate.</Message><Key>storj-partsize_64m-workers_8/hbtest.tmp.2</Key><BucketName>hachbackup-bucket</BucketName><Resource>/storj-partsize_64m-workers_8/hbtest.tmp.2</Resource><RequestId>18CD8C10A4F70844</RequestId><HostId></HostId></Error>

[hashbackup]

Cool!

There are other options that would let you test over a long period of time, like a day. For example:

-d 15m = run 3 tests and show average every 15 minutes
-r 1 = only run 1 round each time, not 3
-n 4 = run test 4 times (stops -d loop)

So this command:

hb dest -c hashbackup test -s 4k 1g -r 1 -d 10m -n 144

would measure one small and large file transfer every 10 minutes for 24 hours to see if performance is consistent.

[arrogantrabbit]

I see a number of issues with the methodology used to obtain results in this topic. To name a few:

• Three samples is nowhere near enough here, especially when the results themselves are noisy. B2 uploading 64 MiB takes 15.0, 14.6 and 36.1 seconds. Averaging those three and reporting 2.9 MiB/s mostly measures one bad run.

• The first request is obviously paying some startup/setup cost. Wasabi 4 KiB upload goes 309 ms, 71 ms, 66 ms. iDrive goes 402 ms, 56 ms, 52 ms. AWS goes 170 ms, 56 ms, 50 ms. Averaging the cold first request together with the warm ones makes the result fairly meaningless.

• 4 KiB is not a throughput test. At that size almost all you are measuring is latency and request overhead. Converting that into KiB/s does not make it bandwidth.

• Storj Uplink is tested with the default single parallel chunk. That is a fairly important detail when trying to draw conclusions about throughput from a system specifically designed to transfer pieces in parallel.

• Everything is tested sequentially from one Internet connection, three times, against different destinations. Routing and transient network conditions become part of the result. There is no repetition over time or randomization of test order to separate those effects from provider performance.

• Later in this same thread changing part size and worker count changes Storj performance dramatically, eventually reaching roughly 100 MiB/s in both directions and running into the ISP limit. That alone should be a fairly large warning sign against treating the original numbers as characteristics of the storage providers.

This is a benchmark of one particular HashBackup configuration, from one particular machine and network connection, at that particular moment. The numbers are nowhere near controlled enough to support broader conclusions about relative performance of Storj, B2, Wasabi, iDrive and S3.

[hashbackup]

More performance numbers, this time from a 2-CPU shared Linode (Akamai) VPS in Chicago that said it had a 4Gbit connection. I did a ping test and most of the services were 19-40ms, but iDrive2 was less than 1ms so maybe my VPS was in the same data center or very nearby.

The first set of results are single part transfers:

# hb dest -c sj test -s 4k 64m 1g
HashBackup #3336 Copyright 2009-2026 HashBackup, LLC
Using destinations in dest.conf
Warning: destination is disabled: sj
Warning: destination is disabled: b2s3
CPU count: 2

2026-08-21 00:35:19 --- Testing b2, type b2, 4 workers, 1 part ---

  4 KiB: 
    Round  1  Up 503.0ms   8.0 KiB/s   Down  77.4ms  51.7 KiB/s   Del 125.5ms  31.9 KiB/s  
    Round  2  Up  97.9ms  40.9 KiB/s   Down  65.0ms  61.6 KiB/s   Del 131.2ms  30.5 KiB/s  
    Round  3  Up 104.3ms  38.4 KiB/s   Down  66.7ms  60.0 KiB/s   Del 128.2ms  31.2 KiB/s  
  > Average   Up 235.0ms  17.0 KiB/s   Down  69.7ms  57.4 KiB/s   Del 128.3ms  31.2 KiB/s  

  64 MiB: 
    Round  1  Up   1.8s   35.1 MiB/s   Down   1.6s   40.6 MiB/s   Del 150.3ms 425.9 MiB/s  
    Round  2  Up   2.1s   31.0 MiB/s   Down   1.2s   53.3 MiB/s   Del 141.8ms 451.2 MiB/s  
    Round  3  Up   1.7s   38.0 MiB/s   Down   1.2s   53.7 MiB/s   Del 147.0ms 435.3 MiB/s  
  > Average   Up   1.9s   34.4 MiB/s   Down   1.3s   48.4 MiB/s   Del 146.4ms 437.2 MiB/s  

  1 GiB: 
    Round  1  Up  23.1s   44.3 MiB/s   Down  19.0s   54.0 MiB/s   Del 320.4ms   3.1 GiB/s  
    Round  2  Up  25.6s   40.1 MiB/s   Down  42.4s   24.1 MiB/s   Del 343.9ms   2.9 GiB/s  
    Round  3  Up  24.7s   41.4 MiB/s   Down  19.6s   52.1 MiB/s   Del 324.6ms   3.1 GiB/s  
  > Average   Up  24.5s   41.8 MiB/s   Down  27.0s   37.9 MiB/s   Del 329.6ms   3.0 GiB/s  


2026-08-21 00:38:14 --- Testing sjs3, type s3, 4 workers, gateway.storjshare.io, 1 part ---

  4 KiB: 
    Round  1  Up 392.5ms  10.2 KiB/s   Down  54.9ms  72.9 KiB/s   Del  57.1ms  70.0 KiB/s  
    Round  2  Up  90.1ms  44.4 KiB/s   Down  58.2ms  68.7 KiB/s   Del  61.4ms  65.2 KiB/s  
    Round  3  Up  85.8ms  46.6 KiB/s   Down  54.6ms  73.3 KiB/s   Del  62.6ms  63.9 KiB/s  
  > Average   Up 189.5ms  21.1 KiB/s   Down  55.9ms  71.6 KiB/s   Del  60.4ms  66.3 KiB/s  

  64 MiB: 
    Round  1  Up   1.7s   37.1 MiB/s   Down   2.7s   23.8 MiB/s   Del 264.3ms 242.1 MiB/s  
    Round  2  Up   2.3s   27.7 MiB/s   Down   3.2s   19.9 MiB/s   Del 212.3ms 301.4 MiB/s  
    Round  3  Up   2.5s   25.7 MiB/s   Down   2.2s   29.0 MiB/s   Del 211.7ms 302.3 MiB/s  
  > Average   Up   2.2s   29.4 MiB/s   Down   2.7s   23.6 MiB/s   Del 229.5ms 278.9 MiB/s  

  1 GiB: 
    Round  1  Up  15.5s   66.2 MiB/s   Down  39.9s   25.6 MiB/s   Del 423.4ms   2.4 GiB/s  
    Round  2  Up  15.4s   66.7 MiB/s   Down  37.3s   27.5 MiB/s   Del 169.4ms   5.9 GiB/s  
    Round  3  Up  14.1s   72.8 MiB/s   Down  37.3s   27.5 MiB/s   Del 341.0ms   2.9 GiB/s  
  > Average   Up  15.0s   68.5 MiB/s   Down  38.2s   26.8 MiB/s   Del 311.3ms   3.2 GiB/s  


2026-08-21 00:41:18 --- Testing wasabi, type s3, 4 workers, s3.us-east-2.wasabisys.com, 1 part ---

  4 KiB: 
    Round  1  Up 280.2ms  14.3 KiB/s   Down  31.8ms 125.6 KiB/s   Del  47.0ms  85.1 KiB/s  
    Round  2  Up  90.3ms  44.3 KiB/s   Down  32.0ms 124.9 KiB/s   Del  45.2ms  88.6 KiB/s  
    Round  3  Up  90.6ms  44.2 KiB/s   Down  33.1ms 121.0 KiB/s   Del  44.2ms  90.5 KiB/s  
  > Average   Up 153.7ms  26.0 KiB/s   Down  32.3ms 123.8 KiB/s   Del  45.5ms  88.0 KiB/s  

  64 MiB: 
    Round  1  Up   2.1s   30.6 MiB/s   Down   1.8s   36.3 MiB/s   Del 175.8ms 364.1 MiB/s  
    Round  2  Up   1.8s   35.1 MiB/s   Down   1.1s   57.1 MiB/s   Del 166.8ms 383.6 MiB/s  
    Round  3  Up   1.9s   34.4 MiB/s   Down   1.0s   61.1 MiB/s   Del 158.6ms 403.4 MiB/s  
  > Average   Up   1.9s   33.2 MiB/s   Down   1.3s   48.9 MiB/s   Del 167.1ms 383.0 MiB/s  

  1 GiB: 
    Round  1  Up  14.4s   71.3 MiB/s   Down  11.2s   91.4 MiB/s   Del 217.7ms   4.6 GiB/s  
    Round  2  Up  22.3s   45.8 MiB/s   Down  11.8s   86.8 MiB/s   Del 195.3ms   5.1 GiB/s  
    Round  3  Up  22.0s   46.6 MiB/s   Down  22.8s   45.0 MiB/s   Del 184.2ms   5.4 GiB/s  
  > Average   Up  19.6s   52.4 MiB/s   Down  15.3s   67.1 MiB/s   Del 199.1ms   5.0 GiB/s  


2026-08-21 00:43:22 --- Testing ide2, type s3, 4 workers, z8i9.ch31.idrivee2-8.com, 1 part ---

  4 KiB: 
    Round  1  Up 176.0ms  22.7 KiB/s   Down   8.4ms 477.2 KiB/s   Del   7.5ms 535.3 KiB/s  
    Round  2  Up  10.8ms 370.8 KiB/s   Down   6.3ms 636.6 KiB/s   Del   4.9ms 818.0 KiB/s  
    Round  3  Up  17.5ms 228.4 KiB/s   Down   5.7ms 697.3 KiB/s   Del   5.0ms 798.9 KiB/s  
  > Average   Up  68.1ms  58.7 KiB/s   Down   6.8ms 588.2 KiB/s   Del   5.8ms 690.9 KiB/s  

  64 MiB: 
    Round  1  Up 781.5ms  81.9 MiB/s   Down 393.7ms 162.5 MiB/s   Del   1.3s   47.7 MiB/s  
    Round  2  Up 420.2ms 152.3 MiB/s   Down 354.7ms 180.4 MiB/s   Del  78.2ms 818.0 MiB/s  
    Round  3  Up 492.3ms 130.0 MiB/s   Down 407.2ms 157.2 MiB/s   Del  22.3ms   2.8 GiB/s  
  > Average   Up 564.7ms 113.3 MiB/s   Down 385.2ms 166.1 MiB/s   Del 481.0ms 133.1 MiB/s  

  1 GiB: 
    Round  1  Up  15.3s   66.9 MiB/s   Down   5.2s  196.3 MiB/s   Del   1.1s  898.1 MiB/s  
    Round  2  Up  17.5s   58.6 MiB/s   Down   5.3s  193.4 MiB/s   Del  24.6ms  40.7 GiB/s  
    Round  3  Up  12.1s   84.4 MiB/s   Down   5.1s  200.9 MiB/s   Del  27.5ms  36.4 GiB/s  
  > Average   Up  15.0s   68.4 MiB/s   Down   5.2s  196.8 MiB/s   Del 397.4ms   2.5 GiB/s  


2026-08-21 00:44:36 --- Testing s3, type s3, 4 workers, s3-us-east-2.amazonaws.com, 1 part ---

  4 KiB: 
    Round  1  Up  86.7ms  46.1 KiB/s   Down  32.2ms 124.2 KiB/s   Del  27.2ms 147.0 KiB/s  
    Round  2  Up  35.7ms 112.2 KiB/s   Down  35.3ms 113.4 KiB/s   Del  26.4ms 151.3 KiB/s  
    Round  3  Up  51.9ms  77.0 KiB/s   Down  34.9ms 114.5 KiB/s   Del  24.5ms 163.5 KiB/s  
  > Average   Up  58.1ms  68.8 KiB/s   Down  34.1ms 117.2 KiB/s   Del  26.0ms 153.7 KiB/s  

  64 MiB: 
    Round  1  Up 824.4ms  77.6 MiB/s   Down 778.6ms  82.2 MiB/s   Del  73.2ms 874.1 MiB/s  
    Round  2  Up 823.3ms  77.7 MiB/s   Down 760.8ms  84.1 MiB/s   Del  76.7ms 834.9 MiB/s  
    Round  3  Up 918.9ms  69.6 MiB/s   Down 776.0ms  82.5 MiB/s   Del 102.8ms 622.4 MiB/s  
  > Average   Up 855.5ms  74.8 MiB/s   Down 771.8ms  82.9 MiB/s   Del  84.2ms 759.8 MiB/s  

  1 GiB: 
    Round  1  Up  12.6s   81.0 MiB/s   Down  11.2s   91.4 MiB/s   Del  77.0ms  13.0 GiB/s  
    Round  2  Up  12.6s   81.2 MiB/s   Down  11.1s   92.4 MiB/s   Del  79.2ms  12.6 GiB/s  
    Round  3  Up  12.6s   81.2 MiB/s   Down  11.1s   92.3 MiB/s   Del  82.2ms  12.2 GiB/s  
  > Average   Up  12.6s   81.1 MiB/s   Down  11.1s   92.0 MiB/s   Del  79.5ms  12.6 GiB/s  

Test complete

The next set are multipart transfers with 6 concurrent workers. I didn’t include B2 here because while B2 has multipart uploads, HashBackup doesn’t support them.

HashBackup #3336 Copyright 2009-2026 HashBackup, LLC
Using destinations in dest.conf
Warning: destination is disabled: sj
s3(wasabi): canceling incomplete upload for file: mbtest/hbtest.tmp.1
Warning: destination is disabled: b2s3
CPU count: 2

2026-08-21 00:53:23 --- Testing sjs3, type s3, 6 workers, gateway.storjshare.io, 64 MiB partsize ---

  4 KiB: 1 part
    Round  1  Up 132.3ms  30.2 KiB/s   Down 103.1ms  38.8 KiB/s   Del  99.6ms  40.2 KiB/s  
    Round  2  Up 129.7ms  30.8 KiB/s   Down 101.1ms  39.6 KiB/s   Del 101.2ms  39.5 KiB/s  
    Round  3  Up 129.0ms  31.0 KiB/s   Down 102.7ms  38.9 KiB/s   Del 103.3ms  38.7 KiB/s  
  > Average   Up 130.4ms  30.7 KiB/s   Down 102.3ms  39.1 KiB/s   Del 101.4ms  39.5 KiB/s  

  64 MiB: 1 part
    Round  1  Up   2.0s   32.6 MiB/s   Down   2.5s   26.0 MiB/s   Del 292.5ms 218.8 MiB/s  
    Round  2  Up   2.1s   31.0 MiB/s   Down   1.7s   37.3 MiB/s   Del 234.0ms 273.5 MiB/s  
    Round  3  Up   1.6s   38.8 MiB/s   Down   2.4s   26.6 MiB/s   Del 170.2ms 375.9 MiB/s  
  > Average   Up   1.9s   33.8 MiB/s   Down   2.2s   29.2 MiB/s   Del 232.2ms 275.6 MiB/s  

  1 GiB: 16 parts @ 64 MiB
    Round  1  Up   8.2s  124.1 MiB/s   Down   8.4s  121.4 MiB/s   Del 161.8ms   6.2 GiB/s  
    Round  2  Up   6.4s  160.4 MiB/s   Down   8.9s  115.6 MiB/s   Del 159.0ms   6.3 GiB/s  
    Round  3  Up   7.0s  146.7 MiB/s   Down   9.3s  109.8 MiB/s   Del 168.3ms   5.9 GiB/s  
  > Average   Up   7.2s  142.1 MiB/s   Down   8.9s  115.4 MiB/s   Del 163.0ms   6.1 GiB/s  


2026-08-21 00:54:34 --- Testing wasabi, type s3, 6 workers, s3.us-east-2.wasabisys.com, 64 MiB partsize ---

  4 KiB: 1 part
    Round  1  Up 291.6ms  13.7 KiB/s   Down  41.9ms  95.4 KiB/s   Del  51.3ms  78.0 KiB/s  
    Round  2  Up  96.6ms  41.4 KiB/s   Down  43.8ms  91.3 KiB/s   Del  50.6ms  79.0 KiB/s  
    Round  3  Up  98.6ms  40.6 KiB/s   Down  31.7ms 126.1 KiB/s   Del  45.1ms  88.6 KiB/s  
  > Average   Up 162.2ms  24.7 KiB/s   Down  39.2ms 102.2 KiB/s   Del  49.0ms  81.6 KiB/s  

  64 MiB: 1 part
    Round  1  Up   1.2s   54.0 MiB/s   Down   1.2s   53.6 MiB/s   Del 177.4ms 360.8 MiB/s  
    Round  2  Up   1.1s   56.2 MiB/s   Down 941.3ms  68.0 MiB/s   Del 170.5ms 375.3 MiB/s  
    Round  3  Up   1.5s   41.5 MiB/s   Down   1.0s   61.6 MiB/s   Del 173.6ms 368.8 MiB/s  
  > Average   Up   1.3s   49.7 MiB/s   Down   1.1s   60.5 MiB/s   Del 173.8ms 368.2 MiB/s  

  1 GiB: 16 parts @ 64 MiB
    Round  1  Up   6.0s  170.0 MiB/s   Down   4.7s  219.0 MiB/s   Del 163.3ms   6.1 GiB/s  
    Round  2  Up   5.7s  180.5 MiB/s   Down   4.4s  233.5 MiB/s   Del 171.5ms   5.8 GiB/s  
    Round  3  Up   8.1s  126.6 MiB/s   Down   4.4s  232.1 MiB/s   Del 168.0ms   6.0 GiB/s  
  > Average   Up   6.6s  155.3 MiB/s   Down   4.5s  228.0 MiB/s   Del 167.6ms   6.0 GiB/s  


2026-08-21 00:55:23 --- Testing ide2, type s3, 6 workers, z8i9.ch31.idrivee2-8.com, 64 MiB partsize ---

  4 KiB: 1 part
    Round  1  Up 115.1ms  34.7 KiB/s   Down   7.8ms 512.0 KiB/s   Del   7.4ms 537.3 KiB/s  
    Round  2  Up  12.5ms 319.8 KiB/s   Down   6.8ms 586.7 KiB/s   Del  12.4ms 322.3 KiB/s  
    Round  3  Up  13.3ms 299.7 KiB/s   Down   6.8ms 590.2 KiB/s   Del   5.7ms 700.4 KiB/s  
  > Average   Up  47.0ms  85.1 KiB/s   Down   7.1ms 560.6 KiB/s   Del   8.5ms 469.4 KiB/s  

  64 MiB: 1 part
    Round  1  Up 447.9ms 142.9 MiB/s   Down 422.1ms 151.6 MiB/s   Del 150.7ms 424.7 MiB/s  
    Round  2  Up 459.6ms 139.2 MiB/s   Down 381.0ms 168.0 MiB/s   Del  21.9ms   2.9 GiB/s  
    Round  3  Up 501.8ms 127.5 MiB/s   Down 420.5ms 152.2 MiB/s   Del  24.2ms   2.6 GiB/s  
  > Average   Up 469.8ms 136.2 MiB/s   Down 407.8ms 156.9 MiB/s   Del  65.6ms 975.6 MiB/s  

  1 GiB: 16 parts @ 64 MiB
    Round  1  Up   5.3s  194.7 MiB/s   Down   3.2s  320.7 MiB/s   Del  19.0ms  52.6 GiB/s  
    Round  2  Up   6.6s  154.7 MiB/s   Down   4.4s  231.8 MiB/s   Del  30.0ms  33.3 GiB/s  
    Round  3  Up   3.8s  272.5 MiB/s   Down   4.2s  241.8 MiB/s   Del  21.9ms  45.6 GiB/s  
  > Average   Up   5.2s  196.5 MiB/s   Down   3.9s  259.4 MiB/s   Del  23.6ms  42.3 GiB/s  


2026-08-21 00:56:01 --- Testing s3, type s3, 6 workers, s3-us-east-2.amazonaws.com, 64 MiB partsize ---

  4 KiB: 1 part
    Round  1  Up 151.1ms  26.5 KiB/s   Down  35.2ms 113.5 KiB/s   Del  24.4ms 164.2 KiB/s  
    Round  2  Up  35.8ms 111.7 KiB/s   Down  35.9ms 111.5 KiB/s   Del  30.0ms 133.1 KiB/s  
    Round  3  Up  48.2ms  83.0 KiB/s   Down  41.3ms  96.9 KiB/s   Del  30.9ms 129.6 KiB/s  
  > Average   Up  78.4ms  51.0 KiB/s   Down  37.5ms 106.8 KiB/s   Del  28.4ms 140.8 KiB/s  

  64 MiB: 1 part
    Round  1  Up 837.2ms  76.4 MiB/s   Down 773.8ms  82.7 MiB/s   Del  78.9ms 811.2 MiB/s  
    Round  2  Up 830.0ms  77.1 MiB/s   Down 764.7ms  83.7 MiB/s   Del  79.0ms 809.9 MiB/s  
    Round  3  Up 943.8ms  67.8 MiB/s   Down 764.1ms  83.8 MiB/s   Del  73.7ms 868.8 MiB/s  
  > Average   Up 870.3ms  73.5 MiB/s   Down 767.6ms  83.4 MiB/s   Del  77.2ms 829.1 MiB/s  

  1 GiB: 16 parts @ 64 MiB
    Round  1  Up   4.2s  242.4 MiB/s   Down   3.4s  298.9 MiB/s   Del  85.6ms  11.7 GiB/s  
    Round  2  Up   4.4s  234.9 MiB/s   Down   3.8s  269.1 MiB/s   Del 107.0ms   9.3 GiB/s  
    Round  3  Up   5.3s  192.6 MiB/s   Down   5.1s  199.2 MiB/s   Del 138.7ms   7.2 GiB/s  
  > Average   Up   4.6s  221.0 MiB/s   Down   4.1s  248.3 MiB/s   Del 110.4ms   9.1 GiB/s  

Test complete

I tested Storj’s uplink a bit, but it didn’t do very well, about 26MBit/s. I think a 2-CPU system doesn’t have the horsepower to get decent performance with uplink. I might try again with a bigger configuration and see about including Cloudflare R2. I know lots of people here rag on S3 because it’s an extra cost for Storj, but from a customer point of view, S3 requires less bandwith and CPU since expansion happens at the gateway, and less fiddling and hassle to “just work”.

[hashbackup]

I asked Google’s AI (a previous question was about the S3 MT gateway, so it knew that): “how is egress accounted for ranged downloads, both for the case of an inline segment and a segment stored on nodes” and it said:

When handling S3 partial range requests through a Storj Multi-Tenant (MT) Gateway, egress accounting scales differently based on whether the requested file segment is classified as an inline segment or a remote segment (stored on nodes). [1, 2]

The accounting breakdown for both architectures is detailed below:

1. The Inline Segment Case

An inline segment contains data that is exceptionally small (typically under 4 KB). Instead of distributing this microscopic piece of data across dozens of worldwide storage nodes, the encrypted data payload is embedded directly inside the file’s metadata pointer on the Storj Satellite. [1, 2]

• The Download Process: The MT Gateway issues a metadata lookup request to the satellite. Because it is an inline segment, the satellite returns the metadata which already contains the full encrypted payload. The gateway receives the whole segment, decrypts it, extracts the specific range requested by your S3 client, and discards the rest. [1, 2]

• Network Egress Accounting:

  • What you are billed: You are billed strictly for the exact byte size of the requested range (plus minimal S3 protocol overhead).

  • Behind the scenes: Even though the gateway had to pull the entire inline payload from the satellite’s metadata database, Storj’s system scales your bill to match the actual S3 range payload delivered to your client. No underlying storage nodes are involved, so no third-party node operator payouts are triggered. [2, 3]

2. The Remote Segment Case (Stored on Nodes)

For larger files broken into standard remote segments (up to 64 MiB each), data is encrypted and divided using Reed-Solomon erasure coding. To rebuild a segment, the gateway needs to successfully download data pieces from a minimum threshold of storage nodes (e.g., 29 out of 80 attempted nodes). [1, 4, 5]

When a custom byte range is requested from a remote segment, the accounting behaves as follows:

• The Download Process: The gateway calculates which specific Reed-Solomon “stripes” correspond to the requested byte range. Instead of downloading the massive 64 MiB segment in its entirety, the gateway communicates with the storage nodes to fetch only the specific encrypted blocks (stripes) that encompass your requested byte offset.

• Network Egress Accounting:

  • What you are billed: You are billed for the exact byte size of the range delivered back to your S3 client.

  • Behind the scenes (The Over-Usage Factor): Because the internet is unpredictable, the gateway always requests data from more than the minimum 29 nodes (often firing requests to 35–40 nodes simultaneously) to cut out slow-moving connections. Whichever 29 nodes finish delivering their fragments first win, and the slower connections are cancelled mid-stream.

  • The Gateway Cushion: The nodes that successfully sent fragments are paid for their partial egress by Storj. This means the total data transferred from nodes to the MT Gateway might slightly exceed the exact byte size of your requested range due to erasure-coding expansion and racing connections. However, because you are using a Storj-hosted S3 MT Gateway, Storj absorbs this internal network amplification cost. Your project account is only dinged for the final, un-amplified S3 payload size. [3, 4, 6]

Summary Comparison

Feature	Inline Segments (< 4 KB)	Remote Segments (Stored on Nodes)
Data Source	Storj Satellite metadata database	Distributed independent storage nodes
Gateway Retrieval	Downloads the entire tiny payload anyway	Targets only the specific stripes housing the range
Client Billing	Actual requested bytes	Actual requested bytes
Node Egress Triggered	None	Only the nodes actively answering the stripe race

[1] https://lup.lub.lu.se

[2] https://storj.dev

[3] https://www.storj.io

[4] https://forum.storj.io

[5] https://forum.storj.io

[6] https://forum.storj.io

While it sounds reasonable, I don’t think it’s correct. I uploaded a 1K file and did 1000 50-byte reads at random offsets. I expected to see 50K egress. My Download graph for the day said I downloaded 0.0B, even though I have other days where it’s as low as 22K. In the billing CSV file, egress is 0.001040 GB, or 1.040 MB, which indicates that even though I only downloaded and received 50 bytes, I’m charged for 1K.

I’m still trying to decipher the data for a small ranged get to storage nodes.

[arrogantrabbit]

I take it you did not give the robot access to the source code? Its musings are invalid.

Im not sure what is the value is posting ai vomit here. Disproving it is a waste of everyone’s time.

You also ignored the methodology problems already pointed out and generated another pile of bogus numbers nobody has time to debunk again.

[Toyoo]

Yeah, my AI claims:

The customer is billed for the sum of the settled orders across every node that was contacted for that stripe. With the default 29/35/80/110 scheme and a 256 B share size, that is 39 nodes × 256 B ≈ 9,984 bytes for a single-byte GET, not 1 byte.

Each storage node is credited 256 bytes (its erasure share), not 1 byte — and not more than 256, because the order the uplink signs is clamped to the requested chunk length.

[…]

One caveat on that last point: I’ve traced the code path, not Storj’s commercial policy. There may be adjustments applied outside this repo (in the invoicing configuration, price-per-GB calibration, or edge-service accounting in the Gateway-MT deployment) that account for the expansion factor. What the code establishes is the mechanism — customer egress and node compensation are the same order.Amount values, summed across every node contacted.

Claude

[hashbackup]

Thanks, that’s closer to what I’ve seen in the billing logs. It does seem like requesting and getting 1 byte but charged for 10K should be disclosed to customers if it does work that way, similar to the 50K minimum file size for storage.

[Alexey]

It’s disclosed here: Object Storage Pricing starting July 1, 2026 - Storj Docs and Egress fees

[hashbackup]

I just read those, twice, and didn’t see anything about minimum egress charges per download request. Where do they say that?

I ran a couple of experiments. On the 24th, I uploaded a 10K file and downloaded 10 bytes at random offsets every 3-4 seconds 1000 times, so I received 10000 bytes. On the Download graph on the Storj dashboard it shows 0.0B for the 24th. The CSV billing file says egress for the 24th was 8.904MB total, or 8904 bytes per request. The minimum download size from nodes is 256 bytes, so this is about 35 nodes for each request, or it could be 29 nodes sending a little more than 256 bytes with signatures, etc.

On the 25th a 1K file had 1000 50-byte download requests, so I received 50000 bytes. This file would be stored as an inline segment on the satellite, not on nodes. On the Download graph it shows 0.0B for the 25th. The CSV billing file says egress for the 25th was 1.040MB, or 1040 bytes per request.

Summary is:

• partial fetches of small files stored as inline segments are billed egress as if the whole file had been fetched
• partial fetches smaller than 256 bytes to files stored on nodes are billed egress at around 9K

[Alexey]

I mean, there’s no point in mentioning this separately if the billing unit is GB. Further detailing that the minimum possible download size is around 10 KB due to the current protocol implementation and configuration will only raise more questions. So, I don’t really understand why this is necessary.
We also mention that all egress traffic is billed. And this will include the overhead of canceling a long tail and won’t differentiate between inline and online segments.
So, in any case, all these additional bytes will be added up at the end of the month, and the amount will be rounded up to whole GB.

For storage, mentioning a minimum object size of 50 KB makes sense—you’ll have an additional line item in the invoice indicating the total number of GB of such objects. And this should help the client understand that larger objects are better than small ones. Because small objects result in higher costs for the satellite operator (including due to the larger number of segments for accounting and storing inline segments) and for node operators—there’s more disk load, more IOPS, etc.

But for Egress, mentioning a minimum possible volume is pointless; it doesn’t create any incentive—we’ll pay the operators for the entire Egress used anyway, so it makes more sense to charge the client for their Egress as well. Why round up each downloaded volume separately? It’ll be rounded up to the full GB in the invoice anyway.

[hashbackup]

I only became interested in this because of the discussion in another thread, “How to attract more customers”, where this was posted:

@littleskunk

We do have at least one customer that is using us as their primary CDN. I looked up their invoice just to understand if that is some kind of ongoing abuse but no they are doing that since a few months now. About 1 PB of download traffic for 7000 USD and just a few hundert USD in storage fees. So much on the theory it works for only storage…

One SNO I talked with said they are getting hundreds of thousands of requests every day for 256 bytes, sometimes 512 bytes (about 10-20%), with an offset into a piece. For that to happen, a customer must be doing very small ranged gets. As a customer, I would expect that if I read 10 bytes, I get billed for 10 bytes; not 9K. My actual concern was that the customer was getting billed only for the bytes he read, but Storj was paying for all the overhead, so Storj was paying 900x more to SNOs than it was receiving for this download.

@Alexey

I mean, there’s no point in mentioning this separately if the billing unit is GB. Further detailing that the minimum possible download size is around 10 KB due to the current protocol implementation and configuration will only raise more questions. So, I don’t really understand why this is necessary.

Sure, it doesn’t matter if I do 1000 of these in a test. But when there are millions of downloads like this every day, it matters.

I’m glad Storj is not paying out more than it is receiving from the customer, and will admit this is a crazy use of Storj. On the flip side, this customer is paying Storj WAY more than he would on other services that only charge for the bytes actually received. If he’s reading 90 bytes per request for example, he is paying Storj about 100x what he would pay on competing services.

@Alexey

But for Egress, mentioning a minimum possible volume is pointless

Not when there are millions of requests per day

@Alexey

we’ll pay the operators for the entire Egress used anyway, so it makes more sense to charge the client for their Egress as well.

I agree, it makes more sense for Storj, but it doesn’t make sense at all for the client. The client is paying for Storj’s backend overhead.

I’m done beating this horse, and glad your customer with millions of small downloads isn’t causing Storj a financial hardship.