Small downloads with offset

[alpharabbit]

I see a lot of downloads where only 256 bytes from a larger piece are requested. This makes no sense to me. Is it some kind of testing or ddos?

2025-03-31T20:38:28+02:00       INFO    piecestore      downloaded      {"Piece ID": "BV2DOKQAI3JEJFSK2WQPNIYZHA7RHSUKYVACUY2XC5QGGSRZ6NYQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Offset": 216832, "Size": 256, "Remote Address": "93.177.65.143:56996"}

[Toyoo]

BUG? Multiple call to on piece

From what I gather from the Engineers looking into some issues, there appears to be a large number of download requests which may be streaming requests. The cause isn’t clear to me right now. It could be a bunch of simulataneous requests coming in at once, or it could be a bunch of backed up requeats being released at once. What I can tell you is that there is all hands on deck working to triage the situation and are working on solutions. At some point when we have more information we’ll in…

[Knowledge]

Could also be an audit. Checking if you have specific pieces. If it is the same piece over and over again, than it is likely something else.

[alpharabbit]

Action is “GET” not “GET_AUDIT”. So I think it’s not audits.

[nerdatwork]

@alpharabbit

only 256 bytes from a larger piece

Can same piece id have different sizes?

That shows the amount the uplink has downloaded or intends to download, not the size of piece being stored. For example, if you only want to read the first 128bytes, it would have Size: 128… (ignoring encryption etc. overheads).

[alpharabbit]

Those are valid requests and downloads are successful done. The Question is why whould someone download such small parts of a piece file?

Splitting pieces into smaller parts and downloading them in parallel could make sense for performance reasons but not with parts that small.

On the other hand, for a dos attack small parts from random files makes sense…

[Toyoo]

@alpharabbit

On the other hand, for a dos attack small parts from random files makes sense…

Nodes themselves respond to these downloads very well: piece data is likely cached and read-ahead by the OS anyway, and any writes necessary for accounting (orders, bandwidth and disk space statistics) are batched across many such requests. It could be a problem for your network infrastructure to handle so many small network connections though. For example, my ISP’s router has problems dealing with thousands of concurrent NATted connections, basically hanging and requiring manual restart—thankfully Storj traffic does not reach this number anymore for my setup.

[alpharabbit]

If files are truly random cache can’t help. Requesting 256 bytes will always result in a much bigger read from whatever storage ist used.

But I agree, router overload might kick in earlier than storage overload. My pfsense showing 150K states atm, I guess not many home routers could handle this.

[Toyoo]

@alpharabbit

If files are truly random cache can’t help. Requesting 256 bytes will always result in a much bigger read from whatever storage ist used.

Ah, sorry, the behavior that was observed November was reading from the same piece 256 bytes at a time, instead of requesting it in one go. Do you observe something different?

[alpharabbit]

I see multiple requests for the same file but a number of file names used. Probably not much randomness.