Many requests (over 20k download request) for same piece id

Node Operators troubleshooting
1

Since two days ago, one of my node starts to receive download request for same piece ID LOSQUU3HGYVSAUJ55OWTY5RRBLZ3V4R57SXLPWJHGMIQQRKBUKTQ. There are over 50k lines of log item from this piece ID over 6 hours.

I know it is part of the customer behaviour. However, for handling these download request took up most part of IO for my disk.

Not sure if there is any work around for that?

2024-07-11T08:24:02Z INFO piecestore download canceled {Process: storagenode, Piece ID: LOSQUU3HGYVSAUJ55OWTY5RRBLZ3V4R57SXLPWJHGMIQQRKBUKTQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: GET, Offset: 0, Size: 512, Remote Address: 79.127.226.100:32796}
2024-07-11T08:24:02Z INFO piecestore download canceled {Process: storagenode, Piece ID: LOSQUU3HGYVSAUJ55OWTY5RRBLZ3V4R57SXLPWJHGMIQQRKBUKTQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: GET, Offset: 0, Size: 512, Remote Address: 109.61.92.65:38086}
2024-07-11T08:24:02Z INFO piecestore download canceled {Process: storagenode, Piece ID: LOSQUU3HGYVSAUJ55OWTY5RRBLZ3V4R57SXLPWJHGMIQQRKBUKTQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: GET, Offset: 0, Size: 512, Remote Address: 79.127.226.98:45730}
2024-07-11T08:24:02Z INFO piecestore download canceled {Process: storagenode, Piece ID: LOSQUU3HGYVSAUJ55OWTY5RRBLZ3V4R57SXLPWJHGMIQQRKBUKTQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: GET, Offset: 0, Size: 512, Remote Address: 79.127.219.42:47900}
2

Is this a windows node? I believe all Linux systems would keep that one piece in memory and don’t consume any additional iops. So I wonder if windows might behave different.

2 Likes
3

It is in Linux Debian.

4

how much memory does the system have in total and how much is used by processes?

Are all of the download requests canceled? Or are there also successful transmissions?

5

half of them are successful, half of them are canceled.

The memory usage is just 20-30%. I think it should not be an issue.

These requests are atomic.

6

Most likely this piece was cached in RAM all that time. The reason for canceled downloads would then be the fact that you are “far” (in network latency terms) from the requestors—which is quite plausible given the requested amount of data was only half a kibibyte. Hence if you have observed elevated I/O at the same time, this piece wouldn’t be the reason for it.

1 Like