Lots of very tiny ingress shards ? are these really suitable for storage nodes

Node Operators
1

Hi All,

Just noticed that @ 13:00hrs today, sudden influx of very tiny shards, which are not closing TCP sessions, any one else seeing this ?

2021-05-17T12:56:11.657Z        INFO    piecestore      uploaded        {"Piece ID": "6CMFVQTSECV3KJPWK3MFMIX6VAZ7OTYCK3LDKGFVQPPITTD47TYA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 3840}
2021-05-17T12:56:11.996Z        INFO    piecestore      upload started  {"Piece ID": "EDOJ4OLMMAPIGSIHBPDZLG5P6KT4U7GGQVBOBTYNYGHHUIHPB4CA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 91979873792}
2021-05-17T12:56:13.146Z        INFO    piecestore      uploaded        {"Piece ID": "EDOJ4OLMMAPIGSIHBPDZLG5P6KT4U7GGQVBOBTYNYGHHUIHPB4CA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 3840}
2021-05-17T12:56:13.227Z        INFO    piecestore      upload started  {"Piece ID": "3PNSBZKVWFYIQ6IYQYBOBOE3ZUXKU6WELCU6LKDSWPOHQGRTZIPA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 91979869440}
2021-05-17T12:56:13.368Z        INFO    piecestore      uploaded        {"Piece ID": "3PNSBZKVWFYIQ6IYQYBOBOE3ZUXKU6WELCU6LKDSWPOHQGRTZIPA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 3840}
2021-05-17T12:56:13.447Z        INFO    piecestore      upload started  {"Piece ID": "HZ3NQIQFFWQAAZMEUUHTS4FCGHHUGEIBAEFAHATWNQT6YEHAVHVA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 91979865088}

The piece is being written, and moved out of temp, but the connection isn’t clearing, so it’s hitting my rate limiter.

2021-05-17T12:56:58.087Z        ERROR   piecestore      upload rejected, too many requests      {"live requests": 17, "requestLimit": 12}
2021-05-17T12:56:58.234Z        ERROR   piecestore      upload rejected, too many requests      {"live requests": 17, "requestLimit": 12}
2021-05-17T12:56:58.368Z        ERROR   piecestore      upload rejected, too many requests      {"live requests": 17, "requestLimit": 12}
2021-05-17T12:56:58.519Z        ERROR   piecestore      upload rejected, too many requests      {"live requests": 17, "requestLimit": 12}

I’ve restarted node, and within 2 minutes it’s maxed out again with connections all with these really tiny shards.

Graphing of inbound connections shows TCP inbound is completely blocked, then I restarted node, then it’s blocked again.

Capture
Capture756×220 22.3 KB

We got any tests going on from that satellite in question ?

#edit - now even more lol, lucky I have rate limiting on else my node would explode :smiley:

2021-05-17T13:15:37.970Z        ERROR   piecestore      upload rejected, too many requests      {"live requests": 32, "requestLimit": 12}
2021-05-17T13:15:37.991Z        ERROR   piecestore      upload rejected, too many requests      {"live requests": 32, "requestLimit": 12}
2021-05-17T13:15:38.650Z        ERROR   piecestore      upload rejected, too many requests      {"live requests": 32, "requestLimit": 12}
2021-05-17T13:15:38.910Z        ERROR   piecestore      upload rejected, too many requests      {"live requests": 32, "requestLimit": 12}
2021-05-17T13:15:40.884Z        ERROR   piecestore      upload rejected, too many requests      {"live requests": 32, "requestLimit": 12}
2021-05-17T13:15:41.432Z        ERROR   piecestore      upload rejected, too many requests      {"live requests": 32, "requestLimit": 12}
2021-05-17T13:15:41.754Z        ERROR   piecestore      upload rejected, too many requests      {"live requests": 32, "requestLimit": 12}
2021-05-17T13:15:42.477Z        ERROR   piecestore      upload rejected, too many requests      {"live requests": 32, "requestLimit": 12}
2021-05-17T13:15:42.542Z        ERROR   piecestore      upload rejected, too many requests      {"live requests": 32, "requestLimit": 12}
2021-05-17T13:15:42.646Z        ERROR   piecestore      upload rejected, too many requests      {"live requests": 32, "requestLimit": 12}
1 Like
2

I see the small files but I havent seen anything getting rejected cause I dont have a limit set.

4

I have a similar rate limit and I don’t see anything like that in the last 3 hours. What timezone is your 13:00hrs in?

5

Node and Dashboards are linked to UTC time UTC+0

6

Then, sorry, no, I’ve got the limit set to 9 (probably a left-over from the time I experimented with btrfs) and I see no upload rejections on any of my nodes.

Maybe something in your local network blocks or delays FIN/ACKs?

1 Like
8

Unless you really know you need this rate limit, you should really remove it. Almost nobody needs it to begin with and you’re just missing out on good transfers and causing issues for yourself and possibly for customers too.