Lots of very tiny ingress shards ? are these really suitable for storage nodes

Hi All,

Just noticed that @ 13:00hrs today, sudden influx of very tiny shards, which are not closing TCP sessions, any one else seeing this ?

2021-05-17T12:56:11.657Z        INFO    piecestore      uploaded        {"Piece ID": "6CMFVQTSECV3KJPWK3MFMIX6VAZ7OTYCK3LDKGFVQPPITTD47TYA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 3840}
2021-05-17T12:56:11.996Z        INFO    piecestore      upload started  {"Piece ID": "EDOJ4OLMMAPIGSIHBPDZLG5P6KT4U7GGQVBOBTYNYGHHUIHPB4CA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 91979873792}
2021-05-17T12:56:13.146Z        INFO    piecestore      uploaded        {"Piece ID": "EDOJ4OLMMAPIGSIHBPDZLG5P6KT4U7GGQVBOBTYNYGHHUIHPB4CA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 3840}
2021-05-17T12:56:13.227Z        INFO    piecestore      upload started  {"Piece ID": "3PNSBZKVWFYIQ6IYQYBOBOE3ZUXKU6WELCU6LKDSWPOHQGRTZIPA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 91979869440}
2021-05-17T12:56:13.368Z        INFO    piecestore      uploaded        {"Piece ID": "3PNSBZKVWFYIQ6IYQYBOBOE3ZUXKU6WELCU6LKDSWPOHQGRTZIPA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 3840}
2021-05-17T12:56:13.447Z        INFO    piecestore      upload started  {"Piece ID": "HZ3NQIQFFWQAAZMEUUHTS4FCGHHUGEIBAEFAHATWNQT6YEHAVHVA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 91979865088}

The piece is being written, and moved out of temp, but the connection isn’t clearing, so it’s hitting my rate limiter.

2021-05-17T12:56:58.087Z        ERROR   piecestore      upload rejected, too many requests      {"live requests": 17, "requestLimit": 12}
2021-05-17T12:56:58.234Z        ERROR   piecestore      upload rejected, too many requests      {"live requests": 17, "requestLimit": 12}
2021-05-17T12:56:58.368Z        ERROR   piecestore      upload rejected, too many requests      {"live requests": 17, "requestLimit": 12}
2021-05-17T12:56:58.519Z        ERROR   piecestore      upload rejected, too many requests      {"live requests": 17, "requestLimit": 12}

I’ve restarted node, and within 2 minutes it’s maxed out again with connections all with these really tiny shards.

Graphing of inbound connections shows TCP inbound is completely blocked, then I restarted node, then it’s blocked again.

We got any tests going on from that satellite in question ?

#edit - now even more lol, lucky I have rate limiting on else my node would explode :smiley:

2021-05-17T13:15:37.970Z        ERROR   piecestore      upload rejected, too many requests      {"live requests": 32, "requestLimit": 12}
2021-05-17T13:15:37.991Z        ERROR   piecestore      upload rejected, too many requests      {"live requests": 32, "requestLimit": 12}
2021-05-17T13:15:38.650Z        ERROR   piecestore      upload rejected, too many requests      {"live requests": 32, "requestLimit": 12}
2021-05-17T13:15:38.910Z        ERROR   piecestore      upload rejected, too many requests      {"live requests": 32, "requestLimit": 12}
2021-05-17T13:15:40.884Z        ERROR   piecestore      upload rejected, too many requests      {"live requests": 32, "requestLimit": 12}
2021-05-17T13:15:41.432Z        ERROR   piecestore      upload rejected, too many requests      {"live requests": 32, "requestLimit": 12}
2021-05-17T13:15:41.754Z        ERROR   piecestore      upload rejected, too many requests      {"live requests": 32, "requestLimit": 12}
2021-05-17T13:15:42.477Z        ERROR   piecestore      upload rejected, too many requests      {"live requests": 32, "requestLimit": 12}
2021-05-17T13:15:42.542Z        ERROR   piecestore      upload rejected, too many requests      {"live requests": 32, "requestLimit": 12}
2021-05-17T13:15:42.646Z        ERROR   piecestore      upload rejected, too many requests      {"live requests": 32, "requestLimit": 12}
1 Like

I see the small files but I havent seen anything getting rejected cause I dont have a limit set.

fixed now by sunking IP’s, looks like some slight too keen clients for my poor node to handle :face_with_monocle: sorry if that was someone with their 768byte backup job.

3 Likes

I have a similar rate limit and I don’t see anything like that in the last 3 hours. What timezone is your 13:00hrs in?

Node and Dashboards are linked to UTC time UTC+0

Then, sorry, no, I’ve got the limit set to 9 (probably a left-over from the time I experimented with btrfs) and I see no upload rejections on any of my nodes.

Maybe something in your local network blocks or delays FIN/ACKs?

1 Like

Thank you for checking, it confirms I’ve probably broken something :slight_smile: yes I think the issue it local to the node, it’s on a 2 hop VPN so could be one of the gateways having a funny. I’ve blocked a group of 4 IP’s that has reduced the traffic rate and it’s fine now - so will have a check… I have a hunch it could be related to me enabling UDP for QUIC, I’ve probably made an error somewhere as was fiddling with pfsense settings… when will I learn to stop tweaking.

2 Likes

Unless you really know you need this rate limit, you should really remove it. Almost nobody needs it to begin with and you’re just missing out on good transfers and causing issues for yourself and possibly for customers too.