ERROR piecestore upload rejected, too many requests

All night have this thing, Connection limit is 40 on config
It is not pisible that during the night all connected and no one made opload or download, looks like counter stacked, made restart, now it geting data.
0 MB during night and all the time writing this below.
2019-12-24T11:06:29.320+0200 ERROR piecestore upload rejected, too many requests {“live requests”: 41}

If needed log file to analize can share it or upload to some location.

i discovered that other nodes in the morning started to make same thing.

Due to the latest updates there is no more connection limitation needed so you can comment it out and check that behavior again next day.

Which OS is running on your Node?
Can’t confirm that behavior on my Nodes

1 Like

Windows GUI Node version 28.x
Looks like problem is that some of uploads stuck for hole night. In one moment there can be 10000 Stuck connections, then processor start eat lot of power.

Had a similar behavior on a low powered Node with small upload bandwidth…
If the bandwidth is full the Node starts to reject and begin to stuck on.

Try to comment out the connection limitation and let the Node regulate by itself.

1 Like

Saw the same thing on my node just now. I had it set to 50 and never hit that limit before. Must be some weird traffic on the network. I took the moment to comment out the setting. Lets see how that goes.

Edit: Over 35000 rejected in one day…

I have I5 on node, it cant be low processor power.
I made limitation 100.

Let’s see how it works. I would appreciate your feedback.

Are you also on Win GUI?

I made also support request and sended log file.
Devs can analize this.

Docker setup on Linux. 500/500mbit, so it has nothing to do with bandwidth. CPU and RAM don’t look any different during this period. Nowhere near to a bottleneck.
I saw a significant drop network and disk IO though, which is to be expected if most connections are rejected.

I have 100/100Mbit connection, router statistics show max 30Mbit was used and averaged 10.
I have more than 1 node here.

Just checked my traffic last night.

Noticed a huge local network IO between Server and Storage >25MB/s for round about 3 sec
but logs look still good at this time. No IO on WAN at same time… Nodes on Linux no limitation.

:thinking:

Check logs at this time, may be was delete some pieces then it was moved to trashbean

Are you use the network connected drives?
If so, please take a look on: https://forum.storj.io/tags/nfs and https://forum.storj.io/tags/smb

No i am not use network drives, problem that i had stucked 40 connections, and all other connnections was rejected, as i had 40 connection limitations in config. After restart node stoped stucked connections and started work again.

It is defenetly not a solution, because next time can be same thing 40 connections stuck is a problem, becase even if 100-1000 connections also can get stack.

Node rejected connections during all night. I think it need to investigate why connetions are stuking.
Becase during hole night i had 0 mb Ingress and all the time rejected connections. After restart i made config of maximum 100 connections, all working now, but question is how long it get to take 100 stucked connections again?

Did you see rejected connections after you have removed the parameter?

I not see how many connections i have now, stucked or not. Not got limit yet.
May be from your side it looks like panic or something, but i just try to make understandable that someone research it until it start be big or very big problem. Sended also Support request with log file.
I understand if i get some data during the night and thats connections over limit would be filtered.
But problem is that All connections was rejected like over limid. I had 0 data during Night.
And 12000 rejected requests. This mean that connections have no timeouts?