ERROR piecestore upload rejected, too many requests

Node Operators troubleshooting
21

If you have not yet done so already, please completely comment out or remove the line with the concurrency parameter. It is not needed any longer since a few releases ago.

22

Yes I comented concurency row in config.

23

This evening again I have all the connections rejected like the other time and a lot of people apparently have had this problem. I have restarted my node to resolve the problem.
where could that prevent? it’s been 2 times now in 1 week that this happens

24

did you comented #retain.concurrency: 100
as it was obsolate, there is looks like some bugs and some connections are stucking.

25

how many concurrent retain requests can be processed at the same time.

retain.concurrency: 5

26

plese coment this line by # mark, and make restart to node.

27

i do this :

how many concurrent retain requests can be processed at the same time.

retain.concurrency: 100

it’s ok ? and I restarted my node

Capture d’écran 2019-12-29 Ă  12.54.47
Capture d’écran 2019-12-29 Ă  12.54.471146×90 4.86 KB

28

this is ok, if there is # befor rown number is not relevant any more.

29

Thanks for your help, happy end of the year :pray:

30

There looks like bug in storagenode.
I discovered after this thread, than i also have
ERROR piecestore upload rejected, too many requests {“live requests”: 41}
But i have commented #retain.concurrency: 100
and made restart after some days ago.
And more than 1 node show this problem.
Just 41 number for some reason.
Can it be some limitation on windows?
Storj Team?
Looks like it not dropping faild ingress loads.
Egress loads works at same time.
@littleskunk ?

31

With no limit set on my node, I have the following “too many requests” errors in my Dec log:

2019-12-19T20:43:40.616Z    ERROR    piecestore upload rejected, too many requests {"live requests": 41}
2019-12-19T21:03:12.779Z    ERROR    piecestore upload rejected, too many requests {"live requests": 42}
2019-12-22T23:12:58.502Z    ERROR    piecestore upload rejected, too many requests {"live requests": 99}
32

is it windows GUI or linux?
@anon27637763

33

How do you explain all the successful uploads between a few failed uploads?

The old answer is still valid. GRPC is still limited by default. You should not change that because GRPC is able to let your storage node crash with an out of memory exception. Everything is working as expected. No issue here.

34
  • Linux Docker client, v 0.28.2
  • Docker version 19.03.5
  • watchtower autoupdates enabled

I don’t think the “too many requests” rejection policy is a problem. It seems that the concurrent requests limit is being stepped up gradually from a default of 7 
 if I remember correctly
 to seems like 100 now.

35

if node bloking all Ingress connections for hole night untill i restart it, it is Very BIG isue

36

As was Announced before it dont have limit from 26-27 version

37

GRPC is still limited to 7. DRPC has no limit.unless you set one yourself.

38

That is only possible if you set a limit yourself. Remove the limit and it should work just fine.

39

I do NOT have this limits also, thans why i cald it BUG.

40

Please count the following messages in your logfile for the current day.

root@storj:~# grep 2019-12-29 storagenode.log | grep 'upload started' | wc -l
23758
root@storj:~# grep 2019-12-29 storagenode.log | grep 'uploaded' | wc -l
23397
root@storj:~# grep 2019-12-29 storagenode.log | grep 'upload failed' | wc -l
295
root@storj:~# grep 2019-12-29 storagenode.log | grep 'too many requests' | wc -l
0