ERROR piecestore upload rejected, too many requests

Node Operators troubleshooting
1

Hello,

I have a log full of this error:

2019-09-18T06:38:20.034131232Z 2019-09-18T06:38:20.033Z ERROR piecestore upload rejected, too many requests {“live requests”: 641}

I have try to increase the value on config.yaml as suggested on other post:

storage2.max-concurrent-requests: 640

but after time (with 640 value) takes about 12 hours the problem occurs again,
now I have se the value to 6400.

What is the problem and how I can solve it ?

thanks

2

You completely misunderstood the point of this configuration.
It is used to keep the concurrent requests within a range that your device can handle so it is able to finish the requests before getting new ones. It will otherwise easily get overwhelmed.
The recommended value is maybe 7 for a Raspberry Pi and 20 for a decent PC.
You have to find out by yourself how many uploads actually finish and how many get cancelled because your device wasn’t able to finish those.

Getting the error “piecestore upload rejected” is therefore totally fine and I wouldn’t recommend setting concurrent requests higher than 20 without knowing your hardware and the succesful/failed upload ratio.

1 Like
3

Thanks,
How I can check my succesful/failed upload ratio ?

4
5

Wow, I’ve never seen anyone actually reach thresholds that high. That’s not a good thing, your node is clearly not able to handle those requests as fast as they come in. You’re most likely also failing almost all uploads and a good chunk of downloads. This is exactly why this concurrency limit exists. To make sure your hardware isn’t overloaded. I created a post a while ago outlining how to find your perfect value. In your case there may even be a benefit of lowering it below the default.

6

I have run the script and after have reduce the max concurrent request to 20 this is the report:

Fri Sep 20 08:00:59 CEST 2019
======== AUDIT ===========
Successful: 2 (2)
Recoverable failed: 0 (1)
Unrecoverable failed: 0 (0)
Success Rate Min: 100.0% (66.6666666667%)
Success Rate Max: 100.0% (100.0%)
======== DOWNLOAD ========
Successful: 0 (20)
Failed: 0 (0)
Success Rate: 0% (100.0%)
======== UPLOAD ==========
Successful: 1111 (193)
Rejected: 8884 (26413)
Failed: 1180 (4220)
Acceptance Rate: 11.1155577789% (0.72540028565%)
Success Rate: 48.4941073767% (4.37344210288%)
======== REPAIR DOWNLOAD =
Successful: 0 (0)
Failed: 0 (0)
Success Rate: 0% (0%)
======== REPAIR UPLOAD ===
Successful: 0 (0)
Failed: 0 (0)
Success Rate: 0% (0%)

What do you think ?

Is this report related to last 24h or to all log file since has been started ?

Thansk

7

It reads the whole log file. That’s why your successrate rate is very low.
Try removing the log file and wait another 24 hours. But try lowering concurrent requests to 7.
The acceptance rate should be >70% and the successrate too.

This is from one of my nodes with concurrent requests at 10 on a decent pc:
========== AUDIT =============
Successful: 8754
Recoverable failed: 17
Unrecoverable failed: 1
Success Rate Min: 99.795%
Success Rate Max: 99.989%
========== DOWNLOAD ==========
Successful: 78748
Failed: 2696
Success Rate: 96.690%
========== UPLOAD ============
Successful: 188014
Rejected: 987
Failed: 8531
Acceptance Rate: 99.478%
Success Rate: 95.660%
========== REPAIR DOWNLOAD ===
Successful: 0
Failed: 0
Success Rate: 0.000%
========== REPAIR UPLOAD =====
Successful: 0
Failed: 0
Success Rate: 0.000%

8

It’s based on the docker logs currently available. Usually since the last update, as removing the container also removes the logs.

I really suggest starting with the default value of 7. It seems to me your hardware is still being squeezed.

9

I don’t agree with this part. That is simply not always achievable depending on your hardware and connection. You’re aiming for the highest success rate possible on your node. You can raise concurrency a bit but only if it wouldn’t drop the success rate or acceptance rate.

@brizio71 can you tell us a little bit about the hardware you are using? What HDD are you using? How is it connected? What is your connection speed? Where are you roughly located?

10

I have a Ubuntu VM running on QNAP with Celern quadcore J1900 with all 4 core dedicated and with 4GB dedicated memory, RAID 5 disk connected in NFS, the connection speed is a fiber connetion to 1GB in donwload and 200MB in upload. I’m in Milan Italy.

11

Ok, a likely issue is your use of NFS. Are you running a VM on the same system? Does QNAP not offer native docker support? If not I suggest using iSCSI instead.

I’m fairly certain your hardware is capable of better performance than what you are seeing right now. And your connection and location are pretty ideal for current tests. But you probably really want to get rid of the use of NFS.

Have a look at container station, I think that might be your solution for native docker on QNAP.

12

I have move from NFS to ISCSI and now the situation in much better:

Thu Sep 26 08:00:36 CEST 2019
======== AUDIT ===========
Successful: 43 (31)
Recoverable failed: 0 (1)
Unrecoverable failed: 0 (0)
Success Rate Min: 100.0% (96.875%)
Success Rate Max: 100.0% (100.0%)
======== DOWNLOAD ========
Successful: 9875 (1873)
Failed: 54 (1)
Success Rate: 99.4561385839% (99.946638207%)
======== UPLOAD ==========
Successful: 3111 (570)
Rejected: 0 (2355)
Failed: 70 (195)
Acceptance Rate: 100.0% (19.4871794872%)
Success Rate: 97.7994341402% (74.5098039216%)
======== REPAIR DOWNLOAD =
Successful: 0 (0)
Failed: 0 (0)
Success Rate: 0% (0%)
======== REPAIR UPLOAD ===
Successful: 0 (0)
Failed: 0 (0)
Success Rate: 0% (0%)

Can I try to increase the storage2.max-concurrent-request ?

13

Thanks for reporting back on that! I know people have been skeptical about this in the past but these numbers show a very clear difference.

As for increasing concurrency, right now it looks like you would not benefit from it as no uploads got rejected to begin with. Wait until your acceptance rate drops. If it does you can increase it slowly while keeping a close eye on the success rate. The danger of increasing it now is that when load on the network increases you might find out that your node can’t actually handle your higher setting and starts failing many more uploads. So increase slowly and only increase when you actually hit the limits and verify your node could still easily handle it.

14

I have increase the value of storage2.max-concurrent-request from 10 to 20 and after 24h this are the results:

Fri Sep 27 08:01:07 CEST 2019
======== AUDIT ===========
Successful: 16 (43)
Recoverable failed: 0 (0)
Unrecoverable failed: 0 (0)
Success Rate Min: 100.0% (100.0%)
Success Rate Max: 100.0% (100.0%)
======== DOWNLOAD ========
Successful: 1820 (9875)
Failed: 2 (54)
Success Rate: 99.8902305159% (99.4561385839%)
======== UPLOAD ==========
Successful: 428 (3111)
Rejected: 0 (0)
Failed: 12 (70)
Acceptance Rate: 100.0% (100.0%)
Success Rate: 97.2727272727% (97.7994341402%)
======== REPAIR DOWNLOAD =
Successful: 0 (0)
Failed: 0 (0)
Success Rate: 0% (0%)
======== REPAIR UPLOAD ===
Successful: 0 (0)
Failed: 0 (0)
Success Rate: 0% (0%)

What do you think ? Can I increase more ?

15

There are not enough uploads at the moment. Keep the requests at 10 until you see a drop in acceptance rate while your success rate still stays close to 100%

1 Like