Node in constant state of upload rejected, no upload failed or uploaded

Node Operators troubleshooting
1

Hello,

So after his updated to 16.1, I noticed that my node is only showing upload rejected. It keeps showing upload rejected continuesly in log file. I know upload rejected is limiting the number of cocurrent connections but it just keeps going. At no point do I see the typical upload failed message and then after an upload started to signal the node is processing another request.

image
image1203×324 223 KB

System resources appear to be fine

image

The dashboard does not show any abnormality

image

2

Hi @xyphos10

At this moment heavy stress test is running, you can try to increase limit, here is text from doccumentation:

Special Node Configuration to Limit Concurrent Uploads

Slower nodes like Raspberry Pi3 for example, may have difficulties getting any data. In previous releases (up to 0.14.3) they were accepting too many concurrent uploads but were unable to finish them in time. In the new releases after 0.14.3, we now have a new configuration option available to fine-tune the number of concurrent uploads. If your node was already previously running, first docker stop -t 300 storagenode and then please edit your config.yaml file (on Windows, use Nodepad++, not Notepad, on MacOS, use TextEdit, not Notes) to add the following line at the end of the file:

storage2.max-concurrent-requests: 7

Save the config.yaml file and then restart your node by docker restart -t 300 storagenode , if it was already running before. If this is the first time you will start your node, please follow the instructions in the following sections to use the proper parameters with your docker run storagenode command.

This will allow slow nodes to focus on a smaller number of uploads and finish them as fast as possible, while refusing the uploads it couldn’t process anyway. You can adjust the number of requests - 7 is just an initial suggestion, but you can modify this number up or down and monitor the performance of your node until you find the right number of requests for your particular node which does not cause your node to be overwhelmed. Faster nodes will be able to function with a higher number of concurrent requests than slow nodes.

1 Like
3

Is your storage on NFS?

4

VMware workstation pro shared folder which is a hgfs mount

5

What the result of this script: Script: Calculate Success Rates for Audit, Download, Upload, Repair?

6

I had rm the node and recreated it not to long ago so the stats will be only for the past 22 hours

image
image1209×54 3.76 KB

image

7

Seems it’s problem with shared folders. What is your host OS?
Try to stop and remove the container and re-add the shared folders. Maybe it could help.

8

I am running Windows 10 LTSB and vmware workstation pro 15 as the hypervisor. I would use docker directly but I have other work related matters that needs vmware. What I find odd is that the node did not behave like this before 0.16.1. In any case I will have to investigate when I get more time

9

In case of Windows perhaps you can’t use a docker for Windows, since it’s requires the Hyper-V enabled.
I think it’s not compatible with VMWare.

You have some other options: connect a whole disk to the VM, or create a Virtual disk and transfer all data to there.

1 Like
10

Thanks for the suggestion, I will try with a virtual disk and see how it goes.

1 Like