QUIC misconfigured and node status offline

also can you show port forward rule on router?
does this internal PC with linux has static internal ip or dynamic?

They have 10.131.101.50 as a WAN IP. It’s an internal diapason.

Thank you @Alexey and @Vadim, we have opted for static IP and our node is up and running.

@Alexey

You sure are dedicated to this project sir.

Here you are

j.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2022-12-03T05:21:58.593Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 5, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2022-12-03T05:22:01.028Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 5, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}

Also,

Since everything is on a drive that is separate from my OS can I do the following?

  1. Stop and remove the existing docker
  2. Reformat my OS drive
  3. Reinstall Ubuntu
  4. Statically mount the STORJ drive again
  5. Setup and run using these instructions: Storage Node - Storj Node Operator Docs

I’m having issues when I boot the computer so I wanted to reinstall my OS anyway.

This is a consequence of your node being offline for these satellites (they cannot connect to it), but your node is trying to check-in without correcting of the issue, so satellites starts rate limiting your node.

Yes, you can, but you must skip SETUP=true step, it should be done only once for entire node’s life and should not be executed for the worked node.
Also make sure that identity for this node on the disk with data, not on your OS’s disk.

@Alexey

So This is what it should look like when running set up?

docker run --rm -e \
    --user $(id -u):$(id -g) \
    --mount type=bind,source="<identity-dir>",destination=/app/identity \
    --mount type=bind,source="<storage-dir>",destination=/app/config \
    --name storagenode storjlabs/storagenode:latest

Or should I change it SETUP=“false”

Also, Why would I be offline for those satellites? And how do I get back online for those satellite? How do I correct the issue if re-installing the OS doesn’t work?

no. You need to run a full docker run command with all parameters

docker run -d --restart unless-stopped --stop-timeout 300 \
    -p 28967:28967/tcp \
    -p 28967:28967/udp \
    -p 127.0.0.1:14002:14002 \
    -e WALLET="0xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" \
    -e EMAIL="user@example.com" \
    -e ADDRESS="domain.ddns.net:28967" \
    -e STORAGE="2TB" \
    --user $(id -u):$(id -g) \
    --mount type=bind,source="<identity-dir>",destination=/app/identity \
    --mount type=bind,source="<storage-dir>",destination=/app/config \
    --name storagenode storjlabs/storagenode:latest

Something is blocking an access - either firewall on your PC, or your router or ISP.

So I skip step 2 on this page all together?

Just download the docker and run it?

Yes, you have performed this step already when you installed your storagenode in a first time.


@Alexey

Is this why my QUIC is misconfigured? It won’t let me run one of the commands in the guide even under sudo.

Did you fix the offline issue?
If so, to configure QUIC on Linux: Linux Configuration for UDP - Storj Node Operator Docs

sudo -s

echo "net.core.rmem_max=2500000" >> /etc/sysctl.conf

sysctl -w net.core.rmem_max=2500000
exit

After that just refresh the dashboard.
If it’s still misconfigured, then check that you have a correct firewall’s and router’s rules: Step 4. Configure QUIC - Storj Node Operator Docs

3 Likes

@Alexey

Yes I am back online but QUIC is still misconfigured.

I was able to run the commands using “sudo -s”.

However, that did not solve the problem and QUIC is still misconfigured. I have everything right for port forwarding on my router. I reset my router too and that didn’t help.

Here are the logs. Does this tell you anything?

j.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2022-12-10T13:29:06.142Z	INFO	piecestore	download started	{"Process": "storagenode", "Piece ID": "OVZGFCVBLMRN6TGMLXVAIPAXEIXWR6WSD72M42ZJ5RL36I5AQHCQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2022-12-10T13:29:06.721Z	INFO	piecedeleter	delete piece sent to trash	{"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "ZCDLQTPUKRPTKCV6YBD662GWB6AC5DPOJCXUTWIJZKCRMQHZAL3Q"}
2022-12-10T13:29:07.094Z	INFO	piecestore	downloaded	{"Process": "storagenode", "Piece ID": "OVZGFCVBLMRN6TGMLXVAIPAXEIXWR6WSD72M42ZJ5RL36I5AQHCQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}

Just double checking you’ve done Ctrl+F5 on dashboard?

Please post your whole docker run command of you are sure the router port forwarding is correct.

The documentation only discusses Linux, and links to someone’s GitHub document (that, by the way, discusses configuration on both, Linux and FreeBSD/Darwin), that in turn links to another article with almost identical content, here: Increase OS UDP Buffers to Improve Performance | by Cameron Sparr | Medium

Why not

  1. Link to that source instead from Linux Configuration for UDP - Storj Node Operator Docs,
  2. Add information for those OSes to STORJ documentation: there are binaries for those extra OSes, and some folks will benefit for having relevant information available.
  3. Step 4. Configure QUIC - Storj Node Operator Docs shall call out FreeBSD/Darwin users as well as Linux, because this is not unique to Linux (“Linux users, please take a look at”).

It seems to me the documentation emphasizes and focuses too much on Linux and docker, and this is not even the best way to run things, albeit most popular…

@Stob

Yes, I did try ctrl+F5 several times.

Sometimes it says “OK” but most of the time i says “Misconfigured”

@Stob @Alexey

I moved the node from Windows to Linux about a month an half a ago. While the node was running in windows I never had an issue with QUIC. The issue only appeared when I switch to Linux. That makes me believe the issue is not with my ISP.

@Stob

Here is the docker run command

sudo docker run -d --restart unless-stopped --stop-timeout 300 \
    -p 28967:28967/tcp \
    -p 28967:28967/udp \
    -p 127.0.0.1:14002:14002 \
    -e WALLET="REDACTED" \
    -e EMAIL="REDACTED" \
    -e ADDRESS="REDACTED.tplinkdns.com:28967" \
    -e STORAGE="3.55TB" \
    --user $(id -u):$(id -g) \
    --mount type=bind,source="/mnt/STORJ/storagenode/identity/storjidentity",destination=/app/identity \
    --mount type=bind,source="/mnt/STORJ/storagenode/data",destination=/app/config \
    --name storagenode storjlabs/storagenode:latest

I have added your favorite OS to the UDP configuration :slight_smile: Thanks for suggestions!

1 Like

Do you have a firewall installed?

sudo ufw status

Please also show your port forwarding rules on your router for this node.