QUIC misconfigured and node status offline

Node Operators troubleshooting
,
41

@Alexey

sudo ufw status said inactive.

The port forwarding rules on my router:

Service Type: DNS
Internal IP: <matches was I get when I do “hostname -I” in terminal>
internal port: 28967
external port: 28967
Protocol: TCP & UDP

That is everything the router app says. Should I change the service type perhaps?

Also, I am using the DDNS option on my router and therefore I never installed NO-IP. Doing it this way worked just fine when the node was on Windows so I assumed that would also be true for linux…

Should I try installing NO-IP?

42

if you mean a no-ip’s application DUC, then it’s not needed, if you configured your router to update your DDNS hostname, otherwise they may influence each other (two updaters).

Did you refresh your dashboard with Ctrl-F5?

p.s. You do not need to tag me in every post, thank you.

43

yes. CTRL-F5 did not work.

44

Did you recreate the storagenode container after you did this configuration:

45

yes.

I used the two commands below and then the whole docker run command

sudo docker stop -t 300 storagenode

sudo docker rm storagenode

46

Then I’m out of ideas, why your UDP connections are unstable.
Please search for errors in your logs related to QUIC and/or UDP.

47

Here is the log, I don’t see anything in there about QUIC. If you are out of ideas that’s disheartening. You have helped me a great deal over this last year though Alexey and for that you have my gratidude.

I’ll try using NO-IP for the DDNS later today and see if that makes a difference.

2022-12-11T13:46:54.465Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "attempts": 3, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
48

The rate limiting error is a consequence of unsuccessful attempts to connect to your node.
So it should have errors “ping satellite failed” from the same satellite before but with actual reason like timeout or some other error except rate limit.

49

I tried changing the ddns domain name and that didn’t help. I still plan to try setting it up using NO-IP this weekend probobly.

That’s the error message I got today

2022-12-15T01:12:37.032Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 2, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2022-12-15T01:12:37.833Z	INFO	piecestore	download started	{"Process": "storagenode", "Piece ID": "NR5QLS3M4JM42AKVLKTZIBEBUDIHFS6TIFXPZRG6C2Q6DHZYPNPA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2022-12-15T01:12:38.569Z	INFO	piecestore	downloaded	{"Process": "storagenode", "Piece ID": "NR5QLS3M4JM42AKVLKTZIBEBUDIHFS6TIFXPZRG6C2Q6DHZYPNPA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2022-12-15T01:12:38.675Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "attempts": 3, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2022-12-15T01:12:38.914Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 3, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2022-12-15T01:12:39.143Z	INFO	piecestore	download started	{"Process": "storagenode", "Piece ID": "OSAFBE5QWP2DJWKXTDSE4YBRZP35AB73B2DBCECHZKUDXQTEAMUA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2022-12-15T01:12:39.478Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 3, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2022-12-15T01:12:39.711Z	INFO	piecestore	downloaded	{"Process": "storagenode", "Piece ID": "OSAFBE5QWP2DJWKXTDSE4YBRZP35AB73B2DBCECHZKUDXQTEAMUA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
50

You need to find errors “ping satellite failed” except “check-in ratelimit”, because they are consequence of a previous failed attempts.

51

when you say “find” do you mean I need to frequently and manually run the command to generate the logs until I see a log with “check-in ratelimit”?

52

the rough example:

docker logs storagenode 2>&1 | grep "ping satellite failed" | grep -v "check-in ratelimit" | head
53

that doesn’t seem to have done anything

image
image1106×62 14.7 KB

54

switching to NO-IP for DDNS didn’t help either. I was able connect just fine and got the “online” but QUIC was misconfigured as usual. I didn’t bother to set up the dynamic update client though.

55

maybe it has something to do with the fact that everytime I restart the computer after statically mounting the drive I end up in emergency mode. This is either the exact mesage or nearly the exact message that appears:

Screenshot from 2022-12-16 23-54-09
Screenshot from 2022-12-16 23-54-09848×153 44.3 KB

I’m able to make it back to ubutnu after after typing “systemctl reboot” and hitting enter. I’m assuming the static mounting is what is cuasing the crash. I don’t know what else about the whole STORJ process would cause it to crash. I have not used this PC for anything else but STORJ after installing the OS.

Also get this error message when trying to restart the docker

Screenshot from 2022-12-17 00-01-31
Screenshot from 2022-12-17 00-01-311367×137 19 KB

56

Did you redirect logs to the file? If so, you need to search in that file, i.e.

cat /mnt/storj/storagenode/storagenode.log | grep "ping satellite failed" | grep -v "check-in ratelimit" | head

If you didn’t, then perhaps you need to use sudo, i.e.

sudo docker logs storagenode 2>&1 | grep "ping satellite failed" | grep -v "check-in ratelimit" | head

This suggest to me that your docker installation could be broken. Did you follow their guide? Install Docker Engine on Ubuntu | Docker Docs

57

sudo worked…

2022-12-17T06:15:41.423Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 6, "error": "ping satellite: context canceled", "errorVerbose": "ping satellite: context canceled\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2022-12-17T06:19:42.574Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 6, "error": "ping satellite: rpc: tcp connector failed: rpc: dial tcp 34.92.23.134:7777: operation was canceled", "errorVerbose": "ping satellite: rpc: tcp connector failed: rpc: dial tcp 34.92.23.134:7777: operation was canceled\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:189"}
2022-12-17T06:28:59.188Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 6, "error": "ping satellite: rpc: tcp connector failed: rpc: dial tcp: lookup ap1.storj.io: operation was canceled", "errorVerbose": "ping satellite: rpc: tcp connector failed: rpc: dial tcp: lookup ap1.storj.io: operation was canceled\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:189"}
2022-12-17T06:38:47.952Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "attempts": 6, "error": "ping satellite: context canceled", "errorVerbose": "ping satellite: context canceled\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2022-12-17T06:41:12.262Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 6, "error": "ping satellite: context canceled", "errorVerbose": "ping satellite: context canceled\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}

As far as the other stuff I don’t know that those issues are related. I’ll start another thread for that one.

58

I never followed the instructions to download and install the docker engine that you posted. I simply ran this command according to the guide

docker pull storjlabs/storagenode:latest

I went and did the docker engine install like you suggested. I followed the instructions for “install using repository”. Everything looked good and I got the “Hello World” message just fine.

To my utter dismay and unbearable frustration, that actually caused the node to go offline. I stopped it, removed, re-ran the QUIC instructions in STEP 4 and Docker Container download in STEP 6. I then used the same docker run command as before. It says offline and QUIC misconfigured now.

The dashboard loads, and everything looks fine in terminal when running the commands. The logs are giving the same ERROR basically as before but for some reason I am completely offline.

Maybe it’s just time to throw in the towel. I have been hanging on to this node because it’s 1.5 years old and I have put so much time into it. However, at this point it’s seems the Universe has sent a clear message.

@Alexey I hope you are being compensated for all the help you give on this forum. I tip my hat to you sir.

59

This is the real issue - your node have network problems, it cannot connect to the satellites and they cannot ping your node back. Something is blocking the traffic. Maybe you have some “smart screening” or “DDOS protection” etc. on your router? If so, try to disable them.

Regarding docker reinstall and offline issue now.
Please post the last 20 lines from your logs after start.

60

well it went back to being online but misconfigured

What does this mean?

Screenshot from 2022-12-18 20-09-09
Screenshot from 2022-12-18 20-09-091491×225 56.8 KB