At. My. Wits. End

Node Operators troubleshooting
1

I have had a hell of a time with my 1.5 year old node over the last few months. First I transferred it from Windows to Linux with some setbacks but ultimate success.

Not shortly after this was done The QUIC problem appeared. Apparently first “QUIC misconfigured” was attributed to an issue with a network update and it affected multiple people. Now it’s only me that seems to have QUIC misconfigured.

At this moment by node is offline and QUIC is misconfigured. My Dashboard loads though.

I am at my wits end. And I’m sure @Alexey hates me by this point because he has been helping me for months and it probobly fed up.

In a last ditch effort to try and get this node back up and running I re-installed Ubuntu and re-installed everything. My data, identify, and orders are on a different drive so this was not too difficult.

Here is exactly what I did I GOT NO ERRORS OR PROBLEMS DURING ANY STEP IN THIS PROCESS. EVERYTHING REPORTED BACK THE WAY IT SHOULD HAVE IN TERMINAL

  1. installed Ubuntu version 22.04.1 LTC
  2. made the STORJ directory to mount the drive too
  3. edited the fstab file and ran “sudo mount -a”
  4. Configured UDP using these directions
  5. ran sudo chown randall:randall /mnt/STORJ/storagenode/data/trust-cache.json
  6. installed docker using repository following these directions
  7. installed the docker container following these directions BUT SKIPPED STEP 2 AS THIS IS AN EXISTING NODE
  8. ran the storage node using the command in the screen shot below.

Screenshot from 2022-12-20 20-45-56
Screenshot from 2022-12-20 20-45-56956×251 42.8 KB

Port forwarding is set up correctly. There is no firewall set up. All security options on my router app have been disabled.

I suppose it’s possible that my ISP started blocking STORJ traffic but I think that is very unlikely because my Node was up and running with out any “QUIC misconfigured” just 2 weeks ago. 2 days ago is was online but QUIC was still misconfigured.

Here are my logs

2022-12-21T01:52:56.953Z	INFO	Invalid configuration file value for key	{"Process": "storagenode-updater", "Key": "log.caller"}
2022-12-21T01:52:56.953Z	INFO	Invalid configuration file value for key	{"Process": "storagenode-updater", "Key": "log.development"}
2022-12-21T01:52:56.953Z	INFO	Invalid configuration file value for key	{"Process": "storagenode-updater", "Key": "log.level"}
2022-12-21T01:52:56.953Z	INFO	Anonymized tracing enabled	{"Process": "storagenode-updater"}
2022-12-21T01:52:56.953Z	INFO	Running on version	{"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.68.2"}
2022-12-21T01:52:56.953Z	INFO	Downloading versions.	{"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2022-12-21T01:52:56.970Z	INFO	Configuration loaded	{"Process": "storagenode", "Location": "/app/config/config.yaml"}
2022-12-21T01:52:56.970Z	INFO	Anonymized tracing enabled	{"Process": "storagenode"}
2022-12-21T01:52:56.971Z	INFO	Operator email	{"Process": "storagenode", "Address": "randallst"}
2022-12-21T01:52:56.971Z	INFO	Operator wallet	{"Process": "storagenode", "Address": "0xe02c962e2"}
2022-12-21T01:52:57.272Z	INFO	Current binary version	{"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.68.2"}
2022-12-21T01:52:57.272Z	INFO	New version is being rolled out but hasn't made it to this node yet	{"Process": "storagenode-updater", "Service": "storagenode"}
2022-12-21T01:52:57.279Z	INFO	Current binary version	{"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.68.2"}
2022-12-21T01:52:57.279Z	INFO	New version is being rolled out but hasn't made it to this node yet	{"Process": "storagenode-updater", "Service": "storagenode-updater"}
2022-12-21T01:52:57.458Z	INFO	Telemetry enabled	{"Process": "storagenode", "instance ID": "12q9SBbHrkRKDECe86XNTRq1YSt9"}
2022-12-21T01:52:57.458Z	INFO	Event collection enabled	{"Process": "storagenode", "instance ID": "12q9SBbHrkRKDECe86XNTRq1YSt99f4"}
2022-12-21T01:52:57.472Z	INFO	db.migration	Database Version	{"Process": "storagenode", "version": 54}
2022-12-21T01:52:57.697Z	INFO	preflight:localtime	start checking local system clock with trusted satellites' system clock.	{"Process": "storagenode"}
2022-12-21T01:52:58.273Z	INFO	preflight:localtime	local system clock is in sync with trusted satellites' system clock.	{"Process": "storagenode"}
2022-12-21 01:52:58,273 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-12-21 01:52:58,273 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-12-21 01:52:58,273 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-12-21T01:52:58.273Z	INFO	bandwidth	Performing bandwidth usage rollups	{"Process": "storagenode"}
2022-12-21T01:52:58.273Z	INFO	Node 12q9SBbHrkRKDECe86X started{"Process": "storagenode"}
2022-12-21T01:52:58.273Z	INFO	Public server started on [::]:28967	{"Process": "storagenode"}
2022-12-21T01:52:58.273Z	INFO	Private server started on 127.0.0.1:7778	{"Process": "storagenode"}
2022-12-21T01:52:58.273Z	INFO	trust	Scheduling next refresh	{"Process": "storagenode", "after": "4h56m52.263503914s"}
2022-12-21T01:52:58.533Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "attempts": 1, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2022-12-21T01:52:58.618Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 1, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}

I have deleted some person identifying information from the log.

Again, right now I am offline and QUIC is configured but my dashboard loads

Sincerely,

Your Desperate and Frustrated Friend Randall

2

if your port is opened on Open Port Check Tool - Test Port Forwarding on Your Router, and IP from there is matching your WAN IP on your router and (please replace “your.ddns.here” to your actual DDNS hostname)

nslookup your.ddns.here 8.8.8.8

returns the same IP as your WAN IP and IP from yougetsignal, then you need to wait until satellites will clear the rate limiting for your node.
If they cannot connect your node you should see errors “ping satellite failed” with clear message - why (please filter out “check-in ratelimit” - they are consequence of a previous error when satellites cannot connect your node).

3

Look like you have

-p 28974:28974/tcp
-p 28974:28974/udp

When it should be 28967 based on your ddns port.

2 Likes
4

Good catch!
It also should be always 28967 on the right side, unless you changed the listening port in the configuration file.
So it should be

-p 28967:28967/tcp \
-p 28967:28967/udp \
1 Like
5

okay. Switched everything back to 28967 on my router app and in my run command.

Stopped, removed, & restarted the container.

Now I am online, but QUIC is misconfigured. At least sometimes it’s misconfigured. If I hit CTRL-F5 or just the refresh button my browser sometimes it says OK. But if I keep hitting refresh it says misconfigured again within a minute.

If I go to yougetsignal and input the IP address that is given on my router (different from the IP address on the Node PC) and the port 28967 then it says the port is open and gives a little green flag. if I click “use current IP” it gives the same IP that my router gives.

doing nslookup returns the same IP as my WAN IP and the IP from yougetsignal BUT it also say “** server can’t find (my ddns): NXDOMAIN”

the logs:

2022-12-21T21:40:48.066Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 1, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2022-12-21T21:40:48.268Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 1, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2022-12-21T21:40:48.987Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "attempts": 2, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2022-12-21T21:40:48.997Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "attempts": 2, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2022-12-21T21:40:49.130Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 2, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2022-12-21T21:40:49.325Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "attempts": 2, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2022-12-21T21:40:49.474Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 2, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2022-12-21T21:40:49.748Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 2, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2022-12-21T21:40:51.142Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "attempts": 3, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2022-12-21T21:40:51.158Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "attempts": 3, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2022-12-21T21:40:51.362Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 3, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2022-12-21T21:40:51.713Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "attempts": 3, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2022-12-21T21:40:51.886Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 3, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2022-12-21T21:40:52.117Z	INFO	piecestore	download started	{"Process": "storagenode", "Piece ID": "FSKBVHIXGVKX4S44KZJAQDSPAWM6I7GG4HXVS3ERWTX5OYR7AWCA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2022-12-21T21:40:55.296Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "attempts": 4, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2022-12-21T21:40:55.312Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "attempts": 4, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2022-12-21T21:40:55.621Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 4, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2022-12-21T21:40:56.316Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 4, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2022-12-21T21:40:57.128Z	INFO	piecestore	downloaded	{"Process": "storagenode", "Piece ID": "FSKBVHIXGVKX4S44KZJAQDSPAWM6I7GG4HXVS3ERWTX5OYR7AWCA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2022-12-21T21:41:01.787Z	INFO	piecestore	download started	{"Process": "storagenode", "Piece ID": "BRT5N76QA7YFAMHNLC5F73O37AAOTA7LJVRQCL3KQBDNQ2Y4YTTA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2022-12-21T21:41:01.903Z	INFO	piecestore	downloaded	{"Process": "storagenode", "Piece ID": "BRT5N76QA7YFAMHNLC5F73O37AAOTA7LJVRQCL3KQBDNQ2Y4YTTA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2022-12-21T21:41:02.470Z	INFO	piecedeleter	delete piece sent to trash	{"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "6NHKCPWCIGSZR242BEIXF3GQU2ZKOXCKPNC7ENMDXGOZRFL5TZ7A"}
2022-12-21T21:41:02.608Z	INFO	piecestore	download started	{"Process": "storagenode", "Piece ID": "OCJ5UPQTGI36JLF2BADVD4F6KCYLQH2QTB7G4HEWTIP4OP3GZ2IA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET_AUDIT"}
2022-12-21T21:41:03.235Z	INFO	piecestore	downloaded	{"Process": "storagenode", "Piece ID": "OCJ5UPQTGI36JLF2BADVD4F6KCYLQH2QTB7G4HEWTIP4OP3GZ2IA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET_AUDIT"}
2022-12-21T21:41:03.434Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "attempts": 5, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2022-12-21T21:41:03.496Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "attempts": 5, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2022-12-21T21:41:03.857Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 5, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2022-12-21T21:41:03.871Z	INFO	piecestore	download started	{"Process": "storagenode", "Piece ID": "F7TJQDL4JPW2AW3IJMONFTB6R6GF746JV7L4DKZITYDU6H3QUE4A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2022-12-21T21:41:04.447Z	INFO	piecestore	downloaded	{"Process": "storagenode", "Piece ID": "F7TJQDL4JPW2AW3IJMONFTB6R6GF746JV7L4DKZITYDU6H3QUE4A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2022-12-21T21:41:04.727Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 5, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
6

please search for “ping satellite failed” but “check-in ratelimit”

docker logs storagenode 2>&1 | grep "ping satellite failed" | grep -v "check-in ratelimit" | tail
7 
2022-12-22T10:54:50.879Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 6, "error": "ping satellite: rpc: tcp connector failed: rpc: dial tcp 34.92.23.134:7777: operation was canceled", "errorVerbose": "ping satellite: rpc: tcp connector failed: rpc: dial tcp 34.92.23.134:7777: operation was canceled\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:189"}
2022-12-22T11:00:43.131Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 6, "error": "ping satellite: rpc: tcp connector failed: rpc: dial tcp 35.228.234.184:7777: operation was canceled", "errorVerbose": "ping satellite: rpc: tcp connector failed: rpc: dial tcp 35.228.234.184:7777: operation was canceled\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:189"}
2022-12-22T11:03:23.509Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 6, "error": "ping satellite: context canceled", "errorVerbose": "ping satellite: context canceled\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2022-12-22T11:04:11.716Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 6, "error": "ping satellite: context canceled", "errorVerbose": "ping satellite: context canceled\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2022-12-22T11:17:59.748Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "attempts": 6, "error": "ping satellite: context canceled", "errorVerbose": "ping satellite: context canceled\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2022-12-22T11:22:54.664Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 6, "error": "ping satellite: rpc: tcp connector failed: rpc: dial tcp 35.228.234.184:7777: operation was canceled", "errorVerbose": "ping satellite: rpc: tcp connector failed: rpc: dial tcp 35.228.234.184:7777: operation was canceled\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:189"}
2022-12-22T11:24:20.152Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 6, "error": "ping satellite: rpc: tcp connector failed: rpc: dial tcp: lookup ap1.storj.io: operation was canceled", "errorVerbose": "ping satellite: rpc: tcp connector failed: rpc: dial tcp: lookup ap1.storj.io: operation was canceled\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:189"}
2022-12-22T11:24:57.845Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 6, "error": "ping satellite: rpc: tcp connector failed: rpc: dial tcp 34.92.23.134:7777: operation was canceled", "errorVerbose": "ping satellite: rpc: tcp connector failed: rpc: dial tcp 34.92.23.134:7777: operation was canceled\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:189"}
2022-12-22T11:29:14.731Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 6, "error": "ping satellite: rpc: tcp connector failed: rpc: dial tcp 34.92.23.134:7777: operation was canceled", "errorVerbose": "ping satellite: rpc: tcp connector failed: rpc: dial tcp 34.92.23.134:7777: operation was canceled\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:189"}
2022-12-22T11:30:02.815Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "attempts": 6, "error": "ping satellite: context canceled", "errorVerbose": "ping satellite: context canceled\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:100\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
8

Response to the satellites took to long. It looks like traffic is blocked or filtered.
Do you have another node in the same local network, which works normally?

Please disable firewall on your router and also any automatic “optimizations” and maybe “DDoS protection”, etc.

Also, is the beginning of the logs contains the same errors?

sudo docker logs storagenode 2>&1 | grep "ping satellite failed" | grep -v "check-in ratelimit" | head
9

The node checkin being rate-limited is very odd. That makes me wonder if it’s possible whether the old copy of the node (the windows one, if I understand right) is still running. Or is there any other possibility of another node somewhere running with the same node id?

10

I do not have another node. I would consider starting one to see if a new node works properly. If you think that is the next best step, let me know and I’ll do it.

I have gone though every menu option on my router app (Deco). There are no security features enabled at all.

Here are the begining lines from the log command you posted

2022-12-21T21:38:18.788Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 6, "error": "ping satellite: rpc: tcp connector failed: rpc: dial tcp 35.221.198.23:7777: operation was canceled", "errorVerbose": "ping satellite: rpc: tcp connector failed: rpc: dial tcp 35.221.198.23:7777: operation was canceled\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:189"}

@thepaul

I made absolutly sure the windows node was shut down because I was worried about losing it due to disqualification for running a duplicate node.

however it’s possible I re-installed the OS without stopping and removing the container. I don’t think this happened because I was trying to be careful but at this point I should consider all possibilities I guess. If this were in fact the case, is there any action I could take?

There is no other possibility of a node running anywhere else but my house. If another node with same node id was running somewhere else, wouldn’t that result in disqualification anyway?

Does it matter that my node is just eggressig? I assumed it’s because I maxed out the storage. I am pretty much at the limit of the drive but I could spare 50GB to see if ingress does more than 2,4 Mb

Screenshot from 2022-12-23 20-43-19
Screenshot from 2022-12-23 20-43-19735×730 42.3 KB

Screenshot from 2022-12-23 20-44-02
Screenshot from 2022-12-23 20-44-02799×737 35.3 KB

11

Yes, if you could have a second node running with the same NodeID (i.e. - with the same identity), then it would be disqualified pretty fast, so unlikely this is the case.
@thepaul I more concerned by errors like posted above

Specifically

Why the node could cancel the request? As far as I understand this is mean that node cannot contact the satellite for some reason (at least it’s not responding or the node cannot receive an answer).

Yes, please, but on different PC or VM with a different OS (maybe it’s the OS issue). You can move it later if you want.

12

I will do a different PC but using ubuntu. I don’t want to mess with Windows any longer at all.

For the second node I will use port 28981

My container command for the second node should look like this then correct?

-p 28981:28967/tcp \
-p 28981:28967/udp \

DDNS will stay the same as I cannot have more than one on my router. On the port forwarding tab in my router app it says “internal port” and also “external port”

Should I have 28981 for both internal and external on my router app? Or I should one be 28981 and another 28967?

13

yes, do not forget the ADDRESS option, i.e.

-e ADDRESS=external.address:28981 \

you may configure it with using 28981 for both, or, since it’s expected to be a different machine you may configure 28981 as external and 28967 as internal, but then you need to use 28967 port in the port mapping in your docker run command and 28981 in the ADDRESS option, i.e.:

-p 28967:28967/tcp \
-p 28967:28967/udp \
-e ADDRESS=external.address:28981 \

See

14

Sorry it took so long.

I have set up a new node on a different PC using linux.

The node runs just fine. But online for 3 days without any QUIC misconfigured.

Online time is 100% on all sattelites.

Something tells me we still won’t be able to figure out what’s wrong though…

1 Like
15

Is it the same distro?

16

Yes Ubuntu 22.

I have noticed on the new node there is no “orders” folder…

The old node has an “orders” folder though.

Was I supposed to delete the “orders” folder after the transition from windows to linux? could that be what is causing the problem?

17

No, it will not cause any problems. The orders folder should be in the same location, where your config.yaml, unless you changed it with a parameter storage2.orders.path.
For docker version it will be in the data location by default.

18

I moved the node to a different machine and I’m still getting QUIC misconfigured.

It’s on the same machine as two other brand new nodes that are both running perfectly fine.

19

If other two are working fine, then something is missed for this one.
Please check again all prerequisites: Step 4. Configure QUIC - Storj Node Operator Docs
You may also just try to refresh a dashboard first (Ctrl-F5).

20

Could it be because I have the node on a SSD and not an HDD?

There is nothing different from the other nodes and this one. Everything is configured the same except the obvious differences in port numbers and pathways.

I could recite to you the 6 steps to setup a node. You and I have looked at every possible avenue over the last two months.

It says online 93% and is still produces revenue. I might just give up and let it run as is.