Ping and untrusted: unable to get signee: trust

Storage Node Operators troubleshooting
#1

I’ve this errors:

2020-05-17T23:55:17.677+0200 ERROR contact:service ping satellite failed {"Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 2, "error": "ping satellite error: rpccompat: context deadline exceeded", "errorVerbose": "ping satellite error: rpccompat: context deadline exceeded\n\tstorj.io/common/rpc.Dialer.dialTransport:264\n\tstorj.io/common/rpc.Dialer.dial:241\n\tstorj.io/common/rpc.Dialer.DialAddressID:151\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:117\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:87\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

and

2020-05-17T23:53:05.028+0200 ERROR piecestore download failed {"Piece ID": "XVKMFKRDRHHWRJNL65ZCRIZYUCEGMDDE3EHNNKWBSN72FDSUQ5MQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "error": "untrusted: unable to get signee: trust: rpccompat: context deadline exceeded", "errorVerbose": "untrusted: unable to get signee: trust: rpccompat: context deadline exceeded\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).VerifyOrderLimitSignature:140\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:62\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:523\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Download:471\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:995\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:66\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}

#2

And another node:

2020-05-17T23:55:38.807+0200    ERROR   nodestats:cache Get stats query failed  {"error": "node stats service error: unable to connect to the satellite 118
UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW: rpccompat: context deadline exceeded; node stats service error: unable to connect to the satellite 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE: rpccompat: context deadline exceeded; node stats service error: unable to connect to the satellite 121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6: rpccompat: context deadline exceeded; node stats service error: unable to connect to the satellite 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S: rpccompat: context deadline exceeded; node stats service error: unable to connect to the satellite 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs: rpccompat: context deadline exceeded; node stats service error: unable to connect to the satellite 12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB: rpccompat: context deadline exceeded", "errorVerbose": "group:\n--- node stats service error: unable to connect to the sat
ellite 118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW: rpccompat: context deadline exceeded\n\tstorj.io/storj/storagenode/nodestats.(*Service).dial:156\n\tstorj.io/storj/storagenode/nodestats.(*Service).GetReputationStats:66\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheReputationStats.func1:139\n\tstorj.io/storj/storagenode/nodestats.(*Cache).satelliteLoop:244\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheReputationStats:138\n\tstorj.io/storj/storagenode/nodestats.(*Cache).Run.func2:105\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- node stats service error: unable to connect to the satellite 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE: rpccompat: context deadline exceeded\n\tstorj.io/storj/storagenode/nodestats.(*Service).dial:156\n\tstorj.io/storj/storagenode/nodestats.(*Service).GetReputa
tionStats:66\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheReputationStats.func1:139\n\tstorj.io/storj/storagenode/nodestats.(*Cache).satelliteLoop:244\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheReputationStats:138\n\tstorj.io/storj/storagenode/nodestats.(*Cache).Run.func2:105\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- node stats service error: unable to connect to the satellite 121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6: rpccompat: context deadline exceeded\n\tstorj.io/storj/storagenode/nodestats.(*Service).dial:156\n\tstorj.io/storj/storagenode/nodestats.(*Service).GetReputationStats:66\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheReputationStats.func1:139\n\tstorj.io/storj/storagenode/nodestats.(*Cache).satelliteLoop:244\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheR
eputationStats:138\n\tstorj.io/storj/storagenode/nodestats.(*Cache).Run.func2:105\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- node stats service error: unable to connect to the satellite 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S: rpccompat: context deadline exceeded\n\tstorj.io/storj/storagenode/nodestats.(*Service).dial:156\n\tstorj.io/storj/storagenode/nodestats.(*Service).GetReputationStats:66\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheReputationStats.func1:139\n\tstorj.io/storj/storagenode/nodestats.(*Cache).satelliteLoop:244\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheReputationStats:138\n\tstorj.io/storj/storagenode/nodestats.(*Cache).Run.func2:105\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- node stats service error: unable to connect to the satellite 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs: rpccompat: context deadline exceeded\n\tstorj.io/storj/storagenode/nodestats.(*Service).dial:156\n\tstorj.io/storj/storagenode/nodestats.(*Service).GetReputationStats:66\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheReputationStats.func1:139\n\tstorj.io/storj/storagenode/nodestats.(*Cache).satelliteLoop:244\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheReputationStats:138\n\tstorj.io/storj/storagenode/nodestats.(*Cache).Run.func2:105\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- node stats service error: unable to connect to the satellite 12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB: rpccompat: context deadline exceeded\n\tstorj.io/storj/storagenode/nodestats.(*Service).dial:156\n\tstorj.io/storj/storagenode/nodestats.(*Service).GetReputationStats:66\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheReputationStats.func1:139\n\tstorj.io/storj/storagenode/nodestats.(*Cache).satelliteLoop:244\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheReputationStats:138\n\tstorj.io/storj/storagenode/nodestats.(*Cache).Run.func2:105\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
#3

Found another thread with your error message, does this help? Asia-east Suspended/Not recognized

Looks to me like your node has connection problems, maybe a firewall problem on your pc or router?

#4

All udp (exclude system DNS + 8.8.8.8 + 8.8.4.4) blocked. All TCP open without any firewall.

#5

However, it’s connection problem.
I have not saw on my nodes, but I saw such messages here on forum from another nodes, which are offline more often than usual.
Could it possible that your router can’t handle a lot of connections?
Have you uptimerobot setup and stat?

#7

I am not yet sure how it happened but I had the same error after I performed apt-get update/upgrade and the Docker got upgraded. This needs user root. However normally I issue the Docker command for the storagenode under a different user.
When I manually stopped Docker again and restarted it under the correct user, then the error message did not show up again any more.
I have not done further testing if this really is the cause for the error but I believe I had this before and solution was the same. I have always wondered, if the Docker run command should be executed as root.

#8

Normally it should not, otherwise you will be forced to chown data to root, unless you did not add your user to the docker group and did not logout/login

#9

Let me do some more testing, if the error is really related to what user I execute the run command with.