A lot of errors!
SnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_REPAIR"}
2020-05-30T15:40:31.069+0200 INFO orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE finished
2020-05-30T15:40:31.069+0200 INFO orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S finished
2020-05-30T15:40:31.069+0200 INFO orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 finished
2020-05-30T15:40:31.069+0200 ERROR orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE failed to settle orders {"error": "order: unable to connect to the satellite: rpccompat: context deadline exceeded", "errorVerbose": "order: unable to connect to the satellite: rpccompat: context deadline exceeded\n\tstorj.io/storj/storagenode/orders.(*Service).settle:281\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:220\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders.func2:199\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-05-30T15:40:31.069+0200 ERROR orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S failed to settle orders {"error": "order: unable to connect to the satellite: rpccompat: context deadline exceeded", "errorVerbose": "order: unable to connect to the satellite: rpccompat: context deadline exceeded\n\tstorj.io/storj/storagenode/orders.(*Service).settle:281\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:220\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders.func2:199\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-05-30T15:40:31.069+0200 INFO orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs finished
2020-05-30T15:40:31.069+0200 ERROR orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 failed to settle orders {"error": "order: unable to connect to the satellite: rpccompat: dial tcp 104.199.150.210:7777: i/o timeout", "errorVerbose": "order: unable to connect to the satellite: rpccompat: dial tcp 104.199.150.210:7777: i/o timeout\n\tstorj.io/storj/storagenode/orders.(*Service).settle:281\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:220\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders.func2:199\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-05-30T15:42:38.980+0200 INFO piecestore download started {"Piece ID": "2WBAZTQ6V3WTJD6ZB2AXFJ2H5KFW4GLLXLAUFB5NYQEVN5KCZA5Q", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_AUDIT"}
2020-05-30T15:42:58.981+0200 ERROR piecestore download failed {"Piece ID": "2WBAZTQ6V3WTJD6ZB2AXFJ2H5KFW4GLLXLAUFB5NYQEVN5KCZA5Q", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET_AUDIT", "error": "untrusted: unable to get signee: trust: rpccompat: context deadline exceeded", "errorVerbose": "untrusted: unable to get signee: trust: rpccompat: context deadline exceeded\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).VerifyOrderLimitSignature:140\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:62\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:515\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Download:463\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1005\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
I’ve no problems with: DNS (hetzner+google) and network.
2020-05-30T15:48:06.977+0200 ERROR contact:service ping satellite failed {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 5, "error": "ping satellite error: rpccompat: context deadline exceeded", "errorVerbose": "ping satellite error: rpccompat: context deadline exceeded\n\tstorj.io/common/rpc.Dialer.dialTransport:264\n\tstorj.io/common/rpc.Dialer.dial:241\n\tstorj.io/common/rpc.Dialer.DialAddressID:151\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:117\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:87\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-05-30T15:48:06.979+0200 ERROR contact:service ping satellite failed {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "attempts": 5, "error": "ping satellite error: rpccompat: context deadline exceeded", "errorVerbose": "ping satellite error: rpccompat: context deadline exceeded\n\tstorj.io/common/rpc.Dialer.dialTransport:264\n\tstorj.io/common/rpc.Dialer.dial:241\n\tstorj.io/common/rpc.Dialer.DialAddressID:151\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:117\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:87\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-05-30T15:48:38.883+0200 INFO piecestore download started {"Piece ID": "R3SECHPWYNG4IR7ZRBCTZUCXFI4P3B2KEWT7JSCGSSRQK3TACFHA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET"}
2020-05-30T15:48:39.104+0200 INFO piecestore downloaded {"Piece ID": "R3SECHPWYNG4IR7ZRBCTZUCXFI4P3B2KEWT7JSCGSSRQK3TACFHA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET"}
2020-05-30T15:48:42.977+0200 ERROR contact:service ping satellite failed {"Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 6, "error": "ping satellite error: rpccompat: context deadline exceeded", "errorVerbose": "ping satellite error: rpccompat: context deadline exceeded\n\tstorj.io/common/rpc.Dialer.dialTransport:264\n\tstorj.io/common/rpc.Dialer.dial:241\n\tstorj.io/common/rpc.Dialer.DialAddressID:151\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:117\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:87\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-05-30T15:48:42.978+0200 ERROR contact:service ping satellite failed {"Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "attempts": 6, "error": "ping satellite error: rpccompat: context deadline exceeded", "errorVerbose": "ping satellite error: rpccompat: context deadline exceeded\n\tstorj.io/common/rpc.Dialer.dialTransport:264\n\tstorj.io/common/rpc.Dialer.dial:241\n\tstorj.io/common/rpc.Dialer.DialAddressID:151\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:117\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:87\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-05-30T15:48:42.978+0200 ERROR contact:service ping satellite failed {"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 6, "error": "ping satellite error: rpccompat: context deadline exceeded", "errorVerbose": "ping satellite error: rpccompat: context deadline exceeded\n\tstorj.io/common/rpc.Dialer.dialTransport:264\n\tstorj.io/common/rpc.Dialer.dial:241\n\tstorj.io/common/rpc.Dialer.DialAddressID:151\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:117\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:87\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-05-30T15:48:42.978+0200 ERROR contact:service ping satellite failed {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 6, "error": "ping satellite error: rpccompat: context deadline exceeded", "errorVerbose": "ping satellite error: rpccompat: context deadline exceeded\n\tstorj.io/common/rpc.Dialer.dialTransport:264\n\tstorj.io/common/rpc.Dialer.dial:241\n\tstorj.io/common/rpc.Dialer.DialAddressID:151\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:117\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:87\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-05-30T15:48:42.979+0200 ERROR contact:service ping satellite failed {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "attempts": 6, "error": "ping satellite error: rpccompat: context deadline exceeded", "errorVerbose": "ping satellite error: rpccompat: context deadline exceeded\n\tstorj.io/common/rpc.Dialer.dialTransport:264\n\tstorj.io/common/rpc.Dialer.dial:241\n\tstorj.io/common/rpc.Dialer.DialAddressID:151\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:117\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:87\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
Terminal:
root@noda1 /srv # ping 104.199.150.210
PING 104.199.150.210 (104.199.150.210) 56(84) bytes of data.
64 bytes from 104.199.150.210: icmp_seq=1 ttl=37 time=283 ms
64 bytes from 104.199.150.210: icmp_seq=2 ttl=37 time=283 ms
64 bytes from 104.199.150.210: icmp_seq=3 ttl=37 time=283 ms
64 bytes from 104.199.150.210: icmp_seq=4 ttl=37 time=283 ms
64 bytes from 104.199.150.210: icmp_seq=5 ttl=37 time=283 ms
64 bytes from 104.199.150.210: icmp_seq=6 ttl=37 time=283 ms
^C
--- 104.199.150.210 ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 1003ms
rtt min/avg/max/mdev = 283.114/283.163/283.212/0.435 ms
root@noda1 /srv # curl 104.199.150.210
^C
root@noda1 /srv # wget 104.199.150.210
--2020-05-30 15:53:25-- http://104.199.150.210/
Connecting to 104.199.150.210:80...
Log:
Looks like network problem on your side.
2020-05-30T01:36:22.931Z ERROR contact:service ping satellite failed {"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 12, "error": "ping satellite error: rpccompat: dial tcp 34.94.104.31:7777: connect: connection refused", "errorVerbose": "ping satellite error: rpccompat: dial tcp 34.94.104.31:7777: connect: connection refused\n\tstorj.io/common/rpc.Dialer.dialTransport:264\n\tstorj.io/common/rpc.Dialer.dial:241\n\tstorj.io/common/rpc.Dialer.DialAddressID:151\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:117\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:87\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
This is a Saltlake satellite, which was in the maintenance:
Hi all,
On May 29th, 2020 we will be performing maintenance to move our satellitedb from CloudSQL to self-hosted postgres for the saltlake.tardigrade.io satellite. The downtime is estimated to be approximately 5 hours. We will update this post when we start and complete the maintenance.
Thanks!
Krista
I have an old records such as
2020-05-29T02:17:49.549+0300 ERROR contact:service ping satellite failed {"SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "attempts": 11, "error": "ping satellite error: rpccompat: context deadline exceeded", "errorVerbose": "ping satellite error: rpccompat: context deadline exceeded\n\tstorj.io/common/rpc.Dialer.dialTransport:279\n\tstorj.io/common/rpc.Dialer.dial:256\n\tstorj.io/common/rpc.Dialer.DialAddressID:166\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:117\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:87\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:152\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
And I think that satellite was in a maintenance before the Saltlake.
No other such messages in all my different nodes - Windows GUI, Windows docker and Raspberry Pi 3B+
Strange situation. I only have such errors on one host, but I do not experience problems with the network. The destination IP is available, and the firewall is disabled.
Alexey
May 30, 2020, 11:29pm
7
Could it be a Roscomnadzor?
Alexey
May 31, 2020, 12:01am
9
Maybe some updates can affect some network services?