Strange node issues - container going offline

Hi, im running a storagenode on my raspberry pi since a few days.
Since around two days i have the strange issue after running the containers for around 45 minutes.
The node seems to be offline, if i would restart the docker container it is immediately back online.
I’ve also removed the whole environment and reinstalled it - Issues are still there.
The Pi is always responsive and also reachable from outside the network.

Hi JoeKa, welcome to the forums.

Could you please post the results of running docker logs --tail 20 storagenode

(I am also going to move this to the Storage Node Operators section of the forum, which is a better place for this topic)

Thats an example how it looks like before the restart:
docker logs --tail 40 storagenode

2020-02-12T19:35:13.502Z ERROR contact:chore ping satellite failed {"Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "attempts": 6, "error": "ping satellite error: rpccompat: context deadline exceeded", "errorVerbose": "ping satellite error: rpccompat: context deadline exceeded\n\tstorj.io/common/rpc.Dialer.dialTransport:242\n\tstorj.io/common/rpc.Dialer.dial:219\n\tstorj.io/common/rpc.Dialer.DialAddressID:138\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatelliteOnce:154\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatellite:124\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:103\n\tstorj.io/common/sync2.(*Cycle).Run:147\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

2020-02-12T19:35:54.941Z ERROR contact:chore ping satellite failed {"Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "attempts": 7, "error": "ping satellite error: rpccompat: context deadline exceeded", "errorVerbose": "ping satellite error: rpccompat: context deadline exceeded\n\tstorj.io/common/rpc.Dialer.dialTransport:242\n\tstorj.io/common/rpc.Dialer.dial:219\n\tstorj.io/common/rpc.Dialer.DialAddressID:138\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatelliteOnce:154\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatellite:124\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:103\n\tstorj.io/common/sync2.(*Cycle).Run:147\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

2020-02-12T19:35:54.941Z ERROR contact:chore ping satellite failed {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 7, "error": "ping satellite error: rpccompat: context deadline exceeded", "errorVerbose": "ping satellite error: rpccompat: context deadline exceeded\n\tstorj.io/common/rpc.Dialer.dialTransport:242\n\tstorj.io/common/rpc.Dialer.dial:219\n\tstorj.io/common/rpc.Dialer.DialAddressID:138\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatelliteOnce:154\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatellite:124\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:103\n\tstorj.io/common/sync2.(*Cycle).Run:147\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

2020-02-12T19:35:54.943Z ERROR contact:chore ping satellite failed {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "attempts": 7, "error": "ping satellite error: rpccompat: context deadline exceeded", "errorVerbose": "ping satellite error: rpccompat: context deadline exceeded\n\tstorj.io/common/rpc.Dialer.dialTransport:242\n\tstorj.io/common/rpc.Dialer.dial:219\n\tstorj.io/common/rpc.Dialer.DialAddressID:138\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatelliteOnce:154\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatellite:124\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:103\n\tstorj.io/common/sync2.(*Cycle).Run:147\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

2020-02-12T19:36:05.504Z ERROR contact:chore ping satellite failed {"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 7, "error": "ping satellite error: rpccompat: context deadline exceeded", "errorVerbose": "ping satellite error: rpccompat: context deadline exceeded\n\tstorj.io/common/rpc.Dialer.dialTransport:242\n\tstorj.io/common/rpc.Dialer.dial:219\n\tstorj.io/common/rpc.Dialer.DialAddressID:138\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatelliteOnce:154\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatellite:124\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:103\n\tstorj.io/common/sync2.(*Cycle).Run:147\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

2020-02-12T19:36:05.504Z ERROR contact:chore ping satellite failed {"Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "attempts": 7, "error": "ping satellite error: rpccompat: context deadline exceeded", "errorVerbose": "ping satellite error: rpccompat: context deadline exceeded\n\tstorj.io/common/rpc.Dialer.dialTransport:242\n\tstorj.io/common/rpc.Dialer.dial:219\n\tstorj.io/common/rpc.Dialer.DialAddressID:138\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatelliteOnce:154\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatellite:124\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:103\n\tstorj.io/common/sync2.(*Cycle).Run:147\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

2020-02-12T19:37:18.943Z ERROR contact:chore ping satellite failed {"Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "attempts": 8, "error": "ping satellite error: rpccompat: context deadline exceeded", "errorVerbose": "ping satellite error: rpccompat: context deadline exceeded\n\tstorj.io/common/rpc.Dialer.dialTransport:242\n\tstorj.io/common/rpc.Dialer.dial:219\n\tstorj.io/common/rpc.Dialer.DialAddressID:138\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatelliteOnce:154\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatellite:124\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:103\n\tstorj.io/common/sync2.(*Cycle).Run:147\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

2020-02-12T19:37:18.946Z ERROR contact:chore ping satellite failed {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "attempts": 8, "error": "ping satellite error: rpccompat: context deadline exceeded", "errorVerbose": "ping satellite error: rpccompat: context deadline exceeded\n\tstorj.io/common/rpc.Dialer.dialTransport:242\n\tstorj.io/common/rpc.Dialer.dial:219\n\tstorj.io/common/rpc.Dialer.DialAddressID:138\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatelliteOnce:154\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatellite:124\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:103\n\tstorj.io/common/sync2.(*Cycle).Run:147\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

2020-02-12T19:37:18.948Z ERROR contact:chore ping satellite failed {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 8, "error": "ping satellite error: rpccompat: context deadline exceeded", "errorVerbose": "ping satellite error: rpccompat: context deadline exceeded\n\tstorj.io/common/rpc.Dialer.dialTransport:242\n\tstorj.io/common/rpc.Dialer.dial:219\n\tstorj.io/common/rpc.Dialer.DialAddressID:138\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatelliteOnce:154\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatellite:124\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:103\n\tstorj.io/common/sync2.(*Cycle).Run:147\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

2020-02-12T19:37:29.506Z ERROR contact:chore ping satellite failed {"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 8, "error": "ping satellite error: rpccompat: context deadline exceeded", "errorVerbose": "ping satellite error: rpccompat: context deadline exceeded\n\tstorj.io/common/rpc.Dialer.dialTransport:242\n\tstorj.io/common/rpc.Dialer.dial:219\n\tstorj.io/common/rpc.Dialer.DialAddressID:138\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatelliteOnce:154\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatellite:124\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:103\n\tstorj.io/common/sync2.(*Cycle).Run:147\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

2020-02-12T19:37:29.507Z ERROR contact:chore ping satellite failed {"Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "attempts": 8, "error": "ping satellite error: rpccompat: context deadline exceeded", "errorVerbose": "ping satellite error: rpccompat: context deadline exceeded\n\tstorj.io/common/rpc.Dialer.dialTransport:242\n\tstorj.io/common/rpc.Dialer.dial:219\n\tstorj.io/common/rpc.Dialer.DialAddressID:138\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatelliteOnce:154\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatellite:124\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:103\n\tstorj.io/common/sync2.(*Cycle).Run:147\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

2020-02-12T19:39:46.948Z ERROR contact:chore ping satellite failed {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "attempts": 9, "error": "ping satellite error: rpccompat: context deadline exceeded", "errorVerbose": "ping satellite error: rpccompat: context deadline exceeded\n\tstorj.io/common/rpc.Dialer.dialTransport:242\n\tstorj.io/common/rpc.Dialer.dial:219\n\tstorj.io/common/rpc.Dialer.DialAddressID:138\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatelliteOnce:154\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatellite:124\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:103\n\tstorj.io/common/sync2.(*Cycle).Run:147\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

2020-02-12T19:39:46.950Z ERROR contact:chore ping satellite failed {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 9, "error": "ping satellite error: rpccompat: context deadline exceeded", "errorVerbose": "ping satellite error: rpccompat: context deadline exceeded\n\tstorj.io/common/rpc.Dialer.dialTransport:242\n\tstorj.io/common/rpc.Dialer.dial:219\n\tstorj.io/common/rpc.Dialer.DialAddressID:138\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatelliteOnce:154\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatellite:124\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:103\n\tstorj.io/common/sync2.(*Cycle).Run:147\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

2020-02-12T19:39:46.952Z ERROR contact:chore ping satellite failed {"Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "attempts": 9, "error": "ping satellite error: rpccompat: context deadline exceeded", "errorVerbose": "ping satellite error: rpccompat: context deadline exceeded\n\tstorj.io/common/rpc.Dialer.dialTransport:242\n\tstorj.io/common/rpc.Dialer.dial:219\n\tstorj.io/common/rpc.Dialer.DialAddressID:138\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatelliteOnce:154\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatellite:124\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:103\n\tstorj.io/common/sync2.(*Cycle).Run:147\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

2020-02-12T19:39:57.509Z ERROR contact:chore ping satellite failed {"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 9, "error": "ping satellite error: rpccompat: context deadline exceeded", "errorVerbose": "ping satellite error: rpccompat: context deadline exceeded\n\tstorj.io/common/rpc.Dialer.dialTransport:242\n\tstorj.io/common/rpc.Dialer.dial:219\n\tstorj.io/common/rpc.Dialer.DialAddressID:138\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatelliteOnce:154\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatellite:124\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:103\n\tstorj.io/common/sync2.(*Cycle).Run:147\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

2020-02-12T19:39:57.509Z ERROR contact:chore ping satellite failed {"Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "attempts": 9, "error": "ping satellite error: rpccompat: context deadline exceeded", "errorVerbose": "ping satellite error: rpccompat: context deadline exceeded\n\tstorj.io/common/rpc.Dialer.dialTransport:242\n\tstorj.io/common/rpc.Dialer.dial:219\n\tstorj.io/common/rpc.Dialer.DialAddressID:138\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatelliteOnce:154\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatellite:124\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:103\n\tstorj.io/common/sync2.(*Cycle).Run:147\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

2020-02-12T19:42:27.197Z INFO bandwidth Performing bandwidth usage rollups

2020-02-12T19:42:44.681Z INFO orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 sending {"count": 6}

2020-02-12T19:42:44.681Z INFO orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S sending {"count": 5}

2020-02-12T19:42:44.681Z INFO orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs sending {"count": 15}

2020-02-12T19:42:47.198Z ERROR version Failed to do periodic version check: version control client error: Get https://version.storj.io: dial tcp: lookup version.storj.io on 192.168.178.1:53: read udp 172.17.0.2:49584->192.168.178.1:53: i/o timeout

2020-02-12T19:43:04.686Z INFO orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs finished

2020-02-12T19:43:04.686Z ERROR orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs failed to settle orders {"error": "order: unable to connect to the satellite: rpccompat: dial tcp: lookup europe-west-1.tardigrade.io on 192.168.178.1:53: read udp 172.17.0.2:44738->192.168.178.1:53: i/o timeout", "errorVerbose": "order: unable to connect to the satellite: rpccompat: dial tcp: lookup europe-west-1.tardigrade.io on 192.168.178.1:53: read udp 172.17.0.2:44738->192.168.178.1:53: i/o timeout\n\tstorj.io/storj/storagenode/orders.(*Service).settle:281\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:220\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders.func2:199\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

2020-02-12T19:43:04.685Z INFO orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S finished

2020-02-12T19:43:04.687Z ERROR orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S failed to settle orders {"error": "order: unable to connect to the satellite: rpccompat: dial tcp: lookup us-central-1.tardigrade.io on 192.168.178.1:53: read udp 172.17.0.2:49931->192.168.178.1:53: i/o timeout", "errorVerbose": "order: unable to connect to the satellite: rpccompat: dial tcp: lookup us-central-1.tardigrade.io on 192.168.178.1:53: read udp 172.17.0.2:49931->192.168.178.1:53: i/o timeout\n\tstorj.io/storj/storagenode/orders.(*Service).settle:281\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:220\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders.func2:199\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

2020-02-12T19:43:04.686Z INFO orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 finished

2020-02-12T19:43:04.687Z ERROR orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 failed to settle orders {"error": "order: unable to connect to the satellite: rpccompat: dial tcp: lookup asia-east-1.tardigrade.io on 192.168.178.1:53: read udp 172.17.0.2:42175->192.168.178.1:53: i/o timeout", "errorVerbose": "order: unable to connect to the satellite: rpccompat: dial tcp: lookup asia-east-1.tardigrade.io on 192.168.178.1:53: read udp 172.17.0.2:42175->192.168.178.1:53: i/o timeout\n\tstorj.io/storj/storagenode/orders.(*Service).settle:281\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:220\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders.func2:199\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

2020-02-12T19:44:22.950Z ERROR contact:chore ping satellite failed {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "attempts": 10, "error": "ping satellite error: rpccompat: context deadline exceeded", "errorVerbose": "ping satellite error: rpccompat: context deadline exceeded\n\tstorj.io/common/rpc.Dialer.dialTransport:242\n\tstorj.io/common/rpc.Dialer.dial:219\n\tstorj.io/common/rpc.Dialer.DialAddressID:138\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatelliteOnce:154\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatellite:124\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:103\n\tstorj.io/common/sync2.(*Cycle).Run:147\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

2020-02-12T19:44:22.952Z ERROR contact:chore ping satellite failed {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 10, "error": "ping satellite error: rpccompat: context deadline exceeded", "errorVerbose": "ping satellite error: rpccompat: context deadline exceeded\n\tstorj.io/common/rpc.Dialer.dialTransport:242\n\tstorj.io/common/rpc.Dialer.dial:219\n\tstorj.io/common/rpc.Dialer.DialAddressID:138\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatelliteOnce:154\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatellite:124\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:103\n\tstorj.io/common/sync2.(*Cycle).Run:147\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

2020-02-12T19:44:22.954Z ERROR contact:chore ping satellite failed {"Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "attempts": 10, "error": "ping satellite error: rpccompat: context deadline exceeded", "errorVerbose": "ping satellite error: rpccompat: context deadline exceeded\n\tstorj.io/common/rpc.Dialer.dialTransport:242\n\tstorj.io/common/rpc.Dialer.dial:219\n\tstorj.io/common/rpc.Dialer.DialAddressID:138\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatelliteOnce:154\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatellite:124\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:103\n\tstorj.io/common/sync2.(*Cycle).Run:147\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

2020-02-12T19:44:33.511Z ERROR contact:chore ping satellite failed {"Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "attempts": 10, "error": "ping satellite error: rpccompat: context deadline exceeded", "errorVerbose": "ping satellite error: rpccompat: context deadline exceeded\n\tstorj.io/common/rpc.Dialer.dialTransport:242\n\tstorj.io/common/rpc.Dialer.dial:219\n\tstorj.io/common/rpc.Dialer.DialAddressID:138\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatelliteOnce:154\n\tstorj.io/storj/storagenode/contact.(*Chore).pingSatellite:124\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:103\n\tstorj.io/common/sync2.(*Cycle).Run:147\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

For some reasons i cannot post code here, it’s protected by the spam filter…
Hopefully my post will be released by the staff :wink:

This is probably because the output contains URLs. A mod should release your post shortly. When the node goes offline, can you still access the dashboard? Or is it completely down where the container has exited or is in a restart loop? Is the storage location an external USB drive connected to your pi?

Do you have any kind of firewall or antivirus software blocking these connections?

Yes i can still access the dashboard, the node ist just marked as offline there.
My disk is connected with usb.
Btw: My Pi is a model 4 with 2gb ram

nothing, just using rapsbian buster lite.
I have 5 more nodes running with firewalls too but not such strange issues there

did you made port forwarding on firewall?

Yes i’ve forwarded the necessary ports - As i said already the node is working after a restart for around one hour since two days

It looks like your node cannot resolve hosts. Can you run docker exec -it storagenode ping -c 10 google.com and see if it can successfully resolve the host from within the container? This might be more useful to do when you are experiencing the issue.

I can do it while the error occurs again - But it did that already today - I had exactly the same thoughts as i’ve seen the log entry with the dns request and the potential resolution error…
I can confirm that it was possible to resolve google.com :smile:

Okay. It sounds like you are an experienced user so you have probably already considered this, but if you are running 5 nodes in the same location (an assumption), have you double checked they are all on different external ports?

Im running 5 nodes on different locations with different external ip addresses so i dont need to double check that

1 Like

One other thing I would also try is running a traceroute from inside the container when you have the issue. This could tell you if traffic is getting stuck inside or outside of the container.

i have seen this on windows gui here, reinstal was helped, but docker not know.

looks like may be some issue incide node.

Not reinstall, just reset of dns cache…
yes, when you reinstall it’s resetting too

on other threan, we tried to flush dns but it not helped. on windows.

I’ve changed the dns server from google to cloudflare yesterday, seems like its more stable - uptime now around 9 hours

The node is again offline…
Here is the output i’ve got

Storage Node Dashboard ( Node Version: v0.31.12 )

======================

ID           12B6nqxC8rgMDGJQS7piieqwq6tk8y83vu167a7ryeuELX49DE2
Last Contact OFFLINE
Uptime       22h0m26s

                   Available        Used     Egress     Ingress
     Bandwidth        2.0 PB     15.7 GB     2.0 GB     13.7 GB (since Feb 1)
          Disk        5.5 TB      5.9 GB
Internal 127.0.0.1:7778
External storjnode2.ddnss.de:28967
^C2020-02-13T20:19:06.978Z	INFO	Got a signal from the OS: "interrupt"
Error: context canceled
root@storjnode:~# docker exec -it storagenode ping -c 10 google.com
PING google.com (172.217.23.78): 56 data bytes
64 bytes from 172.217.23.78: seq=0 ttl=56 time=8.574 ms
64 bytes from 172.217.23.78: seq=1 ttl=56 time=7.127 ms
64 bytes from 172.217.23.78: seq=2 ttl=56 time=7.028 ms
64 bytes from 172.217.23.78: seq=3 ttl=56 time=7.044 ms
^C
--- google.com ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 7.028/7.443/8.574 ms
root@storjnode:~# docker exec -it storagenode traceroute  google.com
traceroute to google.com (172.217.23.78), 30 hops max, 38 byte packets
 1  172.17.0.1 (172.17.0.1)  0.020 ms  0.025 ms  0.020 ms
 2  wpad.fritz.box (192.168.178.1)  1.109 ms  1.284 ms  0.870 ms
 3  dslb-084-059-211-001.084.059.pools.vodafone-ip.de (84.59.211.1)  6.571 ms  6.815 ms  6.663 ms
 4  88.79.10.240 (88.79.10.240)  8.398 ms  88.79.10.238 (88.79.10.238)  6.726 ms  6.839 ms
 5  188.111.216.112 (188.111.216.112)  7.330 ms  188.111.216.100 (188.111.216.100)  6.977 ms  188.111.216.112 (188.111.216.112)  7.712 ms
 6  92.79.202.90 (92.79.202.90)  8.841 ms  92.79.215.16 (92.79.215.16)  8.135 ms  8.323 ms
 7  145.254.2.201 (145.254.2.201)  8.443 ms  8.526 ms  145.254.2.185 (145.254.2.185)  7.836 ms
 8  72.14.214.170 (72.14.214.170)  7.587 ms  8.314 ms  7.986 ms