Node v1.56.4 refuse to start

Node Operators troubleshooting
1

Hi,

Since monday 13th at 17h00 UTC, my node is shut down and refuses to start.
My node (v1.56.4) is running on rpi 4 8GB (120 GB SSD), 12 TB HDD for storage and docker.
I haven’t done anything on it or the hardware or anything else.

I stopped and deleted the storagenode container but i get error to save trust.json and to contact satellite.
I try to change owner of my storj folder (the change stills happen) because now, there is the parameter : “–user $(id -u):$(id -g)”

I checked the ports :

  • 14002 is open
  • 28967 (tcp/udp) and 7777 are closed.
    On my router, all ports are forwarded and open. I disable the firewall on the rpi.

I restarted several times the rpi, node, router but nothing is better.

Could a kind soul help me please ?

Regards

Log :

docker logs --tail 200 storagenode
  6850K .......... .......... .......... .......... .......... 48% 14.5M 1s
  (...)
 14350K .......... .......... ...                             100% 22.5M=1.2s
2022-06-15 18:15:45 (11.9 MB/s) - '/tmp/storagenode.zip' saved [14718174/14718174]
2022-06-15 18:15:46,698 INFO RPC interface 'supervisor' initialized
2022-06-15 18:15:46,700 INFO supervisord started with pid 1
2022-06-15 18:15:47,709 INFO spawned: 'processes-exit-eventlistener' with pid 56
2022-06-15 18:15:47,723 INFO spawned: 'storagenode' with pid 57
2022-06-15 18:15:47,737 INFO spawned: 'storagenode-updater' with pid 58
2022-06-15T18:15:47.855Z        INFO    Invalid configuration file value for key        {"Process": "storagenode-updater", "Key": "log.level"}
2022-06-15T18:15:47.856Z        INFO    Invalid configuration file value for key        {"Process": "storagenode-updater", "Key": "log.output"}
2022-06-15T18:15:47.856Z        INFO    Invalid configuration file value for key        {"Process": "storagenode-updater", "Key": "log.stack"}
2022-06-15T18:15:47.856Z        INFO    Invalid configuration file value for key        {"Process": "storagenode-updater", "Key": "log.development"}
2022-06-15T18:15:47.860Z        INFO    Running on version      {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.56.4"}
2022-06-15T18:15:47.861Z        INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2022-06-15T18:15:47.939Z        INFO    Operator email  {"Process": "storagenode", "Address": "my-email"}
2022-06-15T18:15:47.939Z        INFO    Operator wallet {"Process": "storagenode", "Address": "my-wallet"}
2022-06-15T18:15:48.325Z        INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.56.4"}
2022-06-15T18:15:48.325Z        INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2022-06-15T18:15:48.353Z        INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.56.4"}
2022-06-15T18:15:48.353Z        INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2022-06-15 18:15:49,355 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-06-15 18:15:49,355 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-06-15 18:15:49,356 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-06-15T18:15:49.368Z        INFO    Telemetry enabled       {"Process": "storagenode", "instance ID": "1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A"}
2022-06-15T18:15:49.530Z        INFO    db.migration    Database Version        {"Process": "storagenode", "version": 53}
2022-06-15T18:16:08.440Z        WARN    trust   Unable to save list cache       {"Process": "storagenode", "error": "open config/trust-cache.json4069946084: permission denied", "errorVerbose": "open config/trust-cache.json4069946084: permission denied\n\tstorj.io/common/fpath.AtomicWriteFile:21\n\tstorj.io/storj/storagenode/trust.SaveCacheData:115\n\tstorj.io/storj/storagenode/trust.(*Cache).Save:72\n\tstorj.io/storj/storagenode/trust.(*List).saveCache:130\n\tstorj.io/storj/storagenode/trust.(*List).fetchEntries:107\n\tstorj.io/storj/storagenode/trust.(*List).FetchURLs:49\n\tstorj.io/storj/storagenode/trust.(*Pool).fetchURLs:250\n\tstorj.io/storj/storagenode/trust.(*Pool).Refresh:187\n\tstorj.io/storj/storagenode.(*Peer).Run:871\n\tmain.cmdRun:251\n\tstorj.io/private/process.cleanup.func1.4:372\n\tstorj.io/private/process.cleanup.func1:390\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:93\n\tmain.main:479\n\truntime.main:255"}
2022-06-15T18:16:08.442Z        INFO    preflight:localtime     start checking local system clock with trusted satellites' system clock.        {"Process": "storagenode"}
2022-06-15T18:16:09.162Z        INFO    preflight:localtime     local system clock is in sync with trusted satellites' system clock.    {"Process": "storagenode"}
2022-06-15T18:16:09.163Z        INFO    Node 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A started {"Process": "storagenode"}
2022-06-15T18:16:09.164Z        INFO    Public server started on [::]:7777      {"Process": "storagenode"}
2022-06-15T18:16:09.164Z        INFO    Private server started on 127.0.0.1:7778        {"Process": "storagenode"}
2022-06-15T18:16:09.164Z        INFO    failed to sufficiently increase receive buffer size (was: 176 kiB, wanted: 2048 kiB, got: 352 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.  {"Process": "storagenode"}
2022-06-15T18:16:14.242Z        WARN    contact:service failed PingMe request to satellite      {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "error": "ping satellite: check-in network: failed to ping node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address: my-url:28967, err: contact: failed to dial storage node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address my-url:28967 using QUIC: rpc: quic: timeout: no recent network activity", "errorVerbose": "ping satellite: check-in network: failed to ping node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address: my-url:28967, err: contact: failed to dial storage node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address my-url:28967 using QUIC: rpc: quic: timeout: no recent network activity\n\tstorj.io/storj/storagenode/contact.(*Service).requestPingMeOnce:194\n\tstorj.io/storj/storagenode/contact.(*Service).RequestPingMeQUIC:167\n\tstorj.io/storj/storagenode.(*Peer).addConsoleService:845\n\tstorj.io/storj/storagenode.(*Peer).Run:884\n\tmain.cmdRun:251\n\tstorj.io/private/process.cleanup.func1.4:372\n\tstorj.io/private/process.cleanup.func1:390\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:93\n\tmain.main:479\n\truntime.main:255"}
2022-06-15T18:16:19.461Z        WARN    contact:service failed PingMe request to satellite      {"Process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "error": "ping satellite: check-in network: failed to ping node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address: my-url:28967, err: contact: failed to dial storage node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address my-url:28967 using QUIC: rpc: quic: timeout: no recent network activity", "errorVerbose": "ping satellite: check-in network: failed to ping node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address: my-url:28967, err: contact: failed to dial storage node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address my-url:28967 using QUIC: rpc: quic: timeout: no recent network activity\n\tstorj.io/storj/storagenode/contact.(*Service).requestPingMeOnce:194\n\tstorj.io/storj/storagenode/contact.(*Service).RequestPingMeQUIC:167\n\tstorj.io/storj/storagenode.(*Peer).addConsoleService:845\n\tstorj.io/storj/storagenode.(*Peer).Run:884\n\tmain.cmdRun:251\n\tstorj.io/private/process.cleanup.func1.4:372\n\tstorj.io/private/process.cleanup.func1:390\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:93\n\tmain.main:479\n\truntime.main:255"}
2022-06-15T18:16:24.808Z        WARN    contact:service failed PingMe request to satellite      {"Process": "storagenode", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "error": "ping satellite: check-in network: failed to ping node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address: my-url:28967, err: contact: failed to dial storage node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address my-url:28967 using QUIC: rpc: quic: timeout: no recent network activity", "errorVerbose": "ping satellite: check-in network: failed to ping node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address: my-url:28967, err: contact: failed to dial storage node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address my-url:28967 using QUIC: rpc: quic: timeout: no recent network activity\n\tstorj.io/storj/storagenode/contact.(*Service).requestPingMeOnce:194\n\tstorj.io/storj/storagenode/contact.(*Service).RequestPingMeQUIC:167\n\tstorj.io/storj/storagenode.(*Peer).addConsoleService:845\n\tstorj.io/storj/storagenode.(*Peer).Run:884\n\tmain.cmdRun:251\n\tstorj.io/private/process.cleanup.func1.4:372\n\tstorj.io/private/process.cleanup.func1:390\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:93\n\tmain.main:479\n\truntime.main:255"}
2022-06-15T18:16:30.322Z        WARN    contact:service failed PingMe request to satellite      {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "error": "ping satellite: check-in network: failed to ping node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address: my-url:28967, err: contact: failed to dial storage node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address my-url:28967 using QUIC: rpc: quic: timeout: no recent network activity", "errorVerbose": "ping satellite: check-in network: failed to ping node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address: my-url:28967, err: contact: failed to dial storage node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address my-url:28967 using QUIC: rpc: quic: timeout: no recent network activity\n\tstorj.io/storj/storagenode/contact.(*Service).requestPingMeOnce:194\n\tstorj.io/storj/storagenode/contact.(*Service).RequestPingMeQUIC:167\n\tstorj.io/storj/storagenode.(*Peer).addConsoleService:845\n\tstorj.io/storj/storagenode.(*Peer).Run:884\n\tmain.cmdRun:251\n\tstorj.io/private/process.cleanup.func1.4:372\n\tstorj.io/private/process.cleanup.func1:390\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:93\n\tmain.main:479\n\truntime.main:255"}
2022-06-15T18:16:35.643Z        WARN    contact:service failed PingMe request to satellite      {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "error": "ping satellite: check-in network: failed to ping node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address: my-url:28967, err: contact: failed to dial storage node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address my-url:28967 using QUIC: rpc: quic: timeout: no recent network activity", "errorVerbose": "ping satellite: check-in network: failed to ping node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address: my-url:28967, err: contact: failed to dial storage node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address my-url:28967 using QUIC: rpc: quic: timeout: no recent network activity\n\tstorj.io/storj/storagenode/contact.(*Service).requestPingMeOnce:194\n\tstorj.io/storj/storagenode/contact.(*Service).RequestPingMeQUIC:167\n\tstorj.io/storj/storagenode.(*Peer).addConsoleService:845\n\tstorj.io/storj/storagenode.(*Peer).Run:884\n\tmain.cmdRun:251\n\tstorj.io/private/process.cleanup.func1.4:372\n\tstorj.io/private/process.cleanup.func1:390\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:93\n\tmain.main:479\n\truntime.main:255"}
2022-06-15T18:16:41.453Z        WARN    contact:service failed PingMe request to satellite      {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "error": "ping satellite: check-in network: failed to ping node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address: my-url:28967, err: contact: failed to dial storage node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address my-url:28967 using QUIC: rpc: quic: timeout: no recent network activity", "errorVerbose": "ping satellite: check-in network: failed to ping node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address: my-url:28967, err: contact: failed to dial storage node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address my-url:28967 using QUIC: rpc: quic: timeout: no recent network activity\n\tstorj.io/storj/storagenode/contact.(*Service).requestPingMeOnce:194\n\tstorj.io/storj/storagenode/contact.(*Service).RequestPingMeQUIC:167\n\tstorj.io/storj/storagenode.(*Peer).addConsoleService:845\n\tstorj.io/storj/storagenode.(*Peer).Run:884\n\tmain.cmdRun:251\n\tstorj.io/private/process.cleanup.func1.4:372\n\tstorj.io/private/process.cleanup.func1:390\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:93\n\tmain.main:479\n\truntime.main:255"}
2022-06-15T18:16:41.454Z        WARN    failed QUIC check       {"Process": "storagenode", "error": "ping satellite: failed to ping storage node using QUIC: ping satellite: check-in network: failed to ping node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address: my-url:28967, err: contact: failed to dial storage node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address my-url:28967 using QUIC: rpc: quic: timeout: no recent network activity", "errorVerbose": "ping satellite: failed to ping storage node using QUIC: ping satellite: check-in network: failed to ping node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address: my-url:28967, err: contact: failed to dial storage node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address my-url:28967 using QUIC: rpc: quic: timeout: no recent network activity\n\tstorj.io/storj/storagenode/contact.(*Service).RequestPingMeQUIC:176\n\tstorj.io/storj/storagenode.(*Peer).addConsoleService:845\n\tstorj.io/storj/storagenode.(*Peer).Run:884\n\tmain.cmdRun:251\n\tstorj.io/private/process.cleanup.func1.4:372\n\tstorj.io/private/process.cleanup.func1:390\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:93\n\tmain.main:479\n\truntime.main:255"}
2022-06-15T18:16:41.456Z        INFO    trust   Scheduling next refresh {"Process": "storagenode", "after": "5h54m20.103175038s"}
2022-06-15T18:16:41.459Z        INFO    bandwidth       Performing bandwidth usage rollups      {"Process": "storagenode"}
2022-06-15T18:16:41.593Z        ERROR   contact:service ping satellite failed   {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "attempts": 1, "error": "ping satellite: failed to dial storage node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address my-url:28967: rpc: tcp connector failed: rpc: dial tcp my-ip:28967: connect: connection refused", "errorVerbose": "ping satellite: failed to dial storage node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address my-url:28967: rpc: tcp connector failed: rpc: dial tcp my-ip:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:98\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2022-06-15T18:16:41.694Z        ERROR   contact:service ping satellite failed   {"Process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 1, "error": "ping satellite: failed to dial storage node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address my-url:28967: rpc: tcp connector failed: rpc: dial tcp my-ip:28967: connect: connection refused", "errorVerbose": "ping satellite: failed to dial storage node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address my-url:28967: rpc: tcp connector failed: rpc: dial tcp my-ip:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:98\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2022-06-15T18:16:41.909Z        ERROR   contact:service ping satellite failed   {"Process": "storagenode", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "attempts": 1, "error": "ping satellite: failed to dial storage node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address my-url:28967: rpc: tcp connector failed: rpc: dial tcp my-ip:28967: connect: connection refused", "errorVerbose": "ping satellite: failed to dial storage node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address my-url:28967: rpc: tcp connector failed: rpc: dial tcp my-ip:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:98\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2022-06-15T18:16:42.026Z        ERROR   contact:service ping satellite failed   {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "attempts": 1, "error": "ping satellite: failed to dial storage node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address my-url:28967: rpc: tcp connector failed: rpc: dial tcp my-ip:28967: connect: connection refused", "errorVerbose": "ping satellite: failed to dial storage node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address my-url:28967: rpc: tcp connector failed: rpc: dial tcp my-ip:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:98\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2022-06-15T18:16:42.118Z        ERROR   contact:service ping satellite failed   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 1, "error": "ping satellite: failed to dial storage node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address my-url:28967: rpc: tcp connector failed: rpc: dial tcp my-ip:28967: connect: connection refused", "errorVerbose": "ping satellite: failed to dial storage node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address my-url:28967: rpc: tcp connector failed: rpc: dial tcp my-ip:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:98\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2022-06-15T18:16:42.584Z        ERROR   contact:service ping satellite failed   {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 1, "error": "ping satellite: failed to dial storage node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address my-url:28967: rpc: tcp connector failed: rpc: dial tcp my-ip:28967: connect: connection refused", "errorVerbose": "ping satellite: failed to dial storage node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address my-url:28967: rpc: tcp connector failed: rpc: dial tcp my-ip:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:98\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2022-06-15T18:16:42.734Z        ERROR   contact:service ping satellite failed   {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "attempts": 2, "error": "ping satellite: failed to dial storage node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address my-url:28967: rpc: tcp connector failed: rpc: dial tcp my-ip:28967: connect: connection refused", "errorVerbose": "ping satellite: failed to dial storage node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address my-url:28967: rpc: tcp connector failed: rpc: dial tcp my-ip:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:98\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2022-06-15T18:16:42.904Z        ERROR   contact:service ping satellite failed   {"Process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 2, "error": "ping satellite: failed to dial storage node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address my-url:28967: rpc: tcp connector failed: rpc: dial tcp my-ip:28967: connect: connection refused", "errorVerbose": "ping satellite: failed to dial storage node (ID: 1VjbsodmwtNB8xNDbX2XeFQJtJjzTXBZXY1nCgejso3zQwGi9A) at address my-url:28967: rpc: tcp connector failed: rpc: dial tcp my-ip:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:98\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2

Hi,

What does your node’s dashboard say?

Can you provide the command to start your node (anonymised)?

Connectivity to satellites seems off: any possible reasons why (router configuration, proxy, firewall…)?

I’m not sure what could be wrong with the trust-cache thingy if you already double checked permissions :thinking:

3

Hi @Pac ,

The dashboard doesn’t say more (i think) :

node
node878×1078 60 KB

This is the docker command :

docker run -d --restart unless-stopped --stop-timeout 300 \
    -p 28967:28967/tcp \
    -p 28967:28967/udp \
    -p 14002:14002 \
    -e WALLET="my-wallet" \
    -e EMAIL="my-email" \
    -e ADDRESS="my-url:28967" \
    -e STORAGE="12TB" \
    --user $(id -u):$(id -g) \
    --mount type=bind,source="/home/myuser/.local/share/storj/identity/storagenode",destination=/app/identity \
    --mount type=bind,source="/mnt/storj",destination=/app/config \
    --name storagenode storjlabs/storagenode:latest

I try with sudo but it is not better.
I hesitate to start from 0 and redo the node setup but it doesn’t seem a good idea.

4

Are you using No-IP for IP management with the hopto.org domain? This service was down for part of the day yesterday and I experienced similar connectivity issues.

1 Like
5

Always double check the ports in case your local ip changed, and or using a ddns service.

6

I’m now tempted to ask Storj Labs whether they have estimated the risk of popular DDNS service providers failure. Like, maybe some statistics on how much total stored data is stored on specific domains?

7

Thank you for this information !
And yes unfortunately, my url uses the hopto.org domain :unamused:

I will change the domain to check.

8

My local ip is static otherwise i can’t forward ports with the provider router. (i check again)

9

A temporary fix would be to skip ddns and use the public up. Hopefully your provider won’t change it so often, until hopto.org is fixed.

1 Like
10

So I changed the url using the ddns.net domain and another time with my public IP but nothing improves.
Only port 14002 is open, the other ports are closed (28967 & 7777).

I don’t know what’s going on.

Log

11

Your storage location either corrupted or have a wrong permissions:

If you run your node as non root, then you need to change owner for data to your user and apply the change recursively, i.e. (replace user:group to your user and group):

sudo chown user:group -R /mnt/storj/storagenode

Then change permissions to full access

chmod ug+wrx -R /mnt/storj/storagenode

And your node is still offline. Check your WAN IP, public IP and resolved IP from your DDNS hostname - they all should match.

12

good morning crew, my storage node has been running since April 7 2021 in windows 10 along with watchtower and in the last 8 hours it has giving me the following error screenshot to follow

error storagenode
error storagenode1696×218 20.1 KB
sorry I picked topic docker V 4.2.8 and it has done this again since I posted

13

You still need to fix the filesystem issue. Please stop and remove the container, then check and fix your disk.
Then please check the path from the error, can you open it and list it or not.

14

thank you between my posting and your response a pending update .net type need a restart the node has not given the error but I did stop and remove the container performed a disk check no errors if the errors continue I follow your final step again thank you

15

Hi @Alexey ,
Sorry for the delay …
I followed your recommandations about permissions, checked IP (WAN, dns);
The warning about list cache is disappeared but it is always the same result : only port 14002 is open and ping fails.

16

Now please check the WAN IP on your router and compare it with Open Port Check Tool - Test Port Forwarding on Your Router, they should match otherwise port forwarding will not work.
Then check your DDNS (replace my-url with your DDNS hostname):

nslookup my-url 8.8.8.8

Should return the same IP as WAN IP and the IP from yougetsignal. If the DDNS hostname is resolved to a different IP - you need to check your DDNS updater and update the DDNS hostname. It’s better to configure DDNS updater on your router (it’s usually in the DDNS section on your router).
If they match - please check the local IP of your device and make sure that your port forwarding rule uses the same local IP of your device.

17

Thank you for your patience.
So i checked my url with nslookup. The public IP matches the IP of my router.
But there is still only port 14002 open.
I disable the firewall on my router and IPV6 but nothing is working.
My port forwarding is always good (all TCP and UDP are forwaded from public IP to my local IP). I have not made any changes to the port forwarding in my router since the beginning of my node.

I don’t know where to look anymore.

18

Please show your port forwarding rules for ports.
Also could you elaborate what do you mean by

Where is it open?

You should not expose this port to the internet - anyone will have an access to your private information, use How to remote access the web dashboard - Storj Docs instead.

Please also check your identity: Step 5. Create an Identity - Storj Docs

19

I don’t know what’s being used to display the dashboard, but a simple basic auth configured via config yaml would be so useful.

1 Like
20

You can use reverse proxy for that. This is a normal behavior, if you want to have a publicly available dashboard any cost with supported authorization method (at least basic authorization). In such a case you can also configure SSL (required to have a certificate, preferable from publicly available CA).

The secured way is to use guide mentioned above or use a multinode dashboard as a client on your device.