Hello folks,
I’ve been trying to improve my nodes’ vulnerability against electricity outages (sometimes, some of the folders owner changes right after electricity outages, I need to fix such issues by using chown username:username command). To prevent this issue HDD driver’s hold-up time needs to be increased (I have simple 12V/2A adapter for my USB to Sata driver). While trying my simple resettable fuse + 6mF capacitor buffer circuit the HDD could not start. And, after taking off that intermediate board from the system, node started giving below output:
09:49:04 username@raspberrypi scripts → docker logs --tail 50 storagenode1
2024-09-09T18:49:00Z ERROR pieces used-space-filewalker failed {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Lazy File Walker": false, "error": "filewalker: context canceled", "errorVerbose": "filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:79\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkAndComputeSpaceUsedBySatellite:731\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:81\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-09-09T18:49:00Z ERROR piecestore:cache encountered error while computing space used by satellite {"Process": "storagenode", "error": "filewalker: context canceled", "errorVerbose": "filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:79\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkAndComputeSpaceUsedBySatellite:731\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:81\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "SatelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-09-09T18:49:00Z ERROR piecestore:cache error getting current used space for trash: {"Process": "storagenode", "error": "filestore error: failed to walk trash namespace f474535a19db00db4f8071a1be6c2551f4ded6a6e38f0818c68c68d000000000: context canceled", "errorVerbose": "filestore error: failed to walk trash namespace f474535a19db00db4f8071a1be6c2551f4ded6a6e38f0818c68c68d000000000: context canceled\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).SpaceUsedForTrash:273\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:100\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-09-09T18:49:02Z ERROR failure during run {"Process": "storagenode", "error": "piecestore monitor: error verifying location and/or readability of storage directory: node ID in file (12HFEBiqo4Rkcv9VNHvRxVqMf8VfobtZCQj48b8YzfpnKTnyouW) does not match running node's ID (125ErUdwQuETNYrMWXGuWCp5sGLoAvrAP4mfjhvFyYHmNriSZ66)", "errorVerbose": "piecestore monitor: error verifying location and/or readability of storage directory: node ID in file (12HFEBiqo4Rkcv9VNHvRxVqMf8VfobtZCQj48b8YzfpnKTnyouW) does not match running node's ID (125ErUdwQuETNYrMWXGuWCp5sGLoAvrAP4mfjhvFyYHmNriSZ66)\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func1.1:159\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func1:140\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
Error: piecestore monitor: error verifying location and/or readability of storage directory: node ID in file (12HFEBiqo4Rkcv9VNHvRxVqMf8VfobtZCQj48b8YzfpnKTnyouW) does not match running node's ID (125ErUdwQuETNYrMWXGuWCp5sGLoAvrAP4mfjhvFyYHmNriSZ66)
2024-09-09 18:49:02,547 INFO exited: storagenode (exit status 1; not expected)
2024-09-09 18:49:03,552 INFO spawned: 'storagenode' with pid 52
2024-09-09 18:49:03,553 WARN received SIGQUIT indicating exit request
2024-09-09 18:49:03,554 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2024-09-09T18:49:03Z INFO Got a signal from the OS: "terminated" {"Process": "storagenode-updater"}
2024-09-09 18:49:03,559 INFO stopped: storagenode-updater (exit status 0)
2024-09-09T18:49:03Z INFO Configuration loaded {"Process": "storagenode", "Location": "/app/config/config.yaml"}
2024-09-09T18:49:03Z INFO Anonymized tracing enabled {"Process": "storagenode"}
2024-09-09T18:49:03Z INFO Operator email {"Process": "storagenode", "Address": "nazimyildiz90@gmail.com"}
2024-09-09T18:49:03Z INFO Operator wallet {"Process": "storagenode", "Address": "0x412c63a480cfcc6a3bd771cc33a181ba8e098067"}
2024-09-09T18:49:03Z INFO server kernel support for server-side tcp fast open remains disabled. {"Process": "storagenode"}
2024-09-09T18:49:03Z INFO server enable with: sysctl -w net.ipv4.tcp_fastopen=3 {"Process": "storagenode"}
2024-09-09T18:49:04Z INFO Telemetry enabled {"Process": "storagenode", "instance ID": "125ErUdwQuETNYrMWXGuWCp5sGLoAvrAP4mfjhvFyYHmNriSZ66"}
2024-09-09T18:49:04Z INFO Event collection enabled {"Process": "storagenode", "instance ID": "125ErUdwQuETNYrMWXGuWCp5sGLoAvrAP4mfjhvFyYHmNriSZ66"}
2024-09-09T18:49:04Z INFO db.migration Database Version {"Process": "storagenode", "version": 61}
2024-09-09T18:49:05Z INFO preflight:localtime start checking local system clock with trusted satellites' system clock. {"Process": "storagenode"}
2024-09-09T18:49:06Z INFO preflight:localtime local system clock is in sync with trusted satellites' system clock. {"Process": "storagenode"}
2024-09-09T18:49:06Z INFO Node 125ErUdwQuETNYrMWXGuWCp5sGLoAvrAP4mfjhvFyYHmNriSZ66 started {"Process": "storagenode"}
2024-09-09T18:49:06Z INFO Public server started on [::]:28967 {"Process": "storagenode"}
2024-09-09T18:49:06Z INFO Private server started on 127.0.0.1:7778 {"Process": "storagenode"}
2024-09-09T18:49:06Z INFO failed to sufficiently increase send buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details. {"Process": "storagenode"}
2024-09-09T18:49:06Z INFO bandwidth Persisting bandwidth usage cache to db {"Process": "storagenode"}
2024-09-09T18:49:06Z INFO pieces:trash emptying trash started {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-09-09T18:49:06Z INFO lazyfilewalker.trash-cleanup-filewalker starting subprocess {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-09-09T18:49:06Z INFO collector expired pieces collection started {"Process": "storagenode"}
2024-09-09T18:49:06Z INFO trust Scheduling next refresh {"Process": "storagenode", "after": "5h14m57.015154482s"}
2024-09-09T18:49:06Z INFO lazyfilewalker.trash-cleanup-filewalker subprocess started {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-09-09T18:49:06Z ERROR services unexpected shutdown of a runner {"Process": "storagenode", "name": "piecestore:monitor", "error": "piecestore monitor: error verifying location and/or readability of storage directory: node ID in file (12HFEBiqo4Rkcv9VNHvRxVqMf8VfobtZCQj48b8YzfpnKTnyouW) does not match running node's ID (125ErUdwQuETNYrMWXGuWCp5sGLoAvrAP4mfjhvFyYHmNriSZ66)", "errorVerbose": "piecestore monitor: error verifying location and/or readability of storage directory: node ID in file (12HFEBiqo4Rkcv9VNHvRxVqMf8VfobtZCQj48b8YzfpnKTnyouW) does not match running node's ID (125ErUdwQuETNYrMWXGuWCp5sGLoAvrAP4mfjhvFyYHmNriSZ66)\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func1.1:159\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func1:140\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-09-09T18:49:06Z ERROR version failed to get process version info {"Process": "storagenode", "error": "version checker client: Get \"https://version.storj.io\": context canceled", "errorVerbose": "version checker client: Get \"https://version.storj.io\": context canceled\n\tstorj.io/storj/private/version/checker.(*Client).All:68\n\tstorj.io/storj/private/version/checker.(*Client).Process:89\n\tstorj.io/storj/private/version/checker.(*Service).checkVersion:104\n\tstorj.io/storj/private/version/checker.(*Service).CheckVersion:78\n\tstorj.io/storj/storagenode/version.(*Chore).checkVersion:115\n\tstorj.io/storj/storagenode/version.(*Chore).RunOnce:71\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/storj/storagenode/version.(*Chore).Run:64\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-09-09T18:49:06Z ERROR piecestore:cache error during init space usage db: {"Process": "storagenode", "error": "piece space used: context canceled", "errorVerbose": "piece space used: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceSpaceUsedDB).Init:55\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:60\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-09-09T18:49:06Z ERROR nodestats:cache Get pricing-model/join date failed {"Process": "storagenode", "error": "context canceled"}
2024-09-09T18:49:06Z ERROR gracefulexit:chore error retrieving satellites. {"Process": "storagenode", "error": "satellitesdb: context canceled", "errorVerbose": "satellitesdb: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*satellitesDB).ListGracefulExits:197\n\tstorj.io/storj/storagenode/gracefulexit.(*Service).ListPendingExits:59\n\tstorj.io/storj/storagenode/gracefulexit.(*Chore).AddMissing:55\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/storj/storagenode/gracefulexit.(*Chore).Run:48\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-09-09T18:49:06Z ERROR contact:service ping satellite failed {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 1, "error": "ping satellite: rpc: tcp connector failed: rpc: dial tcp: lookup ap1.storj.io: operation was canceled", "errorVerbose": "ping satellite: rpc: tcp connector failed: rpc: dial tcp: lookup ap1.storj.io: operation was canceled\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2024-09-09T18:49:06Z INFO contact:service context cancelled {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-09-09T18:49:06Z INFO lazyfilewalker.trash-cleanup-filewalker subprocess exited with status {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "status": -1, "error": "signal: killed"}
2024-09-09T18:49:06Z ERROR collector error during expired pieces collection {"Process": "storagenode", "count": 0, "error": "pieces error: context canceled", "errorVerbose": "pieces error: context canceled\n\tstorj.io/storj/storagenode/pieces.(*Store).GetExpiredBatchSkipV0:614\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:99\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:68\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/storj/storagenode/collector.(*Service).Run:64\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-09-09T18:49:06Z ERROR pieces:trash emptying trash failed {"Process": "storagenode", "error": "pieces error: lazyfilewalker: signal: killed", "errorVerbose": "pieces error: lazyfilewalker: signal: killed\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:85\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkCleanupTrash:195\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:436\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1.1:84\n\tstorj.io/common/sync2.(*Workplace).Start.func1:89"}
2024-09-09T18:49:06Z ERROR collector error during collecting pieces: {"Process": "storagenode", "error": "pieces error: context canceled", "errorVerbose": "pieces error: context canceled\n\tstorj.io/storj/storagenode/pieces.(*Store).GetExpiredBatchSkipV0:614\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:99\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:68\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/storj/storagenode/collector.(*Service).Run:64\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-09-09T18:49:06Z ERROR gracefulexit:blobscleaner couldn't receive satellite's GE status {"Process": "storagenode", "error": "context canceled"}
2024-09-09T18:49:06Z ERROR contact:service ping satellite failed {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "attempts": 1, "error": "ping satellite: rpc: tcp connector failed: rpc: dial tcp: lookup eu1.storj.io: operation was canceled", "errorVerbose": "ping satellite: rpc: tcp connector failed: rpc: dial tcp: lookup eu1.storj.io: operation was canceled\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2024-09-09T18:49:06Z INFO contact:service context cancelled {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-09-09T18:49:06Z ERROR contact:service ping satellite failed {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 1, "error": "ping satellite: rpc: tcp connector failed: rpc: dial tcp: lookup saltlake.tardigrade.io: operation was canceled", "errorVerbose": "ping satellite: rpc: tcp connector failed: rpc: dial tcp: lookup saltlake.tardigrade.io: operation was canceled\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2024-09-09T18:49:06Z INFO contact:service context cancelled {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-09-09T18:49:06Z ERROR contact:service ping satellite failed {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "attempts": 1, "error": "ping satellite: rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io: operation was canceled", "errorVerbose": "ping satellite: rpc: tcp connector failed: rpc: dial tcp: lookup us1.storj.io: operation was canceled\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2024-09-09T18:49:06Z INFO contact:service context cancelled {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
Seems like some of the node’s files were changed or broken during my experimental work.
Where I can find my actual node’s identity number, is it inside the identity.key or is it hidden inside the Authorization token?
So far:
- The node container was removed and started again by pulling the docker image…
- I’ve checked the .local/share/storj/identity/storagenode folder to get node identity number by I could not see it there
- The disk seems healthy but I’ll execute fsck to be %100 sure
Thank you very much for any hints.