Hello guys,
on one of my node I’m having the problem right now, that it doesn’t run and keeps crashing and restarting. It must have been hardware errors, I already unmouted it, ran e2fsck, it fixed alot and now it seems to keep having the problem.
2026-03-16T19:27:53Z INFO hashstore loaded hint file {"process": "storagenode", "satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "store": "s0", "path": "config/storage/hashstore/121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6/s0/meta/hint-000000000000028a", "writable": [847, 846, 848, 845, 832, 829, 826, 77], "largest": 848, "skip": false}
2026-03-16 19:27:54,535 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-03-16 19:27:54,536 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-03-16 19:27:54,536 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-03-16T19:27:56Z INFO hashstore unclean shutdown detected: reconciling logs {"process": "storagenode", "satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "store": "s0"}
2026-03-16T19:28:00Z INFO hashstore loaded hint file {"process": "storagenode", "satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "store": "s1", "path": "config/storage/hashstore/121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6/s1/meta/hint-00000000000001ea", "writable": [627, 628, 625], "largest": 628, "skip": false}
2026-03-16T19:28:02Z INFO hashstore unclean shutdown detected: reconciling logs {"process": "storagenode", "satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "store": "s1"}
2026-03-16T19:28:04Z INFO hashstore hashstore opened successfully {"process": "storagenode", "satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "open_time": "10.71617208s", "logs_skipped": 126, "logs_matched": 11, "logs_mismatched": 0}
2026-03-16T19:28:04Z INFO hashstore loaded hint file {"process": "storagenode", "satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s0", "path": "config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/meta/hint-0000000000000141", "writable": [], "largest": 0, "skip": false}
2026-03-16T19:28:13Z ERROR failure during run {"process": "storagenode", "error": "Failed to create storage node peer: hashstore: read config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/meta/hashtbl-0000000000000145: input/output error\n\tstorj.io/storj/storagenode/hashstore.(*roBigPageCache).ReadRecord:649\n\tstorj.io/storj/storagenode/hashstore.(*HashTbl).Range:320\n\tstorj.io/storj/storagenode/hashstore.(*HashTbl).loadEntries:264\n\tstorj.io/storj/storagenode/hashstore.OpenHashTbl:176\n\tstorj.io/storj/storagenode/hashstore.OpenTable:120\n\tstorj.io/storj/storagenode/hashstore.NewStore:261\n\tstorj.io/storj/storagenode/hashstore.New:110\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:272\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:122\n\tstorj.io/storj/storagenode.New:607\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tmain.main:34\n\truntime.main:285", "errorVerbose": "Failed to create storage node peer: hashstore: read config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/meta/hashtbl-0000000000000145: input/output error\n\tstorj.io/storj/storagenode/hashstore.(*roBigPageCache).ReadRecord:649\n\tstorj.io/storj/storagenode/hashstore.(*HashTbl).Range:320\n\tstorj.io/storj/storagenode/hashstore.(*HashTbl).loadEntries:264\n\tstorj.io/storj/storagenode/hashstore.OpenHashTbl:176\n\tstorj.io/storj/storagenode/hashstore.OpenTable:120\n\tstorj.io/storj/storagenode/hashstore.NewStore:261\n\tstorj.io/storj/storagenode/hashstore.New:110\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:272\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:122\n\tstorj.io/storj/storagenode.New:607\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tmain.main:34\n\truntime.main:285\n\tmain.cmdRun:86\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tmain.main:34\n\truntime.main:285"}
Error: Failed to create storage node peer: hashstore: read config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/meta/hashtbl-0000000000000145: input/output error
storj.io/storj/storagenode/hashstore.(*roBigPageCache).ReadRecord:649
storj.io/storj/storagenode/hashstore.(*HashTbl).Range:320
storj.io/storj/storagenode/hashstore.(*HashTbl).loadEntries:264
storj.io/storj/storagenode/hashstore.OpenHashTbl:176
storj.io/storj/storagenode/hashstore.OpenTable:120
storj.io/storj/storagenode/hashstore.NewStore:261
storj.io/storj/storagenode/hashstore.New:110
storj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:272
storj.io/storj/storagenode/piecestore.NewHashStoreBackend:122
storj.io/storj/storagenode.New:607
main.cmdRun:84
main.newRunCmd.func1:33
storj.io/common/process.InitBeforeExecute.func1.2:389
storj.io/common/process.InitBeforeExecute.func1:407
github.com/spf13/cobra.(*Command).execute:985
github.com/spf13/cobra.(*Command).ExecuteC:1117
github.com/spf13/cobra.(*Command).Execute:1041
storj.io/common/process.ExecWithCustomOptions:115
main.main:34
runtime.main:285
2026-03-16 19:28:13,594 INFO exited: storagenode (exit status 1; not expected)
2026-03-16 19:28:14,598 INFO spawned: 'storagenode' with pid 129
2026-03-16 19:28:14,598 WARN received SIGQUIT indicating exit request
2026-03-16 19:28:14,599 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2026-03-16T19:28:14Z INFO Got a signal from the OS: "terminated" {"process": "storagenode-updater"}
2026-03-16 19:28:14,603 INFO stopped: storagenode-updater (exit status 0)
2026-03-16 19:28:14,604 INFO stopped: storagenode (terminated by SIGTERM)
2026-03-16 19:28:14,605 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)
2026-03-16 19:28:14,976 INFO Set uid to user 0 succeeded
2026-03-16 19:28:14,978 INFO RPC interface 'supervisor' initialized
2026-03-16 19:28:14,978 INFO supervisord started with pid 1
2026-03-16 19:28:15,980 INFO spawned: 'processes-exit-eventlistener' with pid 11
2026-03-16 19:28:15,982 INFO spawned: 'storagenode' with pid 12
2026-03-16 19:28:15,983 INFO spawned: 'storagenode-updater' with pid 13
2026-03-16T19:28:16Z INFO Configuration loaded {"process": "storagenode-updater", "location": "/app/config/config.yaml"}
2026-03-16T19:28:16Z INFO Invalid configuration file key {"process": "storagenode-updater", "key": "operator.wallet-features"}
2026-03-16T19:28:16Z INFO Invalid configuration file key {"process": "storagenode-updater", "key": "pieces.enable-lazy-filewalker"}
2026-03-16T19:28:16Z INFO Invalid configuration file key {"process": "storagenode-updater", "key": "console.address"}
2026-03-16T19:28:16Z INFO Invalid configuration file key {"process": "storagenode-updater", "key": "hashstore.compaction.alive-fraction"}
2026-03-16T19:28:16Z INFO Invalid configuration file key {"process": "storagenode-updater", "key": "operator.email"}
2026-03-16T19:28:16Z INFO Invalid configuration file key {"process": "storagenode-updater", "key": "healthcheck.enabled"}
2026-03-16T19:28:16Z INFO Invalid configuration file key {"process": "storagenode-updater", "key": "contact.external-address"}
2026-03-16T19:28:16Z INFO Invalid configuration file key {"process": "storagenode-updater", "key": "hashstore.compaction.delete-trash-immediately"}
2026-03-16T19:28:16Z INFO Invalid configuration file key {"process": "storagenode-updater", "key": "hashstore.compaction.expires-days"}
2026-03-16T19:28:16Z INFO Invalid configuration file key {"process": "storagenode-updater", "key": "operator.wallet"}
2026-03-16T19:28:16Z INFO Invalid configuration file key {"process": "storagenode-updater", "key": "storage.allocated-disk-space"}
2026-03-16T19:28:16Z INFO Invalid configuration file key {"process": "storagenode-updater", "key": "storage2.trust.exclusions"}
2026-03-16T19:28:16Z INFO Invalid configuration file key {"process": "storagenode-updater", "key": "server.address"}
2026-03-16T19:28:16Z INFO Invalid configuration file key {"process": "storagenode-updater", "key": "storage2.piece-scan-on-startup"}
2026-03-16T19:28:16Z INFO Invalid configuration file key {"process": "storagenode-updater", "key": "storage2migration.interval"}
2026-03-16T19:28:16Z INFO Invalid configuration file key {"process": "storagenode-updater", "key": "storage2.database-dir"}
2026-03-16T19:28:16Z INFO Invalid configuration file key {"process": "storagenode-updater", "key": "server.private-address"}
2026-03-16T19:28:16Z INFO Invalid configuration file key {"process": "storagenode-updater", "key": "healthcheck.details"}
2026-03-16T19:28:16Z INFO Anonymized tracing enabled {"process": "storagenode-updater"}
2026-03-16T19:28:16Z INFO Running on version {"process": "storagenode-updater", "service": "storagenode-updater", "version": "v1.148.3"}
2026-03-16T19:28:16Z INFO Downloading versions. {"process": "storagenode-updater", "server_address": "https://version.storj.io"}
2026-03-16T19:28:16Z INFO Configuration loaded {"process": "storagenode", "location": "/app/config/config.yaml"}
2026-03-16T19:28:16Z INFO Anonymized tracing enabled {"process": "storagenode"}
2026-03-16T19:28:16Z INFO Operator email {"process": "storagenode", "address": "walter1@magenta.de"}
2026-03-16T19:28:16Z INFO Operator wallet {"process": "storagenode", "address": "0xba6CFA55a0fCF2e399e3AbA65dBC91d11Cb5E292"}
2026-03-16T19:28:16Z INFO server kernel support for server-side tcp fast open remains disabled. {"process": "storagenode"}
2026-03-16T19:28:16Z INFO server enable with: sysctl -w net.ipv4.tcp_fastopen=3 {"process": "storagenode"}
2026-03-16T19:28:16Z INFO hashstore loaded hint file {"process": "storagenode", "satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "store": "s0", "path": "config/storage/hashstore/121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6/s0/meta/hint-000000000000028b", "writable": [846, 832, 848, 77, 847, 826, 829, 845], "largest": 848, "skip": false}
2026-03-16T19:28:16Z INFO hashstore unclean shutdown detected: reconciling logs {"process": "storagenode", "satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "store": "s0"}
2026-03-16T19:28:16Z INFO Current binary version {"process": "storagenode-updater", "service": "storagenode", "version": "v1.148.3"}
2026-03-16T19:28:16Z INFO Version is up to date {"process": "storagenode-updater", "service": "storagenode"}
2026-03-16T19:28:16Z INFO Current binary version {"process": "storagenode-updater", "service": "storagenode-updater", "version": "v1.148.3"}
2026-03-16T19:28:16Z INFO Version is up to date {"process": "storagenode-updater", "service": "storagenode-updater"}
2026-03-16T19:28:17Z INFO hashstore loaded hint file {"process": "storagenode", "satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "store": "s1", "path": "config/storage/hashstore/121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6/s1/meta/hint-00000000000001eb", "writable": [625, 628, 627], "largest": 628, "skip": false}
2026-03-16 19:28:17,010 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-03-16 19:28:17,010 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-03-16 19:28:17,010 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-03-16T19:28:17Z INFO hashstore unclean shutdown detected: reconciling logs {"process": "storagenode", "satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "store": "s1"}
2026-03-16T19:28:17Z INFO hashstore hashstore opened successfully {"process": "storagenode", "satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "open_time": "1.634638799s", "logs_skipped": 126, "logs_matched": 11, "logs_mismatched": 0}
2026-03-16T19:28:17Z INFO hashstore loaded hint file {"process": "storagenode", "satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "store": "s0", "path": "config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/meta/hint-0000000000000142", "writable": [], "largest": 0, "skip": false}
2026-03-16T19:28:25Z ERROR failure during run {"process": "storagenode", "error": "Failed to create storage node peer: hashstore: read config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/meta/hashtbl-0000000000000145: input/output error\n\tstorj.io/storj/storagenode/hashstore.(*roBigPageCache).ReadRecord:649\n\tstorj.io/storj/storagenode/hashstore.(*HashTbl).Range:320\n\tstorj.io/storj/storagenode/hashstore.(*HashTbl).loadEntries:264\n\tstorj.io/storj/storagenode/hashstore.OpenHashTbl:176\n\tstorj.io/storj/storagenode/hashstore.OpenTable:120\n\tstorj.io/storj/storagenode/hashstore.NewStore:261\n\tstorj.io/storj/storagenode/hashstore.New:110\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:272\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:122\n\tstorj.io/storj/storagenode.New:607\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tmain.main:34\n\truntime.main:285", "errorVerbose": "Failed to create storage node peer: hashstore: read config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/meta/hashtbl-0000000000000145: input/output error\n\tstorj.io/storj/storagenode/hashstore.(*roBigPageCache).ReadRecord:649\n\tstorj.io/storj/storagenode/hashstore.(*HashTbl).Range:320\n\tstorj.io/storj/storagenode/hashstore.(*HashTbl).loadEntries:264\n\tstorj.io/storj/storagenode/hashstore.OpenHashTbl:176\n\tstorj.io/storj/storagenode/hashstore.OpenTable:120\n\tstorj.io/storj/storagenode/hashstore.NewStore:261\n\tstorj.io/storj/storagenode/hashstore.New:110\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:272\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:122\n\tstorj.io/storj/storagenode.New:607\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tmain.main:34\n\truntime.main:285\n\tmain.cmdRun:86\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tmain.main:34\n\truntime.main:285"}
Error: Failed to create storage node peer: hashstore: read config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/meta/hashtbl-0000000000000145: input/output error
storj.io/storj/storagenode/hashstore.(*roBigPageCache).ReadRecord:649
storj.io/storj/storagenode/hashstore.(*HashTbl).Range:320
storj.io/storj/storagenode/hashstore.(*HashTbl).loadEntries:264
storj.io/storj/storagenode/hashstore.OpenHashTbl:176
storj.io/storj/storagenode/hashstore.OpenTable:120
storj.io/storj/storagenode/hashstore.NewStore:261
storj.io/storj/storagenode/hashstore.New:110
storj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:272
storj.io/storj/storagenode/piecestore.NewHashStoreBackend:122
storj.io/storj/storagenode.New:607
main.cmdRun:84
main.newRunCmd.func1:33
storj.io/common/process.InitBeforeExecute.func1.2:389
storj.io/common/process.InitBeforeExecute.func1:407
github.com/spf13/cobra.(*Command).execute:985
github.com/spf13/cobra.(*Command).ExecuteC:1117
github.com/spf13/cobra.(*Command).Execute:1041
storj.io/common/process.ExecWithCustomOptions:115
main.main:34
runtime.main:285
How to fix this error and node?
Thanks and kind regards,
. 2.5 years is very little time for a drive. Yes, it was definitely defective. You may be able to replace it under warranty (it’s 5 years for that drive).