Main node down after update to v1.127.1

[DCinGarage]

Hello,

OS: Windows 10.

I had one node on a separate hard drive for a few months. Didn’t have any problems.

I installed an additional hard disk, Storge node toolbox and started the second node.

Storge node toolbox updated to v1.127.1 and then I started having problems with the first (main) node. Node crashes every 4-6 hours. The second (new) node works without problems.

I disconnected the first disk to check it with chkdsk. No errors were found.

I’m using the new WD hard drives. Not SMR.

Also, I haven’t recorded any network problems.

What could be the cause?

2025-05-10T04:06:30+03:00 ERROR services unexpected shutdown of a runner {“name”: “piecestore:monitor”, “error”: “piecestore monitor: timed out after 1m0s while verifying writability of storage directory”, “errorVerbose”: “piecestore monitor: timed out after 1m0s while verifying writability of storage directory\n\tstorj.io/storj/storagenode/monitor.(*Service).verifyWritability:184\n\tstorj.io/common/sync2.(*Cycle).Run:163\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func2:114\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}
2025-05-10T04:06:42+03:00 ERROR piecestore:cache error getting current used space for trash: {“error”: “filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: context canceled”, “errorVerbose”: “filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: context canceled\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).SpaceUsedForTrash:302\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:104\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}
2025-05-10T04:06:43+03:00 ERROR piecestore error sending hash and order limit {“Piece ID”: “A7JTPJOAI542VGS6I6SRDJG7UGKQHE3G5VOH32FVMNIPIIQZZQKA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 17664, “Remote Address”: “49.12.206.28:14647”, “error”: “EOF”}
2025-05-10T04:06:43+03:00 ERROR piecestore download failed {“Piece ID”: “A7JTPJOAI542VGS6I6SRDJG7UGKQHE3G5VOH32FVMNIPIIQZZQKA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 17664, “Remote Address”: “49.12.206.28:14647”, “error”: “EOF”, “errorVerbose”: “EOF\n\tstorj.io/common/rpc/rpcstatus.NamedWrap:100\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:713\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35”}
2025-05-10T04:06:43+03:00 ERROR piecestore error sending hash and order limit {“Piece ID”: “6UNNCITZVRRT54BC2S4K6ZO4KBBQSBSYQ4ZYHELZP7AFLVSN6BYA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 1280, “Remote Address”: “91.107.230.91:54584”, “error”: “EOF”}
2025-05-10T04:06:43+03:00 ERROR piecestore download failed {“Piece ID”: “6UNNCITZVRRT54BC2S4K6ZO4KBBQSBSYQ4ZYHELZP7AFLVSN6BYA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 1280, “Remote Address”: “91.107.230.91:54584”, “error”: “EOF”, “errorVerbose”: “EOF\n\tstorj.io/common/rpc/rpcstatus.NamedWrap:100\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:713\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35”}
2025-05-10T04:06:43+03:00 ERROR piecestore error sending hash and order limit {“Piece ID”: “HN3EA2U7O3WCNXX3LMCWMTBEMA6ESS36JG6EIQCHAVBFQGGTT2WA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 21504, “Remote Address”: “157.90.18.22:42707”, “error”: “EOF”}
2025-05-10T04:06:43+03:00 ERROR piecestore error sending hash and order limit {“Piece ID”: “3DOO57KQL3WDFVYAHEQB4HVEA5E27O6LOXYRQRRKOJVS6JTJFCUQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 25600, “Remote Address”: “91.99.21.98:26846”, “error”: “EOF”}
2025-05-10T04:06:43+03:00 ERROR piecestore download failed {“Piece ID”: “HN3EA2U7O3WCNXX3LMCWMTBEMA6ESS36JG6EIQCHAVBFQGGTT2WA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 21504, “Remote Address”: “157.90.18.22:42707”, “error”: “EOF”, “errorVerbose”: “EOF\n\tstorj.io/common/rpc/rpcstatus.NamedWrap:100\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:713\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35”}
2025-05-10T04:06:43+03:00 ERROR piecestore download failed {“Piece ID”: “3DOO57KQL3WDFVYAHEQB4HVEA5E27O6LOXYRQRRKOJVS6JTJFCUQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 25600, “Remote Address”: “91.99.21.98:26846”, “error”: “EOF”, “errorVerbose”: “EOF\n\tstorj.io/common/rpc/rpcstatus.NamedWrap:100\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:713\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35”}
2025-05-10T04:06:43+03:00 ERROR piecestore error sending hash and order limit {“Piece ID”: “RDWCQPRXYWJDGUT5HEJHLQKKXX2VU4GVQLLO6FTDQH6UER26TPAA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 181504, “Remote Address”: “45.140.189.201:46095”, “error”: “EOF”}
2025-05-10T04:06:43+03:00 ERROR piecestore download failed {“Piece ID”: “RDWCQPRXYWJDGUT5HEJHLQKKXX2VU4GVQLLO6FTDQH6UER26TPAA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 181504, “Remote Address”: “45.140.189.201:46095”, “error”: “EOF”, “errorVerbose”: “EOF\n\tstorj.io/common/rpc/rpcstatus.NamedWrap:100\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:713\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35”}
2025-05-10T04:06:43+03:00 ERROR piecestore error sending hash and order limit {“Piece ID”: “INHUIRQFCSIREM2JFI6XIAYIY3UFCCLDI6YT4WGKJQPVGEEVREBA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 21248, “Remote Address”: “116.203.74.70:6529”, “error”: “EOF”}
2025-05-10T04:06:43+03:00 ERROR piecestore download failed {“Piece ID”: “INHUIRQFCSIREM2JFI6XIAYIY3UFCCLDI6YT4WGKJQPVGEEVREBA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 21248, “Remote Address”: “116.203.74.70:6529”, “error”: “EOF”, “errorVerbose”: “EOF\n\tstorj.io/common/rpc/rpcstatus.NamedWrap:100\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:713\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35”}
2025-05-10T04:06:43+03:00 ERROR piecestore error sending hash and order limit {“Piece ID”: “KSMHIEG6HQCNCPCL6YQTNFTNUR57IJJB5IMBOGX6EGYFS2A3LSOQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 9472, “Remote Address”: “45.140.189.201:50149”, “error”: “EOF”}
2025-05-10T04:06:43+03:00 ERROR piecestore download failed {“Piece ID”: “KSMHIEG6HQCNCPCL6YQTNFTNUR57IJJB5IMBOGX6EGYFS2A3LSOQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 9472, “Remote Address”: “45.140.189.201:50149”, “error”: “EOF”, “errorVerbose”: “EOF\n\tstorj.io/common/rpc/rpcstatus.NamedWrap:100\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:713\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35”}
2025-05-10T04:06:43+03:00 ERROR piecestore error sending hash and order limit {“Piece ID”: “VPJFZ3IJT5UARFZUW6RRHLT2GXFZX62JLLJWVVLJFDAQ7RSKUU2A”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 33280, “Remote Address”: “49.12.206.28:15704”, “error”: “EOF”}
2025-05-10T04:06:43+03:00 ERROR piecestore download failed {“Piece ID”: “VPJFZ3IJT5UARFZUW6RRHLT2GXFZX62JLLJWVVLJFDAQ7RSKUU2A”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 33280, “Remote Address”: “49.12.206.28:15704”, “error”: “EOF”, “errorVerbose”: “EOF\n\tstorj.io/common/rpc/rpcstatus.NamedWrap:100\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:713\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35”}
2025-05-10T04:06:43+03:00 ERROR piecestore error sending hash and order limit {“Piece ID”: “EY6UYZOQTGU6IPYFEZ4JKAMF72UQHJFDUDETCM5UGN44JNRSMH6Q”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 21248, “Remote Address”: “116.203.74.70:34682”, “error”: “EOF”}
2025-05-10T04:06:43+03:00 ERROR piecestore download failed {“Piece ID”: “EY6UYZOQTGU6IPYFEZ4JKAMF72UQHJFDUDETCM5UGN44JNRSMH6Q”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 21248, “Remote Address”: “116.203.74.70:34682”, “error”: “EOF”, “errorVerbose”: “EOF\n\tstorj.io/common/rpc/rpcstatus.NamedWrap:100\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:713\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35”}
2025-05-10T04:06:43+03:00 ERROR piecestore error sending hash and order limit {“Piece ID”: “7G5F6PVON2RKII442HLBEME52K3B2WJF4SYX5RTRW5QXTBEDB5LA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 2304, “Remote Address”: “91.99.98.54:41145”, “error”: “EOF”}
2025-05-10T04:06:43+03:00 ERROR piecestore download failed {“Piece ID”: “7G5F6PVON2RKII442HLBEME52K3B2WJF4SYX5RTRW5QXTBEDB5LA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 2304, “Remote Address”: “91.99.98.54:41145”, “error”: “EOF”, “errorVerbose”: “EOF\n\tstorj.io/common/rpc/rpcstatus.NamedWrap:100\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:713\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35”}
2025-05-10T04:06:43+03:00 ERROR piecestore error sending hash and order limit {“Piece ID”: “RFD5D3WT3E5KU5CUIZJ45IKR2Q4WZGNLK2JID2OIGCZMRO6PFG6Q”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 2319360, “Remote Address”: “45.140.189.201:7741”, “error”: “EOF”}
2025-05-10T04:06:43+03:00 ERROR piecestore download failed {“Piece ID”: “RFD5D3WT3E5KU5CUIZJ45IKR2Q4WZGNLK2JID2OIGCZMRO6PFG6Q”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 2319360, “Remote Address”: “45.140.189.201:7741”, “error”: “EOF”, “errorVerbose”: “EOF\n\tstorj.io/common/rpc/rpcstatus.NamedWrap:100\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:713\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35”}
2025-05-10T04:06:43+03:00 ERROR piecestore error sending hash and order limit {“Piece ID”: “U5W2XO4NN46CVUF64X75XAORV7VDO4FD6TXAOCMJAX2EPLX3U3HQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 271104, “Remote Address”: “91.99.103.214:40536”, “error”: “EOF”}
2025-05-10T04:06:43+03:00 ERROR piecestore download failed {“Piece ID”: “U5W2XO4NN46CVUF64X75XAORV7VDO4FD6TXAOCMJAX2EPLX3U3HQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 271104, “Remote Address”: “91.99.103.214:40536”, “error”: “EOF”, “errorVerbose”: “EOF\n\tstorj.io/common/rpc/rpcstatus.NamedWrap:100\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:713\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35”}
2025-05-10T04:06:43+03:00 ERROR piecestore error sending hash and order limit {“Piece ID”: “SAG3YWGNTFJNM3K5LLW64GTXC5TUBUOBQQWTFMYOR7BYQ7W7HSYA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 181504, “Remote Address”: “88.99.185.168:54308”, “error”: “EOF”}
2025-05-10T04:06:43+03:00 ERROR piecestore error sending hash and order limit {“Piece ID”: “22XK2YNVLVI46QI7KJQ656Z7H6TM3DSJFRDE6THHZ5MC35F463CA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 39424, “Remote Address”: “138.199.217.172:13229”, “error”: “EOF”}
2025-05-10T04:06:43+03:00 ERROR piecestore download failed {“Piece ID”: “22XK2YNVLVI46QI7KJQ656Z7H6TM3DSJFRDE6THHZ5MC35F463CA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 39424, “Remote Address”: “138.199.217.172:13229”, “error”: “EOF”, “errorVerbose”: “EOF\n\tstorj.io/common/rpc/rpcstatus.NamedWrap:100\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:713\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35”}
2025-05-10T04:06:43+03:00 ERROR piecestore download failed {“Piece ID”: “SAG3YWGNTFJNM3K5LLW64GTXC5TUBUOBQQWTFMYOR7BYQ7W7HSYA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 181504, “Remote Address”: “88.99.185.168:54308”, “error”: “EOF”, “errorVerbose”: “EOF\n\tstorj.io/common/rpc/rpcstatus.NamedWrap:100\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:713\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35”}
2025-05-10T04:06:44+03:00 ERROR piecestore error sending hash and order limit {“Piece ID”: “EEJUAVEUKHJMU45YRV6DKSTX5B3OJ7USFAPJLSSUSLULJU56QMHQ”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 2319360, “Remote Address”: “49.12.206.28:40769”, “error”: “EOF”}
2025-05-10T04:06:44+03:00 ERROR piecestore download failed {“Piece ID”: “EEJUAVEUKHJMU45YRV6DKSTX5B3OJ7USFAPJLSSUSLULJU56QMHQ”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 2319360, “Remote Address”: “49.12.206.28:40769”, “error”: “EOF”, “errorVerbose”: “EOF\n\tstorj.io/common/rpc/rpcstatus.NamedWrap:100\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:713\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:302\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35”}
2025-05-10T04:06:46+03:00 ERROR failure during run {“error”: “piecestore monitor: timed out after 1m0s while verifying writability of storage directory”, “errorVerbose”: “piecestore monitor: timed out after 1m0s while verifying writability of storage directory\n\tstorj.io/storj/storagenode/monitor.(*Service).verifyWritability:184\n\tstorj.io/common/sync2.(*Cycle).Run:163\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func2:114\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}
2025-05-10T04:06:46+03:00 FATAL Unrecoverable error {“error”: “piecestore monitor: timed out after 1m0s while verifying writability of storage directory”, “errorVerbose”: “piecestore monitor: timed out after 1m0s while verifying writability of storage directory\n\tstorj.io/storj/storagenode/monitor.(*Service).verifyWritability:184\n\tstorj.io/common/sync2.(*Cycle).Run:163\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func2:114\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}

[Vadim]

@DCinGarage

FATAL Unrecoverable error {“error”: “piecestore monitor: timed out after 1m0s while verifying writability of storage directory”, “errorVerbose”: “piecestore monitor: timed out after 1m0s while verifying writability of storage directory\n\tstorj.io/storj/storagenode/monitor.(*Service).verifyWritability:184\n\tstorj.io/common/sync2.(*Cycle).Run:163\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func2:114\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}

Looks like your HDD is toggling to response in 1minut. to HDD alive check. My 1.127 on windows work OK

[Roxor]

Yeah if a disk is unresponsive for more than a minute: I’d expect Windows to also be complaining about it in Event Viewer. Any chance the drive is connected by USB?

[EasyRhino]

fwiw i also get a failure to ensure writability error if I have a drive with directories accidentally configured to read only or wrong permissions (linux problems).

or maybe when swapping drives the drive letters got changed around. so the node is looking for a drive that either no longer exists or has swapped to the broken drive.

[DCinGarage]

No, SATA.

I didn’t do anything else, but the node has been running stable for 24 hours now. I don’t know what it was.

Thanks for the answers.

[DCinGarage]

The service is crashing regularly again. I recorded a more detailed log:

2025-05-21T06:59:14+03:00 DEBUG lazyfilewalker.gc-filewalker.trash-handler trashing piece {“satelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “pieceID”: “BKZ4PLRKRB7BCJBR53NDE5KXRQ7P5H5ZVYX5Q3IQ4OLCKDCEY4YQ”}
2025-05-21T06:59:14+03:00 DEBUG retain About to move piece to trash {“cachePath”: “C:\Program Files\Storj\Storage Node/retain”, “BF”: “”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Piece ID”: “BKZ4PLRKRB7BCJBR53NDE5KXRQ7P5H5ZVYX5Q3IQ4OLCKDCEY4YQ”, “Status”: “enabled”}
… repeat meny times
2025-05-21T06:59:24+03:00 DEBUG db blob not found; will not trash {“blob_path”: “F:\storj\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\bp\23ukzfe3qiks7ecl7m7i5wdnpe43etgvnk6fktceakl7pj45zq.sj1”}
…
2025-05-21T06:59:24+03:00 INFO lazyfilewalker.gc-filewalker subprocess exited with status {“satelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “status”: 1, “error”: “exit status 1”}
2025-05-21T06:59:24+03:00 ERROR pieces lazyfilewalker failed {“error”: “lazyfilewalker: exit status 1”, “errorVerbose”: “lazyfilewalker: exit status 1\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:85\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkSatellitePiecesToTrash:164\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePiecesToTrash:625\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:412\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func3:297\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}
2025-05-21T06:59:24+03:00 ERROR filewalker failed to get progress from database
2025-05-21T06:59:24+03:00 DEBUG db number of subdirs {“count”: 1024}
2025-05-21T06:59:24+03:00 ERROR retain retain pieces failed {“cachePath”: “C:\Program Files\Storj\Storage Node/retain”, “error”: “retain: filewalker: context canceled”, “errorVerbose”: “retain: filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:78\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePiecesToTrash:301\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePiecesToTrash:632\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:412\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func3:297\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}
2025-05-21T06:59:24+03:00 ERROR failure during run {“error”: “piecestore monitor: timed out after 1m0s while verifying writability of storage directory”, “errorVerbose”: “piecestore monitor: timed out after 1m0s while verifying writability of storage directory\n\tstorj.io/storj/storagenode/monitor.(*Service).verifyWritability:184\n\tstorj.io/common/sync2.(*Cycle).Run:163\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func2:114\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}
2025-05-21T06:59:24+03:00 FATAL Unrecoverable error {“error”: “piecestore monitor: timed out after 1m0s while verifying writability of storage directory”, “errorVerbose”: “piecestore monitor: timed out after 1m0s while verifying writability of storage directory\n\tstorj.io/storj/storagenode/monitor.(*Service).verifyWritability:184\n\tstorj.io/common/sync2.(*Cycle).Run:163\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func2:114\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}

Windows write:

Unable to find description for event ID 0 from source Storj V3 Storage Node. The component raising this event is not installed on this local machine or is corrupted. Install or repair the component on the local machine.
If the event originated on another machine, you may need to save the display details along with the event.
The following details were added to the event:
starting service
This request is not supported

I can’t understand why this is happening? There are no errors with a full file system test.

[Vadim]

try to update to 1.128

[jammerdan]

@DCinGarage

2025-05-21T06:59:24+03:00 DEBUG db blob not found; will not trash {“blob_path”: “F:\storj\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\bp\23ukzfe3qiks7ecl7m7i5wdnpe43etgvnk6fktceakl7pj45zq.sj1”}

@DCinGarage

2025-05-21T06:59:24+03:00 ERROR filewalker failed to get progress from database

@DCinGarage

2025-05-21T06:59:24+03:00 ERROR failure during run {“error”: “piecestore monitor: timed out after 1m0s while verifying writability of storage directory”

@DCinGarage

2025-05-21T06:59:24+03:00 FATAL Unrecoverable error {“error”: “piecestore monitor: timed out after 1m0s while verifying writability of storage directory”,

It sounds like either a database or a accessibility problem. Did you check if the database it is referring to is available and not corrupted and if the path is accessible and permissions are correct?

[DCinGarage]

I am currently using version 1.128.4
The update occurs automatically

[DCinGarage]

@DCinGarage

F:\storj\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\bp\23ukzfe3qiks7ecl7m7i5wdnpe43etgvnk6fktceakl7pj45zq.sj1

This file really doesn’t exist.

Is there a way to rebuild the database? And is it necessary…

[jammerdan]

@DCinGarage

Is there a way to rebuild the database? And is it necessary…

I am not sure what caused your many error messages. it is complaining that it cannot find the blob and cannot get the progress from database. So there seems to be some issue related to that. And then it cannot perform the write test to the storage directory in 1 minute. So you are having some issue all around this which you need to check 1 by 1.

For check and rebuild database.: Database disk image is malformed error in piece_expiration db - #3 by snorkel

[Alexey]

@DCinGarage

2025-05-21T06:59:24+03:00 FATAL Unrecoverable error {“error”: “piecestore monitor: timed out after 1m0s while verifying writability of storage directory”, “errorVerbose”: “piecestore monitor: timed out after 1m0s while verifying writability of storage directory

This is the root cause. Your disk is unable to respond within 1m to write a small file.
You can check this thread after you would scan and fix errors on the disk and check and fix databases:

Fatal Error on my Node / Timeout after 1 min

Hello Everyone, a week ago i got several errors on my main node. Maybe someone can help me to solve the problem. The Node runs on Windows 10. Thanks for your Help. 2023-03-27T05:11:25.897+0200 INFO piecestore download started {Piece ID: 7WMQ364BMF672RLAQWDD76IQ6EORNXMYXIL5PGN46I4VUT4P4JXA, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: GET, Offset: 110336, Size: 469504, Remote Address: 184.104.224.98:48702} 2023-03-27T05:11:25.916+0200 ERROR services unexp…

[DCinGarage]

I have write cache enabled for HDD in Windows. Storj ignores him?

I performed a full HDD scan with chkdsk, it lasted about 18 hours and no errors were found.

I also did a recording test over the entire surface before add this HDD.

[Alexey]

It’s not related to Storj either. It’s a setting of your OS, so it’s used for everything.
However, If you have a managed UPS (which can gracefully shutdown your Windows), then you may enable the second checkbox.

I think, it likely may be not enough and you would need to perform an optimization tasks (defrag), however, it would not lasts enough due to limitations of NTFS. You may optimize it though:

[Solved] Win10 20GB Ram Usage

Oh, separately, make sure you have last access time disabled on the volume that holds storage node data – this will remove quite a lot of IO: And if you haven’t done so yet – move databases to your system drive. We should have started with this (but I assumed you have already done that, but I"m not sure, so double check

NTFS Disable 8dot3name

Fresh off an ReFS volume that went RAW, I’ve decided to be equally reckless and disable the 8 dot 3 short names for my replacement NTFS volume. I was shocked to not find a single mention of it in this forum, so I’ll report back once this bad decision ends up in another catastrophic fail, but the numbers looks promising for the Storj trillion tiny file configurations. [8484.image_thumb_0763777C] Disable across system: fsutil behavior set disable8dot3 Strip existing: fsutil 8dot3name strip /f…

Help with node optimizations / best practices

After a few months of testing things and then migrating all my nodes, the 100% activity on all drives all the time is fixed, IOdelay in Proxmox is much lower and filewalker only takes a few hours for a 7TB node. After diagnosing that read speeds were reaching about 300KB/s tops for each drive while running Storj (other workloads seemed OK in the same setup) but write speeds were close to bare metal, I took a ~2TB node as a test subject and controlled the conditions so that minimal ingest from S…

So, perhaps the better direction would be to enable a badger cache:

Badger cache filewalker test results

pieces.file-stat-cache: badger and pieces.enable-lazy-filewalker: false in the config, then restart

The first run would be slow as usual, then it would improve. It may also help with the random access and random writes.

However, it’s highly recommended to do any optimizations after you fixed disk and databases issues and did at least once a regular defragmentation.

The complete solution would require to migrate either to Linux or use a Primocache (especially for multiple nodes).

[Vadim]

18h for 4tb HDD? You definitely has some problem somewhere.
Do you use some sata port on Mothrboard or it additional sata card?

[Julio]

Change your SATA cables.

2 cents,
Julio

[Alexey]

It could be not enough. It also could be a poor PSU or the problem with the SATA controller or the extension card.

[Julio]

@DCinGarage

I also did a recording test over the entire surface before add this HDD.

Did you do this with Victoria? Obviously results were acceptable to you, but note how long did it take? Compare time / data of that surface write check vs. 18hr check disk. The chkdsk should have taken twice as long+. The 18 hr test isn’t too bad for a 5200/5400 rpm disk - ~65 megs per sec, but IIRC should be more like 80-90 meg sustained average, and say 120 meg/s for a 7200 rpm.

What Alexey said ^ – true all that, too… PSU/Controller/in-line usb chipset, etc. Even a using a Hub could limit b/w.

Also age/size fragmentation of the old node…MFT could be fragmented out of control on a physical disk, that can start to impede performance fast & hard; especially if it’s a virtual .VHDx on that physical disk and over the 2 to 4 TB mark.

GL, 2 cents more,
Julio

[DCinGarage]

@Vadim

18h for 4tb HDD? You definitely has some problem somewhere.

So that’s normal. It’s a file system check with error correction.

@Julio

Change your SATA cables.

New hdd with a new sata cable. I don’t use old cables with new hdd.

@Alexey

So, perhaps the better direction would be to enable a badger cache:

Thanks, that’s the next thing I’m going to try.

I moved the databases to the system ssd and rebuilt them, since then node has been running again without crashes for a 4 days now.

@Alexey

It could be not enough. It also could be a poor PSU or the problem with the SATA controller or the extension card.

I am using a good Chieftec Proton BDF-500S power supply with the actual system consumption being less than 100Wh.

@Julio

Did you do this with Victoria?

Yes, but I don’t remember how long the whole inspection took. But it was faster than a file system check with error correction.

I want to now wait for the problem to recur, and then I’ll reconnect the hdd to a different sata port.

Thank you all)

[arrogantrabbit]

@DCinGarage

So that’s normal. It’s a file system check with error correction.

Still too long.

@DCinGarage

New hdd with a new sata cable. I don’t use old cables with new hdd.

New cables cannot be faulty? I would use old, known good cables.

@DCinGarage

I am using a good Chieftec Proton BDF-500S

How do you know it’s good?

When stuff like this happens — everything is suspect, especially new parts. New means untested and broken unless proven otherwise. Re: cables — today it’s almost impossible to buy good sata cables. Unless you overpay through the nose for server parts or find one used. Consumer stuff from Amazon and Newegg is all crap. All of it.

Lastly, when you rule out everything else — HDD itself is a suspect — if it’s new, you don’t know if it’s good. Generally it’s a good idea to “burn in” a new system with various stress tests before deploying.

Also ram. Is ram also new? Run memtest then.