Windows node down; tried scanning disk no luck | Fixed

Node Operators troubleshooting
,
1

Hiya Forum

I’ve got a node, that’s suddenly down. I has been running without issues for a small year.
I’ve tried chkdsk.exe /r /f /x on both th C:\ drive, hosting the OS, the D:\ drive hosting the storagenode data and the E:\ drive hosting the databases.

I get a lot of 2024-10-29T17:20:01+01:00 FATAL Unrecoverable error {"error": "filestore error: failed to walk trash namespace in my log, and it’s true, the path’s the node is looking for are indeed not existing. I have not tried repairing (or deleting) databases yet.

Here is a section of my log, filtered to warn

2024-10-29T17:12:51+01:00	WARN	reputation:service	node scores worsened	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Total Audits": 79381, "Successful Audits": 69118, "Audit Score": 0.9960580051044211, "Online Score": 0.7387193415153489, "Suspension Score": 0.999751014728745, "Audit Score Delta": 0.00046301475793741886, "Online Score Delta": -0.06427543825667459, "Suspension Score Delta": 0.07368659233629293}
2024-10-29T17:12:52+01:00	WARN	reputation:service	node scores worsened	{"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Total Audits": 187188, "Successful Audits": 170684, "Audit Score": 0.9956238061978611, "Online Score": 0.7546689418604743, "Suspension Score": 0.987646693716961, "Audit Score Delta": 0.0002475599238211945, "Online Score Delta": -0.06868407494780226, "Suspension Score Delta": 0.19513084010149062}
2024-10-29T17:12:53+01:00	WARN	reputation:service	node scores worsened	{"Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Total Audits": 205459, "Successful Audits": 183269, "Audit Score": 0.9992826318493024, "Online Score": 0.7292749350404752, "Suspension Score": 0.9999570929146524, "Audit Score Delta": 0.0001203374444862515, "Online Score Delta": -0.0657170224997401, "Suspension Score Delta": 0.12167940537034172}
2024-10-29T17:12:53+01:00	WARN	reputation:service	node scores worsened	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Total Audits": 529053, "Successful Audits": 482861, "Audit Score": 0.9988783386915313, "Online Score": 0.7717420636793965, "Suspension Score": 0.9999999985630567, "Audit Score Delta": 0.00047511615860740264, "Online Score Delta": -0.03946151220745342, "Suspension Score Delta": 0.10495592498117012}
2024-10-29T17:13:08+01:00	ERROR	db	failed to stat blob in trash	{"namespace": "ryxCAD78gmq0Nh9z+diQlCFG/g6+gGeG+OcZCAAAAAA=", "key": "AOsZuXGxqmlEmLvinq88y1qbXZkmnVTWJNVfj0NQcMM=", "error": "CreateFile D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-20\\ad\\vrtolrwgvgsreyxprj5lz4znnjwxmze2ovjvre2vpy6q2qodbq.sj1: The system cannot find the file specified."}
2024-10-29T17:13:08+01:00	ERROR	db	failed to stat blob in trash	{"namespace": "ryxCAD78gmq0Nh9z+diQlCFG/g6+gGeG+OcZCAAAAAA=", "key": "AO4PdDjUx18iIILoZr3LbfejDr24TJkTiqmyKi4k1IA=", "error": "CreateFile D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-20\\ad\\xa65by2tdv6iraqlugnpolnx32gdv5xbgjse4kvgzculre2saa.sj1: The system cannot find the file specified."}
2024-10-29T17:13:08+01:00	ERROR	db	failed to stat blob in trash	{"namespace": "ryxCAD78gmq0Nh9z+diQlCFG/g6+gGeG+OcZCAAAAAA=", "key": "AO8L1e9sg5QTw/5bm7SbXAiI/VePXBhB24I5D57AIew=", "error": "CreateFile D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-20\\ad\\xqxvppnsbzie6d7znzxne3lqeir7kxr5obqqo3qi4q7hwaehwa.sj1: The system cannot find the file specified."}
2024-10-29T17:13:10+01:00	ERROR	db	failed to stat blob in trash	{"namespace": "ryxCAD78gmq0Nh9z+diQlCFG/g6+gGeG+OcZCAAAAAA=", "key": "AwCeTRmUkHXXBhtpaHulExKTkKT/gslseg800r+xQnA=", "error": "CreateFile D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-20\\am\\aj4tizssihlvygdnuwq65fcmjjhefe76bms3d2b42nfp5rijya.sj1: The system cannot find the file specified."}
2024-10-29T17:13:33+01:00	ERROR	piecestore	download failed	{"Piece ID": "VVJS7GLEMXNET2SZYAXUG2LW6LM6TUMNYM5KI362UE6DL5J3MCYA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Offset": 0, "Size": 3840, "Remote Address": "79.127.226.104:41248", "error": "trust: rpc: tcp connector failed: rpc: context canceled", "errorVerbose": "trust: rpc: tcp connector failed: rpc: context canceled\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2024-10-29T17:13:42+01:00	ERROR	piecestore	download failed	{"Piece ID": "M2RIVFUGBAGBQHBD5PKITAEJ2YXGGA2PZORG23G2UAFEOTQSAUIA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Offset": 0, "Size": 1536, "Remote Address": "79.127.226.103:38092", "error": "trust: rpc: tcp connector failed: rpc: context canceled", "errorVerbose": "trust: rpc: tcp connector failed: rpc: context canceled\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2024-10-29T17:13:43+01:00	ERROR	piecestore	upload failed	{"Piece ID": "V7H5KIZDHQTCVAD4C4DHTNRK7DT4HBN3XHGSRIKARQ7LKZW52ETA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "Remote Address": "169.150.225.81:52898", "Size": 262144, "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:93\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:583\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-10-29T17:14:06+01:00	ERROR	piecestore	upload failed	{"Piece ID": "EEVJMRUL3QKRLLSARVSPCXOFSCWJHNNFD32TNY6BJXGNPPT5NGFQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.226.107:36172", "Size": 131072, "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:93\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:583\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-10-29T17:14:50+01:00	ERROR	piecestore	upload failed	{"Piece ID": "B7FUPZYAJIVJ7X4HBWVPJGMPLULR57O3MJXO6PJ542JGADU72N3Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "156.146.43.226:55610", "Size": 131072, "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:93\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:583\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-10-29T17:14:52+01:00	ERROR	piecestore	upload failed	{"Piece ID": "EVTDZASJMFWYQLAPDYVG5QTI6TXGI3F3DLFYSUWEIJEROLUKOBPA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Remote Address": "62.248.179.8:54266", "Size": 131072, "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:93\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:583\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-10-29T17:15:06+01:00	ERROR	piecestore	upload failed	{"Piece ID": "MVQJAZBGVSTGWAYQU6M4JHHNCNHTKVVRIWOK6PLSNGAOY47ZTPGQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.226.105:48566", "Size": 327680, "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:93\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:583\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-10-29T17:15:51+01:00	ERROR	piecestore	upload failed	{"Piece ID": "4VCNMTJJ57ZOS4ECC65OC273FJTTVMAD6DGHRKKYCI7A4Q54DF3Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.226.107:60446", "Size": 262144, "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:93\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:583\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-10-29T17:15:58+01:00	ERROR	piecestore:cache	error getting current used space for trash: 	{"error": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: context canceled", "errorVerbose": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: context canceled\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).SpaceUsedForTrash:273\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:100\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-10-29T17:16:27+01:00	ERROR	piecestore	upload failed	{"Piece ID": "2FTXTHXIIVQFJEYNX2ZFNQHIGK7HMO3F4LBXXLEXDSZDJARUL3KA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.226.103:46874", "Size": 131072, "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:93\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:583\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-10-29T17:16:31+01:00	ERROR	piecestore:cache	error getting current used space for trash: 	{"error": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: readdir D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-22\\jx: The system cannot find the path specified.", "errorVerbose": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: readdir D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-22\\jx: The system cannot find the path specified.\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).SpaceUsedForTrash:273\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:100\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-10-29T17:16:31+01:00	ERROR	services	unexpected shutdown of a runner	{"name": "piecestore:cache", "error": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: readdir D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-22\\jx: The system cannot find the path specified.", "errorVerbose": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: readdir D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-22\\jx: The system cannot find the path specified.\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).SpaceUsedForTrash:273\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:100\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-10-29T17:16:31+01:00	ERROR	failure during run	{"error": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: readdir D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-22\\jx: The system cannot find the path specified.", "errorVerbose": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: readdir D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-22\\jx: The system cannot find the path specified.\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).SpaceUsedForTrash:273\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:100\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-10-29T17:16:31+01:00	FATAL	Unrecoverable error	{"error": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: readdir D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-22\\jx: The system cannot find the path specified.", "errorVerbose": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: readdir D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-22\\jx: The system cannot find the path specified.\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).SpaceUsedForTrash:273\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:100\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-10-29T17:20:00+01:00	ERROR	piecestore	download failed	{"Piece ID": "C7EIECWCQ33UZJQZHIEQ2DI2AOKR2E5RZ6UWG5JQRTIDPGGHEQIQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 7936, "Remote Address": "109.61.92.76:52326", "error": "trust: rpc: tcp connector failed: rpc: dial tcp 34.150.199.48:7777: operation was canceled", "errorVerbose": "trust: rpc: tcp connector failed: rpc: dial tcp 34.150.199.48:7777: operation was canceled\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2024-10-29T17:20:00+01:00	ERROR	contact:service	ping satellite failed 	{"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 1, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:202\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:156\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-10-29T17:20:01+01:00	ERROR	piecestore:cache	error getting current used space for trash: 	{"error": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: readdir D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-22\\jx: The system cannot find the path specified.", "errorVerbose": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: readdir D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-22\\jx: The system cannot find the path specified.\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).SpaceUsedForTrash:273\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:100\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-10-29T17:20:01+01:00	ERROR	services	unexpected shutdown of a runner	{"name": "piecestore:cache", "error": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: readdir D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-22\\jx: The system cannot find the path specified.", "errorVerbose": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: readdir D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-22\\jx: The system cannot find the path specified.\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).SpaceUsedForTrash:273\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:100\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-10-29T17:20:01+01:00	ERROR	contact:service	ping satellite failed 	{"Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "attempts": 1, "error": "ping satellite: context canceled", "errorVerbose": "ping satellite: context canceled\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:202\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:156\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-10-29T17:20:01+01:00	ERROR	nodestats:cache	Get pricing-model/join date failed	{"error": "context canceled"}
2024-10-29T17:20:01+01:00	ERROR	contact:service	ping satellite failed 	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 1, "error": "ping satellite: context canceled", "errorVerbose": "ping satellite: context canceled\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:202\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:156\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-10-29T17:20:01+01:00	ERROR	failure during run	{"error": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: readdir D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-22\\jx: The system cannot find the path specified.", "errorVerbose": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: readdir D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-22\\jx: The system cannot find the path specified.\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).SpaceUsedForTrash:273\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:100\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-10-29T17:20:01+01:00	FATAL	Unrecoverable error	{"error": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: readdir D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-22\\jx: The system cannot find the path specified.", "errorVerbose": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: readdir D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-22\\jx: The system cannot find the path specified.\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).SpaceUsedForTrash:273\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:100\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2

Is it Windows GUI or docker ?

3

Windows GUI install - no docker on windows for me :slight_smile:

4

What does storage.path: in the config.yaml file show ?

5

Storage.Path: D:\

Just like

Storage2.database-dir: E:\

additionally, these are the items seen in the root of D:\

image
image987×498 55.9 KB

The dbback folder is from a database related issue I had a year or so ago

6

Can you see this exact path to the directory ?

7

Yes, the directory is filled with trash 2-char directories

8

The 2024-10-20 subfolder is missing in some of the other subdirectories in trash. I’m creating them.

While D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-22\\ does exist, it is completely emtpy.

Perhaps delete some of the trash databases? :thinking:

9

Did it finish successfully ?

You shouldn’t delete any database unless its the only last solution to the issue.

10

Yes, all checks finished succesfully.

C:\ and E:\ very fast both due to their size and NVMe, D:\ took a few days

11

You can try to create this subfolder. Would it resolve an issue?
Also, please make sure that all data folders have a SYSTEM user as an owner and it should have at least Write, List, Delete, Read permissions.

12

Hmmm, suddenly I’m not so sure weather or not there has been an interruption to the windows node or not. the 2024-10-22 did not include a folder named jx - but there was an extensionless file named jx. I removed the file, created the empty folder and restarted the node.

Everything is working as intended now :slight_smile: - I’m marking your post as solution

13

This is usually mean, that you have had a filesystem corruption. Seems you ran chkdsk /f to fix it and it was fixed this way.
I would recommend to perform the scan several times, until it would stop modifying the filesystem. No chkdsk utility will fix all errors in one pass.

14

That’s what I don’t quite understand. The chkdsk finished successfully after running for a few days, but found no errors, and made no changes to the filesystem. The extensionless file, jx was untouched by chkdsk, and modified only when I deleted it.

I’ll write a script searching for extensionless files in all the satellite folders when I get home. I might just have a few more landmines laying around. Cheers friend.