Windows node down; tried scanning disk no luck | Fixed

Node Operators troubleshooting
,
1

Hiya Forum

I’ve got a node, that’s suddenly down. I has been running without issues for a small year.
I’ve tried chkdsk.exe /r /f /x on both th C:\ drive, hosting the OS, the D:\ drive hosting the storagenode data and the E:\ drive hosting the databases.

I get a lot of 2024-10-29T17:20:01+01:00 FATAL Unrecoverable error {"error": "filestore error: failed to walk trash namespace in my log, and it’s true, the path’s the node is looking for are indeed not existing. I have not tried repairing (or deleting) databases yet.

Here is a section of my log, filtered to warn

2024-10-29T17:12:51+01:00	WARN	reputation:service	node scores worsened	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Total Audits": 79381, "Successful Audits": 69118, "Audit Score": 0.9960580051044211, "Online Score": 0.7387193415153489, "Suspension Score": 0.999751014728745, "Audit Score Delta": 0.00046301475793741886, "Online Score Delta": -0.06427543825667459, "Suspension Score Delta": 0.07368659233629293}
2024-10-29T17:12:52+01:00	WARN	reputation:service	node scores worsened	{"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Total Audits": 187188, "Successful Audits": 170684, "Audit Score": 0.9956238061978611, "Online Score": 0.7546689418604743, "Suspension Score": 0.987646693716961, "Audit Score Delta": 0.0002475599238211945, "Online Score Delta": -0.06868407494780226, "Suspension Score Delta": 0.19513084010149062}
2024-10-29T17:12:53+01:00	WARN	reputation:service	node scores worsened	{"Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Total Audits": 205459, "Successful Audits": 183269, "Audit Score": 0.9992826318493024, "Online Score": 0.7292749350404752, "Suspension Score": 0.9999570929146524, "Audit Score Delta": 0.0001203374444862515, "Online Score Delta": -0.0657170224997401, "Suspension Score Delta": 0.12167940537034172}
2024-10-29T17:12:53+01:00	WARN	reputation:service	node scores worsened	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Total Audits": 529053, "Successful Audits": 482861, "Audit Score": 0.9988783386915313, "Online Score": 0.7717420636793965, "Suspension Score": 0.9999999985630567, "Audit Score Delta": 0.00047511615860740264, "Online Score Delta": -0.03946151220745342, "Suspension Score Delta": 0.10495592498117012}
2024-10-29T17:13:08+01:00	ERROR	db	failed to stat blob in trash	{"namespace": "ryxCAD78gmq0Nh9z+diQlCFG/g6+gGeG+OcZCAAAAAA=", "key": "AOsZuXGxqmlEmLvinq88y1qbXZkmnVTWJNVfj0NQcMM=", "error": "CreateFile D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-20\\ad\\vrtolrwgvgsreyxprj5lz4znnjwxmze2ovjvre2vpy6q2qodbq.sj1: The system cannot find the file specified."}
2024-10-29T17:13:08+01:00	ERROR	db	failed to stat blob in trash	{"namespace": "ryxCAD78gmq0Nh9z+diQlCFG/g6+gGeG+OcZCAAAAAA=", "key": "AO4PdDjUx18iIILoZr3LbfejDr24TJkTiqmyKi4k1IA=", "error": "CreateFile D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-20\\ad\\xa65by2tdv6iraqlugnpolnx32gdv5xbgjse4kvgzculre2saa.sj1: The system cannot find the file specified."}
2024-10-29T17:13:08+01:00	ERROR	db	failed to stat blob in trash	{"namespace": "ryxCAD78gmq0Nh9z+diQlCFG/g6+gGeG+OcZCAAAAAA=", "key": "AO8L1e9sg5QTw/5bm7SbXAiI/VePXBhB24I5D57AIew=", "error": "CreateFile D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-20\\ad\\xqxvppnsbzie6d7znzxne3lqeir7kxr5obqqo3qi4q7hwaehwa.sj1: The system cannot find the file specified."}
2024-10-29T17:13:10+01:00	ERROR	db	failed to stat blob in trash	{"namespace": "ryxCAD78gmq0Nh9z+diQlCFG/g6+gGeG+OcZCAAAAAA=", "key": "AwCeTRmUkHXXBhtpaHulExKTkKT/gslseg800r+xQnA=", "error": "CreateFile D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-20\\am\\aj4tizssihlvygdnuwq65fcmjjhefe76bms3d2b42nfp5rijya.sj1: The system cannot find the file specified."}
2024-10-29T17:13:33+01:00	ERROR	piecestore	download failed	{"Piece ID": "VVJS7GLEMXNET2SZYAXUG2LW6LM6TUMNYM5KI362UE6DL5J3MCYA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Offset": 0, "Size": 3840, "Remote Address": "79.127.226.104:41248", "error": "trust: rpc: tcp connector failed: rpc: context canceled", "errorVerbose": "trust: rpc: tcp connector failed: rpc: context canceled\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2024-10-29T17:13:42+01:00	ERROR	piecestore	download failed	{"Piece ID": "M2RIVFUGBAGBQHBD5PKITAEJ2YXGGA2PZORG23G2UAFEOTQSAUIA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Offset": 0, "Size": 1536, "Remote Address": "79.127.226.103:38092", "error": "trust: rpc: tcp connector failed: rpc: context canceled", "errorVerbose": "trust: rpc: tcp connector failed: rpc: context canceled\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2024-10-29T17:13:43+01:00	ERROR	piecestore	upload failed	{"Piece ID": "V7H5KIZDHQTCVAD4C4DHTNRK7DT4HBN3XHGSRIKARQ7LKZW52ETA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "Remote Address": "169.150.225.81:52898", "Size": 262144, "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:93\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:583\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-10-29T17:14:06+01:00	ERROR	piecestore	upload failed	{"Piece ID": "EEVJMRUL3QKRLLSARVSPCXOFSCWJHNNFD32TNY6BJXGNPPT5NGFQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.226.107:36172", "Size": 131072, "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:93\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:583\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-10-29T17:14:50+01:00	ERROR	piecestore	upload failed	{"Piece ID": "B7FUPZYAJIVJ7X4HBWVPJGMPLULR57O3MJXO6PJ542JGADU72N3Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "156.146.43.226:55610", "Size": 131072, "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:93\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:583\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-10-29T17:14:52+01:00	ERROR	piecestore	upload failed	{"Piece ID": "EVTDZASJMFWYQLAPDYVG5QTI6TXGI3F3DLFYSUWEIJEROLUKOBPA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Remote Address": "62.248.179.8:54266", "Size": 131072, "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:93\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:583\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-10-29T17:15:06+01:00	ERROR	piecestore	upload failed	{"Piece ID": "MVQJAZBGVSTGWAYQU6M4JHHNCNHTKVVRIWOK6PLSNGAOY47ZTPGQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.226.105:48566", "Size": 327680, "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:93\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:583\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-10-29T17:15:51+01:00	ERROR	piecestore	upload failed	{"Piece ID": "4VCNMTJJ57ZOS4ECC65OC273FJTTVMAD6DGHRKKYCI7A4Q54DF3Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.226.107:60446", "Size": 262144, "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:93\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:583\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-10-29T17:15:58+01:00	ERROR	piecestore:cache	error getting current used space for trash: 	{"error": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: context canceled", "errorVerbose": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: context canceled\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).SpaceUsedForTrash:273\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:100\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-10-29T17:16:27+01:00	ERROR	piecestore	upload failed	{"Piece ID": "2FTXTHXIIVQFJEYNX2ZFNQHIGK7HMO3F4LBXXLEXDSZDJARUL3KA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.226.103:46874", "Size": 131072, "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:93\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:583\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:62\n\tstorj.io/common/experiment.(*Handler).HandleRPC:43\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:166\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:108\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:156\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-10-29T17:16:31+01:00	ERROR	piecestore:cache	error getting current used space for trash: 	{"error": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: readdir D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-22\\jx: The system cannot find the path specified.", "errorVerbose": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: readdir D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-22\\jx: The system cannot find the path specified.\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).SpaceUsedForTrash:273\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:100\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-10-29T17:16:31+01:00	ERROR	services	unexpected shutdown of a runner	{"name": "piecestore:cache", "error": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: readdir D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-22\\jx: The system cannot find the path specified.", "errorVerbose": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: readdir D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-22\\jx: The system cannot find the path specified.\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).SpaceUsedForTrash:273\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:100\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-10-29T17:16:31+01:00	ERROR	failure during run	{"error": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: readdir D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-22\\jx: The system cannot find the path specified.", "errorVerbose": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: readdir D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-22\\jx: The system cannot find the path specified.\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).SpaceUsedForTrash:273\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:100\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-10-29T17:16:31+01:00	FATAL	Unrecoverable error	{"error": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: readdir D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-22\\jx: The system cannot find the path specified.", "errorVerbose": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: readdir D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-22\\jx: The system cannot find the path specified.\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).SpaceUsedForTrash:273\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:100\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-10-29T17:20:00+01:00	ERROR	piecestore	download failed	{"Piece ID": "C7EIECWCQ33UZJQZHIEQ2DI2AOKR2E5RZ6UWG5JQRTIDPGGHEQIQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 7936, "Remote Address": "109.61.92.76:52326", "error": "trust: rpc: tcp connector failed: rpc: dial tcp 34.150.199.48:7777: operation was canceled", "errorVerbose": "trust: rpc: tcp connector failed: rpc: dial tcp 34.150.199.48:7777: operation was canceled\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2024-10-29T17:20:00+01:00	ERROR	contact:service	ping satellite failed 	{"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 1, "error": "ping satellite: check-in ratelimit: node rate limited by id", "errorVerbose": "ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:202\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:156\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-10-29T17:20:01+01:00	ERROR	piecestore:cache	error getting current used space for trash: 	{"error": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: readdir D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-22\\jx: The system cannot find the path specified.", "errorVerbose": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: readdir D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-22\\jx: The system cannot find the path specified.\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).SpaceUsedForTrash:273\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:100\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-10-29T17:20:01+01:00	ERROR	services	unexpected shutdown of a runner	{"name": "piecestore:cache", "error": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: readdir D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-22\\jx: The system cannot find the path specified.", "errorVerbose": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: readdir D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-22\\jx: The system cannot find the path specified.\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).SpaceUsedForTrash:273\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:100\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-10-29T17:20:01+01:00	ERROR	contact:service	ping satellite failed 	{"Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "attempts": 1, "error": "ping satellite: context canceled", "errorVerbose": "ping satellite: context canceled\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:202\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:156\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-10-29T17:20:01+01:00	ERROR	nodestats:cache	Get pricing-model/join date failed	{"error": "context canceled"}
2024-10-29T17:20:01+01:00	ERROR	contact:service	ping satellite failed 	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 1, "error": "ping satellite: context canceled", "errorVerbose": "ping satellite: context canceled\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:202\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:156\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-10-29T17:20:01+01:00	ERROR	failure during run	{"error": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: readdir D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-22\\jx: The system cannot find the path specified.", "errorVerbose": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: readdir D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-22\\jx: The system cannot find the path specified.\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).SpaceUsedForTrash:273\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:100\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-10-29T17:20:01+01:00	FATAL	Unrecoverable error	{"error": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: readdir D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-22\\jx: The system cannot find the path specified.", "errorVerbose": "filestore error: failed to walk trash namespace af2c42003efc826ab4361f73f9d890942146fe0ebe806786f8e7190800000000: readdir D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-22\\jx: The system cannot find the path specified.\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).SpaceUsedForTrash:273\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:100\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2

Is it Windows GUI or docker ?

1 Like
3

Windows GUI install - no docker on windows for me :slight_smile:

4

What does storage.path: in the config.yaml file show ?

1 Like
5

Storage.Path: D:\

Just like

Storage2.database-dir: E:\

additionally, these are the items seen in the root of D:\

image
image987×498 55.9 KB

The dbback folder is from a database related issue I had a year or so ago

6

Can you see this exact path to the directory ?

1 Like
7

Yes, the directory is filled with trash 2-char directories

8

The 2024-10-20 subfolder is missing in some of the other subdirectories in trash. I’m creating them.

While D:\\trash\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\2024-10-22\\ does exist, it is completely emtpy.

Perhaps delete some of the trash databases? :thinking:

9

Did it finish successfully ?

You shouldn’t delete any database unless its the only last solution to the issue.

3 Likes
10

Yes, all checks finished succesfully.

C:\ and E:\ very fast both due to their size and NVMe, D:\ took a few days

11

You can try to create this subfolder. Would it resolve an issue?
Also, please make sure that all data folders have a SYSTEM user as an owner and it should have at least Write, List, Delete, Read permissions.

1 Like
12

Hmmm, suddenly I’m not so sure weather or not there has been an interruption to the windows node or not. the 2024-10-22 did not include a folder named jx - but there was an extensionless file named jx. I removed the file, created the empty folder and restarted the node.

Everything is working as intended now :slight_smile: - I’m marking your post as solution

13

This is usually mean, that you have had a filesystem corruption. Seems you ran chkdsk /f to fix it and it was fixed this way.
I would recommend to perform the scan several times, until it would stop modifying the filesystem. No chkdsk utility will fix all errors in one pass.

14

That’s what I don’t quite understand. The chkdsk finished successfully after running for a few days, but found no errors, and made no changes to the filesystem. The extensionless file, jx was untouched by chkdsk, and modified only when I deleted it.

I’ll write a script searching for extensionless files in all the satellite folders when I get home. I might just have a few more landmines laying around. Cheers friend.

1 Like