Storage node error - suspended

[ZhiYuan]

hi team,

i am getting this on my node. is it something i need to do?

2022-06-09T11:24:58.314+0800 INFO Public server started on [::]:28967
2022-06-09T11:24:58.314+0800 INFO Private server started on 127.0.0.1:7778
2022-06-09T11:24:59.579+0800 INFO trust Scheduling next refresh {“after”: “8h44m21.847690605s”}
2022-06-09T11:24:59.580+0800 INFO bandwidth Performing bandwidth usage rollups
2022-06-09T11:24:59.583+0800 ERROR collector unable to delete piece {“Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Piece ID”: “6CXSQ6JYAKVLV6NHON6R5EGIWF7BXKLACJCJRAZSS5I62QR7CC2A”, “error”: “pieces error: filestore error: file does not exist”, “errorVerbose”: “pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2022-06-09T11:24:59.584+0800 ERROR collector unable to delete piece {“Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Piece ID”: “G2TEUY2SW6VT37N3QN4TMJZBQ26X6IC4VLNF24SHSV7FCZON6UGQ”, “error”: “pieces error: filestore error: file does not exist”, “errorVerbose”: “pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2022-06-09T11:25:07.702+0800 INFO orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE sending {“count”: 76}
2022-06-09T11:25:07.702+0800 INFO orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 sending {“count”: 13}
2022-06-09T11:25:07.702+0800 INFO orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S sending {“count”: 109}
2022-06-09T11:25:07.702+0800 INFO orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs sending {“count”: 5}
2022-06-09T11:25:07.964+0800 INFO orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 finished
2022-06-09T11:25:08.436+0800 INFO orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs finished
2022-06-09T11:25:08.460+0800 INFO orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE finished
2022-06-09T11:25:08.885+0800 INFO orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S finished
2022-06-09T11:25:18.525+0800 INFO piecestore download started {“Piece ID”: “ZFRSZNUEWXB2T3SPLGHG7HNN4NVZDQJGNHF5XYTBBKU73VY4MQJQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”}
2022-06-09T11:25:30.970+0800 INFO piecestore download started {“Piece ID”: “YYM5I3Z6JOQHMOLIKVW6SPZDQFBQJRXHNNRUB5VBORAV7M6II45A”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “GET”}
2022-06-09T11:25:35.158+0800 INFO piecestore downloaded {“Piece ID”: “YYM5I3Z6JOQHMOLIKVW6SPZDQFBQJRXHNNRUB5VBORAV7M6II45A”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “GET”}
2022-06-09T11:25:39.751+0800 INFO piecestore download started {“Piece ID”: “WC5M5UIDTGPQUNRXHTRMZICUW2ZD2NA4DEAAYOD2JTDCY4FBYVKA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”}
2022-06-09T11:25:40.005+0800 INFO piecestore download started {“Piece ID”: “WC5M5UIDTGPQUNRXHTRMZICUW2ZD2NA4DEAAYOD2JTDCY4FBYVKA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”}
2022-06-09T11:25:41.664+0800 INFO piecestore download canceled {“Piece ID”: “WC5M5UIDTGPQUNRXHTRMZICUW2ZD2NA4DEAAYOD2JTDCY4FBYVKA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”}
2022-06-09T11:25:41.665+0800 INFO piecestore downloaded {“Piece ID”: “WC5M5UIDTGPQUNRXHTRMZICUW2ZD2NA4DEAAYOD2JTDCY4FBYVKA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”}
2022-06-09T11:25:52.306+0800 INFO piecestore download started {“Piece ID”: “FZMKJSLT3WDK26O23EMY5YEDHYDU6EPWVZHPHHE4A7PJ6L22PVZA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “GET”}
2022-06-09T11:25:55.779+0800 INFO piecestore downloaded {“Piece ID”: “FZMKJSLT3WDK26O23EMY5YEDHYDU6EPWVZHPHHE4A7PJ6L22PVZA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “GET”}
2022-06-09T11:26:14.589+0800 INFO piecestore downloaded {“Piece ID”: “ZFRSZNUEWXB2T3SPLGHG7HNN4NVZDQJGNHF5XYTBBKU73VY4MQJQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”}
2022-06-09T11:26:57.850+0800 INFO piecestore download started {“Piece ID”: “4ENUL7YOTOI5RHMREMCV2TDIEAMK3G3AUWA62EFPTZUEEC4HBDRQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “GET”}
2022-06-09T11:27:01.079+0800 INFO piecestore downloaded {“Piece ID”: “4ENUL7YOTOI5RHMREMCV2TDIEAMK3G3AUWA62EFPTZUEEC4HBDRQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “GET”}
2022-06-09T11:27:03.236+0800 INFO piecestore download started {“Piece ID”: “AUUGKPZBMOKBKU24BYJERSOHX5TX6PEOV4FXYFPJVCWBKWPYSZ5A”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”}
2022-06-09T11:27:04.121+0800 INFO piecestore download canceled {“Piece ID”: “AUUGKPZBMOKBKU24BYJERSOHX5TX6PEOV4FXYFPJVCWBKWPYSZ5A”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”}
2022-06-09T11:27:10.742+0800 INFO piecestore download started {“Piece ID”: “EUA6Z6HQPEEENIQMBPD6ZZDBLYXEDF4JTYDPFJFZWWTFCW22MXMQ”, “Satellite ID”: “12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo”, “Action”: “GET_AUDIT”}
2022-06-09T11:27:11.648+0800 INFO piecestore downloaded {“Piece ID”: “EUA6Z6HQPEEENIQMBPD6ZZDBLYXEDF4JTYDPFJFZWWTFCW22MXMQ”, “Satellite ID”: “12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo”, “Action”: “GET_AUDIT”}

[Alexey]

@ZhiYuan

2022-06-09T11:24:59.584+0800 ERROR collector unable to delete piece

This error should gone after a week. If not, you can use this workaround:

ERROR collector unable to delete piece

Each satellite has their own folder in the blobs folder in your storage location. For the satellite this piece is on that’s v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa. The subfolders you see inside there represent the first 2 characters of the piece id. So in your case the folder should be fr. The filename inside that folder is the rest of the piece id with a .sj1 extension. So jgxef52maxhcihttdsbjsez6vpmdqqw6l7zxp2jykasd37pusq.sj1. Path and filenames are in lower case, which is a litt…

For suspension troubleshooting you can this guide:

Suspension mode

However, if you’re using the docker version and haven’t set up log redirection, your past logs will be gone after you re-create the container, and you may not find any past errors that caused the suspension.