Disqualified from all satellites

[Proto03]

Hello all!

Hoping I can find some help here. So I have been running a node for almost a year now. My original 2TB hard drive is now full and so this morning I decided to add another one to the node. I set up the drive according to the instructions in the documentation and created a new docker instance using:

docker run --rm -e SETUP="true" \
    --mount type=bind,source="/home/pi/.local/share/storj/identity/storagenode/",destination=/app/identity \
    --mount type=bind,source="/mnt/HDD2/",destination=/app/config \
    --name storagenode storjlabs/storagenode:latest

I then started both containers and everything looked fine in the node.

To my surprise, I came back to the node this evening only to find the node disqualified from all the satellites and I was wondering if someone can tell me the extent of the damage here. Here is the log from the new container:

$ docker logs Node02 2>&1 | grep -E 'GET_AUDIT' | grep 'failed'
2022-02-11T13:12:59.524Z	ERROR	piecestore	download failed	{"Piece ID": "PGEZABHDBO2XDDNJQROUGO7X5VL35PIJCZGSCUIMRXVOTHB5VBAA", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "Action": "GET_AUDIT", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:545\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-02-11T13:22:11.726Z	ERROR	piecestore	download failed	{"Piece ID": "MLIQUN57WGGMSONGVYEKNGE7UEEKZ7AAMHWXXQFD2CERRE3ECNCQ", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "Action": "GET_AUDIT", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:545\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-02-11T13:23:55.654Z	ERROR	piecestore	download failed	{"Piece ID": "VX3FF55JFJ7DK36RTCSBJU3FFVHDSRQWLLK4NKCYCLT4CE64IGVA", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "Action": "GET_AUDIT", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:545\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-02-11T13:24:57.795Z	ERROR	piecestore	download failed	{"Piece ID": "MC2XODMKAPFE6KKZJQNI7ROV34WGOQ6CEFJ4DGYSSGYNOKCDDTNA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:545\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-02-11T13:25:42.987Z	ERROR	piecestore	download failed	{"Piece ID": "G4E5YD5X3IWSC4JZHQZPSMTPNJUEXV5KSCMUUVEE3KTMUWSJMUSA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:545\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-02-11T13:25:54.753Z	ERROR	piecestore	download failed	{"Piece ID": "5YFKAGUFT3B24Q4G726WU7ML3Q2UDJNXBVXMYSK7JAYRR65HGXCA", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "Action": "GET_AUDIT", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:545\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-02-11T13:30:31.502Z	ERROR	piecestore	download failed	{"Piece ID": "WFE2QEO54Q52VGSUGZCMIYBIUB3CDE6QUDPKZZDHQYY5CWR3HZPQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "GET_AUDIT", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:545\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-02-11T13:32:57.622Z	ERROR	piecestore	download failed	{"Piece ID": "XMGMQIQQJZ4TNIUNGIJH5QDDKSZQM364VSCEVL3ZT42OCJOQPZOA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "GET_AUDIT", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:545\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-02-11T13:49:22.207Z	ERROR	piecestore	download failed	{"Piece ID": "QOAUNFS6ISVXEAAIPIFRERY7SW3TDL7T6F4K3AC2QOCKX4HFRWOQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET_AUDIT", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:545\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-02-11T13:49:26.657Z	ERROR	piecestore	download failed	{"Piece ID": "XDJ66XXIFXN5ZIA7KLTVF7BHPITCY3RIIY2RJSA3ACG45AJ56PEQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:545\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-02-11T13:53:21.609Z	ERROR	piecestore	download failed	{"Piece ID": "GESJH7I5YSP3GBXBRQXK5A6U7AG3OUPF3SX5ILBFBHXOLFSNYWFA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:545\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-02-11T13:59:41.517Z	ERROR	piecestore	download failed	{"Piece ID": "BGGZHWZ7Z22TR2E4TSN5VP4KJLMTVZZTNOMPAVWESYNLKSZJKOWA", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "Action": "GET_AUDIT", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:545\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-02-11T14:01:44.077Z	ERROR	piecestore	download failed	{"Piece ID": "FMC5JQ55O2LKFXKQXMYWXCBZ57LH3DFZGT4HYTJCDQF6EHRTFXOQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:545\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-02-11T14:02:51.461Z	ERROR	piecestore	download failed	{"Piece ID": "X2ORVC53GXCJXP35RE6CILPZYPZMMEXONGBW7PMAFG3NRJBL3M5A", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "GET_AUDIT", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:545\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-02-11T14:18:26.893Z	ERROR	piecestore	download failed	{"Piece ID": "PMD6RQUK3FR3FCTIHOV6ZPXSYYTTSYT2VNPNKCLZT2MR2DBJEJJA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:545\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-02-11T14:25:38.102Z	ERROR	piecestore	download failed	{"Piece ID": "W5XQ7PVLTCDNGPQ67S5ILW74N4VL5OKHURIMPFWY2USURZYIN6RA", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "Action": "GET_AUDIT", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:545\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-02-11T14:28:28.093Z	ERROR	piecestore	download failed	{"Piece ID": "2GN3VJL2HWFP4BY3OBT525EPB76BBYMGWOAJDGNVO6OTT5Z6SF6Q", "Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "Action": "GET_AUDIT", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:545\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-02-11T14:44:12.421Z	ERROR	piecestore	download failed	{"Piece ID": "ESASDT5ULD6JH4VPCPNIJCNXXRGLOE6FYNTTAKH6MMDHS2TVRGXA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:545\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-02-11T14:52:32.471Z	ERROR	piecestore	download failed	{"Piece ID": "WJCZRKCXQ4BEE22NATZLCEZMY4ZB6EDPUFPXX6F77LSLDYLOLPUQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:545\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-02-11T15:44:53.909Z	ERROR	piecestore	download failed	{"Piece ID": "LFI5TCN3B7ZVNQCTIRTYDSIE6TQKTYI6RKQAJ2CG5GTT4EBQQNBQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:545\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-02-11T15:53:37.195Z	ERROR	piecestore	download failed	{"Piece ID": "CHRGHBOYZRPPK553URHG3NVHLE5LQHS7WPE72GRXH4V6YIXSGUIQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:545\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-02-11T16:09:10.002Z	ERROR	piecestore	download failed	{"Piece ID": "B663YD4SAUWLCLDJHXPG56K365GHU6U3SZOFA2DETPMAZ3VK6VWA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "GET_AUDIT", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:545\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-02-11T17:05:49.326Z	ERROR	piecestore	download failed	{"Piece ID": "7KB2J4O3VGX6AR5H4EVICSSDH6C2HY4RZRCZL3P66ONSRRVMUTMQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "GET_AUDIT", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:545\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}

How did adding a new drive cause the node to be disqualified like this? And what is the extent of the damage? If I have to start anew, can I at least do a graceful exit and recoup the earnings?

Thanks for your help!

[baker]

Hi Proto,

Sorry to see you are having trouble. A few questions for you. Is it the new node that was DQ’d, the old node, or both nodes? Did you create a new identity for the new node, or did you use the old identity? Are your identity files located on the respective drives, or in the default location?

Unfortunately disqualification is permanent, and once disqualified the node no longer qualifies for graceful exit. But hopefully we can get to the bottom of why this happened.

[Alexey]

@Proto03

My original 2TB hard drive is now full and so this morning I decided to add another one to the node

This is not possible. You should create a new node with a new generated identity signed with a new authorization token on own external port if the external address is the same.

But you just cloned the identity, this is leading to quick disqualification, because each node reported to the satellites about missed pieces from another disk.

You must generate a new identity for the next node. Only authorization token is not enough - it will be the same identity ergo the same node but with totally missed data.

See How to add an additional drive? - Storj Docs

[anon27637763]

@Alexey

This is not possible.

I gently disagree…
The new 2TB drive could have been added if the drive space had been included in some type of filesystem expansion…

• LVM
• ZFS pool

or something similar.

---

It appears the OP accidentally replaced the original node’s storage location with the new 2TB drive and the node correctly indicated that the new 2TB drive does not have any of the files stored on the original – and still operational – drive.

To OP:

It’s unfortunate, but you’ve misinterpreted the instructions and the general layout architecture of a node. The result is permanent disqualification of your node’s ID. You can obtain 2 new IDs and restart both nodes separately, one on the new drive and on on the older drive.

[Alexey]

@anon27637763

I gently disagree…
The new 2TB drive could have been added if the drive space had been included in some type of filesystem expansion…

RAID0. No way. Please, never do this, unless you want to lose everything with a one disk failure.
As I said - it’s not possible to add a one disk to the existing node.
Technically it’s possible, but not for the node, if you wish it to survive.

[anon27637763]

@Alexey

RAID0. No way. Please, never do this,

Of course not…

ZFS can add new drives, no problem.

https://unix.stackexchange.com/questions/530968/adding-disks-to-zfs-pool

[Alexey]

This is will be still RAID0. Please, ask expert @SGC if you do not believe me.

ZFS doesn’t support adding disks to existing RAIDZ pools with parity re-configuring. So, even this beta feature is technically available for ZFS, it’s useless - the speed will not increase, but risk of something would go wrong is increasing.

You can do only a mirror. This is not possible for disks of different sizes.

The ZFS has a lot more restrictions than LVM for example.
The mirror doesn’t help to extend storage.

So, adding only one disk is not possible. You need to convert the whole setup to something like RAID5/RAID6/RAIDZ/RAIDZ2. And this is requires more than one disk.
And I do not recommend to use RAID5/RAID6 without checksums and auto correction anyway. So the only ZFS is viable at the moment.
Do not offer BTRFS. It’s alpha and not ready for production independently what they are saying. Their RAID implementation is nightmare and the route to data loss.

For the node the RAID is not needed, the network itself has a redundancy, this additional local redundancy will not be paid, thus - waste of resources and wearing disks much quickly than with a 1:1 node:HDD setup.

Please see also

RAID vs No RAID choice

That’s why you use a RAID configuration. If none was present and your drive died you can pretty much start over now. But it’s a good time to learn about RAIDs As an SNO you are responsible that the data you are storing is always available, if it is not you are failing audits which will result in suspension and eventually in disqualification. In your case if there is no RAID configuration and no backup your identity is lost. You will have to create a new identity and start over. And try to …

[anon27637763]

@Alexey

For the node the RAID is not needed,

I admire your knowledge and tenacity…

But when do you sleep?

---

I defer to the master of ZFS…

@SGC

ZFS… how difficult is it to add a drive to a pool ?

[Alexey]

@anon27637763

But when do you sleep?

I’m currently in UTC+7 zone (Thailand) - I need to recover my health after covid.

[Alexey]

@anon27637763

ZFS… how difficult is it to add a drive to a pool ?

It’s not difficult to add to the plain pool, and this is a RAID0 or more like JBOD. You can also add it to the mirror pool (but you need another one disk with the same size).
You can use a beta feature of ZFS on Linux (ZOL) to add a disk to the RAIDZ pool, but it will not change the parity configuration, see

ZFS fans, rejoice—RAIDz expansion will be a thing very soon

There are some tricks with vdev though, but it’s greatly increases complexity.

How to add an additional drive to an existing ZFS RAIDZ-1 in Proxmox?

[SGC]

@anon27637763

ZFS… how difficult is it to add a drive to a pool ?

lol, thanks … i think one can use dRAIDz today, because it writes in a different way more similar to the way that commercial raid’s write across the disks, with dRAIDz one can add and remove HDD’s without issue.

however using the old RAIDz method one may be be able to add or remove disks… but its not recommended… it will make a lot of mess, its more of an emergency procedure if one cannot move the data from one pool to a new pool.

personally i add 6 drives at a time, basically created and adding a full RAIDz1 to my pool each time, and don’t think i’m able to take them out again…

but dRAIDz is the new fancy thing… its very new tho, but its supported in my fully upgrade ZFS on fully upgrade Proxmox.

sadly one cannot convert from RAIDz to dRAIDz
but am considering it for my next upgrade, sure would make some things a lot easier, but usually there are advantages and disadvantages to every solution.
https://openzfs.github.io/openzfs-docs/Basic%20Concepts/dRAID%20Howto.html

[penfold]

Best wishes for a speedy recovery @Alexey.

[Proto03]

Thank you all for the replies. I feel so disappointed that an entire year of reputation building went down the drain with such a mistake.

My next question is if it’s possible to still get my undistributed payout at least? Otherwise this whole year of operating the node will have been for nothing.

[anon27637763]

@Proto03

My next question is if it’s possible to still get my undistributed payout at least?

The monthly payouts are definitely yours. I would imagine that the payout policy for undistributed funds is the same for disqualified nodes as it is for regular nodes. If your payout value is below the threshold, you’ll need to opt-in to either zkSync or Polygon.

It’s important to note that disqualified nodes lose any “Held” amounts in the escrow.

@Proto03

Otherwise this whole year of operating the node will have been for nothing.

If you apply for two new node IDs, you can continue running and earning STORJ… starting again from Month 1 on new nodes instead of just one.

[Proto03]

Ok thank you. And one last question, after I opt-in to zkSync or Polygon, do I have to wait until the end of the month for the payout and therefore to be able to start the new nodes?

[anon27637763]

@Proto03

do I have to wait until the end of the month for the payout

My guess is… yes. But I don’t know for sure. Maybe @littleskunk or @Alexey can answer this one better…

[Alexey]

If you opted-in only recently, then - yes. The change to the wallet or its options will be applied to the next payout period (two first weeks of the next month).

[jammerdan]

@Proto03

and therefore to be able to start the new nodes?

But you don’t have to wait until then to start new node.
@Alexey please correct me if I am wrong.

[Alexey]

You don’t need to wait to start a new node.

[Proto03]

Thank you everyone for the very helpful replies!

Time to start some new nodes I think