Node Suspended?! what do i do!

davidgoderre · January 10, 2022, 11:54pm

I just installed Pihole on my network the other day and now i am getting notified that my node has been suspended. Did i mess up my storj node by installing pi hole? I also go these below when i ran looking at logs?

2-01-10T19:23:41.444Z	ERROR	piecestore	download failed	{"Piece ID": "I7C6ODGHVPLR7M2ZJCW4O6W2VV5WGTYFTONS7RRO247DJETONXDA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/i7/c6odghvplr7m2zjcw4o6w2vv5wgtyftons7rro247djetonxda.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/i7/c6odghvplr7m2zjcw4o6w2vv5wgtyftons7rro247djetonxda.sj1: bad message", "errorVerbose": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/i7/c6odghvplr7m2zjcw4o6w2vv5wgtyftons7rro247djetonxda.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/i7/c6odghvplr7m2zjcw4o6w2vv5wgtyftons7rro247djetonxda.sj1: bad message\n\tstorj.io/storj/storage/filestore.(*Dir).Open:279\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:75\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:262\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:541\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-01-10T19:27:25.458Z	ERROR	piecestore	download failed	{"Piece ID": "ARQ75IJJSSFHWRFVH5IMIMSCHR3LW7UB2LAD2TC4WIFUA633ZVDA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/ar/q75ijjssfhwrfvh5imimschr3lw7ub2lad2tc4wifua633zvda.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/ar/q75ijjssfhwrfvh5imimschr3lw7ub2lad2tc4wifua633zvda.sj1: structure needs cleaning", "errorVerbose": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/ar/q75ijjssfhwrfvh5imimschr3lw7ub2lad2tc4wifua633zvda.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/ar/q75ijjssfhwrfvh5imimschr3lw7ub2lad2tc4wifua633zvda.sj1: structure needs cleaning\n\tstorj.io/storj/storage/filestore.(*Dir).Open:279\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:75\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:262\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:541\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-01-10T19:33:50.207Z	ERROR	piecestore	download failed	{"Piece ID": "6EGTVJSKL5O2DFR2I5CKHCPHAEVMEU74FI2EBK6PSBUIQIHYBYGA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/6e/gtvjskl5o2dfr2i5ckhcphaevmeu74fi2ebk6psbuiqihybyga.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/6e/gtvjskl5o2dfr2i5ckhcphaevmeu74fi2ebk6psbuiqihybyga.sj1: structure needs cleaning", "errorVerbose": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/6e/gtvjskl5o2dfr2i5ckhcphaevmeu74fi2ebk6psbuiqihybyga.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/6e/gtvjskl5o2dfr2i5ckhcphaevmeu74fi2ebk6psbuiqihybyga.sj1: structure needs cleaning\n\tstorj.io/storj/storage/filestore.(*Dir).Open:279\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:75\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:262\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:541\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-01-10T19:33:50.365Z	ERROR	piecestore	download failed	{"Piece ID": "QNYPXKURMXB2I3XKXGTNRP4WLMRIU7H5UFWQ2F3WBRKCPRQTNNXA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/qn/ypxkurmxb2i3xkxgtnrp4wlmriu7h5ufwq2f3wbrkcprqtnnxa.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/qn/ypxkurmxb2i3xkxgtnrp4wlmriu7h5ufwq2f3wbrkcprqtnnxa.sj1: structure needs cleaning", "errorVerbose": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/qn/ypxkurmxb2i3xkxgtnrp4wlmriu7h5ufwq2f3wbrkcprqtnnxa.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/qn/ypxkurmxb2i3xkxgtnrp4wlmriu7h5ufwq2f3wbrkcprqtnnxa.sj1: structure needs cleaning\n\tstorj.io/storj/storage/filestore.(*Dir).Open:279\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:75\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:262\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:541\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-01-10T19:48:04.920Z	ERROR	piecestore	download failed	{"Piece ID": "BJH4KUMB2NH2IPCZOIBTBITGWS5PAU3YMB4WYAMOKWP5YHFZIHYQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/bj/h4kumb2nh2ipczoibtbitgws5pau3ymb4wyamokwp5yhfzihyq.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/bj/h4kumb2nh2ipczoibtbitgws5pau3ymb4wyamokwp5yhfzihyq.sj1: bad message", "errorVerbose": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/bj/h4kumb2nh2ipczoibtbitgws5pau3ymb4wyamokwp5yhfzihyq.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/bj/h4kumb2nh2ipczoibtbitgws5pau3ymb4wyamokwp5yhfzihyq.sj1: bad message\n\tstorj.io/storj/storage/filestore.(*Dir).Open:279\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:75\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:262\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:541\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-01-10T19:51:40.863Z	ERROR	piecestore	download failed	{"Piece ID": "IM64TYG57OCXN3IW5WSUFRQX6NMCKFEHVP6F6WXONICRRVVSF3XA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/im/64tyg57ocxn3iw5wsufrqx6nmckfehvp6f6wxonicrrvvsf3xa.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/im/64tyg57ocxn3iw5wsufrqx6nmckfehvp6f6wxonicrrvvsf3xa.sj1: structure needs cleaning", "errorVerbose": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/im/64tyg57ocxn3iw5wsufrqx6nmckfehvp6f6wxonicrrvvsf3xa.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/im/64tyg57ocxn3iw5wsufrqx6nmckfehvp6f6wxonicrrvvsf3xa.sj1: structure needs cleaning\n\tstorj.io/storj/storage/filestore.(*Dir).Open:279\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:75\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:262\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:541\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-01-10T19:56:50.540Z	ERROR	piecestore	download failed	{"Piece ID": "WUEVD4X2Z46TL6OBHUABDN3WPMZKITBO36CKUA6C3QWU3I2YFAQA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/wu/evd4x2z46tl6obhuabdn3wpmzkitbo36ckua6c3qwu3i2yfaqa.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/wu/evd4x2z46tl6obhuabdn3wpmzkitbo36ckua6c3qwu3i2yfaqa.sj1: bad message", "errorVerbose": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/wu/evd4x2z46tl6obhuabdn3wpmzkitbo36ckua6c3qwu3i2yfaqa.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/wu/evd4x2z46tl6obhuabdn3wpmzkitbo36ckua6c3qwu3i2yfaqa.sj1: bad message\n\tstorj.io/storj/storage/filestore.(*Dir).Open:279\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:75\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:262\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:541\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-01-10T20:06:14.303Z	ERROR	piecestore	download failed	{"Piece ID": "I2IENM4A6IMMVCRJGV7LAMZM5DBJFFQJTSDVMZPDOEHKDORMU2IA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/i2/ienm4a6immvcrjgv7lamzm5dbjffqjtsdvmzpdoehkdormu2ia.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/i2/ienm4a6immvcrjgv7lamzm5dbjffqjtsdvmzpdoehkdormu2ia.sj1: structure needs cleaning", "errorVerbose": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/i2/ienm4a6immvcrjgv7lamzm5dbjffqjtsdvmzpdoehkdormu2ia.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/i2/ienm4a6immvcrjgv7lamzm5dbjffqjtsdvmzpdoehkdormu2ia.sj1: structure needs cleaning\n\tstorj.io/storj/storage/filestore.(*Dir).Open:279\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:75\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:262\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:541\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-01-10T20:09:19.548Z	ERROR	piecestore	download failed	{"Piece ID": "6IVSG3OBNY4EMBWRLR766JA7AMSHBNI6KH2QSGZYVFDBOKXOSFWQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/6i/vsg3obny4embwrlr766ja7amshbni6kh2qsgzyvfdbokxosfwq.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/6i/vsg3obny4embwrlr766ja7amshbni6kh2qsgzyvfdbokxosfwq.sj1: bad message", "errorVerbose": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/6i/vsg3obny4embwrlr766ja7amshbni6kh2qsgzyvfdbokxosfwq.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/6i/vsg3obny4embwrlr766ja7amshbni6kh2qsgzyvfdbokxosfwq.sj1: bad message\n\tstorj.io/storj/storage/filestore.(*Dir).Open:279\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:75\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:262\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:541\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-01-10T20:12:18.608Z	ERROR	piecestore	download failed	{"Piece ID": "BBIZFQB2PIXMUZUFGD7RIOKSTWGGLC564JDDJMBLUYA2JKNR42YA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/bb/izfqb2pixmuzufgd7riokstwgglc564jddjmbluya2jknr42ya.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/bb/izfqb2pixmuzufgd7riokstwgglc564jddjmbluya2jknr42ya.sj1: bad message", "errorVerbose": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/bb/izfqb2pixmuzufgd7riokstwgglc564jddjmbluya2jknr42ya.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/bb/izfqb2pixmuzufgd7riokstwgglc564jddjmbluya2jknr42ya.sj1: bad message\n\tstorj.io/storj/storage/filestore.(*Dir).Open:279\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:75\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:262\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:541\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-01-10T20:13:00.648Z	ERROR	piecestore	download failed	{"Piece ID": "5WJAFHSICLGQ5CFJVWCZM537LD72YAW6L4JCQV2YDAXG7FJA7Z4Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/5w/jafhsiclgq5cfjvwczm537ld72yaw6l4jcqv2ydaxg7fja7z4q.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/5w/jafhsiclgq5cfjvwczm537ld72yaw6l4jcqv2ydaxg7fja7z4q.sj1: structure needs cleaning", "errorVerbose": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/5w/jafhsiclgq5cfjvwczm537ld72yaw6l4jcqv2ydaxg7fja7z4q.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/5w/jafhsiclgq5cfjvwczm537ld72yaw6l4jcqv2ydaxg7fja7z4q.sj1: structure needs cleaning\n\tstorj.io/storj/storage/filestore.(*Dir).Open:279\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:75\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:262\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:541\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-01-10T20:13:38.766Z	ERROR	piecestore	download failed	{"Piece ID": "O34YJ2AERTKTM7C3AC6H6PY34XSDCIYU7ACHWR2HDGA2EIJMLA5A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/o3/4yj2aertktm7c3ac6h6py34xsdciyu7achwr2hdga2eijmla5a.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/o3/4yj2aertktm7c3ac6h6py34xsdciyu7achwr2hdga2eijmla5a.sj1: bad message", "errorVerbose": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/o3/4yj2aertktm7c3ac6h6py34xsdciyu7achwr2hdga2eijmla5a.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/o3/4yj2aertktm7c3ac6h6py34xsdciyu7achwr2hdga2eijmla5a.sj1: bad message\n\tstorj.io/storj/storage/filestore.(*Dir).Open:279\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:75\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:262\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:541\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-01-10T20:18:46.272Z	ERROR	piecestore	download failed	{"Piece ID": "AEQGG7SG3XWHOXTGCOU2TD2NUQEEOF3HII765MBV6XLRZDXI4LOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/ae/qgg7sg3xwhoxtgcou2td2nuqeeof3hii765mbv6xlrzdxi4loq.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/ae/qgg7sg3xwhoxtgcou2td2nuqeeof3hii765mbv6xlrzdxi4loq.sj1: structure needs cleaning", "errorVerbose": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/ae/qgg7sg3xwhoxtgcou2td2nuqeeof3hii765mbv6xlrzdxi4loq.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/ae/qgg7sg3xwhoxtgcou2td2nuqeeof3hii765mbv6xlrzdxi4loq.sj1: structure needs cleaning\n\tstorj.io/storj/storage/filestore.(*Dir).Open:279\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:75\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:262\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:541\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-01-10T20:21:01.415Z	ERROR	piecestore	download failed	{"Piece ID": "NQNQ3PWI2DNMBSMWGDW6ADPRTX6SBXHLL3QPCCFHRUAWTCRQROUQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "error": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/nq/nq3pwi2dnmbsmwgdw6adprtx6sbxhll3qpccfhruawtcrqrouq.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/nq/nq3pwi2dnmbsmwgdw6adprtx6sbxhll3qpccfhruawtcrqrouq.sj1: bad message", "errorVerbose": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/nq/nq3pwi2dnmbsmwgdw6adprtx6sbxhll3qpccfhruawtcrqrouq.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/nq/nq3pwi2dnmbsmwgdw6adprtx6sbxhll3qpccfhruawtcrqrouq.sj1: bad message\n\tstorj.io/storj/storage/filestore.(*Dir).Open:279\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:75\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:262\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:541\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-01-10T22:38:21.399Z	ERROR	piecestore	download failed	{"Piece ID": "IKEW3FNFTH74MTQCAMO4XIV2KFYQ2FGRUB5ONDJ2MEK2YUDCSQXA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_REPAIR", "error": "pieces error: filestore error: unable to open \"config/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/ik/ew3fnfth74mtqcamo4xiv2kfyq2fgrub5ondj2mek2yudcsqxa.sj1\": open config/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/ik/ew3fnfth74mtqcamo4xiv2kfyq2fgrub5ondj2mek2yudcsqxa.sj1: bad message", "errorVerbose": "pieces error: filestore error: unable to open \"config/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/ik/ew3fnfth74mtqcamo4xiv2kfyq2fgrub5ondj2mek2yudcsqxa.sj1\": open config/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/ik/ew3fnfth74mtqcamo4xiv2kfyq2fgrub5ondj2mek2yudcsqxa.sj1: bad message\n\tstorj.io/storj/storage/filestore.(*Dir).Open:279\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:75\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:262\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:541\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-01-10T23:02:59.544Z	ERROR	piecestore	download failed	{"Piece ID": "QXBUGBX542SIBN2FGXK6LASNU4FY3CACS6EU67FI6BDJZY6QKY3Q", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_REPAIR", "error": "pieces error: filestore error: unable to open \"config/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/qx/bugbx542sibn2fgxk6lasnu4fy3cacs6eu67fi6bdjzy6qky3q.sj1\": open config/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/qx/bugbx542sibn2fgxk6lasnu4fy3cacs6eu67fi6bdjzy6qky3q.sj1: bad message", "errorVerbose": "pieces error: filestore error: unable to open \"config/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/qx/bugbx542sibn2fgxk6lasnu4fy3cacs6eu67fi6bdjzy6qky3q.sj1\": open config/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/qx/bugbx542sibn2fgxk6lasnu4fy3cacs6eu67fi6bdjzy6qky3q.sj1: bad message\n\tstorj.io/storj/storage/filestore.(*Dir).Open:279\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:75\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:262\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:541\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-01-10T23:17:59.281Z	ERROR	piecestore	download failed	{"Piece ID": "FLEWCCAKILEFG6EAMZCIOTTLQ3WGR4MYS6GS2STPN5M4BMD5UVVA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_REPAIR", "error": "pieces error: filestore error: unable to open \"config/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/fl/ewccakilefg6eamzciottlq3wgr4mys6gs2stpn5m4bmd5uvva.sj1\": open config/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/fl/ewccakilefg6eamzciottlq3wgr4mys6gs2stpn5m4bmd5uvva.sj1: structure needs cleaning", "errorVerbose": "pieces error: filestore error: unable to open \"config/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/fl/ewccakilefg6eamzciottlq3wgr4mys6gs2stpn5m4bmd5uvva.sj1\": open config/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/fl/ewccakilefg6eamzciottlq3wgr4mys6gs2stpn5m4bmd5uvva.sj1: structure needs cleaning\n\tstorj.io/storj/storage/filestore.(*Dir).Open:279\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:75\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:262\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:541\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}

twl · January 11, 2022, 8:19am

Sounds like a corrupt filesystem to me - or probably a permission issue that prevents your blob folder from being accessed

BrightSilence · January 11, 2022, 12:48pm

This part definitely points to file system issues, please run fsck or equivalent for your file system to fix file system issues.

davidgoderre · January 11, 2022, 3:39pm

How do I run fsck? I’m hosting my node on a rpi4. And should I be running fsck on the rpi4 SD card or on the HDD that is the storj node?

SGC · January 11, 2022, 3:47pm

everything relevant to your storagenode will be on the storagenode hdd.
the sd card will hold stuff like the docker image / software binaries and such…

i would fsck the hdd, but checking the sd card afterwards might also be a good idea, and you should certainly shutdown your node until you have identified the issue.

davidgoderre · January 11, 2022, 3:51pm

ok got it. Am i able to run fsck from the rpi4 that hosts the node? or should i unplug the drive and plug it into a different rpi4 or different linux machine?

Basically the steps for doing this correctly would be to stop the node

docker stop -t 300 storagenode

then run fsck on the hdd?

SGC · January 11, 2022, 3:55pm

yes and then you should be able to run fsck directly on the rpi.
no real advantage in moving the hdd, unless if you feel like doing that for some reason.

davidgoderre · January 11, 2022, 4:04pm

I just didnt know if i needed to move it so it wasnt mounted or in use by the host rpi4?

BrightSilence · January 11, 2022, 4:07pm

You will have to unmount your file system, but you can do it from the rpi just fine. So stop your node, unmount the filesystem and run fsck on it.

Something like

docker stop -t 300 storagenode
umount /dev/sda1
fsck -p /dev/sda1
mount /dev/sda1

Of course please check the name of your storagenode partition. It may not be /dev/sda1 in your case.

Edit: I guess if you use the HDD as root filesystem, then yes you need to connect it to something else to do this.

davidgoderre · January 11, 2022, 4:30pm

Thank for the help! you have been awesome!

/dev/sda contains a file system with errors, check forced.

This is what i get when i run fsck -p /dev/sda, and it seems to be stuck at this point. Is there another command i should run?

davidgoderre · January 11, 2022, 4:35pm

edit, it came back with this

sudo fsck -p /dev/sda
fsck from util-linux 2.33.1
/dev/sda contains a file system with errors, check forced.
/dev/sda: Unattached inode 80921148


/dev/sda: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
	(i.e., without -a or -p options)

davidgoderre · January 11, 2022, 4:48pm

im stepping through it manually and fixing things 1 by 1, entering y. hoping this helps. lol

davidgoderre · January 11, 2022, 5:28pm

so it is just going through a million of the following code

Unattached inode
connect to /lost-found? yes?
inode(some number) ref count is 2 and should be 1. fix? yes?

it is running through this like a billion times. is this good?

davidgoderre · January 11, 2022, 6:54pm

any ideas on how long this process will take on a 5tb drive? its been running a while? i am afriad my node will be disqualified

davidgoderre · January 11, 2022, 7:09pm

i stopped the fsck as it was taking forever. i mounted and started my node and will see what happens.

BrightSilence · January 11, 2022, 7:14pm

I believe you can enter a to do yes for all.

File system issues are caused by something. It could be HDD damage, but more often than not, it’s unsafe shut downs or power cuts etc.

You will only get suspended after being offline for 12 days and you’ll still get a month to recover from that afterwards only if your uptime is still below 60% 30 days later will you be disqualified. Don’t worry about uptime right now. Priority 1 is getting the file system fixed.

It might take hours, just be patient. The issues need to be fixed.

This is a bad idea. If the issues aren’t fixed, you will fail audits and repairs and get disqualified much faster. Please fix your file system issues first if you want to save your node.

davidgoderre · January 11, 2022, 7:26pm

dang it. ok ill restart the fsck. i thought i would be disqualified but your information is really helpful. thanks for the help on this. seriously, i really appreciate it!

davidgoderre · January 11, 2022, 10:49pm

Things seem to be better. Will these suspension, and audit numbers improve?

SGC · January 11, 2022, 10:59pm

audits is what matters, suspension at 60% means no ingress / no growth of the node for a while.
if the audit stay reasonable the node will be fine.
the audit will flux for… well atleast a year… but so long as they don’t reach 60% its fine…
fingers crossed.

BrightSilence · January 11, 2022, 11:15pm

If the issue is fixed they should recover with each audit or repair download. Assuming there was no permanent data loss. Which is probably not the case if you only had filesystem problems. Shouldn’t take too long to recover, but it is dependent on how many audits you get, which in turn is dependent on how much data is stored on your node for each satellite. The scores only update every once in a while though, they’re not real time. So look back over the next few days, but also keep an eye on your logs to make sure those errors don’t pop up anymore.

This would only be true if there is permanent data loss, which I don’t think applies here. And if that is the case there is no end time frame, you’ll basically see fluctuating scores forever, because it’s highly unlikely that all missing data will be removed by customers.