Node suspended :(

Hello!

I started some days ago , it was working fine and now I get the msg :frowning:

Your node has been suspended on 118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE

it is permanent ? 2 satellites from 6 on the list

You got suspended, not disqualified. The suspension is temporary if you manage to sort out the problem.

Find GET_AUDIT messages in the log and try to figure out if there is some error.

I get Some error about Download failed and “userserialsdb error: disk I/o error”

it is a new usb 3 disk

Is it Windows or Linux node ?

1 Like

Download failed is not a problem, disk I/O error is critical. It means that the operating system cannot read from the HDD, probably because of hardware failure.

If using a USB HDD, you should use a split data/power Y cable to give more current to your HDD.

If this is your problem it will be fixed in v1.6.3 that is rolling out right now. Your node will recover automatically. This database will be moved to RAM to solve the locking issue.

GET_AUDIT lines with this error will also be shown as download failed. So it likely is the reason for suspension.

Yes, I meant download failed for normal GET.

Try to look for failed GET_AUDIT specifically. But others have reported the same issue on audits, so it’s likely what caused it. I’m still fairly certain v1.6.3 should fix this issue for you.

thanks for the answers guys!

it is linux

now it is showing 1 satellite suspended not 2

1 Like

This problem raises concerns from me. Can you check your disk for errors?

1 Like

Yep , Going to run test on the disk

—edit–

Do this cause the node get suspended ?

ERROR piecestore download failed {“Piece ID”: “IQU3OSJZIZCVG6FSMVX3MZNRNFNUBSE6RWLEIH2NORV6R2BEUMCA”, “Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “Action”: “GET_AUDIT”, “error”: “usedserialsdb error: disk I/O error”, “errorVerbose”: “usedserialsdb error: disk I/O error\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:459\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1004\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-06-11T06:48:44.385Z INFO piecestore download started {“Piece ID”: “E4RKXCGZGLVOAQPUXSRG3Z35LMXLZ6OBICAAQNCSHOQQM46Z47YA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “GET_AUDIT”}
2020-06-11T06:48:44.393Z ERROR piecestore download failed {“Piece ID”: “E4RKXCGZGLVOAQPUXSRG3Z35LMXLZ6OBICAAQNCSHOQQM46Z47YA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “GET_AUDIT”, “error”: “usedserialsdb error: disk I/O error”, “errorVerbose”: “usedserialsdb error: disk I/O error\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:459\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1004\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-06-11T13:47:05.513Z INFO piecestore download started {“Piece ID”: “BIWFRKIYPA2QPLPH7SZDY6LXITIRALU4ZCOZX7YOLFYF6BJYBYFQ”, “Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “Action”: “GET_AUDIT”}
2020-06-11T13:47:05.524Z ERROR piecestore download failed {“Piece ID”: “BIWFRKIYPA2QPLPH7SZDY6LXITIRALU4ZCOZX7YOLFYF6BJYBYFQ”, “Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “Action”: “GET_AUDIT”, “error”: “usedserialsdb error: disk I/O error: input/output error”, “errorVerbose”: “usedserialsdb error: disk I/O error: input/output error\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:459\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1004\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-06-11T15:21:41.662Z INFO piecestore download started {“Piece ID”: “QM5E6G5R6FX5QJGNUZUPKD5FROEFYMBKTGVOVQRWEZYYXWC37NTA”, “Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “Action”: “GET_AUDIT”}
2020-06-11T15:21:41.674Z ERROR piecestore download failed {“Piece ID”: “QM5E6G5R6FX5QJGNUZUPKD5FROEFYMBKTGVOVQRWEZYYXWC37NTA”, “Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “Action”: “GET_AUDIT”, “error”: “usedserialsdb error: disk I/O error: input/output error”, “errorVerbose”: “usedserialsdb error: disk I/O error: input/output error\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:459\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1004\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-06-11T20:31:49.479Z INFO piecestore download started {“Piece ID”: “BTD7MKCPTKZCBARPDMWZG6YXPJZ577LIQVBQAMILLJM5B6ZVGGJQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “GET_AUDIT”}
2020-06-11T20:31:49.491Z ERROR piecestore download failed {“Piece ID”: “BTD7MKCPTKZCBARPDMWZG6YXPJZ577LIQVBQAMILLJM5B6ZVGGJQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “GET_AUDIT”, “error”: “usedserialsdb error: disk I/O error: input/output error”, “errorVerbose”: “usedserialsdb error: disk I/O error: input/output error\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:459\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1004\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-06-11T22:01:04.505Z INFO piecestore download started {“Piece ID”: “TGXWEFUKQMZCBWGNSLG6ZD34OWENEIJI6YN2UNBB5UEGPPD7JLKQ”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET_AUDIT”}
2020-06-11T22:01:04.517Z ERROR piecestore download failed {“Piece ID”: “TGXWEFUKQMZCBWGNSLG6ZD34OWENEIJI6YN2UNBB5UEGPPD7JLKQ”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET_AUDIT”, “error”: “usedserialsdb error: disk I/O error: input/output error”, “errorVerbose”: “usedserialsdb error: disk I/O error: input/output error\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:459\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1004\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-06-12T03:55:14.382Z INFO piecestore download started {“Piece ID”: “KKQRQVIOK3LKVVFQ55KZGOQUDT5XJJLU3XUSI65LT4ASJNHGZPYA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET_AUDIT”}
2020-06-12T03:55:14.391Z ERROR piecestore download failed {“Piece ID”: “KKQRQVIOK3LKVVFQ55KZGOQUDT5XJJLU3XUSI65LT4ASJNHGZPYA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET_AUDIT”, “error”: “usedserialsdb error: disk I/O error: input/output error”, “errorVerbose”: “usedserialsdb error: disk I/O error: input/output error\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:459\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1004\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}

1 Like

Do I have to do/fix something or just wait ?

1 Like

This is the reason your node is suspended. You need to fix it before your node is disqualified. What did the test return for disk error ?

1 Like

I think I move it accidentally when I was fixing/testing the setup and the USB cable got disconnected

didn’t found errors on the hard drive :slight_smile:

Apparently I no longer have the node suspended :slight_smile:

it is normal on the node stat that the “Last contact” its negative ? shouldn’t it be positive?

Thanks

1 Like

Yes it should be positive. Try restarting your setup once.

1 Like

I reboot and like 5 minutes before it start to show :frowning:

LAST CONTACT

-419m ago

1 Like

Is it showing ONLINE now ?

1 Like

always show it as online, even if it say LAST CONTACT : -420

Could you show the last 20 lines of the log?

2020-06-13T17:30:20.505Z INFO piecestore upload canceled {“Piece ID”: “VZ32O6Y55ZJVDDQYBMMNWQVIRPCJZRR4IXBRR5R3VYEMJLS2UXSA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT_REPAIR”}
2020-06-13T17:31:08.092Z INFO piecestore upload started {“Piece ID”: “RHEHENPIH7PXAGLMSCY722M76A3LQLS25WEBL2HC36SBJJBGRS5Q”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Available Space”: 920250585344}
2020-06-13T17:31:08.609Z INFO piecestore upload canceled {“Piece ID”: “RHEHENPIH7PXAGLMSCY722M76A3LQLS25WEBL2HC36SBJJBGRS5Q”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”}
2020-06-13T17:31:17.244Z INFO piecestore upload started {“Piece ID”: “AL2E56VURJ345BU4WMTFELN2AZPZJSXARXAEEIJ7HCK7USRUWD3A”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”, “Available Space”: 920250578176}
2020-06-13T17:31:17.371Z INFO piecestore upload canceled {“Piece ID”: “AL2E56VURJ345BU4WMTFELN2AZPZJSXARXAEEIJ7HCK7USRUWD3A”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT”}
2020-06-13T17:31:20.121Z INFO piecedeleter deleted {“Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Piece ID”: “AL2E56VURJ345BU4WMTFELN2AZPZJSXARXAEEIJ7HCK7USRUWD3A”}
2020-06-13T17:31:25.885Z INFO piecestore upload started {“Piece ID”: “IL2BDDUWC4SY237DEDHAMXEC22NX5CFQJTJCL44TTRVRE7VUWAHQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 920250578176}
2020-06-13T17:31:26.014Z INFO piecestore upload canceled {“Piece ID”: “IL2BDDUWC4SY237DEDHAMXEC22NX5CFQJTJCL44TTRVRE7VUWAHQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”}
2020-06-13T17:31:59.050Z INFO piecestore upload started {“Piece ID”: “EAYCLQY4E3OB7GTUFGFPBPKUQJMGE3N37ROPGUZDHPV3DXNX7I7A”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Available Space”: 920250575616}
2020-06-13T17:32:06.540Z INFO piecestore upload canceled {“Piece ID”: “EAYCLQY4E3OB7GTUFGFPBPKUQJMGE3N37ROPGUZDHPV3DXNX7I7A”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”}
2020-06-13T17:32:26.900Z INFO piecestore upload started {“Piece ID”: “PKOT65UPIX3QG77LIZFPTF4IAPXET2K3VOBRIIPYIO4MWQ57D2WA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT_REPAIR”, “Available Space”: 920248255744}
2020-06-13T17:32:40.142Z INFO piecestore upload canceled {“Piece ID”: “PKOT65UPIX3QG77LIZFPTF4IAPXET2K3VOBRIIPYIO4MWQ57D2WA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “PUT_REPAIR”}
2020-06-13T17:33:19.648Z INFO piecestore upload started {“Piece ID”: “6D4XKC4GUK2YMU6QZTXXLY4BWIZB54UTEXT7KK474JOAUU2PYAQQ”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “PUT”, “Available Space”: 920245935872}
2020-06-13T17:33:19.751Z INFO piecestore upload canceled {“Piece ID”: “6D4XKC4GUK2YMU6QZTXXLY4BWIZB54UTEXT7KK474JOAUU2PYAQQ”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “PUT”}
2020-06-13T17:33:56.135Z INFO piecestore upload started {“Piece ID”: “PUQVZNS2RT6ASH2UCCEY4XWVUZCICLG6UUJFUD24YMN475C7GRFA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 920245934080}
2020-06-13T17:34:02.186Z INFO piecestore upload canceled {“Piece ID”: “PUQVZNS2RT6ASH2UCCEY4XWVUZCICLG6UUJFUD24YMN475C7GRFA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”}
2020-06-13T17:34:12.750Z INFO piecestore upload started {“Piece ID”: “723KVIA5KAU6WM2HSAUKXQFPVJ77IDJ4CV2L3IGJH2L32367MZKA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 920243614208}
2020-06-13T17:34:14.590Z INFO piecestore upload canceled {“Piece ID”: “723KVIA5KAU6WM2HSAUKXQFPVJ77IDJ4CV2L3IGJH2L32367MZKA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”}
2020-06-13T17:34:29.412Z INFO piecestore upload started {“Piece ID”: “A2DFJMAB62X4QZJDZ7ZS4SAEZKDQ6E5DKZ3SIPHLSAOVTNPG7B4A”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Available Space”: 920242889472}
2020-06-13T17:34:29.527Z INFO piecestore upload canceled {“Piece ID”: “A2DFJMAB62X4QZJDZ7ZS4SAEZKDQ6E5DKZ3SIPHLSAOVTNPG7B4A”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”}