Suspension emails

Hey, i received a Mail that my node has been suspended but on my Dashboard it seems to be online with a Audit Score of 100% and 99,6% uptime. So I guess i can ignore that mail?

you selected the right satellite right?

Yes and now i received the Mail for 3 satellites and everone does have an sufficient uptime and 100% auditscore

i duno how old your node is, but i’ve had like over 10 hours of dt recently (yes i know its terrible and hopefully behind me) and my uptime is at the worst 99.8%

most are still at 99.9% or even 100% kinda makes the number quite useless
so if your node is very new then you could have low uptime numbers, but for older nodes 99.6% is a red flag.

well the dashboard cannot be trusted… but then again recently nor could the mails…
so sadly i’m not sure anyone really knows… you shouldn’t ignore it thats for sure… would be a pity to get DQ

Check your log for download failed AND GET_AUDIT entries to see if you are failing audits. Better be safe than sorry!

I watched through my logs and I’m receiving lots of database locked errors. I restarted the node maybe it is working now

Edit: Nope still receiving lots of database locked errors

2020-05-10T12:46:46.285Z INFO piecestore download started {“Piece ID”: “PRU7JFAORRQBGZKSOSNBM6QHQTB7NW6FY6JE3WXFGKPJMKCQWDMA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”}
2020-05-10T12:46:46.695Z INFO piecestore download started {“Piece ID”: “UYUMPGX3VNIKWSODYBOGRGBILKUAV4JOQIDLKWDM64KZO7ZGRXRA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”}
2020-05-10T12:46:46.912Z INFO piecestore download started {“Piece ID”: “NPH3R66KSR2Y3ZEWNCQ64IBNICMKMNKQ7ITJER7PQOEYABUDTRQQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”}
2020-05-10T12:46:47.077Z ERROR piecestore download failed {“Piece ID”: “UBISSJQJHQM3PQDDHEJGKHVO6UOG4NQL27H3U2RMOUVYWU3YJTPA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET”, “error”: “usedserialsdb error: database is locked”, “errorVerbose”: “usedserialsdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:523\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Download:471\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:995\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:66\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-05-10T12:46:59.766Z ERROR piecestore download failed {“Piece ID”: “CXVPO5QDFZ6J5PCDVZKYW576N24QMG54UHYFSZFXORIKD3EKN2FA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”, “error”: “usedserialsdb error: database is locked”, “errorVerbose”: “usedserialsdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:523\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Download:471\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:995\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:66\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-05-10T12:46:59.843Z INFO piecestore download started {“Piece ID”: “5MMRI45N3HJE67YCV43ZSYIUTQJ3BT5IUVJJ4MUTXJ3PKC2AK6TA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”}
2020-05-10T12:47:01.766Z ERROR piecestore download failed {“Piece ID”: “E7AM3AG73Z3AWTER2FVLDDBKC4DGKJQZBUPETU5NLXAEPUB3URGQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “GET”, “error”: “usedserialsdb error: database is locked”, “errorVerbose”: “usedserialsdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:523\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Download:471\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:995\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:66\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-05-10T12:47:27.421Z INFO piecestore download started {“Piece ID”: “5XISMKTJTLRN2THVNE4T4KAW54POVIBPEBDXQ4AIJOFR3HV3EF2A”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”}
2020-05-10T12:47:40.391Z INFO piecestore download started {“Piece ID”: “ACYRUQEHMHZ5XLKC6KBCL5GOXRAATVTLAPLVMYB5DYHEZDLMV73A”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”}
2020-05-10T12:47:40.621Z ERROR piecestore download failed {“Piece ID”: “3I23V56RCURO7EZRBXFRCV7XDYP7WP4ZWLILVUGRMAQKBDODKSNQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”, “error”: “usedserialsdb error: database is locked”, “errorVerbose”: “usedserialsdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:523\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Download:471\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:995\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:66\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-05-10T12:47:41.412Z INFO piecestore download started {“Piece ID”: “RRTF657MCSZ7CVZSAQOD4DBY3MYJ2QR6DI73WRDVC4Y2XSSXHQYA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET”}
2020-05-10T12:47:41.882Z ERROR piecestore download failed {“Piece ID”: “RJHLSPSDTAX253EDYV2GVSUNSAMAAF6KK3KX6S3RPJNXI3ZP2EMA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “error”: “usedserialsdb error: database is locked”, “errorVerbose”: “usedserialsdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:523\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Download:471\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:995\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:66\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-05-10T12:47:42.158Z INFO piecestore download started {“Piece ID”: “IXDJZCUTPOBVGKQIQCLJXAFU75AUAT5IEVFB4TEBO7JU3D5VKNMQ”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET”}
2020-05-10T12:47:55.098Z INFO piecestore download started {“Piece ID”: “CI2D23OV64DCQBG45F4VMX3BVKYCR7UWYPWZOFB33IGFQHWCVLVQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”}
2020-05-10T12:48:07.994Z ERROR piecestore download failed {“Piece ID”: “DHVKXOBX5PT3N6PKLWICQCPMPBR4RP73SNLQ63YEZVD3EMQAIHFQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “error”: “usedserialsdb error: database is locked”, “errorVerbose”: “usedserialsdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:523\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Download:471\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:995\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:66\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-05-10T12:48:21.227Z INFO piecestore download started {“Piece ID”: “ZAE5JWOHHDEQWGG6OQDF36DVGLUK3HVCPG6SJJS3YP5XENHCRL5Q”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”}
2020-05-10T12:48:34.715Z INFO piecestore download started {“Piece ID”: “NIM6PPXSWNIV534YGXIRNUSMMEROWBMYGBAA5DDAYSSSJFTZFDOA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET”}
2020-05-10T12:48:48.213Z ERROR piecestore download failed {“Piece ID”: “OCZVMJ4BYYQYSXVZ2KHJPRMK7IWXUXKDDBDKMICFDZ7V6I7MEAWQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “GET”, “error”: “usedserialsdb error: database is locked”, “errorVerbose”: “usedserialsdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:523\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Download:471\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:995\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:66\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-05-10T12:49:01.603Z INFO piecestore download started {“Piece ID”: “NCATEHS4B4WTYGUP463PC7CUXELEJPIFMI7NT552RZ6BRO5JTTLQ”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET”}
2020-05-10T12:49:01.645Z ERROR piecestore download failed {“Piece ID”: “C5PQ7BYRGJJ6TBNHVGCMFDNDGFREWI6WWJHUUYD3AZUTA5E3DH6A”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET”, “error”: “usedserialsdb error: database is locked”, “errorVerbose”: “usedserialsdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:523\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Download:471\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:995\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:66\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2020-05-10T12:49:02.751Z INFO piecestore download started {“Piece ID”: “2GBWFWRGD2FP2AYOUOT5JS5HJZYIYBL7Y4TV2E6OCIPOZSJBFM7A”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “GET”}
2020-05-10T12:49:15.963Z ERROR piecestore download failed {“Piece ID”: “5NY67F4SPCYXSYGX4V27Y4FLZWB2EOJICWE6FZOMHFL2KJETFFFA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “error”: “usedserialsdb error: database is locked”, “errorVerbose”: “usedserialsdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:523\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Download:471\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:995\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:66\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}

No, they are working perfectly. Your node failing audits.
But the error

Can’t be solved by SNO currently except do not using the network or USB 2.0 connected drives. This error is related to the slow drive response.
I can suggest only workaround at the moment:

I’m trying this right now, do u have any idea how long this will take? It’s running for 10minutes now. I have arround 7.5TB of data

I guess this is a unraid problem?

Not necessarily, but perhaps their filesystem is little bit slower than others.

I guess i have to wait then, even though it seems stuck at orders.db at the Vacuum part. I haven’t had any problems for 6month

Do you know if it is fine to kill that task and start over again?

Only if you have a reserve copy of that database.

Make sure that your storagenode is stopped.

just took an hour, but it seems to work now, thank you. And it reduced the database file by 200mb :smiley:

I guess i have to upgrade my cpu soon, I think the parity check of unraid killed it

If you can replace the platform, you perhaps will not be need to update the CPU :slight_smile:

slow drives…O.o i get the damn things and im on 5 drives in raidz1 with 512gb ssd arc and ssd SLOG + seperate ssd for the OS, 48gb ram… are you saying i need more drives to keep up with a single node?
the drives are ofc not SMR and 7200RPM…
i guess i should do that upgrade to striped SSD SLOG and add the 4 more drives in raidz1 for a 9 drive nested raidz1 i believe it’s called.

that should basically double my performance on the HDD’s atleast over time. will take a while to rebalance the data

This is assumption. I have no confirmation from the developers. But I read the forum and in most cases it’s related to slow access to drives under the high load.
Perhaps the feature to allow the databases be placed on other disk could help a little bit.
But the root problem should be fixed in code in my opinion.

well it might be… a storagenode on fulltilt puts and amazing stress on the system… i’ve been trying to make it run smooth for quite some time, and i sure cannot keep up… and not for lack of trying and i would like to think my hardware is half decent… i can only speculate about how much trouble those with less gear have in keeping up.

ofc this is essentially large scale random read write io, which has always been some of the slowest to deal with in basically all tech

Suspension is based on an audit suspension score that isn’t currently reported on the dashboard. You can be suspended with a 100 audit score. I already created a feature request to get this info disclosed, you can upvote it here. Report audit suspension scores back to node databases, API, and dashboard

1 Like

so i’ve been pondering… how does this voting system work and how do the votes replenish…??
i just checked the documentation.storj and support.storj for a description of it…without finding any clear mention / definition of it…
also is there anywhere else i should check when looking for information like this?

2 Likes

that was fast suspension is over :smiley:

2 Likes