Node Suspension

[nlitend1]

SATA connection. Why would it say “download started” but “not downloaded”? Were you able to reset the status of the node? Thanks!

[nlitend1]

@Alexey Should I try this? Any harm in doing it? I am on 1.1.1.

No bandwidth usage since update to 1.1.1

Please, stop the storagenode make a backup of satellites.db Execute either with a native sqlite3 or with a docker container (take a look on https://support.storj.io/hc/en-us/articles/360029309111-How-to-fix-a-database-disk-image-is-malformed- for reference): sqlite3 satellites.db CREATE TABLE satellite_exit_progress ( satellite_id BLOB NOT NULL, initiated_at TIMESTAMP, …

[Adavan]

I got this email too. But suspension come from satelite “asia-east-1”. Time is arround in time, when I really have a problem with network traffic (I had a problem with botnet).
After restart docker container and wait some time it seems, that problem was solved for me, again I recieving new traffic (from my statistics it seems, that from this time I only transmitting).
I agreed, some reason, why is my node suspended should be very valuable in future. Thank you for implement it .

[Alexey]

No.
It’s better to check your databases though:

How to fix a "database disk image is malformed"

Perform only the integrity check.

[nlitend1]

@alexey all of my databases checkout okay except used_serial.db

Please see my post about that.

Used_serial database goes from 69MB to 0.3MB after rebuilding. What can i do?

I decided to double check all of my databases for corruption. All of them verified “ok” yesterday, but I am not getting any downloads so I checked again today. All checked out OK except used_serials gives the following: PS C:\Users\Administrator> docker run --rm -it --mount type=bind,source=v:\storagenode\storage\used_serial.db,destination=/used_serial.db sstc/sqlite3 sqlite3 /used_serial.db “PRAGMA integrity_check;” *** in database main *** Page 16352: btreeInitPage() returns error code 11 …

[nlitend1]

@Alexey Update: I replaced my used_serials.db with the one from 2 days ago (that was not corrupt) and I am now seeing “downloaded” in the logs and not just “download starting”.

So it looks like the database was causing an issue with egress. However, I am still not seeing any ingress. Can you please check the node on your end and “un-suspend” it?

[Alexey]

It unsuspend itself with the first successful audit.
Do you have any free space on your node to receive an ingress traffic?

[nlitend1]

Yes, and I appear to be getting ingress now from all nodes.

It appears the reason I was not getting traffic was the used_serial.db corruption. Does that sound like the case to you? Can you explain the purpose of the that database?

Thanks,

nlitend1

[Alexey]

One of the good explanations:

@BrightSilence

keeping track of used serials so uplinks can’t keep downloading your data, without you getting paid

[bdg0296]

So I didn’t get an email but restarted my storjnode and got your node has been suspended on and it list a bunch of characters I’m assuming those are the satellites. Is this part of the problem this thread is talking about? Also do I need to do anything to resolve this? If I look at the log I am seeing uploads and downloads and them completing.

[Alexey]

Now it’s real.
Please, search for GET_AUDIT and failed in your logs

Docker bash

docker logs storagenode 2>&1 | grep GET_AUDIT | grep failed

Docker Powershell

docker logs storagenode 2>&1 | sls GET_AUDIT | sls failed

Windows GUI Powershell

sls GET_AUDIT "C:\Program Files\Storj\Storage Node\storagenode.log" | sls failed

[bdg0296]

@Alexey

sls GET_AUDIT “C:\Program Files\Storj\Storage Node\storagenode.log” | sls failed

I ran it attached is the output.

Here are some of the lines from the output:

C:\Program Files\Storj\Storage Node\storagenode.log:279290:2020-04-28T15:14:37.584-0400 ERROR piecestore download failed {“Piece ID”: “MFV4NE4MVTOFUFF6TOAJSP24YXFYYE3EDLCP2ZZ6MGQS7QXBVANQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”,
“Action”: “GET_AUDIT”, “error”: “usedserialsdb error: disk I/O error: The device is not ready.”, “errorVerbose”: “usedserialsdb error: disk I/O error: The device is not ready.\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagen
ode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:523\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Download:471\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:995\n\tstorj.io/drpc/drp
cmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:66\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Trac
ker).track:51”}
C:\Program Files\Storj\Storage Node\storagenode.log:279358:2020-04-28T15:19:30.916-0400 ERROR piecestore download failed {“Piece ID”: “F2GAJRLDLDTPDEPU4MKS3EOIWTW3SCEARUGSSAJLTZBH3TJVBPRQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”,
“Action”: “GET_AUDIT”, “error”: “usedserialsdb error: disk I/O error: The device is not ready.”, “errorVerbose”: “usedserialsdb error: disk I/O error: The device is not ready.\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagen
ode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:523\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Download:471\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:995\n\tstorj.io/drpc/drp
cmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:66\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Trac
ker).track:51”}
C:\Program Files\Storj\Storage Node\storagenode.log:279440:2020-04-28T15:24:09.689-0400 ERROR piecestore download failed {“Piece ID”: “PLF5YAJDRZBJIEVHKYKEXNFLANQB6LCWF2C37BHAROE56H55TRZA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”,
“Action”: “GET_AUDIT”, “error”: “usedserialsdb error: disk I/O error: The device is not ready.”, “errorVerbose”: “usedserialsdb error: disk I/O error: The device is not ready.\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagen
ode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:523\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Download:471\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:995\n\tstorj.io/drpc/drp
cmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:66\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Trac
ker).track:51”}
C:\Program Files\Storj\Storage Node\storagenode.log:279594:2020-04-28T15:32:53.088-0400 ERROR piecestore download failed {“Piece ID”: “VHGCFP4PTS2H6ZTGA7PH6D3KRSHK5YOOTS4RXTFX2MY5G6RGW2BQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”:
“GET_AUDIT”, “error”: “usedserialsdb error: disk I/O error: The device is not ready.”, “errorVerbose”: “usedserialsdb error: disk I/O error: The device is not ready.\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/pieces
tore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:523\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Download:471\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:995\n\tstorj.io/drpc/drpcmux.(*Mux
).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:66\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track
:51”}
C:\Program Files\Storj\Storage Node\storagenode.log:279598:2020-04-28T15:32:59.538-0400 ERROR piecestore download failed {“Piece ID”: “UPFJF7AZMI7C6LZSYDQKQ5PGX3CQLXXXDP7UBU4OBRJTFSNY5PNA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”,
“Action”: “GET_AUDIT”, “error”: “usedserialsdb error: disk I/O error: The device is not ready.”, “errorVerbose”: “usedserialsdb error: disk I/O error: The device is not ready.\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagen
ode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:523\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Download:471\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:995\n\tstorj.io/drpc/drp
cmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:66\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Trac
ker).track:51”}
C:\Program Files\Storj\Storage Node\storagenode.log:279668:2020-04-28T15:37:44.640-0400 ERROR piecestore download failed {“Piece ID”: “DQHG5Z4HTB6WTCDV2R5XD3NPO72B3RN7YIMN7M463ZTIYHUBQL4Q”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”,
“Action”: “GET_AUDIT”, “error”: “usedserialsdb error: disk I/O error: The device is not ready.”, “errorVerbose”: “usedserialsdb error: disk I/O error: The device is not ready.\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagen
ode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:523\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Download:471\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:995\n\tstorj.io/drpc/drp
cmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:66\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Trac
ker).track:51”}
C:\Program Files\Storj\Storage Node\storagenode.log:279748:2020-04-28T15:43:20.126-0400 ERROR piecestore download failed {“Piece ID”: “YGF7HBUMHIXROJ3543QTJUBK2OEGIQHH4SWNPKH4Y5ZTZUTMOSYA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”,
“Action”: “GET_AUDIT”, “error”: “usedserialsdb error: database is locked”, “errorVerbose”: “usedserialsdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:
76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:523\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Download:471\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:995\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common
/rpc/rpctracing.(*Handler).HandleRPC:66\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}

[deathlessdd]

How is your drive attached?

[bdg0296]

It’s attached via iscsi I have been running with this setup for a long time now. Started with the previous versions of Storj and up to this one. Those errors started yesterday evening it looks like and stopped around 1-2pm my time today. I don’t see anymore, if I tail the logs it only shows successful uploads and downloads.

[deathlessdd]

What kinda drives are they? Do you find you get this error everytime you start your node up from a restart?

[bdg0296]

This is the first I have seen them. Western Digital Red drives running in a Synology 8 bay device. If everything is running correctly now will the node come out of suspended mode if it continues to run correctly? If so how long does it take to come out?

[deathlessdd]

Its suposed to get out of suspension once it starts getting a successful audit, Also Have you been keeping track of SMR drives that could be the main issue since your drives are probably getting full.

[bdg0296]

I have been reading up on that. All the satellites show except one say I’ve been suspended. But all of them seem to show 100% success on audit checks. I just remembered I did have to tweak the lio version on the synology nas right around the time the errors stopped. I did this because I noticed twice today it look like the iscsi mapped drive got hung up 10-20 seconds at a time and I had seen this behavior when I first got the device and found switching the lio version from 4.1 to 4.0 for iscsi fixes this. Maybe that was the problem that started this. But I see everything succeeding in the log nothing fails since between 1-2pm today it is now 7:00pm where I am

[deathlessdd]

Is that one satellite saltlake though because that is the one doing most of the data right now, I would keep an eye on that do you know if your drives are SMR drives or not?

@bdg0296

I just remembered I did have to tweak the lio version on the synology nas right around the time the errors stopped

Oh ok hopefully that fixed it.

[bdg0296]

I think it did and it wasn’t happening often either I only saw it twice and it would recover in like 10-20 seconds. So shouldn’t my node have passed enough audits by now to be unsuspended?