Your node has been disqualified 2

Today node has upgraded from 1.53.1 to 1.54.2 via watchtower. Noticed that no any information is shown on dashboard… Restarted node, bandwidth & used space appeared.
Now I’m back and :
Your node has been disqualified on 12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB .
According to time right a few minutes after restart. Checking logs gives me:

2022-05-09T19:08:30.190Z        ERROR   piecestore      failed to add bandwidth usage   {"Process": "storagenode", "error": "bandwidthdb: database is locked", "errorVerbose": "bandwidthdb: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:723\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:435\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}

Got several failed audits
Pragma security check says dbs: OK

Output of http://127.0.0.1:14002/api/sno/satellite/12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB
{“id”:“12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”,“storageDaily”:[{“atRestTotal”:30226600613975.492,“intervalStart”:“2022-05-01T00:00:00Z”},{“atRestTotal”:30927924096658.773,“intervalStart”:“2022-05-02T00:00:00Z”},{“atRestTotal”:42361585575034.22,“intervalStart”:“2022-05-03T00:00:00Z”},{“atRestTotal”:14135211935947.117,“intervalStart”:“2022-05-04T00:00:00Z”},{“atRestTotal”:56224893734145.72,“intervalStart”:“2022-05-05T00:00:00Z”},{“atRestTotal”:26974814440214.516,“intervalStart”:“2022-05-06T00:00:00Z”},{“atRestTotal”:42942578298851.58,“intervalStart”:“2022-05-07T00:00:00Z”},{“atRestTotal”:26152184373837.01,“intervalStart”:“2022-05-08T00:00:00Z”},{“atRestTotal”:13049595705512.264,“intervalStart”:“2022-05-09T00:00:00Z”}],“bandwidthDaily”:[{“egress”:{“repair”:5500554496,“audit”:32512,“usage”:4068352},“ingress”:{“repair”:4261795328,“usage”:17912832},“delete”:0,“intervalStart”:“2022-05-01T00:00:00Z”},{“egress”:{“repair”:3532323840,“audit”:28672,“usage”:12340736},“ingress”:{“repair”:5882345472,“usage”:29684736},“delete”:0,“intervalStart”:“2022-05-02T00:00:00Z”},{“egress”:{“repair”:3924327168,“audit”:16896,“usage”:12659456},“ingress”:{“repair”:5227652352,“usage”:39492096},“delete”:0,“intervalStart”:“2022-05-03T00:00:00Z”},{“egress”:{“repair”:4182523392,“audit”:17408,“usage”:15247360},“ingress”:{“repair”:4885055488,“usage”:28082688},“delete”:0,“intervalStart”:“2022-05-04T00:00:00Z”},{“egress”:{“repair”:4486583808,“audit”:19712,“usage”:22700288},“ingress”:{“repair”:5349131520,“usage”:41946112},“delete”:0,“intervalStart”:“2022-05-05T00:00:00Z”},{“egress”:{“repair”:4205670144,“audit”:27392,“usage”:9154048},“ingress”:{“repair”:4801351168,“usage”:32929280},“delete”:0,“intervalStart”:“2022-05-06T00:00:00Z”},{“egress”:{“repair”:4161398784,“audit”:20224,“usage”:5187840},“ingress”:{“repair”:4515279104,“usage”:37838592},“delete”:0,“intervalStart”:“2022-05-07T00:00:00Z”},{“egress”:{“repair”:4369883136,“audit”:16128,“usage”:532224},“ingress”:{“repair”:4439604224,“usage”:22079744},“delete”:0,“intervalStart”:“2022-05-08T00:00:00Z”},{“egress”:{“repair”:1729328128,“audit”:4352,“usage”:354304},“ingress”:{“repair”:1738543616,“usage”:3262720},“delete”:0,“intervalStart”:“2022-05-09T00:00:00Z”}],“storageSummary”:282995388774176.6,“bandwidthSummary”:77529007872,“egressSummary”:36175020800,“ingressSummary”:41353987072,“currentStorageUsed”:1390092344832,“audits”:{“auditScore”:0.5893936089106387,“suspensionScore”:1,“onlineScore”:0.9948793363759714,“satelliteName”:“europe-north-1.tardigrade.io:7777”},“auditHistory”:{“score”:0.9948793363759714,“windows”:[{“windowStart”:“2022-04-09T12:00:00Z”,“totalCount”:2209,“onlineCount”:1825},{“windowStart”:“2022-04-10T00:00:00Z”,“totalCount”:2418,“onlineCount”:2418},{“windowStart”:“2022-04-10T12:00:00Z”,“totalCount”:2603,“onlineCount”:2600},{“windowStart”:“2022-04-11T00:00:00Z”,“totalCount”:2618,“onlineCount”:2618},{“windowStart”:“2022-04-11T12:00:00Z”,“totalCount”:3254,“onlineCount”:3253},{“windowStart”:“2022-04-12T00:00:00Z”,“totalCount”:3284,“onlineCount”:3284},{“windowStart”:“2022-04-12T12:00:00Z”,“totalCount”:3183,“onlineCount”:3181},{“windowStart”:“2022-04-13T00:00:00Z”,“totalCount”:3194,“onlineCount”:3194},{“windowStart”:“2022-04-13T12:00:00Z”,“totalCount”:3270,“onlineCount”:3265},{“windowStart”:“2022-04-14T00:00:00Z”,“totalCount”:3366,“onlineCount”:3366},{“windowStart”:“2022-04-14T12:00:00Z”,“totalCount”:3537,“onlineCount”:3537},{“windowStart”:“2022-04-15T00:00:00Z”,“totalCount”:3429,“onlineCount”:3429},{“windowStart”:“2022-04-15T12:00:00Z”,“totalCount”:3451,“onlineCount”:3449},{“windowStart”:“2022-04-16T00:00:00Z”,“totalCount”:3705,“onlineCount”:3705},{“windowStart”:“2022-04-16T12:00:00Z”,“totalCount”:3574,“onlineCount”:3562},{“windowStart”:“2022-04-17T00:00:00Z”,“totalCount”:3537,“onlineCount”:3537},{“windowStart”:“2022-04-17T12:00:00Z”,“totalCount”:3442,“onlineCount”:3207},{“windowStart”:“2022-04-18T00:00:00Z”,“totalCount”:3470,“onlineCount”:3470},{“windowStart”:“2022-04-18T12:00:00Z”,“totalCount”:3279,“onlineCount”:3273},{“windowStart”:“2022-04-19T00:00:00Z”,“totalCount”:4486,“onlineCount”:4486},{“windowStart”:“2022-04-19T12:00:00Z”,“totalCount”:4137,“onlineCount”:4136},{“windowStart”:“2022-04-20T00:00:00Z”,“totalCount”:4291,“onlineCount”:4291},{“windowStart”:“2022-04-20T12:00:00Z”,“totalCount”:3810,“onlineCount”:3810},{“windowStart”:“2022-04-21T00:00:00Z”,“totalCount”:3395,“onlineCount”:3384},{“windowStart”:“2022-04-21T12:00:00Z”,“totalCount”:3357,“onlineCount”:3357},{“windowStart”:“2022-04-22T00:00:00Z”,“totalCount”:3352,“onlineCount”:3352},{“windowStart”:“2022-04-22T12:00:00Z”,“totalCount”:4137,“onlineCount”:4135},{“windowStart”:“2022-04-23T00:00:00Z”,“totalCount”:4311,“onlineCount”:4311},{“windowStart”:“2022-04-23T12:00:00Z”,“totalCount”:4351,“onlineCount”:4347},{“windowStart”:“2022-04-24T00:00:00Z”,“totalCount”:4540,“onlineCount”:4540},{“windowStart”:“2022-04-24T12:00:00Z”,“totalCount”:4578,“onlineCount”:4574},{“windowStart”:“2022-04-25T00:00:00Z”,“totalCount”:4968,“onlineCount”:4968},{“windowStart”:“2022-04-25T12:00:00Z”,“totalCount”:5394,“onlineCount”:5393},{“windowStart”:“2022-04-26T00:00:00Z”,“totalCount”:4944,“onlineCount”:4944},{“windowStart”:“2022-04-26T12:00:00Z”,“totalCount”:4376,“onlineCount”:4372},{“windowStart”:“2022-04-27T00:00:00Z”,“totalCount”:4531,“onlineCount”:4531},{“windowStart”:“2022-04-27T12:00:00Z”,“totalCount”:4468,“onlineCount”:4466},{“windowStart”:“2022-04-28T00:00:00Z”,“totalCount”:4286,“onlineCount”:4286},{“windowStart”:“2022-04-28T12:00:00Z”,“totalCount”:4053,“onlineCount”:4052},{“windowStart”:“2022-04-29T00:00:00Z”,“totalCount”:4596,“onlineCount”:4596},{“windowStart”:“2022-04-29T12:00:00Z”,“totalCount”:4555,“onlineCount”:4555},{“windowStart”:“2022-04-30T00:00:00Z”,“totalCount”:4736,“onlineCount”:4736},{“windowStart”:“2022-04-30T12:00:00Z”,“totalCount”:5088,“onlineCount”:5072},{“windowStart”:“2022-05-01T00:00:00Z”,“totalCount”:4199,“onlineCount”:4120},{“windowStart”:“2022-05-01T12:00:00Z”,“totalCount”:7233,“onlineCount”:7231},{“windowStart”:“2022-05-02T00:00:00Z”,“totalCount”:3025,“onlineCount”:3025},{“windowStart”:“2022-05-02T12:00:00Z”,“totalCount”:2949,“onlineCount”:2945},{“windowStart”:“2022-05-03T00:00:00Z”,“totalCount”:3162,“onlineCount”:3162},{“windowStart”:“2022-05-03T12:00:00Z”,“totalCount”:3495,“onlineCount”:3491},{“windowStart”:“2022-05-04T00:00:00Z”,“totalCount”:3857,“onlineCount”:3857},{“windowStart”:“2022-05-04T12:00:00Z”,“totalCount”:3755,“onlineCount”:3754},{“windowStart”:“2022-05-05T00:00:00Z”,“totalCount”:3494,“onlineCount”:3494},{“windowStart”:“2022-05-05T12:00:00Z”,“totalCount”:4115,“onlineCount”:4113},{“windowStart”:“2022-05-06T00:00:00Z”,“totalCount”:4226,“onlineCount”:4226},{“windowStart”:“2022-05-06T12:00:00Z”,“totalCount”:3794,“onlineCount”:3794},{“windowStart”:“2022-05-07T00:00:00Z”,“totalCount”:3337,“onlineCount”:3337},{“windowStart”:“2022-05-07T12:00:00Z”,“totalCount”:3672,“onlineCount”:3644},{“windowStart”:“2022-05-08T00:00:00Z”,“totalCount”:4215,“onlineCount”:4215},{“windowStart”:“2022-05-08T12:00:00Z”,“totalCount”:4611,“onlineCount”:4609},{“windowStart”:“2022-05-09T00:00:00Z”,“totalCount”:3277,“onlineCount”:3229},{“windowStart”:“2022-05-09T12:00:00Z”,“totalCount”:26,“onlineCount”:0}]},“priceModel”:{“EgressBandwidth”:2000,“RepairBandwidth”:1000,“AuditBandwidth”:1000,“DiskSpace”:150},“nodeJoinedAt”:“2020-07-08T09:46:28.206708Z”}

This is the reason.
Not upgrades or being offline (unless your node were offline for more than 30 days).
If audits are failed - your node lost or corrupted data (or lost an access to data).
To fail audits your node should answer on audit requests (so it should be online) but unable to provide a requested piece (3x times for the same piece in case of 5 minutes timeout), or provides a broken one.
If too many failed audits - (the audit score below 60%), the node will be disqualified.

I see, several days ago I faced network switch bug, it was running, but dropping packets. That time audits have fallen, but recovered until now… Node was actually running, but as I suppose could not send or receive anything…
Yet problem after update persists: again infinite loading of dashboard, stopped node, checked dbs:


/mnt/st6000/storage/piece_expiration.dbok
/mnt/st6000/storage/piece_spaced_used.dbok
/mnt/st6000/storage/notifications.dbok
/mnt/st6000/storage/orders.dbok
/mnt/st6000/storage/bandwidth.dbok
/mnt/st6000/storage/info.dbok
/mnt/st6000/storage/secret.dbok
/mnt/st6000/storage/heldamount.dbok
/mnt/st6000/storage/satellites.dbok
/mnt/st6000/storage/reputation.dbok
/mnt/st6000/storage/pieceinfo.dbok
/mnt/st6000/storage/pricing.dbok
/mnt/st6000/storage/used_serial.dbok
/mnt/st6000/storage/storage_usage.dbok

Restarted node and got

2022-05-10 08:43:30,499 INFO spawned: 'processes' with pid 51
2022-05-10 08:43:30,500 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-05-10 08:43:30,500 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-05-10 08:43:30,516 INFO exited: processes (exit status 0; not expected)
2022-05-10 08:43:32,522 INFO spawned: 'processes' with pid 52
2022-05-10 08:43:32,540 INFO exited: processes (exit status 0; not expected)
2022-05-10 08:43:35,547 INFO spawned: 'processes' with pid 53
2022-05-10 08:43:35,564 INFO exited: processes (exit status 0; not expected)
2022-05-10 08:43:35,565 INFO gave up: processes entered FATAL state, too many start retries too quickly

Removed node, pulled latest, started node and got:

downloading storagenode-updater
--2022-05-10 08:47:45--  https://version.storj.io/processes/storagenode-updater/minimum/url?os=linux&arch=arm
Resolving version.storj.io (version.storj.io)... 35.224.88.204
Connecting to version.storj.io (version.storj.io)|35.224.88.204|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 90 [text/plain]
Saving to: 'STDOUT'

     0K                                                       100% 15.3M=0s
  7400K .......... .......... .......... .......... ........  100%  240K=3m36s

2022-05-10 08:51:23 (34.4 KB/s) - '/tmp/storagenode-updater.zip' saved [7627685/7627685]

and that’s all… No more logs, cannot neither create new container, nor stop running one.

Continuing the discussion from Node could not start since yesterday:

iptables -t mangle -A POSTROUTING -p tcp --tcp-flags SYN,RST SYN -o $IFACE -j TCPMSS --clamp-mss-to-pmtu

This one helped after rebooting. Suddenly got “nice” suspension for 30 mins of offline :neutral_face:

The disqualification is happened when audit score drop below 60%.
The audit score is affected when your node

  1. Answering on audit or GET_REPAIR requests (i.e. it is online)
  2. Cannot provide a requested piece within 5 minutes timeout (3 times for the same piece), or throws an error “file not found” or provided a corrupted piece.
  3. Any other errors change the suspension score instead.

If node was offline, the online score will be affected.

So, in short - being offline does not affect audit score. The disqualification for downtime can be applied if your node were offline for more than 30 days.