Node is Suspended on the saltlake Satellite

[future21]

Hi All.
One of my nodes has been suspended, but I had 100% audition checked and OK. Attached is the log of the last 20 items.

ubuntu@ubuntu:~$ sudo docker logs --tail 20 nodo0
2020-06-28T08:48:45.933Z        INFO    piecestore      upload started  {"Piece ID": "BD3X67HCQ62WJ5KGY53CWE2357GB52OJ76SGOAKCITT4OMRWX2NA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "Available Space": 443697447680}
2020-06-28T08:48:46.192Z        INFO    piecestore      upload canceled {"Piece ID": "AWGS36YGGEZJISVB45VVN6BRYYWZMUTGA4Y3TGKJXQXBQWUJZVLQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT"}
2020-06-28T08:48:46.702Z        INFO    piecestore      upload started  {"Piece ID": "D26YTP3QSLRNTJCQ57MZI7C26DUNJBZ2O77YXXSIU6F3C5JUBBIQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "Available Space": 443697437184}
2020-06-28T08:48:47.016Z        INFO    piecestore      upload canceled {"Piece ID": "5DAFCDQT3BTDI3F5PC2E4BSAXP6VVFI6ROLOF44WQOVSK33PNDQQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-06-28T08:48:47.324Z        INFO    piecestore      upload started  {"Piece ID": "2XV6XA3MMOPMHKXFSFZUBK2CJVC6UWWPV4UY34AUX7IGUNPHCHTQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 443697434880}
2020-06-28T08:48:49.377Z        INFO    piecestore      upload canceled {"Piece ID": "RMFH433KUYG7CUR6TLYH7DSB4AAH6GNNCD3EF3OWRWGY32U7YVPQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2020-06-28T08:48:52.951Z        INFO    piecestore      upload canceled {"Piece ID": "WJDGAFRSKI5OS624Y32VML2PXTN47UZ43YIBWAGG5KE5YZCKTAAA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-06-28T08:48:54.634Z        INFO    piecestore      upload started  {"Piece ID": "L5B7DKWWOFT5OPVNTW62W5CEAPR35WVA37HPPFZECTTD5PRCINDQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 443694069248}
2020-06-28T08:48:56.979Z        INFO    piecestore      upload canceled {"Piece ID": "BD3X67HCQ62WJ5KGY53CWE2357GB52OJ76SGOAKCITT4OMRWX2NA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT"}
2020-06-28T08:48:59.923Z        INFO    piecestore      upload started  {"Piece ID": "TMMK4RPCF2BFTAIO5BOKPDWJC2P35G2L2MKIC2WV25RMXJPGTS2Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 443694067712}
2020-06-28T08:49:01.655Z        INFO    piecestore      uploaded        {"Piece ID": "CX37S6BKKZFYNUJVDF633UKII5DO5FWXPMD52PQNMGOZ7HNR4A3A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT"}
2020-06-28T08:49:03.116Z        ERROR   piecestore      failed to add bandwidth usage   {"error": "bandwidthdb error: database is locked", "errorVerbose": "bandwidthdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:59\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).saveOrder:664\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:401\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:996\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-06-28T08:49:03.117Z        INFO    piecestore      upload canceled {"Piece ID": "D26YTP3QSLRNTJCQ57MZI7C26DUNJBZ2O77YXXSIU6F3C5JUBBIQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT"}
2020-06-28T08:49:06.334Z        INFO    piecestore      upload canceled {"Piece ID": "6SF2DUKK4MGXM22LHZICQGCRYUOXGET3XVR4ZTK5OD67TACBNWCA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-06-28T08:49:06.964Z        ERROR   piecestore      failed to add bandwidth usage   {"error": "bandwidthdb error: database is locked", "errorVerbose": "bandwidthdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:59\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).saveOrder:664\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:401\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:996\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-06-28T08:49:06.964Z        INFO    piecestore      uploaded        {"Piece ID": "JPOLIBET3NI7GJBRKPARNVVWX5J4TIN777BWRGHOV353DIEMGZ3Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2020-06-28T08:49:10.141Z        INFO    piecestore      upload started  {"Piece ID": "W7ADZY5XHFSEC2G6FLXNRCQ7JG5VAPH2HHCMD7KILOEAKCA4ETKQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "Available Space": 443694065920}
2020-06-28T08:49:11.517Z        INFO    piecestore      upload canceled {"Piece ID": "5HJPPIFBTEDIE27QWVL42SC2CQ57IVJ54HT7II2CETM7A4SOJJCA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-06-28T08:49:13.534Z        INFO    piecestore      upload started  {"Piece ID": "OU2WEEPIGY2LHSMVHDERN3SRQDQRMDSCUPUFIETJUPGXMTZ2QKMA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "Available Space": 443694063616}
2020-06-28T08:49:16.731Z        INFO    piecestore      upload canceled {"Piece ID": "L5B7DKWWOFT5OPVNTW62W5CEAPR35WVA37HPPFZECTTD5PRCINDQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT"}

PS: My system is on a Raspberry 4 with ubuntu x64 and a 1TB disk.

[Vadim]

@future21

failed to add bandwidth usage {"error": "bandwidthdb error: database is locked",

This can be the reason.

[nerdatwork]

Welcome to the forum @future21!

Search your log for GET_AUDIT and failed together and post them.

[BrightSilence]

Please check the bandwidth.db for problems and any others you find errors for.

How to fix a "database disk image is malformed"

Suspension is based on a different score than the dashboard shows, so its very well possible you are suspended with a 100% audit score. Please do look at the log lines @nerdatwork pointed to, those will show you the exact reason of the suspension.

[future21]

Hello All.-

Thanks for responses.

I have passed a check of all databases, and all checks show that the databases are OK. Assistant screenshot.

[future21]

Hi, I repass my logs, and i view:

[BrightSilence]

I think your system is IO constrained. These locks tend to appear when the HDD can’t keep up. The upside is that the used serials db will be moved to memory in the next update. That should fix the locks for that db and perhaps also reduce some of the strain on the HDD. In the mean time you can try to vacuum the db files (stop the node first). That helps some users, but it’s not a guaranteed fix and the issue might return.

[nerdatwork]

@future21

and a 1TB disk

Can you tell the model of this disk ?

[future21]

Yes, the model is: Seagate BarraCuda 2.5" 1TB SATA3 (ST1000LM048) https://www.pccomponentes.com/seagate-barracuda-25-1tb-sata3 with adapter https://www.amazon.es/gp/product/B017NIDXF0/ref=ppx_yo_dt_b_asin_title_o05_s00?ie=UTF8&psc=1

[nerdatwork]

@future21

ST1000LM048

This is SMR drive.

This post can shed more light on SMR drives

PSA: Beware of HDD manufacturers submarining SMR technology in HDD's without any public mention

Recently there have been several reports of SNOs running into massive CPU wait issues. It seems the common thread has been the use of HDD’s in which manufacturers have quietly started using SMR technology. [2003549104] Because of these overlapping tracks, when data is written to the disk, the drive has to rewrite several additional tracks. This obviously leads to much slower writes. These HDD’s usually deal with that using large caches and rewriting the shingles tracks during quiet times. How…

[future21]

Oh, What a bad luck. And is there any way to fix this?

[future21]

Maybe wait until the next update and meanwhile leave it as it is?

[Vadim]

if problem will be long you will get DQ soon.
what version do you have? and docker or win GUI?

[BrightSilence]

Disqualification after suspension is for now still not active. At this point it’s best to just see how the new version does. Should be just a couple more days until you get it.

[Mark]

What file system are you using on the storj drive? EXT4? I’ve read that some SMR drives and file systems support the Trim command. I have not found a good source of info about this though.

[future21]

I have NTFS in drive.

[nerdatwork]

@future21

NTFS

Windows node should use NTFS while Linux should use ext4.

[future21]

Yes, i use ntfs with Linux, for compatibility, but, i think that ext4 would have been better.

[future21]

I have ver. 1.5.2 and docker.