Now it hit me: 'Your node has been suspended'

SGC · February 20, 2022, 7:10pm

if the disk can’t keep up stuff will go bad, if not immediately then over time…
but brief periods of problems like bandwidth.db locked won’t destroy a node… i’ve had it happen many times. (but with a lot of effort and trial and error, i have now gotten to a point where i basically never see it)

there really isn’t much to do, except reduce io to the storage media when possible.
here are a few options, i’m sure there are tons more… but i cannot remember them of the top of my head.

don’t install the OS on the same disk as the storagenode
disable xattr because this will cause extra write io
move the storagenode database to an ssd, or other hdd, the database load is rather extensive io and will greatly limit the storagemedia
make sure it’s not an SMR hdd
add more nodes to distribute the load between them.

if you run zfs the disabling atime will also reduce io by 1 per file write i think it is…

alternatively one can also add caches to help reduce the random io.

BrightSilence · February 20, 2022, 8:33pm

Good list, I would add “use noatime mount option”.

SGC · February 20, 2022, 9:03pm

i’m running zfs for everything except my OS partition, so haven’t really had to the need to tinker with the linux partitioning stuff…

i did end up running my OS on xfs because alexey said it was faster, but later research into the topic made me doubt if that was the right choice, but thus far its been fine… tho was i remaking it today i would go with ext4 just because its like half a decade or more recent, than xfs…

after i installed linux i started using zfs, so my knowledge with linux parition and mounting configurations is basically no existing.
but certainly sounds like a good choice, and i added it…

maybe we should write a more specialized storagenode configuration recommendation…

not sure how well it works, but i started using

docker run -e RUN_PARAMS="--filestore.write-buffer-size 4096kiB --pieces.write-prealloc-size 4096kiB"

in my docker run command, i think it should also work for the binary, the run_params supposedly passes it through to the binary inside docker.
that is to reduce disk fragmentation because it shouldn’t write out files fragmented or allocates the space for the file in advance… not sure if it even works… but it seems to atleast from what little verification i’ve done…

not sure how to really verify if it works or not, but i’m sure someone will test that eventually… i haven’t had any problems while running it tho… and been using it for like 3 months or so now.

got it from somebody which was doing a deep dive because they was experiencing massive fragmentation of their data… and it made sense to me to start using it because, i know how much fragmentation can mess up a drive.

anyways seemed relevant, maybe you have a thought on that or maybe its peaked your interest. @BrightSilence

Bivvo · February 20, 2022, 11:28pm

On Debian ext4, too? If so, how?

anon27637763 · February 20, 2022, 11:59pm

Disclaimer and Warning: Messing around with filesystem features can cause lots of problems

I think this will do it:

tune2fs -l /dev/partition |grep -E "Filesystem\ features|Default\ mount"

Check that the following options are set:

Filesystem features:      ext_attr 
Default mount options:    user_xattr

Then remove the ext_attr filesystem feature and check to ensure it worked:

docker stop -t 300 storagenode
umount /dev/partition
tune2fs -O ^ext_attr /dev/partition
mount -a
tune2fs -l /dev/partition |grep -E "Filesystem\ features|Default\ mount"

Alexey · February 21, 2022, 1:29am

It could, if this error is happening during GET_AUDIT or GET_REPAIR.

Bivvo · February 21, 2022, 2:56am

So you’re not doing it yourself?

jammerdan · February 21, 2022, 7:10am

Todays update.
Node1 no further suspension on this one:

docker logs storagenode1 2>&1 | grep -E 'GET_AUDIT' | grep 'failed' | wc -l
0
docker logs storagenode1 2>&1 | grep -E 'GET_REPAIR' | grep 'failed' | wc -l
29

Errors were 23x broken pipe and 6x use of closed network connection

Node 2 suspension got worse again:

docker logs storagenode2 2>&1 | grep -E 'GET_AUDIT' | grep 'failed' | wc -l
0
docker logs storagenode2 2>&1 | grep -E 'GET_REPAIR' | grep 'failed' | wc -l
0

Both 0. How can suspension become worse?

Node3 everything got worse again:

docker logs storagenode3 2>&1 | grep -E 'GET_AUDIT' | grep 'failed' | wc -l
0
docker logs storagenode3 2>&1 | grep -E 'GET_REPAIR' | grep 'failed' | wc -l
2

Errors were 1x "use of closed network connection and 1x broken pipe

I don’t understand why node 2 and 3 are in suspension. They are not failing audits or repairs at all resp. in a significant way.

Alexey · February 21, 2022, 7:18am

Could you please search for GET_REPAIR and ERROR (not failed)?

jammerdan · February 21, 2022, 7:21am

Oh yes:

Node1:

docker logs storagenode1 2>&1 | grep -E 'GET_REPAIR' | grep 'ERROR' | wc -l
29

Node2:

docker logs storagenode2 2>&1 | grep -E 'GET_REPAIR' | grep 'ERROR' | wc -l
0

Node3:

docker logs storagenode3 2>&1 | grep -E 'GET_REPAIR' | grep 'ERROR' | wc -l
2

Alexey · February 21, 2022, 7:51am

And what is it? Could you please show a typical error for GET_REPAIR on affected nodes?

jammerdan · February 21, 2022, 7:58am

You can see that node2 has zero errors and the suspension got worse. So what should the typical error be? There is none.
Node1 has broken pipe error like below, but it did not affect suspension:

2022-02-21T06:03:48.160Z        ERROR   piecestore      download failed     {"Piece ID": "5HRTIZOBHXX6XQW2523IQ3P45POX4TWFCD2ZHLEPS3U5Z4GWQUCA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.2:28967->78.47.121.40:44786: write: broken pipe", "errorVerbose": "write tcp 172.17.0.2:28967->78.47.121.40:44786: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:322\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:400\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:619\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}

Alexey · February 21, 2022, 8:10am

No, It should not.
I’ll ask the team, why there is no errors, but suspension score become worse.
Could you please submit an issue on GitHub and upload zipped logs from node2 to there? I think 5 last days should be enough.

jammerdan · February 21, 2022, 8:13am

I don’t have a Github account.
The logs got reset when I had rebooted yesterday I think. But it showed that since reboot the score got worse without new errors at least on node 2.
I mean I hope they don’t get disqualified but I’ll keep them running and if they don’t get disqualified I’ll have some more days of logs in a couple of days.

Alexey · February 21, 2022, 8:15am

Then please, share logs with me, I would create an issue on behalf you.
You can use https://transfer.sh

Alexey · February 22, 2022, 4:00am

2 posts were split to a new topic: Suspended because of low online score

jammerdan · February 21, 2022, 8:35am

You mean full logs? Ok, no problem.
Let’s just wait another couple of days so we have some meaningful information available.

BrightSilence · February 21, 2022, 8:41am

There is a bit of a delay between issues happening and lowered scores being reported back to the node. So this doesn’t necessarily mean there haven’t been any errors logged that are now gone.

jammerdan · February 21, 2022, 8:46am

Yes I see that, but how much time is that?
Probably not more than 24 hours?
So I think it is fair to keep the node running so we have logs for a good period.

BrightSilence · February 21, 2022, 10:40am

Definitely shouldn’t be more than 24 hours. I would just suggest to ensure you have logs for longer so the response can’t ever be: “maybe the errors happened during that part that is missing in the logs you provided”

If you worry about losing logs in order to enable log redirection to a file you can always save the docker logs to a file prior to enabling the redirection.

If I ever run into an issue like this, my response would be: “Great, I have logs going back to June 2019, how much do you want?”. This also helps when missing file issues pop up. I can prove that a piece was delete many months ago if I have to.