One of my nodes got disqualified early this morning m I had a bad feeling when going to bed and I hope it is not related to changing time here over Europe. The reason why I make this post is after some research on the SNO troubleshooting guide , I havenāt found such any pattern .
Now the facts : during the night I spent some time understanding why the audit rate fell down : the disk was ejected for some reason and the node was up and down, I remounted the volume, but then the node stopped because of sync time issues with the satellite. I finally got the node stable after changing the ntp server . By the way the storage volume was checked and no error wad found in the file system.
Now is still running and it is restarting time to time, and I canāt understand what this means :
FATAL Unrecoverable error {āerrorā: "readdirent: not a directory;
Node is running on macosX with hfs volume.
Node ID is 12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB
I would appreciate if you could help understanding what happened, just to prevent this to happen again.
Regards,
Iām sad to hear that.
I can however assure you, that the time change in europe is not causing this, my node is fine and most nodes are in europe. Besides, time on PC systems is calculated in UTC. So local time changes donāt matter to them.
Time sync issues iirc would only stop your node/prevent it from starting but not DQ it.
It seems like you have some problem with disk reliability and macOS hfs volume stuff. I canāt help with that though, I have no knowledge of that environment.
Hello @nerdatwork
I shutdown my server ; stopped the usb device and started again,
I check the file system was ok using macos disk utility repair functionnality which does almost the same as a fsck.
When you receive this, the node will just crash to prevent a quick DQ.
But if your node is finally started, but missed files during audits - it will fail audits and can be DQ very quickly (a few hours).
I can assume that you run a docker container without logs redirected, so we canāt find a reason in your logs anymore.
If they were exist you could check the reason as described in
Hello @Alexey
Thanks for your prompt answer.
I will leave the node down until I have further instruction.
Here are my last debug actions.
logs are redirected, and I use the latest versions :
docker desktop 2.4.0 with docker engine 19.03.13 and storagenode:latest
Thanks for your help
DSK CHECK with DISK UTILITY macosX Catalina uptodate.
Running First Aid on āstormā (disk4s1)
Repairing file system.
Volume is already unmounted.
Performing fsck_apfs -y -x /dev/rdisk4s1
Checking the container superblock.
Checking the space manager.
Checking the space manager free queue trees.
Checking the object map.
Checking volume.
Checking the APFS volume superblock.
The volume storm was formatted by diskmanagementd (1412.61.1) and last modified by apfs_kext (1412.141.1).
Checking the object map.
Checking the snapshot metadata tree.
Checking the snapshot metadata.
Checking the extent ref tree.
Checking the extent ref tree.
Checking the fsroot tree.
Unfortunately there is no other instructions, if you checked your disk for errors and OS corrected them, then there is nothing we can do.
If your node disqualified on all satellites, there is only one way - to start from scratch with a new identity, new authorization token and clean storage.
If your node disqualified not on all satellites, you can decide to keep it running and receive payouts for service from customers of those satellites.
m I had a bad feeling when going to bed and I hope it is not related to changing time here over Europe. The reason why I make this post is after some research on the SNO troubleshooting guide , I havenāt found such any pattern .