So various people, including me, have had filewalker fail, often with an error of “context canceled”. Alexey has often repeated that this is a symptom of a limited I/O. So much I can hear it wringing in my ears. And yes, in my case the disk are mounted as NFS shares. Also I can hear Alexey’s sco…

[image] EasyRhino: NFS [image] ISCSI vs NFS shares for nodes? Node Operators The network filesystem (NFS, SMB, SSHFS, etc.) have a different files lock mechanism than a local filesystems like NTFS or ext4, thus they behave differently for storagenode. The storagenod…

hmm interesting. in my case the databases are still stored on ssd so I didn’t have lock problems with that. Also the nfs shares are configured ‘async’ not ‘sync’, which was really slow when it was on sync.

ISCSI vs NFS shares for nodes? include storage (during upload for example) I do not recall any locking on the storage directory whatsoever in the source code. [image] EasyRhino: failing filewalkers is like… a problem that affects a significant chunk of SNO’s Do they fail because of co…

i think that’s a common reason I see when people post, yeah… lemme try searching the forum.

okay I searched for used-space-filewalker and looked at the first couple pages of results to see if there were logs. I know, highly scientific statistical sampling. 4 people had context canceled errors: Disk usage discrepancy? - Node Operators / troubleshooting - Storj Community Forum (official) …

Context canceled is just a way to signal to goroutines (like the file walker) there’s no point in them working anymore for external reasons. So for us it’s just an observation suggesting to look for critical failures in other parts of the node leading to node shutdown. [image] EasyRhino: Disk u…

[image] Toyoo: Context canceled is just a way to signal to goroutines (like the file walker) there’s no point in them working anymore for external reasons. Actually, of I run grep -i "context cancelled" against all my logs, they appear to happen now and then without any consequences as far as…

As these bugs have existed for a long time now, in my opinion I don’t think they will be publicly solved any time soon; as a fix. For the ‘bug bounty program’ seems only for security purposes, and has not garnered a single acknowledged reward to date (however anonymous it is), in this forum, or else…

Toyoo, thank you for explaining the meaning of the error more to my smooth brain. [image] JWvdV: Actually, of I run grep -i "context cancelled" Sorry, context canceled happens ALL THE TIME for uploads and downloads, when either the client just aborts the transfer or when it’s finished faster…

“Context canceled” errors specifically in the filewalkers usually a result of a timeout, so it was not able to finish an operation before the timeout is occurred. For network filesystems it’s kind of expected, and this is one of the reasons why we do not recommend them to use and do not support the…

So, how to debug failing filewalkers with "context canceled"

Node Operators troubleshooting

nerdatwork August 14, 2024, 5:28pm 2