Node continuously restart "Error: filewalker: readdirent input/output error"

Node Operators troubleshooting
1

Hello,

I have a node that continuosly restart every 5 minutes, looking the log I see this error:

2024-05-06T10:49:19Z ERROR piecestore:cache error getting current used space: {“Process”: “storagenode”, “error”: “filewalker: readdirent config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/fv: input/output error”, “errorVerbose”: “filewalker: readdirent config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/fv: input/output error\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:79\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:747\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}
2024-05-06T10:49:19Z ERROR services unexpected shutdown of a runner {“Process”: “storagenode”, “name”: “piecestore:cache”, “error”: “filewalker: readdirent config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/fv: input/output error”, “errorVerbose”: “filewalker: readdirent config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/fv: input/output error\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:79\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:747\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}
2024-05-06T10:49:48Z ERROR failure during run {“Process”: “storagenode”, “error”: “filewalker: readdirent config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/fv: input/output error”, “errorVerbose”: “filewalker: readdirent config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/fv: input/output error\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:79\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:747\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}

Last message before the restart is:

Error: filewalker: readdirent config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/fv: input/output error

do you know how I can fix it ?

Thx

2

Check your connection to the hard drive. How is it connected ?

1 Like
3

Its directly connected on SATA port, then disabling filewarker the node doesn’t restart anymore, could be a bad block on HDD ?

4

Try changing the port and cable first.

5

this is not the issue, I have another node running on same physical disk that has no problem

6

In that case, try checking the logical drive for bad sector.

2 Likes
7

This is forbidden by Supplier Terms & Conditions for exactly the reason they inevitable will affect each other.
I would recommend to stop and remove both containers, unmount the disk and check and fix the filesystem. Perhaps you would need to run it several times, until all errors will be fixed. Then mount the disk back and run the node, then monitor it.

1 Like
8

Please refer to your file system documentation for help, as this is a file system problem, not a Storj problem.

1 Like