The file or directory is corrupted and unreadable

Node Operators troubleshooting
1

2021-06-09T23:07:54.444+0300 ERROR piecestore:cache error getting current used space: {“error”: “CreateFile E:\storage\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\7e/27nkshcezhecpvy45ondndp7ges2e2hyiibwtjgtamfh7z64xa.sj1: The file or directory is corrupted and unreadable.”, “errorVerbose”: “CreateFile E:\storage\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\7e/27nkshcezhecpvy45ondndp7ges2e2hyiibwtjgtamfh7z64xa.sj1: The file or directory is corrupted and unreadable.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:280\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:86\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:85\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-06-09T23:07:54.444+0300 ERROR services unexpected shutdown of a runner {“name”: “piecestore:cache”, “error”: “CreateFile E:\storage\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\7e/27nkshcezhecpvy45ondndp7ges2e2hyiibwtjgtamfh7z64xa.sj1: The file or directory is corrupted and unreadable.”, “errorVerbose”: “CreateFile E:\storage\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\7e/27nkshcezhecpvy45ondndp7ges2e2hyiibwtjgtamfh7z64xa.sj1: The file or directory is corrupted and unreadable.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:280\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:86\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:85\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-06-09T23:07:54.444+0300 ERROR orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB rpc client error when closing sender {“error”: “order: CloseAndRecv settlement agreements returned an error: context canceled”, “errorVerbose”: “order: CloseAndRecv settlement agreements returned an error: context canceled\n\tstorj.io/storj/storagenode/orders.(*Service).settleWindow:275\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders.func1:205\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-06-09T23:07:54.444+0300 ERROR contact:service ping satellite failed {“Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “attempts”: 1, “error”: “ping satellite: context canceled”, “errorVerbose”: “ping satellite: context canceled\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:138\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-06-09T23:07:54.444+0300 INFO contact:service context cancelled {“Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”}
2021-06-09T23:07:54.444+0300 INFO orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB finished
2021-06-09T23:07:54.444+0300 ERROR nodestats:cache Get pricing-model/join date failed {“error”: “payouts service: context canceled”, “errorVerbose”: “payouts service: context canceled\n\tstorj.io/storj/storagenode/payouts.(*Endpoint).GetAllPaystubs:114\n\tstorj.io/storj/storagenode/nodestats.(*Cache).Run.func1:81\n\tstorj.io/storj/storagenode/nodestats.(*Cache).satelliteLoop:261\n\tstorj.io/storj/storagenode/nodestats.(*Cache).Run:80\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:86\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:85\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-06-09T23:07:54.445+0300 ERROR orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB failed to settle orders for satellite {“satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “error”: “order: CloseAndRecv settlement agreements returned an error: context canceled”, “errorVerbose”: “order: CloseAndRecv settlement agreements returned an error: context canceled\n\tstorj.io/storj/storagenode/orders.(*Service).settleWindow:275\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders.func1:205\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-06-09T23:07:54.445+0300 INFO piecestore upload canceled {“Piece ID”: “UUMFPUX5B6BTEL3V5U7JDPDB7MO4IWW2QA5I3XDSKJ472BPWMFZA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 0}
2021-06-09T23:07:54.445+0300 INFO piecestore upload canceled {“Piece ID”: “BXSCADWQQDZVYUSKUU43WJA6UOPUABRY4RONIFUV5B4W2AJ2L5GQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 1318912}
2021-06-09T23:07:54.446+0300 INFO piecestore upload canceled {“Piece ID”: “EDOJKL5CQ4KIPVMPKK4QSP7NOKCNTBMSP5O7P2DKSRTQKY2KF4YA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 532480}
2021-06-09T23:07:54.446+0300 INFO piecestore upload canceled {“Piece ID”: “3MMUYYLZ25R7XV57KIDLLCSU3566KG2ZDUY2Z3CZVA7GFOKMX7DA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 163840}
2021-06-09T23:07:54.447+0300 INFO piecestore upload canceled {“Piece ID”: “HVPUBRQNJCSLT7B5V3GVU3OHQ26AQOKRWP45DQ3TRPAVTIB67SOQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 0}
2021-06-09T23:07:54.447+0300 INFO piecestore upload canceled {“Piece ID”: “X2WVOMEQJHXSMENJG6JXNK4IVYTNK33KHH3D37IA6XLU4AIPU7QQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 794624}
2021-06-09T23:07:54.447+0300 INFO piecestore upload canceled {“Piece ID”: “MKLLDPG42NBRBTOQDMCRRJMEAZTQ72EWP6UN74TADLIJ2NT7X5XA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 163840}
2021-06-09T23:07:54.448+0300 INFO piecestore upload canceled {“Piece ID”: “ZH4YFNXOTUS24TWQUWNWLF5TLU64N7SHPXPRDAFDR544XST3SL7Q”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 163840}
2021-06-09T23:07:55.629+0300 FATAL Unrecoverable error {“error”: “CreateFile E:\storage\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\7e/27nkshcezhecpvy45ondndp7ges2e2hyiibwtjgtamfh7z64xa.sj1: The file or directory is corrupted and unreadable.”, “errorVerbose”: “CreateFile E:\storage\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\7e/27nkshcezhecpvy45ondndp7ges2e2hyiibwtjgtamfh7z64xa.sj1: The file or directory is corrupted and unreadable.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:280\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:86\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:85\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

more errors and finaly crashed…

2

You need to stop storagenode service either from Services applet or from the elevated PowerShell

Stop-Service storagenode

And check your disk for errors with automatic correction. When all errors would be fixed, you can try to start the storagenode and check your logs again

3

ok, I will try…

… 20 chars

4

You can use a heart icon to confirm

Please report back how is it going after the fix.

5

scan is in progress but eta is over 100 hours and rising… over 13 tb of data, wont the node be afected by the downtime ?

6

It will, but you need to finish the scan. By the way, the usual check and autofix would be enough at the moment. The surface scan would be needed, if errors would remain.
The online score will recover during the next 30 days online.

1 Like
7

File data verification completed.
An unspecified error occurred (75736e6a726e6c2e 4fa).

:frowning: what now ?

8

Check if the node runs… from an elevated Powershell:

If the node stops, please post the most recent log file entries. If it runs then great!

edit - You can copy the text from the log and put it between new lines with three backticks ``` so it’s more readable:

2021-05-29T11:07:53      INFO   download started .....
9

node seems to be running now for more than an hour without crashing

I have an error in logs and I dont know what to do about it

2021-06-09T23:07:55.629+0300 FATAL Unrecoverable error {“error”: “CreateFile E:\storage\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\7e/27nkshcezhecpvy45ondndp7ges2e2hyiibwtjgtamfh7z64xa.sj1: The file or directory is corrupted and unreadable.”, “errorVerbose”: “CreateFile E:\storage\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\7e/27nkshcezhecpvy45ondndp7ges2e2hyiibwtjgtamfh7z64xa.sj1: The file or directory is corrupted and unreadable.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:280\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:86\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:85\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

and a few errors ERROR collector unable to delete piece

10

It could be worth running the file check again as that may fix the issue.

If the ‘corrupted and unreadable’ FATAL log happens again it might be worth deleting the folder/file mentioned.

11

it may be but by the time I was done scanning for errors my online percent has droped to 93.7 %
it tooked 2 days to scan the hdd so I cant scan again any time soon :frowning:

12

You can be up to 288 hours offline before your node would be suspended.
The corruption error is very dangerous - it’s affect audits and your node could be disqualified much faster because of it.
I would suggest to fix the filesystem corruption before it affects your node.

1 Like
13

Hello!
Your advice helped me. I got an error
FATAL Unrecoverable error {“error”: “FindNextFile X:\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\7u: The file or directory is corrupted and unreadable.”}
The standard disk check service “fixed your disk” in 2 minutes and the service started without problems.
Something else scares me: This happened on a Seagate Exos X16 drive, which was bought new 2 months ago. The disk has passed the warranty check on the site and has not been subjected to overloads and overheating. The node is only 430 GB full.
Could such an error occur for software reasons? It seems to me that the error occurred when the node was being updated.
Or is this a reason to visit a service center?

14

What is the “standard disk check service”?

It looks like you’re on Windows so chkdsk /f on the main system drive and the drive holding the blobs should be sufficient.

1 Like
15

The reason of file system corruption can be power outage or abnormal reboot (BSOD or reset button) and so on.
For Windows it also recommended to disable a write cache for the disk with data.

By the way, what is filesystem on this drive?

1 Like
16

NTFS file system.
The cache has been enabled. I disabled it and also disabled defragmentation.
Should I turn off Search Indexing in Windows?

17

Defragmentation should not be disabled. You may disable Search indexing for this drive, since it is useless for a data drive, but should not affect performance anyway.

2 Likes
18

I have got the same error now. How do I check for disk errors with automatic correction? My PC had an abnormal shutdown, like you said above. A database was corrupted, I fixed that with this https://support.storj.io/hc/en-us/articles/360029309111-How-to-fix-a-database-disk-image-is-malformed. When I start my Storj node, it runs for 50 or 100 hours and then it shuts down and I get this error.

2024-04-11T21:18:44+02:00 ERROR failure during run {"error": "filewalker: CreateFile F:\\blobs\\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\\3d\\htyylynx7wktfysfztw7cxq6hmjjdfeir3uvhzkut6yqrku7qq.sj1: The file or directory is corrupted and unreadable.", "errorVerbose": "filewalker: CreateFile F:\\blobs\\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\\3d\\htyylynx7wktfysfztw7cxq6hmjjdfeir3uvhzkut6yqrku7qq.sj1: The file or directory is corrupted and unreadable.\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:69\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:74\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:727\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"} 2024-04-11T21:18:44+02:00 FATAL Unrecoverable error {"error": "filewalker: CreateFile F:\\blobs\\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\\3d\\htyylynx7wktfysfztw7cxq6hmjjdfeir3uvhzkut6yqrku7qq.sj1: The file or directory is corrupted and unreadable.", "errorVerbose": "filewalker: CreateFile F:\\blobs\\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\\3d\\htyylynx7wktfysfztw7cxq6hmjjdfeir3uvhzkut6yqrku7qq.sj1: The file or directory is corrupted and unreadable.\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:69\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:74\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:727\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}

19

The same way, as usual. Stop the node. Check the disk and fix all issues.
After that start the node back and check your logs.

20

Ah, found it. Was looking for https://kb.acronis.com/content/72322?ckattempt=1 or something like this chkdsk d: /F. Thanks!

1 Like