The file or directory is corrupted and unreadable

2021-06-09T23:07:54.444+0300 ERROR piecestore:cache error getting current used space: {“error”: “CreateFile E:\storage\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\7e/27nkshcezhecpvy45ondndp7ges2e2hyiibwtjgtamfh7z64xa.sj1: The file or directory is corrupted and unreadable.”, “errorVerbose”: “CreateFile E:\storage\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\7e/27nkshcezhecpvy45ondndp7ges2e2hyiibwtjgtamfh7z64xa.sj1: The file or directory is corrupted and unreadable.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:280\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:86\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:85\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-06-09T23:07:54.444+0300 ERROR services unexpected shutdown of a runner {“name”: “piecestore:cache”, “error”: “CreateFile E:\storage\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\7e/27nkshcezhecpvy45ondndp7ges2e2hyiibwtjgtamfh7z64xa.sj1: The file or directory is corrupted and unreadable.”, “errorVerbose”: “CreateFile E:\storage\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\7e/27nkshcezhecpvy45ondndp7ges2e2hyiibwtjgtamfh7z64xa.sj1: The file or directory is corrupted and unreadable.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:280\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:86\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:85\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-06-09T23:07:54.444+0300 ERROR orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB rpc client error when closing sender {“error”: “order: CloseAndRecv settlement agreements returned an error: context canceled”, “errorVerbose”: “order: CloseAndRecv settlement agreements returned an error: context canceled\n\tstorj.io/storj/storagenode/orders.(*Service).settleWindow:275\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders.func1:205\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-06-09T23:07:54.444+0300 ERROR contact:service ping satellite failed {“Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “attempts”: 1, “error”: “ping satellite: context canceled”, “errorVerbose”: “ping satellite: context canceled\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:138\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-06-09T23:07:54.444+0300 INFO contact:service context cancelled {“Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”}
2021-06-09T23:07:54.444+0300 INFO orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB finished
2021-06-09T23:07:54.444+0300 ERROR nodestats:cache Get pricing-model/join date failed {“error”: “payouts service: context canceled”, “errorVerbose”: “payouts service: context canceled\n\tstorj.io/storj/storagenode/payouts.(*Endpoint).GetAllPaystubs:114\n\tstorj.io/storj/storagenode/nodestats.(*Cache).Run.func1:81\n\tstorj.io/storj/storagenode/nodestats.(*Cache).satelliteLoop:261\n\tstorj.io/storj/storagenode/nodestats.(*Cache).Run:80\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:86\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:85\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-06-09T23:07:54.445+0300 ERROR orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB failed to settle orders for satellite {“satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “error”: “order: CloseAndRecv settlement agreements returned an error: context canceled”, “errorVerbose”: “order: CloseAndRecv settlement agreements returned an error: context canceled\n\tstorj.io/storj/storagenode/orders.(*Service).settleWindow:275\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders.func1:205\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-06-09T23:07:54.445+0300 INFO piecestore upload canceled {“Piece ID”: “UUMFPUX5B6BTEL3V5U7JDPDB7MO4IWW2QA5I3XDSKJ472BPWMFZA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 0}
2021-06-09T23:07:54.445+0300 INFO piecestore upload canceled {“Piece ID”: “BXSCADWQQDZVYUSKUU43WJA6UOPUABRY4RONIFUV5B4W2AJ2L5GQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 1318912}
2021-06-09T23:07:54.446+0300 INFO piecestore upload canceled {“Piece ID”: “EDOJKL5CQ4KIPVMPKK4QSP7NOKCNTBMSP5O7P2DKSRTQKY2KF4YA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 532480}
2021-06-09T23:07:54.446+0300 INFO piecestore upload canceled {“Piece ID”: “3MMUYYLZ25R7XV57KIDLLCSU3566KG2ZDUY2Z3CZVA7GFOKMX7DA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 163840}
2021-06-09T23:07:54.447+0300 INFO piecestore upload canceled {“Piece ID”: “HVPUBRQNJCSLT7B5V3GVU3OHQ26AQOKRWP45DQ3TRPAVTIB67SOQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 0}
2021-06-09T23:07:54.447+0300 INFO piecestore upload canceled {“Piece ID”: “X2WVOMEQJHXSMENJG6JXNK4IVYTNK33KHH3D37IA6XLU4AIPU7QQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 794624}
2021-06-09T23:07:54.447+0300 INFO piecestore upload canceled {“Piece ID”: “MKLLDPG42NBRBTOQDMCRRJMEAZTQ72EWP6UN74TADLIJ2NT7X5XA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 163840}
2021-06-09T23:07:54.448+0300 INFO piecestore upload canceled {“Piece ID”: “ZH4YFNXOTUS24TWQUWNWLF5TLU64N7SHPXPRDAFDR544XST3SL7Q”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 163840}
2021-06-09T23:07:55.629+0300 FATAL Unrecoverable error {“error”: “CreateFile E:\storage\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\7e/27nkshcezhecpvy45ondndp7ges2e2hyiibwtjgtamfh7z64xa.sj1: The file or directory is corrupted and unreadable.”, “errorVerbose”: “CreateFile E:\storage\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\7e/27nkshcezhecpvy45ondndp7ges2e2hyiibwtjgtamfh7z64xa.sj1: The file or directory is corrupted and unreadable.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:280\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:86\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:85\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

more errors and finaly crashed…

You need to stop storagenode service either from Services applet or from the elevated PowerShell

Stop-Service storagenode

And check your disk for errors with automatic correction. When all errors would be fixed, you can try to start the storagenode and check your logs again

ok, I will try…

… 20 chars

You can use a heart icon to confirm

Please report back how is it going after the fix.

scan is in progress but eta is over 100 hours and rising… over 13 tb of data, wont the node be afected by the downtime ?

It will, but you need to finish the scan. By the way, the usual check and autofix would be enough at the moment. The surface scan would be needed, if errors would remain.
The online score will recover during the next 30 days online.

1 Like

File data verification completed.
An unspecified error occurred (75736e6a726e6c2e 4fa).

:frowning: what now ?

Check if the node runs… from an elevated Powershell:

If the node stops, please post the most recent log file entries. If it runs then great!

edit - You can copy the text from the log and put it between new lines with three backticks ``` so it’s more readable:

2021-05-29T11:07:53      INFO   download started .....

node seems to be running now for more than an hour without crashing

I have an error in logs and I dont know what to do about it

2021-06-09T23:07:55.629+0300 FATAL Unrecoverable error {“error”: “CreateFile E:\storage\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\7e/27nkshcezhecpvy45ondndp7ges2e2hyiibwtjgtamfh7z64xa.sj1: The file or directory is corrupted and unreadable.”, “errorVerbose”: “CreateFile E:\storage\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\7e/27nkshcezhecpvy45ondndp7ges2e2hyiibwtjgtamfh7z64xa.sj1: The file or directory is corrupted and unreadable.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:280\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:86\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:85\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

and a few errors ERROR collector unable to delete piece

It could be worth running the file check again as that may fix the issue.

If the ‘corrupted and unreadable’ FATAL log happens again it might be worth deleting the folder/file mentioned.

it may be but by the time I was done scanning for errors my online percent has droped to 93.7 %
it tooked 2 days to scan the hdd so I cant scan again any time soon :frowning:

You can be up to 288 hours offline before your node would be suspended.
The corruption error is very dangerous - it’s affect audits and your node could be disqualified much faster because of it.
I would suggest to fix the filesystem corruption before it affects your node.