Unrecoverable error, The file or directory is corrupted and unreadable

Node Operators troubleshooting
, ,
1

Hey folks,

I am getting the error above, and after running chkdsk, I get these errors in the screenshot as well. Any idea how to solve it? Thank you in advance!!

Screen Shot 2021-11-03 at 5.57.23 PM
Screen Shot 2021-11-03 at 5.57.23 PM1920×716 57.9 KB

2021-11-03T17:48:05.618-0700 INFO Configuration loaded {“Location”: “C:\Program Files\Storj\config.yaml”}
2021-11-03T17:48:05.620-0700 INFO Operator email {“Address”: “brandon@brandonernst.com”}
2021-11-03T17:48:05.620-0700 INFO Operator wallet {“Address”: “0x544D6187509F27Eb771F69bE11b7bc3026F0d4CD”}
2021-11-03T17:48:06.035-0700 INFO Telemetry enabled {“instance ID”: “12JtNAfpkrhrBXJZAYMek2dFtUgubtqK2SSEtTvhrMR8Y4p9s7a”}
2021-11-03T17:48:06.063-0700 INFO db.migration Database Version {“version”: 53}
2021-11-03T17:48:07.182-0700 INFO preflight:localtime start checking local system clock with trusted satellites’ system clock.
2021-11-03T17:48:07.825-0700 INFO preflight:localtime local system clock is in sync with trusted satellites’ system clock.
2021-11-03T17:48:07.825-0700 INFO Node 12JtNAfpkrhrBXJZAYMek2dFtUgubtqK2SSEtTvhrMR8Y4p9s7a started
2021-11-03T17:48:07.825-0700 INFO Public server started on [::]:28967
2021-11-03T17:48:07.826-0700 INFO Private server started on [::]:7778
2021-11-03T17:48:07.825-0700 INFO trust Scheduling next refresh {“after”: “5h15m52.073210025s”}
2021-11-03T17:48:07.826-0700 INFO bandwidth Performing bandwidth usage rollups
2021-11-03T17:48:10.325-0700 ERROR pieces:trash emptying trash failed {“error”: “pieces error: filestore error: CreateFile D:\Storj Data\trash\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\7k/tsicdlo737rmrvxgzg3tdvou6l2l4bjujyuhlvrmndjzsoh6sa.sj1: The file or directory is corrupted and unreadable.”, “errorVerbose”: “pieces error: filestore error: CreateFile D:\Storj Data\trash\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\7k/tsicdlo737rmrvxgzg3tdvou6l2l4bjujyuhlvrmndjzsoh6sa.sj1: The file or directory is corrupted and unreadable.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).EmptyTrash:466\n\tstorj.io/storj/storage/filestore.(*blobStore).EmptyTrash:153\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).EmptyTrash:310\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:367\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1:51\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-03T17:48:11.053-0700 INFO piecestore upload started {“Piece ID”: “5B3RA46TTKVUNJTJB77XJHUHU4POEMGETUQDWEVQJFXNJJZMSIWA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 12554413155968}
2021-11-03T17:48:12.915-0700 INFO orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S sending {“count”: 181}
2021-11-03T17:48:12.915-0700 INFO orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs sending {“count”: 7}
2021-11-03T17:48:12.915-0700 INFO orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 sending {“count”: 4}
2021-11-03T17:48:13.054-0700 WARN contact:service Your node is still considered to be online but encountered an error. {“Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Error”: “contact: failed to dial storage node (ID: 12JtNAfpkrhrBXJZAYMek2dFtUgubtqK2SSEtTvhrMR8Y4p9s7a) at address 173.13.172.194:28967 using QUIC: rpc: quic: timeout: no recent network activity”}
2021-11-03T17:48:13.214-0700 WARN contact:service Your node is still considered to be online but encountered an error. {“Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Error”: “contact: failed to dial storage node (ID: 12JtNAfpkrhrBXJZAYMek2dFtUgubtqK2SSEtTvhrMR8Y4p9s7a) at address 173.13.172.194:28967 using QUIC: rpc: quic: timeout: no recent network activity”}
2021-11-03T17:48:13.247-0700 WARN contact:service Your node is still considered to be online but encountered an error. {“Satellite ID”: “12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo”, “Error”: “contact: failed to dial storage node (ID: 12JtNAfpkrhrBXJZAYMek2dFtUgubtqK2SSEtTvhrMR8Y4p9s7a) at address 173.13.172.194:28967 using QUIC: rpc: quic: timeout: no recent network activity”}
2021-11-03T17:48:13.367-0700 INFO orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S finished
2021-11-03T17:48:13.475-0700 INFO orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs finished
2021-11-03T17:48:13.568-0700 INFO orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 finished
2021-11-03T17:48:13.828-0700 WARN contact:service Your node is still considered to be online but encountered an error. {“Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Error”: “contact: failed to dial storage node (ID: 12JtNAfpkrhrBXJZAYMek2dFtUgubtqK2SSEtTvhrMR8Y4p9s7a) at address 173.13.172.194:28967 using QUIC: rpc: quic: timeout: no recent network activity”}
2021-11-03T17:48:13.850-0700 WARN contact:service Your node is still considered to be online but encountered an error. {“Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Error”: “contact: failed to dial storage node (ID: 12JtNAfpkrhrBXJZAYMek2dFtUgubtqK2SSEtTvhrMR8Y4p9s7a) at address 173.13.172.194:28967 using QUIC: rpc: quic: timeout: no recent network activity”}
2021-11-03T17:48:13.965-0700 WARN contact:service Your node is still considered to be online but encountered an error. {“Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “Error”: “contact: failed to dial storage node (ID: 12JtNAfpkrhrBXJZAYMek2dFtUgubtqK2SSEtTvhrMR8Y4p9s7a) at address 173.13.172.194:28967 using QUIC: rpc: quic: timeout: no recent network activity”}
2021-11-03T17:48:17.485-0700 INFO piecestore upload started {“Piece ID”: “QSSQL5HDFY6VYSREWEOAHI6KPBBWE7TUCHELQXU6IGR4BBAMSKGA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 12554412973952}
2021-11-03T17:48:17.521-0700 INFO piecestore uploaded {“Piece ID”: “5B3RA46TTKVUNJTJB77XJHUHU4POEMGETUQDWEVQJFXNJJZMSIWA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 181504}
2021-11-03T17:48:17.623-0700 INFO piecestore upload started {“Piece ID”: “TDN43LSCCKG47ANNUEFDHUWCZY6I7EAO4RORMTNKMIOLB6V6SQ7A”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 12554412973952}
2021-11-03T17:48:21.226-0700 INFO piecestore download started {“Piece ID”: “2EZAQ3QSEHJC2WELXXV7HWF3OUV7TG6EIQRCWUO23ZQUA6PF6XQA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”}
2021-11-03T17:48:21.457-0700 INFO piecestore upload started {“Piece ID”: “FBWIPZ2JDCK2L7DWAIPWXRAS3VNH5IT63NK4F73ZUN2NLTCTFCHA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 12554412609920}
2021-11-03T17:48:21.546-0700 ERROR piecestore:cache error getting current used space: {“error”: “CreateFile D:\Storj Data\blobs\pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa\m7/7bqncfa2gunux7yewcjw723sixzcjpk7nvvtokkqi2vtuwld3a.sj1: The file or directory is corrupted and unreadable.; CreateFile D:\Storj Data\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\2z/rda27rnar2iwtnqh7frudgwxpm3fpu3nyyagfqdp5mfftwhjiq.sj1: The file or directory is corrupted and unreadable.”, “errorVerbose”: “group:\n— CreateFile D:\Storj Data\blobs\pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa\m7/7bqncfa2gunux7yewcjw723sixzcjpk7nvvtokkqi2vtuwld3a.sj1: The file or directory is corrupted and unreadable.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:284\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n— CreateFile D:\Storj Data\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\2z/rda27rnar2iwtnqh7frudgwxpm3fpu3nyyagfqdp5mfftwhjiq.sj1: The file or directory is corrupted and unreadable.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:284\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-03T17:48:21.546-0700 ERROR services unexpected shutdown of a runner {“name”: “piecestore:cache”, “error”: “CreateFile D:\Storj Data\blobs\pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa\m7/7bqncfa2gunux7yewcjw723sixzcjpk7nvvtokkqi2vtuwld3a.sj1: The file or directory is corrupted and unreadable.; CreateFile D:\Storj Data\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\2z/rda27rnar2iwtnqh7frudgwxpm3fpu3nyyagfqdp5mfftwhjiq.sj1: The file or directory is corrupted and unreadable.”, “errorVerbose”: “group:\n— CreateFile D:\Storj Data\blobs\pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa\m7/7bqncfa2gunux7yewcjw723sixzcjpk7nvvtokkqi2vtuwld3a.sj1: The file or directory is corrupted and unreadable.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:284\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n— CreateFile D:\Storj Data\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\2z/rda27rnar2iwtnqh7frudgwxpm3fpu3nyyagfqdp5mfftwhjiq.sj1: The file or directory is corrupted and unreadable.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:284\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-03T17:48:21.549-0700 INFO piecestore downloaded {“Piece ID”: “2EZAQ3QSEHJC2WELXXV7HWF3OUV7TG6EIQRCWUO23ZQUA6PF6XQA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”}
2021-11-03T17:48:21.696-0700 INFO piecestore upload canceled {“Piece ID”: “FBWIPZ2JDCK2L7DWAIPWXRAS3VNH5IT63NK4F73ZUN2NLTCTFCHA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 0}
2021-11-03T17:48:21.698-0700 INFO piecestore uploaded {“Piece ID”: “TDN43LSCCKG47ANNUEFDHUWCZY6I7EAO4RORMTNKMIOLB6V6SQ7A”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 181504}
2021-11-03T17:48:21.723-0700 INFO piecestore uploaded {“Piece ID”: “QSSQL5HDFY6VYSREWEOAHI6KPBBWE7TUCHELQXU6IGR4BBAMSKGA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 181504}
2021-11-03T17:48:23.393-0700 FATAL Unrecoverable error {“error”: “CreateFile D:\Storj Data\blobs\pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa\m7/7bqncfa2gunux7yewcjw723sixzcjpk7nvvtokkqi2vtuwld3a.sj1: The file or directory is corrupted and unreadable.; CreateFile D:\Storj Data\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\2z/rda27rnar2iwtnqh7frudgwxpm3fpu3nyyagfqdp5mfftwhjiq.sj1: The file or directory is corrupted and unreadable.”, “errorVerbose”: “group:\n— CreateFile D:\Storj Data\blobs\pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa\m7/7bqncfa2gunux7yewcjw723sixzcjpk7nvvtokkqi2vtuwld3a.sj1: The file or directory is corrupted and unreadable.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:284\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n— CreateFile D:\Storj Data\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\2z/rda27rnar2iwtnqh7frudgwxpm3fpu3nyyagfqdp5mfftwhjiq.sj1: The file or directory is corrupted and unreadable.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:284\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

2

You need to stop the storagenode and run chkdsk d: /f until it will not throw any errors during the check.
If the directory still not readable, then run chkdsk d: /f /b, it will scan for bad blocks and will try to recover them or at least mark them.

1 Like
3

Also if you have any bad blocks you should replace your drive.

4

@Alexey Thanks, I ran `chkdsk d: /F`` originally, and it seemed to get stuck forever, with the estimated time remaining continuing to count upwards, so I quit it and made this post.

Then I ran it again and it worked within minutes.

6

@konsou, thanks but I don’t think jumping to buy a new drive is the first solution to try if you have bad blocks. I’m pretty sure chkdsk would be good try to first, among other things.

I would quit STORJ before investing in a new drive at this point, as this drive isn’t even 6 months old :slight_smile:

7

If you don’t see any errors from the topic, then chkdsk /f has fixed the problem.
If not, you need to run it several times until it will not throw any errors.
If after that you still have issues - then you need to run it with /b, i.e. chkdsk /f /b d:

Unfortunately the chkdsk from the latest Windows can throw exceptions during execution, this is usually means, that you need to run it again. When all errors are corrected, it should just do a quick scan and give stat, then normally exit (without any errors).
If there are errors, it will perform slowly, this is expected, so be patient.

1 Like
8

If the drive has bad blocks it’s faulty and will likely fail pretty soon.

If it’s that new and has bad sectors it should be under warranty and be replaced for free.

2 Likes
9

2 posts were split to a new topic: Your forum’s emails went into my spam due to DNS misconfiguration