I have a problem that i can’t resolve for weeks now. My node keeps shutting down whith this errors:
2021-10-11T11:57:54.237+0200 ERROR services unexpected shutdown of a runner {"name": "piecestore:cache", "error": "CreateFile D:\\blobs\\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\\4y/4cgoeou3crgckuqyxqdbkqa3jgdk7xm7v7kmpp4z5ynqx4xhlq.sj1: Plik lub katalog jest uszkodzony i nieczytelny.", "errorVerbose": "CreateFile D:\\blobs\\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\\4y/4cgoeou3crgckuqyxqdbkqa3jgdk7xm7v7kmpp4z5ynqx4xhlq.sj1: Plik lub katalog jest uszkodzony i nieczytelny.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:284\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2021-10-11T11:57:54.240+0200 INFO piecestore upload canceled {"Piece ID": "HZTNBFINKNYDQ5CJXFBKKGYEOBPHI3E4BJGLIUH53M2GXT5LTAQQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 0}
2021-10-11T11:57:54.242+0200 INFO piecestore upload canceled {"Piece ID": "7RIAP7QTKFRJXHSWU5TITOJG75GZA4LRBIHG6MU3IJA5S76SYKPQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 65536}
2021-10-11T11:57:54.243+0200 INFO piecestore downloaded {"Piece ID": "W3NQ7BP7DWJFB5KNUA5RG53AAHYC5N4PQXAWBR2ZELXJ7EEABIDQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-10-11T11:57:54.245+0200 INFO piecestore upload canceled {"Piece ID": "NU3QNNNZYE3FP7BKSDOGNRZQQY5FZ6K5QDKDZO7IPUN2BVISBSIQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 65536}
2021-10-11T11:57:54.246+0200 INFO piecestore downloaded {"Piece ID": "F5XZHN2SSI64OEXW2NE5EDH2FTZOCEUVSSPHOXG7PZ3ZSKYPCAMA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-10-11T11:57:54.250+0200 INFO piecestore upload canceled {"Piece ID": "VQYMYGEXFHMVOQQ5RR74WTM5NSJHA3YUESYC25S3CBJFFESX7MBA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 65536}
2021-10-11T11:57:54.250+0200 INFO piecestore downloaded {"Piece ID": "KO3UPL7H64AE7TTSCNPSQNGMXLPOGP7657LG7F6CXDKTOENJVIUA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-10-11T11:57:55.038+0200 FATAL Unrecoverable error {"error": "CreateFile D:\\blobs\\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\\4y/4cgoeou3crgckuqyxqdbkqa3jgdk7xm7v7kmpp4z5ynqx4xhlq.sj1: Plik lub katalog jest uszkodzony i nieczytelny.", "errorVerbose": "CreateFile D:\\blobs\\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\\4y/4cgoeou3crgckuqyxqdbkqa3jgdk7xm7v7kmpp4z5ynqx4xhlq.sj1: Plik lub katalog jest uszkodzony i nieczytelny.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:284\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
I have tried to check disk for erorrs both in properties of disk and thru CMD multiple times and it’s always the same error but in different path. I bought new WD RED HDD so i would be surprised if it’s some problem with disk itself. I’m not super savy in this stuff so i ask for advice for dummies.
Thanks
i would check that the folders and files aren’t read only, and try to push that through.
sometimes when moving data from one windows to another if one don’t have the same user, one might see permissions change, because the new user doesn’t own the files essentially.
duno if this is the case, but it would be my first thing to go through, and do keep in mind changing permissions on millions of files will take a while… hours if not longer.
I’ve tried to uncheck read only dozen times and it keeps
being checked. I have read that since windows XP this feature is ignored for folders and changing it is bigger case.
I’ve read in other post that SYSTEM should be the owner but i will try to change that and see if it helps.
One thing that could mean something:
When i tried to change Read Only after node crash windows said that could not change the properties in some place due to being corrupted and that’s the same file that caused node to shut down. After disk check Read only uncheck went smoothly. And after another shut downs same story. Looks like node itself is damaging some paths in disc or something??? Idk just my first thought.
smart is by far the fastest method, but also the least accurate and only really tells you if it thinks there is a hardware issue.
a disk check tests the data or really the file structure and basic data integrity, far from perfect but should be able to tell you if there is something wrong with the files or file structure.
i really hate this tool, at the first go i wouldn’t allow it to repair anything.
it can sometimes cause more problems than it solves, however if there is a problem it can also sometimes fix it… but if all else fails its usually a perfectly viable way to go.
You need to stop the node and run chkdsk /f D:\ in the cmd.exe/PowerShell as an Administrator.
The command could throw an error, in that case you need to run it again until all errors would be fixed.
After that you can check permissions.
I have run the chkdsk few times and it only made shit worse. The HDD is really messed up and have many bad sectors. Must have been power loss or something. I will try to fix this by some recovery software but i think i will need to wipe the drive… my node lost all the reputation anyway and it’s “only” 600GB :c . I would appreciate any other ideas tho.
You need to run chkdsk until it would not throw any errors.
If you have bad blocks, then you would need to run it with /B /F options. It should find and mark bad blocks.
The storagenode must be stopped during the process.
If you mean the online score, then it can be recovered, you will need to keep your node online for the next 30 days.
If you mean the audit score - it cannot recover if it fall below 60%.
i did run chkdsk like 4-5 times now and always ends with
An unspecified error occurred (6672732e637878 969).
An unspecified error occurred (6e74667363686b2e 17b4).
and everytime it takes like 10 hours minimum… If you say i should continue and it will correct all the bad clusters then i will. In other case i will try to return the HDD and get the new one.
PS. i always run CHKDSK with /f /r parameters. i did not try with /b.
P.S.2 somehow i have 100% audit score. how is that possible? my node is not running for like 3 weeks now. ofc it got like 50% online but i want to understand how i can get no failed audits.
You need to run chkdsk /f, the /r is not needed.
The audit score will be affected only if you lost/corrupt pieces. The downtime affects the online score.
There is only one way to check
If your disk wouldn’t die during the process, if you did not lose pieces during double migration, the node should work normally. You will need to be online for the next 30 days to recover the online score.
I would suggest to run chkdsk /f D: several times until
stops to appear. I have had a similar situation, the chkdsk /f have been executed near the 40 times before it could fix all issues.
I think the chkdsk in Windows 10 is broken somehow.
