Suspended node and damaged file / folder problem

[patryk006]

My service in windows 10 is getting shutdown because of broken file / folder

2022-02-21T23:50:15.198+0100	ERROR	piecestore:cache	error getting current used space: 	{"error": "CreateFile I:\\Storj\\Shared\\blobs\\pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa\\7f/dcb6nelqogqoxlrvwictbbsiwkjxzq5ecx26bhr7qww4sdq6wa.sj1: The file or folder is damaged and cannot be read", "errorVerbose": "CreateFile I:\\Storj\\Shared\\blobs\\pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa\\7f/dcb6nelqogqoxlrvwictbbsiwkjxzq5ecx26bhr7qww4sdq6wa.sj1: The file or folder is damaged and cannot be read\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:284\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2022-02-21T23:50:15.206+0100	ERROR	services	unexpected shutdown of a runner	{"name": "piecestore:cache", "error": "CreateFile I:\\Storj\\Shared\\blobs\\pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa\\7f/dcb6nelqogqoxlrvwictbbsiwkjxzq5ecx26bhr7qww4sdq6wa.sj1: The file or folder is damaged and cannot be read", "errorVerbose": "CreateFile I:\\Storj\\Shared\\blobs\\pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa\\7f/dcb6nelqogqoxlrvwictbbsiwkjxzq5ecx26bhr7qww4sdq6wa.sj1: The file or folder is damaged and cannot be read\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:284\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2022-02-21T23:50:15.296+0100	FATAL	Unrecoverable error	{"error": "CreateFile I:\\Storj\\Shared\\blobs\\pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa\\7f/dcb6nelqogqoxlrvwictbbsiwkjxzq5ecx26bhr7qww4sdq6wa.sj1: The file or folder is damaged and cannot be read", "errorVerbose": "CreateFile I:\\Storj\\Shared\\blobs\\pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa\\7f/dcb6nelqogqoxlrvwictbbsiwkjxzq5ecx26bhr7qww4sdq6wa.sj1: The file or folder is damaged and cannot be read\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:284\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

is there any chance to recover node with 20% uptime now ? And how to try resolve the problem.
Hdd seems to be good.

Any advice what to do now ? Thanks

[Stob]

Hi @patryk006

Is it a 20% Online score? That isn’t good but we can try to keep the node online and fix the problems.

If the S.M.A.R.T reading on the drive looks good then you should run “chkdsk /f /r” on it, usually with the node service stopped.

[patryk006]

Ye, I know thats terrible online time, any good monitoring for nodes ? I cannot look on it every day.

Ok, small update.

1. Full sector scan = 100% good
2. Chkdsk found around 30 broken file indexes, I hope that’s it cause this error.

[Stob]

Yeah, so the physical drive seems fine. It’s just the structure on top which chkdsk had to fix, you should run it again until no errors are found.

Then the node should run reliably and you should keep it online as much as possible before it gets suspended or disqualified. I’m not sure if uptime disqualification is even active at the moment.

Some people use uptimerobot to email them if the node port is down. If you search the forums there are some other tools SNO’s recommend.

[Alexey]

Mail today: You were suspended on 2021-11-23 at 12:20 UTC

Looks like you should be ok here forward. For monitoring, many people like UptimeRobot Some use a phone alert system Or scripts

I would also like to suggest taking a look at

[Tech Preview] Email alerts with Grafana and Prometheus

I would like to setup email alerts for my storage nodes. It turns out the storage node already has a Prometheus endpoint. It is running on the debug endpoint /metrics. If you haven’t set the debug endpoint please visit Guide to debug my storage node, uplink, s3 gateway, satellite I have already a Prometheus instance running and Grafana inclusive email alerts. Here are my configs: docker-compose.yaml version: "3.3" services: prometheus: restart: unless-stopped user: 993:991 ports…

[patryk006]

Slowly recovering and getting new data.
Thanks