Node crashes when updating

Node Operators troubleshooting
,
1

so, i’ve noticed that my storage node starts crashing when updating, so it never restarts. It permanently stays offline, until i manually restart it.
The Hdd is plugged direcly into a sata port on a motherboard.
My drive is a Seagate Ironwolf nas drive.
Here is the log file
I can’t exactly copy the text, because my pc dies when opening it, but it might be useful.
Btw, my online/suspention thingy drops to 90%, but slowly recovers.

2

Hi @Kaupas24
You should start a new log file. Stop the storagenode, rename the existing log, start the storagenode again.

Looking at the most recent errors:

2021-11-08T23:31:52.835+0100 ERROR pieces:trash emptying trash failed {“error”: “pieces error: filestore error: FindNextFile F:\trash\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\24: The file or directory is corrupted and unreadable.”, “errorVerbose”: “pieces error: filestore error: FindNextFile F:\trash\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\24: The file or directory is corrupted and unreadable.\n\tstorj.io/storj/storage/filestore.(*blobStore).EmptyTrash:154\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).EmptyTrash:310\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:367\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1:51\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

2021-11-08T23:47:00.571+0100 ERROR piecestore:cache error getting current used space: {“error”: “open F:\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\qb: Access is denied.; FindNextFile F:\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\2r: The request failed due to a fatal device hardware error.; CreateFile F:\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\kn/vmzw2hcsnot6hgnoe32rwgczzwaivibxlygnsxgzkorzsze52q.sj1: The file or directory is corrupted and unreadable.”, “errorVerbose”: “group:\n— open F:\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\qb: Access is denied.\n— FindNextFile F:\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\2r: The request failed due to a fatal device hardware error.\n— CreateFile F:\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\kn/vmzw2hcsnot6hgnoe32rwgczzwaivibxlygnsxgzkorzsze52q.sj1: The file or directory is corrupted and unreadable.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:284\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T23:47:00.571+0100 ERROR services unexpected shutdown of a runner {“name”: “piecestore:cache”, “error”: “open F:\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\qb: Access is denied.; FindNextFile F:\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\2r: The request failed due to a fatal device hardware error.; CreateFile F:\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\kn/vmzw2hcsnot6hgnoe32rwgczzwaivibxlygnsxgzkorzsze52q.sj1: The file or directory is corrupted and unreadable.”, “errorVerbose”: “group:\n— open F:\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\qb: Access is denied.\n— FindNextFile F:\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\2r: The request failed due to a fatal device hardware error.\n— CreateFile F:\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\kn/vmzw2hcsnot6hgnoe32rwgczzwaivibxlygnsxgzkorzsze52q.sj1: The file or directory is corrupted and unreadable.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:284\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2021-11-08T23:47:02.473+0100 FATAL Unrecoverable error {“error”: “open F:\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\qb: Access is denied.; FindNextFile F:\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\2r: The request failed due to a fatal device hardware error.; CreateFile F:\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\kn/vmzw2hcsnot6hgnoe32rwgczzwaivibxlygnsxgzkorzsze52q.sj1: The file or directory is corrupted and unreadable.”, “errorVerbose”: “group:\n— open F:\blobs\qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa\qb: Access is denied.\n— FindNextFile F:\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\2r: The request failed due to a fatal device hardware error.\n— CreateFile F:\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\kn/vmzw2hcsnot6hgnoe32rwgczzwaivibxlygnsxgzkorzsze52q.sj1: The file or directory is corrupted and unreadable.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:284\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

2021-11-09T12:45:37.081+0100 ERROR piecestore:cache error getting current used space: {“error”: “FindNextFile F:\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\el: The request failed due to a fatal device hardware error.”}
2021-11-09T12:45:37.082+0100 ERROR services unexpected shutdown of a runner {“name”: “piecestore:cache”, “error”: “FindNextFile F:\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\el: The request failed due to a fatal device hardware error.”}
2021-11-09T12:45:37.694+0100 FATAL Unrecoverable error {“error”: “FindNextFile F:\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\el: The request failed due to a fatal device hardware error.”}

These all point to hardware issues, possibly causing corruption or slow speeds accessing the drive. These instructions from @Alexey should be sufficient, just change the D: to F: for your setup.

3

Thanks for the help, i’ve done everything, so now i’m going to wait and see if those errors crop up again. :smile:

4

Hi, after some time, i noticed that the node would stop running. I start the node, and then ~1h later it stops running.
Here is the new log file

5

A node stop is here:

2021-11-09T18:43:15.687+0100 INFO piecestore download started {“Piece ID”: “2P6TBIMARTOGEYVLM5S2YHMISD5TF2JKWQAFQR3APM5QZ2IOIVQA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”}

2021-11-09T18:43:15.921+0100 INFO piecestore upload started {“Piece ID”: “67U4VKQS5OSYBGX22USBIG6OC77RYC2SKRVOXJJ3YEEM73ZSVK7A”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 6975909426432}

2021-11-09T18:43:15.948+0100 INFO piecestore uploaded {“Piece ID”: “67U4VKQS5OSYBGX22USBIG6OC77RYC2SKRVOXJJ3YEEM73ZSVK7A”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 1536}

2021-11-09T18:43:16.275+0100 INFO Stop/Shutdown request received.

2021-11-09T18:43:16.275+0100 INFO piecestore downloaded {“Piece ID”: “2P6TBIMARTOGEYVLM5S2YHMISD5TF2JKWQAFQR3APM5QZ2IOIVQA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”}

2021-11-09T18:43:16.285+0100 INFO piecestore downloaded {“Piece ID”: “2P6TBIMARTOGEYVLM5S2YHMISD5TF2JKWQAFQR3APM5QZ2IOIVQA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”}

2021-11-09T18:50:16.757+0100 INFO Configuration loaded {“Location”: “C:\Program Files\Storj\Storage Node\config.yaml”}

This shows the node was requested to stop, either by the OS or something else. So it was a clean shutdown, not a crash or FATAL issue like before. Presumably you then started the node again 7 minutes later?

There is also this:

2021-11-09T22:15:38.644+0100 INFO piecestore downloaded {“Piece ID”: “VPOR2TFDMSAC3JGVUASY2DBUW6DCX5HPIW55KYR4YI7ZXZBRL3BQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”}

2021-11-09T22:15:38.822+0100 FATAL Unrecoverable error {“error”: “CreateFile F:\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\uu/v6oda2l2x2lhkzgvptc2jax2amnncbszzuykvox5c6fp7flo3q.sj1: The file or directory is corrupted and unreadable.”, “errorVerbose”: “CreateFile F:\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\uu/v6oda2l2x2lhkzgvptc2jax2amnncbszzuykvox5c6fp7flo3q.sj1: The file or directory is corrupted and unreadable.\n\tstorj.io/storj/storage/filestore.walkNamespaceWithPrefix:788\n\tstorj.io/storj/storage/filestore.(*Dir).walkNamespaceInPath:725\n\tstorj.io/storj/storage/filestore.(*Dir).WalkNamespace:685\n\tstorj.io/storj/storage/filestore.(*blobStore).WalkNamespace:284\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePieces:497\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:662\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:54\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

2021-11-10T16:13:47.927+0100 INFO Configuration loaded {“Location”: “C:\Program Files\Storj\Storage Node\config.yaml”}

It still looks like the drive has problems. Did you run chkdsk multiple times until no errors?

6

i ran checkdisk miltiple times, until i had no errors. Kinda wired that there are that many errors, since the drive is only 3 months old.

7

Btw, i ran a drive selftest, and a s.m.a.r.t check on that drive, and got no errors

8

Is this an USB drive? If so, you need to use an external power supply.
Perhaps you should change the cable or check its connection.

Since you have a filesystem corruption, you need to stop the storagenode, then check the disk from the elevated PowerShell or cmd.exe

chkdsk f: /f

Run it until no errors would be thrown. After that start the node back and keep eye on ERROR lines with blobs within.

cat "C:\Program Files\Storage Node\storagenode.log" -last 10 -wait | sls -CaseSensitive "ERROR" | sls "blobs"
1 Like
9

It is an internal data hard-drive. It’s powered by a computer power supply, and connected to the sata ports on my motherboard. I’ve tried changing the sata cable, but nothing changed. Once I get home, I’ll run checkdisk again.
The drive is not USB.
Thanks for the reply.