Node stops after few minutes

Node Operators troubleshooting
1

After weekend I found my node is not running. I manage to run it again using this:
https://support.storj.io/hc/en-us/articles/360029309111-How-to-fix-a-database-disk-image-is-malformed- and this Error during preflight check for storagenode databases: preflight: database "bandwidth" manuals.

Now I can run my node, but it’s bandwidth and current earnings resets to zero, and it stops after few minutes. Last lines of my log file:

2022-06-27T13:47:50.300+0300 ERROR piecestore:cache error getting current used space: {“error”: “readdir F:\Storj_storage\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\at: The system cannot find the path specified.; readdir F:\Storj_storage\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\c6: The system cannot find the path specified.”, “errorVerbose”: “group:\n— readdir F:\Storj_storage\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\at: The system cannot find the path specified.\n— readdir F:\Storj_storage\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\c6: The system cannot find the path specified.”}
2022-06-27T13:47:50.301+0300 ERROR services unexpected shutdown of a runner {“name”: “piecestore:cache”, “error”: “readdir F:\Storj_storage\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\at: The system cannot find the path specified.; readdir F:\Storj_storage\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\c6: The system cannot find the path specified.”, “errorVerbose”: “group:\n— readdir F:\Storj_storage\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\at: The system cannot find the path specified.\n— readdir F:\Storj_storage\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\c6: The system cannot find the path specified.”}
2022-06-27T13:47:51.815+0300 FATAL Unrecoverable error {“error”: “readdir F:\Storj_storage\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\at: The system cannot find the path specified.; readdir F:\Storj_storage\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\c6: The system cannot find the path specified.”, “errorVerbose”: “group:\n— readdir F:\Storj_storage\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\at: The system cannot find the path specified.\n— readdir F:\Storj_storage\blobs\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\c6: The system cannot find the path specified.”}

2

I may be wrong but it seems your config is edited (do you have an f drive?)if so it seems you could of lost some data

Also may need to change permissions on that f drive location if it exists

1 Like
3

Yes, I has drive F and data is here. Only these folders are missing:
\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\c6
\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\at

So if I lost some data, what should I do next? Should I start all from zero on another identity?

4

So you checked drive f and they are gone?
I am unsure but if your not disqualified yet you may be ok don’t try this before @Alexey or @littleskunk reply but maybe deleting some of the orders may work again don’t try unless someone else says it may help(but I think it would be hard to know what order go with blobs)
So maybe start again I am unsure wait for Alexey or little skunk

I am sorry I can only be of this much help

5

Hi @altas
No need to start again until the node is marked as disqualified on the larger satellites. The errors shown suggest data loss or corruption for which you should run drive/disk checking tools. If you are running Windows then:

1 Like
6

I already fixed all errors, and even checked HDD for bads sectors(none was found), but folders still missing and node keeps stopping after few minutes

7

The error listed means you have file system errors, so it’s unlikely they’re fixed.

ERROR piecestore:cache error getting current used space: {“error”: “readdir F:\Storj_storage\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\at: The system cannot find the path specified.

This literally means the node is trying to scan all blobs directories and the OS is telling it to look in a particular folder - F:\Storj_storage\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\at - however when the node tries to read the contents of that folder the OS can’t return file and folder information.

Did you check the disk SMART information? As the original error for the database as well as this error both point to corruption.

8

Disk sentinel shows disk health 100%
chkdsk output:

C:\WINDOWS\system32>chkdsk /f F:
The type of the file system is NTFS.
Volume label is Burst.

Stage 1: Examining basic file system structure ...
  546048 file records processed.
File verification completed.
 Phase duration (File record verification): 59.43 seconds.
  9 large file records processed.
 Phase duration (Orphan file record recovery): 0.00 milliseconds.
  0 bad file records processed.
 Phase duration (Bad file record checking): 0.57 milliseconds.

Stage 2: Examining file name linkage ...
  24 reparse records processed.
  564112 index entries processed.
Index verification completed.
 Phase duration (Index verification): 1.95 minutes.
  0 unindexed files scanned.
 Phase duration (Orphan reconnection): 256.39 milliseconds.
  0 unindexed files recovered to lost and found.
 Phase duration (Orphan recovery to lost and found): 2.29 milliseconds.
  24 reparse records processed.
 Phase duration (Reparse point and Object ID verification): 2.78 milliseconds.

Stage 3: Examining security descriptors ...
Security descriptor verification completed.
 Phase duration (Security descriptor verification): 68.69 milliseconds.
  9032 data files processed.
 Phase duration (Data attribute verification): 0.32 milliseconds.

Windows has scanned the file system and found no problems.
No further action is required.

   3815317 MB total disk space.
 163338436 KB in 500320 files.
    189604 KB in 9034 indexes.
         0 KB in bad sectors.
    731295 KB in use by the system.
     65536 KB occupied by the log file.
   3654908 MB available on disk.

      4096 bytes in each allocation unit.
 976721407 total allocation units on disk.
 935656574 allocation units available on disk.
Total duration: 2.95 minutes (177168 ms).```
9

Hmmm, interesting. Please reboot the system then post the latest log entries.

10

You mean node logs? Or system boot logs?

11

Storagenode log file

12

Rebooted, this time node ran longer
Log file start:

2022-06-28T14:52:59.389+0300	INFO	Configuration loaded	{"Location": "C:\\Program Files\\Storj\\Storage Node\\config.yaml"}
2022-06-28T14:52:59.399+0300	INFO	Operator email	{"Address": "audrius.alt@gmail.com"}
2022-06-28T14:52:59.399+0300	INFO	Operator wallet	{"Address": "0x9aB0bfD9659E19Ac947ded50D22cEC038F336Ebd"}
2022-06-28T14:53:02.048+0300	INFO	Telemetry enabled	{"instance ID": "1nd2G2CiFTfLPTTrxbErwZJL7NuyXJsDZsLXqT9oEVjwEdQZSa"}
2022-06-28T14:53:02.087+0300	INFO	db.migration	Database Version	{"version": 53}
2022-06-28T14:53:03.482+0300	INFO	preflight:localtime	start checking local system clock with trusted satellites' system clock.
2022-06-28T14:53:04.476+0300	INFO	preflight:localtime	local system clock is in sync with trusted satellites' system clock.
2022-06-28T14:53:04.534+0300	INFO	Node 1nd2G2CiFTfLPTTrxbErwZJL7NuyXJsDZsLXqT9oEVjwEdQZSa started
2022-06-28T14:53:04.534+0300	INFO	Public server started on [::]:28967
2022-06-28T14:53:04.534+0300	INFO	Private server started on 127.0.0.1:7778
2022-06-28T14:53:05.405+0300	INFO	piecestore	upload started	{"Piece ID": "ZEKECPZJ5UEXXLAUBMWIHVYZPBD6IH3VLKWVB3YG2F6XSATAC2OQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 2856611035648}
2022-06-28T14:53:05.360+0300	INFO	trust	Scheduling next refresh	{"after": "6h15m31.896814561s"}
2022-06-28T14:53:05.458+0300	INFO	bandwidth	Performing bandwidth usage rollups
2022-06-28T14:53:06.198+0300	ERROR	collector	unable to delete piece	{"Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "RFO662IIHTP6Q35NHB7PG6YMNKUS7HRSHUPY2ZIFRG44GHWQFLNA", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2022-06-28T14:53:06.227+0300	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "C6F46E4LNBHSXEIHCBKRNRDZJNTZBRJNP5RLQLCOY5UQHZF5FEVA", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2022-06-28T14:53:06.227+0300	ERROR	collector	unable to delete piece	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "C6W7RJZ3CQL547ZOHZX5AK36LDVFFJ4SPFXZKZL267VBGUWDZATQ", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2022-06-28T14:53:06.254+0300	INFO	piecestore	download started	{"Piece ID": "SRNZBRHB6J7LQJ663EZRR6WUTLAODSQGFVPGMPADPPKOPA5L7IQQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2022-06-28T14:53:07.066+0300	INFO	piecestore	download started	{"Piece ID": "YUWL2YBBZLQRBW57XC2M7TSICKUTWW6HAYD65ROIYRBD7VBSV6HA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2022-06-28T14:53:07.276+0300	INFO	piecestore	downloaded	{"Piece ID": "SRNZBRHB6J7LQJ663EZRR6WUTLAODSQGFVPGMPADPPKOPA5L7IQQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}

There is some errors in the middle like:

2022-06-28T14:53:12.499+0300	ERROR	blobscache	satPiecesTotal < 0	{"satPiecesTotal": -1812736}
2022-06-28T14:53:12.499+0300	ERROR	blobscache	satPiecesContentSize < 0	{"satPiecesContentSize": -1812224}

and

2022-06-28T15:09:36.129+0300	ERROR	piecestore	download failed	{"Piece ID": "JOCMYFQDIWWABXGFQYPWISTAGA45PPGKYH5VCLJJNQ3JSFMSXDIQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:546\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}

and log file ends with fatal:

2022-06-28T15:18:24.047+0300	ERROR	piecestore:cache	error getting current used space: 	{"error": "readdir F:\\Storj_storage\\blobs\\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\\at: The system cannot find the path specified.; readdir F:\\Storj_storage\\blobs\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\c6: The system cannot find the path specified.", "errorVerbose": "group:\n--- readdir F:\\Storj_storage\\blobs\\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\\at: The system cannot find the path specified.\n--- readdir F:\\Storj_storage\\blobs\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\c6: The system cannot find the path specified."}
2022-06-28T15:18:24.047+0300	ERROR	services	unexpected shutdown of a runner	{"name": "piecestore:cache", "error": "readdir F:\\Storj_storage\\blobs\\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\\at: The system cannot find the path specified.; readdir F:\\Storj_storage\\blobs\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\c6: The system cannot find the path specified.", "errorVerbose": "group:\n--- readdir F:\\Storj_storage\\blobs\\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\\at: The system cannot find the path specified.\n--- readdir F:\\Storj_storage\\blobs\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\c6: The system cannot find the path specified."}
2022-06-28T15:18:24.049+0300	INFO	piecestore	upload canceled	{"Piece ID": "HMM437ZI6ZYM77AVIUMHMMEFOMLQP7AMQUN3B6JUIBBOZBLBEAXA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 1581056}
2022-06-28T15:18:24.049+0300	INFO	piecestore	upload canceled	{"Piece ID": "SHRSWWAZVQM63UBT3FK4JAQU6PRQYIHOLLGNTLI2IH2INKM6Y4NA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 794624}
2022-06-28T15:18:25.510+0300	FATAL	Unrecoverable error	{"error": "readdir F:\\Storj_storage\\blobs\\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\\at: The system cannot find the path specified.; readdir F:\\Storj_storage\\blobs\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\c6: The system cannot find the path specified.", "errorVerbose": "group:\n--- readdir F:\\Storj_storage\\blobs\\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\\at: The system cannot find the path specified.\n--- readdir F:\\Storj_storage\\blobs\\v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa\\c6: The system cannot find the path specified."}
13

If you manually create those two missing folders maybe it would get past them without error.

4 Likes
14

Thanks, 5 hours running so far.