Server just went down - cant boot storagenode back up

Node Operators troubleshooting
22

I ran the start command and this just kept going for super long - just copied some of it in there.

C:\Storj>storagenode run --config-dir C:\Storj --identity-dir C:\Users\andre\AppData\Roaming\Storj\Identity\storagenode --log.output stderr
2022-07-21T10:14:49.287+0200 INFO Configuration loaded {“Process”: “storagenode”, “Location”: “C:\Storj\config.yaml”}
2022-07-21T10:14:49.287+0200 INFO Operator email {“Process”: “storagenode”, “Address”: “andreashalsegravesen@gmail.com”}
2022-07-21T10:14:49.287+0200 INFO Operator wallet {“Process”: “storagenode”, “Address”: “0xFBF22d386c10aeb329A8C245e99F9AB4Db92D16d”}
2022-07-21T10:14:49.988+0200 INFO Telemetry enabled {“Process”: “storagenode”, “instance ID”: “12nTqKMa2knajSvsXJgi9afn4q5gyk97dZgWatDRjaWtHRvtAJ3”}
2022-07-21T10:14:50.402+0200 INFO db.migration Database Version {“Process”: “storagenode”, “version”: 53}
2022-07-21T10:14:51.426+0200 INFO preflight:localtime start checking local system clock with trusted satellites’ system clock. {“Process”: “storagenode”}
2022-07-21T10:14:52.235+0200 INFO preflight:localtime local system clock is in sync with trusted satellites’ system clock. {“Process”: “storagenode”}
2022-07-21T10:14:52.241+0200 INFO Node 12nTqKMa2knajSvsXJgi9afn4q5gyk97dZgWatDRjaWtHRvtAJ3 started {“Process”: “storagenode”}
2022-07-21T10:14:52.244+0200 INFO Public server started on [::]:28967 {“Process”: “storagenode”}
2022-07-21T10:14:52.245+0200 INFO Private server started on 127.0.0.1:7778 {“Process”: “storagenode”}
2022-07-21T10:14:52.290+0200 INFO piecestore download started {“Process”: “storagenode”, “Piece ID”: “GPQIYJJPAKBDDTCIC7JVZM6AMFKGXI4MCKTO2RIDEQ2KH2KSHPMQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET_REPAIR”}
2022-07-21T10:14:52.299+0200 INFO piecestore download started {“Process”: “storagenode”, “Piece ID”: “GX7GTISK7RDYLE5IP7UCXMG6BQW4SLDHIJK5JPXOF75HHOOTSZFA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET_REPAIR”}
2022-07-21T10:14:52.299+0200 INFO piecestore download started {“Process”: “storagenode”, “Piece ID”: “NBXUFVPIG2KGVSFT7MJRMXLTLSBVLNCQQZEF2GC772XJ7ALRIB6A”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET”}
2022-07-21T10:14:52.388+0200 INFO piecestore download started {“Process”: “storagenode”, “Piece ID”: “7S6YL52QLPZAPJ6NDEPQQT2IYYYEJHEGYJ5FYAGTEH3W2MI2GPNQ”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET”}
2022-07-21T10:14:52.388+0200 INFO piecestore download started {“Process”: “storagenode”, “Piece ID”: “R4VUYBOSBA3O5TLWF4JPSSQMVACVQUXM4EYEZJ3JPNE523VQGBEA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET”}
2022-07-21T10:14:52.605+0200 INFO piecestore upload started {“Process”: “storagenode”, “Piece ID”: “HBHLFWQLFP5SRSPE256IBYOTBSV6L6KXE3UAJKAB7JPRTNBM2UTQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT_REPAIR”, “Available Space”: 14388088064}
2022-07-21T10:14:52.605+0200 INFO piecestore upload started {“Process”: “storagenode”, “Piece ID”: “TTGBUVLVV4A4CA4GO7KELBO7PFEJNEDACBX6OTSUXNJH4I7FZYAQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 14388088064}
2022-07-21T10:14:52.843+0200 INFO trust Scheduling next refresh {“Process”: “storagenode”, “after”: “6h9m4.701695803s”}
2022-07-21T10:14:52.856+0200 INFO bandwidth Performing bandwidth usage rollups {“Process”: “storagenode”}
2022-07-21T10:14:53.227+0200 INFO piecestore download started {“Process”: “storagenode”, “Piece ID”: “P3PDFSJMZQX6H2GLNDYIN6I6UDYFBEZYHNCLYTLOHKT6TSOS4VPA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”}
2022-07-21T10:14:53.298+0200 INFO piecestore upload started {“Process”: “storagenode”, “Piece ID”: “JE33C63LZ3XJ27UQY7I66NIZIVDYRYOOD53XXUHFP723VJXQMFBA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “PUT”, “Available Space”: 14387904512}
2022-07-21T10:14:53.364+0200 INFO piecestore download started {“Process”: “storagenode”, “Piece ID”: “YLFMBBQD7FOE6LOROXJLTB7ZZCXUCJHOCZHFLT5NNMVBFSQAE7UA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET”}
2022-07-21T10:14:54.046+0200 INFO piecestore upload started {“Process”: “storagenode”, “Piece ID”: “IF7636PLKBVYYRIDAA2JDCUTLAPEZRBRW2CNHOMB2VUMYFWVL2FA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “PUT”, “Available Space”: 14387891200}
2022-07-21T10:14:54.164+0200 INFO piecestore upload started {“Process”: “storagenode”, “Piece ID”: “RZOC6ZRHKJ7MAYGNKLNHC56C5U5WFEOZ3AENHCOLFNYX2S5OO5KA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “PUT”, “Available Space”: 14387891200}
2022-07-21T10:14:54.171+0200 INFO piecestore upload started {“Process”: “storagenode”, “Piece ID”: “LSJIS5RS4A5XR2LPJCAZYZ4P5PN3JU2OT7WRPJPKXCPTX3IUA2NQ”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “PUT”, “Available Space”: 14387891200}
2022-07-21T10:14:54.897+0200 INFO piecestore download started {“Process”: “storagenode”, “Piece ID”: “AAJCAZAFN7JAOSR54LMQUHBO5QL6GEOEIWUOVHQCOAGXWHXRQQ4A”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”}
2022-07-21T10:14:55.224+0200 INFO piecestore download started {“Process”: “storagenode”, “Piece ID”: “E2FXWIE4FVKHB4KPZOGXZ2EKIEYGNJBGGP24VVVF2Z6FABV7ULBA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET”}
2022-07-21T10:14:55.750+0200 INFO orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB sending {“Process”: “storagenode”, “count”: 0}
2022-07-21T10:14:55.750+0200 INFO orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 sending {“Process”: “storagenode”, “count”: 0}
2022-07-21T10:14:55.750+0200 INFO orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs sending {“Process”: “storagenode”, “count”: 0}
2022-07-21T10:14:55.750+0200 INFO orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S sending {“Process”: “storagenode”, “count”: 0}
2022-07-21T10:14:55.886+0200 INFO orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs finished {“Process”: “storagenode”}
2022-07-21T10:14:55.893+0200 INFO orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB finished {“Process”: “storagenode”}
2022-07-21T10:14:56.002+0200 ERROR collector unable to delete piece {“Process”: “storagenode”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Piece ID”: “6OIQU2XGCXHVNYVNFXRERTNQFTLRD7POUQWB4ARK36NOYYZFUHMQ”, “error”: “pieces error: filestore error: file does not exist”, “errorVerbose”: “pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2022-07-21T10:14:56.178+0200 ERROR collector unable to delete piece {“Process”: “storagenode”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Piece ID”: “5URT5FQP67KLUNXDKXYQ6J5OQJT7KYYTBWC35KCH2UOJPNZGPJKQ”, “error”: “pieces error: filestore error: file does not exist”, “errorVerbose”: “pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2022-07-21T10:14:56.194+0200 INFO orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S finished {“Process”: “storagenode”}
2022-07-21T10:14:56.380+0200 ERROR collector unable to delete piece {“Process”: “storagenode”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Piece ID”: “F7PQWPDIZJPVKL3RPYHPNT67POPCE4JWNAKOYYSCOTMYWXBM7ZCQ”, “error”: “pieces error: filestore error: file does not exist”, “errorVerbose”: “pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2022-07-21T10:14:56.556+0200 INFO orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 finished {“Process”: “storagenode”}
2022-07-21T10:14:56.656+0200 ERROR collector unable to delete piece {“Process”: “storagenode”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Piece ID”: “FUGAKA47NYDOCBDT4OPQPN3CJPDB2H4YAPAOLT7PDOQSHJHLXUWQ”, “error”: “pieces error: filestore error: file does not exist”, “errorVerbose”: “pieces error: filestore error: file does not exist\n\tstorj.io/storj/storage/filestore.(*blobStore).Stat:103\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:239\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Delete:220\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:299\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:97\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

23

Ok, now we seems fixed an underlaying issue, please stop this command and run the service.
You also need to merge all blobs folders from different locations with blobs folder in the correct location (assuming the correct one is StorjData).

24

when you say “run the service” what do you mean excaclty ? just so i dont mess up once again…

And how do i merge blobs?
And lastly - should i be doing any kind of clean up?

25

From the command line as an Administrator:

net start storagenode

or from PowerShell as an Administrator

Start-Service storagenode

or from the Services applet

26

Perfekt - just launched from services with no issue.
Im gonna attempt a reboot and see if it launches automaticly

27

copy content of every wrong blobs folder to the correct blobs folder, allow to join, but not overwrite existing files. It also should not rename anything.

28

Thanks will do.

I REALLY am happy for you help - it means alot.

29

You are welcome!
Do not forget to run storagenode-updater as well.

30

THanks -
So i can see that the updater is running in services and task manager - then all should be ok right?

31

Oh and by the way - i had 3 folders in different location named “blobs” - just like you said - but the two wrong ones are completly empty - so it should be all good right?

32

yes, both services should be running.
But health of both of them you can check in their logs.
Do not procrastinate to merge blobs folders, maybe it’s better to stop the storagenode service until you finish this step, otherwise the disqualification is fast and not reversible.

1 Like
33

but with both “wrong” blobs folders empty - there is nothing i need to do right?

34

Yes. But they should be somewhere. Where is exactly you can figure out from wrong config.yaml files - take a look on storage.path option inside.

35

I see - it doesnt have anything in the blobs folder but its does have a very few in the /orders/archive foldder…
As well as unsent.

36

image
image1452×881 114 KB

Everything is empty in “storage” but there is a few files in “orders”

37

You can try to merge a content of orders to C:\Storj\orders, but do not overwrite.
However it’s a little bit weird, that you run storagenode from a command line without options, and it downloaded pieces and stored them to nowhere.
Perhaps the config file in the %AppData% location have had a correct storage.path?

38

I do belive it actually had the correct storage path - atleast looking at it now it does.

Thanks alot :slight_smile:

1 Like