Hey
I am a STO for over a year, running my home node with Rasbperry Pi 4.
Recently, it appears that I am about to be suspended, due to my node being offline for some time. My dashboard is not working. Docker showed me this:
Error: Error starting master database on storagenode: database: orders opening file "config/storage/orders.db" failed: file is not a database
storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:347
storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:324
storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:299
storj.io/storj/storagenode/storagenodedb.OpenExisting:266
main.cmdRun:62
main.newRunCmd.func1:32
storj.io/private/process.cleanup.func1.4:399
storj.io/private/process.cleanup.func1:417
github.com/spf13/cobra.(*Command).execute:852
github.com/spf13/cobra.(*Command).ExecuteC:960
github.com/spf13/cobra.(*Command).Execute:897
storj.io/private/process.ExecWithCustomOptions:113
main.main:30
runtime.main:250
2023-06-29 16:25:55,017 INFO exited: storagenode (exit status 1; not expected)
Unfortunately, I’m not good with linux (or CLI’s for that matter). So for almost a week I’ve been trying to fix the issue myself - to no avail, hence why I’m making this post.
Context: my node is running on external hard drive
Could you help me, by telling me exactly what kind of commands I need to execute. From the logs I can’t even tell if the malfunction file in on the SD card, or on the external drive
2023-06-29T19:51:35.469Z WARN piecestore:monitor Disk space is less than requested. Allocated space is {"process": "storagenode", "bytes": 138071052288}
2023-06-29T19:51:35.469Z ERROR piecestore:monitor Total disk space is less than required minimum {"process": "storagenode", "bytes": 500000000000}
2023-06-29T19:51:35.469Z ERROR services unexpected shutdown of a runner {"process": "storagenode", "name": "piecestore:monitor", "error": "piecestore monitor: disk space requirement not met", "errorVerbose": "piecestore monitor: disk space requirement not met\n\tstorj.io/storj/storagenode/monitor.(*Service).Run:128\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-06-29T19:51:35.469Z ERROR gracefulexit:blobscleaner couldn't receive satellite's GE status {"process": "storagenode", "error": "context canceled"}
2023-06-29T19:51:35.470Z ERROR gracefulexit:chore error retrieving satellites. {"process": "storagenode", "error": "satellitesdb: context canceled", "errorVerbose": "satellitesdb: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*satellitesDB).ListGracefulExits:149\n\tstorj.io/storj/storagenode/gracefulexit.(*Service).ListPendingExits:59\n\tstorj.io/storj/storagenode/gracefulexit.(*Chore).AddMissing:58\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/storj/storagenode/gracefulexit.(*Chore).Run:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-06-29T19:51:35.470Z INFO Node 12gbm5ZkUtkT2TUJT7tW5psFgLsdkD8HmmnRVxMDN13Aovx3yCp started {"process": "storagenode"}
2023-06-29T19:51:35.470Z INFO Public server started on [::]:28967 {"process": "storagenode"}
2023-06-29T19:51:35.470Z INFO Private server started on 127.0.0.1:7778 {"process": "storagenode"}
2023-06-29T19:51:35.470Z ERROR bandwidth Could not rollup bandwidth usage {"process": "storagenode", "error": "sql: transaction has already been committed or rolled back"}
2023-06-29T19:51:35.470Z ERROR collector error during collecting pieces: {"process": "storagenode", "error": "context canceled"}
2023-06-29T19:51:35.471Z ERROR nodestats:cache Get pricing-model/join date failed {"process": "storagenode", "error": "context canceled"}
2023-06-29T19:51:35.471Z INFO trust Scheduling next refresh {"process": "storagenode", "after": "7h27m4.520456354s"}
2023-06-29T19:51:35.475Z ERROR piecestore:cache error getting current used space: {"process": "storagenode", "error": "filewalker: context canceled; filewalker: context canceled; filewalker: context canceled; filewalker: context canceled; filewalker: context canceled; filewalker: context canceled", "errorVerbose": "group:\n--- filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:69\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:74\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:718\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75\n--- filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:69\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:74\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:718\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75\n--- filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:69\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:74\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:718\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75\n--- filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:69\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:74\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:718\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75\n--- filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:69\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:74\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:718\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75\n--- filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:69\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:74\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:718\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-06-29 19:51:38,480 INFO waiting for storagenode, processes-exit-eventlistener to die
Error: piecestore monitor: disk space requirement not met
2023-06-29 19:51:38,931 INFO stopped: storagenode (exit status 1)
2023-06-29 19:51:38,935 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)
It says something about disk space, but I have allocated less than the disk has (i have put 1,8 TB in docker run, on a 2TB disk), and it wasnt a problem before.
This is because you removed all databases, and now the node cannot recognize its data.
Make sure that your disk is mounted to the same mountpoint:
df --si
If it’s - then you can temporary set the parameter storage2.monitor.minimum-disk-space to be equal to your free space either in the config.yaml file:
storage2.monitor.minimum-disk-space: 100.0GB
or as an argument after the image name: --storage2.monitor.minimum-disk-space=100.0GB in your docker run command.
When the filewalker will finish the scan process and recover content of some databases, you may comment out this parameter or remove from the docker run command.
The whole point of taking backup is to avoid starting from scratch. So, no, you won’t have to start from scratch. Make sure you backup identity and any personal data you may have on SD card then format the SD card. Install the operating system then docker and run the docker run command.
Also remember to check the path of data and identity folders before you execute docker run command after reinstalling OS.
You already did that by running fsck command and getting
The above result is why you are being suggested to format the SD card.
I would suggest to place the identity folder to the disk with data, the identity without its data is useless and data without its identity is useless too, so it advisable to store them together.
However, the backup copy of the identity could help, if it could be corrupted somehow.
After copy/move the identity to the disk with data you would need to update the path to the identity in your docker run command. I would suggest to save your docker run command as a shell script to the disk with data too. This way your node would be completely separated from the OS.
After you reflash your SD card, you would install docker and update /etc/fstab to automount your disk with data and execute the prepared shell script to run your node.
result is why you are being suggested to format the SD card.