Docker Node Keeps Restarting

Node Operators troubleshooting
1

My little node was doing good, then I went to check today and the dashboard would not load so I remoted in and saw it was in restarting status, checked the logs and this is what showed up, not sure how to fix this as I am still learning storj node things

Error: Error starting master database on storagenode: database: unable to open database file: no such file or directory
storj[.]io/storj/storagenode/storagenodedb.(*DB).openDatabase:323
storj[.]io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:305
storj[.]io/storj/storagenode/storagenodedb.(*DB).openDatabases:281
storj[.]io/storj/storagenode/storagenodedb.OpenExisting:248
main.cmdRun:160
storj[.]io/private/process.cleanup.func1.4:363
storj[.]io/private/process.cleanup.func1:381
github[.]com/spf13/cobra.(*Command).execute:852
github[.]com/spf13/cobra.(*Command).ExecuteC:960
github[.]com/spf13/cobra.(*Command).Execute:897
storj[.]io/private/process.ExecWithCustomConfig:88
storj[.]io/private/process.ExecCustomDebug:70
main.main:385
runtime.main:204

2

Seems your storage location have lost files or permissions.
Please, stop the storagenode and check your disk for errors first.
Please also check the path to the data location, maybe your mount point (disk letter) has changed and you should update your docker run command (to apply the changes you need to stop and remove the container, then run it again with all your parameters include changed ones).
Please show result of the command:

docker inspect storagenode --format "{{.Mounts}}"

And result of this command (replace /path/to/data/ to your actual path to data) (bash/PowerShell):

ls /path/to/data/*.db

If you are on Linux, please show result of the command:

df -HT

if you are on Windows, please show result of the command (PowerShell):

Get-WmiObject Win32_LogicalDisk
3

first command
[{bind /root/.local/share/storj/identity/storagenode /app/identity true rprivate} {bind /mnt/storj/node /app/config true rprivate}]

second command
/mnt/storj/node/storage/bandwidth.db /mnt/storj/node/storage/orders.db /mnt/storj/node/storage/pricing.db /mnt/storj/node/storage/storage_usage.db
/mnt/storj/node/storage/heldamount.db /mnt/storj/node/storage/piece_expiration.db /mnt/storj/node/storage/reputation.db /mnt/storj/node/storage/used_serial.db
/mnt/storj/node/storage/info.db /mnt/storj/node/storage/piece_spaced_used.db /mnt/storj/node/storage/satellites.db
/mnt/storj/node/storage/notifications.db /mnt/storj/node/storage/pieceinfo.db /mnt/storj/node/storage/secret.db

third command
/dev/sda1 ext4 5.0T 36G 4.7T 1% /mnt/storj

4

UUID has not changed, i could switch it in fstab from UUID to just /dev/sda1

if I issue ls a bunch, randomly I will get

ls: cannot access ‘/mnt/storj/node/storage/pieceinfo.db-shm’: No such file or directory

then if I run ls again its fine

a drive check is underway and will update this post with the results

5

This is not good. Seems your drive disappears sometimes or stop responding.
This file is a database log, if it’s missed when node is working, then something wrong with the disk or something deletes these files from the storage.
I would like to suggest you to check this disk for errors, check its cables and if it’s an external disk, then check the external case and power supply. Either of them is not reliable enough. It could lead to quick disqualification, so the storagenode is trying to protect you and shutdown itself when the disk is disappearing.

By the way do you have a second node on this device?

6

so I had to start over, bad partition, I wiped the drive and let it stay connected to the system for the past 48 hours and ran fsck once per day and as of today it still shows clean, hoping that fixes it otherwise a new drive it will be, current drive is only about 2 months old as I went to see if I could replace it

7

bad drive, gonna order a new one, as soon as I created the folder after mount the setup command froze trying to bind