Need help. Error starting master database on storagenode

Strange. Maybe the chkdsk did make some change.

It’s your choice. I keep my logs so I can trace any problem back to when it might first have started, but for most node operators they regularly delete them or change the log level to ERROR instead of INFO. You could also revert back to not using a file.

2 Likes

Never mind. It just crashed once again - i really need some help on this one… Its just weird and i dont have the skill to fix it.

The logs:
https://pastebin.mozilla.org/y0XvARYr

It seems that this database is the one thats messed up:

2023-02-24T10:20:13.968Z FATAL Unrecoverable error {Process: storagenode, error: Error starting master database on storagenode: database: reputation opening file \config/storage/reputation.db\ failed: unable to open database file: no such file or directory\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:331\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:308\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:283\n\tstorj.io/storj/storagenode/storagenodedb.OpenExisting:250\n\tmain.cmdRun:193\n\tstorj.io/private/process.cleanup.func1.4:377\n\tstorj.io/private/process.cleanup.func1:395\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:92\n\tmain.main:478\n\truntime.main:250, errorVerbose: Error starting master database on storagenode: database: reputation opening file \config/storage/reputation.db\ failed: unable to open database file: no such file or directory\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:331\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:308\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:283\n\tstorj.io/storj/storagenode/storagenodedb.OpenExisting:250\n\tmain.cmdRun:193\n\tstorj.io/private/process.cleanup.func1.4:377\n\tstorj.io/private/process.cleanup.func1:395\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:92\n\tmain.main:478\n\truntime.main:250\n\tmain.cmdRun:195\n\tstorj.io/private/process.cleanup.func1.4:377\n\tstorj.io/private/process.cleanup.func1:395\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:92\n\tmain.main:478\n\truntime.main:250}
2023-02-24T10:20:16.416Z INFO Configuration loaded {Process: storagenode, Location: /app/config/config.yaml}

I understand that i can recreate the database but then i need to know what DB’s to delete right?

Each time you post an error it references a different database. If you haven’t yet moved all the databases and then tried to start the node, you should.

Okay- i will make a backup and move them out.

I tried moving away all .db files and still the problem is there:

https://pastebin.mozilla.org/hSRecsxa

Is anynoe able to help me fix this issue? when i boot the node it just keep restarting with these errors as posted in the pastebins. I have quite the downtime now so any ideas whould be very nice…

Maybe you can try creating new db’s from scratch so when the node starts up it sees them

I think i did that? moving out the DB files and launching the node will result in a new db? that did not work… :frowning:

I intend to create new ones you, empty, and put them in the right folder, don’t have node recreate them

:confused: sorry can you explain further? i dont quite understand? :confused:

The only way to do that is for the node to recreate them. Manually creating sqlite databases is asking for more trouble.

2 Likes

Hey Alexey - do you have any ideas on how i can troubleshoot?

I am at the point of thinking i might need to give up my node… I have not been able to fix it.

It seems to me that you can stay up to 12 days offline, wait for someone more competent to answer

1 Like

Can you describe these facts in more detail?

I have used these lines every time I have “rebuilt” my node (with existing identity), certainly more than 30 times since May 2021, also for testing scripts and such and I don’t understand what makes this command so bad.

The process has always been:

  1. stop storagenode
  2. remove storagenode
  3. these Command line
  4. docker run …

The error you have is saying the database files aren’t there and can’t be loaded. At least that’s the most recent error I can read. Your pastebins are deleted so i can’t see what they contain.

You might consider just copying the identity to a backup location and keep the data drive/folder and then do a fresh install of the software itself.

if your drive would be unmounted, or automounted to another folder (or another letter in case of Windows), or you messed it up with another node, this command will re-create a full structure and node will start with empty storage, as result it will be quickly disqualified for losing data.

Without setup command the node will not start in this case, because it cannot find needed structure and will crash, this will prevent it from disqualification.

So again - the setup command must be run only once for each new identity and must be never used again.

I do not see any logs. Could you please copy the last 20 lines of your logs here between two new lines with three backticks, like this:

```
logs here
```

Please also describe what you already did.
Right now it looks like a problem with disk itself - seems it disconnects from time to time especially under load. Is it a USB disk? If so, do you have an additional power supply for it?