So i have been running this node for a while now (year +), on an unRAID Docker with no issues till randomly this error came up:
ERROR failure during run {“Process”: “storagenode”, “error”: “Error opening revocation database: revocation database: boltdb: input/output error\n\tstorj.io/storj/private/kvstore/boltdb.New:46\n\tstorj.io/storj/private/revocation.openDBBolt:52\n\tstorj.io/storj/private/revocation.OpenDB:35\n\tstorj.io/storj/private/revocation.OpenDBFromCfg:23\n\tmain.cmdRun:76\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:272”, “errorVerbose”: “Error opening revocation database: revocation database: boltdb: input/output error\n\tstorj.io/storj/private/kvstore/boltdb.New:46\n\tstorj.io/storj/private/revocation.openDBBolt:52\n\tstorj.io/storj/private/revocation.OpenDB:35\n\tstorj.io/storj/private/revocation.OpenDBFromCfg:23\n\tmain.cmdRun:76\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:272\n\tmain.cmdRun:78\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:272”}
Error: Error opening revocation database: revocation database: boltdb: input/output error
I have tried renaming the “revocation” DB as other posts have suggested, and it changes nothing! A new revocation db is created, and the same error loops in the logs! Same with the rest of the DB files (in the default location).
Rebooted machine a few times. Checked the drive; no write/read issues; permission etc match another, working, node on another machine (at another location).
I don’t want to lose this 6tb (and growing) node! HELP!
All the .db files in the standard node are for local stats: so they don’t have anything to do with your payouts. If you were going to move them all out of the way (so the node recreates all of them) now would be a great time… since fresh monthly stats start May 1st.
Just to be super-certain: you moved/renamed all the .db files at once (not trying one-at-a-time between restart attempts). Like the node was down… you moved/renamed so you had zero .db files… and it still complained about “Error opening revocation database” on next startup? (and you would have seen it recreate all the .db files?)
It’s like it’s touching that new file to create it… but can’t open it… so it gives up (and doesn’t proceed to create the other .db files).
Since the node is down anyways, can you unmount and “fsck -y”/scrub the filesystem the .db files are on? If there are no issues it should be quick.
Wait: revocations.db is the one that doesn’t go in the alternate “storage2.database-dir” directory: it’s in the same place as config.yaml? I’ll have to check a local node…
I’m stumped. Is the file getting recreated with the same user:group ownership your docker command is looking for? And it should be “-rw-------” permissions? I’ve seen other posts where boltdb messages mentioned issues with that file… but it specifically said it was read-only… and you’re not seeing those extra read-only logs.
So it can make the file, correct permissions, correct ownership: I don’t know what else could cause an “error opening”.
Edit: Were there any .db-shm/.db-wal left over, like from an unclean shutdown? Or was your alt storage2.database-dir completely empty and it still didn’t make new files there?
Completely empty. I deleted everything in the DB-dir…probably should have backed up the DB files, but got mad and wiped the folder instead.
2025-04-29T21:26:11-05:00 INFO db database does not exist {“Process”: “storagenode”, “database”: “info”}
2025-04-29T21:26:11-05:00 INFO db database does not exist {“Process”: “storagenode”, “database”: “bandwidth”}
2025-04-29T21:26:11-05:00 INFO db database does not exist {“Process”: “storagenode”, “database”: “orders”}
2025-04-29T21:26:11-05:00 INFO db database does not exist {“Process”: “storagenode”, “database”: “piece_expiration”}
2025-04-29T21:26:11-05:00 INFO db database does not exist {“Process”: “storagenode”, “database”: “pieceinfo”}
2025-04-29T21:26:11-05:00 INFO db database does not exist {“Process”: “storagenode”, “database”: “piece_spaced_used”}
2025-04-29T21:26:11-05:00 INFO db database does not exist {“Process”: “storagenode”, “database”: “reputation”}
2025-04-29T21:26:11-05:00 INFO db database does not exist {“Process”: “storagenode”, “database”: “storage_usage”}
2025-04-29T21:26:11-05:00 INFO db database does not exist {“Process”: “storagenode”, “database”: “used_serial”}
2025-04-29T21:26:11-05:00 INFO db database does not exist {“Process”: “storagenode”, “database”: “satellites”}
2025-04-29T21:26:11-05:00 INFO db database does not exist {“Process”: “storagenode”, “database”: “notifications”}
2025-04-29T21:26:11-05:00 INFO db database does not exist {“Process”: “storagenode”, “database”: “heldamount”}
2025-04-29T21:26:11-05:00 INFO db database does not exist {“Process”: “storagenode”, “database”: “pricing”}
2025-04-29T21:26:11-05:00 INFO db database does not exist {“Process”: “storagenode”, “database”: “secret”}
2025-04-29T21:26:11-05:00 INFO db database does not exist {“Process”: “storagenode”, “database”: “garbage_collection_filewalker_progress”}
2025-04-29T21:26:11-05:00 INFO db database does not exist {“Process”: “storagenode”, “database”: “used_space_per_prefix”}
But nothing is created, as i am used to. I have had malformed DB’s before, so i am accustom to fixing the DB files; and the reason i moved them to an SSD and that has made my node much more stable…till now! I have also changed back to the default DB setup, and they are still not being created.
This database is located in a different place than others, I believe that you can remove it without touch any other and restart the node.
However I/o errors usually suggest that you need to run a fsck command, since you ara using unRAID, I would guess that there is xfs, so you need to run xfs_repair instead.
That doesn’t look right: it shouldn’t be root. The docker run command by default has a “–user” flag that should sub in the non-root user (or it’s in docker-compose.yml, if you use that).
Did you maybe initially set up your node as a non-root user… but are coming back to repair things as root? (If you look in your files/identity directory, and do a “ls -l”, who are the files there owned by?)
This FS is BTRFS, but i will have to wait till the weekend to run an fsck. It has to be in maintenance mode to run those, and this is in a production environment. I don’t have any issues creating new and reading files, but it is possible that the FS is messed up.
I have done this a number of times, and get the same error, as i said in earlier posts, i even tried to remove the DB’s in the normal area, and they are not being re-created. I am guessing storj looks at the revocation DB first, and since it cant satisfy that, its not even trying to look for the rest of the DB’s (although the log says they are missing if they are not in the folder there).
I actually made it root to see if that would help any. Root is the master of all; my other node is set up as root, so i was just copying that to see if it would resolve the issue. It was created as root user…but it doesn’t matter what ownership its given, the node is not working!
You may try to run a SETUP command with a wrong data location, then take this database from there and put into an actual data location, it should be near config.yaml.
