The script to automatically fix a malformed database

I have in the last few hours created a script which actually checks / fixes .db files which is from that help guide. ( i found this after i posted. ) :+1:

uploaded here happy for contributions to help solve db issues quicker.

#storj_useful_scripts/fix_malformed_database at main · storjdashboard/storj_useful_scripts · GitHub

2 Likes

Thank you for the contribution.
Unfortunately in my case the script did not help.
Here is an output:

> Do you acknowledge and accept the disclaimer? (yes/no): yes
> Enter the directory containing the SQLite database files: /home/sk/StorjTEST/BAD
> Enter the directory where you want to save backups: BAK
> Enter the directory where you want to save the fixed databases: GOOD
> Checking integrity of /home/sk/StorjTEST/BAD/bandwidth.db...
> The database /home/sk/StorjTEST/BAD/bandwidth.db is not malformed.
> Checking integrity of /home/sk/StorjTEST/BAD/garbage_collection_filewalker_progress.db...
> The database /home/sk/StorjTEST/BAD/garbage_collection_filewalker_progress.db is not malformed.
> Checking integrity of /home/sk/StorjTEST/BAD/heldamount.db...
> The database /home/sk/StorjTEST/BAD/heldamount.db is not malformed.
> Checking integrity of /home/sk/StorjTEST/BAD/info.db...
> The database /home/sk/StorjTEST/BAD/info.db is not malformed.
> Checking integrity of /home/sk/StorjTEST/BAD/notifications.db...
> The database /home/sk/StorjTEST/BAD/notifications.db is not malformed.
> Checking integrity of /home/sk/StorjTEST/BAD/orders.db...
> The database /home/sk/StorjTEST/BAD/orders.db is not malformed.
> Checking integrity of /home/sk/StorjTEST/BAD/piece_expiration.db...
> Integrity check failed for /home/sk/StorjTEST/BAD/piece_expiration.db. Attempting to fix...
> Backing up the original database...
> Exporting the database to SQL...
> Creating a new database from the SQL dump...
> Cleaning up...
> Checking integrity of the fixed database...
> The fixed database GOOD/piece_expiration.db passed the integrity check.
> Do you want to replace the original database with the fixed database? (yes/no): no
> Kept the fixed database in GOOD/piece_expiration.db.
> Checking integrity of /home/sk/StorjTEST/BAD/pieceinfo.db...
> The database /home/sk/StorjTEST/BAD/pieceinfo.db is not malformed.
> Checking integrity of /home/sk/StorjTEST/BAD/piece_spaced_used.db...
> The database /home/sk/StorjTEST/BAD/piece_spaced_used.db is not malformed.
> Checking integrity of /home/sk/StorjTEST/BAD/pricing.db...
> The database /home/sk/StorjTEST/BAD/pricing.db is not malformed.
> Checking integrity of /home/sk/StorjTEST/BAD/reputation.db...
> The database /home/sk/StorjTEST/BAD/reputation.db is not malformed.
> Checking integrity of /home/sk/StorjTEST/BAD/satellites.db...
> The database /home/sk/StorjTEST/BAD/satellites.db is not malformed.
> Checking integrity of /home/sk/StorjTEST/BAD/secret.db...
> The database /home/sk/StorjTEST/BAD/secret.db is not malformed.
> Checking integrity of /home/sk/StorjTEST/BAD/storage_usage.db...
> The database /home/sk/StorjTEST/BAD/storage_usage.db is not malformed.
> Checking integrity of /home/sk/StorjTEST/BAD/used_serial.db...
> The database /home/sk/StorjTEST/BAD/used_serial.db is not malformed.
> Checking integrity of /home/sk/StorjTEST/BAD/used_space_per_prefix.db...
> The database /home/sk/StorjTEST/BAD/used_space_per_prefix.db is not malformed.
> All databases have been processed.

However, the file piece_expiration.db turned out to be zero length :unamused:
I could attach the corrupted database file if you are interested in debugging the script.

This is one of the reason why we never tried to implement an automatic fixer. Too many different cases.
Are you able to fix it manually?

If you have recently had issue with ‘piece_expiration.db’ during a DB migration from the storj node update.

This is an issue reported in this thread. V58 db migrate fail - piece_expiration.db - 1.108.3

Is the BAD piece-expiration file , 0kb? or the new one it created 0kb?

Cheers,

The new one was created 0b

1 Like

No issues with read/write to the directory ?

Has doing it manually fixed it ?

No, I was upset by the failure and simply deleted all the databases.
But I still have a backup, I’ll try to do a manual fix for fun.
Is it possible to replace a new database file (created from scratch) with a fixed database later?

1 Like

@Kopcap

Yes

I have just made an update to the file which checks for read/write before proceeding.

I have also made it so it does check file sizes and copies rather than moves any files so that you will always have the backups

1 Like

Updated version created zero file, too :disappointed:
Will try manual fix

Can you upload it here at all? - not sure if we can - I will try to form a fix

I fixed the corrupted database manually.
Thank you for detailed instructions! :pray:

1 Like

I can’t find the option to upload files here.
I will send you a link to the file in a private message

Glad you managed to fix it… - Yes if you can do that - I can then debug this DB … to see what was causing it to not work

These instructions for docker worked well for me and fixed my broken storage_usage.db - even tho I had no idea what I was doing :slight_smile: Thx a lot.

3 Likes

Nice ! :slightly_smiling_face:

Thanks for the feedback

1 Like