Usedserialsdb error: database disk image is malformed

Yes it should be safe to use.
Auto update is implemented to.

Use the same identity and storage folder

it died again…

edit:
got it fixed but why is it allways happening?

Make sure you don’t have write caching in ram enabled on your system.

What is your OS?
How is your disk connected to the host?
What is filesystem?

i have unraid
my disks are connected via sata
XFS

how do i check if ram caching is on?

Unfortunately Unraid is unreliable platform for the storagenode.

  1. The storagenode uses a sqlite databases, and Unraid will damage the sqlite very often because of their bug:
  1. Unraid mount user’s filesystem after the docker daemon start
  2. The community app for storagenode v3 written by unraid community member uses the -v option instead of --mount and thus loses data every reboot because of p.2 and difference between -v and --mount options

You can’t fix the p.2, but can fix the p.1 as described on their forum, and p.3 if you would run the docker from the command line (bash, ssh+bash, etc) with a right command.

I would recommend to migrate from this unreliable platform to something more robust, even Windows working better.

Update: the latest version of Unraid platform has fixed the sqlite issue. However, it still mounts the user’s disks after the docker start, so the Unraid community app for Storj network is still dangerous, please, do not use it, use the CLI instruction instead: https://documentation.storj.io/setup/cli/storage-node

im not having issues with other docker containers its only storj.

You will have problems with any container which uses the sqlite.

I’m using unraid and had the same problem as you. Downgrading to unraid 6.6.7 solved the problem and Storj has been running stable for a month now. If you don’t want to downgrade, you can follow the patching progress here https://forums.unraid.net/bug-reports/prereleases/sqlite-data-corruption-testing-r664/page/4/?tab=comments#comment-6650. It even seems like in version 6.8.0-rc5 it’s completely fixed.

1 Like

I couldn’t find an exact windows command to rebuild the database using windows. I’ve got sqlite3 but can’t run cat and grep on my windows platform. Is there a conversion somewhere to complete the database rebuild command lines using windows or windows powershell? I discovered some errors in my database running windows. wanted to try and clean up the database.

If you’re talking about step 10 where it removes the lines with TRANSACTION and ROLLBACK. You can just use notepad++ to remove those lines. I’m pretty sure it’s the first and last line in the file. Use an editor that can handle large files though. Notepad would likely not like this file.

1 Like

I would recommend to use the docker-way, it work the same way as on Linux.
Another option - to use the WSL + Ubuntu from the Microsoft store.

If you want to do all with a native commands, then you can use this commands from the Powershell to edit the unloaded sql file (I assume that the path to the file is d:\storage\:

Get-Content d:\storage\dump_all.sql | sls -notmatch "TRANSACTION" | sls -notmatch "ROLLBACK" | Out-File d:\storage\dump_all_notrans.sql
1 Like

Ok,
I tried that line, but didn’t work with my windows version. So, I downloaded sqlite3 into the storage directory. backed up the .sql file, stopped storagenode then ran the SQLite routine. Took the output, edited it with notepad++ and just deleted those two lines at the top and bottom. Ran SQLite read and had to wait 2 hours for it to complete on a large 300MB file. started the node up and all seems to be working fine now. File deletions are happening a lot more often than before. my pieceinfo db shrunk from 300MB to 145MB. Plus one thing I noticed is that database didn’t show date access for over a month. Now it shows access regularly. thanks for all the help.

to know the operation is running you can watch the database file grow in sized as the .read command executes, so you don’t think it hung.

This took care of my file deletion failed and database malformed and also I got a hash order error database malformed as well before this fix.

3 Likes

Updated to 1.3.3, it seems to work but I noticed several:

2020-04-24T20:37:18.233+0200 ERROR piecestore download failed {"Piece ID": "KG2VHBDI4COKKEPOKYPHTTZ5VXNJ3KIEPHE3MG57ZJEJCALHCVXQ", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET", "error": "usedserialsdb error: database disk image is malformed", "errorVerbose": "usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:523\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Download:471\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:995\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:66\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}

1 Like

As described in the changelog. The error was there before. The difference is that you can now see the error message.

1 Like

How do I fix this? Congratulations on having a minimum of 20 characters to post.

Do you run a docker version?
If so, please, replace the latest tag back to the beta.
Also, fix the malformed databases:

Not using Docker.

Thanks, I will follow the instructions.