Usedserialsdb error: database disk image is malformed

[MrYoshii]

I have got an internet outage… now im getting a lot of this errors…
need help.

2019-10-22T16:23:14.625Z e[31mERRORe[0m server gRPC stream error response {“error”: “serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:77\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:214\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:158\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696”}

[John.A]

This might help.

How to fix a "database disk image is malformed"

[MrYoshii]

i have tryed this but im stuck at nummer 9.

[John.A]

I have never done this myself and I dont know what to do. Someone with more knowledge Will Read this later on and help.

[MrYoshii]

i hope so

[MrYoshii]

Warning . If you were not successful with the fix of the database, then your node is lost. You should remove this node, all its data and identity, since it will be disqualified because of too many failed audits. In this case, you need to create a new node.

i see what it will be…

[BrightSilence]

Before giving up entirely on your node you can try renaming the usedserials.db file and let the node create a new one. Obviously stop the node before you do this.

[Alexey]

If you used the local installed sqlite, use a different path to store a dump file, for example /tmp instead of /storage.

[MrYoshii]

i have now renamed the usedserials.db and restarted the node.
i have again checked the .db files.

but now im getting a other error.

[eagleye]

any chance database reconstruction can be automated on startup to automatically check and repair databases before the node starts? My files are all intact so a node shouldn’t go down for a corrupt database.

Even better if there was a routine created to regenerate the database from the core data on the node drive. This could prevent a loss of terabytes of good data all because of a corrupt database.

[MrYoshii]

should i try to clear the node?

[BrightSilence]

if you clear the data, your node will get disqualified and you would have to start over with a new identity and token, so that’s the last resort.

I’ve uploaded a version of the db file with an empty version of the correct table inside it. I’m not sure what happens if you don’t have your previous records in there, but at least it can’t complain about the missing table anymore. Definitely keep your backup though in case someone else has a better idea.
https://alpha.transfer.sh/r3uqF/used_serial.db

[Alexey]

Please add your idea or vote for the existing one: https://ideas.storj.io
Also take a look on this topic:

Why did you choose to use a sqlite db?

Why did you choose to use a sqlite db? Since you ship the software with docker, you could use an internal mariadb database? Would need a little bit more RAM but would be a lot more performant I assume https://community.storj.io/channel/chat?msg=thi7A4evX8swtZ7ZW

and

Storage node performance improvement ideas

Do you have ideas how to improve the storage node performance but you don’t know how to address them? No problem. Post them here. We can disscuss them and I will create Jira tickets for it. Credits: The main performance problem is the chosen dbchema. At the last time i looked in the db i saw that near 30% of the data are used only by the id of one satellite which is written over and over in one table. The db structure need a bit of normalization, not to much or everything, because normalizat…

[MrYoshii]

thx for your db file this got it to work again

[john.danciu]

I have the same issue with used_serial.db malformed.
I checked it as per instructions but the result was “OK”. No error message.
Tried to replace it with the one suggested by BrightSilence and now I’m not getting the error message anymore but I also don’t get any PUT’s at all. Only GET’s and GET_AUDIT’s.
Any ideas why?

[MrYoshii]

i had to wait like 30min.
is it still not working?

[Alexey]

Hello @john.danciu,
Welcome to the forum!
Your node was offline, it must pass all pending audits

Please, update your Watchtower command: https://documentation.storj.io/setup/cli/software-updates#automatic-updates

[john.danciu]

Thanks guys, it’s working now. I get PUT’s again. However, this database corruption issue is a real problem. It happened 4 times for me. 3 times for info.db and now a first for used_serial.db. This is not a user’s fault as I didn’t do anything except stopping the node for a Win update. My sense is that the corruption happens during the “stop” process, even though I do have the 300s delay. And it looks like it is not just a Win issue but Linux as well. Any design plans for solving this, or we have to live with it for ever?

[Alexey]

When you want to stop your node, please use this command:

docker stop -t 300 storagenode

Or you can migrate to the Windows service: https://documentation.storj.io/setup/gui-windows

[eagleye]

is the windows service now reliable enough to switch to? I was going to but there was a warning about switching and possibly losing my node data and I should only try on a new node with fresh data. Will it migrate the data and fix the database?