File delete failed, malformed

Node Operators troubleshooting
1

i’m getting errors that file delete failed yet the file was just downloaded successfully. Database malformed. I’ve gone through the database check and it says OK. Don’t know what else to do. Also, my dashboard shows 280GB of space used, yet my drive has over 1.6TB of data stored. How can that get resolved?

Can there be a clarification on how to rebuild the database. I think my data is secure and I didn’t have any of these problems until the watchtower issue and the most recent update release 22.1.
I see a lot of downloads successful so I don’t know if its accessing the 280GB of seen data or if its also getting the 1.6TB of “missing” data. Most file downloads and deletions are successful and some are not. IN the scripts run on the logs it says all audits that failed are recoverable and no audits were unrecoverable.

How about program or routine to rebuild the database from the actual data stored on the drives and not relying on a possible corrupt info.db file?

2

There are now multiple databases. Since 0.22, the info.db has only a single table called “versions”.

Did you check the rest of databases?

bandwidth.db
info.db
orders.db
piece_expiration.db
pieceinfo.db
piece_spaced_used.db
reputation.db
satellites.db
storage_usage.db
used_serial.db
3

So,

I have to individually backup each database and run a check on each one?

Why doesn’t the storj module do this to automatically check the integrity of each database file before starting up the node program? Then it could either repair the database errors or abnormally terminate with an error until the database was repaired.

A lot of deletions happening right now. What happens or is reported when a file deletion fails? Does that affect node reputation?

4

I don’t have an immediate answer for all of the questions here, but I can say that no, a failure to delete a piece will not impact node reputation. At worst, it will mean your nose is storing more data than it needs to, until we enable garbage collection. The garbage collection functionality exists, but we are working on some extra safety mechanisms because we are very worried about something going wrong in garbage collection causing many pieces to be deleted that shouldn’t be.

I don’t think there is any particular reason why the Storj software can’t recognize db corruption and rebuild databases, other than that we would prefer to make it so db corruption is very rare and automated rebuild would be unnecessary. To get there, it looks like we need to detect when users have placed the database files on a volume that doesn’t support safe SQLite operation.

5

Ok,

Thanks. I’ll just leave it alone and ignore the errors. I think my data is all intact and could just be an error with a database file. Which one, I don’t know.

I think a solution is to inspect the databases upon start of the node. Then repair what is needed in the database. yes, it could be a long process but it should fix the error and make for a more stable node operation. I’ve never had a database problem until this last update of 22.1 and watchtower issue.