They will be caught in garbage collection. But you won’t see individual log lines for that I believe. So yes, they will be cleaned up.
The error does suggest an IO bottleneck though. So it would be good to look into what could be causing that. Could be USB2 connection, remote storage or SMR drive for example.
that’s a database is locked error…
usually means something with user rights on the data, or that they where briefly inaccessible…
maybe your files are read only? i think might be a good guess… if not for you, then for the user that runs the storagenode, if you ain’t running default.
it could be, certainly some sort of issue you need to deal with… i would start by verifying the things that are easy to deal with, like user rights, and deletion permissions on the files…
making your node faster / higher io is a bit more of a troublesome issue to deal with…
high io seems unlikely imo, i mean then you should get the same on uploads and downloads… deletions should be pretty low demand compared to download and upload, ofc they might have different priority…
also i would bet that deletions are not live, they go to the disk cache and then is done when it has time… which i suppose would sort of suggest you could be right… but on 4 drives in raid 5… then people with 1 drive would be ****ed.
you should have a good deal more IO that they have, and then there is the whole raid, depending on what you are using for the raid, something of that would most likely have a cache that would just confirm a deletion and deal with it later…
you should have like 800 iops write or something along those line before your array would even start to give you high latency… and i’ve run with nearly seconds of backlogs on my drives at times…
without any issues… i’m on basically a raid 5 with 5 drives just using ZFS
All databases should be vacuumed once per month or so… more often with more traffic.
If you are running Docker on GNU/Linux, this can be automated quite easily using cron and some simple BASH scripts. Here’s a simple script that should check vacuum all the databases.
dbs=$(ls ./storage/*.db)
c="VACUUM;"
for i in $dbs
do
sqlite3 $i "$c"
done
Is that reflected / recommended anywhere in storj documentation?
The above script will corrupt the database if I’m not mistaken?
Once, the database is opened for writes by storj process in docker. Then, we write to the database from a separate sqlite process. Sqlite does not support that - it can’t end well?
You can also add a "PRAGMA integrity_check;" command loop to ensure that the operation was successful and the databases are still intact.
Vacuuming a database should not corrupt it… the operation purges the db file of old and unused data that may have been kept in the db as a result of numerous row modification operations.
For those who may not understand what’s going on and why this VACUUM procedure may be necessary, you can read about it on sqlite’s documentation page.
i wouldn’t do any changes to my database without atleast getting a “should be fine” from @Alexey
the database is pretty critical infrastructure of your node and should not be tinkered with lightly…
not that i’m saying it cannot be done, or that beast isn’t right… just saying i would take a bit of convincing to go in and manually change my storagenode database…
but i’m sure @anon27637763 knows what he is talking about, else i would hope he wouldn’t suggest the operation…
They should do something about the logs as well, integrate log rotation or something.
Also why email notifications should be re-enabled and accurate.
If our target node isn’t a sysadmin monitoring a datacenter of servers, we need to do as much as possible to help people know when there’s a problem and how to rectify it.
I am no Alexey, but vacuuming a db while the node is stopped is fine. But I would only do it if the db is causing issues for you. Ideally I would like to see vacuum becoming a part of the database migration steps with each new node version. End users shouldn’t need to worry about it at all. I have seen no significant difference in size and performance after vacuuming. But then again, I’m testing on an SSD accelerated array, which may smooth over any issues to begin with.
My orders.db typically shrinks significantly after vacuuming:
Using my posted script in the other similar thread along with an added ls command to check size.
storagenode
-rw-r--r-- 1 root root 162738176 May 4 14:08 /opt/storj/storage/orders.db
ok
ok
ok
ok
ok
ok
ok
ok
ok
ok
ok
ok
ok
-rw-r--r-- 1 root root 162443264 May 4 14:08 /opt/storj/storage/orders.db
storagenode
Difference:
162738176-162443264 = 294912
That’s 288 KB recovered after 6 hours of run time.
288KB of a 162MB file is the opposite of “significant” especially factoring in a 6-hour run time during which your node is offline. It’s a reduction in the total file size by only 0.1%! That doesn’t seem worth the trouble.
