So i got a very concerning issue, which might already be known, im not sure:
The used_serial.db gets “locked” from time to time it seems on my node, the log is then filled with errors about that when a download (upload from my pov) finishes.
The interesting thing is that the db file itself seems to be alright, i ran the PRAGMA integrity_check; comand and it returned “ok”. I tried to restart the nodes a few times so it would “fix itself”, but it didnt.
I then tried vacuuming it via VACUUM; and after that it was working again.
No idea how that alone can fix the issue (or rather postpone it).
Vacuum removes all the old data from the database. It’s a necessary function when running any database. When a database has many writes, updates, and other modifications, the file/memory used can increase dramatically.
On my node, I let the database sit for a month and then vacuum… the DB size generally shrinks by an order of magnitude. Start/stop time as well as general performance slows/worsens towards the end of each month and makes a nice jump after vacuuming. This is to be expected since the DB is updated every few seconds 24/7 …
There should be a nice little button on the dashboard or some easily reachable control for “clean up database of old data”…
For the usual crowd: The above comment is NOT A FEATURE REQUEST … it’s just a comment about something that would at some time in the far future be a nice thing to have.
I get what vacuum does, but what i don’t get is why the storagenode throws locked errors, it shouldnt. even with a bigger db it shouldnt happen unless the storagenode doesnt handle concurrency correctly and just expects the file access to be so fast that concurrency never happens. That would be a major mistake tho
The locked errors are due to the slowness of the database. There are numerous simultaneous connections to the DB… but it can only be written to by a single process at a time. When the DB gets too large, write time also increases… and thus, the database is “locked” by a process which is slowly writing…
That’s what the “lock” is for… The speed of writes to the DB is a factor in overall node performance. The faster your system can perform writes, the more data pieces your can process.
There’s nothing to fix in regard to the “locked” DB error except vacuuming occasionally.
One could make an argument that sqlite is not the appropriate DB choice for the application… but that’s a different thread — and quite controversial… as are so many things. Alas! for the era of too many choices.
the lock error happens when another file write is currently happening, so yes, there is something to fix: implement a queue so writes dont happen at the same time but sequentially
It should be obvious that if a predetermined wait time is introduced than each node will slow down.
The database is already performing the role of gatekeeper through file locking. If a node’s hardware isn’t able to keep up, then that node will not survive… and that’s fine.
Supply and demand will determine both the price charged by Storj for customers and the hardware of most nodes.
If customers are willing to pay for a network that runs on ever cheaper SSDs, then that’s what will be required to run a node.
I hadn’t looked at SSD prices all 2019… and then I took a peek recently… Since Aug 2019, I’ve earned almost enough STORJ to buy my entire allocated space in new SSDs.
You mean to the database? Not necessarily. I just happen to see the same error when I was copying a large file to the disk where Storj data reside. It had nothing to do with Storj data though. But it seemed that it slowed my drive down so much that database could not keep up and has resulted in these errors.
Yes, the database. As said before, sqlite does NOT support parallel writes and storj does exactly that currently: it writes to the db whenever it feels like it, there is no queue to make writes run in sequential order. I created a github issue for this and it is a known bug: https://github.com/storj/storj/issues/3871
That’s why I am trying to tell you that in my case it was not matter of the database. It was a matter of overall disk acticity not related to the node.
It is still the same thing, when you slow down the disk by copying files or whatever you also slow down the writes to the db. This in turn increases the chance for two or more writes to the db happening at the same time because each write takes longer to complete