I don’t know if the title is right, but here’s what I expirienced…
Running 2 nodes on Ubuntu Server, N100, 32GB RAM, Exos 22TB drives, databases and OS on NVMe, nodes on ver 111.
I restarted node1 with startup-piece-scan enabled, lazzy off, badger off, just to correct the usage, because I had a huge used space discrepancy. (Usualy it’s disabled)
The node stores like 5TB of data and the scan completed for all 4 sats in 5 hours.
After like 5 minutes of the last sat scan completion (from log entries), I checked the dashboard and the correct used space was displayed (5TB).
I stopped the container, removed the container and started the container with startup scan disabled.
After like 10 min, I checked the dashboard again and surprise… the used space is again wrong like before the scan, showing 14TB.
How this happened?
Why before restart the dashboard showed updated numbers, and after restart, it forgot all the data from the scan?
I can’t imagine what is the problem; only scenary I can think of is that in those 5 minutes after the last scan, the recordings were stored in a db cache, which upon stop and rm, didn’t got flashed to the database, and got lost… can someone explain it?
Should I wait an hour or so after the scan complition before stopping the node?
Or should I wait a few minutes between stop and remove the container?
Can you just leave used-space-filewalker enabled (like it is out-of-the-box)?
Even if it runs after upgrades every couple weeks… you wouldn’t notice it being busy for 5h… but you would notice your stats remaining correct.
I’ve seen the same thing on my nodes. The used space database does not seem to get updated with the current numbers when the node is shutdown, at least not in docker, and I use the stop -t 300 to give it plenty of time. Seems to be a bug.
I think that would be your best bet. Also I think you can check the timestamp on the piece_spaced_used.db file to see when it last saved.
It’s perhaps a bug, but it’s because the db cache is flushed every hour. So yes, you need to wait a hour before shutdown. The node also contact satellites once a hour. Or you need to reduce the flush interval with the option:
--storage2.cache-sync-interval duration how often the space used cache is synced to persistent storage (default 1h0m0s)
Seems you need to use this option with enabling scan altogether. Perhaps as a command line arguments after the image name would be a best approach, if you usually keep the scan disabled in your config.yaml.
OK, 1 hour waiting… But when I stop with the recommended -t 300 (which I always do), isn’t suppose to flash the cache to the permanent db?
Second… is there any risk putting at 30 min or 15 min sync interval?
Third… if with each stop (system updates, node updates, maintenance, power outages with UPS triggered stop), you loose between 1 s and 1 hour of recordings, it can create a significant discrepancy by itself, without other bugs in storagenode software.
And this happens on a verry powerful system for storagenode purposes; I wonder what happens on weaker ones.
I don’t understant how cache works and why is different from a db, but why recording things in 2 files instead of one? If the cache is like a db, why not recording directly in the main db?
How the command should look like in docker run? Is this OK?
--name storagenode storjlabs/storagenode:latest \
--storage2.cache-sync-interval=15m0s \
performance I think?
Yep, but you have a scan enabled by default, right? If there are a lot of load, ok, the discrepancy could be noted (if the last state has been applied to the database though). By the way, are databases healthy?
to batch some IOPS to a shorter time I guess?
Looks good to me. Except the last \, but I guess you have other options there?
Yes the databases are healty. Did the pragma check and vaccum on them.
I did some testing; after setting the sync inteval at 15 min (in run command or config - both variants work; tested), I see there are 3 files that are updated on regular itervals:
piece_spaced_used.db - updated at 30min;
piece_spaced_used.db-shm - updated at 15min;
piece_spaced_used.db-wal - updated at 15min.
When running the node, all db-es have those shm and wal brothers spawned at start, and after some time, just some.
When stopping the node (Docker - docker stop -t 300 storagenode), some of the shm and wal files disappear, sometimes all.
Is this normal? Aren’t those temp files, that must be ALL synced to main db and removed at stop?
These are the files left after a node stop command…
As you can see, the piece_spaced_used files are all there after stop, no sync>delete, and I can confirm that the shm and wal files waren’t modified at node stop, but after restart.
Is the -t 300 bypassed by Docker? Should there be -t 300s like in the watcher command?
Also, running the docker restart -t 300 storagenode abruptely restarts the node, without any delay. The stop has some delay, of few seconds at best.
These screenshot is taken before setting the 15min sync. You can see that the piece_spaced_used.db lost 4 minutes of records.
I always use stop -t 300 and the node has free space, so it continuously stores new pieces or deletes some, so if I understand it correctly, the db should be updated at stop with the new info.
They should all disappear. Is it possible that the 300s timeout is not enough?
But is stops in less than 5 min, in just a few seconds…