Just to keep this up-to-date, now the startup-used-space-filewalker, aka piece-scan-on-startup, scans each satellite’s data folder one by one, meaning it first scans for US1, then EU1, etc. I don’t know the exact order, so don’t quote me on this. You shoud see in the log:
piecescan started for sat ID xxxx,
piecescan finished for sat ID xxxx,
piecescan started for sat ID yyyy,
piecescan finished for sat ID yyyy, etc.
Second, there is a new system to speed up the second scan; it’s called badger db or badger cache, and it’s a database for matadata cache. The first scan takes the usual time and it caches the matadata, but the second scan will start first by reading the badger db, than continue scanning the rest of the files if the first scan was interrupted.
So it acts like a cache and a progress saver. This cache speeds the piecescan exponentialy, but it works best if it is stored on another drive, a SSD or NVMe.
And is still in a beta, so hickups might crash your node from time to time. That’s why is not enabled by default. Search the forum for badger cache. You should turn off the lazzy mode if you enable badger.
Some new stats for FW run, node ver. 1.114.6.
Synology 18GB RAM, 2 nodes runing, FW only on one, ingress and egress active, lazzy off, no badger cache, db-es on SSD stick. On 5 machines in different subnets, these are the stats:
40H for 7.5TB
68H for 10TB
66H for 10.6TB
58H for 10.7TB
50H for 11TB
Don’t ask me why they look like this. Maybe some caught a trash cleanup, or some BF during the run. I deleted the logs, so can’t check the cause.
10GB RAM machine, 1 node:
FW run: 74h
pieces: 35627929
size: 7.6TB
Why do not use a badger cache if the lazy mode is off anyway? It speedups a FW in several times of magnitude.
Because it can create problems… get corrupted, etc., as we saw from other members, and crash the node. No thanks!
Yes, it could. However, this may happen if the node was restarted abruptly and it was not able to properly close the badger cache database. The fix is easy - just remove it from the disk and restart.
As far as I understand it is still in development and not endorsed by the company.
Has that changed? Is it now ready for “Prime Time”?
I’d rather have a slower node that works rather than a faster one that crashes.
Yes, it’s experimental. However, it’s working pretty fine. Yes, it may be corrupted on unclean shutdown or reboot. The workaround just remove this cache and restart the node. Yes, the first run will be as slow as without it, but all next - much faster. Why do not use it right now?
I enabled it for my node more than a month ago, no issues so far (I do not touch my nodes, so they reboot, when Windows would decide).
I dont feel like it is wise to recommend to everyone to enable experimental features. Yes, it may have been working fine (including for myself), but as long as its marked as experimental, it is not meant for everyone to use.
If Storj believes this feature is already mature enough for everyone to use, then it should not be classified as experimental anymore. There is no point in releasing a feature and leaving it as experimental forever (Im not saying this is the case, just that if the feature is still experimental it is because of a reason), at some point it should be reclassified as a normal feature.
Yes, I understand. However, if your setup is suffering out of not enough IOPS, maybe it’s worth it?
P.S. When we would get enough feedback, we would remove the “experimental” disclaimer.
But I think, it wouldn’t be enabled by default anyway. We have so much variants how the FS may be implemented, so the badger cache wouldn’t be an universal answer for any cases. I’m sure, that ZFS with 256GB of available RAM and SSD as a cache layer would over perform the badger cache. Interesting to see though, what would happen if we enable the badger cache for such kinds of setups?