In my case, it’s pristine!
No, more simple: I was sleeping and had to restart manually next day.
1 Like
Nah, I have a UPS for all devices.
Besides, there is also no apparent relation with total sizes of the databases:
root@VM-HOST:/var/lib/lxc# cd /var/lib/lxc; for i in STORJ[1-9]*; do echo $'\n\n'$i; lxc-attach $i -- ls -lSh /storj/DBs | head -n 1; lxc-attach $i -- docker logs storagenode --since 24h 2> /dev/null | grep locked | wc -l; lxc-attach $i -- docker logs storagenode --since 24h 2> /dev/null | grep 'uploaded' | wc -l; lxc-attach $i -- docker logs storagenode --since 24h 2> /dev/null | grep 'downloaded' | wc -l; lxc-attach $i -- docker logs storagenode --since 24h 2> /dev/null | grep 'started' | wc -l; done
STORJ10
totaal 830M
4
24
35567
35905
STORJ11
totaal 396M
3
4817
3075
8037
STORJ16
totaal 67M
3
24
8724
8773
STORJ17
totaal 56M
5
24
6127
6177
STORJ18
totaal 375M
5
50708
3491
68069
STORJ22
totaal 308M
0
24
2857
2908
STORJ23
totaal 1,6G
0
24
3713
3760
STORJ4
totaal 244M
0
21808
4525
60695
STORJ6
totaal 752M
0
24
3724
3899
There also seem to be no relation between traffic, data drive and so on…
Numbers are pertaining last 24h: amount of database locks, amount of finished uploads, amount of finished downloads, and amount of started downloads/uploads. These numbers also tell, that the data fish probably isn’t all to much involved in this.
Well… the Salt Lake one just finally finished after 56 hours, and… no change to the dashbaord. And no database locked messages to explain it.
Sorry, what is pristine?
no, I mean these log lines:
Do you see a time difference between these two lines?
Do you now have all 4 trusted satellites finished their scans?
This should only fix a discrepancy between the OS reported usage and the piechart on the dashboard.
The Avg Disk would not be updated, this is a satellite-side reporting and currently it has issues: Avg disk space used dropped with 60-70%.
Hah - not even close. Even with the disk full and no longer taking in ingress, it’s still taking forever. After running 56 hours to do just the Salt Lake satellite, it’s been working on the US Satellite for the last 13 hours. It’s up to directory “j7”.
I see. There I could not offer anything better than to disable a lazy mode to speedup a process. This should not be a problem now, because your node is full, so it wouldn’t reduce a success rate, but would finish scans earlier.
I might consider that, if restarting didn’t mean the last 69 hours of progress would be completely wasted and restarted from scratch. I highly doubt that non-lazy could be so much faster that it could make up for having to redo what it took the lazy 69 hours to do.
It would be helpful if the filewalkers would do the most outdated satellite first, instead of doing them in the same order every single time. The fact that the fourth satellite never gets done until all four can be completed in one go is, I think, a big reason that the distortions get maximized.
but you said that some satellites are finished? However, you, probably right. Until we would have a resume for startup scan, it could be waste of time…
Only Salt Lake has finished. I think I may have finished it as often as 3 times in the last couple of weeks - twice for sure. But since that takes around 60 hours or more each time, I’ve never managed to finish a second satellite.
Hm, what is concurrency for the scan?
retain.concurrency option/flag (not sure, though, that it affects used-space-filewalker…)
Whatever the default is. I’ve never seen any guidance to change it. I see it commented out in the config.yaml with a value of 5.
1 Like
By the way, since I moved the databases to ssd (and maybe it was there before, but if so I didn’t notice it), I do see this now in the logs:
"Invalid configuration file key {“Process”: “storagenode-updater”, “Key”; “storage2.database-dir”}
Since changing that value was part of the guide to move the dbs to an ssd, I’ve ignored this error… and frankly, anything that will STOP the node from being updated right now I see as a good thing, so if this is breaking the updater for now, good. For now.
One of my nodes with a slower, but still respectably fast drive started having this issue. I tried a lot of different things and eventually limiting it’s memory usage to 8GB was enough for it stop.
I haven’t been able to replicate this issue since I had it. Initially I wasn’t even bothering to limit ram usage as I have an overwhelmingly high amount for my needs (512GB).
I have a theory that if I were to give it back all 512GB, this problem would show up again. However, I don’t have the equipment nor time to test right now.
Interesting. There’s been a couple of issues discussed in this thread, can you be specific as to which one? And how exactly did you limit the memory usage, please, under what OS?
such INFO messages are comes from the storagenode-updater process as you may see. It doesn’t recognize all storagenode’s options but shares config.yaml.
They likely mean the docker’s --memory option in the docker run command (if you would use it, it should be placed before the image name).
1 Like