It is a bug, I encountered the same thing many many version back
It should be this
This is weird, because my nodes on v1.102.3 have these log records. And I run two docker nodes and one Windows service node. All have lazy and scan enabled.
What’s your storagenode version?
I have --pieces.enable-lazy-filewalker=false
as this was recommended everywhere when doing the startup scan. I wrote this multiple times here too.
All my nodes are on 1.102.3
It’s recommended to fix the not updated trash on the dashboard, otherwise it’s better to run it in a lazy mode. I run it with a lazy mode enabled, because I do not care about cosmetic values for the trash on the dashboard.
However, it could be a culprit of not showing logs entries as mentioned above.
I thought that’s what we’re talking about here. Read the whole sentence you just quoted. Of course I have it enabled otherwise.
Yes. I didn’t explain, what’s that mean, sorry!
If you are on the version before 1.104.x, the trash information is not updated on the dashboard (but accounted properly),
In that case you need to disable the lazy mode if you want to have an updated info related to the trash, even if it’s not important for the node.
if your node is on 1.104.x, it should update this info, thus you do not need to disable a lazy mode anymore.
The main problem is not in cosmetic values, but in the fact that a new data does not come because disk is “full”.
I had to finish filewalker with lazy turned off (it took about a week) and as a result, the values did not change:
But in fact, some of the garbage was removed from the disk even before the filewalker was launched:
How do I start getting ingress? Restart filewalker again and wait another week in the hope that the dashboard will be updated?
Now I will have to run filewalker on nodes with a “full” disk all the time so that the node understands that there is free space?
Then you have an error related to the databases update, there is no other way.
Please check that your databases are not corrupted:
Already checked. All fine.
Delete the Trash and hope you won’t get DQed in the next 7 days. But you need one more Filewalker run after that.
The garbage from a disk is removed normally. The problem is that the node doesn’t see it
I have more than 10 “full” nodes, but in fact there is already a lot of free disk space after garbage removal.
Where are the db-es located? On the storage drive? Or on a SSD or a USB3 SSD flash drive?
If not, you should consider moving them. I have coming some Corsair flash drives with SSD tech. They will be more than enough to keep databases happy.
Try to vacuum them, also.
https://forum.storj.io/t/database-maintenance-integrity-check-vacuum/25047
All DB’s on nvme ssd.
Hmm… that’s strange. Even my 1GB Syno, running 2 nodes of 7TB each, never had problems with db or dashboard.
Totally agree. That’s very strange and I lose ingress every day
Did you check the permissions. Does the node have permission to delete the trash folder and files and write db-es? I’ve seen cases with problems at this points: owner and permissions.
Node removes garbage from the disk, so I assume that there are no rights problems. Each node is run under a unique id. All folders created by gc under this id. DB’s also with this id.
Does they show recent modification time?