I wonder why is it false by default.
In storj select we donāt run the maintenance jobs inside the storage node process. We run them as separate processes. I think that commit is for that cleanup process and the normal migration might still leave empty files behind even if you enable that setting.
2 Likes
Is this something that is possible for a regular node operator to do as well? Iād be interested in experimenting with the compaction options but it seems like the compaction jobs occur on a somewhat random 24 hour timer, which mostly just tests my patience instead of the config changes I make. In other words, is it possible to force a compaction?
Yesterday just finished the migration on the first node, so 17 days it took, but last 12 days it had no ingress, so it speeded up the process. In order to correct de used space calculation, I will delete all db-es and deactivate the badger. Letās hope it will work.
Update: yes, it worked, and the dashboard was updated in less than a minute.
Wow! Hashstore is so fast!
2 Likes
youāre saying deleting the dbs is safe?? i thought deleting dbs is a surefire one-way ticket to disqualify the nodeā¦
You can stop the node and delete the db-es. Itās safe. Iāve done it many times. The node is recreating them. You just loose some stats for the current month, but no payout is affected, or data or scores.
See here:
https://forum.storj.io/t/what-can-i-delete-on-storagenodes/29187?u=snorkel
1 Like
Im curious if those that have migrated and stopped getting ingress have badger cache enabled or not? I dont think that would affect things, but just trying to think through possible correlating issues.
Tried your suggestion on my two smallest nodes(turned badger off as well). It made no difference at all for me. The difference between your setup and mine is that I have āDedicated diskā turned on. I think you said in an earlier post that you didnāt use that.
Great to hear you got your used space calculation to work. I tried deleting all the .db files under storage and also deleted the piece_expirations folder. Unfortunately that did not fix the used space calculation on my node. There is another .db file in the root of the node called revocations.db. Is that safe to delete as well? Could you share exactly which settings you are using on that node?
i had no badger enabled. honestly i did not even know its a thing before stumbling upon this thread
This is the startup file walker on a full migrated 5TB node, on a system with 8GB RAM; usualy it was taking days to finish:
2025-02-18T15:11:01Z INFO pieces used-space-filewalker started {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2025-02-18T15:32:22Z INFO pieces used-space-filewalker completed {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Lazy File Walker": false, "Total Pieces Size": 0, "Total Pieces Content Size": 0, "Total Pieces Count": 0, "Duration": "21m21.60683003s"}
2025-02-18T15:32:22Z INFO pieces used-space-filewalker started {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2025-02-18T15:33:32Z INFO pieces used-space-filewalker completed {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Lazy File Walker": false, "Total Pieces Size": 0, "Total Pieces Content Size": 0, "Total Pieces Count": 0, "Duration": "1m9.207302349s"}
2025-02-18T15:33:32Z INFO pieces used-space-filewalker started {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2025-02-18T15:33:32Z INFO pieces used-space-filewalker completed {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Lazy File Walker": false, "Total Pieces Size": 0, "Total Pieces Content Size": 0, "Total Pieces Count": 0, "Duration": "407.111412ms"}
So, I stopped and rm the node, I deleted all db-es in the storage directory (not in other directories, as I explained in the dedicated post); I also deleted piece-expirations dir and filestatcache dir.
I changed the run command to start without badger cache and I started the node. The correct space was displayed after 1 min.
When you delete databases and those dirs, you have to enable startup piece scan to recreate all you deleted, and let the walker finish.
Synology NAS + Docker:
echo "net.core.rmem_max=2500000" >> /etc/sysctl.conf
sysctl -w net.core.rmem_max=2500000
echo "net.core.wmem_max=2500000" >> /etc/sysctl.conf
sysctl -w net.core.wmem_max=2500000
echo "net.ipv4.tcp_fastopen=3" >> /etc/sysctl.conf
sysctl -w net.ipv4.tcp_fastopen=3
docker run -d --restart unless-stopped \
--stop-timeout 300 \
--network host \
-e WALLET="xxx" \
-e EMAIL="xxx" \
-e ADDRESS="xxx:28901" \
-e STORAGE="7TB" \
--mount type=bind,source="/volume1/Storj/Identity/storagenode/",destination=/app/identity \
--mount type=bind,source="/volume1/Storj/",destination=/app/config \
--log-driver json-file \
--log-opt max-size=25m \
--log-opt max-file=3 \
--name storagenode storjlabs/storagenode:latest \
--server.address=":28901" \
--console.address=":14011" \
--server.private-address="127.0.0.1:14021" \
--debug.addr=":6001" \
--log.level=info \
--log.custom-level=piecestore=FATAL,collector=FATAL,blobscache=FATAL \
--pieces.enable-lazy-filewalker=false \
--storage2.piece-scan-on-startup=true
docker logs storagenode 2>&1 | grep "used-space-filewalker"
Thanks for that info. I now see the used space value 
It was setting dedicated disk to false that did the difference.
But I still donāt understand why there is such a big difference between used space and average disk space used this month. Used space has been above 700GB all of 2025
You see those dips? Thatās why! There are some missing reports from US1 I believe.
1 Like
What about the plotted values on the other days. Shouldnāt they be closer to the actual used space?
For example this piece stor node I can understand. The used space value corresponds with the graph
@PieceKeeper āIt was setting dedicated disk to false that did the difference.ā
So now we can confirm that setting āDedicated diskā to enable destroy the used space calculations.
I leave the dedicated disk turned on and wait for the bugfix. The node is working so the numbers shown in the web-app is not important to me.
The usage calculation for Dedicated enabled is not working
No, the badger is not affecting this, but databases seems are not correctly updated after migration. So, deleting them may fix the issue. I think that this one should be enough:
You may use the latest full report data point as a rough estimation of the used space from the satellites perspective. The exact would be in the payout receipt, so you would need to check it after day 15 of the next month.