[Solved] Trash Reported at 0Bytes but it's clearly there

Node Operators troubleshooting
,
21

The read only attribute has been ignored by windows for years, disregard it; basically, that attribute has been supplanted by security rights.

1 Like
22

thanks for the confirmation

23

Is there anything in the logs indicating that its a startup process? I was under the impression all of the walkers are re-run eventually?

24

ā€œlazyfilewalker.used-space-filewalkerā€ indicates the startup process. All other filewalkers run periodically like these

lazyfilewalker.gc-filewalker
lazyfilewalker.trash-cleanup-filewalker

2 Likes
25

I learn a bit more every day here, thanks!

Iā€™ll leave it tonight, if lazy doesnā€™t finish Iā€™ll prob turn of lazy and restart the node to see if it finishes that way.

For specs I am:
Win10 (I know know, but I canā€™t change this anymore)
32gb ddr3
3 nodes (1 per hdd)
1 ssd for OS + all the databases (3 nodes worth)
CPU: AMD a10-7860k (fake hyperthreaded dual-core ā†’ 4.1ghz)

So specs wise it shouldnā€™t be taking this long but Iā€™ll see

26

Just try to stop restarting the node :wink: . Let it finish its job :slight_smile:

1 Like
27

You can always change.
A Raspberry Pi is a cheap computer and running a node ā€œforcedā€ me to learn Linux.
It was daunting for me at first (I wasnā€™t used to a command line since DOS back in the 90s!) but totally worth it. Itā€™s been great fun (but still SO much to learn!)
For running services like Storj it just seems so much more stable! Most of the times it went wrong was because I fiddled with it :slight_smile:

1 Like
28

FFS. LET. IT. FINISH.

29

well, I have ~15tb of NTFS data and no hard drives to migrate that to ext4 (?). I was also under the impression Linux doesnā€™t like ntfs as much as windows so Iā€™m stuck here for the time being :sweat_smile:

30

Alright, I heard you guys, Iā€™ll let it keep running for a while longerā€¦

31

Hahahaha! Someone needs his morning coffee! :smile:

1 Like
32

So you guys were 100% on the money, I just wasnā€™t letting the lazy filewalker finish (I guess it takes a long time since there is so much activity that it doesnā€™t get the priority it needs (?))

Iā€™ll mark it as closed now - thanks everyone for you suggestions/feedback!

3 Likes
33

Thatā€™s great news!

I still think you should still spend about Ā£100 and get a Raspberry Pi and an SSD and start playing with it.
Maybe even spin up a new node with a small external HDD.
You wonā€™t regret it and weā€™ll be here to help you out :wink:

All the best! :blush:

34

If storj fills up all my current disks (and I have room to add at least 1 more) then my future will be playing around with Linux servers (I really hope I get an excuse to do that) but as is, Iā€™d rather just buy another HDD and throw it in the current case :sweat_smile:

35

PSA: used-space needs to be re-run with the next update (1.105.x), even after running it when updating to 1.104.5. After cleaning up most of the expired trash, nodes are starting to report 0B trash even though there are files in the trash folders.

This is the error:

2024-05-29T23:03:22+03:00        ERROR        blobscache        trashTotal < 0        {"Process": "storagenode", "trashTotal": -421774055460}
2024-05-29T23:06:19+03:00        ERROR        blobscache        trashTotal < 0        {"Process": "storagenode", "trashTotal": -25964643328}

Looking forward to another week of used-space running.

36

It took 7 minutes on a node with ~1TB data including trash on ext4. Another node with ~2TB on xfs took about 14 hours.

Iā€™m moving a node from xfs to ext4 right now. I will run 2 used-space filewalkers on xfs before the last rsync and then Iā€™ll run another used-space filewalkers on ext4 afterwards. This way I can compare the speed more accurately as it will be the same node with approximately the same data.

2 Likes
37

2TB nodes arenā€™t the same as 20TB nodes though.

38

If itā€™s linear it should take 20*7 minutes. And who has 20TB nodes these days? :grin:

39

Itā€™s not linear and let me explain why: Metadata canā€™t be cached when you grow a node that big. If itā€™s a 1TB node, most (if not all) of the metadata would be in RAM (cached). Since a 20TB node has to go out to the disk to get the metadata, and if you have lazy walkers enabled (I do), that means the used-space FW needs to wait for free IO (since it runs at a lower priority) to get the data out of the disk.

Canā€™t say I have 20TB nodes, thanks to all the deletions.

40

This is usually mean, that the database is not updated. Could you, please, restart the node with the enabled used-space-filewalker on start?