By setting it to true in the config.yaml and restart the node. There is no manual trigger otherwse as per my knowledge.
1 Like
I think we need this feature.
You don’t need the FW so often, maybe once a year. I kept it more than half a year off, and I didn’t had any problems. Run it with node restart, and switch it off after it finishes, and again restart. No need for triggers.
1 Like
I would say no need for the second restart. Just change the config yaml and leave it go. When the node restarts next time it will load the changed config anyway.
1 Like
Anyone knows if this is done at a Storj Node level, firewall level, etc.? And if so how? I have one of my five nodes that would be really happy without small files.
Edit: yikes, didn’t know it was against T&C.
Don’t do this. It’s against terms of service you have agreed to. Fix your node performance instead.
3 Likes
Small source code change. Against T&C, so not exactly recommended, but anyone with basic coding skills is capable of making this change.
How big are your trash subdirectories? These always get scanned on startup, even with disabled file walker.
2 Likes
Hello.
- I would like to raise for consideration the issue of introducing in the log file, a report, about in what state the filewalker is at the moment. (or even in SNOs dashboard if possible)
Yes in log im getting starts and exit lines, but nothing in between,
and monitoring it in debug mode is somehow problematic as if not sure why filewalker is not shown in proceses, altho logs assures me the filewalker is started and not exited yet.
https://forum.storj.io/t/guide-to-debug-my-storage-node-uplink-s3-gateway-satellite/1372/33?u=ruskiem
Because for example, now, i dont know if my node is half way in checking all of the 15 milion files my node has, or in what other percentage he is…
For example i need to know, to plan maitance, and simply not waste the work of the filewalker.
Checking all files in Storage folder takes days, even for windows, even with a lot of RAM, here:
it took windows 6-7 days to count all files.
Similar is true for the filewalker function, which in full (not lazy form) is crucial for reporting what is the real usage of the disk.
RAM i allocated 7GB, i can do 14GB but it seems it just does not get any faster in small files counting despite more RAM.
it’s normal PC, quite modern, win10, storagenode GUI.
(processor AMD 4700GE, 8/16 cores,
DDR4 3200MHz RAM, sata III, 8TB HDD HelioSeal WD Ultrastar, NTFS)
quote from Release preparation v1.90 topic
also in other topic i found people confirming the problem:
That’s @jammerdan quotes, from this topic: here
So im being paid for 4,81TB, and my nodes actually holds 6,41TB, and the dashboard shows 6,98TB. That’s all because filewalker wasn’t been able to finish its work without interruption over course of 6-7 days. Because theres always some unexpected shutdown, or service restart in log i notices. I got logs from 2021. From what i saw, the filewalker over course of last months,
… i never was able to determinate, if the process was completed with success even once,
i didn’t saw that even once in the logs.
Probably there’s always something occurring, a restarts or a new version update, i will provide logs maybe later, but that doesn’t change the clue of a problem: that a filewalker can do all the job and a restart in last moment causes it to forget all the work he has done…and to start allover again form 0.
Unexpected shutdowns in log can occur few times in a month, seems its enough to interrupt the file counting, if it takes soo long to complete. I realized i have only 3 maybe 5 chances in a month to perform a full filewalker, to provide actual information, to get paid correctly for actual files the node have, so it sends that info to satellites (if current gaining of space is around 0,4TB to 0,6TB a month, thats a mater of almost a 1$ per disk, per node a lose, if once a month a filewalker wont go to check all the files. This is a arduous task to watch if it was succesfull in every month now, to track it in every node like that. Would be much easier if the process would remember where it stopped, and not start from 0 every time.
Im doing defragmentation and MFT Optimization, just like advised here on forum, but im not sure if my large nodes will be able to ever finish a full normal filewalker (not lazy) before some unexpected restart of service.
- So i rise that to Your attention, now when egress prices are slashed from 20$/TB to planed from Dec 1, 2023 2$/TB. a correct payout from storage is crucial for me as a SNO. Would warmly welcome announcement that STORJ inc. is improving filewalker function so it can SAVE work done, if restart occurs, and IF it was not longer than few minutes ago, so it will knows, it can CONTINUE WHERE IT STOPPED, counting the files on hdd, and not start from the beginning.
Thank You!
You don’t need tuning, you need overhaul.
That means this storage solution you have is not suitable for running storagenode. You cannot get around physics – if touching every file takes 10ms and you need to touch millions of files you can’t finish that in 10 minutes. Does not matter what you tune. You have the same 10ms overhead for fetching data, and are at disadvantage compared to faster nodes.
The only solution therefore is to decrease seek for metadata per file. How? Use some sort of tiered storage or caching solution, you want metadata to live on SSD or be in the cache.
For reference, on a 10TB node filewalker shall not take more than 10 minutes. That’s your target. If you start with 6 days – you will never optimize it down to 10 min.
(Also, windows is shit for hosting services. Use linux or freebsd, you will haveaccess to much better storage technology. It’s much easier to get better performance there. But with windows it is possible too – just cost money and time)
1 Like
Yes, i make my first steps with primocache. Testing new ram right now 
.
Hello together 
,
is there some guide or some possibility to check the status of the filewalker-running on some docker? I got it startet now on one of the older notes and would like to know in advance when it is complete.
Eh, I was fine with my 12 min/TB file walker on default ext4, or 8 min/TB with optimized settings, both with no SSD cache. 1 min/TB would surely be nicer, but unnecessary, especially if it is performed as a low-priority process.
And unfortunately I still don’t fully understand:
Is the “Average Disk Space Used This Month” the real payout basis or the base coming from the satellite? And the “Average Disk Space Used This Month” is just for info for the SNO? Does anyone know this for sure? @Alexey
The information from the satellites are used to calculate payouts. This information your node reports to the satellite as a signed orders.
This information is showed on the graph. The information on the right side is calculated locally (and the usage for the current month on the Payout information - it’s an estimation). This information ideally should match.
For the used space you need that the filewalker is finished the calculation and the garbage collector has removed the deleted data from your node.
1 Like
@Alexey mentioned this, but it’s worth repeating: the filewalker output does not determine or affect your payout. That calculation happens on the satellite and covers every piece your node is holding, every time. The filewalker is used only (a) to have an estimation of how much space is used locally, and (b) to take data you no longer need and send it to the trash. There is also a filewalker that traverses your trash directory in order to clean it up, but that’s probably not your problem.
3 Likes
Plus, if I understand correctly, the file walkers’ values are used to decide whether to accept new uploads. Is this correct?
No, I do not think so. Why do you think is it?
I didn’t see any code related to this, did you?
Not at my workstation now, but IIRC the amount of “free” disk space reported to the satellites is taken as the difference between the allocated amount and the space considered taken by file walkers. Then, if the difference is positive enough, the satellite sends directs new uploads to the node.
As such, it does not matter how much data satellite believes the node stores.
1 Like