@Alexey He is using windows, is there a howto do it with windows?
I want to know too.
linux commandlines won’t help much i guess…
Please educate me, If i’ve moved databases to C: then why is D: still being over used? Like i say i’m only a novice.
This means only the drive is active all the time,propably not using its full potential.
Normal for frequent Filewalks trash deletion,or node freshly started.
the locked database problem is the real problem
1 Like
I’ve followed the instructions to check the databases and your right the bandwidth database was corrupt. so i’ve gone through instructions to recreate database when restarting node. Again all logs are coming back ok. D: is still at 100% but i’m assuming this is because of restart.
The storagenode service must be stopped during the process.
You may use these scripts:
PowerShell
Get-ChildItem X:\storagenode\storage\*.db -File | %{$_.Name + " " + $(c:\sqlite\sqlite3.exe $_.FullName "PRAGMA integrity_check;"; c:\sqlite\sqlite3.exe $_.FullName "VACUUM;")}
Docker (amd64 only)
docker run --rm -it --mount type=bind,source=x:\storagenode\storage,destination=/data sstc/sqlite3 find . -maxdepth 1 -iname "*.db" -print0 -exec sqlite3 '{}' 'PRAGMA integrity_check;' ';' -exec sqlite3 '{}' 'VACUUM;' ';'
You may also run the script for bash from wsl, your disks will be mounted automatically with their letters to /mnt, i.e. for the example path above the path will be /mnt/x/storagenode/storage
1 Like
Moving databases to the SSD will not reduce the load, just your databases should not be locked anymore.
You may only enable a lazy filewalker: How to enable lazy mode filewalker, it could help to reduce the load but will increase the duration.
i assume i have to install sqllite and adapt the paths?
Yes. Or you may use a docker version instead, if you have docker.
i have docker (pain in the… to get it to work also), but sadly zero knowledge, so i go for the win- gui with the 2. node (mid-to end of the month) on my pc. As i did on my mini-pc. But better this time.
Hi daki82,
I have the same problem for a few months. An Ironwolf NAS in my Synology 8TB fully filled operated with 3 nodes on it (3.5, 2.9, 0.6TB) never had problems before and now it can’t keep up. The nodes keep going offline and I’ve been suspended several times still painfully waiting for disqualification.
The Irowolf NAS is not made for so many
multiple accesses. I also run 2 Seagate EXOS 16 TB, which are also full to the brim. On each 5 nodes (4x3.5, 0.6TB). I have decided for multiple nodes, because in case of a sector-wise failure of the hard disk (no Raid) at least the possibility exists to migrate the intact nodes.
Long story short: The 16TB disks EXOS never had a problem with the dataflow. The utilization was quite pleasant. For a few months now, however, I’ve noticed an increase in HDD utilization here as well, but the EXOS still copes well with it. EXOS is an Enterprise HDD with approx. double of the IO capability as the Ironwolf.
I am now migrating all nodes from the Ironwolf down to an EXOS 8TB and hope this will solve the problem. I will report back.
If it is the fragmentation, the migration also brings something, because it is like a defragmentation. Unfortunately I won’t be able to say afterwards if it was the Ironwolf NAS or a strong fragmentation.
Please give me comments, what your Ironwolf 8TB (full) do so.
Alexey has already advised me against running multiple nodes on one physical HDD. However, I am afraid of the risk of losing 16TB at a time in the event of a hard disk failure, where 12TB might still have been salvageable. In return, I also accept a bit less revenue due to the lost races.
There are three problems with your setup:
- 5 nodes on the one disk (because your RAID volume is a one disk for the system), so they evitable affect each other.
- If that RAID would fail, you will lose all 5 nodes, despite your hopes.
- You likely use a BTRFS filesystem, which is not good for storagenode, see Topics tagged btrfs
So if you going with RAID5 it should be one node per pool, otherwise it’s worse than using a one disk for multiple nodes, because array is working with speed of the slowest drive in the pool, plus it usually has less IOPS than a single drive, in case of BTRFS it become even worse.
The better way is to use a one disk per node and use ext4 as a filesystem. This way your nodes would be fully independent of each other regarding storage.
Of course if you already has RAID for something else, you may use it, but only for one node.
I dont have any, I use WD Elements USB3.0 12TB ( 7,8 of 10TB) because of poor existing hardware. I consider running defragmentation all 3TB. Wich Version do you running?
This causes The drives to seek more for new data. response time goes up, its recommended to let 10% free. maybe less. but never fill a drive to the brim it will clog with the massive io. set the node maybe some 500gb less. it will shrink over time and be more responsive(also free place needed for defrag)
1 Like
I have 2 Ironwolfs, 8TB each, on one Syno 1GB RAM. Both ext4. One full, one half. Never had fatal errors, unexpected restarts etc. I keep the load manageable by keeping the FW off.
Exos series from x16 onward are the fastest drives outhere. Besides the 2 IW, I have only Exos 16GB and they work great. With FW off ofcourse. But all are ext4. No matter how good is your drive, if the FS is trash for Storj.
1 Like
Filewalker 
, not firewall
2 Likes
I am using BTRFS. Up till now it worked fine with my workload. I do not have any raid, so speed ist limited by the drive itself. But I once had a crash of a sektor of the harddrive that lead the node to be disqualified. I was damn lucky to have been able to successfully migrate the other nodes on that drive to a new hardware.
But I see your point and will consider in future nodes.
1 Like
I would like to give a little update. Since I moved the nodes to the EXOS disk, the workload is working fine again. The EXOS are simply fast. Nevertheless, in the future I would follow Alexey’s advice and go for the ext4 filesystem.
My 2 cents…
Yesterday (to save electricity costs where I can) I removed my nvme disk lvm cache volume from storj vg on ext4, filewalker time changed from 5 minutes to over 2 hours. Well… numbers don’t lie 
Still I can’t find a good motivation for the FW to be kept ON. I kept it off for a year on all nodes. All worked fine. I activated it, because I thought it will reduce the gap between Satellite’s reported storage and node reported storage. I can’t see any impruvement after 2 weeks, so… FW off and keep annoing devs about buggie satellites.
i have it on for few weeks like 4-5 now.
it fixed some nodes in several days, where FW was able to be completed on all 3 satelities.
2 weeks could be not enough tho, because:
- If You have it in Virtual Machine, thats doubling or quadrupling or more a seek time for HDD,
(don’t get me wrong, the HDD access time is still OK: 7-12ms but, couting files takes more)
(even if You have proper hardware controller like H310, or LSI 92xx, 93xx
and its super fast without the VM, but under VM it gets some penalities anyway, - especially when You run many VM simultaneously on that computer,
- beside You need to cut CPU to 1 procesor / 2 core per node, and RAM to it as well,
so in the end under VM ware i noticed, it counts files like 4-6+ times slower than it could without,
Still like 12 milion files in 24h (~1,5TB), but that would be done without it in minutes(10-15min), with all the resources, in fact i even did a test under only one VM online at the time, and windows 10 was able to count all 3TB storj blobs folder (20milion files) under 30min, so not that VM is bad, but if You run many in one computer, that does the impact somehow and that 30 minuets become unachievable anymore)
so in case of many VM:
- a random node restart = interupting filewalker
(its very interupting, and that can be due to an version update,
can be random, can be to a full disk,
(i move log file from C: to D: every week, it grows like 3-4GB in 7 days,
don’t have more space in C: on SSD)
So avoid restarts You must if You run many VMs, so was forced to leave Filewalker On for some time, it can take 2 months, or 3 months or more for some full nodes.
I think they should do some indicator in the node dashboard to let us know,
when the filewalker was last time completed on each satelite,
just like time online example: “us1. FileWalker status: completed: 72h ago”,
or: “us1 FilerWalker: counting… (dont interupt)”
so it at least dont start couting on sattelies that are freshly done, and even interupted by a restart, the FW could take priorities and start from the satellites that are less up to date!
And gives us Clue, to not restart a node intentionally, if the FileWalker is couting!
if you set the loglevel to fatal, the log is verry small and does not grow noticeable, if you have reoccuring problems, just set back to error level logging. if its not a fatal, the node continues to work.
1 Like