I had a quick look and saw 2 options – bcache and LVMcache. Bcache is what was discussed at the link I posted above. LVMcache is a logical volume manager that is independent from the file system. And as far as I remember Storj’s documentation again doesn’t recommend using LVM.
Yes, this workaround can be implemented on Linux and even may be on Windows via WSL (I’m not sure about that though). But it’s exactly as I said a Linux geek thing, not even close to a regular SNO.
Yes, I’ve spotted before that you persue either Stroj’es or your own selfish interests but not interests of SNOs as a community.
So I can see Storj trash handling hurting regular SNOs and a few cynical SNOs benefiting from it and supporting it.
Just try it and if you get it working you can share your knowledge with everyone else. If you feel like you want to call yourself a Linux geek after that thats fine for me as well.
For comparison for ZFS we have this thread: Notes on storage node performance optimization on ZFS
In the first post is all I need to reproduce it and I don’t think reading that post and copy paste the commands makes me a Linux geek. But I am not going to stop you from calling yourself a Linux geek. Thats fine for me. I just don’t see myself as a Linux geek because there is way too much about Linux that I don’t know about yet.
1 Like
I don’t consider myself a Linux geek though Linux has been my only system for quater a century. I can try and get it working for sure but that is not the right solution. That is another work around on top of work arounds already built around Storj. I strive for a proper solution which is get Storj fix the screwed trash handling. If this is an architectural issue it needs to be solved sooner or later and building workarounds for a dead body is not a solution.
The node doesn’t require ZFS. It’s only for your convenience. ext4 should work fine for most of the setups.
However, if you can improve the setup by adding more RAM (Linux can get an advantage of it without any additional tuning), that’s will improve things.
1 Like
You can. For Linux with LVM, for Windows by using a tiered Storage or Primocache.
Perhaps only for cases when you build a RAID0.
Even if so, the point is with all its advantages ZFS must have disadvantages too. It’s not even supported on a bare minimum Linux install that tells something. If it was better than EXT4 in all respects then probably it would become the default FS on Linux. But for some reason(s) it’s not happenning. And my points still hold – SSD + ZFS is a walkaround and not a solution for the trash handling issues and this workaround is not for regular SNOs who may not be even on Linux.
3 Likes
you are of course correct. However, you can still optimize your setup even on Windows. By at least using the check and fix errors on the filesystem and defragmentation.
The next steps are more advanced, but still doesn’t require to change the hardware confirutaion:
But if these changes would be not enough, then, well, you need to do something more cardinal, like adding an SSD cache as a tiered Storage using a PowerShell or using a Primocache.
We would hope, that our developers could implement something to lower the required IOPS.
to be honest, I not suggest using Primochache it broke my nodes under heavy load.
3 Likes
From what I know, it’s a matter of licenses at the moment
Guys, every few years some file system excites some part of Linux community and they tell it will soon replace EXT4. It happend with XFS, ReiserFS, BTRFS and who knows what other FS else.
I’m not a fan of ZFS or any other FS and not a hater either. All I’m saying the default FS on Linux at the moment is EXT4. I don’t predict if ZFS will replace it or not, neither I care about it. When/if it happens then I’ll stick to it as being the default on Linux. That’s it.
And again my whole point is not about FSes and I see no value in continuing EXT4 vs ZFS discussions. My whole point is let’s strive for a solution and not a workaround.
3 Likes
It is not. My node has been up for a month and all full from the very beginning but Storj tells me it used 60% at maximum. You may solve some local issues with filewalkers using SSDs and ZFS tricks but you can’t solve the issue of trash hadling by Storj that steals significant amounts of space from SNOs by not paying for the amount used.
You guys are talking about different things. ZFS’s native support for handling metadata with SSDs does solve filewalker-runtime issues. And actually having filewalker complete does fix many Storj UI discrepancies. But the fact the Satellites have one idea of disk usage, and the Node has a second idea, and the filesystem has a third idea… definately isn’t a problem that even infinite iops will solve 
2 Likes
This. I first worked with ZFS on Solaris, back in the Sun days. It powered massive storage arrays: and was kicking the vendors of proprietary arrays (like NetApp) in the nuts. It moved down into less-mission-critical environments when Linux distros started to pick it up.
Some may never like the way it violates some purity/layering constraints: by blending the idea of a filesystem / md / lvm together. And it’s not that EXT4 is bad: it’s reliable proven tech. But ZFS is better for Storj: specifically it’s native support for metadata accelleration with SSDs puts filewalker issues to bed.
1 Like
Well on my nodes it looks like I am getting paid for my used space and just the dashboard is off. With infinite iops I could spawn an infinite numbers of used space file walkers to have an accurate reading at all times 
1 Like
To reiterate. There are multiple issues, some under SNOs control some are solely dependent on satellites.
SSD+ZFS tackles SNO local issues only. Let’s say we used it or some other magic and got our filewalkers completing instantly.
Now the issue is on the satellites side. Do satellites claim:
- Sending bloom filters not rare than once a week? Does in reality it take longer?
- That bloom filters leave behind no more than 10% of trash. Is it in reality different?
- The trash is kept for 7 days?
I don’t know and I shouldn’t care. But in reality huge percentage of disk space keeps occupied even if all file walkers are instant but the satellites claim they don’t use that space.
This is the issue.
1 Like
Are you sure? How do you know? Most people make the mistake to take the numbers on the storage node dashboard or from some python script and believe it would be the size they have on disk. That part is wrong. We know that the local storage node is showing incorrect numbers on the dashboard. So you have to spend a bit more time to rule out calculating with incorrect numbers. The numbers look wrong at first but when you start questioning which part is off you have to go down the rabbit hole and surprise you might get paid for almost all the used space and the issue you want to see fixed is a different one than you claimed at first.
And this is part of the issue – how can I know?
I know that bloom filters are not coming daily. Do they come twice per week? Or once? Or once per month? Why don’t satellites stop sending them at all until they run out of all space of all SNOs?
How long does this thread last? How many other threads were before it about similar issues? How long will it take for Storj to fix the statistics reporting?
What responsibility does Storj take? What are the timelines? How can SNOs plan their business if Storj takes zero responsibility about anything related to SNOs?
Ok I tried. Keep venting.