Even with fairly agressive compaction settings resulting in <5% reclaimable overhead it has been far faster than piecestore filewalker for moving stuff to trash and emptying the trash for me.
Unless you have ZFS with special device setup. My disks on XFS reclaim at maybe 50% the max read/write speed due to the node also reading/writing pieces.
This is a tradeoff between power waste vs storage waste. Due to the nature of the project – space waste is much preferable.
If I could be arsed to figure out how to do it – I would increase thresholds to 50-100%. I can double space storage nodes use and still have plenty left. Power on the other hand – expensive.
I believe you can do something like setting the alive fraction to be like 0.05, and the probability power to something large like 10-20. Thus the probability of a log file being selected for compaction would be 100% if the log is 95+% dead, and ~0 otherwise.
Could be quite efficient and I wonder how much overhead there actually would be.
True in theory. But my drives are always busy anyway because of all that random egress traffic. So I guess we can’t save much power by making compaction less aggressive. Also I can use the freed space for things like chia mining.
There is a difference between background chattiness and full speed maximum random IO.
But I agree that this random IO takes half a minute and happens infrequently enough to not matter. Defaults are adequate.
Lol. Right. As if this is not a completely unproductive net negative activity. I would not touch it even if it was temporary profitable.
Without any adjustment reclaimable space was like 30 TB in sum for my nodes. Now I am down to 5 TB.
All but those settings are still default:
STORJ_HASHSTORE_COMPACTION_ALIVE_FRACTION=0.6
STORJ_HASHSTORE_COMPACTION_REWRITE_MULTIPLE=4
I’d say, it would be more useful to have a different tuning knob: “how much available space do you want to have after compaction”.
Reasoning: if I’m dedicating 10 TB for Storj, until I’m close to that 10 TB, I don’t really care how is this space used. So until my nodes are close to that 10 TB limit, there’s no need for any compaction. And when I’m close—I only care to reclaim daily as much as necessary to allow one more day’s worth of uploads. The rest can wait increasing chances that even more pieces will be dead by the time compaction is necessary.
The current code does not make this easy though—compaction is per hashstore and per satellite, so it’s difficult to coördinate which log files to compact.
How much is this in percentages?
I would feel better. The problem for me is that they control the amount of trash but have no incentive to keep it small. Maybe if they paid for it there would be no talk about adjusting how it works and we could be sure that it would be minimal. They would be the ones to worry about the amount of trash. Thats not our job.
Precisely and we are seeing this. The problem is that their tech does not align with their promise to the customer or the customer expectation:
When a customer deletes a file his expectation is that it gets deleted immediately and that he will not get billed for it any longer.
However this is not how the Storj tech is working. I am not sure now with haststore but with piecestore there were those bloomfilters periodically created from backup data, then some time later if the deleted data is included in the bloomfilter the pieces were moved to trash and after that finally deleted. AFAIK all of that unpaid for the SNO. And as you have said for Storj no incentive to create find a different process that uses less disk space, because for them it is free space they can use however they want.
For SNOs of course it is not free space. It is space that is occupied by Storj and cannot used for anything else. But they bear the cost.
There was more or less big discussion about that in the past:
AFAIK select network is using the same software and storj is paying for all allocated space there. So the incentive to save space exists.
Space is traded off for reliability (for trash) and speed (for hashstore compaction). Both are even more important on the select network.
But we are not on Select, are we?
But we are on the same version of the software.
You may also configure how often and how deep the compaction will work.
See Hashstore Migration Guide
Right. Would not it be nice if the compaction was happening in realtime? This would avoid both the space waste and periodic bursts of high IO, maintaining consistent low background IO that does not interfere with anything. Like, for example, purely hypothetically, if we could place each segment into its own file, and let (gasp!) filesystem manage its free space, caching, and access acceleration, as opposed to hiding intent behind opaque containers, requiring ugly manual garbage collection, then we would not need this jumping through hoops. I wonder if this was ever attempted. Oh wait…
Yep. And the file walker was killing select nodes.
Right. On crappy filesystems and storages. I’m not denying that. But in the vertically integrated storage stack the solution should have been swapping a filesystem with the one that does not fail at one job filesystem must do, and not rearchitecting the application to take over part of the filesystem job.
Look, my 10TB node filewalker was taking 20 seconds. On an ancient decade old hardware. No HDDs were accessed. No IO overhead. On 600TB collection of nodes this should take therefore ballpark of 20 minutes.
20 minutes. Once on server start. So, pretty much never.
And what do we have now instead?
And why? I fail to see why.
@Alexey , would it be possible to learn how select network nodes are set up? Like, what OS, what file systems, etc.? Or is it left again to node operators?
Not much of insights.
I only know, that there is Linux, likely Debian, ext4 and hashstore with memtbl.
I do not know, does it use containers or binaries, but I think it’s likely binaries. Servers are normal servers, likely not new. Network is dozens Gbps.
That’s all.
Well here is it.