Bloom dont work or why my trash is empty?

[pangolin]

@Ambifacient

you are definitely risking your node

Really? @littleskunk

[snorkel]

A middle ground would be to put a limit to trash like a fixed X TB or a percent of total storage available. So in a short time, if there is to much data sent to trash, only X TB would be kept for 7 days and the oldest deleted even if it has less than 7 days.
In this way we can better plan our upgrades and etc, and would not have any more discussions about this topic.

[littleskunk]

Yes that is correct. In times with almost no garbage collection changes the risk for bugs is low and a storage node could wipe the trash folder with no penalty. Don’t get me wrong. Even in these times there is a risk. For example a satellite could burn down and we are forced to restore the satellite from backup. Part of that procedure is also to restore all data from the trash folder on the storage node side. Nodes that don’t keep the files in the trash folder could get disqualified basically in any situation that requires us to restore the trash folder.

Now in addition to that risk we are currently also changing a lot around garbage collection. So that increases the risk of a bug and us having to call restore from trash. I have deleted the trash folder on my own storage nodes in the past but right now I don’t risk it. I don’t want to get disqualified for a simple bug. I don’t want to get disqualified for ignoring the fallback option that was implemented for this exact reason. At least not for the next few months. I might wipe my trash folder in a few month when all the garbage collection changes are stable enough to risk it again.

[pangolin]

@littleskunk

I might wipe my trash folder in a few month when all the garbage collection changes are stable enough to risk it again.

But right now nodes suffering the most from the trash thing. When looking at my oldest node I see trash at 3.5 TB and still growing. All those fancy filewalkers working for a week or more and I don’t know when this comes to an end. Meanwhile almost all uploads failing because of race lost.

I will take the risk now for that node and see what happens…

[Ambifacient]

When trash was first proposed it was mentioned that it was supposed to be a temporary solution:

Deleted pieces will be sent to trash by default

Hi SNOs, The next software upgrade includes a change that will by default send piece deletions to the trash rather than immediately removing them from disk. This is a temporary precaution to avoid failing audits for pieces that were deleted between satellite database backups if we have to restore from a previous backup. Each node still has the option to turn this configuration off, but we highly recommend you keep it on. It is very unlikely that we will have to restore from a satellite database…

I wonder if it would better to invest developer hours in this potential other solution than to “cleanup” this trash situation.

[00riddler]

It would be better to do so.
But why should they? Developer hours do cost money. Keeping the trash on our nodes does not cost them money.
It will only be of interest if no more storage space is available for the complete network.

[Toyoo]

@d4rk4

You just use SMR disks as wrong way =) It’s a good and cheap solution for a large ZFS setup with multiple RAIDZ VDEVs.

TrueNAS stopped using them in their ZFS-based NAS, and Seagate explicitly stated this is a bad idea. They are probably amateurs who are also not using SMR drives the right way.

[d4rk4]

I have no intention of arguing with you. If one understands how SMR drives work and how they should be interacted with (what actions need to be taken on the host’s part), they can be successfully utilized. Honestly, I’m rather pleased that there’s a prevailing notion that SMR drives are unsuitable for any use, as it allows me to purchase these drives quite affordably. Just for your information: I have a ZFS setup consisting of 60 SMR drives of 5TB each, which has been running smoothly for 4 years now. Of course, some drives have failed during this time (I think I’ve replaced about 10 drives), but the resilver process has been trouble-free.

[Mitsos]

WD Knew DM-SMR and ZFS were a bad match!?! WD Red SMR 3

[donald.m.motsinger]

@snorkel

Untill now I took Storj’s side, agreeing that they should not pay for trash, because the trash is not payed by clients, and after 7 days you get rid of that trash, and the space is free for new data. But in the recent months, the trash is almost never empty, and takes space from our storage daily, space unpaid for.

If your node is not full, what difference does it make to you if some free space is used for trash? None.

[snorkel]

Well, I have 9 of them full… at least they were full until the changes in bloom filter took 2 TB down from each.

[Alexey]

You may use a [Tech Preview] Multinode Dashboard Binaries to have it

[Alexey]

Yes… But our customers are suffered from a long deletions, so… We decided to use an async approach, which in result created this situation with the garbage collection…