Hi SNOs,
The next software upgrade includes a change that will by default send piece deletions to the trash rather than immediately removing them from disk. This is a temporary precaution to avoid failing audits for pieces that were deleted between satellite database backups if we have to restore from a previous backup. Each node still has the option to turn this configuration off, but we highly recommend you keep it on. It is very unlikely that we will have to restore from a satellite database backup (we have never had to restore from backup thus far), but in the event we do, this will be the only way for you to avoid audit failures and potential disqualification.
Again, we understand this is not an ideal solution but we will be implementing a better solution in the coming months.
You may check out the change here
Just a question - If a database backup was restored would the piece deletions be moved out of trash and back into the original storage location?
Then you have to pay (fo trash storage) for your bad ideas.
Why do you want to be paid twice for storage ?
Why not store the list of recent deletes in tardigrade? Then they can be re-applied after the backup is restored
Yes that is our short term solution while we work on a better solution.
That is what the better solution would do. Storage nodes just keep the delete message and if the satellite requests an audit for a piece that was deleted the storage node can proof it with a signed delete message. Problem right now is that we didn’t had the time to implement it.
Why twice? Deleted = not paid. Trash = free storage.
Don’t all databases support online/hot backups these days: do you need to bring the satellites down for them? (if the problem is that backups aren’t timely: isn’t the solution just to perform them more often?)
I have no problem with having a bit more trash hanging around: I think most of us still have tons of free space 
Hot standby replicas are not backups, they are redundancy. There is a big difference.
Redundancy is for uptime – your hardware is malfunctioning or someone broke the primary config, so we switch to the replica.
Backups are for disaster recovery.
If a human accidentally runs a too-broad update/delete operation on the database, replication is going to happily ship the update to the standby and it won’t help you recover your mistake. Other forms of data loss could also be replicated to the standby replica.
I currently have 21 Tb stored but trash is only 102GB. Maybe some of your pieces already got repaired during your one hour downtime and get deleted now on your node?
Hmm my trash is constant at 102 Gb…
Did you check your logs?
This is really curious, these lines are not garbage collection, but normal deletes, unless they changed the wording for garbage collection to be the same (sure hope not). But I’m not seeing any of these. Barely any deletes at all lately.
I was actually thinking to myself the other day that since the change I haven’t seen an increase in the amount of data in the trash. Pretty much the same levels as before, although I don’t track it so this is just and anecdotal observation. 1 GB / 3.55 TB and 500 MB / 1.74 TB.
Yeah there have been very few deletes recently.
I did not follow on this one. Are we having this ‘better solution’ in the meantime?
Doesn’t seem like it. I still have large amounts of data in trash.
What percentage? I see 3.5% of amount data stored over all nodes.
Percentage of total data isn’t really relevant. Percentage of monthly deletes is more interesting. And since I already calculate that for the earnings estimator (it’s just over 1% on average atm on my node)
But I’ll post both. I have a total of 38.8GB on my largest node. Which is:
0.24% of data stored
Since that’s 7 days worth of trash, lets multiply by 30/7 for 1.03% per month, which pretty much exactly matches the total amount of deletes on my node.
It is now over a year since this claim. Are we still sending every deleted piece into trash instead of deleting them from the disk instantly?
Yes we still sending almost every deleted piece into trash, except expired. Expired pieces are deleted directly.