If I receive this error : ERROR pieces:trash emptying trash failed
Can I empty storage/trash folder on my own. Or data can be rolled back/needs to be deleted by satellite so leave them intact?
Don’t remove it yourself, it may be needed to restore data. The expired parts will be caught in the next bloom filter and removed next time. No need to do anything.
Can you elaborate on that on how it can be used to restore data ?
If an error is made by the satellite and it sends a faulty or empty bloom filter. Pieces may be moved to trash that shouldn’t be removed. This must then be corrected, hence why the trash folder exists. It’s a contingency. Likely not needed, but you should still never delete anything on your own.
1 Like
This is what I was trying to say but in not so many words.
1 Like
I guess I tripped over the wording around offering a feature to recover files. This is a fallback that only Storjlabs could use to trigger recovery of data. Its not something that the customer or node operator would use. It’s only meant to recover from garbage collector problems. Not to recover deleted customer files, but to recover pieces on nodes that were incorrectly removed.
1 Like
Ok, so to recap. Data from trash dir shouldn’t be deleted by hand as it can be used to recover files.
I’m too much of a perfectionist to agree with that recap, but your conclusion is correct. Don’t delete data from trash.
I wonder if you still get paid the storage that is in trash for this 7+ days period it’s sitting in the trash folder before certain files get removed.
I currently have 65GB on a very active node that hasn’t been offline in a while.
The payment for this amount to stay in the trash for 1 week would be 0.024$… Can’t say it’s nothing but it’s not much either.
However, the idea is that most pieces in the trash are only there because a node failed to delete it when the request from the satellite arrived. So basically it’s your node’s fault that there is data that shouldn’t be there. And therefore it is probably not paid. Especially as it could be used to cheat the system if you compile your own software that ignores all deletes and just moves the file to the trash.
Apart from that I am actually suprised that I have 65GB in my trash! My node is quite performant so it shouldn’t have missed that many deletes.
1 Like
Do you mean if such an empty bloom filter is sent or during normal operation? During normal operation data only goes to trash if your node was either offline or unable to respond to a delete request in time. I doubt you’ll be paid for that data, since the satellite has already marked it as deleted and informed your node of that.
I have over 158 GB in trash sitting there for a long time and it just increases over time, which to me is wasted space hosted on the node.
I actually checked this morning and it’s now 169.8 GB…
The issue that I see is, you don’t get paid upload ingress and if data is uploaded and deleted, uploaded and deleted… you are utilizing a lot of bandwidth, hdd writes and deletes and SNO’s are not paid for X GB’s of data.
Also if GB’s of data sits in the trash folder for 10+ days, again this is wasted storage not available to SNO.
Looking at OP’s post I also checked my log and I didn’t find any type of error. So I don’t have the reported issue but I have noticed large amounts of data sitting in the trash folder for a long time.
date format: DD/MM/YYYY
I actually didn’t notice this until now, but some of these folders are VERY VERY OLD… 
Folder age doesn’t indicate anything, the file age is relevant.
It’s not a problem if customers upload and delete and upload and delete… That’s how the system is designed and is a legitimate usage. You still get paid for the stored data.
However, I agree that it needs investigating why there are so many pieces in the trash although a node doesn’t see an error and is online all the time. Mine still has 67GB.
But I think they are/will check it out.
Node has to delete a piece under a time limit if it doesn’t then it gets moved in to trash.
The folders are created only once. There are 1024 folders per satellite that will be reused over and over. Deleting them would be a waste of resources since those same folders will be needed again and they don’t take up space anyway. The files are deleted after 7 days. Normal upload and delete cycles don’t end up in trash. So yes you won’t get paid much if data is uploaded and immediately deleted. But it also won’t end up in trash taking up space.
1 Like
Trash folder now has grown to 273.6 GB
That’s not true, there are no files older than 8 days in there. Check for yourself with
for i in {1..10}; do c=$(find /mnt/storagenode1/node/storage/trash/ -mindepth 1 -type f -mtime -$i |wc -l); echo "Day $i: $c"; done
Of course, you’ll need to change the path to your trash folder accordingly.
So how to interpret this output of your command:
for i in {1..30}; do c=$(find /media/STORJ3/STORJ/storage/trash/ -mindepth 1 -type f -mtime -$i |wc -l); echo "Day $i: $c"; done
Day 1: 2186
Day 2: 2966
Day 3: 4677
Day 4: 25144
Day 5: 25144
Day 6: 25144
Day 7: 26374
Day 8: 26374
Day 9: 26374
Day 10: 26374
There are 2186 files younger than 1 day, 2966 files younger than 2 days, etc. Files younger than 1 days are of course also younger than 2 days, that’s why the numbers increase.