When will "Uncollected Garbage" be deleted?

littleskunk · August 9, 2024, 4:19pm

Turns out I am kind of lucky that I was able to observe it on my storage nodes. For the last 2 days there was no entry missing in the TTL DB beside a few hundert caused by a storage node restart. I had to go back 3 days or so to find a good time period.

github.com/storj/storj

Missing inserts into TTL DB

opened 04:11PM - 09 Aug 24 UTC

littleskunk

Bug

I have writen a script to compare my storage node log entries with my TTL db and… it turns out I am missing some inserts. ``` sqlite3 /home/storagenode/ssd/sn8/dbs/piece_expiration.db "SELECT piece_expiration, hex(piece_id) FROM piece_expirations WHERE hex(satellite_id)='7B2DE9D72C2E935F1918C058CAAF8ED00F0581639008707317FF1BD000000000' AND piece_expiration>='2024-09-05 21:00:00' AND piece_expiration<='2024-09-06 23:00:00' ORDER BY piece_expiration;" > ttl.txt for piece in `grep uploaded /home/storagenode/logs/storagenode8.log | grep 2024-08-07 | grep "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE" | grep '"PUT"' | cut -d '"' -f 8` do hex=`echo $piece==== | base32 -d | xxd -ps | tr -d '\n'` ttl=`grep -i $hex ttl.txt` if (($?==0 )) then echo found $piece else echo missing $piece fi done #total sort compare.txt | cut -d ' ' -f 1 | uniq -c 134809 found 18885 missing #ordered by time cat compare.txt | cut -d ' ' -f 1 | uniq -c 577 found 1 missing 1 found 592 missing 2027 found 1 missing 1598 found 613 missing 7786 found 682 missing 2826 found 682 missing 1427 found 738 missing 3565 found 686 missing 3582 found 726 missing 2171 found 729 missing 698 found 717 missing 1468 found 762 missing 23903 found 778 missing 8728 found 639 missing 4339 found 574 missing 1839 found 576 missing 1204 found 673 missing 649 found 1359 missing 3722 found 691 missing 684 found 672 missing 4655 found 1303 missing 14641 found 957 missing 461 found 444 missing 5227 found 283 missing 17 found 126 missing 281 found 367 missing 6305 found 328 missing 3562 found 382 missing 4059 found 295 missing 632 found 326 missing 3472 found 392 missing 1492 found 386 missing 3490 found 405 missing 13721 found ``` I don't see any pattern why this is happening. It could be that what ever is in the cache doesn't get persistet into the TTL DB for some reason. I see no oustanding error message that would explain it. Also in the last 2 days I found no missing pieces while still seeing the usual error messages like upload failed or upload canceld. I am running out of ideas why these inserts don't make it into the TTL DB. I am going to attach my log file and the result file of my comparison. That might help to find the root cause.