One more trash problem…
It seems that after GE of SL sat, the data from the exited satellite is moved to trash as expected, but the database with disk usage is not updated accordingly.
Yesterday GE finished successfuly and the cleaners removed the data blob directory, which has been moved to trash blob directory, as expected.
But the Dashboard still shows the trashed data in the data directory, not in trash.
There is like 1.3-1.5TB of SL data. I kow that a successful startup piece scan will correct things, but this should not be necessary.
1 Like
I tested with storj-up, after GE is successfully completed, the node removes blobs data permanently, not to the trash.
Is the trash usage consistent with the OS report (in SI units)?
Is this the command?
du --si -s /mnt/hdd2p1/Storj2/storage/trash/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa
Yes.
I tried to reproduce when the trash is exist
Storage Node Dashboard ( Node Version: v0.0.0 )
======================
ID 1Ut4Jhqqp7Y1gHz7Rgwc84fSQXZDtWsarzMtcMPRiU6cVJon6e
Status ONLINE
Uptime 4m55s
Available Used Egress Ingress
Bandwidth N/A 67.25 MB 0 B 67.25 MB (since Dec 1)
Disk 0.93 GB 50.44 MB
Internal :30102
External storagenode11:30101
$ docker compose exec storagenode11 du --si -d 1 /var/lib/storj/.local/share/storj/storagenode/storage/
WARN[0000] /home/ubuntu/build/docker-compose.yaml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion
13k /var/lib/storj/.local/share/storj/storagenode/storage/piece_expirations
17M /var/lib/storj/.local/share/storj/storagenode/storage/trash
4.1k /var/lib/storj/.local/share/storj/storagenode/storage/temp
17k /var/lib/storj/.local/share/storj/storagenode/storage/hashstore
33k /var/lib/storj/.local/share/storj/storagenode/storage/filestatcache
51M /var/lib/storj/.local/share/storj/storagenode/storage/blobs
71M /var/lib/storj/.local/share/storj/storagenode/storage/
$ docker compose exec storagenode11 storagenode exit-satellite
WARN[0000] /home/ubuntu/build/docker-compose.yaml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion
2024-12-10T04:43:57Z INFO process/exec_conf.go:318 Configuration loaded {"Process": "storagenode", "Location": "/var/lib/storj/.local/share/storj/storagenode/config.yaml"}
2024-12-10T04:43:57Z INFO process/tracing.go:73 Anonymized tracing enabled {"Process": "storagenode"}
2024-12-10T04:43:57Z DEBUG tracing collector monkit-jaeger@v0.0.0-20240221095020-52b0792fa6cd/thrift.go:149 started {"Process": "storagenode"}
2024-12-10T04:43:57Z INFO storagenode/cmd_gracefulexit.go:113 Identity loaded. {"Process": "storagenode", "Node ID": "1Ut4Jhqqp7Y1gHz7Rgwc84fSQXZDtWsarzMtcMPRiU6cVJon6e"}
By starting a graceful exit from a satellite, you will no longer receive new uploads from that satellite.
This action can not be undone.
Are you sure you want to continue? [y/n]
:y
Domain Name Node ID Space Used
satellite-api:7777 12whfK1EDvHJtajBiAUeajQLYcWqxcQmdYQU5zX5cCf6bAxfgu4 16.81 MB
Please enter a space delimited list of satellite domain names you would like to gracefully exit. Press enter to continue
:
satellite-api:7777
Domain Name Node ID Percent Complete Successful Completion Receipt
satellite-api:7777 12whfK1EDvHJtajBiAUeajQLYcWqxcQmdYQU5zX5cCf6bAxfgu4 0.00% N N/A
2024-12-10T04:44:09Z DEBUG tracing collector monkit-jaeger@v0.0.0-20240221095020-52b0792fa6cd/thrift.go:206 stopped {"Process": "storagenode"}
$ docker compose exec storagenode11 storagenode exit-status
WARN[0000] /home/ubuntu/build/docker-compose.yaml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion
2024-12-10T04:44:31Z INFO process/exec_conf.go:318 Configuration loaded {"Process": "storagenode", "Location": "/var/lib/storj/.local/share/storj/storagenode/config.yaml"}
2024-12-10T04:44:31Z INFO process/tracing.go:73 Anonymized tracing enabled {"Process": "storagenode"}
2024-12-10T04:44:31Z DEBUG tracing collector monkit-jaeger@v0.0.0-20240221095020-52b0792fa6cd/thrift.go:149 started {"Process": "storagenode"}
2024-12-10T04:44:31Z INFO storagenode/cmd_gracefulexit.go:190 Identity loaded. {"Process": "storagenode", "Node ID": "1Ut4Jhqqp7Y1gHz7Rgwc84fSQXZDtWsarzMtcMPRiU6cVJon6e"}
Domain Name Node ID Percent Complete Successful Completion Receipt
satellite-api:7777 12whfK1EDvHJtajBiAUeajQLYcWqxcQmdYQU5zX5cCf6bAxfgu4 100.00% Y 0a46304402202f7b18d30c6936960c29cdf7046e10e6cdb04b691370550c117bcfd841bcd4ba02201b340e2153e6c933dafa10833e4eb439f851005e1bd4a02513e740a903ae49c51220ffe6c91f476e70f5d71e3c3878e0592f33b82f01ddfa766f406205cbaea373001a203f4e3cd3fae936387aa12c093889dbade50c82e16bb18d668993c11ee1c90d00220b08aa8edfba06109087c261
2024-12-10T04:44:31Z DEBUG tracing collector monkit-jaeger@v0.0.0-20240221095020-52b0792fa6cd/thrift.go:206 stopped {"Process": "storagenode"}
$ docker compose exec storagenode11 du --si -d 1 /var/lib/storj/.local/share/storj/storagenode/storage/
WARN[0000] /home/ubuntu/build/docker-compose.yaml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion
13k /var/lib/storj/.local/share/storj/storagenode/storage/piece_expirations
51M /var/lib/storj/.local/share/storj/storagenode/storage/trash
4.1k /var/lib/storj/.local/share/storj/storagenode/storage/temp
17k /var/lib/storj/.local/share/storj/storagenode/storage/hashstore
33k /var/lib/storj/.local/share/storj/storagenode/storage/filestatcache
4.1k /var/lib/storj/.local/share/storj/storagenode/storage/blobs
54M /var/lib/storj/.local/share/storj/storagenode/storage/
Seems you are correct, some data moved to the trash, but not all. Looks like the trash has been removed, but blobs moved instead.
And only in the case, if there was a trash.
It seems that after 2 days, the graphs corrected and was shown the correct space, in the left and in the right. The du command showed 27M in trash. So I think I rised a false alarm. There was no trash to display in the graph.
But maybe you discovered something I missed?
Anyway, does your test proves that GE finish triggers a cleanup job that moves all data to trash for the exited sat?
Because, the only thing that could do that would be a bloom filter, but I doubt that both of us got one just after GE finished. To big of a coincidence. Or maybe the satellite sends the bloom filter with zero pieces once the GE finished? 
I discovered, that if the node has something in the trash, when you call a GE, it deletes a trash, but moves all from the blobs to the trash.
If there is nothing in trash initially, then the blobs just deleted.
For me it looks weird, but perhaps a low priority: if you GE from the satellite, you likely delete all remaining data of that satellite anyway.
I reproduced this in storj-up where I control BF generation and GC process. The initial trash has been created this way, but later, no more BF were sent, and I call a GE on one of the nodes with trash and on the another one with a TTL data, but without a trash.
I didn’t touched the data or trash. The data blob deleted itself. I’m watchind the trash for one more week to see if that cleans up too.
It should, because it’s a storagenode’s side.