How To Forget Untrusted Satellites

[Vadim]

Ok thank you for help, then for now I will hardcode this 2 sattelites to toolbox, that people can remove them with 1 click.

[Vadim]

Win GUI users can now clean it all with one click.

Win GUI Storj Node Toolbox - Node Operators - Storj Community Forum (official)
Toolbox V1.9

[Nokotopulele]

Hello, my command did not return any error, but it is processing since a while. Is it normal ? I should just let it like that ?

[Vadim]

yes let it like that for some time. it takes time to delete gigabytes of data.

[Nokotopulele]

Well, actually I have 5 TB of data to clean…

[Alexey]

For these satellites?! They should not have more than a few GB.

[Nokotopulele]

I cant check the data on those satellite anymore, they dont appear on dashboard. But it’s almost 5 TB yes. Cant tell you why, the node is old maybe

[Alexey]

My nodes are old enough (2019), but I have had only few GB from these satellites after their shutdown.

[Nokotopulele]

Maybe depending on your location ?
Anyway it seems to work fine, already 150GB have been deleted this night.

[Alexey]

I hope that’s all. You would need to check a blobs folder.
See

Satellite info (Address, ID, Blobs folder, Hex)

I noticed one of my responses in an old topic being linked to quite frequently listing this info. I figured it might be a good idea to post it in an easier to find place where I can also keep it up to date. Active satellites Customer facing Satellite Address: ap1.storj.io:7777 (formerly: asia-east-1.tardigrade.io:7777) Satellite ID: 121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 Sat.ID hex: 84A74C2CD43C5BA76535E1F42F5DF7C287ED68D33522782F4AFABFDB40000000 Blobs Folder: qstuylguhrn2ozjv4h…

[Nokotopulele]

There is 855 folders of more than 4000 files each in the europe-north-1 blob folder, before cleaning starts it was 1024 folders

Already 400GB of data cleaned since I runned it yesturday. Still processing

I dont know how I can calculate the space used by the blob folder, since I cant see the 2 satellite in dashboard anymore. Windows calculation doesnt work, too many files, it would take almost the same time than deleting them. I tried with powershell aswell, the process wont end.

[nerdatwork]

@Nokotopulele

I dont know how I can calculate the space used by the blob folder

You know the space used by non-decommissioned satellites. Subtracting that from total space gives you space used by decommissioned satellite.

In short, Total space - (saltlake + ap1 + us1 + eu1 + trash) = Space used by decommissioned satellite

[Nokotopulele]

Yes, this gived 4.8TB

[unrealSpeedy]

Wait. Storj shuts down the satellites which leads to a lot of trash-Data left on our nodes and we have to manually tidy them up by running a command?
Or did i get this wrong?^^

Are there Plans to do this automatically?

[Alexey]

We deleted all data before shutdown of these satellites. However, if your node wasn’t online when this request has come to your nodes, it can do not have any effect. We run them for several months more to allow to accept all deletion requests. But if the satellites were shutdown, there is no any service, which would send a repeat of the bloom filter to remove all data (with help of a garbage collector).

So, yeah. There is a one command to remove it from your node(s), if it (they) missed the party.

[daki82]

So here is my line that did not retturn an error, as i had still blobs folders of the decomissioned test sats left.
syntax sample
windows GUI, powershell, admin rights, node running, cd to folder where the storagenode.exe is. with

cd “C:\Program Files\Storj\Storage Node”

and:

.\storagenode.exe forget-satellite --force 12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB 12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo --config-dir “C:\Program Files\Storj\Storage Node” --identity-dir “C:\Users\Mini\Documents\storj\Identity\storagenode”

After a day (i let the shell window open, just to be sure), the blob folders of the decommisioned sats were gone. Adapting the paths behind -dir may be sure neccesary.

[BrightSilence]

If you have your logs redirected to a file in the config.yaml, it will apply to this command as well. Add --log.output="stdout" to the command if you want it to be verbose without having to change your config.

@clement I encountered on 2 of my nodes an error that there were no untrusted satellites. Do you know how this could happen? I haven’t done anything different with these 2 nodes, no manual exit or anything. I ended up (carefully) cleaning up the data manually.

I also noticed that because satellite data is removed from the reputation table, my earnings calculator currently throws an error for months where the forgotten satellites were active. I’ll post a fix for that soon. But more importantly, the dashboard pretends those satellites never even existed when you look at those past months and the data is simply omitted. That doesn’t seem right to me.

[Alexey]

@BrightSilence

@clement I encountered on 2 of my nodes an error that there were no untrusted satellites. Do you know how this could happen? I haven’t done anything different with these 2 nodes, no manual exit or anything. I ended up (carefully) cleaning up the data manually.

You need to provide them explicitly either in the command line or with this option:

@clement

• Add the satellites (Europe-north-1 and US2) to the exclusions list in your storagenode’s config.yaml file.

storage2.trust.exclusions: "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB@europe-north-1.tardigrade.io:7777,12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo@us2.storj.io:7777"

To delete data you need to use a --force flag.

@BrightSilence

But more importantly, the dashboard pretends those satellites never even existed when you look at those past months and the data is simply omitted. That doesn’t seem right to me.

Yes, I’m agree with you here. On the multinode dashboard they are some kind of visible but with empty names.

[BrightSilence]

@Alexey

You need to provide them explicitly either in the command line or with this option:

Yeah, I realized that later, but already did manual cleanup. I should have checked back here first. I was just curious how this could have happened in the first place?

@Alexey

Yes, I’m agree with you here. On the multinode dashboard they are some kind of visible but with empty names.

That’s because the data has also been removed from the satellites table, so to the node the stats now belong to a satellite for which it doesn’t know the name. I have a hardcoded fallback in my earnings calculator.

[clement]

@BrightSilence

But more importantly, the dashboard pretends those satellites never even existed when you look at those past months and the data is simply omitted. That doesn’t seem right to me.

I created a ticket for this [storagenode]: Payout history should have previous payout information for untrusted satellites · Issue #6574 · storj/storj · GitHub