How To Forget Untrusted Satellites

[clement]

Following the announcement to decommission Europe-north-1 and US2 satellites, manually deleting the remaining data of these satellites from the node can be a tedious task as it requires finding the blobs folder name for each satellite, so we wanted to improve the storagenode CLI to semi-automate cleaning up a satellite data after it is marked as untrusted.

The storagenode CLI (versions from 1.88.0) provides a forget-satellite subcommand to remove satellites from the trust cache and clean up the available data.

• General Usage

  storagenode forget-satellite [satelliteID] --config-dir <config-dir> --identity-dir <identity-dir>
  

• You can specify multiple satellite IDs to remove multiple satellites at once.

   storagenode forget-satellite [satelliteID1] [satelliteID2] [satelliteID3] --config-dir <config-dir> --identity-dir <identity-dir>
  

• To remove all untrusted satellites, you can use the --all-untrusted flag.

   storagenode forget-satellite --all-untrusted --config-dir <config-dir> --identity-dir <identity-dir>
  

Docker nodes

For storagenodes running in a docker container, you can use the following command to remove all untrusted satellites.

docker exec -it storagenode /app/config/bin/storagenode forget-satellite --all-untrusted --config-dir <config-dir> --identity-dir <identity-dir>

Or specify the satellite IDs to remove.

docker exec -it container_name /app/config/bin/storagenode forget-satellite [satelliteID1] [satelliteID2] [satelliteID3] ...

Cleaning Up Data for the Decommissioned Satellites

Method 1

1. Add the satellites (Europe-north-1 and US2) to the exclusions list in your storagenode’s config.yaml file.

   storage2.trust.exclusions: "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB@europe-north-1.tardigrade.io:7777,12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo@us2.storj.io:7777"
   

2. Run the storagenode forget-satellite command

   storagenode forget-satellite --all-untrusted --config-dir /path/to/config --identity-dir /path/to/identity
   

   Or for docker nodes:

   docker exec -it storagenode /app/config/bin/storagenode forget-satellite --all-untrusted --config-dir config --identity-dir identity
   

3. Restart the node

Method 2

You can directly specify the satellite IDs of the decommissioned satellites as arguments and --force flag to force cleanup

storagenode forget-satellite --force 12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB  12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo   --config-dir /path/to/config --identity-dir /path/to/identity

For docker nodes:

docker exec -it storagenode /app/config/bin/storagenode forget-satellite --force 12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB  12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo   --config-dir config  --identity-dir identity

[zip]

@clement

storagenode forget-satellite --all-untrusted --config-dir <config-dir> --identity-dir <identity-dir>

Can this command be issued while storagenode binary process is already running and serving requests?

[ACarneiro]

Thank you for that, guys!
As I still have a lot of free space on my nodes I think I’ll wait a few weeks to make sure everything works fine and no bugs are uncovered before doing mine

[jammerdan]

@clement

Method 1

1. Add the satellites (Europe-north-1 and US2) to the exclusions list in your storagenode’s config.yaml file.

storage2.trust.exclusions: "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB@europe-north-1.tardigrade.io:7777,1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE@saltlake.tardigrade.io:7777"

2. Run the storagenode forget-satellite command

storagenode forget-satellite --all-untrusted --config-dir /path/to/config --identity-dir /path/to/identity

Or for docker nodes:

docker exec -it storagenode ./storagenode forget-satellite --all-untrusted --config-dir config --identity-dir identity

3. Restart the node

Method 2

You can directly specify the satellite IDs of the decommissioned satellites as arguments and --force flag to force cleanup

storagenode forget-satellite --force 12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB  1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE   --config-dir /path/to/config --identity-dir /path/to/identity

For docker nodes:

docker exec -it storagenode ./storagenode forget-satellite --force 12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB  1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE   --config-dir config  --identity-dir identity

I think it is important to note that 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE is not a satellite to be decommissioned.

The satellite ids should be:

• 12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB for Europe-north-1

• 12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo for US2

[clement]

@zip Yes, you can run this command while the process is still running.

[clement]

Oops! It was a mistake. Thanks @jammerdan! The post has been updated

[penfold]

If a new node is created as of today will it still add Europe-north-1 and US2?

[clement]

Yes they will be in the trust cache since we have EUN and US2 listed at https://storj.io/dcs-satellites, unless these two satellites are explicitly excluded when setting up the node but there won’t be any new orders from these satellites.

These satellites will be removed from the storj.io/dcs-satellites source soon.

[Walter1]

Is there a check to not forget currently running satellites? Like you accidently typed in US1 as the satellite to forget but you didn’t even trigger the GE or put it into the untrusted list.

[snorkel]

Shouldn’t be this handled by the network? Not to put preessure on us, making changes that can ruin the entire node by mistake?
If the nodes will be removed from the official DCS trust list, that can be the trigger to remove the data on our end, without us making any changes to settings.
Here, only 7% of SNOs enter and read posts. The rest are left with unwanted data.

[daki82]

I have 2 nodes, at 1.87.1 one has 4 blob folders, one has 6.

will the one with 6 be able to delete the decommissioned 2 soon?

fine, i will answer myself, yes its looking good, the dashboard already does not show the stats but the blobs folders are still there. will wait for GC to get work done.

[penfold]

That seems somewhat ass about. I would have thought the first step would have been to avoid any new nodes creating the mess that then needs to be cleaned up.

[Walter1]

Some of my nodes are having now the version 1.88.3 and I tried to forget. Unfortunately I’m getting the error message that the identity couldn’t be loaded. It is there and the right folder but somehow not fetching the identity.cert.

[pietro]

I don’t know which is your setup, but If your node is running inside a container you must use the path inside the container, not the path on the host

[Walter1]

It is Docker and yes leaving it as /app/identity works fine. Now it’s working with this command:

docker exec -it storagenode /app/storagenode forget-satellite 12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB 12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo --identity-dir /app/identity --config-dir /app/config

Unfortunately even after decommissioning the entries still remain on the dashboard. So leave them in the untrust seems to be better:

And the to be decommissioned satellites must be in the untrusted-list:

[Egaria]

Good morning,

I would like to know if after executing this command, the files of the 2 satellites are deleted.

Because it’s been 24 hours since I ran this command and the files are still there.

[Alexey]

did you try with --force?

@clement

Method 2

You can directly specify the satellite IDs of the decommissioned satellites as arguments and --force flag to force cleanup

@clement

For docker nodes:

docker exec -it storagenode ./storagenode forget-satellite --force 12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB  12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo   --config-dir config  --identity-dir identity

[Egaria]

The command executed more than 48 hours ago must have passed because I just relaunched it and in the logs it shows me this, and the files of the 2 satellites have been deleted

Thanks

[DanielTB80]

What are the commands to remove the satellites through the GUI (Powershell)?

[Alexey]

Hello @DanielTB80,
Welcome to the forum!

From the first topic

@clement

storagenode forget-satellite --force 12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB  12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo   --config-dir /path/to/config --identity-dir /path/to/identity

You should provide paths to the config file and to the identity. If you didn’t move the identity, the default location is in %AppData%\Storj\Identity\storagenode, so the command would be:

& "C:\Program Files\Storj\Storage Node\storagenode.exe" forget-satellite --force 12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB 12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo  --config-dir "C:\Program Files\Storj\Storage Node\\" --identity-dir "$env:AppData\Storj\Identity\storagenode\\"