How To Forget Untrusted Satellites

Node Operators FAQ
1

Following the announcement to decommission Europe-north-1 and US2 satellites, manually deleting the remaining data of these satellites from the node can be a tedious task as it requires finding the blobs folder name for each satellite, so we wanted to improve the storagenode CLI to semi-automate cleaning up a satellite data after it is marked as untrusted.

The storagenode CLI (versions from 1.88.0) provides a forget-satellite subcommand to remove satellites from the trust cache and clean up the available data.

  • General Usage

    storagenode forget-satellite [satelliteID] --config-dir <config-dir> --identity-dir <identity-dir>

  • You can specify multiple satellite IDs to remove multiple satellites at once.

     storagenode forget-satellite [satelliteID1] [satelliteID2] [satelliteID3] --config-dir <config-dir> --identity-dir <identity-dir>

  • To remove all untrusted satellites, you can use the --all-untrusted flag.

     storagenode forget-satellite --all-untrusted --config-dir <config-dir> --identity-dir <identity-dir>

Docker nodes

For storagenodes running in a docker container, you can use the following command to remove all untrusted satellites.

docker exec -it storagenode ./storagenode forget-satellite --all-untrusted --config-dir <config-dir> --identity-dir <identity-dir>

Or specify the satellite IDs to remove.

docker exec -it container_name ./storagenode forget-satellite [satelliteID1] [satelliteID2] [satelliteID3] ...

Cleaning Up Data for the Decommissioned Satellites

Method 1

  1. Add the satellites (Europe-north-1 and US2) to the exclusions list in your storagenode’s config.yaml file.

    storage2.trust.exclusions: "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB@europe-north-1.tardigrade.io:7777,12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo@us2.storj.io:7777"

  2. Run the storagenode forget-satellite command

    storagenode forget-satellite --all-untrusted --config-dir /path/to/config --identity-dir /path/to/identity

    Or for docker nodes:

    docker exec -it storagenode ./storagenode forget-satellite --all-untrusted --config-dir config --identity-dir identity

  3. Restart the node

Method 2

You can directly specify the satellite IDs of the decommissioned satellites as arguments and --force flag to force cleanup

storagenode forget-satellite --force 12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB  12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo   --config-dir /path/to/config --identity-dir /path/to/identity

For docker nodes:

docker exec -it storagenode ./storagenode forget-satellite --force 12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB  12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo   --config-dir config  --identity-dir identity
12 Likes
2

Can this command be issued while storagenode binary process is already running and serving requests?

3

Thank you for that, guys!
As I still have a lot of free space on my nodes I think I’ll wait a few weeks to make sure everything works fine and no bugs are uncovered before doing mine :slight_smile:

3 Likes
4

I think it is important to note that 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE is not a satellite to be decommissioned.

The satellite ids should be:

  • 12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB for Europe-north-1

  • 12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo for US2

3 Likes
5

@zip Yes, you can run this command while the process is still running.

2 Likes
6

Oops! It was a mistake. Thanks @jammerdan! The post has been updated

2 Likes
7

If a new node is created as of today will it still add Europe-north-1 and US2?

8

Yes they will be in the trust cache since we have EUN and US2 listed at https://storj.io/dcs-satellites, unless these two satellites are explicitly excluded when setting up the node but there won’t be any new orders from these satellites.

These satellites will be removed from the storj.io/dcs-satellites source soon.

10

Is there a check to not forget currently running satellites? Like you accidently typed in US1 as the satellite to forget but you didn’t even trigger the GE or put it into the untrusted list.

11

Shouldn’t be this handled by the network? Not to put preessure on us, making changes that can ruin the entire node by mistake?
If the nodes will be removed from the official DCS trust list, that can be the trigger to remove the data on our end, without us making any changes to settings.
Here, only 7% of SNOs enter and read posts. The rest are left with unwanted data.

12 Likes
12

I have 2 nodes, at 1.87.1 one has 4 blob folders, one has 6.

will the one with 6 be able to delete the decommissioned 2 soon?

fine, i will answer myself, yes its looking good, the dashboard already does not show the stats but the blobs folders are still there. will wait for GC to get work done.

13

That seems somewhat ass about. I would have thought the first step would have been to avoid any new nodes creating the mess that then needs to be cleaned up.

14

Some of my nodes are having now the version 1.88.3 and I tried to forget. Unfortunately I’m getting the error message that the identity couldn’t be loaded. It is there and the right folder but somehow not fetching the identity.cert.

grafik
grafik2345×113 12 KB

15

I don’t know which is your setup, but If your node is running inside a container you must use the path inside the container, not the path on the host

2 Likes
16

It is Docker and yes leaving it as /app/identity works fine. Now it’s working with this command:

docker exec -it storagenode /app/storagenode forget-satellite 12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB 12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo --identity-dir /app/identity --config-dir /app/config

grafik
grafik1815×378 45.7 KB

Unfortunately even after decommissioning the entries still remain on the dashboard. So leave them in the untrust seems to be better:

grafik
grafik1278×513 41.6 KB

And the to be decommissioned satellites must be in the untrusted-list:

grafik
grafik1660×244 20.1 KB

1 Like
17

Good morning,

I would like to know if after executing this command, the files of the 2 satellites are deleted.

image
image1887×37 4.11 KB

Because it’s been 24 hours since I ran this command and the files are still there.

18

did you try with --force?

1 Like
19

The command executed more than 48 hours ago must have passed because I just relaunched it and in the logs it shows me this, and the files of the 2 satellites have been deleted

image
image1869×53 7.91 KB

Thanks

1 Like
20

What are the commands to remove the satellites through the GUI (Powershell)?

1 Like
21

Hello @DanielTB80,
Welcome to the forum!

From the first topic

You should provide paths to the config file and to the identity. If you didn’t move the identity, the default location is in %AppData%\Storj\Identity\storagenode, so the command would be:

& "C:\Program Files\Storj\Storage Node\storagenode.exe" forget-satellite --force 12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB 12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo  --config-dir "C:\Program Files\Storj\Storage Node\\" --identity-dir "$env:AppData\Storj\Identity\storagenode\\"
1 Like