There are like 2917 unique operators and emails. I don’t know the ban limits, but this seems like a lot too.
Sorry but that is not accurate. Those are only the ones that met the payout threshold, there are many that did not and therefore did not get payout in this particular month.
3 Likes
I was not suggesting that you send an email specific to this. However there are routine emails that go out which this could be (could have been) included in. Likewise, you obviously have the ability to push alerts/announcements to SNO’s via the SNO interface. Something important like this could be pushed out through that.
Regardless of means, the point is that this forgetting untrusted satellites–which appears to not be automatic–should be made known more directly rather than expecting every SNO to monitor this particular site.
My opinion…thank you for considering it.
1 Like
It can send millions, but if most of it would be bounced or marked as a spam… well, we will not be able to send any (no, we could try, but all these emails will be bounced even before reach users).
No, as far as I know, only hardcode like you saw regarding changes in payouts.
What’s in the Notification button is generated by the node itself, and they are limited to some messages (like suspended, disqualified, etc.)
Easy to use docker run command to forget all 3 test satellites, below, after Graceful Exit.
You should remove the last one aka Saltlake, if you didn’t Graceful Exited from it.
Also, check the server private address in run or config and addapt the ports accordingly.
This command is pretty much like the Graceful Exit one, except there you use exit-satellite.
I don’t know if db-es were moved, you should also use this flag:
--database-dir /app/dbs
EDIT: The database-dir flag is not supported, so you can skip it. It works without it.
# Node 1:
docker exec -it storagenode /app/storagenode forget-satellite \
--force \
12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB \
12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo \
1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE \
--config-dir /app/config \
--identity-dir /app/identity \
--server.private-address 127.0.0.1:7778
# Node 2:
docker exec -it storagenode2 /app/storagenode forget-satellite \
--force \
12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB \
12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo \
1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE \
--config-dir /app/config \
--identity-dir /app/identity \
--server.private-address 127.0.0.1:7779
You don’t need to specify anything in config.
To check the status, see next posts. After it shows success, you should stop and restart the node. I encountered some problems so do my way and all will be ok:
- run the command;
- check the status until it’s shows success;
- wait one more minute;
- stop the node;
- restart the node;
- wait 2 minutes;
- check the dashboard.
1 Like
Question 1:
How can I check the status of forget-satellite command?
Is there a forget-status command, like for Graceful Exit?
Does the log shows something?
Question 2:
What happens if the node restarts or machine reboots when forget satellite is still procerssing?
Question 3:
If I moved databases to another drive, should I specify the new location of db-es too, like?
--database-dir /app/dbs
Question 4:
If I deleted the folders manualy, I set the satellites as untrusted in config and restart the node, do I must run forget-satellite too? Does it do more than just removing the folders?
No, as far as I know. It’s working, then exiting.
I guess you may try to execute it again.
doesn’t matter, the main point - the node should be running, because this command executed via API at the end.
You can do it. This command just will error out, that there is nothing to delete if you used a --force flag, but it also should clean databases.
1 Like
Yes there’s a forget-satellite-status command to check the progress
4 Likes
So I think it should look like this:
# Node 1:
docker exec -it storagenode /app/storagenode forget-satellite-status \
--config-dir /app/config \
--identity-dir /app/identity \
--server.private-address 127.0.0.1:7778
# Node 2:
docker exec -it storagenode2 /app/storagenode forget-satellite-status \
--config-dir /app/config \
--identity-dir /app/identity \
--server.private-address 127.0.0.1:7779
1 Like
So I tested my commands as above and they deleted the untrusted sats data including payout from dashboard and databases.
Steps:
I Graceful Exited all 3 sats.
I stopped the node.
I manualy deleted all the folders of the untrusted satellites.
I edited the config and added the untrusted satellites.
I restarted the node.
I tryed the --all-untrusted method, but it says that there are no untrusted satellites.
I tryed the --force ... sats xxx, yyy, zzz method and it worked.
Status shows In Progress and, when it’s finished, it shows Successful. It took like 10 seconds on the slowest node.
I restarted the node.
1 Like
For that you need to explicitly add them to the untrusted list in your config or with a --storage2.trust.exclusions option after the image name in your docker run command.
In the latter case it’s simpler to list them in the forget-satellite command of course.
Yes, that is what I did. I added this in config and restarted:
storage2.trust.exclusions: "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB@europe-north-1.tardigrade.io:7777,12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo@us2.storj.io:7777,1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE@saltlake.tardigrade.io:7777"
Using the first method, didn’t work. It sais “… no satellites in untrused list…”.
This is weird. It’s tested and must work. Notified the team.
After forget sat, I get this error in log, probably linked to Saltlake (?):
ERROR console:endpoint failed to encode json response {"Process": "storagenode", "error": "consoleapi payouts: json: unsupported value: NaN", "errorVerbose": "consoleapi payouts: json: unsupported value: NaN\n\tstorj.io/storj/storagenode/console/consoleapi.(*Payout).PayoutHistory:198\n\tnet/http.HandlerFunc.ServeHTTP:2136\n\tgithub.com/gorilla/mux.(*Router).ServeHTTP:210\n\tnet/http.serverHandler.ServeHTTP:2938\n\tnet/http.(*conn).serve:2009"}
This is the output of forget-satellite for all 3 test sats, for anyone wondering:
2024-05-12T07:12:11Z INFO forgetsatellite:endpoint initializing forget satellite {"Process": "storagenode", "satelliteID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "action": "InitForgetSatellite"}
2024-05-12T07:12:11Z INFO forgetsatellite:endpoint initializing forget satellite {"Process": "storagenode", "satelliteID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "action": "InitForgetSatellite"}
2024-05-12T07:12:11Z INFO forgetsatellite:endpoint initializing forget satellite {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "action": "InitForgetSatellite"}
2024-05-12T07:12:39Z INFO forgetsatellite:cleaner removing satellite from trust cache {"Process": "storagenode", "satelliteID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo"}
2024-05-12T07:12:39Z INFO forgetsatellite:cleaner cleaning up satellite data {"Process": "storagenode", "satelliteID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo"}
2024-05-12T07:12:39Z INFO forgetsatellite:cleaner cleaning up the trash {"Process": "storagenode", "satelliteID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo"}
2024-05-12T07:12:39Z INFO forgetsatellite:cleaner removing satellite info from reputation DB {"Process": "storagenode", "satelliteID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo"}
2024-05-12T07:12:39Z INFO forgetsatellite:cleaner removing satellite v0 pieces if any {"Process": "storagenode", "satelliteID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo"}
2024-05-12T07:12:39Z INFO forgetsatellite:cleaner cleanup completed {"Process": "storagenode", "satelliteID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo"}
2024-05-12T07:12:39Z INFO forgetsatellite:cleaner removing satellite from trust cache {"Process": "storagenode", "satelliteID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB"}
2024-05-12T07:12:39Z INFO forgetsatellite:cleaner cleaning up satellite data {"Process": "storagenode", "satelliteID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB"}
2024-05-12T07:12:39Z INFO forgetsatellite:cleaner cleaning up the trash {"Process": "storagenode", "satelliteID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB"}
2024-05-12T07:12:39Z INFO forgetsatellite:cleaner removing satellite info from reputation DB {"Process": "storagenode", "satelliteID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB"}
2024-05-12T07:12:39Z INFO forgetsatellite:cleaner removing satellite v0 pieces if any {"Process": "storagenode", "satelliteID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB"}
2024-05-12T07:12:39Z INFO forgetsatellite:cleaner cleanup completed {"Process": "storagenode", "satelliteID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB"}
2024-05-12T07:12:39Z INFO forgetsatellite:cleaner removing satellite from trust cache {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-12T07:12:39Z INFO forgetsatellite:cleaner cleaning up satellite data {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-12T07:12:39Z INFO forgetsatellite:cleaner cleaning up the trash {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-12T07:12:39Z INFO forgetsatellite:cleaner removing satellite info from reputation DB {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-12T07:12:39Z INFO forgetsatellite:cleaner removing satellite v0 pieces if any {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-12T07:12:39Z INFO forgetsatellite:cleaner cleanup completed {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
or just Saltlake:
2024-05-12T07:12:11Z INFO forgetsatellite:endpoint initializing forget satellite {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "action": "InitForgetSatellite"}
2024-05-12T07:12:39Z INFO forgetsatellite:cleaner removing satellite from trust cache {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-12T07:12:39Z INFO forgetsatellite:cleaner cleaning up satellite data {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-12T07:12:39Z INFO forgetsatellite:cleaner cleaning up the trash {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-12T07:12:39Z INFO forgetsatellite:cleaner removing satellite info from reputation DB {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-12T07:12:39Z INFO forgetsatellite:cleaner removing satellite v0 pieces if any {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-12T07:12:39Z INFO forgetsatellite:cleaner cleanup completed {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
When you got this response? Before or after the forget-satellite command?
After forget-satellite and node restart. It has only 1 entry.
And the Payout history on Dashboard is gone for every month… 
It should look like this, maybe without Saltlake:
Sould I do something to databases? Like some cleaning or something? It’s not so important for me if the rest is functioning properly, but maybe others will start complaining.
I believe if you called a forget-satellite command, all data related to them will be purged.
Yes, but there are still 3 active sats with history, that is missing on display, eventhough in the upper table it is showed, for each month. This is for apr.2024.
And?
What’s the point? I just missed it, sorry…
The Payout history panel is empty after forget sat. There should be the other sats at least. I hope the database is not broken in some way.