Removing data for disqualified satellite

When a node is disqualified on a satellite the data still remains on the node. SNO is forced to figure out which folder belongs to which satellite and then take necessary action. It is still risky if not done with utmost accuracy. This should be done by the software itself.

Let GC move it to the trash and be deleted after 7 days or in case of rare errors, restore it.

So there’s no way for Storj to free up space illegally occupied by disqualified satellites? Are dead storage blocks destined to hang around forever stealing precious space which could be allocated to not disqualified satellites? I can’t believe that storj node software doesn’t know which blocks belong to which satellites.

The node should know the data to satellite relationship since it is able to display the total data stored/used by each satellite (additionally, each satellite gets its own folder under storage/blobs, you will need to identify which folder goes to which satellite though).

This is likely just a case of ‘not my problem’ that only affects a few people and has therefore been given low developer priority (it is also high risk to low reward, as getting it wrong could wipe legitimate data from many/all satellites while only helping a few less reliable nodes). For completeness sake, it would be nice to have at some point though. StorJ does accept external pull requests if anyone wants to code it themselves (StorJ Contributors License Agreement (CLA))

Reference to finding folder to satellite mapping:

This is more elaborate

Maybe SNO should be able to set the days or set a flag in config.yaml wether or when automated deletion will happen. I think it would be a bad experience for a SNO if Storj decides to reinstate a node but data has already been wiped.

2 Likes

This is why the software needs to take care of it with no interference from SNO.

1 Like

The software should wipe it, but I think the SNO should be able to tell when.

2 Likes

What will that achieve ?

What if the node shall be reinstated on day 8? I understand you suggestion that after 7 days the data gets deleted automatically.

Reinstatement is done by Storj so they are aware that this decision needs to be taken before 7 days. I mentioned 7 days because trash is emptied after 7 days.

I am not sure if that is a realistic duration.
From what I understand reinstating a node mainly happens if disqualification was an error mainly due to a software bug.
This means: SNO needs to note node has been disqualified. Then SNO probably asks in the forum. Someone at Storj has to pick that up and if there is a bug has to find and confirm it.
7 days simply sound very short for these processes to take place. That’s all.

I totally agree that current manual data removal is a pain with finding the right folders etc. and that it shall be improved.
What would make that already much easier is if there was some button in the dashboard next to the disqualification message that SNO could press and that would perform or initiate the deletion process.

I agree 7 days is short and auto reinstatement will only take place if its a major bug and many SNOs are affected.

That would be in version 36.1.1 along with notifications explaining why node was disqualified with an excerpt from satellite log.

1 Like

Now look how long it took from disqualification to reinstating in this case: Got Disqualified from saltlake - #116 by jammerdan

I really don’t think there should be an automatic deletion after a fixed period of time unless the SNO has acknowledged.

1 Like

Well another way to look at that would be that if Storj knows there is a time limit, they will have an incentive to resolve these issues quicker. Worst case they could always disable final deletion while fixes are being worked on if they can’t disable it in time.

Because the other side of the equation is that if there is no automatic clean up, people would be incentivized to do it themselves and they can shoot themselves in the foot if they aren’t aware Storj is working on reversing something. In this case, that would have really sucked.

I would much prefer to just be able to tell anyone to never delete anything manually. But as it is now, SNOs won’t like permanently wasting space on satellites that disqualified them.

1 Like

I see that, but this also plays against the SNO who would need to note, investigate and report the disqualification almost instantly. I don’t think that’s a good way to go.

I’m not sure that really applies. The only scenario in which such reversals ever happen are when there are wider spread problems. They never do it for individual cases to begin with (and for good reasons). And with wider spread problems you can be sure that other SNOs will notify them of the issue.

But I agree that it’s not that clean cut what the right method is. I would just prefer for it to be in control of the satellite operator so that they can determine that the node is definitively not going to be reinstated. SNOs themselves can never be sure of that.

I am not against the satellite operator to have the final decision. I am against removing the data too early in an automatic fashion before the final decision has been made.

1 Like