Storagenode Recovery Mode

Ideas & Suggestions
1

Short:
Give SNOs the ability to fully restore their node via their identity files if something breaks.

Problem:
Unfortunately, there are many ways a storagenode can fail. (database malformed, hard disk fails, update not testet enough, etc.)

The SNO must then start at 0. This is particularly annoying if the storagenode is already very full and is a risk for any SNO.

It is expensive for Storj because repair traffic is generated.

The advantages:
The SNO is glad that if something goes wrong, not everything is lost.

Repair traffic savings: If the database fails and the SNO uses a database backup, it may be missing 1% of the data. If he now starts in recovery mode, the missing 1% of data would only need to be uploaded to the Storagenode.

If a hard disk fails and the SNO has really lost all data, there would be a lot of repair traffic. However, if the SNO buys a new hard drive within a deadline and replaces it in its server and starts its storagenode in recovery mode with its identity files and re-downloads the entire node, this would have 2 advantages:
SNO: Would not need to start from 0.
Storj: There is no need to pay for expensive repair traffic.

Procedure:
SNO starts in recovery mode.

Storagenode signals all satellites that it is currently in restore mode.

The database is scanned and checks whether all data is available. All files in blobs/temp/trash/garbage that are not refered in the database are deleted.

Transmission to the satellites which data is available. The satellite checks which data is missing and sends it to the storage node. Missing data is transmitted. It proceeds as if the data is sent normally to the storagenode.

After completion, the SNO is informed by e-mail and the SNO is switched off. The SNO now only needs to switch on the storage node as usual.

SNO: Happy noise

2

The missed pieces can be re-generated, but they should be downloaded first. To recover a one file the satellite is need to have 29 pieces of the said file, so it should download them and pay to SNO $10/TB for that. Then it should regenerate remained 51 pieces of the file and upload them to nodes. The payment for repair traffic is taken from the held amount of failed nodes.

  1. Who will pay to SNO for the repair traffic in your case?
  2. Why the satellite should prefer the node which already managed to lost pieces once?
3 Likes
3

Hello,
I really appreciate your quick response.

What I wanted to say is this. It is currently possible for a storagenode to become corrupt and have to start over for many reasons.

I believe this is a risk for a SNO.

For example, if I hold 10 TB and something goes wrong and I have to start over, that would be bad for me.

  1. If I can recover my storagenode by downloading missing parts or the complete node, I would be happy to get paid less for the current month. It’s more important for me not to start from scratch because it can take months to refill your node.

  2. you are right. But I could also just create new identity files and wait to be vetted.

There would be two goals:
Some or all of the data is lost.
The database is corrupt. If the database is corrupt, in my simple world I could just iterate all my blobs and refill the database. Probably not that simple.

Thanks for your time.

4

For honestly your node will be paid less while it vetting. So, the result will be the same.
The recovering is an expensive operation and will be performed anyway, but the satellite operator should pay for that. In your suggestion - from own pocket. If were me - I would not pay that myself, sorry.

The vetting there is for the reason - your node should be vetted to do not lost a significant amount of data.

So the current solution with disqualifying your node and forcing you start from scratch with a new node looks more reliable and coherent.

2 Likes
5

in theory wouldn’t it be better if the same identity was used, then the network would have more information about reliability ofc that would mean that a node might as well start fresh…

i think the knowledge if a node is stable or not should be worth something… ofc the node restorjration should cost as regular tardigrade downloads or something a kin to that…

ofc it wouldn’t exactly be a restoration because the node would get different data, which would be basically taken away from the successful nodes then…

there are some issues with how this should work, but i do kinda like the idea of a restoration … but then again… what does one want to restore, everything aside from the data and id is expendable.
so it would always be data restoration… which entire involved corrupted data which is like poison to the network i would assume (please correct me if i’m wrong)

or a full loss of data which means big downloads…

yeah i kinda have to give it to alexey, i don’t think this makes sense, tho i do think the information of node inreliablity by keeping the ID would be useful… and maybe there could be built some advantages of reusing the same id…

i mean the network needs to be able to rely on the nodes to store data, knowing that a node is more unreliable than another, would be better than a unreliable node entering the network on a new id… and most likely not much wiser…

6

I would like to add an example of a backup. Let’s say I messed something up and had to restore the backup from yesterday. Unless my node was doing a lot of ingress, the difference is probably not that big, so I would like it if there was a way to repair the missing data since the last backup, even though the payment for it would be taken out of my next payout (or more than one if I lost something like 200GB and had zero egress traffic that month).

My node could signal the satellite “hey, I was restored from a backup and do not have anything past timestamp 12345”

1 Like
7

Then we should move it to the voting category?

8

agreed there is merit to the idea, but it would need to be conceptually sketched out…
might work pretty well with snapshots…incase shit hits the fan

9

i would say yes, but i would expect it to be a rather long project, but could be very useful long term and we might need to hear some storj engineers if it’s even remotely possible, but it should be… in theory… xD
i mean no point in voting about projects that might be technically out of reach without many months of work… no matter how beneficial the idea seems.

1 Like
10

Unfortunately it is worth nothing. If your node is failed it is better do not have a dirt mark on node’s reputation. Otherwise it should be accounted too. And as a result - more low reputation.
I believe, start from clean page looks more fair.

1 Like
11

That’s my idea at least, to have a snapshot in case I mess something up. While I use raidz2 for the node and the data would survive a failed hard drive, if the database gets corrupted my node is done and there are examples of SNOs with corrupted databases.

So, I find out that my node is not starting because of a corrupt db or is failing audits, restore the last snapshot and try to recover. The satellite has a list of files my node is supposed to have and could restore the files since last backup and run some more audits on the older data to make sure my node still has it.

After all, if I mess up something and lose my customers emails, the customer would be less unhappy if I could restore a backup and have most of his emails back compared to having no emails at all.

1 Like
12

i’m not thinking from a SNO’s perspective… i’m think from a network data integrity standpoint …
a SNO starting fresh and has a habit of loosing data, can’t be good for the network… then the network cannot adapt to it… with the added information of reusing id’s if a SNO has a habit of loosing data that can be taken into account…

1 Like
13

really… the db can break… i should drive my system more carefully…
but yeah i like that idea… stuff can always go wrong… a database recovery should be a feature if nothing else… and with snapshots we could shot it like hourly or something… and then delete them after a week or whatever… i duno… not really that well versed in snapshots yet…= i got no clue… i took one tho… didn’t save my OS tho… but it might as well have…

14

This seems a bit strange to me:
I have a node with 10TB of data, messed something up and have to restore from a backup, but the backup does not have, say, 100GB of the most recent files. Well, I guess I can just delete the remaining 9.9TB and just set up a new node.

The network cannot trust my node that has lost some data (I do not know how much data you can lose and not be DQ, I guess it matters a lot whether the audits for the missing files come one after the other or are dispersed among the audits for files that still exist), but it can trust my new node?

This is especially if the data loss is because of a corrupt db - well, can I move db to a cluster so it does not get corrupted? No? But if it does get corrupted I have no way of recovering it and my node gets disqualified essentially because of something outside my control.

I understand if my node got DQ because I do not have a second connection, a UPS or a generator and my node spends too much time offline. I understand my node getting DQ because I cheaped out and used RAID5 (or no RAID at all) instead of RADI6 and had two drives fail on me.
That is something I have control over. I do not have control over the database, well, unless I modified the node and ran my own version that can use db cluster.

2 Likes
15

i full agree on that point… it would make sense to be able to do some kind of node restorations, throwing away 9.9tb verifiable good data is a big waste, and not good for network data integrity either.

16

But we do not feel strange if 100GB of data from the RAR (tar.gz) archive were missing and you lose a whole 9.9TB archive, do we? :slight_smile:

If serious, I think the current system can tolerate losing of a few GB, but may be not hundreds.

17

Use the held amount (if any) to restore the data.

1 Like
18

Depends on what was in it and which part of the data was lost. I think with a tar format it would be possible to recover the remaining files :slight_smile:
Also, if I ever decided to have such an archive, I would probably enable RAR ECC and set that to some large number or use separate ECC.

To me, this is more like if I lost your emails from today, I might as well delete the rest of them. Or would you prefer I had a backup from yesterday?

I do not really like this “perfection or nothing” idea - people do make mistakes and a lot of the mistakes are small enough to be possible to recover from. This, however, makes me feel like I am performing surgery - a small mistake and the patient dies. Well, there are many reasons why I am not a doctor and this is one of them.

Especially if it happens because of something I have no control over, for example, db corruption. If the node could use MySQL, I would have set up a cluster a long time ago.

That does not cost anything to the SNO and reduces the held amount, so I am pretty sure Storj would not want to do that.

But hey, you can take it out my next payout (or more than one if one is not enough).

1 Like
19

Yes, but as a satellite operator I should pay to SNO first for the repair from my own pocket. And then, maybe, if your node would still be there from your future payout…
I, as a satellite operator, want to have more guarantees than that.
Would you like to pay in advance to be able to recover?

20

Sure. While it would be annoying to fire up my coin server to send some of the tokens, it would be much less annoying than setting up the node again.

1 Like