Is the Backup of a Node (and its contents) reqired?

[Vadim]

i took lot of time to get it, old nodes getimg much more egress than new ones, during test, now is all flat.

[BrightSilence]

My node has been around since the last network wipe (and long before) and I have 4.6TB as well, just like @Pentium100, who’s node I’m pretty sure has also been around since long before the last network wipe. So it seems you clearly got more data.

[Vadim]

In some point about a month i used Proxies to nodes, it is posible that significant amount of data i got there.
But i got strongly minus in money, as it is expencive configuration and i stoped to use it.

[BrightSilence]

False alarm everyone. I verified in the code that node selection is done in the right way and every /24 subnet receives the same amount of data. Differences must have come from the temporary proxy setup.

More info in this post:

IP Filtering, Deep Dive

I can confirm that @Alexey must be correct that this was just because of the proxies used. I had a little dig through the code to find the actual code that’s responsible for selecting nodes. You can find it here: Bottom line; the code first picks one random node per /24 for all nodes with a known IP, and then picks as many nodes as needed for the transfer from that list. The result is no doubt that every /24 subnet will be picked at the same rate. I guess we got all excited over nothing. …

[Mad_Max]

@Pentium100

I do not know how many files you can lose and not be disqualified, but restoring any backup will mean that you lose some files, unless the node was completely idle since the last backup.

You loose some files(all recent) if current data folder is replaced by outdated backup version. Then yes - you will loose some most recent files (uploaded after backup was made).
But what if you merge backup with current folder? Simplest way is just writing over it.
It will overwrite files which exits both in backup and in current(possible damaged) data, add missing files from backup and keeps files creates after backup was made. It also add some unnecessary data (like files deleted after last backup) but it not a big deal - just some harmless junk.

It will not help in case of total data loss (like complete HDD failure due to dead motor for example ou burnaout controller) as backup is all you have in such case. But can help in case of minor data damages / corruptions like bad sectors on HDD or transient software/hardware glitches.

[Mad_Max]

@cdhowie

What is the true cost of this insurance policy? If you divide your storage in half for redundancy (RAID1/RAID10) then keep in mind that you’re losing half of your income potential. If that potential is more than the cost of restarting a node from scratch then you’re actually losing money by adding your own redundancy.

This is why the official advice is to run one node per HDD. Yes, if a hard drive dies you lose the escrow and you have to start that node over from scratch. However, if you have many HDDs, in the long run you will make more money running a node per HDD instead of wasting half of the space on redundancy that the network already provides.

It s not true. Increasing/decreasing HDD space 2 folds do NOT increase/reduce your income potential 2 folds. Because most of the income supposed to flow from traffic/bandwidth used not from HDD space rented (compare 1.5 USD per TB of HDD space used for month with 10-20 USD per TB of egress traffic served).
Reducing available space 2 folds sure affect potential income but much less than 2 times. And it start actually affect income only after node is already full (used all allocated space). Which takes a lot of time if we talking about large volumes of data. It will take many months from node creation to fill even small RAID arrays if SNO shoose bundle up all disks in same machine into RAID instead of running separate node per each HDD.

Also only most primitive raid 1 scheme reduce available space 2 folds. RAID 5 with 3 or 4 disk will reduce space only by 1/3 or 1/4. Eg 3x4 TB disks in RAID5 gives 2x4 = 8 TB of usable space instead of 12 TB without RAID. So we actually should compare about 25%-33% space “loss” and may be about 10-15% of total income (composed from payment from space + bandwidth) loss due to it AFTER node is already full vs risk of losing all accumulated escrow.

@Pentium100

And yet, if there is a problem and I lose 100MB out of 5TB, I cannot do graceful exit and upload the remaining data back to the network. I have to just shut down the node and allow the network to repair the whole 5TB. If I don’t shut down the node, I may just be disqualified later with the same results - network repairing the whole 5TB.

Actually you still can call GE in such situation. Satellite will mark you failed it due to errors in data / missing of some data but still gets all data remaining from your node transferred to other nodes .
So its a win-win situation for a network as it get both at same time: all the data is still usable (not lost/damaged) and your escrow.

Network will loose data and may encounter need of significant repair only if you surely know in advance that you already lost/damaged some of stored data, so there is no chance for you to finish GE with succes and because of that instead of starting GE just shut down node and wipe data stored without waste of time and bandwidth for GE.
But if you are “good fellow” and nonetheless still want to support network you still can call GE and satellite will gladly collect all undamaged data from your node.

[BrightSilence]

@Mad_Max

Reducing available space 2 folds sure affect potential income but much less than 2 times.

You can’t be so certain either way. It’s fairly reasonable to think that if you hold twice as much data that data serves twice as much egress traffic as well and since you’re not paid for ingress which would not be correlated with storage, the assumption of it scaling 1:1 is actually pretty fair. Unless it turns out recently uploaded data is downloaded more often. (like it was during some of the tests that were in no way representative of actual customers usage)

[Pentium100]

@Mad_Max

Network will loose data and may encounter need of significant repair only if you surely know in advance that you already lost/damaged some of stored data, so there is no chance for you to finish GE with succes and because of that instead of starting GE just shut down node and wipe data stored without waste of time and bandwidth for GE.

Well, if I didn’t think the data was damaged, I probably would not start GE. If I start failing audits or notice bad sectors in the hard drive, I would start GE, but, since the requirement for GE is even more strict than normal (you can fail at least a few audits and not be disqualified, but a single missing file and GE fails), my options are to keep the node running and hope that it does not get disqualified (not enough failed audits) or just shut it down and create a new node. There is no reason to start GE, I might as well start the new node right now so it can start the vetting process immediately.

[Eleos]

What if let’s say the OS drive fails or for some reason the identity files get lost. A backup of the identity files would be viable to reinstall the OS and setup storj on a new drive and transfer over the identity files from backup?

[Pentium100]

You can also store the identity files on the same drive as the node data.

[Eleos]

That’s what I’m doing currently. From my understanding, if the drive with the data fails, there’s no point in having a backup of the identity files anyways.

[Toyoo]

Drive failures come in many varieties. Some make all data unavailable, but some only hit a small number of files, making the rest recoverable. In the latter case if the failure hits identity files, the node is lost despite that there’s enough data to recover the node.

Identity files are small. It doesn’t hurt to back them up.

[Eleos]

How would you be able to tell if enough data is recoverable?

[Toyoo]

Standard data recovery practices apply. You just try to copy data using a data recovery tool like ddrescue, or whatever tool you are familiar with.