Question of fue hd

Node Operators FAQ
21

Yes, but that redundancy does not help me as a node operator. In fact, the current requirements are really strict and at least datacenter-level and actually even more strict than that, considering that

  1. a datacenter usually has multiple employees who look after the servers, so somebody going on vacation is not problem, where I am alone and if I go on vacation, my node has to survive without any physical intervention
  2. datacenters usually have scheduled maintenance where it is allowed to take the server down for some time (usually negotiated with the customer when it is least annoying to do so)
  3. datacenters have backups, so in the unlikely event of something really bad happening, you can restore the data.
  4. datacenters have different contracts with ISPs, where problems are solved fairly quickly, while a residential connection may stay down until the next working day.

Well, I do not really want to do something where I know I will not be able to meet the requirements and need to start over multiple times. For example, if I had only one internet line, if the ONU failed during the weekend, it would not be replaced until Monday, getting 40 or so hours of downtime.
The “no backups” part worries me a bit though - what if my node loses data because of a bug in the software? It’s something that I cannot really control and RAID would obviously not help there.

1 Like
22

I really appreciate your efforts in creating a “near” datacenter stable solution. I mean there’s enough RPis with USB 3.0 HDDs already…

For the strict uptime requirements: if any component in any of my nodes fails while I’m at work or asleep, they will easily ramp up 5h of downtime…most likely before I even notice. Also, 2 of my nodes are only accessible on weekends…so if somethings goes bad, the node is mostly done - be it with or without RAID.

23

I still don’t get it… you’re spending much more money on an overkill of hardware, than the value you would actually lose should this 2% chance occurrence happen.

Just take the small risk, live a little. Chances are you don’t need any of the protections. And even if you do lose a node, I doubt your held amount adds up to the cost of server hardware + UPS + 2 additional HDD’s for RAID6.

2 Likes
24

And that’s where the other hardware comes in - UPS, redundant power supplies etc. The goal would be to make it so that any single component could fail and that would not affect the operation and that you can fix other problems remotely (iLO etc). For example, a PSU failed on my server (not the one with Storj), so I just bought a new one on ebay and replaced it when it arrived with no downtime. Otherwise I would have had hours (if I had a spare PSU) or days (if I didn’t) of downtime.
Storj node does not work exactly like that sadly. I cannot have a backup copy to spin up in case the whole server decides to let the magic smoke out for example. So, it’s either a single server or a HA setup with shared storage, which would increase the latency somewhat.

Basically, I have two modes of operation - either try to make it as reliable as possible (the way I run my own VMs etc) or run it like a miner, where if it is stopped for a few hours or a day, I only lose the income for the time it was offline, so just rig a relay to reboot it once a day and it’s good. The way Storj operates removes the second option - I have to make it reliable.

A lot of the stuff use for the node I either already had (UPS) or it was on my list, but Storj just gave me the last push to do (backup internet connection). The only things I bought specifically for Storj were the hard drives and faster CPUs for the server.

Now, I could have used some old PCs I had and put one hard drive in each of them and run multiple nodes that way. That would result in more frequent problems, desktop hardware freezing etc, slower nodes overall, essentially more management overhead than one big and reliable node. It was nice with v2 when my node could be offline for a week and nobody really cared.

I had hard drives fail, I had PSUs fail and the 2% hard drive failure rate is for one year, so if I was running the node for longer, the probability increases. Especially if my node got a lot of traffic and put high load on the node.

I like getting more tokens, I would not want to start from scratch, with an empty node and little traffic for months and also high escrow percentage.

1 Like
25

You’re making a philosophical argument, not a $$$ argument. Can you actually make all this extra protection and cost make monetary sense? Maybe in your case you already had the hardware. In that case, sure, go ahead and use it, but that’s not the case for most others and thus not a great basis for good advice. I can’t see how making all those extra costs is worth it, especially considering there are a LOT of set it and forget it SNOs running on rpi.

2 Likes
26

Well, if I set up a node and had to regularly start from scratch, I would get fed up with it (what’s the point if I never make the promised $/TB because I get little traffic and most of that is eaten up by escrow?).

I actually do not know how profitable running a node will be in production. Right now, the traffic is so low that it’s not really worth it - RAID or not, well, with the exception of January. However, if the traffic increases in production to make it worth it, then losing a few months of traffic would mean a higher loss. Also, right now, my node is almost old enough to receive the full payment, so I probably have less USD tied up in escrow than I would have if I created a new node right now. Though I guess it would make sense to run a second node (or more than one) with 500GB (or whatever the minimum is) of space to have an (old) replacement if my current node is disqualified. That could be on a single hard drive etc - if it fails, not a lot is lost.

Also, running multiple nodes (in the same /24) means more management overhead - more monitoring, having to update all of the nodes etc. At least right now my node does not use a lot of space, so splitting up the RAID would not make any sense.

But all in all, if Storj did not want the nodes to be very reliable, why publish the rules/requirements that say that the node has to be reliable?

3 Likes
27

Actually, telling from the numerous threads in here, RPi setups are anything but set it and forget it :wink:

1 Like
28

I think you’ve made a miscalculation if you think the “set it and forget it” people post here.

4 Likes
29

But it looks like quite a few rpi setups have problems. Then again, Rpi setup are probably more common and the operators who have them are probably less experienced.

1 Like
30

If there are twice as many threads concerning RPi problems compared to other hardware, I think it’s safe to assume that RPis encounter problems twice as often as other hardware. (Don’t pin me on this number, I’m just observing a pattern there…)

31

Well they are dirt cheap for a standalone node, yeah. But if an unexperienced user has a running windows machine, they’d be better off just going with the windows GUI node.

Fiddling around in the depths of linux isn’t for many users, I mean…

32

Why would you “regularly start from scratch”? The failure rate per hard disk is about 2% per year. Ok, if you mean by regularly “after statistically many many years”, then you’re right. You over-exaggerating a very small problem here.

33

It takes 10 months for the escrow percentage to go down to zero and 16 months for me to get half of the escrow money back.

Can a hard drive survive that long? Yes, absolutely, it can survive even longer.
Would I like waiting a month to get vetted, then a few months until traffic picks up, then a few more months for escrow to drop to zero, then 6 more months to get half of escrow back? No, not at all.

There are also other things to consider. Let’s say my current node gets disqualified, so I can do whatever I want with the 6 drives.

  1. Do I keep them in the same server, but run 6 nodes on them? There would not be much point, since right now, the stored data is 4.7TB, whether the total capacity is 16TB or 24TB it won’t matter.
  2. Do I use some old PCs for each node and one drive? In that case I may have less reliable nodes due to the PCs and if the cases etc do not have proper cooling, the drives may fail faster. In addition, multiple PCs would use more physical space than my current 4U server.
  3. What about performance? Right now, I have RAID6 out of 6 drives, but also SSD read and write cache - two SSDs for the array. If I split the drives up to different PCs, I’ll need more SSDs, if I keep them all in one server, I’ll have to split the SSDs into lots of partitions, which means that there will be less cache available for each node.

But, all in all, Storj made the requirements that strict for a reason and a single hard drive cannot really be expected to meet those requirements. So, why do something knowing that it will fail and I’ll have to start over? Might as well do it properly the first time, so I do not have to do the same thing multiple times (and expect different results).

1 Like
34

On the Storj Network side

Let’s say there are 2000 SNOs.

Let’s say that the HDD failure rate is 2% per year.

Statistically we should expect about 2000*.02 = 40 SNOs to experience a failed HDD.

Let’s say that those 40 SNOs each store about 3 TB of data …

That’s 40*3 = 120 TB of lost data that needs to go through the repair process.

If the bandwidth is paid at $20 USD per TB, then the required expected and normal yearly repair cost just for lost nodes through normal operation is: 120 * 20 = $2400

On the SNOs side

Let’s say I was running node from Sept 2019 through Dec 2019 and had a drive fail at the beginning of Jan 2020.

What would the payout implication be for this specific scenario given that Jan 2020 had an Egress stress test?

If that node operator was myself, I would have lost $150.00 of Egress traffic, which is very unlikely to be repeated in the near future. So, HDD failures throughout the network may not be very consequential to the network itself. However, I contend that since individual node traffic is not continuous nor the same from month to month, that running a single HDD drive per node may result in significant loss of potential payout.

GNU Octave m-file

fullpay=[2.16, 9.06, 9.13, 14.54, 108.32,7.81];
surge=[4,4,4,3,3,1];
unescrow=[0.25,0.25,0.25,0.5,0.5,0.5];
pay=fullpay.*surge.*unescrow

octave pays.m

Realized Pay

 Sept       Oct        Nov       Dec        Jan         Feb
2.1600     9.0600     9.1300    21.8100   162.4800     3.9050

So, if my node’s HDD decided to be one of those 40 per year in the network that are expected to fail, what would I have lost? And would I ever expect to recoup the lost $162 ? Would it have been worth it to run an extra drive at the the cost of about $50 to $80 to ensure that the $162 random payout wouldn’t disappear ?

I’ve given a practical and real life scenario in which the potential failure of an HDD in a one drive per node situation would have resulted in loss of a huge portion of an SNOs yearly return.

EDIT: Updated to include Feb payout for comparison to Jan… as well as the correction from 4x to 3x surge for Dec 2019 … The basic point is that the uncertainty of traffic in any given month means that running one HDD per node is inherently risky and may result in significant unrecoverable income loss.

1 Like
35

And who profits from the repair? Right, other SNO’s like you and I.

Again, the chance of that to happen is 2%/year.

You’re saying the 2% chance of losing $xxx weighs more than the 98% chance of earning another $xxx with a second node on the hard disk you’re wasting on redundancy for your first node? I don’t get it.

Also, statistics doesn’t work like this. You can’t prove your decision was right in hindsight after an event happened. So what payout do you expect in “the future”? Let’s say you have 2 drives with 1TB each and get $x for 1TB stored (including expected egress). It’s safe to assume that the egress increases linear the more data you store. So you rather earn $x with 100% likelihood than 2 times $x with 96% (100% - 2x2%) likelihood?

Yes, 40 people would lose money, but 1600 would make more. I know which queue I would join if I can chose :slightly_smiling_face:

RAID makes only sense if the first node is not full yet and you have a spare second empty hard disk. As soon as the first hard disk gets ~75% full, it makes you more money in the long run if you start a second node instead. Actually I would start the second node earlier to spread the data more evenly and therefore spread the risk more evenly.

But I see this discussion leads to nothing and we’re running in circles. People who think RAID is better can’t be persuaded with arguments and math…

3 Likes
36

If I’m running a node with a single HDD… and have the exact payout I actually did have…

Here’s the lost escrow payment as well:

 Sep         Oct       Nov        Dec       Jan         Feb
6.4800    27.1800    27.3900    21.8100   162.4800     3.9050

Total Lost Escrow: $249.24

Total loss if my node happens to be in the 2% of failed drives:

$411.72

That randomly placed $162 month puts a big wrench in the idea that one should simply not worry about node DQs due to failed HDDs. It affects the payout and escrow, and one can not determine from the past what the future traffic will look like. Running two nodes on the same Internet subnet creates contention and the nodes fill up slower. Running a single HDD per node makes one quite vulnerable to large potential losses in income…

So…

I still contend, using my node’s real world data, that it’s better from an SNO’s perspective to run a RAID array rather than a single drive.

1 Like
37

It’s not just the lost escrow money. Let’s say the drive failed in December. You would have lost less escrow money, but your new node would be empty on January, so, no egress at all.

And if my node manages to fill 16TB, I’d rather expand the array than split it off to separate nodes. Storj said earlier that there may be some kind of reputation system in addition to audits and uptime (though that may have been scrapped), in that case, having an older node would be better.

Also, my current node probably has less money held in escrow than a new node would have (due to little traffic in March etc).

Running multiple nodes n the same /24 would make sense if someone else was running nodes in the same /24 (more nodes would mean higher chance that my node is chosen). But I could do that on the same array.

Unless your node is full, I recommend RAID. And if your node is full, you should expand the storage to always have free space.

1 Like
38

Of course it is… You’re arguing a straw man argument that nobody here made. The comparison should be against running 2 nodes on separate HDDs. You’re also intent on going with the worst case scenario of missing the best payout month. That’s fine, but since you reduced the failure to a specific month, I’m going to reduce the chance of that risk to less than 0.2% to make it fit your scenario.

0.2% * 411.72 = a whopping 82 cents. That’s the value of the risk you’re worried about in the worst month ever. It’s lower for all other months. As soon as a second node could make an extra 82 cents (or less for average months) running that second node is more profitable on average.

2 Likes
39

This is an invalid use of observational data.

A node operator who runs a single HDD drive per node is risking loss of the current month payment, future months payout, and escrow.

Observational data show that any given month may yield a significant percentage of an entire year’s payout. Therefore any given node that fails may lose most of the potential payout for that year. Since this is true based on real world operation and the operational observation is that the egress traffic for any given node is:

  1. Not evenly distributed across all months.
  2. May be mostly contained within a single month of the year.
  3. Has no memory of the past and no knowledge of the future.

it is invalid to claim that the lost payout should be evenly distributed across potential future nodes that may be run as replacement nodes for the lost HDD.

40

I should point out that my RAID 5 node has 31% space left with its original allocation after running for 6 months. So, I would have not accumulated any extra payout over that 6 months by running each drive as a separate node.

My guess is that once all test data has been removed from the network, my node will fill even more slowly than my 69% over 6 months.