Best Practices for multiple hard disks

[hondasean]

Hey guys

So i have just discovered the project and have spent the last couple of days getting my head around the technology and how it works etc in anticipation of starting a node.

One thing in particular that i have been not totally understanding is what is the best way of treating multiple drives in a single server.

I have had a bunch of drives sitting in a cupboard for around a year now with the plans to build myself a nice NAS server but i never got around to it.

after discovering storj it gave me a good excuse to finish the NAS because i really liked what the project is trying to offer, I also have a decently fast 50/10MB line which is soon being upgraded to 1gbps/250 as my isp are installing cables at present.

after building my NAS and deciding how to handle my personal data i was left with 2 brand new 4tb NAS drives along with 2 older 1tb drives,

I had initially planned to just put all of these drives into one big storage pool in order to run a single node, but there did not seem to be an easy way to remove drives to upgrade the storage pool or replace a failing drive, (windows sever 2019) so overall i did not like the solution(although it did work for my personal data solution) since i am very aware of the dangers of a single drive failure killing the whole node.

I also could not see much of an option to shrink the node in the event of me having to remove a drive for one of the above reasons, or if i wanted to set up a second node from another internet connection which i may explore further in time.

So it makes sense to recommend one drive per node, however I have some concerns that a smaller node would take longer to accept the same amount of data(taking the same or longer to fill) and may also see less traffic as a whole due to it holding less data. Is this still an issue with storj at present or is it no longer an issue in the current release?

of course one node per drive does hold the advantage of less drive wear as the nodes could be kept offline until the online ones were nearly full.

Another major concern with one node per drive that would hold me back in the long run is the price of electricity, as the nodes must at least break even in terms of running costs in order to stay online.

Is it possible for me to run more than one node on one computer, but still keeping with the ethos of one node per drive? this way in event of drive failure it would still only result in the loss of one node and for me it would still have the benefit of only using one system to host the data with.

the server in question has 4 cores/4 threads and 24GB of ECC ram installed at present there should be ample resources to run additonal nodes. this system is just an old sandy bridge pc, i might move storj the drives to an itx box for even lower power which would allow me to sleep mode this oversized NAS when not needed.

[Pavmer]

If you have several nodes behind one IP range, in this case the network will consider that you have one node and the traffic will go as for one node, being distributed on all. At the same time, audits and checks etc will be considered as for each node separately, which is much longer and longer.
The best option is to fill one node and open the next, but it is best for the network, it is different locations of nodes…
I am a newcomer and my node is assembled on a knee with one disk 2.5, but I plan RAID6 from BU disks

[cdhowie]

@hondasean

I had initially planned to just put all of these drives into one big storage pool in order to run a single node, but there did not seem to be an easy way to remove drives to upgrade the storage pool or replace a failing drive, (windows sever 2019) so overall i did not like the solution(although it did work for my personal data solution) since i am very aware of the dangers of a single drive failure killing the whole node.

Note that the network stores data redundantly. A lost node doesn’t mean the data is lost, it just means you don’t hold it and therefore don’t profit from it anymore.

To put it another way, you will almost certainly make more revenue running a node on each disk rather than building a RAID1. With a RAID1 your node could last forever, but you are nearly halving your revenue potential. (If you have many disks, RAID5 could make sense just because it will save you time having to set up a new node each time a disk fails. In effect, you are “spending” the potential revenue of the redundant disk as an insurance policy against your time.)

@hondasean

I also could not see much of an option to shrink the node in the event of me having to remove a drive for one of the above reasons

This feature was in the design doc but it does not exist yet. The “total graceful exit” feature is in active development right now, so there will soon be a way to shut down an entire node without losing escrow, but it may be some time before you can ask storagenode to remove some amount of data without any penalty. Right now it’s an all-or-nothing deal.

This is another point in favor of running a node per disk. If you need the disk back, just shut down the node and accept the escrow loss (right now) or gracefully exit the node (once that feature is ready).

@hondasean

So it makes sense to recommend one drive per node, however I have some concerns that a smaller node would take longer to accept the same amount of data(taking the same or longer to fill) and may also see less traffic as a whole due to it holding less data. Is this still an issue with storj at present or is it no longer an issue in the current release?

The amount of free space on the node isn’t really relevant; however, running two nodes in the same /24 subnet will cause Storj to equally share data among them. This means each node will probably take longer to be vetted. However, once the nodes are vetted, the “N nodes with M storage each” scenario and “1 node with N*M storage” will see pretty much the same traffic patterns.

To put it another way, if you are running 3 nodes, each node gets 1/3 the traffic that a single node would. You’d still get the same amount of traffic running a single node. (And, running a single node with redundancy like RAID1 would see reduced revenue potential due to the decrease in total capacity.)

So, aside from the additional time to be vetted, running multiple nodes will not affect your revenue.

This is why it’s recommended to turn up one node at a time until it is nearly full (90% or so) and then start up the second node. If a node is full, it doesn’t get ingress traffic anymore. Therefore, this minimizes the amount of time that the nodes fight for ingress data during the vetting period – once the first node fills up, the second node is not losing ingress traffic to the first node, helping it to be vetted faster.

@hondasean

Is it possible for me to run more than one node on one computer, but still keeping with the ethos of one node per drive?

Yes. Docker can run multiple containers from the storagenode image, with different data directories and mapping the container’s internal port to a different host port. You’d effectively be running multiple services on different ports. (I currently do this, it’s not just theoretical.)

[Krey]

@hondasean

windows sever 2019

This OS have Storage Spaces, nice soft for building data storage.
Personally i prefer raid redundancy, but you disk collection Not very comfortable for this. Raid5 on 4x4TB much better.

[direktorn]

@cdhowie

o put it another way, if you are running 3 nodes, each node gets 1/3 the traffic that a single node would. You’d still get the same amount of traffic running a single node. (And, running a single node with redundancy like RAID1 would see reduced revenue potential due to the decrease in total capacity.)

How does that work? Your saying a single node will get the same traffic as splitting it up in multiple nodes, but decreed income?

[direktorn]

@Krey

Personally i prefer raid redundancy, but you disk collection Not very comfortable for this. Raid5 on 4x4TB much better.

I’m running Raid-5, or multiple Raid-5’s on my drives, but also storage spaces as it allows me the manipulate my storage pools and on top gain redundancy (Datacenter edition) as duplication is built in, even across multiple instances/nodes of storage spaces.

[Pentium100]

@cdhowie

To put it another way, you will almost certainly make more revenue running a node on each disk rather than building a RAID1. With a RAID1 your node could last forever, but you are nearly halving your revenue potential.

That is only applicable if you can fill the available space. Empty space does not generate profit.

I personally use RAID6 (raidz2) with 6 drives, because data loss/disqualified node means multiple levels of problems:

1. Loss of escrow for the failed node
2. The new node will be vetted for a month and get almost no traffic
3. The new node will start at low reputation and high escrow percentage.

[cdhowie]

@direktorn

How does that work? Your saying a single node will get the same traffic as splitting it up in multiple nodes, but decreed income?

I’m not sure what “decreed income” means.

Yes, running a single node will get the same amount of ingress traffic as multiple nodes in the same /24 subnet. Piece uploads are divided among them for the health of the network (you don’t want two of the redundant chunks stored in the same physical location – that defeats the purpose of redundancy, especially when the network does not require redundant storage).

@Pentium100

That is only applicable if you can fill the available space. Empty space does not generate profit.

Correct. Right now I have a 4TB node that is running 2x4TB disks in a RAID1. But as soon as the node fills to 90%, I plan to reduce the RAID1 to 1 device and create a new node on the second disk. That is, I might as well have the redundancy while I don’t need the extra storage, but once I could profit from the extra storage I certainly will.

@Pentium100

I personally use RAID6 (raidz2) with 6 drives, because data loss/disqualified node means multiple levels of problems:

These problems are really only problems if you expect your disks to fail frequently.

Let’s put it another way. Depending on the capacity of your disk, it could take a certain amount of time to fill up. Let’s say that it takes 6 months to fill up.

After 15 months, all of the held escrow that will ever be paid out (without gracefully exiting the node) will have been paid out. You will never realize the remaining escrow unless you leave the network and it doesn’t sound like you plan to, so we don’t consider that escrow in this calculation. If a node is disqualified after 15 months then it will take another 6 months with a new disk of the same capacity to get back to full. Sum that to get 21 months.

So, with RAID6, if you expect two disks to fail every 21 months, yes, RAID6 is a good idea. If you expect a lower failure rate, the redundancy is wasted and you’d realize more profit by using those two disks to run their own nodes.

The ultimate question is which is larger: the lost revenue from disk failures, or the additional revenue that could be realized by using the redundant disks. This is a calculation you’d have to do based on the capacity and expected lifetime of your disks.

The smaller the disks and the less frequent the failures, the more revenue you’d get by running multiple nodes with no redundancy. On the other hand, the bigger the disks and the more frequent the failures, the more revenue you’d get by running a single node with a redundant storage configuration.

[Pentium100]

@cdhowie

You will never realize the remaining escrow unless you leave the network and it doesn’t sound like you plan to, so we don’t consider that escrow in this calculation.

But if I create a new node, there will be additional escrow that I will never get. Depending on the traffic of the network at the time, the new unusable escrow amount can even be bigger than the current one.

@cdhowie

If you expect a lower failure rate, the redundancy is wasted and you’d realize more profit by using those two disks to run their own nodes.

But again, only if the disks fill up. Right now I have 16TB of storage and only 4TB of that is used. When they decide to delete the test data, it will probably free up a lot of space.

Also, as I understand it, right now, all nodes get about the same opportunity to store data (the faster ones get to store more of it), but the plan is to have a reputation system, not just new/vetted/disqualified, which means that a new node will most likely get less data, even after being vetted, compared to an old, reliable node.

RAID5 is not recommended with large drives, so I use RAID6.

Unlike v2, where up to a week of downtime was acceptable (and I used desktop grade hardware that sometimes crashed), I use proper hardware here and that includes RAID, so that I could have the best reputation (and the most traffic) that I can.

I do not expect hard drives to fail all that often, but I’d rather be safe.

[direktorn]

@cdhowie

Yes, running a single node will get the same amount of ingress traffic as multiple nodes in the same /24 subnet. Piece uploads are divided among them for the health of the network (you don’t want two of the redundant chunks stored in the same physical location – that defeats the purpose of redundancy, especially when the network does not require redundant storage).

OSX spellchecker… decreased traffic. So how can you get increased revenue by having multiple nodes running on the same IP?

[cdhowie]

@direktorn

So how can you get increased revenue by having multiple nodes running on the same IP?

It’s not about having multiple nodes directly, but rather running multiple nodes on each disk instead of using additional disks for unnecessary redundancy.

It’s important to note that ingress traffic is not decreased but rather is spread across all of the nodes. In aggregate, multiple nodes are going to see about the same amount of ingress traffic as a single node.

For example, if you have three 1TB disks, you have a few options:

1. Put the disks in a RAID0 or make a linear LVM volume across them, then run a single node.

   Pros: shares all 3TB of storage with the network, easier to manage since there is one node.

   Cons: a single disk failing will destroy the volume and disqualify the node.

2. Put the disks in a RAID5/6 and run a single node on that.

   Pros: can sustain a disk (RAID5) or two disks (RAID6) failing without the node failing, therefore it’s easier to recover from a failed disk. Replacing the disk and repairing the array is easy and is transparent to the Storj network.

   Cons: One (RAID5) or two (RAID6) disk(s) are used for redundancy, reducing the amount of data you can store for the network and therefore reduces your storage revenue (and egress traffic revenue opportunity).

3. Run a node on each disk separately.

   Pros: utilizes all available storage instead of dedicating some amount for redundancy, which is already built-in to the Storj network.

   Cons: when a disk fails (assuming that you want to replace the node) you have to get a new auth token and identity and configure a new node. Any payments held in escrow are lost. The new node must start from scratch, unvetted and holding no data.

Other redundancy mechanisms built into LVM, ZFS, btrfs, etc. will fall into one of the categories above.

Category 1 is a non-starter. Don’t do that. The benefit of simpler maintenance is not even close to outweighing the inevitable disk failure from killing the whole node.

The question is whether you’d pick category 2 or category 3. If you have big disks that are failing very often, category 2 will provide you more revenue. If you are using smaller disks that don’t fail very often, category 3 will provide you more revenue. This is what I meant by:

@cdhowie

The ultimate question is which is larger: the lost revenue from disk failures, or the additional revenue that could be realized by using the redundant disks. This is a calculation you’d have to do based on the capacity and expected lifetime of your disks.

The variables “disk capacity” and “disk failure” rate ultimately will decide which to use, but I don’t think anyone has come up with a formula to determine this yet. It would be an estimate anyway since network usage patterns also play a part and those can’t be predicted, only estimated.

[hondasean]

Thanks for the responses guys i really found this discussion insightful in finally coming to my decision.

I have decided best course of action is to run a single node per disk, as actually, being able to run multiple nodes from a single server this option becomes the most flexible because it gives me freedom down the road to decide where to set up future nodes be they on my server or another one in another location(more profit).

it also means I am only ever using the power of a single motherboard and cpu etc which will cut overall consumption by a considerable amount.

I will not be concerned with drive redundancy as the drive is a new WD red it should last a long enough time

I am currently reinstalling using docker as my gui node had only been online briefly as an initial test and I want to be able to run multiple nodes if needed which means the docker install is needed anyways. btw, is docker some type of virtual machine?

[cdhowie]

@hondasean

is docker some type of virtual machine?

Docker is a tool for managing Linux containers¹ as well as managing and acquiring so-called “images” which form the (read-only) beginning state for a new container.

The storjlabs/storagenode:beta image on Docker Hub, for example, contains the storagenode software and all necessary dependencies, so you can download the image and launch it from Docker without installing anything but Docker on your host system. The extracted image can be shared by multiple containers since aufs is used to create the container’s root filesystem. (aufs layers multiple directories on top of each other to create a view – only the topmost layer, the container’s own “root volume” is mutable. This way, containers can share lower layers which are effectively read-only. So you might be running ten nodes in different containers, but you only have one copy of the storagenode software.)

---

¹On non-Linux platforms, other technologies might be used. On Windows, both native Windows containers or Hyper-V might be used.

[nerdatwork]

@hondasean

the docker install is needed

Storj in production would be docker-less so getting used to Windows GUI should definitely help.

[Krey]

Read this post and think twise

Restore a node with hard drive fail

Thanks for the fast response… fu** this was a full 8TB drive with a lot of traffic… okay, to a new one…

[BrightSilence]

There is another option that hasn’t been discussed here yet. It doesn’t apply to everyone, but it might apply to you.

If you already have an existing RAID5 or RAID6 array for other uses, you can simply add the disks to that array to expand it. You will not be wasting additional disks for redundancy and are able to run just a single node to make use of all additional space. Furthermore this will also protect your node against HDD failure, basically for free. It will also make it easier to free up more space when partial graceful exit is implemented. I mostly prefer it for this type of flexibility in how you can move around allocated space for different purposes.

There are some things to consider. Since you’ll be sharing that array between Storj and other uses, they will impact each others performance. If this is simply a personal NAS with mostly static storage, that likely doesn’t matter. You’ll also be adding more disks, so more opportunities of failure to the array. This is not as much of a problem with RAID6 as it is with RAID5. The increase in risk is usually not that large, but it is something to consider.

This is actually the setup I’m using on my NAS. I don’t often advise this to others as it’s not that common that people have a large enough NAS to simply add more disks. But if this is applicable I would say it’s the best of both worlds.

[thaistar]

To have additional nodes on my same network what ports should I forward??? E.g., 28967 then 28968, 28969.

[BrightSilence]

You can pick which port to use. So those are logical options. If you’re using docker, make sure you change the port in the address and the first of the two ports in the port mapping. Example -p 28968:28967. You’ll want to do something similar for the dashboard port. Also use a different container name.

[thaistar]

Thanks. I guess this thread is about having multiple drives on the same computer. I’m adding a second node on a second computer on the same network.
docker run -d --restart unless-stopped -p 28968:28967
-p 127.0.0.1:14003:14002
Is the 14003:14002 parameter right???
Also, where should I put the data???
Replace it to the local directory where you want files to be stored on your hard drive for the network.
I wish it said put the data here. Recommendations???

No idea what to do with this BoltDB stuff. Am I supposed to install something???
Note: the current database backend is BoltDB, which requires mmap, hence you have to use a file system which supports mmap.

BTW, the docker repository instructions do not work with Linux Mint 19.3. I tried both Ubuntu and Debian. I installed this way:
sudo apt install docker.io
and I’m sure I’ll have problems later when it fails to upgrade.

[Alexey]

@thaistar

Also, where should I put the data???

Instead of <storage-dir> in the Storage Node - Storj Docs
The identity folder you should place instead of <identity-dir>

@thaistar

No idea what to do with this BoltDB stuff. Am I supposed to install something???
Note: the current database backend is BoltDB, which requires mmap, hence you have to use a file system which supports mmap .

You do not need to install anything additional. Just do not use a network connected drives neither NFS nor SMB.