RAID6 array expansion and Storj

I’m currently expanding my RAID6 array and noticed the CPU couldn’t keep up with the background expansion operation and three storj nodes , even setting the expansion task to low:


Note: 6 CPU cores, 12 threads

I’ve managed to have just one node running but even then it would have some iowait (10~20%).

The RAID6 is currently using 64KB as stripe size, should this get better with a bigger stripe size or do you have any other recommendation ?

The nodes I have on this machine are turned off because it is preferable I end the expansion and have data availability than by being suspended/DQ because I failed audits in acceptable time due to the iowait.

2 Likes

No idea about CPU usage or iowait, but…

Some compromise could be running the nodes with disk space reduced to below your current disk utilization. This way you will allow downloads, but not uploads, so it should not be as taxing to your system.

Being offline will not cause failed audits.

DQ for downtime currently requires around 2 weeks (I think, someone should confirm this as it recently changed).

However, why not just let everything take a little longer but stay online? Whenever I rebuild an array that StorJ is stored on, it does take extra memory and processor - and by the nature of it maxes out the drive I/O.

But so what? It takes twice as long and the server runs a little slower, but no need for downtime.

Good luke!

1 Like

Why are you running more then one node on the same array anyway? You will not getting more data compared to one node.

Aren’t nodes limited to 20TB ?

no that is just a recommended size. Depending on the ingress and the deletion ratio it might take 1 or more years to fill up, so more than 20TB doesn’t really make sense to have at the start. You can however expand it to any size you like.

64kbit stripe size maybe a bit small… but most likely not worth the effort to change it…
nor can i tell you that you will for sure benefit from that… you will have to get some benchmarks on that, also depends on where you database is located, how many RAM you system has and if you are using any large caching solutions.

sorry that it isn’t a simply answer… 64kbit stripe size was the recommended for a long time if people don’t know what to set it to… to really optimize it you will have to account for the number of disks and the sector size so a full stripe writes on every drive in the array.

20% iowait shouldn’t make your storagenodes fail audits… tho if you are running 3 nodes on the same raid6 array i’m pretty sure i can tell you where your problem is…

a raid6 array has the same IOPS as 1 hdd because they run in harmony / sync… on top of that if your stripe size isn’t the correct size compared to your array, then you will also get writes that overlap all of the array and then starts over again… without writing the full way through the array on the last part of the stripe… which means you will end up having drives running out of sync, which will cause further latency.

on top of that all the drives should be the same to futher reduce issues in the harmony / sync of the drives, else you might see excessive wear, performance decreases or both.

it’s very difficult to fail audits, ofc if it’s a cpu issue which it might be… but that would mean you are doing software raid6, because with a conventional raid card you shouldn’t see a massive cpu utilization just because you are doing work on the array, the raid controller should do that… and because it’s hardware is designed for the tasks it can do the job much more efficient than a cpu.

raid6 calculations are fairly demanding

you shouldn’t run multiple nodes on the same raid array and you really should use a raid card if you want to run raid6

1 Like

@SGC thanks for the comprehensive reply. I’m using an Adaptec ASR72405 RAID adapter, and 48 GB of RAM. The OS is working on a Soft RAID 1 with 2x 256GB SSDs.

I will add two SSDs to the array and see if their maxCache solution help, I haven’t thought on it honestly as I’m already using the RAID adapter cache (1GB) for R/W.

Edit: Nevermind, maxCache is not suported by this adapter.

there is an option in storj / storagenodes to move the database to another location, since you are running a mirror Boot / OS solution it would make good sense and be perfectly safe to move the databases of your storagenodes to this location and thus potentially save a lot of IO that is difficult for your array to do.

not sure how much it actually helps… to answer that i would search the forum or ask the crowd mind. :smiley: but it might make a huge difference.

That’s a good idea, will try it, tks :wink:

@SGC suggestion worked wonders. Now I get 0~5% iowait after some initial garbage collection where I got 5~15%.

1 Like

2 weeks cumulative? So during the entire lifespan of the node? Or two weeks every 6 months for example?

1 Like

Makes sense, basically its an allotted amount per 35 days, upon which it gets reset to 0. Fair enough!

1 Like