Looking for help to set up RAID6+Dual SSD cache

Storage Node Operators
,
#1

I have 12 4TB SMR drives that I wish to set up with RAID6, and use two SSDs as a cache for the array. I know how to set up RAID6 in Win 10, but I am not sure at all how I would connect an SSD cache to it.

All my drives are external, using USB 3. They are connected to twelve ports, which share 6 individual controllers.

I read somewhere that setting up the SSDs in RAID1 is a good idea, for better reliability.

I have a few questions:

  1. How do I set up the SSD cache in connection with the RAID6 array?
  2. Does using an SSD cache in this setup add additional corruption risk compared to just using the HDDs in RAID6 without SSD cache?
  3. Pros and cons?
#2
  1. For Windows you should probably get a hardware RAID controller that has this feature.
  2. If you are using it as write cache, it could be. Depends on the implementation.
  3. SSD cache would improve performance, but SSDs and the controller cost money.

SMR drives are not that suitable for this as they could have problems with heavier loads. Then again, using a big array would distribute the load.

1 Like
#3

The cache is available on server version of Windows

1 Like
#4

I wonder how Storage Spaces Direct handles SMR drives.

1 Like
#5

I wonder the same :slight_smile:
I think it could work but it’s better to test

1 Like
#6

Also, isn’t S2D supposed to be a cluster? Minimum two servers, 10G network cards with RDMA as requirements. Running the node on a cluster would be great IMO, but may not be what the OP wants.

Alternative for Windows would a a RAID controller, but some require special SSDs for the cache, while others can work with any SSDs.

1 Like
#7

Hi, new SNO here - I’ve had my node online now for about 20 days and been lurking on the forums. I’ve got my storage on Windows server Hyper-V Core 2019 and am running Storage Spaces with a spare capacity SMR drive to use for some SNO storage.

Initial setup had the storage node data just on REFS on the SMR drive, read performance poor ( sub 100) write performance sketchy (40-60) - 2nd week in, write performance died on SMR periodically down to (1-5) and causing write latency > 200ms !

I’ve now put in a SSD caching tier, so SSD cache with 7TB SMR drive - layout is non-fault tolerant. I’ve given 95GB of SSD for read/write cache, and 10GB of SSD for write-back cache (normally would be 1-2GB). It’s been in this config for 3 days so far, and while it’s still not great it looks like it’s flattened the write performance removing the spikes and averaging < 30ms write.

I’m hoping that once my node is vetted, the write’s will increase and I might have to review the write back cache again. Annoyingly the only way to change the write back cache is to blow the drive away and start again :frowning:

Forgot to answer posters questions…

  1. Think you will need to use powershell to setup tiering - new-storagetier and new-virtual disk once you created a pool, not sure if those available on Windows 10.

  2. Adding Ssd increases complexity, and points where corruption might happen.

  3. pros - gives a nice persistent buffer for read/writes and write back. keeps hot reads away from smr so it can fiddle with writing. cons - complex to setup on windows, reliant on os and drivers playing nice, changing tier parameters means starting from scratch

1 Like
#8

With a SMR drives I would like to recommend to use one node per drive. They will spread the load between them and they should work much better than in RAID.

2 Likes
#9

Thanks for all the great answers. It got me thinking a bit, and I think I am now considering to go for multiple nodes instead. That leaves me with some questions though. My setup is on a single machine on one network, but I must use VPN since I cannot access local router for setting up my ports. Notably, then, my VPN provider auto-assigns one single port depending on which VPN server I choose, and as far as I can see I cannot open up more ports.

  1. Doesn’t each SNO node require a unique port? If yes, does my VPN setup prevent me from setting up multiple nodes?

  2. How do I go about dedicating one CPU core to each node, if I need to set up 12 nodes and only have 4 cores (8 logical processors)?

  3. Will dedicating all cores to SNO nodes make my machine unusable for other activities? I use it for heavy browsing (50+ tabs, multiple windows and browsers) and running 5 displays.

  4. Can I monitor all nodes in a single dashboard?

  5. Is Docker-based setup the only realistic way of doing this, or can I use GUI?

#10

The storagenode (as any published service) requires the unique external port (on the same IP). So, yes - with your VPN provider your can publish only one storagenode directly.
The recommendation regarding CPU is one core per node, however you can run more nodes if your CPU is powerful enough.
Usually you can use the same PC for a regular tasks, but if your bandwidth is used on 100%, you likely will have timeouts on sites or slow response.
You cannot monitor all nodes in one official dashboard at the moment, however, you can use other monitoring solutions for that: https://forum.storj.io/tag/monitoring
The docker is an officially supported way, but you can use a Windows Toolbox (search here on the forum) instead. It’s supported by the Community.

1 Like
#11

running raid6 on smr is a bad plan thats for sure… your iops will be the same as 1 smr hdd…
any conventional raid setup on smr is generally a bad idea… as it compounds the already great weakness of the SMR tech

i would go multiple nodes

1 Like
#12

I looked at Windows Toolbox and Windows docker, and they both seem like viable solutions for setting up multiple nodes. But, if I understand it correctly, there is no “escaping” using a unique port for each node, not even when using docker? Because if that is true, then my only option is to set up Raid and make a single large node, due to my single port restriction from my VPN.

#13

Unfortunately no. Each node has to use its own public port.

1 Like