ISCSI vs NFS shares for nodes?

Node Operators
, , ,
1

Hello StorJ forum.

I’ve encountered a serious dataloss due to human error, and am rebuilding all my nodes. I am currently running a few VMware hosts that have Windows VMs with StorJ on them. The Windows install is on local NVMe Storage, and the Synology box presents ISCSI targets for E:\StorJ drives.

Whenever I reach another TB, I increase the ISCSI LUN, update the configuration file and restart the StorJ service. This has worked fine for a while, but I now have the opportunity to redo my setup.

My VMhosts have limited capability to have physical storage directly attached, so I would like to continue using networked storage.

Therefore:

  • Would you change NFTS formatted ISCSI drives to simple NFS remote shares?

While I know it’s the right way going forward, now is not the time to upgrade to a full FC setup, so IP storage will remain for the time being.

Thank you

2

In Short: No NFS (or SMB) shares for Storj, only iSCSI works for network storage.

1 Like
3

That is a great and precise answer. Can I get the long version? :slight_smile:

4

NFS is not quick enough for node data access, so you will experience many errors and high memory usage.

5

I would also get rid of windows guest (and maybe also host, unless you also use that computer for something else other than VM host). All the ram you will no longer waste by doing so can be used for filesystem caching to improve responsiveness, and the latter directly affects winning races and payouts.

2 Likes
6

I know windows is not the preferred way of running StorJ, but it is what I am comfortable managing, and the system is also used for other things.

As for RAM, I have no problem dedicating 16GB to the VM alone. I’ve never seen more than 6GB usage

7

This is not about usage, but about what stays unused: I’m not sure if windows behaves the same way and to what degree, but most unix and linux systems utilize the unused ram to cache filesystem data and/or metadata. This significantly reduces random IO pressure on the disk array, and that directly translates into lower access latency. This happens transparently, so when applications request more ram—portions of the disk cache is evicted. Storj creates a lot of small files, so having the entire file system metadata in RAM may halve the disk IO required.

Empirically I’d say you want at least 8GB of free ram for 2-3TB size node, but I did not measure preciselsy,.

On the other hand, if you just give all that ram to one VM guest – nobody else can use it.

So perhaps you might want to not run storagenode in the vm on your host, but directly. Anyway, it depends on what other services you run, and what separation is required; you’ll need to balance those requirements; all I wanted to say that since storagenode usecase benefits from filesystem caching, deisgn your usecases in a way to facilitate that.

8

based on the previous topic i would recommend to run new single virtual machine directly using synology vmm with no network attached drive but creating volume with vmm itself and start multinode installation (topic starter already mentioned that he feel more comfortable with windows appliance not linux or docker- so its the only OS option)
i have several nodes running that way- no issue for 1.5-2 years

9

The long version is that networked file systems introduce new modes of failure that the storage node code is not tested against. Hence nobody will officially support it. Besides, to be reliable enough and fast enough, networked file systems require careful configuration, and therefore skills to make it so—probably nobody wants to have yet another flood of questions coming from misunderstanding basics of networking.

You would probably break some T&C items as well, but…

It is technically possible to set up a reliable and fast node with NFS or SMB, but you’ll be on your own. I have some notes here on this idea.

There used to be a major obstacle in form of sqlite3 databases just not working correctly on networked file systems, but for some time now it is possible to configure nodes to keep them in a separate location. They’re small, so they can be placed on the local OS drive.

BTW, with regards to Windows, make sure you’ve got the right license for setting up the node.

10

I would suggest always keep DBs on local fast drive, it will reduce unnecessary IO much.

11

Lots of great answers. I’ll try to answer them all in one comment, to not clutter the thread.

@Stob - I’ve tried checking my other nodes logs, and besides one being on NFS and one being on ISCSI, no of them are experiencing many errors or especially high Memory usage. It could be because they are sub 1TB nodes, or it might be something else. Do you care to elaborate? Thanks!

@arrogantrabbit - RAM is exactly what about gets used. Unused memory is wasted memory, I want as much to go in cache as I can - and even with only 9 GB of RAM, windows is not utilizing the entire RAM pool as caching yet. I suspect this is just because my node is not very large. I would love to hear other operators, do you also have 80GB woth of RAM for a 30 node?

I completely agree on the point on designing the host so most possible filesystem operations can be cached.

@vladro - good Idea, but my Synology does not support VVM. I can install some harddrives in my VMware boxes no problem, but each VMhost will as of right now be limited to a single HDD, greatly limiting capacities.

@Toyoo - I am leaning more and more towards just taking some of the older FC gear from work and using that. I don’t want to if I don’t have to. My largest node is only around 7TB, so I am still very much a small fish.

You would probably break some T&C items as well, but…

I cannot find anything in the T&C about this. Why would I break T&C by using a different storage backend? I’ve tried reading through them all. @Alexey I’ll allow myself to ping you on this matter. Additionally, T&Cs point 5.3 states that “[node operator cannot] Operate more than one (1) Storage Node behind the same IP address”. As I understand from the forums, this is the suggested way of extending beyond a single HDD - how is this a rule?

BTW, with regards to Windows, make sure you’ve got the right license for setting up the node.

What do you mean by this? I am running my nodes on Win10 Pro, but running on home is supported

@Vadim - I agree. I am running my VMs on VMHost local NVMe storage, which houses all my Windows machines C:\ drives. This thread is not about the DBs

12

I believe it would be this:

  • 4.1.4.2.5. meet all performance requirements referred to in this Agreement, as well as any performance requirements set forth in the Documentation or in other instructions made available with the Storage Node Software or otherwise hereunder;

The documentation clearly states that networked storage is not supported. I conservatively consider this as not meeting requirements, but optimistically assume this rule is simply outdated, like many other rules in that document.

It’s pretty simple—you can operate multiple HDDs in a single PC, as long as you have multiple external IPs assigned to the PC :stuck_out_tongue: Again, there were many comments stating this rule is pretty much obsolete.

What you link is a technical availability. What I have in mind is this entry from Microsoft’s retail EULA (which might have changed since last time I checked it—too lazy to verify current terms of use):

this license does not give you any right to, and you may not: […] (v) use the software as server software, for commercial hosting, make the software available for simultaneous use by multiple users over a network, install the software on a server and allow users to access it remotely, or install the software on a device for use only by remote users;

Given that a storage node is software making significant pieces of Windows networking and OS stack to act as servers, and its owner receives compensation for services making operation qualify as commercial, I can’t imagine this rule would not be broken with just a standard Windows Home license.

What’s more, in your specific case you need to be careful with running under VMWare, as each single Windows Home license only grants you rights to a single VM.

2 Likes
13

The network filesystem (NFS, SMB, SSHFS, etc.) have a different files lock mechanism than a local filesystems like NTFS or ext4, thus they behave differently for storagenode. The storagenode software uses files locks in many places, starting from SQLite databases and include storage (during upload for example). So you will always have problems either with databases or with storage (especially for uploads - they may stuck), this is if we even will not talk about higher RAM usage due to worse latency.
So if you forced to use the network to attach your storage, the only working network protocol is iSCSI, because it provides a block storage not the filesystem and you then uses the normal filesystem like NTFS or ext4.
If your storage is attached locally to the host, then you may also create a virtual disk file on it and attach this virtual disk to the VM, this way you may remove a Middleware such as iSCSI.

14

I also use an iSCSI LUN for my node’s storage. The storage array is hard drives, but has SSDs for read and write caching. Latency is ~1-1.5 ms.

15

Yes, the iSCSI protocol also have a less latency than any network filesystem, but local is better :slight_smile: at least from availability perspective, the network could blip, especially if not setup with HA.

16

Are NFS locks the only concern about using it as storage backend ? If so that can be sorted out, because there are no issues at all with regards NFS performance. It can perform as good as iSCSI.
There are very robust and critical systems that support NFS for mission critical systems like VMware ESXi hosts. At the end all comes down to the NAS that plays the NFS role server and from experience it is unnoticeable any performance penalty between NFS or iSCSI if the backend device is prepared for that.

There are tunnings that can be done when mounting NFS volumes, so if we can understand what limitations are around we can sort it out and make NFS a viable option.

17

Since it hasn’t been mentioned (surprisingly) I have to say; your Synology is almost certainly perfectly capable of running a docker node without any virtualization at all. This would allow it to freely use RAM where it’s needed most because it isn’t pre-assigned to any VM. I understand that you are more comfortable with windows, but there are very clear instructions to get it up and running with docker and we’ll gladly help you set it up. This would eliminate all virtualization overhead and the use of network protocols. It would also speed up transfers, likely leading to better income from your nodes. You can just install docker from the package center and connect over SSH and follow the docs from there. This is how I run my nodes and it’s by far the smoothest way to do it on Synology.

5 Likes
18

Thank you for all the great responses.

@Toyoo, good points on Windows licensing. I will be sure to read up on that. I am running licenced Win10 Pro, so I think I am all good. It never hurt to check again

@Alexey I’ve redone my storage backend. VMhosts has storage mounted directly to them. I use that to make virtual directly attached Harddisks for the VMs as suggests. Local datastores are just raw disks passed directly to the VMhost without any formatting. I am keeping an eye on IO performance.

For the largest nodes, I will need to continue with the ISCSI stores for now, but that can change in the future.

@cosmo thank you for your point. How large is your node on ISCSI storage?

@BrightSilence my Synology cannot run docker natively (unsupported ARM chip), and I will not be using docker from the commandline. I could very easily spin up a ubuntu server VM and let that be docker host, but Windows is what I am comfortable with.

19

It is 6 TB, with 3.25 TB used.

1 Like