I have an old NAS that was still in good shape and had a useful amount of storage, but wasn’t able to run Docker containers. It does offer iSCSI, so I set up a lightweight Alpine Linux VM on my Proxmox cluster and connected it to the NAS using iSCSI.
Along the way, I noticed that while I could manually connect to the iSCSI target and, after adding a suitable fstab entry, could mount the filesystem to a local mountpoint, whenever the VM would reboot it wouldn’t mount the filesystem and the node would fail to start. I wrote an OpenRC init script that waits until the iSCSI target is fully connected and then mounts the filesystem before Docker loads.
I did a write-up here that goes through the whole process and includes the init script. I hope some people find it useful. It’s mostly written as a set of notes to myself, so I’d appreciate any feedback, comments, suggestions, etc. that could improve its usefulness to others.
Moreover, you likely might be able to run the node directly on NAS, but without docker (you can also configure the storagenode-updater service as well):
Thanks! It’s been a while, but it’s good to be back.
Good thinking. I tried this earlier, but I found that it was unsatisfactory for a few reasons:
It requires modifying the Docker init script, which could be modified by future updates of the Docker package. This could overwrite the changes made to make it dependent on iscsid. I wanted to avoid modifying third-party scripts, so making my own script that won’t get clobbered by third-party updates seemed better.
The way Alpine handles mounting of filesystems complicates things. My understanding (which could be mistaken, I welcome any corrections) is that the system treats iSCSI-connected disks as if they were local, at least when it comes to mounting them. localmount runs early in the boot process, typically before iscsid. Thus, having the entry in /etc/fstab to mount the iSCSI-connected disk early as a local disk fails because the disk isn’t available (the network isn’t up and iscsid hasn’t connected).
Adding _netdev to the line in /etc/fstab for the iSCSI disk didn’t work either. This tells the system to wait until the network is connected before trying to mount a network drive, which works well for NFS/SMB/CIFS connections, but would fail for iSCSI. After doing some testing, it appears the problem is that the network comes up and there’s a race condition between (a) iscsid connecting to the target and showing the iSCSI disk as available for mounting and (b) the system process finding and mounting network shares. Since iscsid takes a moment to connect to the iSCSI target, it usually loses this race, so the system doesn’t see or mount the iSCSI disk. (In other words, the network being up is necessary but not sufficient for the iSCSI disk to be mountable. iscsid must also be connected.)
The script resolves all three issues by telling OpenRC that it depends on the network and iscsid and that it must run before Docker starts, and it checks to make sure that iscsid is properly connected to the target and that the iSCSI disk is mountable before trying to mount the disk. Once mounted, it lets the system continue to boot, load Docker, etc. (It also has some timeouts that will let the script fail noisily with errors if there’s problems and not block the system from booting.)
As for running the node directly on the NAS without Docker, that’s an interesting idea. I’ll have to look into it. Thanks!