Yeah, unfortunately, NFS has always suffered from problems with file locking (and probably always will). See https://sqlite.org/lockingv3.html for some of the specific problems with SQLite databases like the ones used in node software.
Ideally, no one should be putting their database dir on remote storage at all; it would fit better just to mount the blobs dir for remote storage.
Ideally the database executable would be run somewhere else with a local storage and be accessed over the network, but that would be more complicated. We should at least have a separate mount point, as I, and many others, have suggested. I know there is a way to do it ourselves, but I would prefer to have an official and supported way and instructions for it as I am not too savvy with Linux.
For most of us this will get solved with native packages for our NAS/SAN systems. Windows native package is out, so hopefully it isn’t too far now.
The relevant quote is (emphasis mine):
SQLite uses POSIX advisory locks to implement locking on Unix. On Windows it uses the LockFile(), LockFileEx(), and UnlockFile() system calls. SQLite assumes that these system calls all work as advertised. If that is not the case, then database corruption can result. One should note that POSIX advisory locking is known to be buggy or even unimplemented on many NFS implementations (including recent versions of Mac OS X) and that there are reports of locking problems for network filesystems under Windows. Your best defense is to not use SQLite for files on a network filesystem.
The advice “don’t use SQLite on SMB/NFS” is therefore not really originating with Storj support, but rather comes directly from the SQLite developers.
iSCSI is obviously not susceptible to these kinds of issues, as it presents a raw block device instead of a filesystem and a “local” filesystem is used instead. (The local filesystem driver is not aware that one of the below layers is remote, and it doesn’t need to care.)
I find it amazing that a known bug can go on for decades…
The sqlite is incompatible with any network connected storage with a network filesystem.
But it could sometimes work. For the migration it doesn’t work unfortunately. The iSCSI do not have such problem
I do agree. I’m trying to narrow it all down to something that doesn’t make the setup too complicated, but still leaves the databases on a separate local mount point. It’s hard. I may have to ask for help from the ever-capable @Alexey or @BlackDuck or @Knowledge.
With a docker you could mount any file separately. However, the sqlite uses log journals too, for example, info.db-wal and info.db-shm, it doesn’t makes it simpler.