Hi!
How do I set Docker-based Linux deployment so that databases are stored in a different directory than actual data? Both are by default stored in the /app/config bind-mounted directory. I want to put databases on a different storage — small, but fast SSD.
I tried setting symlinks to data directories, but then storj complains that Error: piecestore monitor: disk space requirement not met despite that the data directories symlink to a file system with plenty of space.
Please, do not use symlinks, you can destroy your databases: https://www.sqlite.org/howtocorrupt.html#_multiple_links_to_the_same_file
I would not recommend to move your databases to other disk - you will increase a number of points of failure.
Up to 12 times to be precise
Even if the current directory is an SMB share? Is SMB share a better place to store SQLite databases than a small, but local drive?
Especially if you use SMB which doesn’t supported at all, as a NFS.
The only working network protocol for storage is iSCSI
Alexey, I do not have enough disk space to store files on a local drive. I’m now running Storj from SMB. I would like to move databases to local storage.
Sorry, but SMB and NFS shares are not supported. You will have problems sooner or later.
You can study these topics right here, on forum:
https://forum.storj.io/tag/smb
https://forum.storj.io/tag/nfs
I’ve read these. They only state that running databases is not supported on SMB. That’s why I’d like to move only them.
I’m not sure that you will not have any problem with remained storage. It could not work too.
Try to switch your NAS to offer the space via iSCSI
I can’t do iSCSI in this system. But I have extremely well-maintained SMB, as evidenced by the fact, that I’ve run this node already for two months with zero problems and all success rates at >95%. Which is why I’m asking the question above. Is there any answer to that question?
Ok, it seems that without changing the source code, this is impossible. Thankfully, changing the source code if you don’t mind hard-coding paths to the database files is not difficult.
I agree with Alexey that you shouldn’t use symlinks or SMB at all, but if you are going to use them, use a local folder as your data folder and symlink blobs, temp and trash to SMB locations. That way all db’s inclusing shm and wal files will be local and only piece storage is remote. It would be safer than symlinking the databases, but I have no guarantee that this will work or work reliably and you’re still dealing with added latency for your disk writes and reads. You’d be on your own if you have any issues with this setup.
If I may ask, what system are those SMB shares on? Can that system run docker?
I cannot use a local folder because of the error message I posted in the opening post.
That SMB system is not under my control.
My bad, I should have read further back.
The only thing I can think of beyond that is set up a local thin provisioning file system which shows more space than is actually available. It starts to quickly become seriously hacky and a patchwork or less than ideal things.
Yeah, I settled on modifying these two lines and recompiling. Turns out, defining paths to all databases is mediated through these two lines.
why do you want the database on a SSD? It’s not required. Its a very lightweight database. What do you do if an upgrade does not detect what you have done and your node crashes and gets disqualified. Is if worth it?
Because the only local drive in this system is an SSD. The only alternative is SMB, which is risky for database files.
At this point I’ve disabled automatic upgrades. They’re rare enough that it’s fine for me to do them manually.
Have you tried mounting /config to a local folder and /config/storage and /config/trash (paths may not be correct) to SMB? I don’t know if docker supports nested mounting but that would solve your issue. That way everything would be local except the large storage folder
Docker does support nested mounts, I used it for putting one of database files onto the SSD while keeping everything else on HDD.
Yes, that’s what I described in the first post.
Didn’t you say you created symlinks instead? According to @xopok this method can be used to do separate databases from blob storage, so I recommend trying it