I need to setup a node that works with a drive over internet (drive location upload speed is 200mbit/s). I understand that it’s not the suggested configuration but I need to know if there are any problems running with this configuration.
I think speed is enough for a basic node (4tb)
I think that iscsi is the only remote protocol supported. (Others might work)
I, personally, without evidence think that it would work with small sizes but 4 TB would not be optimal.
Assuming that:
M1 is the machine housing the HDD.
M2 is the machine running the Storj node.
M3 is the machine exposed to the Storj network.
Do you mean the configuration M1=M2, or the configuration M2=M3?
I am successfully running a Storj node (size: 2TB) using the configuration M1=M2. It works well.
I am not running the configuration M2=M3.
I’ve tried a setup like that about 2 years ago, and at that time it was close to unusable even at hundreds of gigabytes. The node consumed large amounts of memory, and was failing most requests at times, even despite that the storage was basically next rack, not going over the Internet.
Today I suspect it may work in some circumstances with a very, very careful setup. As you noted, and @Alexey will quickly confirm, this is a not supported configuration, and as such cannot be operated on the main network by T&C, but you can try it on, let say, the test network. You need to:
- Ensure that your remote file system will not fail randomly. Usually this means a hard mount, i.e. a way to make sure that in case there is a problem connecting to your remote drive, the file system will retry the connection, as opposed to passing an error to application. See some explanation on ServerFault.
- Store databases locally. They’re small, so it’s usually not a problem.
- Disable the initial piece scan.
- Disable forced syncing (requires manual recompilation, the official releases do not allow this).
- Enable all file attribute caching you can afford, even at a risk of lack of consistency. Even with disabled initial piece scan, the node will need to periodically scan all piece metadata, essentially listing all files stored. Here’s some example for NFS.
- Not use Windows. I just can’t imagine most of the above to work reliably on Windows
Of course, with all these changes your node will be a lot less reliable. Whether this is worth your time, and how strong is your “need to setup a node” this way, you should decide on your own.
2 Likes
M2=M3 but it’s interesting M1=M2 too… will be the same I think
Not at all. The node software requires responsive storage with low latency, but it doesn’t assume responsive internet connections. This is why M1=M2 setup is usually not an issue, but M2=M3 is. I would only advise the latter on iSCSI setups with solid reliable and low latency connections.
Which protocol are you thinking of?
If you are willing to try using a cloud service like S3 or GDrive, it won’t work (and it is not supporteed by T&C as pointed above).
The main reason why it wont work is that Storj is creating a vast amount of files, which will require vast amounts of RAM in order to cache the filesystm structure and metadata for it to operate, at least with solutions that allow you to mount those virtual FS locally, like Rclone.
If you are looking to use NFS or iSCSI, as it was said above, the most sucessful stories are with iSCSI, but it will be difficult to have it working smoothly if you are connecting to a remote network. It will be difficult because your access to the FS will be much more delayed because of networks ping, jitter, etc.
I suppose that M1!=M2 (that is: HDD (M1) is on a different machine than the Storj Docker process (M2)) would work fine, but I believe it requires a lot of system-level programming and testing to make it run reliably and with low latency, and it most likely requires M2 to cache (example size: 32 GB) some of the filesystem data stored on M1 (example size: 2 TB) and the cache would most likely need to be a write-back cache (not a write-through cache). In essence, a distributed filesystem spread over M1 and M2.