It would not be pointless, because usually space requirements do not change rapidly. If you have a 20TB server and 5TB of data, you can allocate 11TB to the node, but later gradually shrink it when you fill up the server with your own data. Right now there is no way to do it, so you have to be absolutely sure that you won’t need that space, ever.
Hence why I said it’s more likely for those who bought a too big drive by mistake, rather than with future usage in mind.
Multiple small nodes increase the time needed to monitor and set them up, driving down the “hourly wage”.
Not with the reliability requirements and no direct use for myself. I can seed a torrent I downloaded because, well, I downloaded it already and seeding is just the matter of not turning off the client (or moving the files to another server).
Small margins are OK, as long as volume makes up for it.
Right now we have smaller payments and small usage:
Well, in the worst recent month, I still got 600GB of ingress. For our larger nodes, that’s not a lot, since we are hit with deletes as well and it leads to small net growth, but for smaller nodes, this is a meaningful amount that leads to significantly higher net growth. This is why I test with new nodes from time to time.
Egress is unfortunately barely relevant anymore.
They did a survey. I believe they reported it was 70% existing hardware. Though they haven’t clarified exactly what that number means. I run on an existing always online NAS, but bought an external 10-bay USB enclosure and quite a few additional HDD’s. I don’t know how they would count my setup.
Ehhh. Zero times ten is sill zero. What monitoring are you talking about? Do you spend time monitoring your node? Why?!
Initial setup is fixed cost, it does not matter how many nodes you setup.
Reliability requirements of storj is much much lower than other things I run the server for. Both in uptime and data integrity. Hence, it makes no difference.
If you can find other use for extra space for yourself — it’s not “unused resources”.
Well, as I said, multiple small nodes solves it. Run 10. Yank them one by one once you need space. You get space back instantly. No waiting involved. What can be better than that?
Once the setup is complete, I still need to monitor the node once in a while to see if it running correctly and sometimes update it, expand the virtual disk when the space goes down etc.
Every new node requires some setup - creating the VM, port forwarding, adding to the monitoring system.
Also, Storj apparently does not like when people run multiple nodes on the same drive/array/pool.
I can have backups of everything that’s in my server, except for the node. Though maybe now that ingress is super low it would be possible to get away with it, still, it is not supported by Storj.
Being able to control the allocated space in smaller increments. And being able to reclaim the space sooner than waiting however long for the node to qualify for GE.
Mmm. The former is accomplished by automation — kuma or uptime robot and the likes. Expanding virtual disk is strange — give it all free space. Then you don’t have to expand.
Node in VM is a bad idea from performance perspective. You don’t benefit from shared filesystem cache. But that’s beyond the topic; either way it’s a one time setup. Setup all nodes at once with a script. Literally for loop in bash
Why are you creating individual port forwards? Forward entire port block, and when starting the node — just increment port number. No other changes needed.
Storj is not a person, it does not matter what it likes. What matters is published operator terms and conditions document. And it says — max one node per hard drive. So if you have an array of 10 drives you can run 10 nodes on it. At least in the current incarnation of ToS.
They all will share space so you don’t have to micromanage space allocations either, going back to the first point.
Node data is ever evolving live should not be backed up. It would make no sense. From one hand, restoring old data will disqualify your node, unless you also unwind the rest of the universe along with customer data and satellites. From another — node data is not that important. Storage network is designed to tolerate massive number of lost segments, by design. You could run node on a single HDD with ext4 without data consistency checks if you wanted to. It is not realistic usecase — because that the only acceptable use of a non-redundant storage, and we agree not to build hardware just to storj. So you already have a redundant array for your primary purpose and hence nodes get to benefit from overkill reliable storage.
That would be ideal indeed. But it’s not implemented today. Hence, multiple small nodes is the only way to get more granular control.
I don’t think implementing it is difficult, just low priority. Node could just keep dropping least frequently used or most redundant/healthy segments and forcing repair, until desired space is reached. With some safeguards to prevent abuse. It’s better, because alternative is user killing entire node. So, win-win.
Since I am suposed to use the hardware that I already own and use, I am running the node in a VM along with my own VMs on that server. I have to allocate space on the pool for the virtual disk and I do not want to overbook the space because if the space on the pool runs out, there would be big problems.
So, I would allocate a few TB to the node and, as the node grew, I would periodically expand the virtual drive (assuming I had space in the pool). The recent shutdown of europe-north-1 left my node with 9TB of free space though, so I won’t be expanding that virtual disk for a while.
Let’s say I lose my 20.9TB node that earns about $30/month. Assuming no deletes, no held amount and no vetting period, with 600GB/month ingress, it would take about 35 months for my node to grow back to 20.9TB, during which time it would earn less (on average over the entire period about half), so I would lose about $520 of income. In reality, it could be much worse. That’s why I would want to be able to back up the node and restore it if something happens.
That’s fine, I was just saying thid is sub-optimal, compared to all nodes sharing the same pool and the same filesystem cache.
To be clear, we are taking about catastrophic failure of your storage system such as flood or volcano eruption, where your whole array is destroyed, because losing a few files here and there is absolutely harmless.
In this case, you need to multiple your loss of income by the probability of this happening. You will find it’s about ten cents worth of risk, and therefore the whole point is moot.
By the way, absolutely nothing prevents you from sending hourly incremental filesystem updates to remote server that could take over running the node if the first server dies. But it would be much better bang for buck to run second node there. Because wasting resources for a very remotely probably event that otherwise could be used for guaranteed income is silly. This is true for any other system you would backup node to, if that was possible: just run another node on that system instead.
This assumes you have perfect knowledge on the operation of a node. In practice I had to add new parameters to monitoring, tune OS settings in order to keep the node operating as it gained more data and handled more traffic, simply because operating a 2 TB node is a different task than operating a 20 TB node.
Once you gained the knowledge (not sure if there is anything specific to storj – all those optimizations for file access are fairly generic and applicable to wide range of scenarios) to setup one, it takes no extra effort to setup another few:
You need external monitoring solution for one node, and this is where bulk of work goes into and once you have it adding 14 more is trivial,
You need to learn how to init, configure, and start the node. This is where bulk of work go, and once you start one – starting 14 more is trivial.
See what I mean? Experience is highly reusable.
Anecdotally, I did not find any difference or had to do any additional tweaks since the node was 1 month old to its current size of about 14TB. It was using negligible resource then, and it uses negligible resources now. Once you realize you need “improve random access to small files” – you do that, and that’s the end of configuration. Note, number of nodes is not a factor - its’ the same for every one of them.
Realizing that docker network proxy was, due to a bug in docker, eating 20% of CPU, stealing it from other activities which needed as much CPU as possible, then rewriting my scripts to use host network and testing them in practice (did you know that for this to work for multiple nodes you need --server.private-address?) was alone two-three hours of work. At my current hourly rate that’s 5 months of my nodes’ revenue.
Well, no, I avoided the whole distraction by not using docker where it is not needed in the first place… Still baffles me why people waste time and take on more risk containerizing go apps, but I digress.
But again, this is a docker bug you hit, it’s not specific to storj in any way. If you were using 15 other different containers you woudl have hit the same bugs.
No, on one hand I had fair share of issues with docker in the past to avoid it as a plaque. If I need to run container on linux I do that with podman, rootless.
On the other hand – general principle of “simple is better than complex”, “lean is better than bloated”, etc… It’s a general approach of “keeping things simple” and not bringing unnecessarily dependencies into the system.
Why do you say it’s a recommended approach? Recommended by who? Storagenode is a command line self-contained executable, with self-describing command reference. What do I need docker for? Seems like 100% waste of time, resources, and inviting extra bugs, as your experience demonstrates.
Hmm… Maybe I should do it, maybe not. I wonder if an hour old backup would still be OK.
Running a second node would only make sense if my first node was full or if I had another public IP. Otherwise, I am not getting any more traffic.
As for why I would want to have backups - well, some major disaster is not the only way to lose data, it is possible to lose data by mistake - deleting the wrong file, messing up the file system etc.
There was a time when docker was the only option available.
Oh yes, this confused me at first too. Why are they talking about docker in the “CLI install” section. But then docker is just wrapper, so I was going to figure out what do they run there from a Dockerfile, and then it turned out the command line utility is enough so I did not bother looking there.
They shall fix the documentation. Rename current “CLI Install” into “docker install” and create a proper “CLI configuration” section. Ideally, with podman, and point out how to generate service wrappers. Why use commercial product when open source alternative exists?
Wasn’t storagenode always open source? What did they put into container then?
They did not want to make different binaries for different distributions, so they used docker in the beginning. Docker was the only option - for Linux and for Windows.
Separate binaries came much later.
Storage node was always open source, but at that time, if you did not want to use docker, you would have to compile it yourself and IIRC the self-compiled version may not have worked with the official satellites (I do not know this part exactly, I never tried to compile the node, docker seemed simpler than compiling).
Podman is exist since 2018 (and for RHEL only on that time), but docker since 2013 and it has had a wider adoption and knowable by the Community. Podman also was not available for Windows and WSL (even v1) was not exist, and WSL did not work with the containers until wsl2 anyway, but there was a Docker Desktop.
We did not want to scatter resources to support many different platforms and methods to run it in different container solutions and explain to each novice how to run a binary, so docker was a tradeoff, simple distribution and manage and also cross-platform.
Now Operators have much more wide choices how to run storagenode, or containers, include Podman, Rancher Desktop and many other alternatives like Kubernetes, Nomad or docker swarm.
