Bare metal Linux, Ubuntu, and Docker: Which isetup is best?

In another post, I’m in the process of compiling best practice based on users’ feedback, and there are a few issues which I’m trying to ascertain whether they are simply based on personal preference, or if they are actually grounded in fact.

One of which is the base setup: I don’t even know the correct way to phrase this, but I’m trying to understand which is best, I think these are the options, but please correct me if I’m wrong:

  1. Bare metal Linux installation, no Docker
  2. Bare metal Linux WITH Docker
  3. Ubuntu (or similar) with Docker

So, I’d love to open up the debate here:

Firstly, it would be really helpful to me (and hopefully others, I’ll include it in the best practice summary) to understand the first principals: how each of these interacts with Storj.

Then, subsequent to that: what are the pros and cons of each, and how should a new user determine which setup is best?!

Thanks!

I’ll start the debate with the baremetal pros and cons:

Pros

  • Easy to setup and forget. There isn’t any fiddling you need to do with it to get it to run properly.
  • Easy to scale up. You copy the systemd service file, change the user/group/path and you have another node (after you generate a different identity of course)
  • Easy to properly keep up-to-date. You don’t have to worry about a version downgrade nuking your node while you are at work and can’t deal with it. You update when you want, it stays on that version for as long as you want (=as long as it’s not below the minimum version).
  • No overhead whatsoever. You run the node as fast as your host system can.
  • Configuration changes are just a config file edit and a service restart.

Cons

  • Requires a bit of knowledge. As long as you can use adduser, edit a file, cp, mv, you can get it running.
2 Likes

You can install and run docker for other services like grafana but because you want to be the fastest node for winning the long tail race you better don’t run the storagenode in docker. Even with networking host setting it still is a latency penalty. A small one that shouldn’t matter to much but why not avoid it?

Is that latency penalty relevant, though?
My nodes are running on docker and I’m still seeing success rates consistently in excess of 98%. I would suspect there are other considerations that are far more relevant for race wins than the docker latency, surely?

Try it out. Setup a node outside docker and test if it might get you to 99% success rate.

1 Like

I don’t care about tiny differences in success rate. The reason for migrating my nodes to bare metal is storj’s update policy for docker. Waiting weeks for a much-needed bug fix to be deployed is unacceptable.

1 Like

The other thing I like about the docker option is the “portability” of the nodes.

I can move a self-contained HDD with all the node data from one machine to another and all I need to do is a docker command on the recipient end. :man_shrugging:t2:

1 Like

You will need to manually update the node using bare metal, and you can do the same with docker so I am confused with this sentiment.

2 Likes

How? Please enlighten me. :thinking:

I have an example here:

The gist is that you simply replace the storagenode binary in the docker container with the updated version.

There is also a flag to disable the version updater, but according to an entry in Gerrit it’s bugged at the moment. But so far I have not had my nodes downgraded to an older version.

1 Like

I’m assuming you meant “VM” in opposition to “Bare Metal”. Ubuntu is just another Linux distribution.

Docker is just significantly easier to manage, especially for multiple nodes. A docker-compose file is far cleaner than any other setup. For example, if I want to change my payout address for all of my nodes, I change one line in the file, run a single command, and that’s it.

If you use VMs for multiple nodes on a single box, then the additional overhead of that is probably going to kill any miniscule performance gain you could hope to squeeze out by not containerizing the node.

You can also use Docker inside of a VM so that you only get the overhead of a single VM rather than multiple. You can optimize the performance of this by passing through an HBA and NIC (or a VF) directly to the VM, giving it close to baremetal network/storage performance.

1 Like

Thanks @Ambifacient. TrueNAS Scale seems to be not plain vanilla docker but I will give this a try.

1 Like

I have them on Docker and they are easy to setup and manage. But Docker sometimes has some nasty issues where it cannot stop a container and needs a Docker restart itself or even a reboot.
I think at some point I will move my nodes to bare metal but not yet. There are other issues that needs to be solved first.

1 Like

The suggested procedure to force a docker upgrade did not work for TrueNAS Scale. I ended up replacing the storj version info server with my own one (VERSION_SERVER_URL).

It would be even more convenient, if you would use a docker-compose.yaml. I’m store it on a smallest node’s HDD.

I’m running lxc container without docker.

I never got proper running TrueNAS storj app all the time was a problem with QUIC. :frowning_face:

Hello @salna,
Welcome to the forum!

You may ignore this problem, since

But usually it affects either a FreeBSD or ARM systems (do not ask me why). If you have a AMD64, you just need to check that’s everything correct:

1 Like