Bare metal Linux, Ubuntu, and Docker: Which isetup is best?

[lookitsbenfb]

In another post, I’m in the process of compiling best practice based on users’ feedback, and there are a few issues which I’m trying to ascertain whether they are simply based on personal preference, or if they are actually grounded in fact.

One of which is the base setup: I don’t even know the correct way to phrase this, but I’m trying to understand which is best, I think these are the options, but please correct me if I’m wrong:

1. Bare metal Linux installation, no Docker
2. Bare metal Linux WITH Docker
3. Ubuntu (or similar) with Docker

So, I’d love to open up the debate here:

Firstly, it would be really helpful to me (and hopefully others, I’ll include it in the best practice summary) to understand the first principals: how each of these interacts with Storj.

Then, subsequent to that: what are the pros and cons of each, and how should a new user determine which setup is best?!

Thanks!

[Mitsos]

I’ll start the debate with the baremetal pros and cons:

Pros

• Easy to setup and forget. There isn’t any fiddling you need to do with it to get it to run properly.
• Easy to scale up. You copy the systemd service file, change the user/group/path and you have another node (after you generate a different identity of course)
• Easy to properly keep up-to-date. You don’t have to worry about a version downgrade nuking your node while you are at work and can’t deal with it. You update when you want, it stays on that version for as long as you want (=as long as it’s not below the minimum version).
• No overhead whatsoever. You run the node as fast as your host system can.
• Configuration changes are just a config file edit and a service restart.

Cons

• Requires a bit of knowledge. As long as you can use adduser, edit a file, cp, mv, you can get it running.

[littleskunk]

@lookitsbenfb

Bare metal Linux WITH Docker

You can install and run docker for other services like grafana but because you want to be the fastest node for winning the long tail race you better don’t run the storagenode in docker. Even with networking host setting it still is a latency penalty. A small one that shouldn’t matter to much but why not avoid it?

[ACarneiro]

Is that latency penalty relevant, though?
My nodes are running on docker and I’m still seeing success rates consistently in excess of 98%. I would suspect there are other considerations that are far more relevant for race wins than the docker latency, surely?

[littleskunk]

Try it out. Setup a node outside docker and test if it might get you to 99% success rate.

[pangolin]

I don’t care about tiny differences in success rate. The reason for migrating my nodes to bare metal is storj’s update policy for docker. Waiting weeks for a much-needed bug fix to be deployed is unacceptable.

[ACarneiro]

The other thing I like about the docker option is the “portability” of the nodes.

I can move a self-contained HDD with all the node data from one machine to another and all I need to do is a docker command on the recipient end.

[Ambifacient]

You will need to manually update the node using bare metal, and you can do the same with docker so I am confused with this sentiment.

[pangolin]

@Ambifacient

and you can do the same with docker

How? Please enlighten me.

[Ambifacient]

I have an example here:

How to force update to 1.104.1? (docker)

If you are running docker on linux, this is pretty easy to do, since storagenodes are packaged as binaries, so we simply overwrite the storagenode binary inside /app in the container, and then restart the container. For example this pulls a specific version from Github: #!/bin/bash VERSION=v1.104.1 CONTAINER=storagenode_satet_d1n2-node-1 ARCH=$(docker exec $CONTAINER printenv GOARCH) cat <<EOF | docker exec -i $CONTAINER bash wget https://github.com/storj/storj/releases/download/$VERSION/sto…

The gist is that you simply replace the storagenode binary in the docker container with the updated version.

There is also a flag to disable the version updater, but according to an entry in Gerrit it’s bugged at the moment. But so far I have not had my nodes downgraded to an older version.

[mattventura]

I’m assuming you meant “VM” in opposition to “Bare Metal”. Ubuntu is just another Linux distribution.

Docker is just significantly easier to manage, especially for multiple nodes. A docker-compose file is far cleaner than any other setup. For example, if I want to change my payout address for all of my nodes, I change one line in the file, run a single command, and that’s it.

If you use VMs for multiple nodes on a single box, then the additional overhead of that is probably going to kill any miniscule performance gain you could hope to squeeze out by not containerizing the node.

You can also use Docker inside of a VM so that you only get the overhead of a single VM rather than multiple. You can optimize the performance of this by passing through an HBA and NIC (or a VF) directly to the VM, giving it close to baremetal network/storage performance.

[pangolin]

Thanks @Ambifacient. TrueNAS Scale seems to be not plain vanilla docker but I will give this a try.

[jammerdan]

I have them on Docker and they are easy to setup and manage. But Docker sometimes has some nasty issues where it cannot stop a container and needs a Docker restart itself or even a reboot.
I think at some point I will move my nodes to bare metal but not yet. There are other issues that needs to be solved first.

[pangolin]

The suggested procedure to force a docker upgrade did not work for TrueNAS Scale. I ended up replacing the storj version info server with my own one (VERSION_SERVER_URL).

[Alexey]

It would be even more convenient, if you would use a docker-compose.yaml. I’m store it on a smallest node’s HDD.

[salna]

I’m running lxc container without docker.

[salna]

I never got proper running TrueNAS storj app all the time was a problem with QUIC.

[Alexey]

Hello @salna,
Welcome to the forum!

You may ignore this problem, since

QUIC misconfiguration in v1.86.1

There hasn’t been any progress on those QUIC bugs. It’s considered low priority because apparently uplinks quite rarely use QUIC in production. I suppose we’re recommending not to worry about what the dashboard says about your QUIC status for now.

But usually it affects either a FreeBSD or ARM systems (do not ask me why). If you have a AMD64, you just need to check that’s everything correct:

Step 4. Configure QUIC - Storj Docs

[arrogantrabbit]

@Mitsos

Requires a bit of knowledge.

I’d argue this requires less knowledge than using the docker. Storagenode has extensive help, it’s already self-contained executable with no dependencies, and you don’t need to fight against container orchestrator.

The best way is always the one that involves fewer layers.

@littleskunk

Try it out. Setup a node outside docker and test

If you are running docker specifically on windows or macOS then you have a VM in between. Of course everything will be horseshit.

Containers on the OS have no computational overhead, they share kernel with the host. There is no difference literally.

@Ambifacient

You will need to manually update the node using bare metal

No, you run storagenode updater to take care of that.

@pangolin

docker but I will give this a try

I can’t comprehend why is everyone so hell bent on docker. Advertising? It’s a commercial, shitty solution. There are other containerization solutions that are lower overhead and much better. Read about podman. It literally generates systemd wrappers for you, it does not have a daemon, etc.

Edit. I just realized it’s an ancient thread… sorry for muddying the water. I got misled by helpful discourse again.