Considering shifting from Docker to Podman on my Ubuntu 22.04.1 LTS - Has anyone taken this on? Is it really as simple as “apt get podman-docker” or were there additional steps to get the STORJ container up and running? Does wachtower still work? Any gotchas?
Thanks in advance.
I haven’t tried it, but from what I know it’s a replacement-ready. Like mariaDB and mysql.
The only difference is to explicitly pull the image from docker.io
Agreed, my concern was the lacking documentation here for it and how tools like watchtower are configured to work in partnership.
Tried running this, seems like Watchtower shut it down for an upgrade, but didn’t start it back up for me…
I would like to move to podman because some of my machines are running RHEL9 based distros. I read this guide prepared by @arrogantrabbit [link], however, somehow can’t make it work with storj. Im receiving:
Job for container-storagenode.service failed because the control process exited with error code.
See "systemctl --user status container-storagenode.service" and "journalctl --user -xeu container-storagenode.service" for details.
And almost no info in those logs.
I am wondering, is there maybe a guide specifically written about running storagenodes with podman?
Wondering, maybe you, @arrogantrabbit, may have some additional info. I have seen several of your posts advocating for podman, thus this question.
What’s in your storagenode’s logs?
Almost nothing, because as indicated above cant make it working:
Error: no container with name or ID "storagenode" found: no such container.
I can try to clean and paste the whole sequence of commands I tried that were influenced by @arrogantrabbit post, however, I was wondering if there is maybe a guide on this topic somewhere. Pls let me know, if there is no guide, I’ll clean and paste here all the commands I tried.
I’ve used podman with CentOS 8 and storagenode was more or less equivalent to docker procedure. The only difference was it needed --privileged flag because of some restrictions back then. Can’t tell if this is still true.
In my case I found it easiest to install cockpit and setup the container using this tool. Pairing this up with a Dozzle container to monitor/search logs realtime was a also a huge help.
IIR - I was able to pull the container, set my variables and container rules and fire it up, the larger problem was keeping it up to date with Watchtower.
Thanks @revyte n @cpare. Re: --privileged and Watchtower, thats why I asked this question in relation to the method proposed by @arrogantrabbit (please see the link in my post above #20199). It seems that he is resolving both of those issues, running storj container with podman in rootless mode with systemd and also dealing with SELinux and updates. It seems to look like an elegant solution. It is also somehow similar to running official native storagenode binnaries with systemd as proposed here [link].
However, the case is: i) Im not super familiar nor with podman, nor with RHEL, nor with SELinux; ii) the examples provided by him are useful but not exactly about storj; iii) never run storj with native binnaries and is seems there is no official guide on this (or is there?) thus I decided to ask this question with a hope of receiving opinionated solution from him or any of you guys.
Or in the other words, Im just a hobby RHEL user, after the lecture of his notes it seemed rather trivial, however, if something does not work as expected from the very beginning, there are more then two questions. So I do think that simpler would be to have an opinionated piece of code directly related to storj and podman.
Only for docker but the link to the guide you provided looks like it should work. systemd is not the only option but running as service is a good one.
Overread the “service” part and tought you are running it just like docker.
I’m not sure where in the process you get the issues. Generate was fine but you can’t start now`?
Looks like your service don’t exist in “.config/systemd/user/”
Generate was fine but you can’t start now`? Looks like your service don’t exist in “.config/systemd/user/”
It seemed to look that the service files were indeed generated but when trying to run the setup I got the message that I included in my initial post, it is:
Job for container-storagenode.service failed because the control process exited with error
code.
I will look at my notes soon and hope to provide the commands that I was trying (cant do it immediately).
How did you create service definition? Did you pass --new flag? The container shall not exist prior to the service start; the service will create it. So, the content of the message is correct and expected; but the fact that you see the message means service wrapper expects the container to exist, which likely means you did not pass --new flag to the podman generate systemd call
Tried to avoid it with a hope of receiving clean lines, however, as you were insisting @revyte [:-)] so here is my snippet. But seriously, would you have any comments?
What Im doing different than @arrogantrabbit in his guide:
mkdir -p ~/.config/systemd/user)--label "io.containers.autoupdate=registry"” (seemed not to work in my case?)-e PUID=$(id -u) and -e PGID=$(id -g) instead of hard coded values of 1000 (not sure if this is ok, but would be my preference)-e TZ=America/Los_Angeles” (prefer my server UTC time):Z after --mount (tried to used it with it as well - dont remember what was the outcome)storj-zone” in order to allow access to dashboard only from a specified ip (not sure if this is correct - doing it for the first time with firewall-cmd - its probably not, because it looks I almost lost ssh access to my server)## Change user
# We will be running containers as a user, rootless, and therefore we need to allow processes launched by our user to persist2:
sudo loginctl enable-linger user
## SELinux
# For systemd to be able to manage container add container_manage_cgroup permission3:
sudo setsebool -P container_manage_cgroup on
## Create directory for systemd files
mkdir -p ~/.config/systemd/user
## Open firewall ports
sudo firewall-cmd \
--permanent \
--add-port=$port/tcp \
--add-port=$port/udp
## Create a new zone and allow connection from desired IP in orer to access dashboard
sudo firewall-cmd --new-zone=storj --permanent
sudo firewall-cmd --reload
sudo firewall-cmd --zone=storj --add-source=$ctlhostip/32 --permanent
sudo firewall-cmd --zone=storj --add-port=$dport/tcp --permanent
sudo firewall-cmd --reload
## Firewall check
sudo firewall-cmd --list-all
sudo firewall-cmd --get-active-zones
sudo firewall-cmd --zone=storj --list-all
## Storagenode setup
podman run --rm -e SETUP="true" \
-e PUID=$(id -u) \
-e PGID=$(id -g) \
--mount type=bind,source="$storjidlocation",destination=/app/identity \
--mount type=bind,source="$datapath",destination=/app/config \
--name $snname storjlabs/storagenode:latest
## Create
podman create \
-p $port:28967/tcp \
-p $port:28967/udp \
-p $dip:$dport:14002 \
-e WALLET="$wallet" \
-e EMAIL="$email" \
-e ADDRESS="$pip:$port" \
-e STORAGE="$ssize" \
-e STORJ_HEALTHCHECK_DETAILS="true" \
-e PUID=$(id -u) \
-e PGID=$(id -g) \
--mount type=bind,source="$storjidlocation",destination=/app/identity \
--mount type=bind,source="$storjconfiglocation",destination=/app/config \
--name $snname storjlabs/storagenode:latest --operator.wallet-features=zksync
## Generate service files
podman generate systemd \
--new --name $snname \
--restart-policy=always \
> ~/.config/systemd/user/container-$snname.service
## Remove container
podman rm $snname
## Enable and start services
systemctl --user enable container-$snname.service
systemctl --user start container-$snname.service
systemctl --user status container-$snname.service
The outcome is:
Job for container-storagenode227.service failed because the control process exited with error code.
See "systemctl --user status container-storagenode227.service" and "journalctl --user -xeu container-storagenode227.service" for details.
● container-storagenode227.service - Podman container-storagenode227.service
Loaded: loaded (/home/user/.config/systemd/user/container-storagenode227.service; enabled; preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since Mon 2023-07-10 18:20:10 GMT; 18ms ago
Docs: man:podman-generate-systemd(1)
Process: 2706395 ExecStart=/usr/bin/podman run --cidfile=/run/user/1000/container-storagenode227.service.ctr-id --cgroups=no-conmon --rm --sdnotify=conmon -d --replace -p 28967:28967/tcp -p 28967:28967/udp -p someip:14>
Process: 2706438 ExecStopPost=/usr/bin/podman rm -f --ignore -t 10 --cidfile=/run/user/1000/container-storagenode227.service.ctr-id (code=exited, status=0/SUCCESS)
Main PID: 2706395 (code=exited, status=126)
$ systemctl --user status container-storagenode227.service
container-storagenode227.service - Podman container-storagenode227.service
Loaded: loaded (/home/user/.config/systemd/user/container-storagenode227.service; enabled; preset: disabled)
Active: deactivating (stop-post) (Result: exit-code)
Docs: man:podman-generate-systemd(1)
Process: 2713635 ExecStart=/usr/bin/podman run --cidfile=/run/user/1000/container-storagenode227.service.ctr-id --cgroups=no-conmon --rm --sdnotify=conmon -d --replace -p 28967:28967/tcp -p 28967:28967/udp -p someip:14>
Main PID: 2713635 (code=exited, status=126); Control PID: 2713676 (podman)
Tasks: 6 (limit: 150091)
Memory: 5.7M
CPU: 147ms
CGroup: /user.slice/user-1000.slice/user@1000.service/app.slice/container-storagenode227.service
└─2713676 /usr/bin/podman rm -f --ignore -t 10 --cidfile=/run/user/1000/container-storagenode227.service.ctr-id
Jul 10 18:30:58 hostname systemd[2567138]: Starting Podman container-storagenode227.service...
Jul 10 18:30:58 hostname podman[2713635]:
Jul 10 18:30:58 hostname podman[2713635]: 2023-07-10 18:30:58.284361915 +0000 GMT m=+0.060460683 container create b02cd874afbb8e916544f2a909c058b6fa4f10d5c9b920712958f29f6622ca5f (image=docker.io/storjlabs/storagen>
Jul 10 18:30:58 hostname podman[2713635]: 2023-07-10 18:30:58.25576147 +0000 GMT m=+0.031860278 image pull storjlabs/storagenode:latest
Jul 10 18:31:01 hostname podman[2713635]: 2023-07-10 18:31:01.319437228 +0000 GMT m=+3.095535956 container remove b02cd874afbb8e916544f2a909c058b6fa4f10d5c9b920712958f29f6622ca5f (image=docker.io/storjlabs/storagen>
Jul 10 18:31:01 hostname podman[2713635]: Error: rootlessport listen tcp 10.1.0.193:14002: bind: cannot assign requested address
lines 1-18...skipping...
● container-storagenode227.service - Podman container-storagenode227.service
Loaded: loaded (/home/user/.config/systemd/user/container-storagenode227.service; enabled; preset: disabled)
Active: deactivating (stop-post) (Result: exit-code)
Docs: man:podman-generate-systemd(1)
Process: 2713635 ExecStart=/usr/bin/podman run --cidfile=/run/user/1000/container-storagenode227.service.ctr-id --cgroups=no-conmon --rm --sdnotify=conmon -d --replace -p 28967:28967/tcp -p 28967:28967/udp -p someip:14>
Main PID: 2713635 (code=exited, status=126); Control PID: 2713676 (podman)
Tasks: 6 (limit: 150091)
Memory: 5.7M
CPU: 147ms
CGroup: /user.slice/user-1000.slice/user@1000.service/app.slice/container-storagenode227.service
└─2713676 /usr/bin/podman rm -f --ignore -t 10 --cidfile=/run/user/1000/container-storagenode227.service.ctr-id
Jul 10 18:30:58 hostname systemd[2567138]: Starting Podman container-storagenode227.service...
Jul 10 18:30:58 hostname podman[2713635]:
Jul 10 18:30:58 hostname podman[2713635]: 2023-07-10 18:30:58.284361915 +0000 GMT m=+0.060460683 container create b02cd874afbb8e916544f2a909c058b6fa4f10d5c9b920712958f29f6622ca5f (image=docker.io/storjlabs/storagen>
Jul 10 18:30:58 hostname podman[2713635]: 2023-07-10 18:30:58.25576147 +0000 GMT m=+0.031860278 image pull storjlabs/storagenode:latest
Jul 10 18:31:01 hostname podman[2713635]: 2023-07-10 18:31:01.319437228 +0000 GMT m=+3.095535956 container remove b02cd874afbb8e916544f2a909c058b6fa4f10d5c9b920712958f29f6622ca5f (image=docker.io/storjlabs/storagen>
Jul 10 18:31:01 hostname podman[2713635]: Error: rootlessport listen tcp someip:14002: bind: cannot assign requested address
Jul 10 18:31:01 hostname systemd[2567138]: container-storagenode227.service: Main process exited, code=exited, status=126/n/a
$ journalctl --user -xeu container-storagenode227.service
No journal files were found.
Looks fine on a first glance.
Can you show the ~/.config/systemd/user/container-$snname.service file?
If you look at this:
Process: 2706395 ExecStart=/usr/bin/podman run --cidfile=/run/user/1000/container-storagenode227.service.ctr-id --cgroups=no-conmon --rm --sdnotify=conmon -d --replace -p 28967:28967/tcp -p 28967:28967/udp -p someip:14>
It does not look healthy to me.
Oh, you also need to specify container registry you are using, i.e. docker.io/storjlabs/storagenode, not just storjlabs/storagenode
Hi, yeah, sure, here is the output:
$ cat ~/.config/systemd/user/container-$snname.service
# container-storagenode227.service
# autogenerated by Podman 4.4.1
# Mon Jul 10 18:19:54 GMT 2023
[Unit]
Description=Podman container-storagenode227.service
Documentation=man:podman-generate-systemd(1)
Wants=network-online.target
After=network-online.target
RequiresMountsFor=%t/containers
[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=always
TimeoutStopSec=70
ExecStart=/usr/bin/podman run \
--cidfile=%t/%n.ctr-id \
--cgroups=no-conmon \
--rm \
--sdnotify=conmon \
-d \
--replace \
-p 28967:28967/tcp \
-p 28967:28967/udp \
-p somedashbordip:14002:14002 \
-e WALLET=mywallet \
-e EMAIL=myemail \
-e ADDRESS=someserverip:28967 \
-e STORAGE=20000G \
-e STORJ_HEALTHCHECK_DETAILS=true \
-e PUID=1000 \
-e PGID=1000 \
--mount type=bind,source=/home/user/.local/share/storj/identity/storagenode227,destination=/app/identity \
--mount type=bind,source=/home/user/sj/storagenode227,destination=/app/config \
--name storagenode227 storjlabs/storagenode:latest \
--operator.wallet-features=zksync
ExecStop=/usr/bin/podman stop \
--ignore -t 10 \
--cidfile=%t/%n.ctr-id
ExecStopPost=/usr/bin/podman rm \
-f \
--ignore -t 10 \
--cidfile=%t/%n.ctr-id
Type=notify
NotifyAccess=all
[Install]
WantedBy=default.target
Will do and will report back, however, I’m recalling that I was asked about it by podman and made this choice interactively. Maybe the firewall? Frankly, really don’t know, feel a bit lost.
I had to do that on one of my containers recently as it failed to start following image update one day. Turned out, I missed specifying the registry on that one. Adding it fixed it.
The service definition looks fine, I guess it just tractated it there weirdly.
Your service definition looks fine on the
Yes, I am confirming, it is definitely better to specify: <docker.io/storjlabs/storagenode>, however, only <storjlabs/storagenode> seems to make no difference (at least as for now, and in this particular case). With <docker.io/storjlabs/storagenode> I’m receiving the same errors.
I see that the problem might be that:
Error: rootlessport listen tcp somepodmanip:14002: bind: cannot assign requested address
but not sure.
Should you guys have any suggestions, it would be great. Also I think that some opinionated piece of code directly related to podman and storj would be extremely useful. On the server Im doing those tests I am running a full LXD Ceph Cloud stack. Sometimes hard to diagnose whats wrong. EDIT: Just to be clear, I am running those tests not within LXD, apart to LXD its almost a vanilla server, without anything special.