Moving from Docker to Podman

Node Operators
,
21

Oh dang it, of course it is! How did I miss that in your original log!

I would try launching the container manually using the command from ExecStart and see if this works. If that does not work either with the same failure – see which process is holding that port open on your host. Maybe you have some stale storj container hanging from earlier tries that did not die?

If it does work manually, but fails when systemd runs it – I’ll be surprised, but maybe there are some weird SELinux restrictions on you system on who can listen on ports?

22

I would try launching the container manually using the command from ExecStart and see if this works.

Will try.

[…] maybe there are some weird SELinux restrictions on you system on who can listen on ports?

Frankly, I am not sure if this is SELinux, but you are probably right that something is not ok, currently my guess is that its related to firewall configuration, particularly to the way I tried to set up this storj-zone and restrict access to dashboard from one selected ip. Will have to investigate. Just in any case you may have any suggestions Im providing some info about firewall below.

And thanks so much for all the info. Really appreciate. Particularly the confirmation (this is how I understand it) that the config I prepared based on the blog post seems to be correct.

Will also try to run it on another machine with clean OS installation soon.

Firewall config:

$ sudo firewall-cmd --get-active-zones
public
  interfaces: enp0s6
storj
  sources: AnotherServerIPtoAccessDashboard/32
trusted
  interfaces: lxdbr0

$ sudo firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: enp0s6
  sources: 
  services: cockpit dhcpv6-client ssh
  ports: 28968/tcp 28968/udp 28967/tcp 28967/udp 22/tcp
  protocols: 
  forward: yes
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:

$ sudo firewall-cmd --list-all-zones
block
  target: %%REJECT%%
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: 
  ports: 
  protocols: 
  forward: yes
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

dmz
  target: default
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: ssh
  ports: 
  protocols: 
  forward: yes
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

drop
  target: DROP
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: 
  ports: 
  protocols: 
  forward: yes
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

external
  target: default
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: ssh
  ports: 
  protocols: 
  forward: yes
  masquerade: yes
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

home
  target: default
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: cockpit dhcpv6-client mdns samba-client ssh
  ports: 
  protocols: 
  forward: yes
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

internal
  target: default
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: cockpit dhcpv6-client mdns samba-client ssh
  ports: 
  protocols: 
  forward: yes
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

nm-shared
  target: ACCEPT
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: dhcp dns ssh
  ports: 
  protocols: icmp ipv6-icmp
  forward: no
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 
        rule priority="32767" reject

public (active)
  target: default
  icmp-block-inversion: no
  interfaces: enp0s6
  sources: 
  services: cockpit dhcpv6-client ssh
  ports: 28968/tcp 28968/udp 28967/tcp 28967/udp 22/tcp
  protocols: 
  forward: yes
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

storj (active)
  target: default
  icmp-block-inversion: no
  interfaces: 
  sources: AnotherServerIPtoAccessDashboard/32
  services: 
  ports: 14003/tcp 14002/tcp
  protocols: 
  forward: no
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

trusted (active)
  target: ACCEPT
  icmp-block-inversion: no
  interfaces: lxdbr0
  sources: 
  services: 
  ports: 
  protocols: 
  forward: yes
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

unifi
  target: default
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: 
  ports: 8080/tcp 3478/tcp 8443/tcp
  protocols: 
  forward: no
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

work
  target: default
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: cockpit dhcpv6-client ssh
  ports: 
  protocols: 
  forward: yes
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:
23

Ah, just one more question @arrogantrabbit. Actially three questions.

Is this --label "io.containers.autoupdate=registry obligatory, or can it be committed and the container still be updated?

Is my understanding that using -e PUID=$(id -u) and -e PGID=$(id -g) instead of hard-coded values is ok?

Is :Z obligatory in case of --mount? Or can it be committed and applies only when using -v?

24

No problem! yes, the config looks good. The issue is clearly podman fails to start container because it can’t open the port you ask it to port container port into. Let’s check if anyone holds that port open on the host. e.g. sudo lsof -i or sudo netstat -an

Yes, this is what tells podman that the containers are updatable: https://www.redhat.com/sysadmin/podman-auto-updates-rollbacks

Yes, but note, that if you are running with SELinux, the current user ID is meaningless for the container. I.e. If your user ID is 500, and you run that container passing 500 or $(id -u), the app will be running under user 500 in the container, that is not the same user 500 on the host. On the host the user will be id=100500 likely.

According to Bind mounts | Docker Docs

  • The --mount flag does not support z or Z options for modifying selinux labels.

I use volume mounts exclusively, so I did not look into how to make it work with --mount.

25

It seems not SElinux. I set it to permissive mode <sudo setenforce 0> and its the same.

It seems not firewall, added 14002 permanently with sudo firewall-cmd --permanent --add-port=14002/tcp, its the same.

It seems nothing is holding 14002, checked as per your suggestions and also with nc -zv and sudo ss -ltn, however, in general I dont see this Error: rootlessport listen tcp somepodmanip:14002: bind: cannot assign requested address anymore.

It seems I got config.yaml created correctly. To be sure, lowered the storage size requirement as I am mostly running the nodes remotely but its the same.

With my early attempts when trying podman create --label "io.containers.autoupdate=registry" […] Im recalling some problems. Tried it again with the above --label, now seems to be ok.

When executing directly with ExecStart command I am receiving: Error: creating idfile: open %t/%n.ctr-id: no such file or directory.

However, I got:
ca.1*.cert ca.cert ca.key identity.1*.cert identity.cert identity.key
and it seems the identity is ok:

grep -c BEGIN ~/.local/share/storj/identity/$snname/ca.cert
2
grep -c BEGIN ~/.local/share/storj/identity/$snname/identity.cert
3

So, I do not know … will probably try tomorrow on a different machine and/or on my home computer.

26

Seems somepodmanip is not available for podman. Could you please try to remove it?

When you trying to execute a command from ExecStart you need either remove

or update it to a normal name, because these template variables are not working in the shell.

27

I did some additional checks on one of my home computers with Fedora 37. In general, following all the info provided here I managed to start it as a service, however, it was not fully successful as I was receiving “ERROR contact:service ping satellite failed” and the node was not running correctly.

What I noticed:

When:

podman run --rm -e SETUP="true" \
    -e PUID=1000 \
    -e PGID=1000 \
    --mount type=bind,source="$storjidlocation",destination=/app/identity \
    --mount type=bind,source="$datapath",destination=/app/config \
    --name $snname docker.io/storjlabs/storagenode:latest

Receiving: Error: storagenode configuration already exists (/app/config)
Solution: Reason seems to be SELinux, setting it to permissive mode <sudo setenforce 0> seems to create a new config.yaml. Otherwise the setup does not work. Also when trying to run the service in SELinux enforced mode, the node is continuously restarting itself. Of course as per suggestion from the blog I set: sudo loginctl enable-linger $USER and sudo setsebool -P container_manage_cgroup on respectively.

When:

podman create \                                                                                                                                           [14:46:15]
  --label "io.containers.autoupdate=registry" \
    -p $port:28967/tcp \
    -p $port:28967/udp \
    -p $dip:$dport:14002 \
    -e WALLET="$wallet" \
    -e EMAIL="$email" \
    -e ADDRESS="$pip:$port" \
    -e STORAGE="$ssize" \
    -e STORJ_HEALTHCHECK_DETAILS="true" \
    -e PUID=$(id -u) \
    -e PGID=$(id -g)\
    --mount type=bind,source="$storjidlocation",destination=/app/identity \
    --mount type=bind,source="$storjconfiglocation",destination=/app/config \
    --name $snname storjlabs/storagenode:latest --operator.wallet-features=zksync

Receiving: Error: short name: auto updates require fully-qualified image reference: "storjlabs/storagenode:latest"
Solution: Full name as suggested by @arrogantrabbit: docker.io/storjlabs/storagenode:latest

I also abandoned my attempt to use $(id -u) and $(id -g) in favor of hard coded values. Also I abandoned (for a while) my attempt to restrict access to Dashboard to a particular IP with a separate firewall zone. During those tests on Fedora 37 my firewall looked like this:

$ sudo firewall-cmd --zone=public --list-all                                                                                                                                
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: enp0s25
  sources: 
  services: cockpit dhcpv6-client iscsi-target mdns mountd nfs rpc-bind ssh
  ports: 28967/tcp 28967/udp 14002/tcp
  protocols: 
  forward: yes
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:

The other noticeable difference from the code described in the blog by @arrogantrabbit was the use of --mount instead of -v, which in general is my preference.

1 Like
28

To sum up, I would like to sustain the interest in a very direct piece of code that enables running storj node on Fedora / RHEL derived distros, with i) SELinux on, ii) full access to config.yaml and iii) ability to use --mounts and iv) ideally restricted IP access to dashboard. Will try to do some additional tests but this might not happen immediately. Should I be able to run it correctly I will post the code here. I will monitor this thread in case of any additional suggestions.

29

I got it running but in SELinux Permissive mode but as for now not on fedora. Are you sure @arrogantrabbit that it should run fully in Enforcing mode? Every time I set setenforce 1, Im receiving ERROR piecestore upload failed. EDIT1: Tried to create SELinux policy for storagenode container as described here, but still it does not work with Enforcing mode (seems like storj container is loosing communication). EDIT 2: Got it running in SELinux Enforcing mode with custom SELinux created by udica as indicated in EDIT 1. The trick was to use systemctl --user daemon-reload to reload units before starting the service. Don’t know how safe it is (and whats the difference between this and --Privileged as the label of storagenode container running as a service is also unconfined) however, in general its running and SELinux is … Enforcing. EDIT 3: So to sum up again, AFAIUI, the code provided by @arrogantrabbit in his blog and slightly adapted and presented above, together with udica seems to be solving the problem (at least from my current perspective) and --Privileged flag is not required to run storj with podman. Would be interested in any comments as this field is not my area of expertise, just a hobby.

30

I would exclude a file permissions problem first, to make sure that user-id and group-id are matched between the host and the container and also to the file permissions on desired paths.

31

this is weird. I don’t have an explanation. But I only had selinux in enforcing mode since install, so I don’t have much experience with that.

That means some permissions are not set properly; does it fail for the same reason of being unable to map a socket or something else?

I’m not sure how can this work: you need to label the namespace, and mount does not support Z. Have you tried with --volume, just to see if this solves all the issues?

I was doing that on oracle linux 8. Maybe things changed since then? Fedora is cutting edge, perhaps there is additional configuration required? I did not look into “what’s new” in selinux; I just figured out enough to get my services running.

It woudl be awesome if you publish your notes once you figure it out. The mount and its lack of namespace management bothers me. But this is in no way specific to storj, it is not a unique requirement and therefore there should be some simple solution. I’m in no way linux expert so maybe someone who is may chime in?

32

SELinux will remap the user IDs. i.e. user id 100 in container is not the same as user id 100 on the host. therefore all permissions need to either be set in the container or via podman unshare command; ultimately, the user owning files will be different than that on the host.

But this should not matter – all files storagenode creates are created from scratch, so whatever user ID node runs under that’s what they will get.

Unless someone edits files on the host, and messes up with permissions. Then those need to be restored with under podman unshare. Maybe that’s what happened with the config file in the message above?

1 Like
33

Im running the code that I provided above (I extended this code also with the use of udica). When I was making those tries described above of course I made some typos in my terminal. Frankly, I tried to avoid that, thus I asked if there is any opinionated code on this topic directly related to storj nodes.

Basically, it seems to be running, however, I noticed that in the night the node went offline. Its not perfect because it seems that I have to put SELinux into Permissive mode to get the node started. So it is a very preliminary approach.

As for the OSes, I do not think there is that much difference between RHEL 8 or 9 with regard to this thing, probably also not with regard to Fedora 37 (the latest now is 38).

What I do differently than you is that I am using --mount instead of -v. I have to read more on those topics. The major reason that I am using --mount is that usually I am storing data on remote computers, thus my current understanding is that this might be easier. This difference might be the reason for all those problems. I have to read more on those topics, however, please take into account, that I literally spent almost the last two evenings trying those things, so this additional reading might not take place immediately.

Based on my knowledge and understanding, the approach you are taking in your blog is really great, thus I liked it and wanted to try. On the other hand, my knowledge suggest that the other option is to handle SELinux with a custom policy. Probably best would be to get such a policy written and reviewed by storj team.

To sum up, I will read more about those topic, will try also -v option. Should there be any conclusions, or I have additional questions I write here. However, again, this might not happen immediately. A lot is going on apart to storj and I have limited amount of time but in general this is to my interest and would like to keep this topic active.