VPN port forwarding for Pi nodes?

[snorkel]

You locked the car with the keys inside.

[MarkFeuer]

Aw man… Seriously? Is there any way to fix it? I guess I can rebuild the instance if I have to

[arrogantrabbit]

@MarkFeuer

I have a Wireguard server set up on a free instance in a PAYG account

You will blow through 10TB free egress in a couple of days and traffic is very expensive.

Instead use commercial vpn with unlimited traffic. I recommend AirVPN.

[arrogantrabbit]

Oracle provides you with remote access via cloud shell to the virtual server’s serial console. You would need to boot to single user mode and turn off your firewall. The same you would do with the physical machine.

Troubleshooting Instances Using Instance Console Connections

Here are correct ip tables rules for your usecase

Hosting services behind a restrictive firewall/CG-NAT using DNAT with...

[MarkFeuer]

Thank you, this script actually fixed my issues and allowed my Raspberry Pi to connect to the Oracle instance! I uninstalled the copy of Wireguard I had set up with the previous script and then ran this one. Generated a client and copied that .conf to the wg0.conf on my Pi, pinged and confirmed it’s working. I did have to install a missing depency, openresolv, in order for the Pi to actually set up wg0 properly.

[MarkFeuer]

Thank you, I utilized the Cloud Shell to get back into my instance and correct the connection issues, so it’s back up and working. Thanks to that script that @Krawi posted, I was able to get Wireguard working between it and the Pi.

[MarkFeuer]

So, I have connection between Oracle and the Pi Storj node, and my docker run command for storagenode appears to be configured properly. I set the external address as the Oracle instance’s public IP.

When I check the docker container with sudo docker ps, it is no longer restarting every 10 seconds and appears to be stable. However, I can’t access the node dashboard even though I had the run command set to -p 14002:14002.

When I try to run the dashboard in CLI with docker exec -it storagenode /app/dashboard.sh on the Pi node, I get this error: /app/dashboard.sh: 3: /app/dashboard.sh: ./storagenode: not found

[Alexey]

Perhaps something wrong with your docker run command.
You may try to stop and remove the container, remove the image and run the node back.

[MarkFeuer]

Stopping, deleting and then restarting the container with the docker run command unfortunately yields the same result. Here’s my current run command:

docker run -d --restart always --stop-timeout 300 \
-p 28967:28967/tcp \
-p 28967:28967/udp \
-p 14002:14002 \
-e WALLET="[REDACTED]" \
-e EMAIL="[REDACTED]" \
-e ADDRESS="10.66.66.1:28967" \ (This is the Wireguard IP of my Oracle instance, not the external IP)
-e STORAGE="18TB" \
--memory=800m \
--log-opt max-size=50m \
--log-opt max-file=10 \
--mount type=bind,source=/mnt/storj/storagenode/identity,destination=/app/identity \
--mount type=bind,source=/mnt/storj/storagenode,destination=/app/config \
--name storagenode storjlabs/storagenode:latest

I also previously got an error that memory limits weren’t supported, but now when I run the docker command it just gets stuck thinking and I have to exit SSH. When I come back I can start the docker container but I still can’t access the CLI dashboard or the web interface one.

[Alexey]

@MarkFeuer

I also previously got an error that memory limits weren’t supported

That’s usually mean that you didn’t enable it. Depending on the used OS and HW you need to use either a proper configuration option or a proper kernel. In some cases it’s just not enabled, you may check this article for Pi:

Install storagenode on Raspberry Pi3 or higher version

Specifically:

If you are using a Pi3 B/B+ please add cgroup_enable=memory cgroup_memory=1 swapaccount=1 to the end of the line. The resulting string should look like this:

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=PARTUUID=XXXXXX-XX rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait cgroup_enable=memory cgroup_memory=1 swapaccount=1

If you use another model, this could work too, but not tested. You may also google the solution of that problem for the used OS/device.

@MarkFeuer

but now when I run the docker command it just gets stuck thinking and I have to exit SSH

That’s usually mean that the shell is waiting for input (it should just create a container and that’s all). So, please review your command, perhaps you missed a quote or used a not supported symbol somewhere (curly quotes for example). In your posted excerpt I do not see a problem though, except comment

@MarkFeuer

(This is the Wireguard IP of my Oracle instance, not the external IP)

which shouldn’t be here. Or you may try to use a comment character #, not sure that it could work or not in this case.

@MarkFeuer

Stopping, deleting and then restarting the container

I mentioned that you also need to remove the image too.

docker stop -t 300 storagenode
docker rm storagenode
docker rmi storjlabs/storagenode:latest

After that try to run the node again, just make sure that you didn’t miss something, like a not closed quote.

[Krawi]

@MarkFeuer

-e ADDRESS="10.66.66.1:28967" \ (This is the Wireguard IP of my Oracle instance, not the external IP)

10.x.x.x/8 is the A-Class network reserved for private use only, it will NOT be routed over the internet and nobody could reach it.

Next question: Have you enabled routing in the kernel of your Oracle host?
One more question: Have you created proper rules for port forwarding in iptables on your Oracle host?

[MarkFeuer]

Okay, I ended up wiping my Raspberry Pi and completely started from scratch. I installed Wireguard and confirmed that was working and pinged my Oracle instance successfully.

Then I created a new Storj identity, got a new key, reinstalled Docker and went through the setup instructions for Storj on a Pi. I now have a working Storj container and can pull up both the CLI dashboard and the web interface dashboard on another computer. However, on the web dashboard I have an error that QUIC is misconfigured, and weirdly Wireguard on the client side stopped working, either when I installed Docker or when I finished setting up the Storj node. I just got Wireguard up and running again between the Pi and the Oracle instance, but QUIC is still coming up misconfigured.

@Alexey to answer your questions: Since I am using a Pi 4 with 8 gigabytes of RAM and that memory fix is for Pi 3 B/B+, I decided to skip the memory limit. Since I wiped out the old Pi OS, I think I am now working with a fresh Docker image of Storj. To clarify, those parentheses on the Docker run command were just in my post here, not in the actual run command.

@Krawi I have changed the ADDRESS value to my Oracle instance’s public IP and that worked in my latest setup run for Storj. However, now Wireguard then stopped working on the client side, and pinging it from the Oracle instance is failing. Just got Wireguard back up and running but QUIC is still showing misconfigured.

I think routing is enabled on the Oracle instance, because I have a separate Mysterium node running on it in a separate Wireguard configuration (myst0). I made wg0 for Storj. In /etc/sysctl.conf I have IPv4 packet forwarding enabled.

In my wg0.conf file, this was what I had under [Interface] that was working up until I got Storj working on the client side. This was the config that @JWvdV listed that was working for him.

(The immediate first PostUp and PostDown values that point to .sh files were generated by this Wireguard setup that you linked to, which did get it working initially)

[Interface]
PrivateKey = [REDACTED]
Address = 10.66.66.1/24, fd55:1a33:34e3::1/64
ListenPort = 51820
PostUp = /etc/wireguard/helper/add-nat-routing.sh
PostDown = /etc/wireguard/helper/remove-nat-routing.sh

# Input from outside, mind the ingress-rule from Oracle-cloud!
# In my case the Wireguard-port is 51820
PostUp = iptables -I INPUT -p udp --dport 51820 -j ACCEPT
PostDown = iptables -D INPUT -p udp --dport 51820 -j ACCEPT

# Forwarding bidirectional, %i is substituted for the name of the wireguard-adapter
PostUp = iptables -I FORWARD -o %i -j ACCEPT
PostUp = iptables -I FORWARD -i %i -j ACCEPT
PostUp = ip6tables -I FORWARD -i %i -j ACCEPT
PostDown = iptables -D FORWARD -o %i -j ACCEPT
PostDown = iptables -D FORWARD -i %i -j ACCEPT
PostDown = ip6tables -D FORWARD -i %i -j ACCEPT

# In my case the ethernet adapter is enp0s6, change according to your situation
PostUp = iptables -t nat -A POSTROUTING -o enp0s6 -j MASQUERADE
PostUp = ip6tables -t nat -A POSTROUTING -o enp0s6 -j MASQUERADE
PostDown = iptables -t nat -D POSTROUTING -o enp0s6 -j MASQUERADE
PostDown = ip6tables -t nat -D POSTROUTING -o enp0s6 -j MASQUERADE

# STORJ, in my case it's the client with IP 10.66.66.2.
PostUp = iptables -I INPUT -p udp --dport 28967 -j ACCEPT
PostUp = iptables -I INPUT -p tcp --dport 28967 -j ACCEPT
PostUp = iptables -t nat -A PREROUTING -p udp --dport 28967 -j DNAT --to-destination 10.66.66.2
PostUp = iptables -t nat -A PREROUTING -p tcp --dport 28967 -j DNAT --to-destination 10.66.66.2
PostDown = iptables -D INPUT -p udp --dport 28967 -j ACCEPT
PostDown = iptables -D INPUT -p tcp --dport 28967 -j ACCEPT
PostDown = iptables -t nat -D PREROUTING -p udp --dport 28967 -j DNAT --to-destination 10.66.66.2
PostDown = iptables -t nat -D PREROUTING -p tcp --dport 28967 -j DNAT --to-destination 10.66.66.2

[MarkFeuer]

Actually, I stand corrected! Once I got Wireguard up and running again (after it stopped working when Docker or Storj started), QUIC briefly showed up as “OK” but is now showing as Misconfigured again. However, the node status is still showing as Online.

[arrogantrabbit]

Setting listening address to wireguard ip may fix the QUIC connectivity.

This is a question about the "Misconfigured" error

Old thread on this issue: You may get rid of this error by specifying the interface IP to listen to explicitly in the config file, like so: This consistently helps me avoid the issue on FreeBSD. YMMV.

[MarkFeuer]

Is there a value for this that can be entered in the initial Docker run command? Like: -e SERVER.ADDRESS=“10.66.66.2:28967” \

I tried to edit config.yaml and restart the Docker container but that sent me back to my issues where I can’t get the Storj dashboard to load.

[arrogantrabbit]

@MarkFeuer

“10.66.66.2:28967”

These are wrong quotes. Please use text editor that does not try to autocorrect. Use ether ' or ", but not “ or ”. In fact, you don’t need quotes there at all.

[MarkFeuer]

Would it work like so?

-e SERVER.ADDRESS=10.66.66.2:28967 \

[arrogantrabbit]

I meant you don’t need quotes in the config file. Not sure about the command line. Just use the "

-e "server.address=10.66.66.2:28967"

[MarkFeuer]

Well, I haven’t been able to test that out yet because now I can’t set the Docker container back up. I have completely wiped my Pi three times at this point and tried again with a fresh install of Raspbian. I tried with my existing signed Storj credentials and also with new signed ones. Both ways, as soon as I run the initial Docker setup command for the storage node, it gets stuck on this message.

Connecting to version.storj.io (version.storj.io) | 34.173.164.90|:443... connected.

I’ve just ran it again and am going to let it sit overnight to see if that makes any difference.

[Krawi]

Could be the different MTU sizes of docker0 and wg0 interfaces.

Try to change the MTU of your docker0 interface on your Pi:

Edit:

/lib/systemd/system/docker.service

Add this paramater:

–mtu 1420

ExecStart=/usr/bin/dockerd --mtu 1420 -H fd:// --containerd=/run/containerd/containerd.sock

Restart docker.

But be warned, it seems to me this change is overwritten by docker updates, so keep an eye on this.