Wireguard + VPS: need help for QUIC

[smokemodel]

Hi!

I tried to setup a wireguard tunnel but it seems that something is wrong with QUIC confiuration.

Here is my Docker compose:
services:

  storj1:
    image: "storjlabs/storagenode:${IMAGE_TAG}"
    container_name: storj1
    restart: unless-stopped
    stop_grace_period: 300s
    user: "${UID}:${GID}"
    ports:
      - "28967:28967/tcp"
      - "28967:28967/udp"
      - "14001:14002"
    environment:
      WALLET: "${WALLET}"
      EMAIL: "${EMAIL}"
      ADDRESS: "<VPS PUBLIC IP>:28967"
      STORAGE: "3.5TiB"
    volumes:
      - /mnt/storj1/storj1_identity:/app/identity
      - /mnt/storj1/storj1_data:/app/config

Here is my wireguard configuration on Client:

[Interface]
PrivateKey = <WIREGUARD CLIENT PRIVATE KEY>
Address = 10.10.0.2/24
Table = off


[Peer]
PublicKey = <WIREGUARD SERVER PUBLIC KEY>
AllowedIPs = 10.10.0.0/0
Endpoint = <VPS PUBLIC IP>:51820
PersistentKeepalive = 25

Here is my wireguard configuration on VPS (Hetzner):

[Interface]
PrivateKey = <WIREGUARD SERVER PRIVATE KEY>
Address = 10.10.0.1/24
ListenPort = 51820

PostUp = iptables -A FORWARD -i wg0 -j ACCEPT
PostUp = iptables -A FORWARD -o wg0 -j ACCEPT
PostUp = iptables -t nat -A POSTROUTING -s 10.10.0.0/24 -o eth0 -j MASQUERADE

# Port Storj
PostUp = iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 28967 -j DNAT --to 10.10.0.2:28967
PostUp = iptables -t nat -A PREROUTING -i eth0 -p udp --dport 28967 -j DNAT --to 10.10.0.2:28967

PostDown = iptables -F
PostDown = iptables -t nat -F

[Peer]
PublicKey = <WIREGUARD CLIENT PUBLIC KEY>
AllowedIPs = 10.10.0.2/32

I suppose that I need to add some configuration to allow UDP connection on port 28967 but don’t know how to do that.

How can I fix this?

Thanks for your help!

[arrogantrabbit]

TLDR: your QUIC issue may not be related to routing. But it’s not worth debugging, ignore.

This approach works and preserves source IPs for log analissys later:

Hosting services behind a restrictive firewall/CGNAT using DNAT on a VPS...

This may help workaround QUIC binding:

Binding to a specific interface to stabilize QUIC connectivity

TLDR: if you are having flaky QUIC connectivity, go to the config file, and add an explicit interface address to the line server address line. e.g.: server.address: 10.0.17.120:28967 Backstory: I noticed that this morning, at 7:30PM PST according to the logs all three my nodes on three separate machines, in three separate states, connected in two different ways (one directly, and the other two over VPS) all lost ability to see QUICK. It was rather bizarre, and I don’t know why that happened. Th…

Ultimately, the guidance is to ignore:

QUIC misconfiguration in v1.86.1

There hasn’t been any progress on those QUIC bugs. It’s considered low priority because apparently uplinks quite rarely use QUIC in production. I suppose we’re recommending not to worry about what the dashboard says about your QUIC status for now.

[snorkel]

Increase the read and write buffers to 7500000. See QUIC docs and Storj node FAQ about QUIC.

[smokemodel]

Thanks, I will try this!

[smokemodel]

So I followed your tutorial and adapted the wireguard server config file to my setup (replacing 10.0.60.2 by 10.10.0.2 ; and ens3 by eth0).

It seemed to work for a few minutes but then, I started to see some ERRORS in the logs, like that:

2026-01-08T19:45:20Z ERROR contact:service ping satellite failed {“Process”: “storagenode”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “attempts”: 4, “error”: “ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 46.224.42.102:28967: connect: connection timed out”, “errorVerbose”: “ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 46.224.42.102:28967: connect: connection timed out\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:232\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:169\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:89\n\tstorj.io/common/sync2.(*Cycle).Run:102\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93”}

Also, I don’t know if it’s normal or not but with your wireguard configuration, when I try to ping from wireguard client to wireguard server (or the other way around), it doesn’t work.

[arrogantrabbit]

@smokemodel

wireguard client to wireguard server (or the other way around), it doesn’t work.

That’s the problem — the tunnel does not work. It does not make sense to triage further until tunnel works.

How come it works first and then stops? Or you mean logss log failure after some time? Then it’s normal - tunnel is broken so responses don’t go back.

Make sure the wireguard tunnel works. You shall be able to ping host and client from each other. See if you have stale iptables rules or firewall config.

[smokemodel]

With my initial configuration (in my original post), wireguard tunnel works.

My client wireguard config:

[Interface]
PrivateKey = <wireguard client private key>
Address = 10.10.0.2/24
Table = off


[Peer]
PublicKey = <Wireguard server public key>
AllowedIPs = 10.10.0.0/0
Endpoint = 46.224.42.102:51820
PersistentKeepalive = 25

My server wireguard config:

[Interface]
PrivateKey = <Wireguard server private key>
Address = 10.10.0.1/24
ListenPort = 51820

# Allow WireGuard's own traffic to reach the server.
PreUp = iptables -I INPUT -p udp --dport 51820 -j ACCEPT
PostDown = iptables -D INPUT -p udp --dport 51820 -j ACCEPT

# Allow incoming Storj connections on the public interface BEFORE they are forwarded.
PreUp = iptables -I INPUT -p tcp --dport 51820 -j ACCEPT
PostDown = iptables -D INPUT -p tcp --dport 51820 -j ACCEPT
PreUp = iptables -I INPUT -p udp --dport 51820 -j ACCEPT
PostDown = iptables -D INPUT -p udp --dport 51820 -j ACCEPT

# Port forward incoming Storj traffic to the VPN client.
PreUp = iptables -t nat -I PREROUTING -i eth0 -p tcp --dport 51820 -j DNAT --to-destination 10.10.0.2:28967
PostDown = iptables -t nat -D PREROUTING -i eth0 -p tcp --dport 51820 -j DNAT --to-destination 10.10.0.2:28967
PreUp = iptables -t nat -I PREROUTING -i eth0 -p udp --dport 51820 -j DNAT --to-destination 10.10.0.2:28967
PostDown = iptables -t nat -D PREROUTING -i eth0 -p udp --dport 51820 -j DNAT --to-destination 10.10.0.2:28967

# Allow the now-forwarded traffic to pass from the public interface to the VPN interface.
PreUp = iptables -I FORWARD -i eth0 -o %i -m state --state RELATED,ESTABLISHED -j ACCEPT
PreUp = iptables -I FORWARD -i eth0 -o %i -p tcp -d 10.10.0.2 --dport 28967 -j ACCEPT
PreUp = iptables -I FORWARD -i eth0 -o %i -p udp -d 10.10.0.2 --dport 28967 -j ACCEPT
PostDown = iptables -D FORWARD -i eth0 -o %i -m state --state RELATED,ESTABLISHED -j ACCEPT
PostDown = iptables -D FORWARD -i eth0 -o %i -p tcp -d 10.10.0.2 --dport 28967 -j ACCEPT
PostDown = iptables -D FORWARD -i eth0 -o %i -p udp -d 10.10.0.2 --dport 28967 -j ACCEPT

# Allow outbound traffic from the VPN client out to the internet
PreUp = iptables -I FORWARD -i %i -o eth0 -j ACCEPT
PostDown = iptables -D FORWARD -i %i -o eth0 -j ACCEPT

# Perform NAT for traffic from the VPN client going to the internet
PreUp = iptables -t nat -I POSTROUTING -s 10.10.0.2/32 -o eth0 -j MASQUERADE
PostDown = iptables -t nat -D POSTROUTING -s 10.10.0.2/32 -o eth0 -j MASQUERADE

[Peer]
PublicKey = <Wireguard client public key>
AllowedIPs = 10.10.0.2/32

Ping from client (10.10.0.2):

user@storj:~/app $ ping 10.10.0.1
PING 10.10.0.1 (10.10.0.1) 56(84) bytes of data.
^C
--- 10.10.0.1 ping statistics ---
7 packets transmitted, 0 received, 100% packet loss, time 6130ms

Ping from server (10.10.0.1):

root@storjvpn:~# ping 10.10.0.2
PING 10.10.0.2 (10.10.0.2) 56(84) bytes of data.
From 10.10.0.1 icmp_seq=1 Destination Host Unreachable
ping: sendmsg: Destination address required

[RecklessD]

This is your problem. (Wire guard client config). Something you missed in the tutorial.

@smokemodel

[Peer]
PublicKey = <Wireguard server public key>
AllowedIPs = 10.10.0.0/0

[smokemodel]

I just changed it for:

[Interface]
PrivateKey = <Wireguard client private key>
Address = 10.10.0.2/24
Table = off   # Important : wg-quick ne touche pas à la table de routage principale


[Peer]
PublicKey = <Wireguard server public key>
AllowedIPs = 10.10.0.1/32
Endpoint = 46.224.42.102:51820
PersistentKeepalive = 25

But it still doesn’t work.

Though, on the server config, the tunnel works when I remove the follwing:

PreUp = iptables -t nat -I PREROUTING -i eth0 -p tcp --dport 51820 -j DNAT --to-destination 10.10.0.2:28967
PostDown = iptables -t nat -D PREROUTING -i eth0 -p tcp --dport 51820 -j DNAT --to-destination 10.10.0.2:28967
PreUp = iptables -t nat -I PREROUTING -i eth0 -p udp --dport 51820 -j DNAT --to-destination 10.10.0.2:28967
PostDown = iptables -t nat -D PREROUTING -i eth0 -p udp --dport 51820 -j DNAT --to-destination 10.10.0.2:28967

I’m not really sure why we need to redirect incoming traffic on 51820 to Storj server (10.10.0.2)

[Mitsos]

Question: Why are you forwarding port 51820 (wireguard) to 28967 (storj)?
Answer: Because that’s the easiest way to break wireguard which uses UDP port 51820.

[arrogantrabbit]

@smokemodel

I’m not really sure why we need to redirect incoming traffic on 51820 to Storj server (10.10.0.2)

Holy crap. My bad. I command/replaced wrong value in my opus. I’ll fix it asap. Sorry about that! It was meant to be my non-standard storj port, not wireguard port, obviously…

(i.e. I took my config, and meant to replace 2618 with 28967, but instead replaced it with 51820. )

The fixed version shall be live, you may need to Option+Reload (Reload from Origin) to see the changes.

Thank you @Mitsos and @smokemodel for noticing the glaring enmorontment of my post.

[RecklessD]

Still wrong. You want all traffic initiating from node to enter internet from VPN Server?

AllowedIPs = 0.0.0.0/0

[arrogantrabbit]

@RecklessD

You want all traffic initiating from node to enter internet from VPN Server?

Yes. There are no downsides, and if you use inadyn or similar tool to update DDNS – you want all traffic from node to go via tunnel (this would include inadyn traffic).

Node is supposed to run in a separate jail/container, with a separate networking.

[smokemodel]

Thank you all!

So, I change the Wireguard server configuration. Now, it looks like this:

[Interface]
PrivateKey = <Wireguard server private key>
Address = 10.10.0.1/24
ListenPort = 51820

# Allow WireGuard's own traffic to reach the server.
PreUp = iptables -I INPUT -p udp --dport 51820 -j ACCEPT
PostDown = iptables -D INPUT -p udp --dport 51820 -j ACCEPT

# Allow incoming Storj connections on the public interface BEFORE they are forwarded.
PreUp = iptables -I INPUT -p tcp --dport 28967 -j ACCEPT
PostDown = iptables -D INPUT -p tcp --dport 28967 -j ACCEPT
PreUp = iptables -I INPUT -p udp --dport 28967 -j ACCEPT
PostDown = iptables -D INPUT -p udp --dport 28967 -j ACCEPT

# Port forward incoming Storj traffic to the VPN client.
PreUp = iptables -t nat -I PREROUTING -i eth0 -p tcp --dport 28967 -j DNAT --to-destination 10.10.0.2:28967
PostDown = iptables -t nat -D PREROUTING -i eth0 -p tcp --dport 28967 -j DNAT --to-destination 10.10.0.2:28967
PreUp = iptables -t nat -I PREROUTING -i eth0 -p udp --dport 28967 -j DNAT --to-destination 10.10.0.2:28967
PostDown = iptables -t nat -D PREROUTING -i eth0 -p udp --dport 28967 -j DNAT --to-destination 10.10.0.2:28967

# Allow the now-forwarded traffic to pass from the public interface to the VPN interface.
PreUp = iptables -I FORWARD -i eth0 -o %i -m state --state RELATED,ESTABLISHED -j ACCEPT
PreUp = iptables -I FORWARD -i eth0 -o %i -p tcp -d 10.10.0.2 --dport 28967 -j ACCEPT
PreUp = iptables -I FORWARD -i eth0 -o %i -p udp -d 10.10.0.2 --dport 28967 -j ACCEPT
PostDown = iptables -D FORWARD -i eth0 -o %i -m state --state RELATED,ESTABLISHED -j ACCEPT
PostDown = iptables -D FORWARD -i eth0 -o %i -p tcp -d 10.10.0.2 --dport 28967 -j ACCEPT
PostDown = iptables -D FORWARD -i eth0 -o %i -p udp -d 10.10.0.2 --dport 28967 -j ACCEPT

# Allow outbound traffic from the VPN client out to the internet
PreUp = iptables -I FORWARD -i %i -o eth0 -j ACCEPT
PostDown = iptables -D FORWARD -i %i -o eth0 -j ACCEPT

# Perform NAT for traffic from the VPN client going to the internet
PreUp = iptables -t nat -I POSTROUTING -s 10.10.0.2/32 -o eth0 -j MASQUERADE
PostDown = iptables -t nat -D POSTROUTING -s 10.10.0.2/32 -o eth0 -j MASQUERADE

[Peer]
# Machine chez toi
PublicKey = <Wireguard client public key>
AllowedIPs = 10.10.0.2/32

And the Wireguard client config:

[Interface]
PrivateKey = <Wireguard client private key>
Address = 10.10.0.2/24
Table = off   # Important : wg-quick ne touche pas à la table de routage principale


[Peer]
PublicKey = <Wireguard server public key>
AllowedIPs = 0.0.0.0/0
Endpoint = 46.224.42.102:51820
PersistentKeepalive = 25

Results:
It worked for a few minutes, while the QUIC statut on the Dashboard GUI was “Refreshing”. Then, it turned to “Misconfigured” and the ERRORS started to show up:

2026-01-09T01:23:07Z ERROR contact:service ping satellite failed {“Process”: “storagenode”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “attempts”: 2, “error”: “ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 46.224.42.102:28967: connect: connection timed out”, “errorVerbose”: “ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 46.224.42.102:28967: connect: connection timed out\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:232\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:169\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:89\n\tstorj.io/common/sync2.(*Cycle).Run:102\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93”}

[RecklessD]

At the point when the errors starts:

• can you still ping 10.10.0.1 from node?

[smokemodel]

Yes.

From client (10.10.0.2) to server (10.10.0.1):

user@storj1:~/app $ ping 10.10.0.1
PING 10.10.0.1 (10.10.0.1) 56(84) bytes of data.
64 bytes from 10.10.0.1: icmp_seq=1 ttl=64 time=97.8 ms
64 bytes from 10.10.0.1: icmp_seq=2 ttl=64 time=98.2 ms
^C
--- 10.10.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 97.847/98.047/98.248/0.200 ms

It seems that something is wrong with the QUIC configuration

[arrogantrabbit]

It still does not work if you see ping satellite failed — it has nothing to do with QUIC. It’s node connectivity not working. It’s tcp.

I assume you are stopping node immediately when it fails to work? Otherwise satellites will be throttling attempts.

Check few things.

1. Can you ping Google.com from your node host?
2. curl ipinfo.io from node host returns wireguard server external IP?
3. Curl node ip:port from wireguard client/node machine
4. Curl node ip:port from your wireguard server machine (through the tunnel)
5. Curl using external wireguard server ip:port from an unrelated network. (Avoid hairpin)
6. TCP forwarding enabled?
7. Does port appear open? I recommend testing with simple http server, not node, to avoid satellites throttling connection attempts. Only start node once networking is fully working.

Depending on where does the process break next step would be tracking packets with tcpdump at each endpoint.

Also, are you reporting the client and server to purge state iptables?

What is the client and server OS? Is firewall-cmd available? What zones are your interfaces in? Did you read the article in its entirety?

[smokemodel]

(Thank you so much for your help!)

I assume you are stopping node immediately when it fails to work? Otherwise satellites will be throttling attempts.

For example, here is my last try:

|2026-01-09T02:00:27Z|INFO|piecestore|downloaded|{Process: storagenode, Piece ID: 5MDKXHCHKN4UGUZARUIWH5CGDRVY74UYF3E3G43LNFA2ZKVXEWKA, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: GET, Offset: 0, Size: 256, Remote Address: 156.146.43.227:34292}|
|---|---|---|---|---|
|2026-01-09T02:00:27Z|INFO|piecestore|downloaded|{Process: storagenode, Piece ID: QAOMPPOUSH3GMFHZ2FOUA5LHQWBDJZVT3EE22LTJ3DE7CBVLP43A, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: GET, Offset: 0, Size: 1280, Remote Address: 79.127.205.226:52774}|
|2026-01-09T02:00:35Z|DEBUG|piecestore|download started|{Process: storagenode, Piece ID: BVDNOODWSU5E5SH52JUKBJ3ZPCW3KC76UEWOVGFFPENI6JOXGMUA, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: GET, Offset: 0, Size: 5120, Remote Address: 156.146.43.227:42274}|
|2026-01-09T02:00:35Z|INFO|piecestore|downloaded|{Process: storagenode, Piece ID: BVDNOODWSU5E5SH52JUKBJ3ZPCW3KC76UEWOVGFFPENI6JOXGMUA, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: GET, Offset: 0, Size: 5120, Remote Address: 156.146.43.227:42274}|
|2026-01-09T02:00:46Z|INFO|piecestore|downloaded|{Process: storagenode, Piece ID: QAOMPPOUSH3GMFHZ2FOUA5LHQWBDJZVT3EE22LTJ3DE7CBVLP43A, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: GET, Offset: 0, Size: 1280, Remote Address: 109.61.92.87:38380}|
|2026-01-09T02:00:59Z|DEBUG|piecestore:monitor|readability check done|{Process: storagenode, Duration: 272.72µs}|
|2026-01-09T02:01:27Z|ERROR|contact:service|ping satellite failed |{Process: storagenode, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, attempts: 2, error: ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 46.224.42.102:28967: connect: connection timed out, errorVerbose: ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 46.224.42.102:28967: connect: connection timed out\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:232\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:169\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:89\n\tstorj.io/common/sync2.(*Cycle).Run:102\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93}|
|2026-01-09T02:01:31Z|ERROR|contact:service|ping satellite failed |{Process: storagenode, Satellite ID: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE, attempts: 2, error: ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 46.224.42.102:28967: connect: connection timed out, errorVerbose: ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 46.224.42.102:28967: connect: connection timed out\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:232\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:169\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:89\n\tstorj.io/common/sync2.(*Cycle).Run:102\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93}|
|2026-01-09T02:01:31Z|ERROR|contact:service|ping satellite failed |{Process: storagenode, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, attempts: 2, error: ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 46.224.42.102:28967: connect: connection timed out, errorVerbose: ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 46.224.42.102:28967: connect: connection timed out\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:232\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:169\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:89\n\tstorj.io/common/sync2.(*Cycle).Run:102\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93}|
|2026-01-09T02:01:32Z|ERROR|contact:service|ping satellite failed |{Process: storagenode, Satellite ID: 121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6, attempts: 2, error: ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 46.224.42.102:28967: connect: connection timed out, errorVerbose: ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 46.224.42.102:28967: connect: connection timed out\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:232\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:169\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:89\n\tstorj.io/common/sync2.(*Cycle).Run:102\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93}|
|2026-01-09T02:01:59Z|DEBUG|piecestore:monitor|readability check done|{Process: storagenode, Duration: 327.126µs}|
|2026-01-09T02:01:59Z|DEBUG|piecestore:monitor|writability check done|{Process: storagenode, Duration: 409.7µs}|
|2026-01-09T02:02:16Z|DEBUG|orders|cleaning|{Process: storagenode}|
|2026-01-09T02:02:16Z|DEBUG|orders|cleanup finished|{Process: storagenode}|
|2026-01-09T02:02:59Z|DEBUG|piecestore:monitor|readability check done|{Process: storagenode, Duration: 179.258µs}|
|2026-01-09T02:03:42Z|ERROR|contact:service|ping satellite failed |{Process: storagenode, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, attempts: 3, error: ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 46.224.42.102:28967: connect: connection timed out, errorVerbose: ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 46.224.42.102:28967: connect: connection timed out\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:232\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:169\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:89\n\tstorj.io/common/sync2.(*Cycle).Run:102\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93}|
|2026-01-09T02:03:48Z|ERROR|contact:service|ping satellite failed |{Process: storagenode, Satellite ID: 121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6, attempts: 3, error: ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 46.224.42.102:28967: connect: connection timed out, errorVerbose: ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 46.224.42.102:28967: connect: connection timed out\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:232\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:169\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:89\n\tstorj.io/common/sync2.(*Cycle).Run:102\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93}|
|2026-01-09T02:03:50Z|ERROR|contact:service|ping satellite failed |{Process: storagenode, Satellite ID: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE, attempts: 3, error: ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 46.224.42.102:28967: connect: connection timed out, errorVerbose: ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 46.224.42.102:28967: connect: connection timed out\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:232\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:169\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:89\n\tstorj.io/common/sync2.(*Cycle).Run:102\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93}|
|2026-01-09T02:03:51Z|ERROR|contact:service|ping satellite failed |{Process: storagenode, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, attempts: 3, error: ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 46.224.42.102:28967: connect: connection timed out, errorVerbose: ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 46.224.42.102:28967: connect: connection timed out\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:232\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:169\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:89\n\tstorj.io/common/sync2.(*Cycle).Run:102\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93}|
|2026-01-09T02:03:59Z|DEBUG|piecestore:monitor|readability check done|{Process: storagenode, Duration: 207.165µs}|
|2026-01-09T02:04:59Z|DEBUG|piecestore:monitor|readability check done|{Process: storagenode, Duration: 298.867µs}|
|2026-01-09T02:05:59Z|DEBUG|piecestore:monitor|readability check done|{Process: storagenode, Duration: 272.924µs}|
|2026-01-09T02:06:00Z|ERROR|contact:service|ping satellite failed |{Process: storagenode, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, attempts: 4, error: ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 46.224.42.102:28967: connect: connection timed out, errorVerbose: ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 46.224.42.102:28967: connect: connection timed out\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:232\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:169\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:89\n\tstorj.io/common/sync2.(*Cycle).Run:102\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93}|
|2026-01-09T02:06:07Z|ERROR|contact:service|ping satellite failed |{Process: storagenode, Satellite ID: 121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6, attempts: 4, error: ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 46.224.42.102:28967: connect: connection timed out, errorVerbose: ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 46.224.42.102:28967: connect: connection timed out\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:232\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:169\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:89\n\tstorj.io/common/sync2.(*Cycle).Run:102\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93}|
|2026-01-09T02:06:09Z|ERROR|contact:service|ping satellite failed |{Process: storagenode, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, attempts: 4, error: ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 46.224.42.102:28967: connect: connection timed out, errorVerbose: ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 46.224.42.102:28967: connect: connection timed out\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:232\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:169\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:89\n\tstorj.io/common/sync2.(*Cycle).Run:102\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93}|
|2026-01-09T02:06:09Z|ERROR|contact:service|ping satellite failed |{Process: storagenode, Satellite ID: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE, attempts: 4, error: ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 46.224.42.102:28967: connect: connection timed out, errorVerbose: ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: dial tcp 46.224.42.102:28967: connect: connection timed out\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:232\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:169\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:89\n\tstorj.io/common/sync2.(*Cycle).Run:102\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93}|

We see it is working well, until it doesn’t anymore. I only included the last few success (the first 6 lines), but then, it keeps failing.

@arrogantrabbit

• Can you ping Google.com from your node host?

It works from the OS host (I’m running the node in a Docker container hosted on a Raspberry (bookworm).).

@arrogantrabbit

• curl ipinfo.io from node host returns wireguard server external IP?

From the node host (Raspberry) and from the container, it returns my ISP IP. I suppose this is not normal.

@arrogantrabbit

• Curl node ip:port from wireguard client/node machine

I run curl 172.18.0.4:28967 (IP of the storj docker container) from the Raspberry (where the container runs) and it returns:

{
  "Statuses": null,
  "Help": "To access Storagenode services, please use DRPC protocol!",
  "AllHealthy": false
}

@arrogantrabbit

• Curl node ip:port from your wireguard server machine (through the tunnel)

curl 10.10.0.2:28967 from the Wireguard server:
{
“Statuses”: null,
“Help”: “To access Storagenode services, please use DRPC protocol!”,
“AllHealthy”: false
}

@arrogantrabbit

• Curl using external wireguard server ip:port from an unrelated network. (Avoid hairpin)

Sorry, could you elaborate? I don’t understand what I need to run.

@arrogantrabbit

• TCP forwarding enabled?

Yes, on the Wireguard server with the following command.
The last line of /etc/sysctl.conf is net.ipv4.ip_forward=1. Then, I run sudo sysctl --system.

@arrogantrabbit

1. Does port appear open? I recommend testing with simple http server, not node, to avoid satellites throttling connection attempts. Only start node once networking is fully working.

On Hetzner, the VM is running without firewall, so everything is open by default.
On my local machine (Raspberry), I confirm the port 28967 is open and redirects incoming traffic to the Raspberry.

@arrogantrabbit

Also, are you reporting the client and server to purge state iptables?

How should I do that? I run all of these commands on both Wireguard server and client to make sure everything is clean:

sudo iptables -F

sudo iptables -t nat -F

sudo iptables -t mangle -F

sudo iptables -X

sudo iptables -P INPUT ACCEPT
sudo iptables -P FORWARD ACCEPT
sudo iptables -P OUTPUT ACCEPT

And then, stop & start Wireguard.

@arrogantrabbit

What is the client and server OS? Is firewall-cmd available? What zones are your interfaces in? Did you read the article in its entirety?

Client OS:

PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Wireguard server:

PRETTY_NAME="Ubuntu 24.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.3 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

On both machines, I’m using iptables, not firewall-cmd (sudo systemctl status firewalld => “Unit firewalld.service could not be found.”)

@arrogantrabbit

What zones are your interfaces in?

How can I know that?

@arrogantrabbit

Did you read the article in its entirety?

Yes.

[RecklessD]

@smokemodel

@arrogantrabbit

• curl ipinfo.io from node host returns wireguard server external IP?

From the node host (Raspberry) and from the container, it returns my ISP IP. I suppose this is not normal

This should return VPS external IP and details, not your local router (ISP) external IP details.
Did you run this after node stopped working?

If you restart wireguard on node and run curl command, do you get different result?

[smokemodel]

@RecklessD

Did you run this after node stopped working?

Before and after it stopped working. What I did exactly (just did it again):

1. Install curl and ping on the container, with a specific Dockerfile:

FROM storjlabs/storagenode:latest

RUN apt update && \
    apt install -y curl iputils-ping && \
    apt clean

2. Wait for the first download activity in the logs, like this:

2026-01-09T15:06:40Z	DEBUG	piecestore	download started	{"Process": "storagenode", "Piece ID": "NHGB7HBQTSKHJABV7XTGMOZL6WRHPEFIQ3YAD2JJOHWR2CZOF7AQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 6656, "Remote Address": "109.61.92.75:51432"}
2026-01-09T15:06:40Z	INFO	piecestore	downloaded	{"Process": "storagenode", "Piece ID": "NHGB7HBQTSKHJABV7XTGMOZL6WRHPEFIQ3YAD2JJOHWR2CZOF7AQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 6656, "Remote Address": "109.61.92.75:51432"}
2026-01-09T15:06:41Z	DEBUG	piecestore	upload started	{"Process": "storagenode", "Piece ID": "2NLLB6N5T2YRAMETOR2AXY5ICSOXT4CSD5EHVWONOYOWQSMKMJ5Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.205.239:48292", "Available Space": 1405808899404}
2026-01-09T15:06:42Z	INFO	piecestore	uploaded	{"Process": "storagenode", "Piece ID": "2NLLB6N5T2YRAMETOR2AXY5ICSOXT4CSD5EHVWONOYOWQSMKMJ5Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.205.239:48292", "Size": 1792}

3. Run curl ipinfo.io from the docker container. It keeps returning my ISP IP. Though, when I try to ping the Wireguard server (10.10.0.1) from the containe, it works.

Maybe something is wrong with the routes on the Storj machine (machine = the Raspberry that hosts the Storj container)? How can I check that?

@RecklessD

If you restart wireguard on node and run curl command, do you get different result?

Every time I try a new configuration, this is what I do:

1. stop Wireguard server: systemctl stop wg-quick@wg0
2. stop Wireguard client: systemctl stop wg-quick@wg0
3. stop the Storj container: sudo docker compose down
4. start Wireguard server: systemctl start wg-quick@wg0
5. start Wireguard client: systemctl start wg-quick@wg0
6. make sure the docker compose file uses the wireguard server public IP with the following line in the environment section: ADDRESS: "46.224.42.102:28967"
7. start Storj container: sudo docker compose up -d --build

---

---

EDIT:
Some news.

I commented the following line in the Wireguard client config:

Table = off

(It is not present in the tutorial from @arrogantrabbit but I added it during my troubleshooting).
Now, the curl ipinfo.io returns the Wireguard server IP!

BUT, the Storj container still doesn’t work as expected and returns the following errors:

2026-01-09T15:24:05Z	DEBUG	trust	Satellite is trusted	{"Process": "storagenode", "id": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2026-01-09T15:24:05Z	INFO	preflight:localtime	start checking local system clock with trusted satellites' system clock.	{"Process": "storagenode"}
2026-01-09T15:26:19Z	ERROR	preflight:localtime	unable to get satellite system time	{"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "error": "rpc: tcp connector failed: rpc: dial tcp 35.212.10.183:7777: connect: connection timed out", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp 35.212.10.183:7777: connect: connection timed out\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2026-01-09T15:26:19Z	ERROR	preflight:localtime	unable to get satellite system time	{"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "error": "rpc: tcp connector failed: rpc: dial tcp 35.207.121.91:7777: connect: connection timed out", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp 35.207.121.91:7777: connect: connection timed out\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2026-01-09T15:26:19Z	ERROR	preflight:localtime	unable to get satellite system time	{"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "error": "rpc: tcp connector failed: rpc: dial tcp 34.2.157.232:7777: connect: connection timed out", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp 34.2.157.232:7777: connect: connection timed out\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2026-01-09T15:26:19Z	ERROR	preflight:localtime	unable to get satellite system time	{"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "error": "rpc: tcp connector failed: rpc: dial tcp 35.215.108.32:7777: connect: connection timed out", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp 35.215.108.32:7777: connect: connection timed out\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2026-01-09T15:26:19Z	ERROR	Failed preflight check.	{"Process": "storagenode", "error": "system clock is out of sync: system clock is out of sync with all trusted satellites", "errorVerbose": "system clock is out of sync: system clock is out of sync with all trusted satellites\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check:96\n\tstorj.io/storj/storagenode.(*Peer).Run:1100\n\tmain.cmdRun:127\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tmain.main:34\n\truntime.main:283"}
2026-01-09T15:26:20Z	ERROR	failure during run	{"Process": "storagenode", "error": "system clock is out of sync: system clock is out of sync with all trusted satellites", "errorVerbose": "system clock is out of sync: system clock is out of sync with all trusted satellites\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check:96\n\tstorj.io/storj/storagenode.(*Peer).Run:1100\n\tmain.cmdRun:127\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tmain.main:34\n\truntime.main:283"}
2026-01-09T15:26:20Z	FATAL	Unrecoverable error	{"Process": "storagenode", "error": "system clock is out of sync: system clock is out of sync with all trusted satellites", "errorVerbose": "system clock is out of sync: system clock is out of sync with all trusted satellites\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check:96\n\tstorj.io/storj/storagenode.(*Peer).Run:1100\n\tmain.cmdRun:127\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tmain.main:34\n\truntime.main:283"}