Unable to contact satellites through open port

I’ve set up everything according to official docs and I cannot connect to satellites as I can see from the logs.
The interesting thing is that when I check port 32400 (plex) here, it is open. I used same options by forwarding TCP/UDP port in my ISP router → Mi Router → Raspberry ufw (has storj, plex, pivpn, pihole, portainer etc.). For storj after I had no luck with port 28967, I tried 38967.

Here is the docker-compose I used:

---
version: '3.3'
services:
  Node1:
    image: storjlabs/storagenode:latest
    container_name: Node1
    volumes:
      - type: bind
        source: /home/artur/.local/share/storj/identity/storagenode
        target: /app/identity
      - type: bind
        source: /mnt/exdisk/storj
        target: /app/config
    ports:
      - "38967:38967/tcp"
      - "38967:38967/udp"
      - "14002:14002"
    restart: unless-stopped
    stop_grace_period: 300s
    environment:
      - WALLET=0xyyyyyyy
      - EMAIL=aartasas@yoyoyoyo
      - ADDRESS=some.ddns.net:38967
      - STORAGE=500GB

And my last logs:
https://gist.github.com/neone35/4fe071c410780629a3d3167f1be3bd1d

If you used a different external port (like 38967) that’s fine. But unless you also changed the config file for your node it will still be running on 28967 inside its container. Try this:

“38967:28967/tcp”
“38967:28967/udp”

It’s very common to run with different internal and external ports, because many people prefer to leave config options inside the container at their defaults. Docker takes care of all the networking changes for you.

1 Like

Same log errors after changing docker-compose.yml to 38967:28967 and also specifying address in config.yaml:
contact.external-address: some.ddns.net:38967

yougetsignal.com still shows that 38967 is not open. So this is the cause I bet? But I’ve opened it the same way as plex port. Maybe I need to restart raspberry or what?

Hello @neone35,
Welcome to the forum!

Please also make sure that the IP from Open Port Check Tool - Test Port Forwarding on Your Router is matched your WAN IP on the router, and

nslookup some.ddns.net 8.8.8.8

is matched them too.

yes.

if you changed the port in the docker-compose.yaml file, you also need to re-create a container:

docker compose up -d

and do not forget to forward TCP+UDP port 38967 to TCP+UDP 28967 and your raspberry’s local IP as a destination.
You also need to allow incoming connections to 28967 TCP+UDP ports in your firewall on raspberry and 38967 TCP+UDP ports on your router’s firewall.

Yes, 78.xx.xx.xxx matches WAN and some.ddns.net:

nslookup some.ddns.net 8.8.8.8
Non-authoritative answer:
Name:    some.ddns.net
Address:  78.xx.xx.xxx

ISP router admin panel (192.168.3.154 is static Mi router):
DHCP on
WAN IP is 78.xx.xx.xxx and netmask is 255.255.192.0

Mi router (192.168.3.154). Raspberry is static at 192.168.31.50:

Name  Protocol     External ports  Internal IP address  Internal port
storj TCP&UDP      38967	       192.168.31.50	    38967

Raspberry ufw rules:

artur@artur-pi:~$ sudo ufw status | grep "8967"
28967                      ALLOW       Anywhere
28967/udp                  ALLOW       Anywhere
38967                      ALLOW       Anywhere
28967 (v6)                 ALLOW       Anywhere (v6)
28967/udp (v6)             ALLOW       Anywhere (v6)
38967 (v6)                 ALLOW       Anywhere (v6)

Still no access to satellites after restarting container. Port is also shown closed on yougetsignal.com

Any suggestions what could be wrong, anyone?

Your logs show “tcp connector failed: rpc: dial tcp 78.xx.xx.xxx:28967: connect: connection refused”… which means Storj is still starting up thinking it should tell satellites it lives on external port 28967 (instead of 38967). Are you sure your docker-compose.yml has ADDRESS = some.ddns.net:38967 in it? I’d avoid editing the Storj config/config.yaml file directly if you can just pass params (like that ADDRESS variable) in docker-compose.yml instead… for what you’re setting up you shouldn’t need to touch config/config.yaml at all.

But you’re right that until yougetsignal says the port is open, things won’t work properly.

1 Like

I restored config.yaml to default. My newest logs now don’t contain 28967, only 38967. I also tried opening only TCP, not TCP/UDP for 38967 - no luck. How can I debug it further, why port is not open yet on my ISP router?

At admin panel I see this:

You showed port-forwarding on your mi-router: but does it also run a firewall that needs port 38967 opened?

Do you have any other web service on your mi-router (maybe a web server, FTP, ssh service: really anything) that you can temporarily move to 38967 to see if the connection is at least making it that far (i.e. making it past your ISP router)?

Did you restart every device already? Everything up2date?
From running nmap against your network there seems to be nothing filtered on port 38967, which is good. But the port is also not open, which is bad.
This doesn’t necessarily mean, that your ISP router is not forwarding correctly. Just that there is no answer on this port, which could be the fault of either ISP router, MI router or raspberry.

For further troubleshooting you need to take one of the devices out of the equation. Either look if there is any service you can run on the MI router and put it on port 38967 as suggested by @Roxor.
Or
If you have another device you could install some ftp server on it and let it use port 38967. Hook this device directly to ISP router and forward the port to this device. If it works try the same again but put the MI router inbetween. If this also works you know it’s something with the raspberry. Then you can install ftp server on raspberry with port 38967(directly, no docker) and see if this works.
Basically confirm step by step starting as close to your WAN as possible and take out as many variables as possible to narrow down the problem.

So, you have a double NAT. Is it possible to switch the ISP’s router to the bridge mode? In this case you would have to use NAT only on your MI router.
If it’s not possible, can you switch your MI router to the access point mode? In this case you would need to update your port forwarding rules on ISP’s router to route to new IPs in your network.

I would support the @Roxor’s question - do you maybe have a firewall on your MI router and need to add a rule for 38967 port as well?
By the way you didn’t allow 38967/udp in the firewall on your Raspberry Pi, is it intended?
Did you change the server.address: option in the config.yaml? (hint - you shouldn’t, unless you use a network: host in your compose file).

I cannot put ISP router into another mode (bridge, AP) because it’s configured to work with ISP’s IPTV service. I couldn’t use another DNS servers, that’s why I moved my systems up to another router (Mi) where I have more freedom.

Mi Router 4A Gigabit edition has proprietary OS, not an linux one I think, which configuration I cannot directly modify. And it has no firewall rules as ISP router has, which I have set to lowest mode:

I’ve contacted ISP provider for any remote assitance because after restarting all routers and raspberry there’s no effect on yugetsignal result.

So port must have running service which sends answer, otherwise yougetsignal won’t show me any result like for plex whose server runs on raspberry?

So, some progress.

I already had SFTP/SSH running and open at raspberry ufw and after forwarding ports on both mi router and ISP router, I got it shown as open at yougetsignal.

So the problem is in docker container itself it turns out?

Is this port configuration correct?

Found one more thing:

artur@artur-pi:~/storj$ curl -I http://localhost:38967
curl: (56) Recv failure: Connection reset by peer

Also, when I bypassed bridge network mode by removing ports from docker-compose.yml and specifying:
network_mode: "host"
it started uploading data as I see from the logs, but as ports were not assigned, I cannot access admin panel.

The docker ps output looks correct to me. Did “curl -I http://localhost:14002” work?

I don’t think even a working node will respond to “curl -I http://localhost:38967” - it probably just blocks with no output even when the node is there. But if it’s not there I’d expect it a failed-to-connect message (and not connection-reset-by-peer message like you saw).

Is this the first docker container you’ve ever configured on your rpi (like has any docker with port forwarding worked on that system before)?

It sounds like all your port-forwarding and firewall stuff is working: and docker is seeing you try an incoming connection on 38967… but there’s no service inside the container listening on 28967… so your curl command is kinda bouncing off of it.

When you have things set up normally (using port-forwarding, not “network_mode: host”)… and storjNode1 is running… can you run a “netstat -an | grep 28967” on your rpi? You should see a LISTEN line like this:

tcp 0 0 0.0.0.0:28967 0.0.0.0:* LISTEN

…and if you look at your storjNode1 docker logs you should see a startup line like this:

 ... Public server started on [::]:28967 ...

That would make sure the node is on 28967 (and didn’t move to 38967 during troubleshooting). I guess you could also do a “netstat -an | grep 38967” and you should get no results because nothing should be on 38967.

1 Like

curl -I http://localhost:14002
works

netstat -an | grep 28967
no output

netstat -an | grep 38967
image

I guess I need to somehow allow docker to access localhost network. I never configured ports to be accessible in docker from outside home network. I tried some, but they don’t work:

Anywhere                   ALLOW OUT   172.17.0.0/16 on docker0
Anywhere                   ALLOW OUT   172.21.0.0/16 on docker0

storjNode1 runs on 172.21.0.2 inside docker

If 14002 works, then your node is running and listening on its UI port: hooray!

But for some reason the nodes main port is on 38967… which I’m almost positive means you still have an altered config/config.yaml file for storjNode1. Please check it again for a “server.address:” override line (and remove it)

It feels like you’re almost there!

Edit: you don’t need to do anything special with the 172.x.y.z networks: they’re little pocket-universe networks that docker punches holes into for you (like opening 14002 outside to 14002 inside etc)

I’ve commented out these lines in config.yaml:

# server.address: :38967
# contact.external-address: ""

same errors after restarting container…

So “netstat -an | grep 28967” doesn’t now show anything, and “netstat -an | grep 38967” still shows the node is on 38967? That should be… impossible. Without an override it should revert to 28967.

Now it looks like this, had changes
image