Tailscale funnel ping satellite failed

boelle · September 5, 2024, 3:55pm

i used this guide https://support.storj.io/hc/en-us/articles/360026612332-Install-storagenode-on-Raspberry-Pi3-or-higher-version

and i got the client running, and as i understand it i got tailscale funnel working too

i can use Open Port Check Tool - Test Port Forwarding on Your Router to check that the port is open

i opted to use port 10000

sudo docker run -d --restart always --stop-timeout 300
-p 10000:28967/tcp
-p 10000:28967/udp
-p 192.168.0.2:14002:14002 \

dashboard also works on the internal ip of the pi

where can i have F’ed up?

i also did an test where i setup a simple webserver on the same port and no problem getting that to show a simple hello world on port 10000

last entry in the log:

2024-09-05T15:24:51Z    ERROR   contact:service ping satellite failed   {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 10, "error": "ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: EOF", "errorVerbose": "ping satellite: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: EOF\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:208\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:156\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}

i checked the identtity: …rj.dev/node/get-started/identity#confirm-the-identity

came out correct

Alexey · September 6, 2024, 2:49am

Hello @boelle,
Welcome to the forum!

Please check your -e ADDRESS option, it must use either the public IP which matches the WAN IP and IP from yougetsignal or DDNS hostname and have a public port (in your case 10000), e.g. -e ADDRESS=my.public.address:10000

Please note, if you have a dynamic public IP, it’s better to use a DDNS hostname instead, otherwise you would be forced to update your docker run command with a new public IP every time, when the ISP would change it. This requires also to stop and remove the container and run it back with all your parameters include the changed one.

boelle · September 6, 2024, 6:35am

it does…
-e ADDRESS=“raspberrypi.tail9ee2.ts.net:10000” \

hence i use the hostname that tailscale provides

boelle · September 6, 2024, 6:39am


CONTAINER ID   IMAGE                          COMMAND                  CREATED        STATUS        PORTS                                                                                                                            NAMES
47451e87b617   storjlabs/storagenode:latest   "/entrypoint"            15 hours ago   Up 15 hours   192.168.0.2:14002->14002/tcp, 0.0.0.0:10000->28967/tcp, 0.0.0.0:10000->28967/udp, [::]:10000->28967/tcp, [::]:10000->28967/udp   storagenode
4f27619801bc   storjlabs/watchtower           "/watchtower storage…"   17 hours ago   Up 16 hours                                                                                                                                    watchtower

so at least it does not crash, i had issues with that yesterday

Alexey · September 6, 2024, 6:45am

Did you change the listening address (server.address:) in your docker run command or in the config.yaml file?
Is it possible that it is not 28967 any longer?

Please also check your DDNS hostname:

nslookup raspberrypi.tail9ee2.ts.net 8.8.8.8

It should return the same IP as a WAN IP and the IP from yougetsignal.

boelle · September 6, 2024, 7:03am

no, that as far as i could see not mentioned in the guide i followed… maybe it was not updated?

my docker run is this (i removed email and wallet…)

sudo docker run -d --restart always --stop-timeout 300 \
-p 10000:28967/tcp \
-p 10000:28967/udp \
-p 192.168.0.2:14002:14002 \
-e WALLET="****" \
-e EMAIL="****" \
-e ADDRESS="raspberrypi.tail9ee2.ts.net:10000" \
-e STORAGE="1TB" \
--memory=800m \
--log-opt max-size=50m \
--log-opt max-file=10 \
--mount type=bind,source=/home/pi/.local/share/storj/identity/storagenode,destination=/app/identity \
--mount type=bind,source=/media/storj/storagenode,destination=/app/config \
--name storagenode storjlabs/storagenode:latest

the guide did not tell how to change the config.yaml so that one i have not touched either

strange enough yougetsignal reports the IP of the CGNAT i’m behind and not the IP of the tailscale funnel

boelle · September 6, 2024, 7:10am

btw… my pi is headless hence some stuff is done in webbrowser and that might be why yousignal reports the IP of the CG-NAT?

boelle · September 6, 2024, 7:48am

about the tailscale funnel…

i will contact them and ask if both udp and tcp is forwarded

boelle · September 6, 2024, 7:58am

udp should be open

i think tcp also, thou here it says connected

Alexey · September 6, 2024, 8:14am

And this is correct.

That’s the culprit, if they do not match, it will never work. Please check your DDNS updater, it should update this hostname to your current external IP.

In that case you need to contact your ISP and ask for a public IP, it could be a dynamic, but must be a public (the WAN IP will match the IP on yougetsignal).

The DDNS will not solve the issue with a CGNAT, it can solve only the problem when your public IP is dynamic. With CGNAT you do not have a public IP either.

boelle · September 6, 2024, 8:22am

i do not have one, tailscale handles that… ie match the external ip to the hostname they give me

i did that a year ago or so… no luck, and that is why i started using tailscale so that a friend can backup to my openmediavault and i can back to him… ie we use tailscale to get past the cg-nat issue

boelle · September 6, 2024, 8:26am

side Q… would using the ipv6 adresses help anything if that is even an option?

reason i ask is that i have read or misread that it is always direct?

boelle · September 6, 2024, 8:29am

also i do this testing from my windows machine that do not use tailscale at all… so my windows outgoing connection is different that the raspberry pi…

that makes me think… what inerface/local ip does the storj client listen to? if it picks the local network one i can see why it might fail… it needs to listen on the interface that tailscale creates

4: tailscale0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc pfifo_fast state UNKNOWN group default qlen 500
    link/none
    inet 100.69.111.59/32 scope global tailscale0
       valid_lft forever preferred_lft forever
    inet6 fd7a:115c:a1e0::9301:6f3b/128 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::caa8:4da8:5213:3903/64 scope link stable-privacy
       valid_lft forever preferred_lft forever

EDIT: small clue

boelle · September 6, 2024, 8:59am

just for fun i tried to change where it listens but now the app is gone… ie the web interface

# server address of the api gateway and frontend app
console.address: 192.168.0.2:14002

# the public address of the node, useful for nodes behind NAT
contact.external-address: "raspberrypi.tail9ee2.ts.net"

# public address to listen on
server.address: 100.69.111.59:28967

maybe it was the private address i should change?

 private address to listen on
server.private-address: 127.0.0.1:7778

hmm i tried to change that in the config.yaml to

# private address to listen on
server.private-address: 100.69.111.59:10000

now the interface is dead again…

EDIT… decided to start all over with new identity… i still have a backup of identity but since i have not reached “production” yet i take there is no harm… and i will take carefull notes of everything i have done

Q… i issued ./identity create storagenode --identity-dir /media/storj/identity as i want the files to be created on the external drive of the pi rather than on the sd card

when i sign with my token… does the identity dir part come before or after email/token? i tried to google this but no luck and i dont want to F it up

Alexey · September 7, 2024, 3:52am

Only in a case of a dual stack, the public IPv4 is mandatory, IPv6 is optional.

the node will bind to all interfaces by default, however, if your tunnel is not established when the node is started, I guess it will not listen it.

You may try to bind the node only to tailscale0 interface using its IP in your docker run command with the option -p 100.69.111.59:10000:28967/tcp -p 100.69.111.59:10000:28967/udp -e ADDRESS=raspberrypi.tail9ee2.ts.net:10000
But if the IP would change, you will need to update it. Also the node wouldn’t start, if the tunnel is not open.

boelle:

# public address to listen on
server.address: 100.69.111.59:28967
maybe it was the private address i should change?
# private address to listen on
server.private-address: 127.0.0.1:7778
hmm i tried to change that in the config.yaml to
# private address to listen on
server.private-address: 100.69.111.59:10000

You shouldn’t change that, especially using the same ports for all these private addresses, they must be unique. In the case of docker it’s also makes no sense, because the container is running in an isolated network behind the docker NAT called the “bridge”, so it wouldn’t be possible to bind to the external address from the docker container, unless you would disable the bridge with the option --network host in your docker run command before the image name. In the latter case you may remove all -p options, because they wouldn’t have any effect and update only

server.address: 100.69.111.59:10000

option either in the config.yaml file or as a command line argument --server.address=100.69.111.59:10000 after the image name in your docker run command (the command line options have a precedence above the options in the config file).

You may use the --network host option before the image name in your docker run command, then all addresses would be exposed to the host, because the NAT will not be used in that case, but perhaps it’s not needed. So, please revert back your changes and do not touch config.yaml anymore. The mentioned change to -p 100.69.111.59:10000:28967/tcp -p 100.69.111.59:10000:28967/udp -e ADDRESS=raspberrypi.tail9ee2.ts.net:10000 should be enough.

boelle · September 7, 2024, 5:48am

they do never change once a machine is registered with tailscale

i forgot about docker, but is my thinking wrong if i wanted to disable the NAT? i mean since this is a purpose build pi that will never run anything else, why have an extra nat? to me that just makes it more complex… just a thought, i could be barking up the wrong tree

Alexey · September 7, 2024, 5:52am

You can. But with downsides - you would need to change more addresses and ports for that node and any next on the same device.
See How to add an additional drive? - Storj Docs

boelle · September 7, 2024, 6:01am

there will only be one node on the pi, once i get it running (if ever) the most change i will do is a larger disk

Alexey · September 7, 2024, 7:01am

Then you may try, just update the server.address option altogether with the --network host.

docker run -d ...\
--network host \
-e ADDRESS=raspberrypi.tail9ee2.ts.net:10000 \
...
storjlabs/storagenode:latest \
--server.address=100.69.111.59:10000

or with a docker compose and a docker-compose.yaml file:

services:
  storagenode2:
    container_name: storagenode2
    restart: always
    stop_grace_period: 300s
    image: storjlabs/storagenode:latest
    volumes:
      - type: bind
        source: /mnt/x/storagenode2/identity
        target: /app/identity
      - type: bind
        source: /mnt/x/storagenode2
        target: /app/config
    environment:
      - WALLET=0xYYYY
      - EMAIL=YYY@YYYY
      - ADDRESS=raspberrypi.tail9ee2.ts.net:10000
      - STORAGE=7TB
    command:
    #   - --pieces.enable-lazy-filewalker=false
    #   - --pieces.file-stat-cache=badger
      - --server.address=100.69.111.59:10000
    #   - --healthcheck.details=true
    #   - --operator.wallet-features=zksync-era
  watchtower:
    container_name: watchtower
    image: storjlabs/watchtower:latest
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    command: storagenode2 watchtower --stop-timeout 300s

boelle · September 7, 2024, 7:25am

so like:

sudo docker run -d --restart always --stop-timeout 300 \
--network host \
-p 100.69.111.59:10000:28967/tcp \
-p 100.69.111.59:10000:28967/udp \
-p 192.168.0.2:14002:14002 \
-e WALLET="****" \
-e EMAIL="****" \
-e ADDRESS="raspberrypi.tail9ee2.ts.net:10000" \
-e STORAGE="1TB" \
--memory=800m \
--log-opt max-size=50m \
--log-opt max-file=10 \
--mount type=bind,source=/home/pi/.local/share/storj/identity/storagenode,destination=/app/identity \
--mount type=bind,source=/media/storj/storagenode,destination=/app/config \
--name storagenode storjlabs/storagenode:latest
--server.address=100.69.111.59:10000

but remove:

-p 100.69.111.59:10000:28967/tcp \
-p 100.69.111.59:10000:28967/udp \
-p 192.168.0.2:14002:14002 \

?