Nodes Offline!?

svet0slav · January 9, 2022, 2:19pm

Just set up 4 nodes, 2 of which cannot start.

docker run -d --restart unless-stopped --stop-timeout 300 -p 28967:28967/tcp -p 28967:28967/udp -p aa.bb.cc.dd:14002:14002 -e WALLET="XXXX" -e EMAIL="mail@mail.mail" -e ADDRESS="node1.domain.com:28967" -e STORAGE="14TB" --mount type=bind,source="/home/user/.local/share/storj/identity/node1",destination=/app/identity --mount type=bind,source="/nodes/STORJ/node1",destination=/app/config --name node1 storjlabs/storagenode:latest

docker run -d --restart unless-stopped --stop-timeout 300 -p 28968:28967/tcp -p 28968:28967/udp -p ee.ff.gg.hh:14002:14002 -e WALLET="XXXX" -e EMAIL="mail@mail.mail" -e ADDRESS="node2.domain.com:28968" -e STORAGE="14TB" --mount type=bind,source="/home/user/.local/share/storj/identity/node2",destination=/app/identity --mount type=bind,source="/nodes/STORJ/node2",destination=/app/config --name node2 storjlabs/storagenode:latest

docker run -d --restart unless-stopped --stop-timeout 300 -p 28969:28967/tcp -p 28969:28967/udp -p ii.jj.kk.ll:14002:14002 -e WALLET="XXXX" -e EMAIL="mail@mail.mail" -e ADDRESS="node3.domain.com:28969" -e STORAGE="14TB" --mount type=bind,source="/home/user/.local/share/storj/identity/node3",destination=/app/identity --mount type=bind,source="/nodes/STORJ/node3",destination=/app/config --name node3 storjlabs/storagenode:latest

docker run -d --restart unless-stopped --stop-timeout 300 -p 28970:28967/tcp -p 28970:28967/udp -p mm.nn.oo.pp:14002:14002 -e WALLET="XXXX" -e EMAIL="mail@mail.mail" -e ADDRESS="node4.domain.com:28970" -e STORAGE="14TB" --mount type=bind,source="/home/user/.local/share/storj/identity/node4",destination=/app/identity --mount type=bind,source="/nodes/STORJ/node4",destination=/app/config --name node4 storjlabs/storagenode:latest

1 and 4 are already online, but 2 and 3 are offline. DNS is properly set on each. Ports are open.
Logs show errors like this:

ERROR contact:service ping satellite failed {“Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “attempts”: 12, “error”: “ping satellite: failed to dial storage node (ID: XXXX) at address node2.domain.com:28968: rpc: context deadline exceeded”, “errorVerbose”: “ping satellite: failed to dial storage node (ID: XXXX) at address node2.domain.com:28968: rpc: context deadline exceeded\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

I can open only cli dashboard on offline nodes, but there it says they are offline. Any idea what is going on?

svet0slav · January 9, 2022, 2:45pm

Now node2 started for some reason, but node3 not starting for some reason…
Maybe DNS did not propagate? How to validate this from satelite point of view?

svet0slav · January 9, 2022, 3:15pm

Edited. It was a copy-paste thing…

svet0slav · January 9, 2022, 3:16pm

It is named uniquely.

svet0slav · January 9, 2022, 3:19pm

Tried with telnet and I can connect to the node IP and port!

telnet aa.bb.cc.dd 28969
Trying aa.bb.cc.dd…
Connected to aa.bb.cc.dd.
Escape character is ‘^]’.

netcat also works!

nc -vz subdomain.domain.com 28969
Connection to subdomain.domain.com 28969 port [tcp/*] succeeded!

Node still offline, though.

Huston, we have a problem!

svet0slav · January 9, 2022, 3:24pm

What do you mean???

svet0slav · January 9, 2022, 3:25pm

A-ha! At the same time it works on 3 other nodes on the same machine. This 14002 port is for the dashboard only.

svet0slav · January 9, 2022, 3:33pm

This makes me think it is DNS related. Like node3.domain.tld does not resolve for satellites, yet.

svet0slav · January 9, 2022, 5:13pm

Hmmm… Any help, please? Maybe it is some weirdo routing problem. Restarted server and node1, node2 and node4, can’t start, but node3 started. What the… !?

OK. Definitely a firewall thing. Managed to temporarily solve by flushing iptables and re-running the iptables rules. Rebooting server and it all fails again.

SGC · January 10, 2022, 8:41am

did you try and check if the ip and port number responds from the internet.
thats usually a good indicator if the network routing is configured correctly.

svet0slav · January 10, 2022, 8:42am

Yes. This works for sure, but the driver says there is no such chain for docker.
Last night I completely disabled firewall and all 4 nodes were working, so it is some firewall thing… Only to find all nodes offline just now when I checked on them. When I stop and delete the container and try to start it again I get such errors.

docker: Error response from daemon: driver failed programming external connectivity on endpoint storagenode (3ceabf1abd47cc2585296527a009f8b38dcf3c81a7293d30bb6b9936b3f4aabd): (iptables failed: iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.6 --dport 28967 -j ACCEPT: iptables: No chain/target/match by that name.

SGC · January 10, 2022, 8:49am

i’m so bad at using iptables, i mostly just copy paste mine lol they are so tricky to learn to read and write… or they atleast mostly seem like nonsense to me still lol

doesn’t the -A mean add and if it does then why would it show up in an error message, unless if its input wrong or something.

svet0slav · January 10, 2022, 8:53am

Sometimes I do feel the same. Tried to completely reset them with this

IPv4

iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
iptables -t nat -F
iptables -t mangle -F
iptables -F
iptables -X

IPV6

ip6tables -P INPUT ACCEPT
ip6tables -P FORWARD ACCEPT
ip6tables -P OUTPUT ACCEPT
ip6tables -t nat -F
ip6tables -t mangle -F
ip6tables -F
ip6tables -X

To no avail… Unless I stop, remove containers and run their commands again. And looks like this is only a temporary fix and thus I also leave my machine unprotected, which is not acceptable as a solution.

SGC · January 10, 2022, 8:56am

install UFW its much easier to use
if just for firewall purposes

svet0slav · January 10, 2022, 8:59am

I have no experience with it. I come from CentOS → Ubuntu and never played with UFW that much. so I am not sure how to translate to UFW, yet.
What I need it to do:

Reset all rules and chains
Force SYN packets check
Drop XMAS packets
Drop null packets
Drop incoming packets with fragments
Block all traffic
Keep established connections
Allow loopback
Allow known used ports for needed protocols (TCP/UDP) INPUT and OUTPUT

SGC · January 10, 2022, 9:02am

no idea about what many of those even are… but if you block all then you will not have a connection, you will always have to allow something.

ufw is pretty easy to use, you basically just turn it on i think… and then if you need something specific open you open it up.
with some simple commands

ofc for something uncomplicated just isn’t advanced enough

svet0slav · January 10, 2022, 9:05am

This is why I first block everything, while allowing established connections and ONLY what is needed is allowed. With iptables, first you block then allow. There is an order. Here is an example:

####IPv4###

#Reset all rules (F) and chains (X)
iptables -t filter -F
iptables -t filter -X

#Force SYN packets check
iptables -A INPUT -p tcp ! --syn -m state --state NEW -j DROP
#Drop XMAS packets
iptables -A INPUT -p tcp --tcp-flags ALL ALL -j DROP
#Drop null packets
iptables -A INPUT -p tcp --tcp-flags ALL NONE -j DROP
#Drop incoming packets with fragments
iptables -A INPUT -f -j DROP

#Block all traffic
iptables -t filter -P INPUT DROP
iptables -t filter -P FORWARD DROP
iptables -t filter -P OUTPUT DROP
#Keep established connections
iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
#Enable loopback
iptables -t filter -A INPUT -i lo -j ACCEPT
iptables -t filter -A OUTPUT -o lo -j ACCEPT
#SNMP
iptables -t filter -A OUTPUT -p udp --dport 161 -j ACCEPT
iptables -t filter -A INPUT -p udp --dport 161 -j ACCEPT
#Ubuntu
iptables -t filter -A INPUT -p tcp --dport 1022 -j ACCEPT
#HTTP
iptables -t filter -A OUTPUT -p tcp --dport 80 -j ACCEPT
iptables -t filter -A INPUT -p tcp --dport 80 -j ACCEPT
#SMTP
iptables -t filter -A OUTPUT -p tcp --dport 443 -j ACCEPT
iptables -t filter -A INPUT -p tcp --dport 443 -j ACCEPT
iptables -t filter -A INPUT -p tcp --dport 25 -j ACCEPT
iptables -t filter -A OUTPUT -p tcp --dport 25 -j ACCEPT
iptables -t filter -A INPUT -p tcp --dport 465 -j ACCEPT
iptables -t filter -A OUTPUT -p tcp --dport 465 -j ACCEPT
#IMAP
iptables -t filter -A INPUT -p tcp --dport 143 -j ACCEPT
iptables -t filter -A OUTPUT -p tcp --dport 143 -j ACCEPT
#ICMP
iptables -t filter -A INPUT -p icmp -j ACCEPT
iptables -t filter -A OUTPUT -p icmp -j ACCEPT
#SSH
iptables -t filter -A INPUT -p tcp --dport XXXX -j ACCEPT
iptables -t filter -A OUTPUT -p tcp --dport XXXX -j ACCEPT
#DNS
iptables -t filter -A OUTPUT -p tcp --dport 53 -j ACCEPT
iptables -t filter -A OUTPUT -p udp --dport 53 -j ACCEPT
iptables -t filter -A INPUT -p tcp --dport 53 -j ACCEPT
iptables -t filter -A INPUT -p udp --dport 53 -j ACCEPT
#GPG Keys
iptables -t filter -A OUTPUT -p tcp --dport 11371 -j ACCEPT
iptables -t filter -A OUTPUT -p udp --dport 11371 -j ACCEPT
iptables -t filter -A INPUT -p tcp --dport 11371 -j ACCEPT
iptables -t filter -A INPUT -p udp --dport 11371 -j ACCEPT
#WEBMIN
iptables -I INPUT -p tcp --dport xxxx -j ACCEPT
#NTP
iptables -t filter -A OUTPUT -p tcp --dport 123 -j ACCEPT
iptables -t filter -A OUTPUT -p udp --dport 123 -j ACCEPT
iptables -t filter -A INPUT -p tcp --dport 123 -j ACCEPT
iptables -t filter -A INPUT -p udp --dport 123 -j ACCEPT
#HTTPS
iptables -t filter -A OUTPUT -p tcp --dport 443 -j ACCEPT
iptables -t filter -A INPUT -p tcp --dport 443 -j ACCEPT
iptables -t filter -A OUTPUT -p udp --dport 443 -j ACCEPT
iptables -t filter -A INPUT -p udp --dport 443 -j ACCEPT
#STORJ Node1
iptables -t filter -A INPUT -p tcp --dport 28967 -j ACCEPT
iptables -t filter -A OUTPUT -p tcp --dport 28967 -j ACCEPT
iptables -t filter -A INPUT -p udp --dport 28967 -j ACCEPT
iptables -t filter -A OUTPUT -p udp --dport 28967 -j ACCEPT
#STORJ Node2
iptables -t filter -A INPUT -p tcp --dport 28968 -j ACCEPT
iptables -t filter -A OUTPUT -p tcp --dport 28968 -j ACCEPT
iptables -t filter -A INPUT -p udp --dport 28968 -j ACCEPT
iptables -t filter -A OUTPUT -p udp --dport 28968 -j ACCEPT
#STORJ Node3
iptables -t filter -A INPUT -p tcp --dport 28969 -j ACCEPT
iptables -t filter -A OUTPUT -p tcp --dport 28969 -j ACCEPT
iptables -t filter -A INPUT -p udp --dport 28969 -j ACCEPT
iptables -t filter -A OUTPUT -p udp --dport 28969 -j ACCEPT
#STORJ Node4
iptables -t filter -A INPUT -p tcp --dport 28970 -j ACCEPT
iptables -t filter -A OUTPUT -p tcp --dport 28970 -j ACCEPT
iptables -t filter -A INPUT -p udp --dport 28970 -j ACCEPT
iptables -t filter -A OUTPUT -p udp --dport 28970 -j ACCEPT
#STORJ Dashboard
iptables -t filter -A INPUT -p tcp --dport 14002 -j ACCEPT
iptables -t filter -A OUTPUT -p tcp --dport 14002 -j ACCEPT
iptables -t filter -A INPUT -p udp --dport 14002 -j ACCEPT
iptables -t filter -A OUTPUT -p udp --dport 14002 -j ACCEPT
#STORJ
iptables -t filter -A INPUT -p tcp --dport 8888 -j ACCEPT
iptables -t filter -A OUTPUT -p tcp --dport 8888 -j ACCEPT
iptables -t filter -A INPUT -p udp --dport 8888 -j ACCEPT
iptables -t filter -A OUTPUT -p udp --dport 8888 -j ACCEPT
iptables -t filter -A INPUT -p tcp --dport 7778 -j ACCEPT
iptables -t filter -A OUTPUT -p tcp --dport 7778 -j ACCEPT
iptables -t filter -A INPUT -p udp --dport 7778 -j ACCEPT
iptables -t filter -A OUTPUT -p udp --dport 7778 -j ACCEPT
#iperf
iptables -t filter -A INPUT -p tcp --dport 5001 -j ACCEPT
iptables -t filter -A OUTPUT -p tcp --dport 5001 -j ACCEPT
iptables -t filter -A INPUT -p udp --dport 5001 -j ACCEPT
iptables -t filter -A OUTPUT -p udp --dport 5001 -j ACCEPT
#speedtest-cli
iptables -t filter -A INPUT -p tcp --dport 8080 -j ACCEPT
iptables -t filter -A OUTPUT -p tcp --dport 8080 -j ACCEPT
iptables -t filter -A INPUT -p udp --dport 8080 -j ACCEPT
iptables -t filter -A OUTPUT -p udp --dport 8080 -j ACCEPT
#TFTP
iptables -t filter -A INPUT -p tcp --dport 69 -j ACCEPT
iptables -t filter -A OUTPUT -p tcp --dport 69 -j ACCEPT
iptables -t filter -A INPUT -p udp --dport 69 -j ACCEPT
iptables -t filter -A OUTPUT -p udp --dport 69 -j ACCEPT

####IPv6###

#Reset all rules (F) and chains (X)
ip6tables -t filter -F
ip6tables -t filter -X

#Force SYN packets check
ip6tables -A INPUT -p tcp ! --syn -m state --state NEW -j DROP
#Drop XMAS packets
ip6tables -A INPUT -p tcp --tcp-flags ALL ALL -j DROP
#Drop null packets
ip6tables -A INPUT -p tcp --tcp-flags ALL NONE -j DROP
#Drop incoming packets with fragments
ip6tables -A INPUT -f -j DROP

#Block all traffic
ip6tables -t filter -P INPUT DROP
ip6tables -t filter -P FORWARD DROP
ip6tables -t filter -P OUTPUT DROP
#Keep established connections
ip6tables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
ip6tables -A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
#Enable loopback
ip6tables -t filter -A INPUT -i lo -j ACCEPT
ip6tables -t filter -A OUTPUT -o lo -j ACCEPT
#SNMP
ip6tables -t filter -A OUTPUT -p udp --dport 161 -j ACCEPT
ip6tables -t filter -A INPUT -p udp --dport 161 -j ACCEPT
#Ubuntu
ip6tables -t filter -A INPUT -p tcp --dport 1022 -j ACCEPT
#HTTP
ip6tables -t filter -A OUTPUT -p tcp --dport 80 -j ACCEPT
ip6tables -t filter -A INPUT -p tcp --dport 80 -j ACCEPT
#SMTP
ip6tables -t filter -A OUTPUT -p tcp --dport 443 -j ACCEPT
ip6tables -t filter -A INPUT -p tcp --dport 443 -j ACCEPT
ip6tables -t filter -A INPUT -p tcp --dport 25 -j ACCEPT
ip6tables -t filter -A OUTPUT -p tcp --dport 25 -j ACCEPT
ip6tables -t filter -A INPUT -p tcp --dport 465 -j ACCEPT
ip6tables -t filter -A OUTPUT -p tcp --dport 465 -j ACCEPT
#IMAP
ip6tables -t filter -A INPUT -p tcp --dport 143 -j ACCEPT
ip6tables -t filter -A OUTPUT -p tcp --dport 143 -j ACCEPT
#ICMP
ip6tables -t filter -A INPUT -p icmp -j ACCEPT
ip6tables -t filter -A OUTPUT -p icmp -j ACCEPT
#SSH
ip6tables -t filter -A INPUT -p tcp --dport XXXX -j ACCEPT
ip6tables -t filter -A OUTPUT -p tcp --dport XXXX -j ACCEPT
#DNS
ip6tables -t filter -A OUTPUT -p tcp --dport 53 -j ACCEPT
ip6tables -t filter -A OUTPUT -p udp --dport 53 -j ACCEPT
ip6tables -t filter -A INPUT -p tcp --dport 53 -j ACCEPT
ip6tables -t filter -A INPUT -p udp --dport 53 -j ACCEPT
#GPG Keys
ip6tables -t filter -A OUTPUT -p tcp --dport 11371 -j ACCEPT
ip6tables -t filter -A OUTPUT -p udp --dport 11371 -j ACCEPT
ip6tables -t filter -A INPUT -p tcp --dport 11371 -j ACCEPT
ip6tables -t filter -A INPUT -p udp --dport 11371 -j ACCEPT
#WEBMIN
ip6tables -I INPUT -p tcp --dport xxxx -j ACCEPT
#NTP
ip6tables -t filter -A OUTPUT -p tcp --dport 123 -j ACCEPT
ip6tables -t filter -A OUTPUT -p udp --dport 123 -j ACCEPT
ip6tables -t filter -A INPUT -p tcp --dport 123 -j ACCEPT
ip6tables -t filter -A INPUT -p udp --dport 123 -j ACCEPT
#HTTPS
ip6tables -t filter -A OUTPUT -p tcp --dport 443 -j ACCEPT
ip6tables -t filter -A INPUT -p tcp --dport 443 -j ACCEPT
ip6tables -t filter -A OUTPUT -p udp --dport 443 -j ACCEPT
ip6tables -t filter -A INPUT -p udp --dport 443 -j ACCEPT
#STORJ Node1
ip6tables -t filter -A INPUT -p tcp --dport 28967 -j ACCEPT
ip6tables -t filter -A OUTPUT -p tcp --dport 28967 -j ACCEPT
ip6tables -t filter -A INPUT -p udp --dport 28967 -j ACCEPT
ip6tables -t filter -A OUTPUT -p udp --dport 28967 -j ACCEPT
#STORJ Node2
ip6tables -t filter -A INPUT -p tcp --dport 28968 -j ACCEPT
ip6tables -t filter -A OUTPUT -p tcp --dport 28968 -j ACCEPT
ip6tables -t filter -A INPUT -p udp --dport 28968 -j ACCEPT
ip6tables -t filter -A OUTPUT -p udp --dport 28968 -j ACCEPT
#STORJ Node3
ip6tables -t filter -A INPUT -p tcp --dport 28969 -j ACCEPT
ip6tables -t filter -A OUTPUT -p tcp --dport 28969 -j ACCEPT
ip6tables -t filter -A INPUT -p udp --dport 28969 -j ACCEPT
ip6tables -t filter -A OUTPUT -p udp --dport 28969 -j ACCEPT
#STORJ Node4
ip6tables -t filter -A INPUT -p tcp --dport 28970 -j ACCEPT
ip6tables -t filter -A OUTPUT -p tcp --dport 28970 -j ACCEPT
ip6tables -t filter -A INPUT -p udp --dport 28970 -j ACCEPT
ip6tables -t filter -A OUTPUT -p udp --dport 28970 -j ACCEPT
#STORJ Dashboard
ip6tables -t filter -A INPUT -p tcp --dport 14002 -j ACCEPT
ip6tables -t filter -A OUTPUT -p tcp --dport 14002 -j ACCEPT
ip6tables -t filter -A INPUT -p udp --dport 14002 -j ACCEPT
ip6tables -t filter -A OUTPUT -p udp --dport 14002 -j ACCEPT
#STORJ
ip6tables -t filter -A INPUT -p tcp --dport 8888 -j ACCEPT
ip6tables -t filter -A OUTPUT -p tcp --dport 8888 -j ACCEPT
ip6tables -t filter -A INPUT -p udp --dport 8888 -j ACCEPT
ip6tables -t filter -A OUTPUT -p udp --dport 8888 -j ACCEPT
ip6tables -t filter -A INPUT -p tcp --dport 7778 -j ACCEPT
ip6tables -t filter -A OUTPUT -p tcp --dport 7778 -j ACCEPT
ip6tables -t filter -A INPUT -p udp --dport 7778 -j ACCEPT
ip6tables -t filter -A OUTPUT -p udp --dport 7778 -j ACCEPT
#iperf
ip6tables -t filter -A INPUT -p tcp --dport 5001 -j ACCEPT
ip6tables -t filter -A OUTPUT -p tcp --dport 5001 -j ACCEPT
ip6tables -t filter -A INPUT -p udp --dport 5001 -j ACCEPT
ip6tables -t filter -A OUTPUT -p udp --dport 5001 -j ACCEPT
#speedtest-cli
ip6tables -t filter -A INPUT -p tcp --dport 8080 -j ACCEPT
ip6tables -t filter -A OUTPUT -p tcp --dport 8080 -j ACCEPT
ip6tables -t filter -A INPUT -p udp --dport 8080 -j ACCEPT
ip6tables -t filter -A OUTPUT -p udp --dport 8080 -j ACCEPT
#TFTP
ip6tables -t filter -A INPUT -p tcp --dport 69 -j ACCEPT
ip6tables -t filter -A OUTPUT -p tcp --dport 69 -j ACCEPT
ip6tables -t filter -A INPUT -p udp --dport 69 -j ACCEPT
ip6tables -t filter -A OUTPUT -p udp --dport 69 -j ACCEPT

Works perfectly fine on machines with 1 node, but somehow docker does not like this on machine with multiple nodes with several ports with 1 usable IP from different subnet each - no matter routing is also done properly in rc.local at boot.

ip rule add from IP1 table 1
ip route add default via GW1 table 1

ip rule add from IP2 table 2
ip route add default via GW2 table 2

ip rule add from IP3 table 3
ip route add default via GW3 table 3

ip rule add from IP4 table 4
ip route add default via GW4 table 4

For you - maybe, because you seem familiar with it. I will check… Thanks!

SGC · January 10, 2022, 9:11am

nah i’m even less proficient with UFW than i am with iptables, but i just remember how much easier it was when i was trying to use iptables as a firewall.
kinda makes me think of the advice about not to run one’s own mail server nor make ones own logging solution.

seems simple enough at first, but there are just so many things to take into account… with ufw you basically just tell it what ports you need open and it will sort out the rest… like with pfsense and such… default state is good security and still usable…
blocking everything means no access.
then one open ports and such.

svet0slav · January 10, 2022, 9:12am

Indeed, but back in the day it was very very hard to understand all this. There were only books on it, and you had to have the chance to even get them somewhere. Now there is info everywhere and you can ask on so many places and it is not much easier to fathom it all…

SGC · January 10, 2022, 9:16am

like getting advice online on ZFS, every damn idiot with ZFS is an expert… i’ve been digging into ZFS for 2 years now and just about gotten to the level that every time i find an online guide to help me with something it’s incorrect can causes side effects which isn’t obvious from normal light usage.

so there are just tons and tons of people recommending how to wreck one’s ZFS configuration and they have no idea lol