Iptables on a new pine64 system

[thebadcat]

Ok, so after struggling for a while I decided to do a rip and replace. So, I have updated my node to a newer Linux and Docker versions:

storj@pine64 : ~ $ uname -a

Linux pine64 5.4.43-sunxi64 #20.05.2 SMP Tue Jun 2 17:20:17 CEST 2020 aarch64 aarch64 aarch64 GNU/Linux

storj@pine64 : ~ $ docker --version

Docker version 19.03.12, build 48a6621

I’ve now removed the experimental flag from docker and everything has started up fine. However, I think because I’ve been down for so long I think I have been locked out. I am seeing the following error when starting my node:

2020-08-03T01:03:59.266216306Z 2020-08-03T01:03:59.265Z INFO Configuration loaded {“Location”: “/app/config/config.yaml”}

2020-08-03T01:03:59.274312022Z 2020-08-03T01:03:59.273Z INFO Operator email {“Address”: “galewis@yaddatech.com”}

2020-08-03T01:03:59.274641657Z 2020-08-03T01:03:59.274Z INFO Operator wallet {“Address”: “0x0000000000000000000000000000000000000”}

2020-08-03T01:04:10.431166292Z 2020-08-03T01:04:10.430Z INFO Telemetry enabled

2020-08-03T01:04:10.508318495Z 2020-08-03T01:04:10.507Z INFO db.migration Database Version {“version”: 42}

2020-08-03T01:04:11.454759799Z 2020-08-03T01:04:11.454Z INFO preflight:localtime start checking local system clock with trusted satellites’ system clock.

2020-08-03T01:04:12.346559907Z 2020-08-03T01:04:12.345Z INFO preflight:localtime local system clock is in sync with trusted satellites’ system clock.

2020-08-03T01:04:12.347000421Z 2020-08-03T01:04:12.346Z INFO bandwidth Performing bandwidth usage rollups

2020-08-03T01:04:12.348359297Z 2020-08-03T01:04:12.347Z INFO Node 12mYxtBsxSKpbZrh1bwZ9kMrrPx2W9SxDu62qrJWaFHcsCb4xkV started

2020-08-03T01:04:12.348475343Z 2020-08-03T01:04:12.348Z INFO Public server started on [::]:28967

2020-08-03T01:04:12.348502760Z 2020-08-03T01:04:12.348Z INFO Private server started on 127.0.0.1:7778

2020-08-03T01:04:12.349881054Z 2020-08-03T01:04:12.349Z INFO trust Scheduling next refresh {“after”: “6h29m52.79753552s”}

2020-08-03T01:05:10.021706761Z 2020-08-03T01:05:10.021Z INFO orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW sending {“count”: 1}

2020-08-03T01:05:10.021901851Z 2020-08-03T01:05:10.021Z INFO orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S sending {“count”: 169}

2020-08-03T01:05:10.021934227Z 2020-08-03T01:05:10.021Z INFO orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 sending {“count”: 38}

2020-08-03T01:05:10.022233820Z 2020-08-03T01:05:10.021Z INFO orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs sending {“count”: 79}

2020-08-03T01:05:10.025392170Z 2020-08-03T01:05:10.024Z INFO orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB sending {“count”: 4}

2020-08-03T01:05:10.028893406Z 2020-08-03T01:05:10.021Z INFO orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE sending {“count”: 14}

2020-08-03T01:05:10.420367105Z 2020-08-03T01:05:10.419Z INFO orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S finished

2020-08-03T01:05:10.513063886Z 2020-08-03T01:05:10.512Z INFO orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE finished

2020-08-03T01:05:10.568901911Z 2020-08-03T01:05:10.568Z INFO orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB finished

2020-08-03T01:05:10.734171041Z 2020-08-03T01:05:10.733Z INFO orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW finished

2020-08-03T01:05:10.750544937Z 2020-08-03T01:05:10.750Z INFO orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs finished

2020-08-03T01:05:10.931540234Z 2020-08-03T01:05:10.931Z INFO orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 finished

2020-08-03T01:05:20.944215747Z 2020-08-03T01:05:20.943Z ERROR orders archiving orders {“error”: “ordersdb error: database is locked”, “errorVerbose”: “ordersdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).archiveOne:238\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).Archive:202\n\tstorj.io/storj/storagenode/orders.(*Service).handleBatches.func2:238\n\tstorj.io/storj/storagenode/orders.(*Service).handleBatches:262\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders.func1:189\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

Is there anything I can do to fix this? I haven’t seen any email indicating that I had been locked out.

[mrkeyboardcommando]

wait…

database is locked means the system is busy… wait a couple of minutes to see if the error goes away.

Probably your disk is trashing now also.

[thebadcat]

@mrkeyboardcommando

It’s now been sitting for a while and I don’t see any activity from top or iotop that would indicate anything is happening. The dashboard also shows as being OFFLINE:

Storage Node Dashboard ( Node Version: v1.6.4 )

======================

ID 12mYxtBsxSKpbZrh1bwZ9kMrrPx2W9SxDu62qrJWaFHcsCb4xkV
Last Contact OFFLINE
Uptime 47m30s

               Available       Used       Egress     Ingress
 Bandwidth           N/A     2.2 GB     223.7 MB      2.0 GB (since Aug 1)
      Disk        2.6 TB     1.0 TB

Internal 127.0.0.1:7778
External storj.yaddatech.com:28968

[nerdatwork]

Your port is closed. Try this checklist

My Node is OFFLINE for 2 hours

It should connect in seconds. There are a few things you should check. Start with checking the port. On https://www.yougetsignal.com/tools/open-ports/?port=28967 you can fill in the domain or IP you are using in the run command. Then check the port. If it’s closed, it could still be several things. Make sure the WAN IP you see in your router settings matches the IP shown on the above page. Make sure your port forward is set up correctly. That means no filter on incoming IPs, forwarding to th…

[mrkeyboardcommando]

@thebadcat

storj.yaddatech.com:28968

[thebadcat]

I’ve disabled the firewall totally so my iptables output now looks like this:

storj@pine64:~/bin$ sudo ip6tables -L
[sudo] password for storj:
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
storj@pine64:~/bin$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

However, when I run nc I still get this:

[sysprog@nucky ~] $ nc -vz storj.yaddatech.com 28968

Ncat: Version 7.50 ( Ncat - Netcat for the 21st Century )

Ncat: Connection refused.

I am certain that my port forwarding setting are ok in my router because I can see this for another host/port in my network:

[sysprog@nucky ~] $ nc -vz storj.yaddatech.com 3022

Ncat: Version 7.50 ( Ncat - Netcat for the 21st Century )

Ncat: Connected to 99.244.218.26:3022.

Ncat: 0 bytes sent, 0 bytes received in 0.38 seconds.

I have no expertise with iptables so I am totally lost at the moment. Any help would be appreciated…

[deathlessdd]

Im a bit curious why your running the port 28968 is this your second node? Also what does your run command look like?

[thebadcat]

@deathlessdd

This is my run command:

docker run -d --restart unless-stopped --stop-timeout 300 -p 28968:28967 -p 14003:14002 -p 17777:7777 -e WALLET="0x00000000000000000000000000000000000000" -e EMAIL="galewis@yaddatech.com" -e ADDRESS="storj.yaddatech.com:28968" -e STORAGE="3.6TB" --mount type=bind,source="/home/storj/.local/share/storj/identity/storjnode",destination=/app/identity --mount type=bind,source="/storj/fs0",destination=/app/config --name storagenode storjlabs/storagenode:latest

The point is I can bring up the node, however, because the port forwarding is being blocked (at least that is what I am told earlier) the status is always OFFLINE. I think the only thing that could be blocking me is the firewall so I’ve cleared the iptables so nothing will be blocked. However, even doing that does not seem to show the port as being open.

BTW, I did have this setup on another machine that was already using 28967 which for unknown reasons started failing about 2 or 3 days ago. I decided to setup a new machine so I could not use the same port.

[deathlessdd]

@thebadcat

-p 17777:7777

what is this for? I dont remember there being a 3rd port. Did you switch from a windows node to linux?

[nerdatwork]

Try deleting your config.yaml file and restart the node.

[Alexey]

Please, show the

sudo iptables -L

(not ip6tables)

[thebadcat]

@Alexey

This is it with the node down:

storj@pine64-storj00:~$ sudo iptables -L
[sudo] password for storj: 
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy DROP)
target     prot opt source               destination         
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (1 references)
target     prot opt source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere

And this is with the node started:

storj@pine64-storj00:~$ sudo iptables -L
[sudo] password for storj: 
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy DROP)
target     prot opt source               destination         
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (1 references)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             172.17.0.3           tcp dpt:28967
ACCEPT     tcp  --  anywhere             172.17.0.3           tcp dpt:7777
ACCEPT     tcp  --  anywhere             172.17.0.3           tcp dpt:14002

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere

[Alexey]

Please, check your port forwarding rule on your router. Make sure that you forward the 28968 to this Pine64
Check its local IP for sure.

[thebadcat]

Also netstat shows this:

storj@pine64-storj00:~$ netstat -nlta
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN     
tcp        0      0 192.168.0.15:22         192.168.0.23:53187      ESTABLISHED
tcp        0    172 192.168.0.15:22         192.168.0.22:54253      ESTABLISHED
tcp        0      0 192.168.0.15:22         192.168.0.22:54299      ESTABLISHED
tcp        0      0 192.168.0.15:22         192.168.0.20:53843      ESTABLISHED
tcp6       0      0 :::28968                :::*                    LISTEN     
tcp6       0      0 :::17777                :::*                    LISTEN     
tcp6       0      0 :::14003                :::*                    LISTEN     
tcp6       0      0 :::22                   :::*                    LISTEN  

Note that there doesn’t seem to be IPv4 listeners for the 3 ports that I specified. However, from researching I am told this is normal.

[thebadcat]

@Alexey

I’ve got 2 pine64 running Linux. One is older and does not have the firewall. For that system I have a forwarder for SSH so that when I am away from home I can ssh into it to get access to my other systems. This forwarding is done in my router. I’ve done exactly the same thing for the 2nd pine64, however, that one does not seem to work. Again, I think it is related to the firewall but even when I remove all the firewall rules to what I am told is the default (open to everyone) the port still stays closed. I was going to attempt to remove the iptables firewall but that also would remove the dependent docker-ce package and I don’t know what other issues that would cause.

[BrightSilence]

@thebadcat

I decided to setup a new machine so I could not use the same port.

Did you remember to change the IP in the port forward on your router to this new machine?

[thebadcat]

@BrightSilence

Yep…

[thebadcat]

ok, found the problem after a boat load of google searches. Just wish I had more iptables knowledge. Anyway finally found someone who suggested doing this:

storj@pine64-storj00 : ~ $ sudo iptables -S
-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 172.17.0.3/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 28967 -j ACCEPT
-A DOCKER -d 172.17.0.3/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 7777 -j ACCEPT
-A DOCKER -d 172.17.0.3/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 14002 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN

The 2nd like of output has a FORWARD DROP so I changed that to a FORWARD ACCEPT:

sudo iptables -P FORWARD ACCEPT

And eurika my node is back up.