New node offline. Running ngrok on MacOS

carovonoliver · April 21, 2021, 11:01am

Node appears offline after trying to setup a new storage node on MacOS 11.2.3. Running ngrok since it seems like my ISP doesn’t support port forwarding. Exact error logs will be attached in the comments.

I have turned off the firewall for both my Mac and our router, which is a ZTE F660. I did try port forwarding on our router initially but I think that my ISP doesn’t allow it since upon checking the ports it always says closed even after trying different options (I can’t seem to find my WAN address on the router page so I can’t verify this). This is why I opted into using ngrok. I have since deleted the port forwarding rules on the router.

I used the following command when running the storage node:

docker run -d --restart unless-stopped --stop-timeout 300 \
    -p 18257:28967 \
    -p 127.0.0.1:14002:14002 \
    -e WALLET="REDACTED" \
    -e EMAIL="REDACTED" \
    -e ADDRESS="2.tcp.ngrok.io:18257" \
    -e STORAGE="900GB" \
    --mount type=bind,source="/Users/voncaro/Library/Application Support/Storj/Identity/storagenode",destination=/app/identity \
    --mount type=bind,source="/Volumes/Backup Plus/vonstorj",destination=/app/config \
    --name storagenode storjlabs/storagenode:latest

This is what the terminal displays when running ngrok:

I have confirmed that the ports are open when using ngrok

Identity also seems to be intact:

172:~ voncaro$ grep -c BEGIN ~/Library/Application\ Support/Storj/identity/storagenode/ca.cert

2

172:~ voncaro$ grep -c BEGIN ~/Library/Application\ Support/Storj/identity/storagenode/identity.cert

3

Other details:

NoIP isn’t currently running since from my understanding you don’t need it when running ngrok, but even when it is turned on I encounter the same problem
My storage directory is a USB-connected 1TB hard drive. Not sure if that has a negative effect on the set up
When trying to run the storage node it says in the dashboard that Last Contact was 17709610h 45m ago. Others have pointed out that this might be due to a wrong system time but my machine is using automatic time so it is correct
I have tried restarting my machine and the router as well
I have tried increasing the buffer size using the link outlined in the error logs but it was giving me an error (unkown oid). After a bit of searching I found that the command only works on Linux so I searched for its MacOS counterpart but I was still getting the error after running the command
I have not yet tried redoing the whole setup process from scratch. I figured I’d get help here first before attempting to do so

I think I’ve spent about 10 hours here in the forum looking for answers to no avail so hopefully someone can point me in the right direction. Thanks!

carovonoliver · April 21, 2021, 11:03am

Error logs:
    > 2021-04-21T10:31:12.031Z	INFO	Configuration loaded	{"Location": "/app/config/config.yaml"}
    > 2021-04-21T10:31:12.049Z	INFO	Operator email	{"Address": "REDACTED"}
    > 2021-04-21T10:31:12.049Z	INFO	Operator wallet	{"Address": "REDACTED"}
    > 2021-04-21T10:31:14.207Z	INFO	Telemetry enabled	{"instance ID": "12XvyBfGi74G3wJCenXVwC7Bmui3FZ4ZMeALXFpZ4a4xBr4rQVm"}
    > 2021-04-21T10:31:14.585Z	INFO	db.migration	Database Version	{"version": 51}
    > 2021-04-21T10:31:16.041Z	INFO	preflight:localtime	start checking local system clock with trusted satellites' system clock.
    > 2021-04-21T10:31:17.250Z	INFO	preflight:localtime	local system clock is in sync with trusted satellites' system clock.
    > 2021-04-21T10:31:17.251Z	INFO	bandwidth	Performing bandwidth usage rollups
    > 2021-04-21T10:31:17.251Z	INFO	Node 12XvyBfGi74G3wJCenXVwC7Bmui3FZ4ZMeALXFpZ4a4xBr4rQVm started
    > 2021-04-21T10:31:17.251Z	INFO	Public server started on [::]:7777
    > 2021-04-21T10:31:17.251Z	INFO	Private server started on 127.0.0.1:7778
    > 2021-04-21T10:31:17.251Z	INFO	failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
    > 2021-04-21T10:31:17.252Z	INFO	trust	Scheduling next refresh	{"after": "5h42m51.707942238s"}
    > 2021-04-21T10:31:18.181Z	ERROR	contact:service	ping satellite failed 	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 1, "error": "ping satellite error: failed to dial storage node (ID: 12XvyBfGi74G3wJCenXVwC7Bmui3FZ4ZMeALXFpZ4a4xBr4rQVm) at address 2.tcp.ngrok.io:18257: rpc: EOF", "errorVerbose": "ping satellite error: failed to dial storage node (ID: 12XvyBfGi74G3wJCenXVwC7Bmui3FZ4ZMeALXFpZ4a4xBr4rQVm) at address 2.tcp.ngrok.io:18257: rpc: EOF\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:141\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

peem · April 22, 2021, 6:19am

ngrok already redirects to this port:

tcp://2.tcp.ngrok.io:18257 -> localhost:28967

therefore no

-p 18257:28967

only

-p 28967:28967

in the docker startup command

carovonoliver · April 22, 2021, 6:35am

Tried your suggestion but it was still giving me the same error

peem · April 22, 2021, 6:38am

You run ngrok and what do you do with the terminal window?

carovonoliver · April 22, 2021, 6:39am

i leave ngrok running in the background and open a new terminal window and run the docker command from the new window

peem · April 22, 2021, 6:41am

see also if you have an established connection:
https://dashboard.ngrok.com/endpoints/status

and on which port

carovonoliver · April 22, 2021, 6:52am

this means that a connection was established right? note that the port number is different because i tried a new connection in the Asia Pacific server but error logs were the same. i changed the docker command when i created this new connection so we’re good on that end.

peem · April 22, 2021, 7:10am

maybe the us region doesn’t add “us” in the address
I have

tcp://0.tcp.eu.ngrok.io:18772

maybe the firewall in macOS is blocking the port …

carovonoliver · April 22, 2021, 7:13am

I already thought of the Mac firewall possibly blocking the connection so I tried disabling it entirely in System Preferences but the same error appears (unless MacOS has another firewall I’m not aware of). Did you do any additional configuration with the MacOS firewall for your setup?

peem · April 22, 2021, 7:16am

I don’t have MacOS, I have ngrok on linux

I have an interface for managing this linux on port 80 and I pre-checked how ngrok works by redirecting port 80

carovonoliver · April 22, 2021, 7:18am

Ah I see. Well I just checked and it looks like my node got disqualified so it’s good a time as any to re-do the process from scratch. Will update here what happens

peem · April 22, 2021, 7:25am

ngrok is difficult to manage (e.g. a different port each time it is started)
it’s easier to test portmap … but there are lots of Storj nodes in the subnet …
or use e.g. Synology and its QuickConnect
it’s best to get an external IP from your ISP

carovonoliver · April 22, 2021, 7:30am

i did try portmap.io first before ngrok but I can’t quite seem to understand how it works. might give it another shot later. as for contacting my ISP, ISP customer support is terrible where I live so I’d rather save that as a last-resort option. thanks for the feedback though!

peem · April 22, 2021, 7:34am

As others have already written here: tell your ISP that you must have access to a security camera in your home, and without an external IP you cannot access it …
or change your ISP to one that provides you with an external IP (doesn’t have to be a fixed IP, but has to be external) …

carovonoliver · April 22, 2021, 10:58am

Got it to work after a clean reinstall using portmap and SSH! Not sure if ngrok just wasn’t working for me, I did something wrong during my initial setup or a combination of both, but I’m just glad that it’s finally displaying a green “Online” text. For now I’ll mark peem’s suggestion of using portmap instead as the solution. Thanks everyone!