Contact:chore pingSatellites failed

Node Operators troubleshooting
1

2019-10-26T00:12:15.619+0200 ERROR contact:chore pingSatellites failed {“error”: “rpc error: code = Internal desc = contact: couldn’t connect to client at addr: xxx.xxx.com:28968 due to internal error.”}
2019-10-26T00:13:11.135+0200 ERROR nodestats:cache Get disk space usage query failed {“error”: “node stats service error: rpc error: code = PermissionDenied desc = node not found: 126LHX36So7mDYtxVqSHJMvT2tdLo31Uj3nAJRdaNefEev4ktBU; node stats service error: rpc error: code = PermissionDenied desc = node not found: 126LHX36So7mDYtxVqSHJMvT2tdLo31Uj3nAJRdaNefEev4ktBU; node stats service error: rpc error: code = PermissionDenied desc = node not found: 126LHX36So7mDYtxVqSHJMvT2tdLo31Uj3nAJRdaNefEev4ktBU; node stats service error: rpc error: code = PermissionDenied desc = node not found: 126LHX36So7mDYtxVqSHJMvT2tdLo31Uj3nAJRdaNefEev4ktBU”, “errorVerbose”: “group:\n— node stats service error: rpc error: code = PermissionDenied desc = node not found: 126LHX36So7mDYtxVqSHJMvT2tdLo31Uj3nAJRdaNefEev4ktBU\n\tstorj.io/storj/storagenode/nodestats.(*Service).GetDailyStorageUsage:112\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheSpaceUsage.func1:130\n\tstorj.io/storj/storagenode/nodestats.(*Cache).satelliteLoop:170\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheSpaceUsage:129\n\tstorj.io/storj/storagenode/nodestats.(*Cache).Run.func2:91\n\tstorj.io/storj/internal/sync2.(*Cycle).Run:87\n\tstorj.io/storj/internal/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n— node stats service error: rpc error: code = PermissionDenied desc = node not found: 126LHX36So7mDYtxVqSHJMvT2tdLo31Uj3nAJRdaNefEev4ktBU\n\tstorj.io/storj/storagenode/nodestats.(*Service).GetDailyStorageUsage:112\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheSpaceUsage.func1:130\n\tstorj.io/storj/storagenode/nodestats.(*Cache).satelliteLoop:170\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheSpaceUsage:129\n\tstorj.io/storj/storagenode/nodestats.(*Cache).Run.func2:91\n\tstorj.io/storj/internal/sync2.(*Cycle).Run:87\n\tstorj.io/storj/internal/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n— node stats service error: rpc error: code = PermissionDenied desc = node not found: 126LHX36So7mDYtxVqSHJMvT2tdLo31Uj3nAJRdaNefEev4ktBU\n\tstorj.io/storj/storagenode/nodestats.(*Service).GetDailyStorageUsage:112\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheSpaceUsage.func1:130\n\tstorj.io/storj/storagenode/nodestats.(*Cache).satelliteLoop:170\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheSpaceUsage:129\n\tstorj.io/storj/storagenode/nodestats.(*Cache).Run.func2:91\n\tstorj.io/storj/internal/sync2.(*Cycle).Run:87\n\tstorj.io/storj/internal/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n— node stats service error: rpc error: code = PermissionDenied desc = node not found: 126LHX36So7mDYtxVqSHJMvT2tdLo31Uj3nAJRdaNefEev4ktBU\n\tstorj.io/storj/storagenode/nodestats.(*Service).GetDailyStorageUsage:112\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheSpaceUsage.func1:130\n\tstorj.io/storj/storagenode/nodestats.(*Cache).satelliteLoop:170\n\tstorj.io/storj/storagenode/nodestats.(*Cache).CacheSpaceUsage:129\n\tstorj.io/storj/storagenode/nodestats.(*Cache).Run.func2:91\n\tstorj.io/storj/internal/sync2.(*Cycle).Run:87\n\tstorj.io/storj/internal/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

2

Please, check all bullets from this checklist:

3

I have checked NAT and ports continously for the past 3 hours.
My old node is on 28967, new on 28968

[01:19:31] Host is up (0.022s latency).
[01:19:31] PORT STATE SERVICE
[01:19:31] 28966/tcp filtered unknown
[01:19:31] 28967/tcp open unknown
[01:19:31] 28968/tcp open unknown
[01:19:31] 28969/tcp filtered unknown

Port 28968 is open on 79...*.

I have tried connecting to them both from the outside using telnet and they appear to behave the same.
Firewall shows states with traffic in both directions to the new node. Old node is linux, new node is
windows GUI.

4

In case of Windows, you should edit the "%ProgramFiles%\Storj\Storage Node\config.yaml" with a text editor such as Notepad++ (please, do not use a regular Notepad) and change the port in the server.address: option, save the configuration file and restart the node either from the Services applet or from the elevated powershell:

Restart-Service storagenode
5

already done a long time ago:

public address to listen on

server.address: :28968

6

Please, check your identity:

7

PS C:\Users\Non-ICE> (sls BEGIN $env:appData\Storj\Identity\storagenode\ca.cert).count
2
PS C:\Users\Non-ICE> (sls BEGIN $env:appData\Storj\Identity\storagenode\identity.cert).count
3

EDIT:
This is where I checked the generated files, not the files i was using in another location on the machine to run the node.

8

And add a firewall exception in the elevated powershell:

New-NetFirewallRule -DisplayName "Storj v3 28968" -Direction Inbound -Protocol TCP -LocalPort 28968 -Action allow
9

When you check your port on the https://www.yougetsignal.com/tools/open-ports/ please use your DDNS hostname as an Remote address and port as the Port.

Perhaps your DDNS ie not updated

10

windows firewall was already done via gui.

11

I have had the same IP since 2 months+ ago and all my other services are working as intended.
I have tried telnetting myself on port 28968 from a remote computer using the hostname and IP is correct and the port appears to be open.

12

Then make sure that your node is on v0.23.5 version and keep it online.
Also do not forget to add this port and all storagenode traffic to the exceptions list of your antivirus.

13

Let me get this straight:

Is this error a satellite complaining about connections to my node?

2019-10-26T02:05:17.769+0200 ERROR contact:chore pingSatellites failed {“error”: “rpc error: code = Internal desc = contact: couldn’t connect to client at addr: xxx.xxx.com:28968 due to internal error.”}

14

yes, your node tried to contact the satellite to update its info and satellite should be able to contact it too. This error mean that your node is still offline

15

the kademlia.external-address: should contain your external IP/address with a port via colon

16

the public address of the node, useful for nodes behind NAT

contact.external-address: xxx.xxx.com:28968

EDIT: all kademlia. parameters in config are depreciated

Port is open, i have tried running a custom telnet server on the same port and tested from the outside:

stolen pythoncode alert

socket_echo_server.py

import socket
import sys

Create a TCP/IP socket

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

Bind the socket to the port

server_address = (’’, 28968)
print(‘starting up on {} port {}’.format(*server_address))
sock.bind(server_address)

Listen for incoming connections

sock.listen(1)

while True:
# Wait for a connection
print(‘waiting for a connection’)
connection, client_address = sock.accept()
try:
print(‘connection from’, client_address)
connection.sendall(‘Port is open!\n’.encode(‘UTF-8’))
# Receive the data in small chunks and retransmit it
while True:
data = connection.recv(16)
print(‘received {!r}’.format(data))
connection.sendall(‘Closing port!\n’.encode(‘UTF-8’))
break
# if data:
# # print(‘sending data back to the client’)
# connection.sendall(data)
# else:
# print(‘no data from’, client_address)
# break
except:
pass

finally:
    # Clean up the connection
    connection.close()

response from telnet from outside:

Port is open!

Closing port!

Connection to host lost.

17

Solved

problem with the cert files. somehow the files i pointed the client to was different from the ones generated in profile/appdata

2019-10-26T15:46:46.763+0200 INFO version running on version v0.23.5
2019-10-26T15:46:47.044+0200 DEBUG contact:endpoint pinged {“by”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “srcAddr”: “35.195.93.217:33714”}
2019-10-26T15:46:47.097+0200 DEBUG contact:endpoint pinged {“by”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “srcAddr”: “78.94.240.189:45180”}
2019-10-26T15:46:47.622+0200 DEBUG contact:endpoint pinged {“by”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “srcAddr”: “35.192.33.38:38584”}
2019-10-26T15:46:49.070+0200 DEBUG contact:endpoint pinged {“by”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “srcAddr”: “35.201.222.48:35656”}

18

Now that my node has been down for 17 hours I assume its disqualified?

EDIT: Nevermind, suddenly LOTS OF PIECES!

19

The disqualification for downtime is currently disabled. Just keep your node online to recover its reputation.

20

Maybe have the devs have a look at proper error messages.