Multinode addition doesn't work

After trying to add my node to the multinode, I gave up and decided to post here:

This is what I did:

root@STORJ9:/storj/DBs# sqlite3 secret.db
SQLite version 3.40.1 2022-12-28 14:03:47
Enter ".help" for usage hints.
sqlite> SELECT * FROM secret
   ...> ;
sqlite>

root@STORJ9:/storj/DBs# docker exec -it storagenode /app/storagenode issue-apikey --config-dir config --identity-dir identity --storage2.database-dir="dbs"
2024-07-06T20:37:42Z    INFO    Configuration loaded    {"Process": "storagenode", "Location": "/app/config/config.yaml"}
2024-07-06T20:37:42Z    INFO    Anonymized tracing enabled      {"Process": "storagenode"}
2024-07-06T20:37:42Z    INFO    Identity loaded.        {"Process": "storagenode", "Node ID": "1ghkS2nPpDdLPXRajjQaUqzH99b4NuGQoMg1WXp1FoQDTGvetS"}
KajkpWJJzPbVursW0_fgh-pHim03_8Fj2bZg_HiyBHQ=
root@STORJ9:/storj/DBs# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 4a:52:4f:54:53:09 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.1.10/24 brd 192.168.1.255 scope global dynamic eth0
       valid_lft 70145sec preferred_lft 70145sec
    inet6 fe80::4852:4fff:fe54:5309/64 scope link
       valid_lft forever preferred_lft forever
3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:f1:ca:d2:de brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:f1ff:feca:d2de/64 scope link
       valid_lft forever preferred_lft forever
8: vetha7a4e22@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default
    link/ether a6:c4:1c:88:6c:32 brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::a4c4:1cff:fe88:6c32/64 scope link
       valid_lft forever preferred_lft forever
root@STORJ9:/storj/DBs# sqlite3 secret.db
SQLite version 3.40.1 2022-12-28 14:03:47
Enter ".help" for usage hints.
sqlite> SELECT * FROM secret
   ...> ;
)��I�պ���G�m7��ٶ`�x�t|2024-07-06 20:37:42.14852215+00:00
sqlite

This is what I filled out (no additional spaces and so on):

The node is really running on the filled out address:

But the node wasn’t being accepted anyway. Multiple times. I even created some additional api-keys (after truncating the table). Without any luck.

And yeah, it worked this way before:

root@STORJ-MULTINODE:/storj/data# sqlite3 master.db
SQLite version 3.40.1 2022-12-28 14:03:47
Enter ".help" for usage hints.
sqlite> SELECT * FROM nodes
   ...> ;
&;�j
    {���
XW6���L&D3��STORJ2|192.168.1.3:28967|�3��j�
6���N���$Nc6T�4
               �
��������
�       �;(h��~��|STORJ3|192.168.1.4:28967|�������/��<┐��
�T�┬��π
�R�?£N���≥ɚ5�L�����|STORJ5|192.168.1.6:28967|�3m�
N�                                               �as���3F������|H��
��.ח��T=沋`� .��I��|STORJ8|192.168.1.9:28967|�S����� ��0��Q�jbM:1]mE6�
2j�¤-�o���m���m+|��w�STORJ10|192.168.1.11:28967|&0kF����JHa��O�o
?+ø�9T}1 B��y`u-��@�yt�f�|STORJ12|192.168.1.13:28967|z�����H�?F�ȳ�;�X-ZA(C
���^����Q�P
           w��5}ha��|STORJ13|192.168.1.14:28967|�o�18�F2GB*���g,����ܮ^�<-:
��ԡX)��≠�B��!���≠STORJ14≠192▼168▼1▼15:28967≠�␊
                                              ����␍1���▲��(\۔≠8��⎺

6S─Z▒S└Y└│�STORJ15≠192▼168▼1▼16:28967≠�⬥⎼C8┬OQ K�█°⎼Ğ���┘�␉��┬▼^
�O��)8�┼��@��±◀3P��ý≠STORJ16≠192▼168▼1▼17:28967≠�? �Z˛�]�⎺����3�����P�
흗␊�°#���█7␊�O�␍��Ā�1@≠STORJ17≠192▼168▼1▼18:28967≠��P����▼�E    �&�S└�␉�≤⎽P�
�D�7��█�_�}���ވ�DX�STORJ18|192.168.1.19:28967|q��Y'��D�������v�n��Fm
c����<  P��bK��
               G|STORJ19|192.168.1.20:28967|��ġ⹨O�I��)�aT�L��X│M≤�
$�=��T�┼܆◀─��ļ��@≠STORJ21≠192▼168▼1▼22:28967≠E�␊��^W�㲀[)
               I�≠STORJ22≠192▼168▼1▼23:28967≠��<▼�⎼^�≤4X���␍����␊ͧ£�
≤��Z�I���FS���┴├β°"��≠STORJ23≠192▼168▼1▼24:28967≠�L���  =�=�─�R
                                                               ���⎺W�±)��
� ���┬�)�┴─�˹�$����▶≠STORJ6≠192▼168▼1▼7:28967≠��������
                                                      �������⎻!G�
���⎽�␉���C│␊�������& �STORJ11≠192▼168▼1▼12:28967≠׊������⎻▶�5̱��N ⎻)U Ƒ)R
�؂]@█≠STORJ24≠192▼168▼1▼25:28967≠�Z�R;�␉�R K[��4[���[;•
�*�␊ȑ#│���▒7��├#)# ڰ≠STORJ4≠192▼168▼1▼5:28967≠␍��^'�I
                                                     4��ؿ�HE��]▶��O>^Q)G┴
�Oշ┼�18%
        �W�"���P≠STORJ7≠192▼168▼1▼8:28967≠�L�K]�IA����@6��]���(�
���'␋���A����⬥Y�┤
                 �█≠STORJ2█≠192▼168▼1▼21:28967≠K≥�%���� E#
⎽─┌␋├␊>

So, what’s going on here?

And please devs, can we fix this issue?

I tried to reproduce and it’s added. However, usually you need to use the public IP and port, but it is working for the local IP too, I checked on my node.
I also tried to remove the multinode’s DB and added this node as a first node.

So, please, check the log of the multinode dashboard when you trying to add a node.
Also, I would suggest to use the info command instead of issue-apikey, because it also prints a NodeID. If the NodeID doesn’t match the API key - it will not be added.

Your public url it is wrong, it have to be something like your-ddns-url.com:port

I did that error myself several times because isn’t very well indicated.

Yeah, because it’s not a public URL.
The multinode dashboard and storagenodes are on the same LAN. And if you looked more down in my post, you could see that it worked before. There are several other storagenodes with LAN-IP in it.

@Alexey , the multinode Los aren’t that informative:

root@STORJ-MULTINODE:~# docker logs multinode
2024-07-06T19:06:12Z    INFO    process/exec_conf.go:318        Configuration loaded                                   {"Location": "/app/config/config.yaml"}
2024-07-06T19:06:12Z    DEBUG   process/tracing.go:104  Could not read identity for tracing setup                      {"error": "open ./identity.cert: no such file or directory"}
2024-07-06T19:06:12Z    INFO    process/tracing.go:73   Anonymized tracing enabled
2024-07-06T19:06:12Z    DEBUG   tracing collector       monkit-jaeger@v0.0.0-20240221095020-52b0792fa6cd/thrift.go:149 started
2024-07-06T19:06:12Z    DEBUG   db      multinodedb/database.go:72      Connected to:                                  {"db source": "file:config/master.db?_journal=WAL&_busy_timeout=10000"}
2024-07-06T19:06:12Z    INFO    db      migrate/versions.go:221 Database Version                                       {"version": 0}
2024-07-06T19:06:12Z    DEBUG   servers lifecycle/group.go:100  started {"items": ["console:endpoint"]}
2024-07-06T19:50:43Z    ERROR   console:endpoint        controllers/nodes.go:245                                       list node trusted satellites internal error      {"error": "nodes: rpc: tcp connector failed: rpc: dial tcp 192.168.1.20:28967: connect: connection refused", "errorVerbose": "nodes: rpc: tcp connector failed: rpc: dial tcp 192.168.1.20:28967: connect: connection refused\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
storj.io/storj/multinode/console/controllers.(*Nodes).TrustedSatellites
        /go/src/storj.io/storj/multinode/console/controllers/nodes.go:245
net/http.HandlerFunc.ServeHTTP
        /usr/local/go/src/net/http/server.go:2166
github.com/gorilla/mux.(*Router).ServeHTTP
        /go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210
net/http.serverHandler.ServeHTTP
        /usr/local/go/src/net/http/server.go:3137
net/http.(*conn).serve
        /usr/local/go/src/net/http/server.go:2039

Before and after exactly the same.

The info didn’t help either:

root@STORJ9:/storj/DBs# sqlite3 secret.db
SQLite version 3.40.1 2022-12-28 14:03:47
Enter ".help" for usage hints.
sqlite> SELECT * FROM secret
  ...> ;
sqlite>

root@STORJ9:/storj/DBs# docker exec -it storagenode /app/storagenode info --config-dir config --identity-dir identity --storage2.database-dir="dbs"
2024-07-07T06:14:23Z    INFO Configuration loaded    {"Process": "storagenode", "Location": "/app/config/config.yaml"}
2024-07-07T06:14:23Z    INFO    Anonymized tracing enabled      {"Process": "storagenode"}
2024-07-07T06:14:23Z    INFO    Identity loaded.        {"Process": "storagenode", "Node ID": "1ghkS2nPpDdLPXRajjQaUqzH99b4NuGQoMg1WXp1FoQDTGvetS"}
ID: 1ghkS2nPpDdLPXRajjQaUqzH99b4NuGQoMg1WXp1FoQDTGvetS
API Secret: 4UKReCXZTd7VdULA3kF4IN5vM01DXfawcG2nN16snXE=
Public Address:

root@STORJ9:/storj/DBs# sqlite3 secret.db                                                                               SQLite version 3.40.1 2022-12-28 14:03:47
Enter ".help" for usage hints.
sqlite> SELECT * FROM secret
...> ;
��x%��uB�Ax �3MC]�m�7^��q|2024-07-07 06:14:23.306859476+00:00

The fact public address is empty, of interesting. But also kind of expected, since I normally send it by command line in one of the parameters.

This is what I filled out:

No luck either.

Is there a way to impute is into the database myself?

actually it’s. Does this identity is signed?

perhaps, but it’s not useful.
Please try the info command with the --json option and save this info to the file. Then you may import it to the multinode dashboard using the command

./multinode add node.json

Of course you may modify the file before that.

For sure, they’re all long functioning nodes. Moreover this is about 192.168.1.20 instead of 192.168.1.10 and already there before I tried appending this node.

For those, encountering the same problem the solution. Apparently dev is too busy with all this other problems, so:

b58CharToDicimal() {
    i=0
    for char in $(echo {1..9} {A..H} {J..N} {P..Z} {a..k} {m..z}) ; do
        if test $1 == $char ;then
            echo $i
            return 0
        fi
        i=$(( ${i}+1 ))
    done
    echo 'invalidate base58 char'
    return 1
}

decode58() {
    num58=$1

    # remain leading 00 in hexadecimal
    prefix=''
    if test "1" == "$(echo -n $num58 | cut -c 1)" ; then
        prefix=00
    fi

    exponent=$((${#num58} - 1))
    numDecimal=0
    for char in $(echo $num58 | sed 's/./& /g') ;do
        digit=$(b58CharToDicimal $char)
        numDecimal=$(echo "$digit * 58 ^ $exponent + $numDecimal" | bc)
        exponent=$(($exponent-1))
    done
    numHex=$prefix$(echo "obase=16;$numDecimal" | bc | tr A-Z a-z)
    echo $numHex
}

# ID
sNodeID="1ghkS2nPpDdLPXRajjQaUqzH99b4NuGQoMg1WXp1FoQDTGvetS"
# Name you like it to have
sName="STORJ9"
# Secret
sSecret="4UKReCXZTd7VdULA3kF4IN5vM01DXfawcG2nN16snXE="
# IP+Port
sAddress="192.168.1.10:28967"

sSecret="$( echo "$sSecret" | base64 --decode | sed "s/'/''/g" )"
sNodeID="$( decode58 "$sNodeID" )"
sNodeID="${sNodeID:2:64}"
sqlite3 "/path/to/multinodeDNs/master.db" "INSERT INTO nodes (id,name,public_address,api_secret) VALUES (X'${sNodeID^^}' )', '$sName', '$Address', '$sSecret');"

And yeah, it works! :+1:

So much again for the many unresolved bugs. Indeed, starting to begin a bit sarcastic…

@Alexey , might that bug be resolved quite soon and this added to the list? Is there a check or something again local addresses?

I didn’t get where bug is, I’m sorry. It’s working for my nodes no matter what I do.
Would you mind to open a bug on the GitHub? (to be able to validate that it’s solved, since I cannot reproduce it)

Where is the addition code of the multinode? It might also be a timeout issue or something. If I see the code, I can try to see for myself why it didn’t work here.

Multinode works fine for me too. If I remember right, I even added the nodes with localhost (127.0.0.1) and it worked.
Make sure that your machine can reach the address you trying to add. Maybe check first (with curl if you are using Linux) if you can reach the address and port.

Maybe the all healthy=false is the problem because it’s generating a http error 500 if I am correct.
But that would have @Alexey to confirm if this could cause this

Edit: checked my status page and all healthy=false generates a 503 http error

Could be, but the reason for it is an online score at the moment of 85%.

if stat.DisqualifiedAt != nil || stat.SuspendedAt != nil || stat.OnlineScore < 0.9 {

Would be really strange if your can’t add a node with lower OnlineScore.

It for certain can, because after forcefully adding it to the database or turned out to work. So…

No, unlikely. My node which I added like this has AllHealthy: false too. It doesn’t prevent to add a node.
I only have had a one guess, that the NodeID doesn’t match the API key or the IP:port.

You don’t believe it yourself, do you? :wink:

Let’s start easy: if IP or port would be wrong, it wouldn’t be able to reach the node after forcefully injecting it into the database.

If API-key or NodeID would match (which is really the same as in the code), it wouldn’t work either. Otherwise, we are having another bug / minor security issue.

I actually think it had something to do with timeout. Because I now have sometimes also a row of zeroes, when I open the dashboard. It usually changes after some seconds.

Although always stating it’s online, but that’s another bug of the whole dashboard.

I do not fully know your configuration :person_shrugging:
It’s possible to specify a correct NodeID, a correct API key but incorrect IP/port, if you have several nodes on the same local IP (and it’s a very often situation), so it’s quitely easy to provide a port from the another node. The response on http://192.168.10.1:28967 doesn’t contain a NodeID.

By “match” I don’t mean equality, but rather that the API key belongs to that NodeID, sorry for confusion. However, I just expressed that I have had a such guess, but after you posted the same with the info command (where NodeID is visible altogether with the API key), it was become a less possible. Except that you might be using a port from the another node.
However, since you were able to add it forcibly via a database, then this guess was not confirmed too.

it’s possible, but the latency should not be so great to have a timeout in the local network I think. Do you use a wifi or similar unstable connection?

But all these explanations have one manco: it’s working now and then with correct figures, and then it’s not. So, it’s for sure not a problem of IP, port, API-key, and nodeID.

Since I had some time yesterday, I looked into it. And to my surprise also the dashboard of this node isn’t working. Although being online. Also recreating the docker container didn’t work. So still debugging, before crying wolf with a bug report.

Even the CLI one? If so, then it should be in a failed state then (or misconfigured/overlapped ports).
The CLI dashboard uses the address and port from the server.private-address option.
The web dashboard uses the address and port from the console.address option.