First time user, can't run the storjlabSnContainer: "ping satellite error: rpc: dial tcp: operation was canceled"

Node Operators troubleshooting
,
1

Not sure what to do with this one.

2021-03-14T17:31:51.872Z        ERROR   contact:service ping satellite failed   {"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 1, "error": "ping satellite error: rpc: dial tcp: operation was canceled", "errorVerbose": "ping satellite error: rpc: dial tc
p: operation was canceled\n\tstorj.io/common/rpc.TCPConnector.DialContextUnencrypted:106\n\tstorj.io/common/rpc.TCPConnector.DialContext:70\n\tstorj.io/common/rpc.Dialer.dialEncryptedConn:180\n\tstorj.io/common/rpc.Dialer.DialNodeURL.func1:101\n\tstorj.io/common/rpc/rpcpool.(*Pool).Get:87
\n\tstorj.io/common/rpc.Dialer.dialPool:146\n\tstorj.io/common/rpc.Dialer.DialNodeURL:100\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:124\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.fu
nc1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}                                                                                                                                            
2021-03-14T17:31:51.872Z        ERROR   contact:service ping satellite failed   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "attempts": 1, "error": "ping satellite error: rpc: dial tcp: operation was canceled", "errorVerbose": "ping satellite error: rpc: dial t
cp: operation was canceled\n\tstorj.io/common/rpc.TCPConnector.DialContextUnencrypted:106\n\tstorj.io/common/rpc.TCPConnector.DialContext:70\n\tstorj.io/common/rpc.Dialer.dialEncryptedConn:180\n\tstorj.io/common/rpc.Dialer.DialNodeURL.func1:101\n\tstorj.io/common/rpc/rpcpool.(*Pool).Get:8
7\n\tstorj.io/common/rpc.Dialer.dialPool:146\n\tstorj.io/common/rpc.Dialer.DialNodeURL:100\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:124\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.f
unc1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}                                                                                                                                           
2021-03-14T17:31:51.872Z        INFO    contact:service context cancelled       {"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}                                                                                                                                           
2021-03-14T17:31:51.872Z        INFO    contact:service context cancelled       {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}                                                                                                                                          
2021-03-14T17:31:51.872Z        ERROR   contact:service ping satellite failed   {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 1, "error": "ping satellite error: rpc: dial tcp: operation was canceled", "errorVerbose": "ping satellite error: rpc: dial t
cp: operation was canceled\n\tstorj.io/common/rpc.TCPConnector.DialContextUnencrypted:106\n\tstorj.io/common/rpc.TCPConnector.DialContext:70\n\tstorj.io/common/rpc.Dialer.dialEncryptedConn:180\n\tstorj.io/common/rpc.Dialer.DialNodeURL.func1:101\n\tstorj.io/common/rpc/rpcpool.(*Pool).Get:8
7\n\tstorj.io/common/rpc.Dialer.dialPool:146\n\tstorj.io/common/rpc.Dialer.DialNodeURL:100\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:124\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.f
unc1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}                                                                                                                                           
2021-03-14T17:31:51.873Z        INFO    contact:service context cancelled       {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}                                                                                                                                          
2021-03-14T17:31:51.873Z        ERROR   contact:service ping satellite failed   {"Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 1, "error": "ping satellite error: rpc: dial tcp: operation was canceled", "errorVerbose": "ping satellite error: rpc: dial t
cp: operation was canceled\n\tstorj.io/common/rpc.TCPConnector.DialContextUnencrypted:106\n\tstorj.io/common/rpc.TCPConnector.DialContext:70\n\tstorj.io/common/rpc.Dialer.dialEncryptedConn:180\n\tstorj.io/common/rpc.Dialer.DialNodeURL.func1:101\n\tstorj.io/common/rpc/rpcpool.(*Pool).Get:8
7\n\tstorj.io/common/rpc.Dialer.dialPool:146\n\tstorj.io/common/rpc.Dialer.DialNodeURL:100\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:124\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.f
unc1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}                                                                                                                                           
2021-03-14T17:31:51.874Z        INFO    contact:service context cancelled       {"Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB"}                                                                                                                                          
Error: piecestore monitor: error verifying location and/or readability of storage directory: content of file is not a valid node ID:                                
2021-03-14T17:31:58.566Z        INFO    Operator email  {"Address": "xxx"}                                                                                                                                                                                                       
2021-03-14T17:31:58.567Z        INFO    Operator wallet {"Address": "xxx"}                                                                                                                                                                                
2021-03-14T17:32:00.213Z        INFO    Telemetry enabled       {"instance ID": "xxx"}                                                                                                                                                           
2021-03-14T17:32:00.278Z        INFO    db.migration    Database Version        {"version": 51}                                                                                                                                                                                                  
2021-03-14T17:32:01.956Z        INFO    preflight:localtime     start checking local system clock with trusted satellites' system clock.                                                                                                                                                         
2021-03-14T17:32:02.858Z        INFO    preflight:localtime     local system clock is in sync with trusted satellites' system clock.                                                                                                                                                             
2021-03-14T17:32:02.858Z        INFO    Node xxx started                                                                                                                                                                                         
2021-03-14T17:32:02.858Z        INFO    bandwidth       Performing bandwidth usage rollups                                                                                                                                                                                                       
2021-03-14T17:32:02.858Z        INFO    Public server started on [::]:7777                                                                                                                                                                                                                       
2021-03-14T17:32:02.858Z        INFO    Private server started on 127.0.0.1:7778                                                                                                                                                                                                                 
2021-03-14T17:32:02.859Z        INFO    trust   Scheduling next refresh {"after": "4h49m22.102804469s"}                                                                                                                                                                                          
2021-03-14T17:32:02.859Z        INFO    failed to sufficiently increase receive buffer size (was: 224 kiB, wanted: 2048 kiB, got: 448 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.                                                              
2021-03-14T17:32:02.861Z        ERROR   services        unexpected shutdown of a runner {"name": "piecestore:monitor", "error": "piecestore monitor: error verifying location and/or readability of storage directory: content of file is not a valid node ID: ", "errorVerbose": "piecestore mon
itor: error verifying location and/or readability of storage directory: content of file is not a valid node ID: \n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func1.1:131\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func1:128\n\tgola
ng.org/x/sync/errgroup.(*Group).Go.func1:57"}                                                                                                                                                                                                                                                    
2021-03-14T17:32:02.861Z        ERROR   gracefulexit:chore      error retrieving satellites.    {"error": "satellitesdb error: context canceled", "errorVerbose": "satellitesdb error: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*satellitesDB).ListGracefulExits:100\n\tstor
j.io/storj/storagenode/gracefulexit.(*service).ListPendingExits:89\n\tstorj.io/storj/storagenode/gracefulexit.(*Chore).Run.func1:53\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/storj/storagenode/gracefulexit.(*Chore).Run:50\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:
81\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:80\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}                                                                                                                                                         
2021-03-14T17:32:02.862Z        ERROR   piecestore:cache        error during init space usage db:       {"error": "piece space used error: context canceled", "errorVerbose": "piece space used error: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceSpaceUsedDB).Init:73\n
\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:75\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:81\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:80\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}                                     
2021-03-14T17:32:02.862Z        ERROR   contact:service ping satellite failed   {"Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "attempts": 1, "error": "ping satellite error: rpc: dial tcp: operation was canceled", "errorVerbose": "ping satellite error: rpc: dial tc
p: operation was canceled\n\tstorj.io/common/rpc.TCPConnector.DialContextUnencrypted:106\n\tstorj.io/common/rpc.TCPConnector.DialContext:70\n\tstorj.io/common/rpc.Dialer.dialEncryptedConn:180\n\tstorj.io/common/rpc.Dialer.DialNodeURL.func1:101\n\tstorj.io/common/rpc/rpcpool.(*Pool).Get:87
\n\tstorj.io/common/rpc.Dialer.dialPool:146\n\tstorj.io/common/rpc.Dialer.DialNodeURL:100\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:124\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.fu
nc1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}                                                                                                                                            
2021-03-14T17:32:02.863Z        ERROR   nodestats:cache Get pricing-model/join date failed      {"error": "context canceled"}                                                                                                                                                                    
2021-03-14T17:32:02.863Z        ERROR   contact:service ping satellite failed   {"Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "attempts": 1, "error": "ping satellite error: rpc: dial tcp: operation was canceled", "errorVerbose": "ping satellite error: rpc: dial t
cp: operation was canceled\n\tstorj.io/common/rpc.TCPConnector.DialContextUnencrypted:106\n\tstorj.io/common/rpc.TCPConnector.DialContext:70\n\tstorj.io/common/rpc.Dialer.dialEncryptedConn:180\n\tstorj.io/common/rpc.Dialer.DialNodeURL.func1:101\n\tstorj.io/common/rpc/rpcpool.(*Pool).Get:8
7\n\tstorj.io/common/rpc.Dialer.dialPool:146\n\tstorj.io/common/rpc.Dialer.DialNodeURL:100\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:124\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.f
unc1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}                                                                                                                                           
2021-03-14T17:32:02.863Z        ERROR   contact:service ping satellite failed   {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 1, "error": "ping satellite error: rpc: dial tcp: operation was canceled", "errorVerbose": "ping satellite error: rpc: dial t
cp: operation was canceled\n\tstorj.io/common/rpc.TCPConnector.DialContextUnencrypted:106\n\tstorj.io/common/rpc.TCPConnector.DialContext:70\n\tstorj.io/common/rpc.Dialer.dialEncryptedConn:180\n\tstorj.io/common/rpc.Dialer.DialNodeURL.func1:101\n\tstorj.io/common/rpc/rpcpool.(*Pool).Get:8
7\n\tstorj.io/common/rpc.Dialer.dialPool:146\n\tstorj.io/common/rpc.Dialer.DialNodeURL:100\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:124\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:95\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.f
unc1:87\n\tstorj.io/common/sync2.(*Cycle).Run:92\n\tstorj.io/common/sync2.(*Cycle).Start.func1:71\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}                                                                                                                                           
2021-03-14T17:32:02.863Z        INFO    contact:service context cancelled       {"Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo"}                                                                                                                                           
2021-03-14T17:32:02.863Z        INFO    contact:service context cancelled       {"Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB"}                                                                                                                                          
2021-03-14T17:32:02.863Z        INFO    contact:service context cancelled       {"Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2

Are you using static IP or DDNS ?

1 Like
3

You missed setup step: Storage Node - Storj Docs or changed the identity. If you changed identity, you need to repeat this step once. Make sure to use same paths as your storagenode container.

2 Likes
4

DDNS I believe. I vaguely remember having asked my ISP this and their answer being that it wasnā€™t static, but try to have the IP addresses remain the same as long as possible.
I havenā€™t seen my IP change since I switched to this ISP around 5 years ago.

Do I need to use that one then?
Because Iā€™m on a qnap and Iā€™ve used this one, plus the wizard: QNAP Storage Node App - Storj Docs

After finishing the wizard, and fixing one or two issues, I had several more errors to deal with concerning missing files and folders that I created manually.
blobs, temp, garbage, trash and the storage-dir-verification file.

5

You need to remove storage-dir-verification file and run the setup command once. Make sure to use the same paths as in regular storagenode.

These file and directories are creating with the setup step. You do not need to create them manually. This step is introduced to protect your node from disqualification. If the file or folders are missed - the storagenode will crash instead of starting from scratch and receive a disqualification for losing data.

1 Like
6

Can I create a new identity using the same authorization token via the SSN Setup?
Does that work?
I already created an identity before I did the SSN setup because it was step 2.

7

You should generate a new identity and use a new authorization token. The authorization token is one time usage, the unique certificate is your identity, not the token. The token is needed only to sign your identity (certificate).

1 Like
8

When do I know if the identity has been generated?

Iā€™m looking at this right now:

Identity generation STATUS(2021-03-15 20:24:36):
Process ID: , Started at: 2021-03-15 20:15:36

Do I need to wait or can I press finish?

[update]

After an hour of waiting and inactivity, the screen is locked and Storj not installed.

9

You tried to generate an identity on your QNAP? If so, it should take several hours, maybe even days (depends on CPU and luck).
Maybe itā€™s better to generate it on your PC/Mac, sign and transfer to QNAP.

1 Like
10

Yes, and it fails.

==== Generate Identity called (mail@domain.tld:xxx) ============
ERROR(generateIdentity.sh): sufficient params not supplied (mail@domain.tld:xxx)
Usage(generateIdentity.sh): generateIdentity.sh <IdentityKeyString> <keyBase>

It only supplies the token to this script.

It would be if that doesnā€™t fail as well, because the identity is not found, despite pointing to the correct folder.

{
 "Identity":"\/path\/to\/folder\/with\/identity.key&identity.cert",
 "AuthKey":"",
 "Port":"mydomain.tld:28967",
 "Wallet":"xxx","Allocation":"1024",
 "Email":"xxx@mydomain.tld",
 "Directory":"\/path\/to_storage",
 "last_log":"\na40169a50ca2   storjlabs\/storagenode:latest   \"\/entrypoint\"   9 seconds ago   Up Less than a second   192.168.x.x:9000->9000\/tcp, 192.168.x.x:14002->14002\/tcp, 0.0.0.0:28967->28967\/tcp   storjlabsSnContainer\n"
}

Also doing so creates the following issue:

2021-03-16T05:33:39.778Z        INFO    Operator email  {"Address": "xxx"}                                                                                                                                                                                                       
2021-03-16T05:33:39.778Z        INFO    Operator wallet {"Address": "xxx"}                                                                                                                                                                                
Error: Error starting master database on storagenode: group:                                                                                                                                                                                                                                     
--- stat config/storage/blobs: no such file or directory                                                                                                                                                                                                                                         
--- stat config/storage/temp: no such file or directory                                                                                                                                                                                                                                          
--- stat config/storage/garbage: no such file or directory                                                                                                                                                                                                                                       
--- stat config/storage/trash: no such file or directory

Also Iā€™d like to point out that the SSN download link on the ā€˜Become a hostā€™ page leads to an older release version.

11

If you generated an identity on your PC/Mac and then moved it to a different folder, you need to specify where is it when you authorize it. For example, you moved identity to the /home/folaht/identity, then authorization command would looks like:

identity --identity-dir /home/folaht authorize identity mail@doamin.tld:xxxxx

Here is a name of the service ā€œidentityā€, because the end dir is /home/folaht/identity. It just takes the service name as a folder name in path specified by --identity-dir option.

Thanks for head-ups regarding link on the registration site. I created an internal issue to the team.

Mean, that you didnā€™t setup with your identity path and storage path: Storage Node - Storj Docs

You should run this command when you created a new storage path or new identity. This command will create all needed structure and file for this identity.

1 Like
12

I created a new identity this way and I still recieve the same message.

The identity files don't exist at the path selected. Please create identity or copy the identity folder at the given path.
13

Please, show the content of the specified folder in --identity-dir

14 
[folaht@pjehrsohmehj Stohrje]$ ls -lha
total 0
drwxr-xr-x 1 folaht folaht  16 16 mrt 09:39 .
drwxr-xr-x 1 folaht folaht 368 16 mrt 08:45 ..
drwxr-xr-x 1 folaht folaht 160 16 mrt 09:40 identity
[folaht@pjehrsohmehj Stohrje]$ ls -lha identity
total 24K
drwxr-xr-x 1 folaht folaht  160 16 mrt 09:40 .
drwxr-xr-x 1 folaht folaht   16 16 mrt 09:39 ..
-rw-r--r-- 1 folaht folaht  562 16 mrt 09:40 ca.1615884001.cert
-rw-r--r-- 1 folaht folaht 1,1K 16 mrt 09:40 ca.cert
-rw------- 1 folaht folaht  241 16 mrt 09:36 ca.key
-rw-r--r-- 1 folaht folaht 1,1K 16 mrt 09:40 identity.1615884001.cert
-rw-r--r-- 1 folaht folaht 1,6K 16 mrt 09:40 identity.cert
-rw------- 1 folaht folaht  241 16 mrt 09:36 identity.key
15

I get the feeling that you didnā€™t create the storj qnap app and that in both cases, the identitypath is returned empty.

Iā€™m going to dig into the source code.

[update]

Before I even looked into the source code, I looked into the logs.

Mon Mar 15 22:30:09 CET 2021  (identityLib)  (file_exist) File /share/homes/admin/idaqtite/nodpozazjstohrje/storagenode/identity.key or others don't exists !

This explains something. So itā€™s always looking into the storagenode folder.

[update]

It would also be nice to know who the ā€œothersā€ are.

16

This command should work:

identity --identity-dir /home/folaht/Stohrje authorize identity mail@doamin.tld:xxxxx

But I see, that you already signed it, so no need to sign it another time.

Yes, itā€™s a third party software. Itā€™s a wrapper around docker.
So, for me personally itā€™s much simpler to run the storagenode with a docker from CLI (via ssh) directly, than trying to force the outdated app to work.
It will be one command to setup (should be done once for nodeā€™s lifetime):

docker run --rm -e SETUP="true" \
    --mount type=bind,source="/home/folaht/Stohrje/identity",destination=/app/identity \
    --mount type=bind,source="/Volume/storagenode",destination=/app/config \
    --name storagenode storjlabs/storagenode:latest

And then finally run

docker run -d --restart unless-stopped --stop-timeout 300 \
    -p 28967:28967 \
    -p 14002:14002 \
    -e WALLET="0xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" \
    -e EMAIL="user@example.com" \
    -e ADDRESS="domain.ddns.net:28967" \
    -e STORAGE="2TB" \
    --mount type=bind,source="/home/folaht/Stohrje/identity",destination=/app/identity \
    --mount type=bind,source="/Volume/storagenode",destination=/app/config \
    --name storagenode storjlabs/storagenode:latest

To update storagenode automatically you can then use a watchtower: Software Updates - Storj Docs

3 Likes