Rpc: dial tcp 127.0.0.1:7778: connect: connection refused

Hi everyone,

I recently got this error message in my node rpc: dial tcp 127.0.0.1:7778: connect: connection refused I tried restarting Docker and removing the image but didn’t work.

See logs below:

2022-05-03T12:14:59.446575700Z  12550K .......... .......... .......... .......... .......... 87% 19.7M 0s
2022-05-03T12:14:59.449096706Z  12600K .......... .......... .......... .......... .......... 87% 18.0M 0s
2022-05-03T12:14:59.451639840Z  12650K .......... .......... .......... .......... .......... 88% 19.5M 0s
2022-05-03T12:14:59.454152771Z  12700K .......... .......... .......... .......... .......... 88% 19.6M 0s
2022-05-03T12:14:59.457836262Z  12750K .......... .......... .......... .......... .......... 88% 18.8M 0s
2022-05-03T12:14:59.460138973Z  12800K .......... .......... .......... .......... .......... 89% 14.4M 0s
2022-05-03T12:14:59.463168194Z  12850K .......... .......... .......... .......... .......... 89% 18.7M 0s
2022-05-03T12:14:59.465395555Z  12900K .......... .......... .......... .......... .......... 89% 18.4M 0s
2022-05-03T12:14:59.468056540Z  12950K .......... .......... .......... .......... .......... 90% 19.4M 0s
2022-05-03T12:14:59.470533824Z  13000K .......... .......... .......... .......... .......... 90% 18.6M 0s
2022-05-03T12:14:59.473076884Z  13050K .......... .......... .......... .......... .......... 90% 19.2M 0s
2022-05-03T12:14:59.475682240Z  13100K .......... .......... .......... .......... .......... 91% 18.9M 0s
2022-05-03T12:14:59.478937791Z  13150K .......... .......... .......... .......... .......... 91% 19.5M 0s
2022-05-03T12:14:59.481403353Z  13200K .......... .......... .......... .......... .......... 92% 15.1M 0s
2022-05-03T12:14:59.483933858Z  13250K .......... .......... .......... .......... .......... 92% 19.4M 0s
2022-05-03T12:14:59.486512974Z  13300K .......... .......... .......... .......... .......... 92% 19.1M 0s
2022-05-03T12:14:59.489070163Z  13350K .......... .......... .......... .......... .......... 93% 19.9M 0s
2022-05-03T12:14:59.491406875Z  13400K .......... .......... .......... .......... .......... 93% 19.6M 0s
2022-05-03T12:14:59.493737864Z  13450K .......... .......... .......... .......... .......... 93% 20.9M 0s
2022-05-03T12:14:59.496195240Z  13500K .......... .......... .......... .......... .......... 94% 19.9M 0s
2022-05-03T12:14:59.499211943Z  13550K .......... .......... .......... .......... .......... 94% 20.7M 0s
2022-05-03T12:14:59.501520802Z  13600K .......... .......... .......... .......... .......... 94% 16.6M 0s
2022-05-03T12:14:59.503895606Z  13650K .......... .......... .......... .......... .......... 95% 20.4M 0s
2022-05-03T12:14:59.506359575Z  13700K .......... .......... .......... .......... .......... 95% 20.7M 0s
2022-05-03T12:14:59.508860469Z  13750K .......... .......... .......... .......... .......... 95% 19.8M 0s
2022-05-03T12:14:59.511205310Z  13800K .......... .......... .......... .......... .......... 96% 19.7M 0s
2022-05-03T12:14:59.513673556Z  13850K .......... .......... .......... .......... .......... 96% 19.6M 0s
2022-05-03T12:14:59.516136840Z  13900K .......... .......... .......... .......... .......... 96% 20.5M 0s
2022-05-03T12:14:59.519130969Z  13950K .......... .......... .......... .......... .......... 97% 20.5M 0s
2022-05-03T12:14:59.521472828Z  14000K .......... .......... .......... .......... .......... 97% 16.5M 0s
2022-05-03T12:14:59.523820854Z  14050K .......... .......... .......... .......... .......... 97% 20.6M 0s
2022-05-03T12:14:59.526178102Z  14100K .......... .......... .......... .......... .......... 98% 20.6M 0s
2022-05-03T12:14:59.528643997Z  14150K .......... .......... .......... .......... .......... 98% 20.8M 0s
2022-05-03T12:14:59.530995560Z  14200K .......... .......... .......... .......... .......... 98% 19.6M 0s
2022-05-03T12:14:59.533335993Z  14250K .......... .......... .......... .......... .......... 99% 20.7M 0s
2022-05-03T12:14:59.535738888Z  14300K .......... .......... .......... .......... .......... 99% 21.0M 0s
2022-05-03T12:14:59.538255949Z  14350K .......... .......... .......... .......... .......... 99% 20.8M 0s
2022-05-03T12:14:59.538538075Z  14400K                                                       100% 2.55M=0.8s
2022-05-03T12:14:59.538562686Z 
2022-05-03T12:14:59.538761313Z 2022-05-03 12:14:59 (16.6 MB/s) - '/tmp/storagenode.zip' saved [14746241/14746241]
2022-05-03T12:14:59.538787387Z 
2022-05-03T12:15:00.804762834Z 2022-05-03 12:15:00,804 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
2022-05-03T12:15:00.833364124Z 2022-05-03 12:15:00,832 INFO RPC interface 'supervisor' initialized
2022-05-03T12:15:00.833544381Z 2022-05-03 12:15:00,833 CRIT Server 'inet_http_server' running without any HTTP authentication checking
2022-05-03T12:15:00.834432759Z 2022-05-03 12:15:00,834 INFO supervisord started with pid 1
2022-05-03T12:15:01.843671606Z 2022-05-03 12:15:01,841 INFO spawned: 'processes' with pid 55
2022-05-03T12:15:01.852909714Z 2022-05-03 12:15:01,851 INFO spawned: 'storagenode' with pid 56
2022-05-03T12:15:01.858895158Z 2022-05-03 12:15:01,857 INFO spawned: 'storagenode-updater' with pid 57
2022-05-03T12:15:01.863367658Z 2022-05-03 12:15:01,862 INFO exited: processes (exit status 0; not expected)
2022-05-03T12:15:01.904093135Z 2022-05-03T12:15:01.903Z	INFO	Running on version	{"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.53.1"}
2022-05-03T12:15:01.904994716Z 2022-05-03T12:15:01.904Z	INFO	Downloading versions.	{"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2022-05-03T12:15:01.987283529Z 2022-05-03T12:15:01.986Z	INFO	Operator email	{"Process": "storagenode", "Address": "eduardo.sequeira.hernandez@gmail.com"}
2022-05-03T12:15:01.987913502Z 2022-05-03T12:15:01.986Z	INFO	Operator wallet	{"Process": "storagenode", "Address": "0x290Eb454a283048ae0378f82bca20266C52eeB0e"}
2022-05-03T12:15:02.008282547Z Error: Error starting master database on storagenode: group:
2022-05-03T12:15:02.008381027Z --- stat config/storage/blobs: no such file or directory
2022-05-03T12:15:02.008408434Z --- stat config/storage/temp: no such file or directory
2022-05-03T12:15:02.008433045Z --- stat config/storage/garbage: no such file or directory
2022-05-03T12:15:02.008456748Z --- stat config/storage/trash: no such file or directory
2022-05-03T12:15:02.012350497Z 2022-05-03 12:15:02,011 INFO exited: storagenode (exit status 1; not expected)
2022-05-03T12:15:02.117521352Z 2022-05-03T12:15:02.116Z	INFO	Current binary version	{"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.53.1"}
2022-05-03T12:15:02.117726423Z 2022-05-03T12:15:02.116Z	INFO	Version is up to date	{"Process": "storagenode-updater", "Service": "storagenode"}
2022-05-03T12:15:02.146670956Z 2022-05-03T12:15:02.145Z	INFO	Current binary version	{"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.53.1"}
2022-05-03T12:15:02.146760288Z 2022-05-03T12:15:02.146Z	INFO	Version is up to date	{"Process": "storagenode-updater", "Service": "storagenode-updater"}
2022-05-03T12:15:03.156053320Z 2022-05-03 12:15:03,154 INFO spawned: 'processes' with pid 88
2022-05-03T12:15:03.165170357Z 2022-05-03 12:15:03,164 INFO spawned: 'storagenode' with pid 89
2022-05-03T12:15:03.165728332Z 2022-05-03 12:15:03,165 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-05-03T12:15:03.174590946Z 2022-05-03 12:15:03,174 INFO exited: processes (exit status 0; not expected)
2022-05-03T12:15:03.290283573Z 2022-05-03T12:15:03.289Z	INFO	Operator email	{"Process": "storagenode", "Address": "eduardo.sequeira.hernandez@gmail.com"}
2022-05-03T12:15:03.290780327Z 2022-05-03T12:15:03.289Z	INFO	Operator wallet	{"Process": "storagenode", "Address": "0x290Eb454a283048ae0378f82bca20266C52eeB0e"}
2022-05-03T12:15:03.293278870Z Error: Error starting master database on storagenode: group:
2022-05-03T12:15:03.293353092Z --- stat config/storage/blobs: no such file or directory
2022-05-03T12:15:03.293374091Z --- stat config/storage/temp: no such file or directory
2022-05-03T12:15:03.293393610Z --- stat config/storage/garbage: no such file or directory
2022-05-03T12:15:03.293412776Z --- stat config/storage/trash: no such file or directory
2022-05-03T12:15:03.297186063Z 2022-05-03 12:15:03,296 INFO exited: storagenode (exit status 1; not expected)
2022-05-03T12:15:05.308252901Z 2022-05-03 12:15:05,306 INFO spawned: 'processes' with pid 98
2022-05-03T12:15:05.317270719Z 2022-05-03 12:15:05,316 INFO spawned: 'storagenode' with pid 99
2022-05-03T12:15:05.326857215Z 2022-05-03 12:15:05,326 INFO exited: processes (exit status 0; not expected)
2022-05-03T12:15:05.442013226Z 2022-05-03T12:15:05.441Z	INFO	Operator email	{"Process": "storagenode", "Address": "eduardo.sequeira.hernandez@gmail.com"}
2022-05-03T12:15:05.442451628Z 2022-05-03T12:15:05.441Z	INFO	Operator wallet	{"Process": "storagenode", "Address": "0x290Eb454a283048ae0378f82bca20266C52eeB0e"}
2022-05-03T12:15:05.444771674Z Error: Error starting master database on storagenode: group:
2022-05-03T12:15:05.444865044Z --- stat config/storage/blobs: no such file or directory
2022-05-03T12:15:05.444887617Z --- stat config/storage/temp: no such file or directory
2022-05-03T12:15:05.444908580Z --- stat config/storage/garbage: no such file or directory
2022-05-03T12:15:05.444928691Z --- stat config/storage/trash: no such file or directory
2022-05-03T12:15:05.448321224Z 2022-05-03 12:15:05,447 INFO exited: storagenode (exit status 1; not expected)
2022-05-03T12:15:08.460906961Z 2022-05-03 12:15:08,459 INFO spawned: 'processes' with pid 108
2022-05-03T12:15:08.471751909Z 2022-05-03 12:15:08,470 INFO spawned: 'storagenode' with pid 109
2022-05-03T12:15:08.481290149Z 2022-05-03 12:15:08,480 INFO exited: processes (exit status 0; not expected)
2022-05-03T12:15:08.482332341Z 2022-05-03 12:15:08,481 INFO gave up: processes entered FATAL state, too many start retries too quickly
2022-05-03T12:15:08.596463510Z 2022-05-03T12:15:08.595Z	INFO	Operator email	{"Process": "storagenode", "Address": "eduardo.sequeira.hernandez@gmail.com"}
2022-05-03T12:15:08.596867783Z 2022-05-03T12:15:08.596Z	INFO	Operator wallet	{"Process": "storagenode", "Address": "0x290Eb454a283048ae0378f82bca20266C52eeB0e"}
2022-05-03T12:15:08.599415383Z Error: Error starting master database on storagenode: group:
2022-05-03T12:15:08.599495845Z --- stat config/storage/blobs: no such file or directory
2022-05-03T12:15:08.599516845Z --- stat config/storage/temp: no such file or directory
2022-05-03T12:15:08.599536271Z --- stat config/storage/garbage: no such file or directory
2022-05-03T12:15:08.599555104Z --- stat config/storage/trash: no such file or directory
2022-05-03T12:15:08.603111674Z 2022-05-03 12:15:08,602 INFO exited: storagenode (exit status 1; not expected)
2022-05-03T12:15:09.605434957Z 2022-05-03 12:15:09,604 INFO gave up: storagenode entered FATAL state, too many start retries too quickly

Does anybody has an idea what is going on?

Thanks
Edu

Hi Edu,

Is this docker on windows or linux?

This looks like a permissions or access issue. Please verify the data exists in those locations and the file permissions are correct.

It looks like it cannot find the directory anymore, Make sure its mounted and the drive is accessible.

Hi @baker this is Linux Ubuntu 21.10. I noticed this morning but I am currently at the office so, not able to dive in the details yet but that’s a good suggestion.

1 Like

Hi @deathlessdd thanks for the suggestion. I will try to troubleshoot it this afternoon and see if I can get it fixed. Will try to reconfigure everything and see if that makes a change. I will keep you posted.

I am still having the problem and I don’t know why is going on. Just in case I am using a Rasp Pi 4 with Ubuntu 21.10 and I don’t know if that’s a problem. The node was working fine and then suddenly showed up that error.

Cheers
Edu

Check your usb connection and make sure its mounted correctly.

1 Like

Either you missed the setup step (if this is a new node), or specified a wrong path for data (if that was a working node).
If this was a working node, please do not execute the setup step! You need to check your data location and fix an underlaying issue.

1 Like

Hi @Alexey,

My data location is fine. Now I got this error message open /app/config/config.yaml: permission denied which I don’t know why is showing up when I tried to run the dashboard
sudo docker exec -it storagenode /app/dashboard.sh

I changed the permission and still get the RPC error

2022-05-06T19:17:38.892Z	INFO	Configuration loaded	{"Process": "storagenode", "Location": "/app/config/config.yaml"}
2022-05-06T19:17:38.893Z	INFO	Identity loaded.	{"Process": "storagenode", "Node ID": "17rUiCzph4fnAGUWwCuUaidxU4o2xGBtnvTyWzwfEbSiQ3Wk8B"}
Error: rpc: dial tcp 127.0.0.1:7778: connect: connection refused```

By the way these are my current logs, I don’t know if it rings any bells to somebody:

2022-05-06T19:16:36.106936166Z       --metrics.app-suffix string        application suffix. Ignored for certain applications. (default "-release")
2022-05-06T19:16:36.106955074Z       --metrics.instance-prefix string   instance id prefix
2022-05-06T19:16:36.106973407Z       --metrics.interval duration        how frequently to send up telemetry. Ignored for certain applications. (default 1m0s)
2022-05-06T19:16:36.106992258Z       --tracing.agent-addr string        address for jaeger agent (default "agent.tracing.datasci.storj.io:5775")
2022-05-06T19:16:36.107010962Z       --tracing.app string               application name for tracing identification (default "storagenode")
2022-05-06T19:16:36.107030054Z       --tracing.app-suffix string        application suffix (default "-release")
2022-05-06T19:16:36.107048647Z       --tracing.buffer-size int          buffer size for collector batch packet size
2022-05-06T19:16:36.107067165Z       --tracing.enabled                  whether tracing collector is enabled
2022-05-06T19:16:36.107085461Z       --tracing.interval duration        how frequently to flush traces to tracing agent (default 0s)
2022-05-06T19:16:36.107104183Z       --tracing.queue-size int           buffer size for collector queue size
2022-05-06T19:16:36.107122609Z       --tracing.sample float             how frequent to sample traces
2022-05-06T19:16:36.107140442Z
2022-05-06T19:16:36.107434753Z 2022-05-06 19:16:36,106 INFO exited: storagenode (exit status 1; not expected)
2022-05-06T19:16:37.109735828Z 2022-05-06 19:16:37,108 INFO gave up: storagenode entered FATAL state, too many start retries too quickly
2022-05-06T22:06:00.022439499Z 2022-05-06 22:06:00,018 INFO RPC interface 'supervisor' initialized
2022-05-06T22:06:00.023981536Z 2022-05-06 22:06:00,023 CRIT Server 'inet_http_server' running without any HTTP authentication checking
2022-05-06T22:06:00.025248370Z 2022-05-06 22:06:00,024 CRIT could not write pidfile /run/supervisord.pid
2022-05-06T22:06:01.041989814Z 2022-05-06 22:06:01,031 INFO spawned: 'processes' with pid 11
2022-05-06T22:06:01.049962017Z 2022-05-06 22:06:01,045 INFO spawned: 'storagenode' with pid 12
2022-05-06T22:06:01.073197276Z 2022-05-06 22:06:01,070 INFO spawned: 'storagenode-updater' with pid 13
2022-05-06T22:06:01.084403184Z 2022-05-06 22:06:01,081 INFO exited: processes (exit status 0; not expected)
2022-05-06T22:06:01.786283813Z 2022-05-06T22:06:01.785Z INFO Configuration loaded {"Process": "storagenode-updater", "Location": "/app/config/config.yaml"}
2022-05-06T22:06:01.786424646Z 2022-05-06T22:06:01.785Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "contact.external-address"}
2022-05-06T22:06:01.786454628Z 2022-05-06T22:06:01.785Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "storage.allocated-bandwidth"}
2022-05-06T22:06:01.786482091Z 2022-05-06T22:06:01.785Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "operator.wallet"}
2022-05-06T22:06:01.786507554Z 2022-05-06T22:06:01.785Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "server.private-address"}
2022-05-06T22:06:01.786532665Z 2022-05-06T22:06:01.785Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "operator.email"}
2022-05-06T22:06:01.786557295Z 2022-05-06T22:06:01.785Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "server.address"}
2022-05-06T22:06:01.786581480Z 2022-05-06T22:06:01.785Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "storage.allocated-disk-space"}
2022-05-06T22:06:01.786605943Z 2022-05-06T22:06:01.785Z INFO Invalid configuration file value for key {"Process": "storagenode-updater", "Key": "log.encoding"}
2022-05-06T22:06:01.786813109Z 2022-05-06T22:06:01.785Z INFO Invalid configuration file value for key {"Process": "storagenode-updater", "Key": "log.level"}
2022-05-06T22:06:01.786877313Z 2022-05-06T22:06:01.785Z INFO Invalid configuration file value for key {"Process": "storagenode-updater", "Key": "log.output"}
2022-05-06T22:06:01.786905369Z 2022-05-06T22:06:01.785Z INFO Invalid configuration file value for key {"Process": "storagenode-updater", "Key": "log.stack"}
2022-05-06T22:06:01.786931813Z 2022-05-06T22:06:01.785Z INFO Invalid configuration file value for key {"Process": "storagenode-updater", "Key": "log.caller"}
2022-05-06T22:06:01.799568258Z 2022-05-06T22:06:01.798Z INFO Running on version {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.53.1"}
2022-05-06T22:06:01.800417035Z 2022-05-06T22:06:01.799Z INFO Downloading versions. {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2022-05-06T22:06:02.047682980Z 2022-05-06T22:06:02.046Z INFO Configuration loaded {"Process": "storagenode", "Location": "/app/config/config.yaml"}
2022-05-06T22:06:02.047792035Z 2022-05-06 22:06:02,047 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-05-06T22:06:02.048486461Z 2022-05-06 22:06:02,047 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-05-06T22:06:02.050054609Z 2022-05-06T22:06:02.049Z INFO Operator email {"Process": "storagenode", "Address": "eduardo.sequeira.hernandez@gmail.com"}
2022-05-06T22:06:02.050854591Z 2022-05-06T22:06:02.049Z INFO Operator wallet{"Process": "storagenode", "Address": "0x290Eb454a283048ae0378f82bca20266C52eeB0e"}
2022-05-06T22:06:02.177015517Z Error: Error starting master database on storagenode: group:
2022-05-06T22:06:02.177132868Z --- stat config/storage/blobs: permission denied
2022-05-06T22:06:02.177156535Z --- stat config/storage/temp: permission denied
2022-05-06T22:06:02.177177368Z --- stat config/storage/garbage: permission denied
2022-05-06T22:06:02.177197054Z --- stat config/storage/trash: permission denied
2022-05-06T22:06:02.182781331Z 2022-05-06 22:06:02,181 INFO spawned: 'processes' with pid 30
2022-05-06T22:06:02.184730998Z 2022-05-06 22:06:02,184 INFO exited: storagenode (exit status 1; not expected)
2022-05-06T22:06:02.192151054Z 2022-05-06 22:06:02,191 INFO spawned: 'storagenode' with pid 31
2022-05-06T22:06:02.200916683Z 2022-05-06 22:06:02,200 INFO exited: processes (exit status 0; not expected)
2022-05-06T22:06:02.262010702Z 2022-05-06T22:06:02.261Z INFO Current binary version {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.53.1"}
2022-05-06T22:06:02.262112498Z 2022-05-06T22:06:02.261Z INFO New version is being rolled out but hasn't made it to this node yet {"Process": "storagenode-updater", "Service": "storagenode"}
2022-05-06T22:06:02.291151128Z 2022-05-06T22:06:02.290Z INFO Current binary version {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.53.1"}
2022-05-06T22:06:02.291266331Z 2022-05-06T22:06:02.290Z INFO New version is being rolled out but hasn't made it to this node yet {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2022-05-06T22:06:02.327071202Z 2022-05-06T22:06:02.326Z INFO Configuration loaded {"Process": "storagenode", "Location": "/app/config/config.yaml"}
2022-05-06T22:06:02.328756424Z 2022-05-06T22:06:02.328Z INFO Operator email {"Process": "storagenode", "Address": "eduardo.sequeira.hernandez@gmail.com"}
2022-05-06T22:06:02.329324276Z 2022-05-06T22:06:02.328Z INFO Operator wallet{"Process": "storagenode", "Address": "0x290Eb454a283048ae0378f82bca20266C52eeB0e"}
2022-05-06T22:06:02.331606757Z Error: Error starting master database on storagenode: group:
2022-05-06T22:06:02.331705794Z --- stat config/storage/blobs: permission denied
2022-05-06T22:06:02.331728757Z --- stat config/storage/temp: permission denied
2022-05-06T22:06:02.331748461Z --- stat config/storage/garbage: permission denied
2022-05-06T22:06:02.331767479Z --- stat config/storage/trash: permission denied
2022-05-06T22:06:02.335381979Z 2022-05-06 22:06:02,334 INFO exited: storagenode (exit status 1; not expected)
2022-05-06T22:06:03.348394368Z 2022-05-06 22:06:03,346 INFO spawned: 'storagenode' with pid 57
2022-05-06T22:06:03.477294831Z 2022-05-06T22:06:03.476Z INFO Configuration loaded {"Process": "storagenode", "Location": "/app/config/config.yaml"}
2022-05-06T22:06:03.479485090Z 2022-05-06T22:06:03.478Z INFO Operator email {"Process": "storagenode", "Address": "eduardo.sequeira.hernandez@gmail.com"}
2022-05-06T22:06:03.480065608Z 2022-05-06T22:06:03.479Z INFO Operator wallet{"Process": "storagenode", "Address": "0x290Eb454a283048ae0378f82bca20266C52eeB0e"}
2022-05-06T22:06:03.482290997Z Error: Error starting master database on storagenode: group:
2022-05-06T22:06:03.482394108Z --- stat config/storage/blobs: permission denied
2022-05-06T22:06:03.482418331Z --- stat config/storage/temp: permission denied
2022-05-06T22:06:03.482437997Z --- stat config/storage/garbage: permission denied
2022-05-06T22:06:03.482457016Z --- stat config/storage/trash: permission denied
2022-05-06T22:06:03.486099516Z 2022-05-06 22:06:03,485 INFO exited: storagenode (exit status 1; not expected)
2022-05-06T22:06:04.493079793Z 2022-05-06 22:06:04,491 INFO spawned: 'processes' with pid 65
2022-05-06T22:06:04.508278867Z 2022-05-06 22:06:04,507 INFO exited: processes (exit status 0; not expected)
2022-05-06T22:06:05.522083107Z 2022-05-06 22:06:05,520 INFO spawned: 'storagenode' with pid 66
2022-05-06T22:06:05.654036681Z 2022-05-06T22:06:05.653Z INFO Configuration loaded {"Process": "storagenode", "Location": "/app/config/config.yaml"}
2022-05-06T22:06:05.655503737Z 2022-05-06T22:06:05.654Z INFO Operator email {"Process": "storagenode", "Address": "eduardo.sequeira.hernandez@gmail.com"}
2022-05-06T22:06:05.656032181Z 2022-05-06T22:06:05.655Z INFO Operator wallet{"Process": "storagenode", "Address": "0x290Eb454a283048ae0378f82bca20266C52eeB0e"}
2022-05-06T22:06:05.658802866Z Error: Error starting master database on storagenode: group:
2022-05-06T22:06:05.658898422Z --- stat config/storage/blobs: permission denied
2022-05-06T22:06:05.658922348Z --- stat config/storage/temp: permission denied
2022-05-06T22:06:05.658943940Z --- stat config/storage/garbage: permission denied
2022-05-06T22:06:05.658965663Z --- stat config/storage/trash: permission denied
2022-05-06T22:06:05.663351403Z 2022-05-06 22:06:05,662 INFO exited: storagenode (exit status 1; not expected)
2022-05-06T22:06:07.674113013Z 2022-05-06 22:06:07,671 INFO spawned: 'processes' with pid 74
2022-05-06T22:06:07.692320958Z 2022-05-06 22:06:07,691 INFO exited: processes (exit status 0; not expected)
2022-05-06T22:06:07.693109384Z 2022-05-06 22:06:07,692 INFO gave up: processes entered FATAL state, too many start retries too quickly
2022-05-06T22:06:08.706211883Z 2022-05-06 22:06:08,704 INFO spawned: 'storagenode' with pid 75
2022-05-06T22:06:08.836707013Z 2022-05-06T22:06:08.835Z INFO Configuration loaded {"Process": "storagenode", "Location": "/app/config/config.yaml"}
2022-05-06T22:06:08.838998920Z 2022-05-06T22:06:08.838Z INFO Operator email {"Process": "storagenode", "Address": "eduardo.sequeira.hernandez@gmail.com"}
2022-05-06T22:06:08.839779494Z 2022-05-06T22:06:08.838Z INFO Operator wallet{"Process": "storagenode", "Address": "0x290Eb454a283048ae0378f82bca20266C52eeB0e"}
2022-05-06T22:06:08.842047698Z Error: Error starting master database on storagenode: group:
2022-05-06T22:06:08.842160457Z --- stat config/storage/blobs: permission denied
2022-05-06T22:06:08.842187124Z --- stat config/storage/temp: permission denied
2022-05-06T22:06:08.842210679Z --- stat config/storage/garbage: permission denied
2022-05-06T22:06:08.842232790Z --- stat config/storage/trash: permission denied
2022-05-06T22:06:08.846275976Z 2022-05-06 22:06:08,845 INFO exited: storagenode (exit status 1; not expected)
2022-05-06T22:06:09.849003531Z 2022-05-06 22:06:09,848 INFO gave up: storagenode entered FATAL state, too many start retries too quickly```

Thanks
Edu

This is a picture how my HD

You have not fixed a permission issue. The node still doesn’t have an access to data.
If you used --user $(id -u):$(id -g) option in your docker run command, you need to change permissions to data location to your user instead of root and add your user to the docker group.
To apply the changes you need to re-login. After that you can run the node.

You can also remove the --user $(id -u):$(id -g) option and use sudo docker run.... as before, this is less secure, but doesn’t require any changes. However, you said that you fixed permissions, so, I can assume that you applied something, but not finished.

Please show your current docker run command (you can replace personal information with placeholders)

1 Like

Hi @Alexey I think my config.yaml are corrupted or something when I run the command ls -l there are no files in the mounting directory.

My docker run command is the following:

-p 28967:28967/tcp \
-p 28967:28967/udp \
-p 127.0.0.1:14002:14002 \
-e WALLET=0x290Eb454a283048ae0378f82bca20266C52eeB0e \
-e EMAIL=ssss.xxxx.cccc@gmail.com \
-e ADDRESS=xxxxxx.ddns.net:28967 \
-e STORAGE=6TB \
--mount type=bind,source=/home/edu/.local/share/storj/identity/storagenode,destination=/app/identity \
--mount type=bind,source=/mnt/hd1/,destination=/app/config \
--name storagenode storjlabs/storagenode:latest

Please copy result of the command:

ls -l /mnt/hd1/
ls -l /mnt/hd1/storage
1 Like

I updated my OS and restarted my PI and apparently that worked. I don’t know why but my node is up and running.

Hi Alexey, I am facing the same issues, so posting the below for your reference. I have only one node that with an external that is still visible when I lsblk.

Blockquote
total 860865
-rwxrwxrwx 1 root root 38727680 Mar 25 14:42 bandwidth.db
-rwxrwxrwx 1 root root 65536 Mar 25 23:16 bandwidth.db-shm
-rwxrwxrwx 1 root root 24777712 Mar 25 14:42 bandwidth.db-wal
drwxrwxrwx 1 root root 4096 Apr 17 2021 blobs
drwxrwxrwx 1 root root 4096 Mar 25 10:01 garbage
-rwxrwxrwx 1 root root 90112 Mar 25 14:14 heldamount.db
-rwxrwxrwx 1 root root 16384 Mar 25 14:14 info.db
-rwxrwxrwx 1 root root 32768 Mar 25 23:16 info.db-shm
-rwxrwxrwx 1 root root 0 Mar 25 23:16 info.db-wal
-rwxrwxrwx 1 root root 24576 Mar 25 14:14 notifications.db
-rwxrwxrwx 1 root root 32768 Mar 25 14:14 orders.db
-rwxrwxrwx 1 root root 2199552 Mar 25 14:14 piece_expiration.db
-rwxrwxrwx 1 root root 24576 Mar 25 14:14 pieceinfo.db
-rwxrwxrwx 1 root root 24576 Mar 25 14:14 piece_spaced_used.db
-rwxrwxrwx 1 root root 24576 Mar 25 14:14 pricing.db
-rwxrwxrwx 1 root root 36864 Mar 25 14:14 reputation.db
-rwxrwxrwx 1 root root 32768 Mar 25 14:14 satellites.db
-rwxrwxrwx 1 root root 24576 Mar 25 14:14 secret.db
-rwxrwxrwx 1 root root 32 May 25 2022 storage-dir-verification
-rwxrwxrwx 1 root root 1044480 Mar 25 14:14 storage_usage.db
drwxrwxrwx 1 root root 814313472 Mar 25 14:42 temp
drwxrwxrwx 1 root root 4096 May 5 2021 trash
-rwxrwxrwx 1 root root 20480 Mar 25 14:14 used_serial.db
-rwxrwxrwx 1 root root 0 Nov 12 06:55 write-test1109815670
-rwxrwxrwx 1 root root 0 Nov 10 07:52 write-test1120282404
-rwxrwxrwx 1 root root 0 Mar 22 14:33 write-test1188848786
-rwxrwxrwx 1 root root 0 Nov 3 21:26 write-test1259268418
-rwxrwxrwx 1 root root 0 Jan 4 2022 write-test126884359
-rwxrwxrwx 1 root root 0 Nov 6 06:17 write-test1300619108
-rwxrwxrwx 1 root root 0 Nov 18 14:54 write-test1325884544
-rwxrwxrwx 1 root root 0 Mar 14 15:36 write-test1329997340
-rwxrwxrwx 1 root root 0 Mar 4 12:05 write-test1396576487
-rwxrwxrwx 1 root root 0 Nov 22 19:09 write-test1452929591
-rwxrwxrwx 1 root root 0 Nov 5 11:18 write-test1620098429
-rwxrwxrwx 1 root root 0 Jan 31 07:48 write-test1622734390
-rwxrwxrwx 1 root root 0 Jan 24 08:13 write-test1622854292
-rwxrwxrwx 1 root root 0 Mar 4 18:57 write-test1662442217
-rwxrwxrwx 1 root root 0 Nov 6 16:53 write-test1664954349
-rwxrwxrwx 1 root root 0 Feb 24 06:57 write-test172217338
-rwxrwxrwx 1 root root 0 Oct 1 13:12 write-test1826623611
-rwxrwxrwx 1 root root 0 Nov 21 17:01 write-test1846506755
-rwxrwxrwx 1 root root 0 Nov 17 15:50 write-test1963002738
-rwxrwxrwx 1 root root 0 Nov 22 15:34 write-test2041625895
-rwxrwxrwx 1 root root 0 Nov 18 00:12 write-test2067139619
-rwxrwxrwx 1 root root 0 Feb 24 07:44 write-test2132644136
-rwxrwxrwx 1 root root 0 Nov 18 15:38 write-test2150397856
-rwxrwxrwx 1 root root 0 Sep 29 2021 write-test225354694
-rwxrwxrwx 1 root root 0 Nov 22 08:01 write-test2288378928
-rwxrwxrwx 1 root root 0 Oct 25 2021 write-test2341921769
-rwxrwxrwx 1 root root 0 Nov 8 19:46 write-test2431708074
-rwxrwxrwx 1 root root 0 Feb 23 15:36 write-test2530229782
-rwxrwxrwx 1 root root 0 Nov 22 03:48 write-test2580951013
-rwxrwxrwx 1 root root 0 Feb 23 20:30 write-test2641540002
-rwxrwxrwx 1 root root 0 Nov 6 08:03 write-test2782286531
-rwxrwxrwx 1 root root 0 Nov 10 01:16 write-test2784524424
-rwxrwxrwx 1 root root 0 Jan 19 10:49 write-test2937843095
-rwxrwxrwx 1 root root 0 Dec 16 13:23 write-test2980425123
-rwxrwxrwx 1 root root 0 Mar 15 13:45 write-test322744172
-rwxrwxrwx 1 root root 0 Feb 23 07:54 write-test3234672473
-rwxrwxrwx 1 root root 0 Nov 19 03:45 write-test3244131442
-rwxrwxrwx 1 root root 0 Jan 14 05:47 write-test333770945
-rwxrwxrwx 1 root root 0 Nov 22 12:14 write-test3524899648
-rwxrwxrwx 1 root root 0 Nov 3 13:31 write-test3658476246
-rwxrwxrwx 1 root root 0 Nov 11 14:55 write-test3773996549
-rwxrwxrwx 1 root root 0 Oct 21 04:39 write-test3804177396
-rwxrwxrwx 1 root root 0 Jun 12 2022 write-test3828108486
-rwxrwxrwx 1 root root 0 Nov 9 08:45 write-test3881567811
-rwxrwxrwx 1 root root 0 Nov 9 17:08 write-test3882311010
-rwxrwxrwx 1 root root 0 Nov 8 22:29 write-test3961071732
-rwxrwxrwx 1 root root 0 Oct 7 2021 write-test439354448
-rwxrwxrwx 1 root root 0 Mar 18 16:05 write-test47658094
-rwxrwxrwx 1 root root 0 Feb 7 12:14 write-test578069253
-rwxrwxrwx 1 root root 0 Nov 9 10:51 write-test65882175
-rwxrwxrwx 1 root root 0 Nov 17 09:48 write-test712073789
-rwxrwxrwx 1 root root 0 Oct 1 04:05 write-test747837926
-rwxrwxrwx 1 root root 0 Sep 6 2021 write-test895010751
-rwxrwxrwx 1 root root 0 Mar 1 16:55 write-test962775448
-rwxrwxrwx 1 root root 0 Sep 18 2021 write-test964806808

Can you provide the last ten or so lines of your logs?

docker logs --tail 20 storagenode

2023-03-26 03:17:52,362 INFO RPC interface 'supervisor' initialized
2023-03-26 03:17:52,363 INFO supervisord started with pid 1
2023-03-26 03:17:53,371 INFO spawned: 'processes-exit-eventlistener' with pid 12
2023-03-26 03:17:53,385 INFO spawned: 'storagenode' with pid 13
2023-03-26 03:17:53,397 INFO spawned: 'storagenode-updater' with pid 14
2023-03-26T03:17:53.467Z	INFO	Configuration loaded	{"Process": "storagenode-updater", "Location": "/app/config/config.yaml"}
2023-03-26T03:17:53.467Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "operator.email"}
2023-03-26T03:17:53.467Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "server.private-address"}
2023-03-26T03:17:53.468Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "operator.wallet-features"}
2023-03-26T03:17:53.468Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "operator.wallet"}
2023-03-26T03:17:53.468Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "storage.allocated-disk-space"}
2023-03-26T03:17:53.468Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "server.address"}
2023-03-26T03:17:53.468Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "console.address"}
2023-03-26T03:17:53.468Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "storage.allocated-bandwidth"}
2023-03-26T03:17:53.468Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "contact.external-address"}
2023-03-26T03:17:53.468Z	INFO	Invalid configuration file value for key	{"Process": "storagenode-updater", "Key": "log.level"}
2023-03-26T03:17:53.468Z	INFO	Invalid configuration file value for key	{"Process": "storagenode-updater", "Key": "log.output"}
2023-03-26T03:17:53.470Z	INFO	Anonymized tracing enabled	{"Process": "storagenode-updater"}
2023-03-26T03:17:53.508Z	INFO	Running on version	{"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.75.2"}
2023-03-26T03:17:53.509Z	INFO	Downloading versions.	{"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}

Here is another.

docker logs --tail 20 storagenode
2023-03-26 03:18:10,294 INFO spawned: 'storagenode' with pid 46
2023-03-26T03:18:10.495Z	INFO	Configuration loaded	{"Process": "storagenode", "Location": "/app/config/config.yaml"}
2023-03-26T03:18:10.496Z	INFO	Anonymized tracing enabled	{"Process": "storagenode"}
2023-03-26T03:18:10.519Z	INFO	Operator email	{"Process": "storagenode", "Address": "myemail@gmail.com"}
2023-03-26T03:18:10.519Z	INFO	Operator wallet	{"Process": "storagenode", "Address": "0x00-> I removed my address"}
Error: Error starting master database on storagenode: database: bandwidth opening file "config/storage/bandwidth.db" failed: disk I/O error: input/output error
	storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:331
	storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:308
	storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:283
	storj.io/storj/storagenode/storagenodedb.OpenExisting:250
	main.cmdRun:193
	storj.io/private/process.cleanup.func1.4:377
	storj.io/private/process.cleanup.func1:395
	github.com/spf13/cobra.(*Command).execute:852
	github.com/spf13/cobra.(*Command).ExecuteC:960
	github.com/spf13/cobra.(*Command).Execute:897
	storj.io/private/process.ExecWithCustomConfigAndLogger:92
	main.main:478
	runtime.main:250
2023-03-26 03:18:10,740 INFO exited: storagenode (exit status 1; not expected)