Rpc: dial tcp 127.0.0.1:7778: connect: connection refused

Alexey · March 26, 2023, 4:38am

This suggest that your filesystem is corrupted.
Stop and remove the container, unmount this disk and check it with fsck. After that mount it back.

If you did setup your docker to be able to run without sudo and you have used --user $(id -u):$(id -g) option in your docker run command, you need to change permissions to data location to your user instead of root (assuming that your data location is /mnt/storj/storagenode):

chown $(id -u):$(id -g) -R /mnt/storj/storagenode

and run your container back.

If you still want to run docker with sudo instead, then you do not need to change permissions in the data location, but you also should not use --user $(id -u):$(id -g) option in your sudo docker run command.

samitjaiswal · March 26, 2023, 3:38pm

Look, I did this. Yet, I am having a hard time connecting my node

Alexey · March 27, 2023, 3:06am

Does this drive have an external power supply? If not - you need to do that, or at least use a more powerful AC/DC adapter for your raspi, especially if you use any other USB devices. However an additional power supply for the drive is better.

Please show result of these commands:

ls -l /mnt/mydisk_seagate/
ls -l /mnt/mydisk_seagate/storage/

samitjaiswal · March 27, 2023, 3:25am

Yes, I am using an external separate power supply for both Pi and Drive.

ls -l /mnt/mydisk_seagate/
total 36
drwxrwxrwx 1 root root     0 Apr 11  2021  orders
-rwxrwxrwx 1 root root 32768 Apr 16  2021  revocations.db
drwxrwxrwx 1 root root  8192 Mar 26 21:27  storage
drwxrwxrwx 1 root root     0 Apr  7  2021 'System Volume Information'
-rwxrwxrwx 1 root root  1377 Apr 16  2021  trust-cache.json

ls -l /mnt/mydisk_seagate/storage/
total 424
-rwxrwxrwx 1 root root 32768 Apr 16  2021 bandwidth.db
drwxrwxrwx 1 root root     0 Apr 11  2021 blobs
-rwxrwxrwx 1 root root  9337 Mar 26 01:25 config.yaml
drwxrwxrwx 1 root root     0 Apr 11  2021 garbage
-rwxrwxrwx 1 root root 32768 Apr 16  2021 heldamount.db
-rwxrwxrwx 1 root root 16384 Apr 16  2021 info.db
-rwxrwxrwx 1 root root 24576 Apr 16  2021 notifications.db
drwxrwxrwx 1 root root     0 Apr 16  2021 orders
-rwxrwxrwx 1 root root 32768 Apr 16  2021 orders.db
-rwxrwxrwx 1 root root 36864 Apr 16  2021 piece_expiration.db
-rwxrwxrwx 1 root root 24576 Apr 16  2021 pieceinfo.db
-rwxrwxrwx 1 root root 24576 Apr 16  2021 piece_spaced_used.db
-rwxrwxrwx 1 root root 24576 Apr 16  2021 pricing.db
-rwxrwxrwx 1 root root 24576 Apr 16  2021 reputation.db
-rwxrwxrwx 1 root root 32768 Mar 25 13:44 revocations.db
-rwxrwxrwx 1 root root 32768 Apr 16  2021 satellites.db
-rwxrwxrwx 1 root root 24576 Apr 16  2021 secret.db
drwxrwxrwx 1 root root 20480 Mar 27 08:52 storage
-rwxrwxrwx 1 root root 20480 Apr 16  2021 storage_usage.db
drwxrwxrwx 1 root root     0 Apr 11  2021 temp
drwxrwxrwx 1 root root     0 Apr 11  2021 trash
-rwxrwxrwx 1 root root  2730 Mar 25 13:44 trust-cache.json
-rwxrwxrwx 1 root root     0 Nov  6 02:13 trust-cache.json1674622673
-rwxrwxrwx 1 root root     0 Nov 17 09:13 trust-cache.json1715502711
-rwxrwxrwx 1 root root     0 Nov  9 01:57 trust-cache.json3030368690
-rwxrwxrwx 1 root root     0 Nov  3 10:08 trust-cache.json3399177787
-rwxrwxrwx 1 root root 20480 Apr 16  2021 used_serial.db

Alexey · March 27, 2023, 3:39am

Seems you have used docker commands with sudo, so you need either remove --user $(id -u):$(id -g) from your docker run command or configure your docker to do not require sudo and change an owner, as I mentioned before.

Since it’s an NTFS disk, you need to stop and remove the container, unmount this drive and plug-in to your Windows PC and run chkdsk /f d:\ command (assuming that it would be assigned to the D: letter). You may be forced to run it several times until all errors will be fixed. After that safe eject it, unplug and plug to raspi again and mount it, then try to run your full docker run command.

I would highly recommend to backup all data from this drive and format it to ext4 filesystem on your raspi, then restore data back: NTFS is a foreign filesystem for Linux and it working much slower and damages data.

samitjaiswal · March 27, 2023, 11:36am

Hi Alexey, This worked. I am greatly thankful for your assistance. One last query, my node was offline for more than 48 hours, will this now be disqualified eventually?

Stob · March 27, 2023, 3:07pm

No. Keep the node online and it will recover the Online score. Current rules give SNO’s up to 12 days downtime per month before going into suspension, and then after 30days if the Online score stays below 60% then your node could be disqualified.

samitjaiswal · April 4, 2023, 6:54pm

I am getting this connection refused signal again. I did the “CHKDSK /f D:” to my drive this time. But after several attempts, I am not able to connect.

ls -l /mnt/mydisk_seagate/
total 40
drwxrwxrwx 1 root root     0 Apr 11  2021  orders
-rwxrwxrwx 1 root root 32768 Apr 16  2021  revocations.db
drwxrwxrwx 1 root root  8192 Apr  2 17:14  storage
drwxrwxrwx 1 root root  4096 Apr  4 12:03 'System Volume Information'
-rwxrwxrwx 1 root root  1377 Apr 16  2021  trust-cache.json

And One more,

ls -l /mnt/mydisk_seagate/storage/
total 392
-rwxrwxrwx 1 root root 32768 Apr 16  2021 bandwidth.db
drwxrwxrwx 1 root root     0 Apr 11  2021 blobs
-rwxrwxrwx 1 root root  9337 Mar 26 01:25 config.yaml
drwxrwxrwx 1 root root     0 Apr 11  2021 garbage
-rwxrwxrwx 1 root root 32768 Apr 16  2021 heldamount.db
-rwxrwxrwx 1 root root 16384 Apr 16  2021 info.db
-rwxrwxrwx 1 root root 24576 Apr 16  2021 notifications.db
drwxrwxrwx 1 root root     0 Apr 16  2021 orders
-rwxrwxrwx 1 root root 36864 Apr 16  2021 piece_expiration.db
-rwxrwxrwx 1 root root 24576 Apr 16  2021 pieceinfo.db
-rwxrwxrwx 1 root root 24576 Apr 16  2021 piece_spaced_used.db
-rwxrwxrwx 1 root root 24576 Apr 16  2021 pricing.db
-rwxrwxrwx 1 root root 24576 Apr 16  2021 reputation.db
-rwxrwxrwx 1 root root 32768 Apr  5 00:24 revocations.db
-rwxrwxrwx 1 root root 32768 Apr 16  2021 satellites.db
-rwxrwxrwx 1 root root 24576 Apr 16  2021 secret.db
drwxrwxrwx 1 root root 20480 Apr  5 00:24 storage
-rwxrwxrwx 1 root root 20480 Apr 16  2021 storage_usage.db
drwxrwxrwx 1 root root     0 Apr 11  2021 temp
drwxrwxrwx 1 root root     0 Apr 11  2021 trash
-rwxrwxrwx 1 root root  2730 Apr  2 17:14 trust-cache.json
-rwxrwxrwx 1 root root     0 Nov  6 02:13 trust-cache.json1674622673
-rwxrwxrwx 1 root root     0 Nov 17 09:13 trust-cache.json1715502711
-rwxrwxrwx 1 root root     0 Nov  9 01:57 trust-cache.json3030368690
-rwxrwxrwx 1 root root     0 Nov  3 10:08 trust-cache.json3399177787
-rwxrwxrwx 1 root root 20480 Apr 16  2021 used_serial.db

Alexey · April 5, 2023, 3:57am

when you get this response on attempt to open a CLI dashboard, you usually need to check the node’s logs like

docker logs --tail 10 storagenode 2>&1 | grep "ERROR"

The FATAL ERROR there would indicate a reason why your node is stopped.

samitjaiswal · April 5, 2023, 4:14am

I did that

pi@pi-sam:~ $ docker logs --tail 10 storagenode 2>&1 | grep "ERROR"
pi@pi-sam:~ $ docker logs --tail 20 storagenode
	storj.io/private/process.cleanup.func1.4:377
	storj.io/private/process.cleanup.func1:395
	github.com/spf13/cobra.(*Command).execute:852
	github.com/spf13/cobra.(*Command).ExecuteC:960
	github.com/spf13/cobra.(*Command).Execute:897
	storj.io/private/process.ExecWithCustomConfigAndLogger:92
	main.main:478
	runtime.main:250
2023-04-05 04:10:33,072 INFO exited: storagenode (exit status 1; not expected)
2023-04-05 04:10:34,085 INFO spawned: 'storagenode' with pid 47
2023-04-05 04:10:34,088 WARN received SIGQUIT indicating exit request
2023-04-05 04:10:34,090 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2023-04-05T04:10:34.091Z	INFO	Got a signal from the OS: "terminated"	{"Process": "storagenode-updater"}
2023-04-05 04:10:34,096 INFO stopped: storagenode-updater (exit status 0)
2023-04-05T04:10:34.284Z	INFO	Configuration loaded	{"Process": "storagenode", "Location": "/app/config/config.yaml"}
2023-04-05T04:10:34.285Z	INFO	Anonymized tracing enabled	{"Process": "storagenode"}
2023-04-05T04:10:34.307Z	INFO	Operator email	{"Process": "storagenode", "Address": "MYEMAIL@GMAIL.COM"}
2023-04-05T04:10:34.308Z	INFO	Operator wallet	{"Process": "storagenode", "Address": "MYWALLETADDRESS"}
2023-04-05T04:10:34.797Z	INFO	server	kernel support for server-side tcp fast open remains disabled.	{"Process": "storagenode"}
2023-04-05T04:10:34.797Z	INFO	server	enable with: sysctl -w net.ipv4.tcp_fastopen=3	{"Process": "storagenode"}

Alexey · April 5, 2023, 8:16am

need to see a beginning of this error
try

docker logs storagenode 2>&1 | grep "FATAL ERROR" | tail

samitjaiswal · April 5, 2023, 12:18pm

Here

pi@pi-sam:~ $ docker logs storagenode 2>&1 | grep "FATAL ERROR" | tail
pi@pi-sam:~ $ docker logs --tail 20 storagenode
2023-04-05T12:16:43.570Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "console.address"}
2023-04-05T12:16:43.570Z	INFO	Invalid configuration file value for key	{"Process": "storagenode-updater", "Key": "log.level"}
2023-04-05T12:16:43.570Z	INFO	Invalid configuration file value for key	{"Process": "storagenode-updater", "Key": "log.stack"}
2023-04-05T12:16:43.570Z	INFO	Invalid configuration file value for key	{"Process": "storagenode-updater", "Key": "log.development"}
2023-04-05T12:16:43.571Z	INFO	Anonymized tracing enabled	{"Process": "storagenode-updater"}
2023-04-05T12:16:43.594Z	INFO	Running on version	{"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.75.2"}
2023-04-05T12:16:43.595Z	INFO	Downloading versions.	{"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2023-04-05T12:16:43.668Z	INFO	Configuration loaded	{"Process": "storagenode", "Location": "/app/config/config.yaml"}
2023-04-05T12:16:43.669Z	INFO	Anonymized tracing enabled	{"Process": "storagenode"}
2023-04-05T12:16:43.691Z	INFO	Operator email	{"Process": "storagenode", "Address": "MYEMAIL@GMAIL.COM"}
2023-04-05T12:16:43.691Z	INFO	Operator wallet	{"Process": "storagenode", "Address": "MYADDRESS"}
2023-04-05T12:16:44.567Z	INFO	Current binary version	{"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.75.2"}
2023-04-05T12:16:44.567Z	INFO	New version is being rolled out but hasn't made it to this node yet	{"Process": "storagenode-updater", "Service": "storagenode"}
2023-04-05 12:16:44,568 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-04-05 12:16:44,568 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-04-05 12:16:44,569 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-04-05T12:16:44.594Z	INFO	Current binary version	{"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.75.2"}
2023-04-05T12:16:44.594Z	INFO	New version is being rolled out but hasn't made it to this node yet	{"Process": "storagenode-updater", "Service": "storagenode-updater"}
2023-04-05T12:16:44.915Z	INFO	server	kernel support for server-side tcp fast open remains disabled.	{"Process": "storagenode"}
2023-04-05T12:16:44.916Z	INFO	server	enable with: sysctl -w net.ipv4.tcp_fastopen=3	{"Process": "storagenode"}

Alexey · April 6, 2023, 3:18am

Then please try to search for any errors:

docker logs storagenode 2>&1 | grep -i "error" | tail

samitjaiswal · April 6, 2023, 7:54am

docker logs storagenode 2>&1 | grep -i "error" | tail
Error: Error during preflight check for storagenode databases: preflight: database "bandwidth": expected schema does not match actual:   &dbschema.Schema{
Error: Error during preflight check for storagenode databases: preflight: database "bandwidth": expected schema does not match actual:   &dbschema.Schema{
Error: Error during preflight check for storagenode databases: preflight: database "bandwidth": expected schema does not match actual:   &dbschema.Schema{
Error: Error during preflight check for storagenode databases: preflight: database "bandwidth": expected schema does not match actual:   &dbschema.Schema{
Error: Error during preflight check for storagenode databases: preflight: database "bandwidth": expected schema does not match actual:   &dbschema.Schema{
Error: Error during preflight check for storagenode databases: preflight: database "bandwidth": expected schema does not match actual:   &dbschema.Schema{
Error: Error during preflight check for storagenode databases: preflight: database "bandwidth": expected schema does not match actual:   &dbschema.Schema{
Error: Error during preflight check for storagenode databases: preflight: database "bandwidth": expected schema does not match actual:   &dbschema.Schema{
Error: Error during preflight check for storagenode databases: preflight: database "bandwidth": expected schema does not match actual:   &dbschema.Schema{
Error: Error during preflight check for storagenode databases: preflight: database "bandwidth": expected schema does not match actual:   &dbschema.Schema{

Below is the latest attempt with dock run

pi@pi-sam:~ $ docker logs storagenode 2>&1 | grep -i "error" | tail
Error: Error during preflight check for storagenode databases: preflight: database "bandwidth": expected schema does not match actual:   &dbschema.Schema{
Error: Error during preflight check for storagenode databases: preflight: database "bandwidth": expected schema does not match actual:   &dbschema.Schema{
Error: Error during preflight check for storagenode databases: preflight: database "bandwidth": expected schema does not match actual:   &dbschema.Schema{
Error: Error during preflight check for storagenode databases: preflight: database "bandwidth": expected schema does not match actual:   &dbschema.Schema{
Error: Error during preflight check for storagenode databases: preflight: database "bandwidth": expected schema does not match actual:   &dbschema.Schema{
Error: Error during preflight check for storagenode databases: preflight: database "bandwidth": expected schema does not match actual:   &dbschema.Schema{

Alexey · April 6, 2023, 8:40am

I see. The database bandwidth.db become corrupted.
You need to recreate it:

After that I would recommend to check all other databases and fix them: