Node dashboard won't load. Setup failure?

@peem

that worked…

>     --name storagenode storjlabs/storagenode:latest
27965d6a84f5d5a3c3728d8

@peem

dashboard still not loading.

Here is the log file

2022-10-30 14:00:11,401 INFO exited: storagenode (exit status 1; not expected)
2022-10-30 14:00:13,405 INFO spawned: 'storagenode' with pid 72
2022-10-30T14:00:13.437Z	INFO	Anonymized tracing enabled	{"Process": "storagenode"}
2022-10-30T14:00:13.449Z	INFO	Operator email	{"Process": "storagenode", "Address": "(deleted)"}
2022-10-30T14:00:13.449Z	INFO	Operator wallet	{"Process": "storagenode", "Address": "(deleted)"}
Error: trust: open config/trust-cache.json: permission denied
2022-10-30 14:00:13,504 INFO exited: storagenode (exit status 1; not expected)

“permission denied”

ls -la /mnt/STORJ/storagenode/data

@peem

total 44
drwxrwxrwx 4 randall randall  4096 Oct 30 09:31 .
drwxrwxr-x 5 randall randall  4096 Oct 16 19:52 ..
drwx------ 4 randall randall  4096 Oct 29 16:17 orders
-rw------- 1 randall randall 32768 Oct 30 10:14 revocations.db
drwx------ 7 randall randall  4096 Oct 30 10:14 storage
-rw------- 1 root    root     1374 Oct 30 09:31 trust-cache.json

you have corrected:

–mount type=bind,source=“/mnt/STORJ/storagenode/data/storage”,destination=/app/config \

@peem

I just made the change.

Stopped the node and removed the container:

sudo docker stop -t 300 storagenode

AND THEN…

sudo docker rm storagenode

I then reran the docker command with the changes you just gave…

The dashboard still will not load

@peem @Alexey

latest log file

Error: Error starting master database on storagenode: group:
--- stat config/storage/blobs: no such file or directory
--- stat config/storage/temp: no such file or directory
--- stat config/storage/garbage: no such file or directory
--- stat config/storage/trash: no such file or directory
2022-10-30 14:19:46,126 INFO exited: storagenode (exit status 1; not expected)
2022-10-30 14:19:47,127 INFO gave up: storagenode entered FATAL state, too many start retries too quickly
2022-10-30 14:19:48,129 WARN received SIGQUIT indicating exit request
2022-10-30 14:19:48,129 INFO waiting for processes-exit-eventlistener, storagenode-updater to die
2022-10-30T14:19:48.129Z	INFO	Got a signal from the OS: "terminated"	{"Process": "storagenode-updater"}
2022-10-30 14:19:48,131 INFO stopped: storagenode-updater (exit status 0)
2022-10-30 14:19:49,132 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)
2022-10-30 14:19:49,878 INFO RPC interface 'supervisor' initialized
2022-10-30 14:19:49,878 INFO supervisord started with pid 1

then:

ls -la /mnt/STORJ/storagenode/data/storage

@peem

otal 51532
drwx------ 7 randall randall     4096 Oct 30 10:16 .
drwxrwxrwx 4 randall randall     4096 Oct 30 09:31 ..
-rwxrwxrwx 1 randall randall 37732352 Oct 30 09:39 bandwidth.db
drwxrwxrwx 8 randall randall     4096 Apr 26  2021 blobs
-rw------- 1 randall randall     9705 Oct 26 21:45 config.yaml
drwxrwxrwx 2 randall randall     4096 Oct 30 09:32 garbage
-rwxrwxrwx 1 randall randall    77824 Oct 30 09:39 heldamount.db
-rwxrwxrwx 1 randall randall    16384 Oct 30 09:39 info.db
-rwxrwxrwx 1 randall randall    24576 Oct 30 09:39 notifications.db
drwx------ 4 randall randall     4096 Oct 26 21:52 orders
-rwxrwxrwx 1 randall randall    32768 Oct 30 09:39 orders.db
-rwxrwxrwx 1 randall randall 13516800 Oct 30 09:39 piece_expiration.db
-rwxrwxrwx 1 randall randall    24576 Oct 30 09:39 pieceinfo.db
-rwxrwxrwx 1 randall randall    24576 Oct 30 09:39 piece_spaced_used.db
-rwxrwxrwx 1 randall randall    24576 Oct 30 09:39 pricing.db
-rwxrwxrwx 1 randall randall    36864 Oct 30 09:39 reputation.db
-rw------- 1 randall randall    32768 Oct 29 16:10 revocations.db
-rwxrwxrwx 1 randall randall    32768 Oct 30 09:39 satellites.db
-rwxrwxrwx 1 randall randall    24576 Oct 30 09:39 secret.db
-rwxrwxrwx 1 randall randall       32 Apr 24  2021 storage-dir-verification
-rwxrwxrwx 1 randall randall  1032192 Oct 30 09:39 storage_usage.db
drwxrwxrwx 2 randall randall    69632 Oct 23 23:25 temp
drwxrwxrwx 8 randall randall     4096 Apr 27  2021 trash
-rw------- 1 randall randall     1374 Oct 29 16:10 trust-cache.json
-rwxrwxrwx 1 randall randall    20480 Oct 30 09:39 used_serial.db

Stop the container, remove it, log out of the system, log in, start the container…

I have to leave, @Alexey look at these data paths and permissions

@peem @Alexey

I did that and it didn’t solve the problem

It is the path that is supposed to be:

--mount type=bind,source="/mnt/STORJ/storagenode/data",destination=/app/config \

Before doing so, you need to swap the rights for the file:

sudo chown randall:randall /mnt/STORJ/storagenode/data/trust-cache.json

Because:

Error: trust: open config/trust-cache.json: permission denied

and

-rw------- 1 root    root     1374 Oct 30 09:31 trust-cache.json
1 Like

@peem @Alexey

Okay. I ran “sudo Chown” and fixed the command line back to how it should be.

I’m getting the dashboard again but it’s saying offline

Here is the log

2022-10-30T18:37:24.613Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "attempts": 7, "error": "ping satellite: failed to dial storage node (ID: 12q9SBbHrkRKDECe86XNTRq1YSt99f4RZqFXZZHsiYuUZpEWdy5) at address (deleted):28967: rpc: tcp connector failed: rpc: dial tcp 73.144.175.37:28967: connect: connection refused", "errorVerbose": "ping satellite: failed to dial storage node (ID: 12q9SBbHrkRKDECe86XNTRq1YSt99f4RZqFXZZHsiYuUZpEWdy5) at address (deleted):28967: rpc: tcp connector failed: rpc: dial tcp 73.144.175.37:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:98\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2022-10-30T18:37:27.537Z	ERROR	contact:service	ping satellite failed 	{"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 7, "error": "ping satellite: failed to dial storage node (ID: 12q9SBbHrkRKDECe86XNTRq1YSt99f4RZqFXZZHsiYuUZpEWdy5) at address (deleted):28967: rpc: tcp connector failed: rpc: dial tcp 73.144.175.37:28967: connect: connection refused", "errorVerbose": "ping satellite: failed to dial storage node (ID: 12q9SBbHrkRKDECe86XNTRq1YSt99f4RZqFXZZHsiYuUZpEWdy5) at address (deleted):28967: rpc: tcp connector failed: rpc: dial tcp 73.144.175.37:28967: connect: connection refused\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:139\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:98\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

Besides, you don’t have

config.yaml

in the folder

/mnt/STORJ/storagenode/data

and it must be there

(You have it in /mnt/STORJ/storagenode/data/storage but I don’t know if the correct one…)

@peem

Well I’ll be damned…

That worked! We are online and OK!!!

Thank you kindly for your patience and assistance.

1 Like

I was very pleased to hear this, good luck with the running of SN, regards :slight_smile:

2 Likes
I have a similar issue:

2023-04-28T12:57:15.972Z	WARN	piecestore:monitor	Disk space is less than requested. Allocated space is	{"Process": "storagenode", "bytes": 115851190272}
2023-04-28T12:57:15.972Z	ERROR	piecestore:monitor	Total disk space is less than required minimum	{"Process": "storagenode", "bytes": 500000000000}
2023-04-28T12:57:15.972Z	ERROR	services	unexpected shutdown of a runner	{"Process": "storagenode", "name": "piecestore:monitor", "error": "piecestore monitor: disk space requirement not met", "errorVerbose": "piecestore monitor: disk space requirement not met\n\tstorj.io/storj/storagenode/monitor.(*Service).Run:127\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-04-28T12:57:15.972Z	INFO	pieces:trash	emptying trash started	{"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2023-04-28T12:57:15.972Z	ERROR	nodestats:cache	Get pricing-model/join date failed	{"Process": "storagenode", "error": "context canceled"}
2023-04-28T12:57:15.972Z	ERROR	collector	error during collecting pieces: 	{"Process": "storagenode", "error": "pieceexpirationdb: context canceled", "errorVerbose": "pieceexpirationdb: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).GetExpired:39\n\tstorj.io/storj/storagenode/pieces.(*Store).GetExpired:556\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:88\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:57\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/storj/storagenode/collector.(*Service).Run:53\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-04-28T12:57:15.973Z	ERROR	piecestore:cache	error during init space usage db: 	{"Process": "storagenode", "error": "piece space used: context canceled", "errorVerbose": "piece space used: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceSpaceUsedDB).Init:73\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:81\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
2023-04-28T12:57:15.973Z	ERROR	gracefulexit:blobscleaner	couldn't receive satellite's GE status	{"Process": "storagenode", "error": "context canceled"}
2023-04-28T12:57:15.973Z	ERROR	bandwidth	Could not rollup bandwidth usage	{"Process": "storagenode", "error": "sql: transaction has already been committed or rolled back"}
2023-04-28T12:57:15.973Z	ERROR	gracefulexit:chore	error retrieving satellites.	{"Process": "storagenode", "error": "satellitesdb: context canceled", "errorVerbose": "satellitesdb: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*satellitesDB).ListGracefulExits:149\n\tstorj.io/storj/storagenode/gracefulexit.(*Service).ListPendingExits:59\n\tstorj.io/storj/storagenode/gracefulexit.(*Chore).AddMissing:58\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/storj/storagenode/gracefulexit.(*Chore).Run:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
Error: piecestore monitor: disk space requirement not met
2023-04-28 12:57:16,210 INFO stopped: storagenode (exit status 1)
2023-04-28 12:57:16,211 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)

If I am running a df --si I can’t see my mount folder of 11TB

df --si
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       126G  4.6G  116G   4% /
devtmpfs        3.9G     0  3.9G   0% /dev
tmpfs           4.1G     0  4.1G   0% /dev/shm
tmpfs           1.7G  1.5M  1.7G   1% /run
tmpfs           5.3M  4.1k  5.3M   1% /run/lock
/dev/mmcblk0p1  268M   33M  236M  13% /boot
tmpfs           820M   21k  820M   1% /run/user/1000

I should see something like this

lsblk
NAME                      MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sdb                         8:16   0   4.5T  0 disk
└─sdb1                      8:17   0   4.5T  0 part
  └─storj_crypto-storj_lv 254:0    0    10T  0 lvm
sdc                         8:32   0 931.5G  0 disk
└─sdc1                      8:33   0 931.5G  0 part
  └─storj_crypto-storj_lv 254:0    0    10T  0 lvm
mmcblk0                   179:0    0 119.1G  0 disk
├─mmcblk0p1               179:1    0   256M  0 part /boot
└─mmcblk0p2               179:2    0 118.8G  0 part /

You need to create a static mount in /etc/fstab.

Also, did you create a logical volume spanning 2 drives? You shouldn’t do this. If one disk fails all data from both disks will be gone. You better should setup 1 node per disk.

1 Like