Pricing.db Error: unable to open database file

Node Operators troubleshooting
1

Hi!

Main problem: docker container restarts every minute. Logs:

2021-07-21T13:34:28.974Z INFO Got a signal from the OS: “terminated”
Error: Error starting master database on storagenode: database: context canceled
storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:323
storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:305
storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:281
storj.io/storj/storagenode/storagenodedb.OpenExisting:248
main.cmdRun:160
storj.io/private/process.cleanup.func1.4:363
storj.io/private/process.cleanup.func1:381
github.com/spf13/cobra.(*Command).execute:852
github.com/spf13/cobra.(*Command).ExecuteC:960
github.com/spf13/cobra.(*Command).Execute:897
storj.io/private/process.ExecWithCustomConfig:88
storj.io/private/process.ExecCustomDebug:70
main.main:385
runtime.main:204

Storj corrupts db file immediately after docker container starts.

When I do integrity check of DB files I see this:

pricing.db Error: unable to open database file

After repairing (link: https://support.storj.io/hc/en-us/articles/360029309111-How-to-fix-a-database-disk-image-is-malformed-) I do one more integrity check:

pricing.db ok

Then I start and stop docker container and check integrity again:

pricing.db Error: unable to open database file

Why?

P.S.
Disk is OK - no bad blocks.

Best regards, Alex.

2

Hi @AGu , The team will look into this and have requested additional log messages from prior to the errors as to get some context. Thanks!

3

Looks like a permissions issue.
How do you run your storagenode container? With sudo or without?
If without - make sure that the database file have the same permissions as others (replace path /mnt/storj/storagenode1/storage to actual one):

sudo ls -l /mnt/storj/storagenode1/storage/*.db
4

With sudo or without

Using root user.

ls -l *.db
-rw-r–r-- 1 root root 757760 Jun 4 04:40 bandwidth.db
-rw-r–r-- 1 root root 32768 Jun 4 04:40 heldamount.db
-rw-r–r-- 1 root root 16384 Jun 4 04:40 info.db
-rw-r–r-- 1 root root 24576 Jun 4 04:40 notifications.db
-rw-r–r-- 1 root root 32768 Jun 4 04:40 orders.db
-rw-r–r-- 1 root root 73728 Jun 4 04:40 piece_expiration.db
-rw-r–r-- 1 root root 24576 Jun 4 04:40 piece_spaced_used.db
-rw-r–r-- 1 root root 24576 Jun 4 04:40 pieceinfo.db
-rw-r–r-- 1 root root 4096 Jul 21 16:26 pricing.db
-rw-r–r-- 1 root root 24576 Jun 4 04:40 reputation.db
-rw-r–r-- 1 root root 32768 Jun 4 04:40 satellites.db
-rw-r–r-- 1 root root 24576 Jun 4 04:40 secret.db
-rw-r–r-- 1 root root 53248 Jun 4 04:40 storage_usage.db
-rw-r–r-- 1 root root 20480 Jun 4 04:40 used_serial.db

lsattr *.db
--------------e----- bandwidth.db
--------------e----- heldamount.db
--------------e----- info.db
--------------e----- notifications.db
--------------e----- orders.db
--------------e----- piece_expiration.db
--------------e----- piece_spaced_used.db
--------------e----- pieceinfo.db
--------------e----- pricing.db
--------------e----- reputation.db
--------------e----- satellites.db
--------------e----- secret.db
--------------e----- storage_usage.db
--------------e----- used_serial.db

5

Then it’s more like a disk issue. How is it connected to the host with storagenode?
Keep in mind - network filesystems are not supported, include but not limited to Topics tagged smb Topics tagged nfs Topics tagged sshfs, only iSCSI network protocol for storage is supported.
It’s much better to use only local attached drives and ext4 filesystem in Linux (NTFS if you use Windows).

What is your filesystem on storage?

df -HT
6

Internal HDD, SATA cable, ext4.

df -HT
/dev/sdd1   ext4   3.0T  48G  2.8T  2% /mnt/storj/node

Oh, just found errors in dmsg :frowning:

dmesg | grep sdd1     
[64988.445506] EXT4-fs error (device sdd1): __ext4_get_inode_loc:4712: inode #168826989: block 675283366: comm storagenode: unable to read itable block
7

Sorry to read that. Please, stop the storagenode and check your disk and fix issues, then try to start again.