FATAL Unrecoverable error {“error”: “Error creating tables for master database on storagenode: migrate: storage node database error: migrate tables:: attempt to write a readonly database

Node Operators troubleshooting
1

I am getting the following error message in one of my nodes:

2019-10-03T19:23:17.242Z        INFO    db.migration.23 Split into multiple sqlite databases

2019-10-03T19:23:17.269Z FATAL Unrecoverable error {“error”: “Error creating tables for master database on storagenode: migrate: storage node database error: migrate tables:: attempt to write a readonly database\n\tstorj.io/storj/internal/dbutil/sqliteutil.backupConns:94\n\tstorj.io/storj/internal/dbutil/sqliteutil.backupDBs.func3.1:71\n\tdatabase/sql.(*Conn).Raw:1888\n\tstorj.io/storj/internal/dbutil/sqliteutil.backupDBs.func3:65\n\tdatabase/sql.(*Conn).Raw:1888\n\tstorj.io/storj/internal/dbutil/sqliteutil.backupDBs:59\n\tstorj.io/storj/internal/dbutil/sqliteutil.MigrateTablesToDatabase:27\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).migrateToDB:361\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).Migration.func4:805\n\tstorj.io/storj/internal/migrate.Func.Run:219\n\tstorj.io/storj/internal/migrate.(*Migration).Run:132\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).CreateTables:255\n\tmain.cmdRun:178\n\tstorj.io/storj/pkg/process.cleanup.func1.2:275\n\tstorj.io/storj/pkg/process.cleanup.func1:293\n\tgithub.com/spf13/cobra.(*Command).execute:762\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:852\n\tgithub.com/spf13/cobra.(*Command).Execute:800\n\tstorj.io/storj/pkg/process.Exec:74\n\tmain.main:349\n\truntime.main:203”, “errorVerbose”: “Error creating tables for master database on storagenode: migrate: storage node database error: migrate tables:: attempt to write a readonly database\n\tstorj.io/storj/internal/dbutil/sqliteutil.backupConns:94\n\tstorj.io/storj/internal/dbutil/sqliteutil.backupDBs.func3.1:71\n\tdatabase/sql.(*Conn).Raw:1888\n\tstorj.io/storj/internal/dbutil/sqliteutil.backupDBs.func3:65\n\tdatabase/sql.(*Conn).Raw:1888\n\tstorj.io/storj/internal/dbutil/sqliteutil.backupDBs:59\n\tstorj.io/storj/internal/dbutil/sqliteutil.MigrateTablesToDatabase:27\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).migrateToDB:361\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).Migration.func4:805\n\tstorj.io/storj/internal/migrate.Func.Run:219\n\tstorj.io/storj/internal/migrate.(*Migration).Run:132\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).CreateTables:255\n\tmain.cmdRun:178\n\tstorj.io/storj/pkg/process.cleanup.func1.2:275\n\tstorj.io/storj/pkg/process.cleanup.func1:293\n\tgithub.com/spf13/cobra.(*Command).execute:762\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:852\n\tgithub.com/spf13/cobra.(*Command).Execute:800\n\tstorj.io/storj/pkg/process.Exec:74\n\tmain.main:349\n\truntime.main:203\n\tmain.cmdRun:180\n\tstorj.io/storj/pkg/process.cleanup.func1.2:275\n\tstorj.io/storj/pkg/process.cleanup.func1:293\n\tgithub.com/spf13/cobra.(*Command).execute:762\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:852\n\tgithub.com/spf13/cobra.(*Command).Execute:800\n\tstorj.io/storj/pkg/process.Exec:74\n\tmain.main:349\n\truntime.main:203”}

2

Hello @JoshGarza,
Welcome to the forum!

Please, show the output of the command: docker inspect storagenode -f '{{ .Mounts }}'

3

thanks!

[{bind  /shared3/storj/SN1 /app/config   true rprivate} {volume f6b75a4c0cfd665a7b00f86de26b27ede411192d533859f837a776f338b66e49 /var/lib/docker/volumes/f6b75a4c0cfd665a7b00f86d                               e26b27ede411192d533859f837a776f338b66e49/_data /root/.local/share/storj/storagenode local  true } {bind  /shared3/storj/SN1/identity/storagenode /app/identity   true rprivate}]

During update, it creates the “bandwidth.db” file but no data is written in this db and then it tries the update again… in loop.

4

Please, give me an output of these commands

df -h
ls -l /shared3/
ls -l /shared3/storj
ls -l /shared3/storj/SN1
5 
$ df -h

/dev/sdb1 7,3T 1,1T 5,9T 15% /shared3

$ ls -l /shared3/

drwxrwxrwx 2 root root 16384 feb 15 2019 lost+found
drwxrwxrwx 5 nas nas 4096 jul 18 18:46 storj

$ ls -l /shared3/storj
drwxrwxrwx 5 user user 4096 oct 3 23:01 SN1

$ ls -l /shared3/storj/SN1

-rwxrwxrwx 1 root root 262144 oct 3 21:21 kademlia
-rw-r–r-- 1 root root 532563235 oct 3 23:19 node.log
-rwxrwxrwx 1 root root 32768 oct 3 23:19 revocations.db
drwxrwxrwx 7 root root 4096 oct 3 23:19 storage

6

I hope you have running the docker with a sudo. If not - then chown the SN1 to the user.

7

yep… always with sudo. So i dont see where it is there permission problem

8

It seems the problem is in the bandwidth.db newly created. log.level=debug shows the following:

image
image1245×90 6.82 KB

9

How is your storage device connected to the host with docker?

10

internal HDD, no virtual machines

btw, i am sure it is a issue with sqlite database permissions, not filesystem permissions,

11

What is filesystem on that drive?

12

Please, show a deeper:

ls -l /shared3/storj/SN1/storagenode
13 
-rw-r--r-- 1 root root      4096 oct  4 00:51 bandwidth.db

drwxrwxrwx 2 user root 4096 jul 17 02:21 blob
drwxrwxrwx 6 user root 4096 jul 30 22:07 blobs
drwxrwxrwx 2 user root 4096 oct 3 20:44 garbage
-rwxrwxrwx 1 user user 451696640 oct 4 00:27 info.db
-rwxrwxrwx 1 user user 32768 oct 4 00:51 info.db-shm
-rwxrwxrwx 1 user user 0 oct 4 00:27 info.db-wal
-rwxrwxrwx 1 user root 13316096 abr 15 12:35 piecestore.db
drwxrwxrwx 2 user root 4096 oct 3 21:19 temp
drwxrwxrwx 2 user root 4096 jul 16 20:31 trash

i have run all with “sudo” and i never had problems like this

14

I am moving the node to another HDD… but im afraid the problem is in the database.
Tomorrow morning i will check again.

Thanks!

Note: this info.db is from the first node I started in February. All newer nodes have not shown any issue with this database.

15

Please, tell me, what is filesystem on that drive?
You don’t need to move data!

chown the whole mountpoint to the root:

sudo docker stop -t 300 storagenode 
sudo chown -R root:root /shared3/storj/
sudo docker rm storagenode 
sudo docker run ...there whole command..

if your user user in the docker group, then you can do the reverse: chown all to the user:user and never use the sudo with docker

16

in this machine is ext4

17

seems you run the docker without a sudo at least once or have moved your node to there under the user

18

maybe the very first time… I have moved the node 2-3 times without any issue.
chown’ing the whole directory didnt fix it. I will copy all the node during the night and i will check in the morning.

Thanks for your support.

19

chowning should be with -R (recursive ). The removal of the container is a mandatory operation.
You must stop and better to remove the container before the chowning.

21

As I supposed, the permission issue was in the sqlite database itselfm not in the filesystem.

I got it working following next steps:

  • Backup info.db
  • Dump all info.db sql tables to a file
  • Open new info.db
  • Increase page size to 4096: PRAGMA page_size=4096;
  • read previously dumped tables.
  • Save

Love all the vacuums and split in several dbs (one per satellite would be enough tho).
Cheers!

1 Like