FATAL Unrecoverable error {“error”: “Error creating tables for master database on storagenode: migrate: storage node database error: migrate tables:: database is locked

SN is throwing a fatal error on startup after updating to latest version:

2019-10-03T21:09:45.795Z FATAL Unrecoverable error {“error”: “Error creating tables for master database on storagenode: migrate: storage node database error: migrate tables:: database is locked\n\tstorj.io/storj/internal/dbutil/sqliteutil.backupDBs:50\n\tstorj.io/storj/internal/dbutil/sqliteutil.MigrateTablesToDatabase:27\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).migrateToDB:361\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).Migration.func4:805\n\tstorj.io/storj/internal/migrate.Func.Run:219\n\tstorj.io/storj/internal/migrate.(*Migration).Run:132\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).CreateTables:255\n\tmain.cmdRun:178\n\tstorj.io/storj/pkg/process.cleanup.func1.2:275\n\tstorj.io/storj/pkg/process.cleanup.func1:293\n\tgithub.com/spf13/cobra.(*Command).execute:762\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:852\n\tgithub.com/spf13/cobra.(*Command).Execute:800\n\tstorj.io/storj/pkg/process.Exec:74\n\tmain.main:349\n\truntime.main:203”, “errorVerbose”: “Error creating tables for master database on storagenode: migrate: storage node database error: migrate tables:: database is locked\n\tstorj.io/storj/internal/dbutil/sqliteutil.backupDBs:50\n\tstorj.io/storj/internal/dbutil/sqliteutil.MigrateTablesToDatabase:27\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).migrateToDB:361\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).Migration.func4:805\n\tstorj.io/storj/internal/migrate.Func.Run:219\n\tstorj.io/storj/internal/migrate.(*Migration).Run:132\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).CreateTables:255\n\tmain.cmdRun:178\n\tstorj.io/storj/pkg/process.cleanup.func1.2:275\n\tstorj.io/storj/pkg/process.cleanup.func1:293\n\tgithub.com/spf13/cobra.(*Command).execute:762\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:852\n\tgithub.com/spf13/cobra.(*Command).Execute:800\n\tstorj.io/storj/pkg/process.Exec:74\n\tmain.main:349\n\truntime.main:203\n\tmain.cmdRun:180\n\tstorj.io/storj/pkg/process.cleanup.func1.2:275\n\tstorj.io/storj/pkg/process.cleanup.func1:293\n\tgithub.com/spf13/cobra.(*Command).execute:762\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:852\n\tgithub.com/spf13/cobra.(*Command).Execute:800\n\tstorj.io/storj/pkg/process.Exec:74\n\tmain.main:349\n\truntime.main:203”}

Container is start looping.

Please, stop docker stop -t 300 storagenode and remove the container docker rm storagenode, then run it again. Please, do not exec the dashboard, just look into logs.

Same thing. Congratulations on having a minimum of 20 characters to post.

How is your storage device connected to the host with docker?

It is connected over SMB.

I’m noticing the node is creating bandwidth.db file and then removing it and then recreating it again constantly as it’s restarting.

This is just after:

2019-10-03T21:46:14.439Z INFO db.migration.23 Split into multiple sqlite databases
2019-10-03T21:46:14.442Z DEBUG db opened database bandwidth

Is it a Linux host?
Is the SMB server Linux too?

Seems we have another incompatibility issue with a network connected drive…
Can you connect it with iSCSI?

SN host is Ubuntu.
SMB host is FreeBSD (FreeNAS, which is finally supported thankfully).

Incompatibility? Again? I’ve had issues with NFS and that was all over the Internet, regarding SQL. But for SMB I have not found such issues. iSCSI is a gigantic hassle and inflexible, I would avoid it if possible.

Based on previous experience it doesn’t look like incompatibility. When I had NFS issues there was nothing in the log - nothing about database locked, just simply uploads piling up and timing out.

I’ve had database locked messages each time when starting the container - the node did some heavy activity on the storage for 2-3 hours and allowed downloads but uploads were canceled. Then the database got unlocked, activity stopped and uploads worked normally.

The description for any network connected drive because of high latency.
However, each of high level protocol such as SMB or NFS have had own locking methods and do not suitable for database processing. They may work, but not guarantee.
Seems you are unlucky and now you meet a different problem with it.
Try to connect this storage directly or via iSCSI to finish migration, maybe it could work after via SMB, but better to avoid such usages, the latency likely killing your ability to compete with other nodes

There is basically no latency as a result of network share, nearly all latency is drive seek time, because I’m using hard drives - and mostly everyone has these. But if there was latency issue, then iSCSI would have it as well. A node like this not working because of a too slow hard drive would be unheard of…

It cannot be connected directly. I can try to move it to iSCSI but I have a feeling it will be a gigantic waste of time.

Can you give me any checks to do regarding the database before I do anything as lengthy as trying to migrate to iSCSI?

unfortunately only start the storagenode.
The latency should not be there from the logic, but it’s here from the practice.
iSCSI works much better than NFS or SMB, you even notice the significance in the acceptance ratio as other users who switched to ISCSI.

@Storgeez if you’re not yet convinced by @Alexey have a look at the results this user saw.


The old values before switching to iSCSI are in brackets.

1

After automatic updating, one node became in such an error
After a while, the second node was updated and now it is in the same error.
Prior to the upgrade, everything worked perfectly.
What should I do ?

The network connected drives are not supported. Neither NFS nor SMB.
Please, use a local connection or at least iSCSI.

Hi, how i can fix this error?

[root@storj ~]# docker run -d --restart unless-stopped -p 28967:28967 -p 192.168.0.10:14002:14002 -e WALLET=“0xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx” -e EMAIL="guarez@gmail.com" -e ADDRESS=“2xx.2xx.2xx.2xx:28967” -e BANDWIDTH=“6TB” -e STORAGE=“1.5TB” --mount type=bind,source="/root/.local/share/storj/identity/storagenode",destination=/app/identity --mount type=bind,source="/mnt/storj",destination=/app/config --name storagenode storjlabs/storagenode:beta
b1510795620d0ba3e7db6a3cd0c84c0fb2f0af3416dc66ca81a9eb4f3bd514dc
[root@storj ~]# docker logs -f storagenode
2019-10-04T02:01:08.419Z INFO Configuration loaded from: /app/config/config.yaml
2019-10-04T02:01:08.443Z INFO Operator email: xxxxxxxx@gmail.com
2019-10-04T02:01:08.443Z INFO operator wallet: 0xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx3
2019-10-04T02:01:09.163Z INFO version running on version v0.22.1
2019-10-04T02:01:09.236Z INFO db.migration.23 Split into multiple sqlite databases
2019-10-04T02:01:19.292Z FATAL Unrecoverable error {“error”: “Error creating tables for master database on storagenode: migrate: storage node database error: migrate tables:: database is locked\n\tstorj.io/storj/internal/dbutil/sqliteutil.backupDBs:50\n\tstorj.io/storj/internal/dbutil/sqliteutil.MigrateTablesToDatabase:27\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).migrateToDB:361\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).Migration.func4:805\n\tstorj.io/storj/internal/migrate.Func.Run:219\n\tstorj.io/storj/internal/migrate.(*Migration).Run:132\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).CreateTables:255\n\tmain.cmdRun:178\n\tstorj.io/storj/pkg/process.cleanup.func1.2:275\n\tstorj.io/storj/pkg/process.cleanup.func1:293\n\tgithub.com/spf13/cobra.(*Command).execute:762\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:852\n\tgithub.com/spf13/cobra.(*Command).Execute:800\n\tstorj.io/storj/pkg/process.Exec:74\n\tmain.main:349\n\truntime.main:203”, “errorVerbose”: “Error creating tables for master database on storagenode: migrate: storage node database error: migrate tables:: database is locked\n\tstorj.io/storj/internal/dbutil/sqliteutil.backupDBs:50\n\tstorj.io/storj/internal/dbutil/sqliteutil.MigrateTablesToDatabase:27\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).migrateToDB:361\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).Migration.func4:805\n\tstorj.io/storj/internal/migrate.Func.Run:219\n\tstorj.io/storj/internal/migrate.(*Migration).Run:132\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).CreateTables:255\n\tmain.cmdRun:178\n\tstorj.io/storj/pkg/process.cleanup.func1.2:275\n\tstorj.io/storj/pkg/process.cleanup.func1:293\n\tgithub.com/spf13/cobra.(*Command).execute:762\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:852\n\tgithub.com/spf13/cobra.(*Command).Execute:800\n\tstorj.io/storj/pkg/process.Exec:74\n\tmain.main:349\n\truntime.main:203\n\tmain.cmdRun:180\n\tstorj.io/storj/pkg/process.cleanup.func1.2:275\n\tstorj.io/storj/pkg/process.cleanup.func1:293\n\tgithub.com/spf13/cobra.(*Command).execute:762\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:852\n\tgithub.com/spf13/cobra.(*Command).Execute:800\n\tstorj.io/storj/pkg/process.Exec:74\n\tmain.main:349\n\truntime.main:203”}
[root@storj ~]#

Try this and do update your post if it helps

mmm i have the same error

2019-10-04T18:37:11.165Z INFO Configuration loaded from: /app/config/config.yaml
2019-10-04T18:37:11.190Z INFO Operator email: xxxxx@gmail.com
2019-10-04T18:37:11.190Z INFO operator wallet: 0x4xdxxxxxxxxxxxd3
2019-10-04T18:37:11.868Z INFO version running on version v0.22.1
2019-10-04T18:37:12.071Z INFO db.migration.23 Split into multiple sqlite databases
2019-10-04T18:37:22.191Z FATAL Unrecoverable error {“error”: “Error creating tables for master database on storagenode: migrate: storage node database error: migrate tables:: database is locked\n\tstorj.io/storj/internal/dbutil/sqliteutil.backupDBs:50\n\tstorj.io/storj/internal/dbutil/sqliteutil.MigrateTablesToDatabase:27\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).migrateToDB:361\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).Migration.func4:805\n\tstorj.io/storj/internal/migrate.Func.Run:219\n\tstorj.io/storj/internal/migrate.(*Migration).Run:132\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).CreateTables:255\n\tmain.cmdRun:178\n\tstorj.io/storj/pkg/process.cleanup.func1.2:275\n\tstorj.io/storj/pkg/process.cleanup.func1:293\n\tgithub.com/spf13/cobra.(*Command).execute:762\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:852\n\tgithub.com/spf13/cobra.(*Command).Execute:800\n\tstorj.io/storj/pkg/process.Exec:74\n\tmain.main:349\n\truntime.main:203”, “errorVerbose”: “Error creating tables for master database on storagenode: migrate: storage node database error: migrate tables:: database is locked\n\tstorj.io/storj/internal/dbutil/sqliteutil.backupDBs:50\n\tstorj.io/storj/internal/dbutil/sqliteutil.MigrateTablesToDatabase:27\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).migrateToDB:361\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).Migration.func4:805\n\tstorj.io/storj/internal/migrate.Func.Run:219\n\tstorj.io/storj/internal/migrate.(*Migration).Run:132\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).CreateTables:255\n\tmain.cmdRun:178\n\tstorj.io/storj/pkg/process.cleanup.func1.2:275\n\tstorj.io/storj/pkg/process.cleanup.func1:293\n\tgithub.com/spf13/cobra.(*Command).execute:762\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:852\n\tgithub.com/spf13/cobra.(*Command).Execute:800\n\tstorj.io/storj/pkg/process.Exec:74\n\tmain.main:349\n\truntime.main:203\n\tmain.cmdRun:180\n\tstorj.io/storj/pkg/process.cleanup.func1.2:275\n\tstorj.io/storj/pkg/process.cleanup.func1:293\n\tgithub.com/spf13/cobra.(*Command).execute:762\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:852\n\tgithub.com/spf13/cobra.(*Command).Execute:800\n\tstorj.io/storj/pkg/process.Exec:74\n\tmain.main:349\n\truntime.main:203”}

Same error here.
Running on Linux (Latest Ubuntu)
2019-10-04T19:21:52.473Z FATAL Unrecoverable error {“error”: “Error creating tables for master database on storagenode: migrate: storage node database error: migrate tables:: database is locked\n\tstorj.io/storj/internal/dbutil/sqliteutil.backupDBs:50\n\tstorj.io/storj/internal/dbutil/sqliteutil.MigrateTablesToDatabase:27\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).migrateToDB:361\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).Migration.func4:805\n\tstorj.io/storj/internal/migrate.Func.Run:219\n\tstorj.io/storj/internal/migrate.(*Migration).Run:132\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).CreateTables:255\n\tmain.cmdRun:178\n\tstorj.io/storj/pkg/process.cleanup.func1.2:275\n\tstorj.io/storj/pkg/process.cleanup.func1:293\n\tgithub.com/spf13/cobra.(*Command).execute:762\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:852\n\tgithub.com/spf13/cobra.(*Command).Execute:800\n\tstorj.io/storj/pkg/process.Exec:74\n\tmain.main:349\n\truntime.main:203”, “errorVerbose”: “Error creating tables for master database on storagenode: migrate: storage node database error: migrate tables:: database is locked\n\tstorj.io/storj/internal/dbutil/sqliteutil.backupDBs:50\n\tstorj.io/storj/internal/dbutil/sqliteutil.MigrateTablesToDatabase:27\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).migrateToDB:361\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).Migration.func4:805\n\tstorj.io/storj/internal/migrate.Func.Run:219\n\tstorj.io/storj/internal/migrate.(*Migration).Run:132\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).CreateTables:255\n\tmain.cmdRun:178\n\tstorj.io/storj/pkg/process.cleanup.func1.2:275\n\tstorj.io/storj/pkg/process.cleanup.func1:293\n\tgithub.com/spf13/cobra.(*Command).execute:762\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:852\n\tgithub.com/spf13/cobra.(*Command).Execute:800\n\tstorj.io/storj/pkg/process.Exec:74\n\tmain.main:349\n\truntime.main:203\n\tmain.cmdRun:180\n\tstorj.io/storj/pkg/process.cleanup.func1.2:275\n\tstorj.io/storj/pkg/process.cleanup.func1:293\n\tgithub.com/spf13/cobra.(*Command).execute:762\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:852\n\tgithub.com/spf13/cobra.(*Command).Execute:800\n\tstorj.io/storj/pkg/process.Exec:74\n\tmain.main:349\n\truntime.main:203”}

I updated another node without error.

I did this to troubleshoot:

  1. Remove container and create again (no result)
  2. Reboot the PC, restarted container (no result)

I can try

GO TO 1.

… but I’m not sure that will help.

Edit 2:
I did that. It is still running:

2019-10-04T19:26:00.591Z INFO Configuration loaded from: /app/config/config.yaml
2019-10-04T19:26:00.617Z INFO Operator email: mymail@mydomain.com
2019-10-04T19:26:00.617Z INFO operator wallet: 0x6986yodaghla9698608dad087d0F
2019-10-04T19:26:01.205Z INFO version running on version v0.22.1
2019-10-04T19:26:01.286Z INFO db.migration.23 Split into multiple sqlite databases

EDIT 3:
THE ERROR IS BACK.
So… this node is toast. Too bad.

I’m getting tired of updates that break things. Just when my nodes are running and stable, updates throw sand in the machine. UGh.

EDIT 4
It retries and throws the same error every 12-15 seconds.

EDIT 5

The storage is on a NAS. So I did:

  1. Update NAS
  2. Reboot NAS
  3. Update Linux (latest Kernel)
  4. Reboot server
  5. Recreate storagenode

RESULT:::::

2019-10-04T19:50:01.295Z FATAL Unrecoverable error {“error”: “Error creating tables for master database on storagenode: migrate: storage node datab ase error: migrate tables:: database is locked\n\tstorj.io/storj/internal/dbutil /sqliteutil.backupDBs:50\n\tstorj.io/storj/internal/dbutil/sqliteutil.MigrateTab lesToDatabase:27\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).migrateToDB:3 61\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).Migration.func4:805\n\tstor j.io/storj/internal/migrate.Func.Run:219\n\tstorj.io/storj/internal/migrate.(*Mi gration).Run:132\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).CreateTables: 255\n\tmain.cmdRun:178\n\tstorj.io/storj/pkg/process.cleanup.func1.2:275\n\tstor j.io/storj/pkg/process.cleanup.func1:293\n\tgithub.com/spf13/cobra.(*Command).ex ecute:762\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:852\n\tgithub.com/spf13/ cobra.(*Command).Execute:800\n\tstorj.io/storj/pkg/process.Exec:74\n\tmain.main: 349\n\truntime.main:203”, “errorVerbose”: “Error creating tables for master data base on storagenode: migrate: storage node database error: migrate tables:: data base is locked\n\tstorj.io/storj/internal/dbutil/sqliteutil.backupDBs:50\n\tstor j.io/storj/internal/dbutil/sqliteutil.MigrateTablesToDatabase:27\n\tstorj.io/sto rj/storagenode/storagenodedb.(*DB).migrateToDB:361\n\tstorj.io/storj/storagenode /storagenodedb.(*DB).Migration.func4:805\n\tstorj.io/storj/internal/migrate.Func .Run:219\n\tstorj.io/storj/internal/migrate.(*Migration).Run:132\n\tstorj.io/sto rj/storagenode/storagenodedb.(*DB).CreateTables:255\n\tmain.cmdRun:178\n\tstorj. io/storj/pkg/process.cleanup.func1.2:275\n\tstorj.io/storj/pkg/process.cleanup.f unc1:293\n\tgithub.com/spf13/cobra.(*Command).execute:762\n\tgithub.com/spf13/co bra.(*Command).ExecuteC:852\n\tgithub.com/spf13/cobra.(*Command).Execute:800\n\t storj.io/storj/pkg/process.Exec:74\n\tmain.main:349\n\truntime.main:203\n\tmain. cmdRun:180\n\tstorj.io/storj/pkg/process.cleanup.func1.2:275\n\tstorj.io/storj/p kg/process.cleanup.func1:293\n\tgithub.com/spf13/cobra.(*Command).execute:762\n\ tgithub.com/spf13/cobra.(*Command).ExecuteC:852\n\tgithub.com/spf13/cobra.(*Comm and).Execute:800\n\tstorj.io/storj/pkg/process.Exec:74\n\tmain.main:349\n\trunti me.main:203”}

EDIT 6
See also this topic:

EDIT 7
I never had problems like this before on this machine, the latest update seems to cause it.

Network storage does not work anymore. Neither SMB, nor NFS.
SMB could work in rare cases.
You should use the iSCSI or local connection (PCI, SATA, USB).

wow! a tremendous change! Maybe more than one will be left out :frowning: I will see how I change the configuration, you could give a more descriptive error message … thanks for the information

@Alexey How do you plan on supporting FreeNAS and other NAS systems then? I sure hope iSCSI won’t be the only protocol supported…