Error starting master database on storagenode: database: unable to open database file: no such file or directory

Node Operators troubleshooting
1

My node was fine yesterday and is now in an infinite restart loop. I didn’t do any storj operations other than docker exec -it storagenode /app/dashboard.sh.

My OS is Linux and my storj setup is standard: https://documentation.storj.io/setup/cli/storage-node#running-the-storage-node.

Please help.

$ docker pull storjlabs/storagenode
Using default tag: latest
latest: Pulling from storjlabs/storagenode
Digest: sha256:ec3cea868ce23fb2e528d5232aa2ed535c10ab8bd6b67151f5033c6df9319d25
Status: Image is up to date for storjlabs/storagenode:latest
docker.io/storjlabs/storagenode:latest

$ docker image ls
REPOSITORY                           TAG                          IMAGE ID            CREATED             SIZE
storjlabs/storagenode                latest                       ae1314be9e57        8 days ago          29.9MB

$ docker ps
CONTAINER ID        IMAGE                          COMMAND                  CREATED             STATUS                         PORTS                                                                                         NAMES
74e389d4b1ff        storjlabs/storagenode:latest   "/entrypoint"            51 minutes ago      Restarting (1) 8 seconds ago                                                                                                 storagenode
00f1751c8d95        storjlabs/watchtower           "/watchtower storage…"   52 minutes ago      Up About a minute                                                                                                            watchtower

$ df
Filesystem                 Size  Used Avail Use% Mounted on
...
/dev/sda1                  932G  788G  144G  85% /home/storj/sda1

# pwd
/home/storj/sda1/storage

# ls -lh *.db
-rw-r--r--. 1 root root  14M Oct 10 12:50 bandwidth.db
-rw-r--r--. 1 root root  68K Oct 10 12:50 heldamount.db
-rw-r--r--. 1 root root  16K Oct 10 12:50 info.db
-rw-r--r--. 1 root root  24K Oct 10 12:50 notifications.db
-rw-r--r--. 1 root root 137M Oct 10 12:50 orders.db
-rw-r--r--. 1 root root  84K Oct 10 12:50 piece_expiration.db
-rw-r--r--. 1 root root 1.9M Oct 10 12:50 pieceinfo.db
-rw-r--r--. 1 root root  24K Oct 10 12:50 piece_spaced_used.db
-rw-r--r--. 1 root root  24K Oct 10 12:50 pricing.db
-rw-r--r--. 1 root root  24K Oct 10 12:50 reputation.db
-rw-r--r--. 1 root root  32K Oct 10 12:47 satellites.db
-rw-r--r--. 1 root root 360K Oct 10 12:50 storage_usage.db
-rw-r--r--. 1 root root  40M Oct 10 12:50 used_serial.db

Here’s the log:

Oct 10 14:42:59 storj[3199]: 2020-10-10T12:42:59.966Z        INFO        Configuration loaded        {"Location": "/app/config/config.yaml"}
Oct 10 14:42:59 storj[3199]: 2020-10-10T12:42:59.975Z        INFO        Operator email        {"Address": "redacted"}
Oct 10 14:42:59 storj[3199]: 2020-10-10T12:42:59.975Z        INFO        Operator wallet        {"Address": "redacted"}
Oct 10 14:43:00 storj[3199]: Error: Error starting master database on storagenode: database: unable to open database file: no such file or directory
Oct 10 14:43:00 storj[3199]:         storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:350
Oct 10 14:43:00 storj[3199]:         storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:336
Oct 10 14:43:00 storj[3199]:         storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:313
Oct 10 14:43:00 storj[3199]:         storj.io/storj/storagenode/storagenodedb.Open:245
Oct 10 14:43:00 storj[3199]:         main.cmdRun:151
Oct 10 14:43:00 storj[3199]:         storj.io/private/process.cleanup.func1.4:353
Oct 10 14:43:00 storj[3199]:         storj.io/private/process.cleanup.func1:371
Oct 10 14:43:00 storj[3199]:         github.com/spf13/cobra.(*Command).execute:840
Oct 10 14:43:00 storj[3199]:         github.com/spf13/cobra.(*Command).ExecuteC:945
Oct 10 14:43:00 storj[3199]:         github.com/spf13/cobra.(*Command).Execute:885
Oct 10 14:43:00 storj[3199]:         storj.io/private/process.ExecWithCustomConfig:88
Oct 10 14:43:00 storj[3199]:         storj.io/private/process.ExecCustomDebug:70
Oct 10 14:43:00 storj[3199]:         main.main:330
Oct 10 14:43:00 storj[3199]:         runtime.main:204

Any ideas?

2

Welcome to the forum @pooch!

First of all, excellent first post. I wish everyone posted their issue this way and gave all necessary information just like you.

You are missing piecestore.db. Can you figure out how it got deleted or moved (random mouse movement) ?

3

Thanks for the quick reply.

I have no idea how it could have been deleted. I only ever run the dashboard command, and not in the storage directory. The only time I’ve cd'ed into the storage directory was today just to do the ls.

Is it possible for me to re-create piecestore.db?

4

Weird. After taking storj down for several hours, doing nothing, and then restarting it, it magically broke out of the restart loop. However, I’m getting strange no space left on device related errors given that I have 144GB of raw disk space available and the dashboard reports that storj has 92GB available:

Storage Node Dashboard ( Node Version: v1.13.3 )

======================

ID     12eiX9fyZBiRjWabUUCQc8K8EbCy1XtdQy9ADRVhh7CstQud1Dq
Status ONLINE
Uptime 11m10s

                   Available         Used       Egress      Ingress
     Bandwidth           N/A     81.71 GB     37.24 GB     44.47 GB (since Oct 1)
          Disk      92.21 GB      0.84 TB
Internal 127.0.0.1:7778
External redacted

2020-10-10T18:25:53.758Z        ERROR        piecestore        upload failed        {"Piece ID": "L5CWXSBORNJ4HLXDO7LELNXLKVUNB53O3LWZJE6OO5BJTC25G5AA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "pieces error: filestore error: rename config/storage/temp/blob-786734932.partial config/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/l5/cwxsbornj4hlxdo7lelnxlkvunb53o3lwzje6oo5bjtc25g5aa.sj1: no space left on device", "errorVerbose": "pieces error: filestore error: rename config/storage/temp/blob-786734932.partial config/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/l5/cwxsbornj4hlxdo7lelnxlkvunb53o3lwzje6oo5bjtc25g5aa.sj1: no space left on device
   storj.io/storj/storage/filestore.(*blobWriter).Commit:120
   storj.io/storj/storagenode/pieces.(*Writer).Commit.func1:130
   storj.io/storj/storagenode/pieces.(*Writer).Commit:198
   storj.io/storj/storagenode/piecestore.(*Endpoint).Upload:383
   storj.io/common/pb.DRPCPiecestoreDescription.Method.func1:996
   storj.io/drpc/drpcmux.(*Mux).HandleRPC:107
   storj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56
   storj.io/drpc/drpcserver.(*Server).handleRPC:111
   storj.io/drpc/drpcserver.(*Server).ServeOne:62
   storj.io/drpc/drpcserver.(*Server).Serve.func2:99
   storj.io/drpc/drpcctx.(*Tracker).track:51"}
5

Please show result of the command:

df -H
6 
# df -H
(standard input):Filesystem                 Size  Used Avail Use% Mounted on
(standard input):devtmpfs                    32G     0   32G   0% /dev
(standard input):tmpfs                       32G  229M   32G   1% /dev/shm
(standard input):tmpfs                       32G  2.0M   32G   1% /run
(standard input):tmpfs                       32G     0   32G   0% /sys/fs/cgroup
(standard input):/dev/mapper/root  937G  762G  128G  86% /
(standard input):tmpfs                       32G  224K   32G   1% /tmp
(standard input):/dev/nvme0n1p1            1008M  270M  688M  29% /boot
(standard input):/dev/sda1                  932G  788G  144G  85% /home/storj/sda1

I can write to the disk, eg in the storage directory I did dd if=/dev/zero of=zero.log bs=4k count=10000.

7

I think it is worth to run a fsck in the safe mode on that disk to fix possible issues with a file system.

8 
# btrfsck /dev/sda1
Opening filesystem to check...
Checking filesystem on /dev/sda1
UUID: c3b72c18-e0d9-4fab-8bc0-000463568999
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 842393653248 bytes used, no error found
total csum bytes: 819793012
total tree bytes: 2682273792
total fs tree bytes: 1363574784
total extent tree bytes: 155222016
btree space waste bytes: 726666434
file data blocks allocated: 839767564288
 referenced 839542702080

Is it possible to re-create piecestore.db? If not then do I have any other options than starting anew?

9

No, it is not possible. There is no such database at all.

I would point in the second time that this error:

suggest that there is a problem with filesystem.
Please, run the actual disk check with fixing.

10

I did run the filesystem check here.

So do I have to start from scratch?

11

I don’t think so.
Did you try to stop and remove the container and run a new one?

Also, please read this article:

1 Like
12

Yes, I stopped and removed the container.

You’ve identified the problem for me:

# btrfs fi show /dev/sda1
Label: 'sda1'  uuid: c3b72c18-e0d9-4fab-8bc0-000463568999
	Total devices 1 FS bytes used 784.54GiB
	devid    1 size 931.51GiB used 931.51GiB path /dev/sda1

Thank-you so much! Please DM me with your WALLET address so that I can give you some well deserved rewards.

13

The “Thank you” is enough :slight_smile:, you can also use a heart icon on the post and mark it as a solution.

14

The lesson learned is that you should not use btrfs with storj:

(I setup my node before that discussion came to life.)