Lost DBs during migration

So, I migrated node to another HDD. Accidently - all .dbs & identity where removed with all backups which where nearby.
All other data is fine. Identity is recovered. Any chanses to restart a node without dbs?

Storagenodes will not start without dbs in the folder, If there completely gone you can try to run setup command to readd the dbs back.

That’s the point, setup command created empty databases. Thus node is able to run but it keeps restarting because dbs are completely empty…

Maybe this will help, Error: Error creating tables for master database on storagenode: migrate: no such table: main.order_archive_

Checked, I already created empty dbs but node restarts each time with:
Error: piecestore monitor: error verifying location and/or readability of storage directory: content of file is not a valid node ID:

I wish I could be more help but I havent experienced this error myself I have started my node with fresh database files with no issues before so im not really sure what the issue could be.

What configuration is your node? Windows? Docker? other?

content of file is not a valid node ID:

This error points to the ‘storage-dir-verification’ file missing, it should be in the same folder as the db’s. This file is generated on first setup and needs to exist for the node to run. If you run the setup process again using the same identity the file should be recreated.

Docker. Thx for pointing at this - thus I found that I did not copy complete dirrectory …

This post should be what you require (with the correct paths):

Many thanks, already done that and that helped. Node is running yet suffered badly because of audits. Hopefully all data remains, so I hope score will recover to 100% soon!

Great. As long as all the blobs are present and the node is online the audit score should recover.

This is usually mean that you have used an identity which doesn’t belongs to data.

This is not a good advise, sorry. If you do that on corrupted storagenode - you could provide a wrong identity and disqualify the original node.
The setup step must not be executed on existing node.

To re-create databases you can use this guide: https://support.storj.io/hc/en-us/articles/4403032417044-How-to-fix-database-file-is-not-a-database-error

This doesnt actually work since the node wont start without databases so you have to run setup to get the dbs back. I ran into the issue when my bandwidth.db corrupted.

The way your talking doesnt actually readd the database files if you loose the files.

Have you even tried? The node will start without any databases and will create the empty ones. But I should repeat - “start without any databases”, this is a mandatory requirement to re-create databases. If some databases are still exist, it will not create any databases.

The setup step did not create databases at all. It creates a folders structure and generates the file storage/storage-dir-verification with NodeID.
This step will fail if the config.yaml is exist.

Yeah of course I tried I had to delete everything to get the databases back by running the setup. It was the only way I was able to get my node running again.

But I think its because i removed everything including the storage-dir-verification file.

This is impossible to create databases with the setup step. Just no way in the code.
Please note in the example below I have used a docker volume instead of real storage location!

docker run --rm -e SETUP="true" --mount type=bind,source=/home/username/.local/share/identity/storagenode/,destination=/app/identity -v storagenode3:/app/config --name storagenode storjlabs/storagenode:latest
...
2022-03-30 23:09:39 (2.32 MB/s) - '/tmp/storagenode.zip' saved [16258195/16258195]

Running ./storagenode setup  --config-dir config --identity-dir identity --metrics.app-suffix=-alpha --metrics.interval=30m --version.server-address=https://version.storj.io --console.address=:14002 --storage.allocated-disk-space=2.0TB

Now check

docker run -it --rm -v storagenode3:/app/config ubuntu
root@4fa749f1276e:/# ls -l /app/config/**
-rw------- 1 root root 9337 Mar 30 23:09 /app/config/config.yaml

/app/config/storage:
total 20
drwx------ 2 root root 4096 Mar 30 23:09 blobs
drwx------ 2 root root 4096 Mar 30 23:09 garbage
-rw-r--r-- 1 root root   32 Mar 30 23:09 storage-dir-verification
drwx------ 2 root root 4096 Mar 30 23:09 temp
drwx------ 2 root root 4096 Mar 30 23:09 trash

As you can see - no databases have been created. Only folders structure, config.yaml and verification file

Looks a bit different then I remember, But say you backup all the databases and move the storage-dir-verification, Then try to start the node it doesnt create the databases correct?

No.
You should not touch storage-dir-verification.
If you have moved all databases to the backup and started the node - it will create an empty databases.

See

docker run -e WALLET=0x... --mount type=bind,source=/home/user/.local/share/storj/identity/storagenode,destination=/app/identity -v storagenode3:/app/config --name storagenode storjlabs/storagenode:latest
...
2022-03-30 23:18:19 (93.1 KB/s) - '/tmp/storagenode.zip' saved [16258195/16258195]

2022-03-30 23:18:19,658 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
2022-03-30 23:18:19,664 INFO RPC interface 'supervisor' initialized
2022-03-30 23:18:19,664 CRIT Server 'inet_http_server' running without any HTTP authentication checking
2022-03-30 23:18:19,664 INFO supervisord started with pid 1
2022-03-30 23:18:20,670 INFO spawned: 'processes' with pid 36
2022-03-30 23:18:20,673 INFO spawned: 'storagenode' with pid 37
2022-03-30 23:18:20,676 INFO spawned: 'storagenode-updater' with pid 38
2022-03-30T23:18:20.698Z        INFO    Configuration loaded    {"Location": "/app/config/config.yaml"}
2022-03-30T23:18:20.699Z        INFO    Invalid configuration file key  {"Key": "console.address"}
2022-03-30T23:18:20.699Z        INFO    Invalid configuration file key  {"Key": "operator.wallet-features"}
2022-03-30T23:18:20.699Z        INFO    Invalid configuration file key  {"Key": "storage.allocated-bandwidth"}
2022-03-30T23:18:20.699Z        INFO    Invalid configuration file key  {"Key": "server.address"}
2022-03-30T23:18:20.699Z        INFO    Invalid configuration file key  {"Key": "operator.wallet"}
2022-03-30T23:18:20.699Z        INFO    Invalid configuration file key  {"Key": "contact.external-address"}
2022-03-30T23:18:20.699Z        INFO    Invalid configuration file key  {"Key": "operator.email"}
2022-03-30T23:18:20.699Z        INFO    Invalid configuration file key  {"Key": "storage.allocated-disk-space"}
2022-03-30T23:18:20.699Z        INFO    Invalid configuration file key  {"Key": "server.private-address"}
2022-03-30T23:18:20.699Z        INFO    Invalid configuration file value for key        {"Key": "log.level"}
2022-03-30T23:18:20.706Z        INFO    Running on version      {"Service": "storagenode-updater", "Version": "v1.50.4"}
2022-03-30T23:18:20.707Z        INFO    Downloading versions.   {"Server Address": "https://version.storj.io"}
2022-03-30T23:18:20.724Z        INFO    Configuration loaded    {"Location": "/app/config/config.yaml"}
2022-03-30T23:18:20.732Z        WARN    Operator email address isn't specified.
2022-03-30T23:18:20.732Z        INFO    Operator wallet {"Address": "0x..."}
2022-03-30T23:18:20.732Z        INFO    db      database does not exist {"database": "info"}
2022-03-30T23:18:20.732Z        INFO    db      database does not exist {"database": "bandwidth"}
2022-03-30T23:18:20.732Z        INFO    db      database does not exist {"database": "orders"}
2022-03-30T23:18:20.732Z        INFO    db      database does not exist {"database": "piece_expiration"}
2022-03-30T23:18:20.732Z        INFO    db      database does not exist {"database": "pieceinfo"}
2022-03-30T23:18:20.732Z        INFO    db      database does not exist {"database": "piece_spaced_used"}
2022-03-30T23:18:20.732Z        INFO    db      database does not exist {"database": "reputation"}
2022-03-30T23:18:20.732Z        INFO    db      database does not exist {"database": "storage_usage"}
2022-03-30T23:18:20.732Z        INFO    db      database does not exist {"database": "used_serial"}
2022-03-30T23:18:20.732Z        INFO    db      database does not exist {"database": "satellites"}
2022-03-30T23:18:20.732Z        INFO    db      database does not exist {"database": "notifications"}
2022-03-30T23:18:20.732Z        INFO    db      database does not exist {"database": "heldamount"}
2022-03-30T23:18:20.732Z        INFO    db      database does not exist {"database": "pricing"}
2022-03-30T23:18:20.732Z        INFO    db      database does not exist {"database": "secret"}
2022-03-30T23:18:21.441Z        INFO    Current binary version  {"Service": "storagenode", "Version": "v1.50.4"}
...

and now the check:

docker run -it --rm -v storagenode3:/app/config ubuntu
root@006c3e77dd08:/# ls -l /app/config/**
-rw------- 1 root root  9337 Mar 30 23:09 /app/config/config.yaml
-rw------- 1 root root 32768 Mar 30 23:18 /app/config/revocations.db
-rw------- 1 root root  1374 Mar 30 23:18 /app/config/trust-cache.json

/app/config/orders:
total 8
drwx------ 2 root root 4096 Mar 30 23:18 archive
drwx------ 2 root root 4096 Mar 30 23:18 unsent

/app/config/storage:
total 384
-rw-r--r-- 1 root root 32768 Mar 30 23:18 bandwidth.db
drwx------ 2 root root  4096 Mar 30 23:09 blobs
drwx------ 2 root root  4096 Mar 30 23:09 garbage
-rw-r--r-- 1 root root 32768 Mar 30 23:18 heldamount.db
-rw-r--r-- 1 root root 16384 Mar 30 23:18 info.db
-rw-r--r-- 1 root root 24576 Mar 30 23:18 notifications.db
-rw-r--r-- 1 root root 32768 Mar 30 23:18 orders.db
-rw-r--r-- 1 root root 36864 Mar 30 23:18 piece_expiration.db
-rw-r--r-- 1 root root 24576 Mar 30 23:18 piece_spaced_used.db
-rw-r--r-- 1 root root 24576 Mar 30 23:18 pieceinfo.db
-rw-r--r-- 1 root root 24576 Mar 30 23:18 pricing.db
-rw-r--r-- 1 root root 24576 Mar 30 23:18 reputation.db
-rw-r--r-- 1 root root 32768 Mar 30 23:18 satellites.db
-rw-r--r-- 1 root root 24576 Mar 30 23:18 secret.db
-rw-r--r-- 1 root root    32 Mar 30 23:09 storage-dir-verification
-rw-r--r-- 1 root root 20480 Mar 30 23:18 storage_usage.db
drwx------ 2 root root  4096 Mar 30 23:09 temp
drwx------ 2 root root  4096 Mar 30 23:09 trash
-rw-r--r-- 1 root root 20480 Mar 30 23:18 used_serial.db

Hmm it still creates the database files without the storage storage-dir-verification, Has anything changes since when the setup first started? I just remember having so much issues when my bandwidth.db corrupted.

Please do not touch the storage-dir-verification.
This file is used against disqualification of your node. And has nothing to do with databases.
It contains the NodeID and used to check that:

  1. You did setup
  2. Your data belongs to the provided identity