When I try to start the container again I get the message: " Error: Error starting master database on storagenode: database: database heldamount does not exist stat config/storage/heldamount.db: no such file or directory"
Does any one know what I need to do to get the nodes running again?
With the most recent release, create config backup mv config.yaml config.yaml.bak. Run the container so it goes through setup process and create missing dbs. Stop the container, restore config mv config.yaml.bak > config.yaml, start container again.
That should help!
So what command do I need to use to start the container again? When I follow your recipe and using docker run to start the container I get an error: “container name already in use”.
Can you point me to step by step documentation on how to get this right? I had configured automatic upgrades and didn’t look back. Did I miss an announcement that automatic updates were broken or not supported anymore?
I now have more than 24 hours downtime and I feel this is through no fault of my own. This is quite frustrating.
You should be able to start the container by using docker start storagenode
You only need to use the docker run command when you have removed the container. Afterward can you please show the result of docker ps -a so we can try to see why your automatic updates failed?
—edit— Step by step following Yarolav’s procedure.
docker stop -t 300 storagenode
create config backup
mv config.yaml config.yaml.bak
run the container
docker start storagenode
Watch the logs to see when the container is done the setup process
docker logs -f --tail 40 storagenode
stop storagenode when(/if) it is done migration. Look for the start up uploads/downloads.
docker stop -t 300 storagenode
If you never modified any of the config.yaml options, you do not need to restore the old config. If you did modify config.yaml options, restore the old config.yaml, or modify the new one.
Let’s start with one of the nodes. I am assuming they are all running on different machines. Can you please run docker ps -a
on the node you are currently working on?
When you see a sqlite> prompt execute this script:
CREATE TABLE versions (version int, commited_at text);
CREATE TABLE paystubs (
period text NOT NULL,
satellite_id bytea NOT NULL,
created_at timestamp NOT NULL,
codes text NOT NULL,
usage_at_rest double precision NOT NULL,
usage_get bigint NOT NULL,
usage_put bigint NOT NULL,
usage_get_repair bigint NOT NULL,
usage_put_repair bigint NOT NULL,
usage_get_audit bigint NOT NULL,
comp_at_rest bigint NOT NULL,
comp_get bigint NOT NULL,
comp_put bigint NOT NULL,
comp_get_repair bigint NOT NULL,
comp_put_repair bigint NOT NULL,
comp_get_audit bigint NOT NULL,
surge_percent bigint NOT NULL,
held bigint NOT NULL,
owed bigint NOT NULL,
disposed bigint NOT NULL,
paid bigint NOT NULL,
PRIMARY KEY ( period, satellite_id )
);
CREATE TABLE payments (
id bigserial NOT NULL,
created_at timestamp NOT NULL,
satellite_id bytea NOT NULL,
period text,
amount bigint NOT NULL,
receipt text,
notes text,
PRIMARY KEY ( id )
);
.exit
following your suggestions I seem to get one step further. The error message is now:
Error: Error starting master database on storagenode: database: database pricing does not exist stat config/storage/pricing.db: no such file or directory
So heldamount.db seems to be now opened correctly. The issue is now pricing.db
Could you please share the scripts needed to create this database and any others that I’ll probably be needing?
You should backup only databases, not blobs folder (because there almost all your node’s data).
To move 11 databases it should not take more than a few seconds.
I’ve backed up the database for node-05 and node-06 and then I get my original error message. Error: Error starting master database on storagenode: database: database heldamount does not exist stat config/storage/heldamount.db: no such file or directory
So it appears that for some reason, the databases I need are not being created.
As I would have expected docker ps -a gives
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8f7740a5fabd storjlabs/storagenode:latest “/entrypoint” 4 minutes ago Exited (1) 2 minutes ago node-05
88b7ae6727df storjlabs/storagenode:latest “/entrypoint” 12 minutes ago Exited (1) 4 minutes ago node-06
bca89f7088e1 storjlabs/storagenode:beta “/entrypoint” 6 months ago Exited (1) 45 hours ago node-07
931ef584cfb9 storjlabs/storagenode:beta “/entrypoint” 6 months ago Exited (1) 45 hours ago node-04
4aa3de4ca0ad storjlabs/storagenode:beta “/entrypoint” 6 months ago Exited (1) 45 hours ago node-03
acd7e07a9698 storjlabs/storagenode:beta “/entrypoint” 6 months ago Exited (1) 45 hours ago node-02
e8de23cc398d storjlabs/storagenode:beta “/entrypoint” 6 months ago Exited (1) 45 hours ago node-01
This is what I would have expected as I’ve been trying to solve the problem for node-05 and node-06 first before I mess with the other nodes. All the nodes have their own public IP address, but I moved them to their the same machine to save power when I noticed there was very light load on the machines.
If you would follow the instruction, it will create an empty databases.
Stop the storagenode (only one)
Move its databases to backup subfolder
Move config.yaml to config.yaml.bak
Run a new temporary container below (it has limited number of arguments, it’s important!), replace the /path/to to your actual data path for that node and run the container:
docker run -it --rm -v /path/to:/app/config storjlabs/storagenode:latest
It will throw an error and exit.
Copy databases and config back from the backup to their places (use your paths!) with overwrite:
You should be able to see your audit score and suspension score on the dashboard.
The corrupted or lost databases do not prevent to pass audits. So, if your data is intact - there is nothing to worry about.
On edge case you are lost a local stat.