Help! Error 1067 and some fatal error in the log

I’ve started having problems with dashboard as it stopped updating, I restarted my PC and I couldn’t start the node. I checked the logs and there is this error. I need help fast.

Don’t panic. This error message is in place to prevent disqualification. As long as the node keeps crashing it will not fail audits. Take your time to fix it.

The message is saying that one of the database files is corrupted.

It’s probably my fault as I didn’t know that you have to shutdown Storj service first before PC reboot. Now, what can I do? I read some similar threads about corrupted files, but it’s hard to understand.

I’m too afraid of doing anything by myself, so I don’t make things worse. I guess, I just have to wait for someone kind enough to help me step by step.

5 hours of node downtime now…

A few details regarding your node setup will be needed for others to try to figure this out.

  1. Which host OS are you using?
  2. Docker install or Windows GUI ?
  3. Regular x86 PC or something else ?
  4. Raspberry Pi … if so, which version ?
  5. Storage allocation and is the node full ?
  6. Any power outage before node stopped working ?
  7. Storage connection type: USB , SATA, etc ?

At the moment, downtime is not a problem. So, you can carefully go through the node setup and work through repairs without worrying about the clock running out.

  1. Windows 10
  2. Windows GUI
  3. x64 PC
  4. Nope
  5. 2.5 TB with around 1.25 TB stored (it’s been running for nearly 2 months)
  6. No, but when I had to update my OS I just restarted it without shutting down Storj. I thought it would close properly considering it has it’s own Windows service build in.
  7. External HDD with USB cable

Unfortunately, I don’t run Windows… so, I’ll probably be of limited help here.

However, it sounds like the USB connected drive was not properly managed by the automated Windows shutdown. Perhaps, Windows disconnected the USB drive before the Storj service finished writing to the disk. There are other topics around here regarding disabling disk caching in Windows to prevent data loss…

So, it seems like the combination of points 6 and 7 with the auto-update/restart corrupted one of the database files.

The best I can do is point you to another thread… such as the following:

There are many Windows node operators here. So if the above doesn’t help, I’m sure they’ll assist in getting you back up and running.


EDIT: Changed the link to more appropriate thread.

1 Like

I tried to use some of PowerShell commands from your link, but I got errors. I’ll just wait for someone to guide me through it.

If I make the node unusable, then there will be no going back. And I was so happy, that I got a lot of traffic in the second month…I don’t want to start again. Besides, the knowledge to repair corrupted database would come in handy in the future if it would happen again.

What were the errors?

I don’t know exactly, I’m not even sure that I copied it properly. I just put old orders.db file back and stopped doing anything.

I’m going to sleep soon, I feel really bad. I don’t have a job right now and prospect of making some extra money with Storj made me very happy until today.

I expect you’ll be able to find help here, but we probably need to know what specific problem you’re having first.

Always a good idea to get some sleep and start fresh. Don’t worry too much, a lot of people around here can help with problems like the one you are describing and chances are your node can be recovered one way or the other.

Start with this page. We’re looking specifically for the results of step 5.2.5.2.
Lots of nested lists there, but it’s the command that looks like this:

Get-ChildItem X:\storagenode\storage\*.db -File | %{$_.Name + " " + $(sqlite3.exe $_.FullName "PRAGMA integrity_check;")}

The error you’re seeing can’t be repaired with the steps that follow after this, so there is no need to continue further. We just want to know which database is giving you issues so we can help you recreate it.

If you run into any errors before you get to this step, let us know and we will try to help. Please include everything you have done so far, the exact command that returned an error and the actual error you are getting.

And don’t worry, you can take a little time to fix this. You’re node won’t be in trouble.

1 Like

Untitled

Seems it’s the orders.db file

Ok, please rename this file to orders.db.bak and run the following commands to recreate it.

sqlite3 orders.db

CREATE TABLE unsent_order (
                                                satellite_id  BLOB NOT NULL,
                                                serial_number BLOB NOT NULL,

                                                order_limit_serialized BLOB      NOT NULL, -- serialized pb.OrderLimit
                                                order_serialized       BLOB      NOT NULL, -- serialized pb.Order
                                                order_limit_expiration TIMESTAMP NOT NULL, -- when is the deadline for sending it

                                                uplink_cert_id INTEGER NOT NULL,

                                                FOREIGN KEY(uplink_cert_id) REFERENCES certificate(cert_id)
                                        );
CREATE TABLE order_archive_ (
                                                satellite_id  BLOB NOT NULL,
                                                serial_number BLOB NOT NULL,

                                                order_limit_serialized BLOB NOT NULL,
                                                order_serialized       BLOB NOT NULL,

                                                uplink_cert_id INTEGER NOT NULL,

                                                status      INTEGER   NOT NULL,
                                                archived_at TIMESTAMP NOT NULL,

                                                FOREIGN KEY(uplink_cert_id) REFERENCES certificate(cert_id)
                                        );
CREATE UNIQUE INDEX idx_orders ON unsent_order(satellite_id, serial_number);
CREATE TABLE versions (version int, commited_at text);
CREATE INDEX idx_order_archived_at ON order_archive_(archived_at);

.quit

This should create a new empty version of the db. The loss of data that used to be in there won’t be critical for your node and with the new empty replacement your node should start again.

I only get this. What does it mean?

Try it in 3 steps.

first:

sqlite3 orders.db

You should then get prompt from sqlite3. If not, please post what you see.

if you get the prompt, continue with:

CREATE TABLE unsent_order (
                                                satellite_id  BLOB NOT NULL,
                                                serial_number BLOB NOT NULL,

                                                order_limit_serialized BLOB      NOT NULL, -- serialized pb.OrderLimit
                                                order_serialized       BLOB      NOT NULL, -- serialized pb.Order
                                                order_limit_expiration TIMESTAMP NOT NULL, -- when is the deadline for sending it

                                                uplink_cert_id INTEGER NOT NULL,

                                                FOREIGN KEY(uplink_cert_id) REFERENCES certificate(cert_id)
                                        );
CREATE TABLE order_archive_ (
                                                satellite_id  BLOB NOT NULL,
                                                serial_number BLOB NOT NULL,

                                                order_limit_serialized BLOB NOT NULL,
                                                order_serialized       BLOB NOT NULL,

                                                uplink_cert_id INTEGER NOT NULL,

                                                status      INTEGER   NOT NULL,
                                                archived_at TIMESTAMP NOT NULL,

                                                FOREIGN KEY(uplink_cert_id) REFERENCES certificate(cert_id)
                                        );
CREATE UNIQUE INDEX idx_orders ON unsent_order(satellite_id, serial_number);
CREATE TABLE versions (version int, commited_at text);
CREATE INDEX idx_order_archived_at ON order_archive_(archived_at);

Again, please post the response if there is any.

Then finish with:

.quit

Yay, it worked, no errors and new file has been created. Is that all?

Yes, use this new empty db. Make sure it’s in the right location and then start your node again. Keep an eye on the logs to see if there are any errors.

And what about old file, should I leave it there?

Something is wrong…