Unclean shutdown detected: reconciling logs (how long does it take?)

Node Operators
1

I had an abrubt power off of my storage system, and now when storj restarts I get a message in the log “INFO hashstore unclean shutdown detected: reconciling logs”

Every several minutes I will get a message like “WARN hashstore mismatched log tail”.

It seems to be taking a VERY long time. like… almost two days so far.

How long should I expect it to take? I think the node had around 6TB of data. And is there something else I should be doing to fix it?

This node was using memtbl by the way.

2

Yes. Buy an UPS.

1 Like
3

Well cleverly, I had a UPS but that doesn’t protect against me intentionally shutting down the storage while the node is still running.

4

Mmm. You mean node is not running on the same host as the storage? Why? And if that’s the case – why do you use memtable, that is guaranteed to go out of sync?

5

sometimes I ask “how?” rather than “why?”…

But yeah two other nodes were not using memtbl and they did not suffer the same fate after the unclean storage shutdown. At least on this anecdote of one occurrence.

But the rebuild, or whatever it is, is STILL running…

6

… it’s STILL running. IT’s been like 4 days. Every 7 or so minutes I get a “mismatched log tail” WARN message, but the node has not yet come up.

I feel like I’m definitely getting a suspension or disqualification.

Is there a way to know how long this… whatever this is… will take or make it run faster? I’ve seen mention of a dedicated write-hashtbl program, but a) I don’t understand it and b) dunno if it will fix the particular problem I’m having.

7

It will be slower than the filewalker on piecestore. Because it’s literary doing the same thing with extra steps and without the benefit of metadata space locality.

You cannot make it faster because it’s limited by seek latency.

You are experiencing one of the tradeoffs of hashstore. It’s a worthwhile tradeoff, because abrupt power loss or kernel panic on a stable system is not a thing in modern times.

8

oh that’s a bummer. it’s a slow node so it’s going to be a while.

I also got these new messages I hadn’t seen before:

INFO contact:amnesty sending amnesty batch to satellite

  ERROR   contact:amnesty ailed to get satellite URL for amnesty report
9

I would assume, that since the node is not started yet, it didn’t have a satellite’s URL.