I see Unclean shutdown detected quite often after a regular shutdown of a node.
After restart it attempts to repair.
Docker command contains --stop-timeout 300 but still this is happening.
Are there other options?
You may try to increase a timeout.
I don’t observe that on freebsd – just restarted every node just to test, and grepped a week worth of logs. But I use neither docker nor windows. I can imagine two things maybe happening:
- container does not send the correct signal to the node
- filesystem does not flush properly before shutting down (windows does it all the time)
- both – if the container is on windows it’s a lost cause – sync across filesystems is broken. So if you shut down windows – I don’t see how woudl you not lose data.
I don’t think timeout matters.. All my nodes exit in under a second. 300 seconds is an eternity for the process to exit. There is simply nothing for it to do for five minutes.
As far as I remember, @jammerdan runs nodes on some SOC device with a small amount of RAM, so stopping nodes may be slower than on a powerful server.
Quite right. But not only on these.
What I have just observed is what @arrogantrabbit has written:
nodes exit in under a second. 300 seconds is an eternity for the process to exit. There is simply nothing for it to do for five minutes.
It is definitely not waiting 5 mins for the shutdown.
Could you please check the system journal for errors related to the disk?
Also how do you invoke this shutdown to get these errors?
I don’t see disk related errors.
docker stop and docker rm
I have just tried it again repeatedly without luck. It does not happen on every shutdown restart cycle but I have seen this several times now on different nodes.
It happened to me too, on Synology DS220+ with 18GB RAM, during hashstore migration. The system didn’t wait 300s and just killed the node, during a shutdown by UPS. 2 or 3 times. The other migrated node stopped nicely. Maybe it’s a Docker issue, not a storagenode issue.
maybe hashstore related chores, like compaction process, is not cancellable gracefully? If, that could be a bug. Whoever has tokens left this week can ask codex to look at the source to check if that’s the case 
Observed compaction average durations approach 300 seconds occasionally (from past week):
It just happened again on a different node for updating:
2026-04-24T08:06:54Z INFO hashstore unclean shutdown detected: reconciling logs {"process": "storagenode", "satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "store": "s0"}
I am still seeing this error a lot. This can’t be correct in a way.
Occasionally I also see this output when I restart the node to change the config. I actually ran into it yesterday when the DDNS didn’t update and the node didn’t receive traffic for half an hour. The server has 32 GB of RAM. The disk is fine and the journal contain no errors.
Please submit a bug on our GitHub, I shared this issue with the team, but seems a filled bug will be better.
The issue is still there and happends frequently now.
I had just restarted a node and was able to grab the shutdown and start sequence from the logs:
2026-07-04 06:36:19,921 WARN received SIGTERM indicating exit request
2026-07-04 06:36:19,935 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2026-07-04T06:36:20Z INFO Got a signal from the OS: "terminated" {"process": "storagenode-updater"}
2026-07-04 06:36:20,672 INFO stopped: storagenode-updater (exit status 0)
2026-07-04 06:36:23,675 INFO waiting for storagenode, processes-exit-eventlistener to die
2026-07-04 06:36:26,678 INFO waiting for storagenode, processes-exit-eventlistener to die
2026-07-04 06:36:29,682 INFO waiting for storagenode, processes-exit-eventlistener to die
2026-07-04 06:36:30,683 WARN killing 'storagenode' (61) with SIGKILL
2026-07-04 06:36:30,920 INFO stopped: storagenode (terminated by SIGKILL)
2026-07-04 06:36:30,920 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)
And upon restart i came back online with
2026-07-04T06:42:00Z INFO hashstore unclean shutdown detected: reconciling logs {"process": "storagenode", "satellite":
I stopped the node with a regular docker stop and docker remove and it responded instantly to it. Unless these commands are wrong the container should stop in a safe way and not restart with an error.
It seems that either the chosen unclean shutdown detection or the way how the container stops is flawd.
I do not have an account on Github.
Maybe it’s a time. I can submit an issue, but it will have a lower priority and I cannot answer on questions from developers, since I do not have this issue.
I notified the team again.
I see just hair over 10s (default grace period) between SIGTERM and SIGKILL. Are you sure you have --stop-timeout applied correctly? Perhaps share you docker run or compose yaml - whatever you use.
Thank you.
I think yes. The command starts with:
docker run -d --restart unless-stopped --stop-timeout 300 \
.....
Hmm, seems to be fine. Just to be sure, does this command report 300?
docker inspect --format='{{.Config.StopTimeout}}' containername # use your container name
Looks fine to me:
# docker inspect --format='{{.Config.StopTimeout}}' storagenode
300
To be sure I have repeated it again:
docker stop storagenode
2026-07-04 08:25:30,684 WARN received SIGTERM indicating exit request
2026-07-04 08:25:30,713 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2026-07-04T08:25:31Z INFO Got a signal from the OS: "terminated" {"process": "storagenode-updater"}
2026-07-04 08:25:31,263 INFO stopped: storagenode-updater (exit status 0)
2026-07-04 08:25:34,302 INFO waiting for storagenode, processes-exit-eventlistener to die
2026-07-04 08:25:37,305 INFO waiting for storagenode, processes-exit-eventlistener to die
2026-07-04 08:25:40,308 INFO waiting for storagenode, processes-exit-eventlistener to die
2026-07-04 08:25:41,309 WARN killing 'storagenode' (73) with SIGKILL
2026-07-04 08:25:41,599 INFO stopped: storagenode (terminated by SIGKILL)
2026-07-04 08:25:41,600 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)
docker start storagenode
2026-07-04T08:26:54Z INFO hashstore unclean shutdown detected: reconciling logs {"process": "storagenode", "satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "store": "s0"}
I noticed this error on my nodes too, this appears when im starting up a node, even though the nodes were shut down properly with docker command. I was wondering why this appears. I think it’s a bug, cause every node has that, but shutting down a node takes like 5 sec…