Unclean shutdown detected

[Alexey]

Because we need this change:

supervisord.conf: allow more time for clean shutdown

The build is successful, but a new tag has not pushed for some reason, asked the team.

[mcanto73]

@Alexey
Hi ,
could we modify manually our nodes to allow the change in advance ?

pi@mcanto:~$ docker exec -it storagenode7 bash
root@e086a5e8919a:/etc# apt-get update
…
root@e086a5e8919a:/etc# apt install vi

root@e086a5e8919a:/app# vi /etc/supervisord.conf
root@e086a5e8919a:/app# cat /etc/supervisord.conf | grep stopwa
stopwaitsecs=280
root@e086a5e8919a:/app# exit
exit

will be the changes persistent after restart ?

best regards

[Alexey]

They will not survive the container recreation. But you can do it in another way, mount a local file to the container.

docker run ...
...
--mount type=bind,source=/mnt/storagenode/supervisord.conf,target=/etc/supervisord.conf
...
storjlabs/storagenode:latest

[mcanto73]

Hello,
unfortunately, it seems that the proposed fix is really NOT working.
It’s persistent across the docker stop/start , of course a docker compose (down / up -d) will loose it

I noticed this one in the log , not sure is related to the added parameter (ubuntu@hpool6:~$ docker exec -it storagenode6 bash
root@d2acc9ed96c8:/app# cat /etc/supervisord.conf |grep wait
stopwaitsecs=280
) :
2026-09-29T07:10:53Z WARN services service takes long to shutdown {“process”: “storagenode”, “name”: “piecemigrate:chore”}
2026-09-29T07:10:53Z INFO services slow shutdown {“process”: “storagenode”, “stack”: “goroutine 726\n\tstorj.io/storj/private/lifecycle.(*Group).logStackTrace.func1:107\n\tsync.(*Once).doSlow:78\n\tsync.(*Once).Do:69\n\tstorj.io/storj/private/lifecycle.(*Group).logStackTrace:104\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func1:77\n\truntime/pprof.Do:57\n\ngoroutine 1\n\tsync.runtime_SemacquireWaitGroup:114\n\tsync.(*WaitGroup).Wait:206\n\tgolang.org/x/sync/errgroup.(*Group).Wait:56\n\tstorj.io/storj/storagenode.(*Peer).Run:1130\n\tmain.cmdRun:121\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.InitBeforeExecute.func1.2:389\n\tstorj.io/common/process.InitBeforeExecute.func1:407\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tmain.main:34\n\ngoroutine 9\n\tgo.opencensus.io/stats/view.(*worker).start:292\n\ngoroutine 11\n\tstorj.io/monkit-jaeger.(*ThriftCollector).Run:174\n\tstorj.io/common/process.initTracing.func1:120\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93\n\ngoroutine 13\n\tos/signal.signal_recv:152\n\tos/signal.loop:23\n\ngoroutine 58\n\tdatabase/sql.(*DB).connectionOpener:1261\n\ngoroutine 82\n\tdatabase/sql.(*DB).connectionOpener:1261\n\ngoroutine 727\n\tsync.runtime_SemacquireWaitGroup:114\n\tsync.(*WaitGroup).Wait:206\n\tstorj.io/common/errs2.(*Group).Wait:33\n\tstorj.io/storj/storagenode/piecemigrate.(*Chore).Run:273\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:93\n\ngoroutine 83\n\tdatabase/sql.(*DB).connectionOpener:1261\n\ngoroutine 51\n\tdatabase/sql.(*DB).connectionOpener:1261\n\ngoroutine 55\n\tdatabase/sql.(*DB).connectionOpener:1261\n\ngoroutine 56\n\tdatabase/sql.(*DB).connectionOpener:1261\n\ngoroutine 57\n\tdatabase/sql.(*DB).connectionOpener:1261\n\ngoroutine 59\n\tdatabase/sql.(*DB).connectionOpener:1261\n\ngoroutine 60\n\tdatabase/sql.(*DB).connectionOpener:1261\n\ngoroutine 61\n\tdatabase/sql.(*DB).connectionOpener:1261\n\ngoroutine 62\n\tdatabase/sql.(*DB).connectionOpener:1261\n\ngoroutine 63\n\tdatabase/sql.(*DB).connectionOpener:1261\n\ngoroutine 73\n\tdatabase/sql.(*DB).connectionOpener:1261\n\ngoroutine 102\n\tdatabase/sql.(*DB).connectionOpener:1261\n\ngoroutine 87\n\tdatabase/sql.(*DB).connectionOpener:1261\n\ngoroutine 109\n\tdatabase/sql.(*DB).connectionOpener:1261\n\ngoroutine 693\n\ttime.Sleep:363\n\tgithub.com/spacemonkeygo/monkit/v3.(*ticker).run:201\n\ngoroutine 433\n\tstorj.io/storj/storagenode/hashstore.(*DB).backgroundCompactions:575\n\ngoroutine 432\n\tstorj.io/storj/storagenode/hashstore.(*DB).backgroundCompactions:575\n\ngoroutine 643\n\tstorj.io/storj/storagenode/pieces.(*PieceExpirationStore).flushOnTicks:117\n\ngoroutine 746\n\tsyscall.Syscall:74\n\tsyscall.unlinkat:166\n\tsyscall.Rmdir:319\n\tos.Remove.func2:369\n\tos.ignoringEINTR:256\n\tos.Remove:368\n\tstorj.io/storj/storagenode/piecemigrate.(*Chore).cleanupEmptyDirectories:569\n\tstorj.io/storj/storagenode/piecemigrate.(*Chore).runOnce:294\n\tstorj.io/common/sync2.(*Cycle).Run:163\n\tstorj.io/storj/storagenode/piecemigrate.(*Chore).Run.func1:267\n\tstorj.io/storj/storagenode/piecemigrate.(*Chore).Run.(*Group).Go.func5:21\n\tsync.(*WaitGroup).Go.func1:258\n”}
2026-09-29T07:51:12Z INFO Configuration loaded {“process”: “storagenode”, “location”: “/app/config/config.yaml”}
2026-09-29T07:51:12Z INFO Anonymized tracing enabled {“process”: “storagenode”}
2026-09-29T07:51:12Z INFO Operator email {“process”: “storagenode”, “address”: “marco.canto01@gmail.com”}
2026-09-29T07:51:12Z INFO Operator wallet {“process”: “storagenode”, “address”: “0x7C23e0f610c455475ADE140324E37EB7bb168452”}
2026-09-29T07:51:12Z INFO db.migration Database Version {“process”: “storagenode”, “version”: 62}
2026-09-29T07:51:13Z INFO server existing kernel support for server-side tcp fast open detected {“process”: “storagenode”}
2026-09-29T07:51:15Z INFO hashstore loaded hint file {“process”: “storagenode”, “satellite”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “store”: “s0”, “path”: “config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/meta/hint-00000000000000ad”, “writable”: [43, 126, 128, 310, 313, 666, 849, 856, 880, 882, 900, 907, 916, 961, 1001, 1002, 1018, 1049, 1053, 1081, 1090, 1138, 1181, 1190, 1206, 1243, 1278, 1297, 1317, 1320, 1343, 1364, 1396, 1450, 1503, 1601, 1627, 1656, 1705, 1706, 1737, 1958, 1959, 1960, 1985, 1994, 1995, 1999, 2007, 2008, 2025, 2026, 2040, 2047, 2048, 2053, 2060, 2061, 2063, 2064], “largest”: 2065, “skip”: false}
2026-09-29T07:54:52Z INFO hashstore unclean shutdown detected: reconciling logs {“process”: “storagenode”, “satellite”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “store”: “s0”}
2026-09-29T07:55:53Z WARN hashstore mismatched log tail {“process”: “storagenode”, “satellite”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “store”: “s0”, “id”: 1278, “path”: “config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/fe/log-00000000000004fe-00005130”, “table”: [{“Key”:“ITKDYUCKKNP3R7PMUSI2Z56PGS6JPDBIWGXWMOKTYKD55TP3HG2A”,“Offset”:1721536,“Log”:1278,“Length”:179712,“Created”:20693,“Expires”:41568},{“Key”:“OKKXGFGVDASF4Q5FMV7PWVOLFH7SSK5KKLCBLFU7JTJXKCNFNQHQ”,“Offset”:1184640,“Log”:1278,“Length”:536832,“Created”:20693,“Expires”:41568},{“Key”:“5DACTNLTNTTRKJ5L3L55KLUYZGOMYB6AUL3BHLLD6VS2QJKC5M5Q”,“Offset”:693312,“Log”:1278,“Length”:491264,“Created”:20693,“Expires”:41568},{“Key”:“AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA”,“Offset”:0,“Log”:0,“Length”:0,“Created”:0,“Expires”:0},{“Key”:“AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA”,“Offset”:0,“Log”:0,“Length”:0,“Created”:0,“Expires”:0}], “log”: [{“Key”:“26XGVAXELOIHXDEPRVAP2TD7GCBSVXAO7J23GX2MJOF24YFOKF6A”,“Offset”:1901312,“Log”:1278,“Length”:479744,“Created”:20693,“Expires”:41568},{“Key”:“ITKDYUCKKNP3R7PMUSI2Z56PGS6JPDBIWGXWMOKTYKD55TP3HG2A”,“Offset”:1721536,“Log”:1278,“Length”:179712,“Created”:20693,“Expires”:41568},{“Key”:“OKKXGFGVDASF4Q5FMV7PWVOLFH7SSK5KKLCBLFU7JTJXKCNFNQHQ”,“Offset”:1184640,“Log”:1278,“Length”:536832,“Created”:20693,“Expires”:41568},{“Key”:“5DACTNLTNTTRKJ5L3L55KLUYZGOMYB6AUL3BHLLD6VS2QJKC5M5Q”,“Offset”:693312,“Log”:1278,“Length”:491264,“Created”:20693,“Expires”:41568},{“Key”:“MMEHOANE6VK4N4YPEEI2ZIXSFLP3KTZ5WQESTRCHWYJYAXUKL23Q”,“Offset”:0,“Log”:1278,“Length”:693248,“Created”:20693,“Expires”:41568}]}

Is there something else to be tried ?
Is the fix applied correctly ?

Best regards

[program:storagenode]
command=/app/bin/storagenode run --config-dir config --identity-dir identity --version.server-address=https://version.storj.io --storage.allocated-disk-space=5700GB --contact.external-address=89.168.21.73:28967 --operator.email=marco.canto01@gmail.com --operator.wallet=0x7C23e0f610c455475ADE140324E37EB7bb168452
autorestart=true
stopwaitsecs=280
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stdout
stderr_logfile_maxbytes=0

[Alexey]

There is no fix currently. This is what I’m trying to post everytime. There is a workaround, which, obviously doesn’t work.
Please wait when the image got published in that case.
Or you can use your suggested way to preserve it - exec to the container and add this setting, then restart the container. It should survive a restart, but will be reset on re-create.

[mcanto73]

Hello,
it’s clear that it is a workaround and I was saying that it is not working.
The only doubt that I have is that this new parameter “stopwaitsecs” could require also a new binary that we will get only with the new release
Do you have any clue ? otherwise we are waiting for nothing that will help to solve

thanks

[Alexey]

I see, I thought that my suggestion didn’t work.
If you used an exec method, then already started supervisord likely wouldn’t re-read its config even after restarting the container.
Please try my suggestion. Please make sure that the docker timeout is greater than used in supervisord. Then do docker compose down and docker compose up -d.

[mcanto73]

@Alexey
Hello,
I added the change to one docker container but it’s not starting

 - type: bind
   source: /NODE7/NODE7/supervisord.conf
   target: /etc/supervisord.conf

pi@mcanto:~/storj-docker7$ ls -l /NODE7/NODE7/supervisord.conf
-rw-r–r-- 1 root root 2959 30 set 10.13 /NODE7/NODE7/supervisord.conf
pi@mcanto:~/storj-docker7$ grep -i stopwait /NODE7/NODE7/supervisord.conf
stopwaitsecs=280
pi@mcanto:~/storj-docker7$

pi@mcanto:~/storj-docker7$ docker logs -f storagenode7
sed: cannot rename /etc/supervisor/sedlhwYUp: Device or resource busy
sed: cannot rename /etc/supervisor/sedhGw9Ar: Device or resource busy
sed: cannot rename /etc/supervisor/sed0qdnNB: Device or resource busy
sed: cannot rename /etc/supervisor/sedWNZ3kI: Device or resource busy

any idea ?

thanks

[jammerdan]

I have again a node here that had a normal shutdown. After restart it detected again an unclean shutdown and is now creating warning after warning with mismatched log tail for 5 hours already.

[EasyRhino]

Just chiming in that this happens to me sporadically. (It seems to shutdown clean but then says unclean detected on next start) It’s most likely to happen on one node where the storage is mounted over a slowish NFS connection (which yes, I know you are not supposed to do).

[Toyoo]

@EasyRhino

happens to me sporadically

Yeah, me as well, especially when the whole docker daemon shuts down as part of a reboot or sth. To the point where when I need to reboot, I first shut down nodes manually.

[ItsHass]

Yes I did see this too recently on 2 nodes - I did fsck and bad sector check , just in case.. it did take the log mismatch timings down alot since doing fsck..

[Alexey]

Seems it didn’t work.
Let’s try to build a local copy from our repo:

services:
  storagenode:
    # image: storjlabs/storagenode:latest
    image: storagenode:latest
    build: https://github.com/storj/storagenode-docker.git
    restart: always
    stop_grace_period: 300s
...

Then docker compose build and then docker compose up -d

[jammerdan]

@Alexey @elek

As a first step can we at least start debug output earlier so that a node doing its startup chore will not be marked not responding or dead?

[Alexey]

Unclean shutdown detected

As turned out, it’s fixed in a modular storagenode and will not be fixed in legacy. The network will be migrated to a modular storagenode eventually, but currently it’s a low priority and can happen only when migration to hashstore will be completed.

So, nothing to debug. We need to switch to a modular version.
Currently I don’t have a documentation, how to enable it.

[jammerdan]

@Alexey

We need to switch to a modular version.

The question is what is possible without the modular version which is probably unrealistiv to expect any time soon.

So without knowing any details how the debug server is implemented my suggestion would be if the web server on the debug port can start sooner. We don’t necessarily need a full debug output during startup phase just something that indicates that the node is running normally and is not down.

[Alexey]

You should see messages in the logs. Also, the container should be up and running.
Also I think that we might switch to a modular storagenode sooner than you and me thinks. The migration is almost finished, only 1PB is remained.
We also started to implement to disable piecestore. So, I guess it should happen soon.

[jammerdan]

@Alexey

We also started to implement to disable piecestore. So, I guess it should happen soon.

It depends on the disabling of piecestore?

[Alexey]

Yes. At least this is what I was told.
Also, the feature flag to disable a centralized migration to hashstore will be removed to finish migration as soon as possible.

[Alexey]

We updated the base image.
You need to do docker pull storjlabs/storagenode:latest then stop and remove the container

docker stop -t 300 storagenode
docker rm storagenode

and run it back using all your parameters.

For docker-compose it’s a little bit simpler:

docker compose up -d --pull