Slow shutdown on node, it dont shutdown

[olympios]

i do a shutdown from services but i get this

2024-07-19T18:37:02+03:00 INFO Stop/Shutdown request received.
2024-07-19T18:37:02+03:00 INFO lazyfilewalker.trash-cleanup-filewalker subprocess exited with status {“satelliteID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “status”: 1, “error”: “exit status 1”}
2024-07-19T18:37:02+03:00 ERROR pieces:trash emptying trash failed {“error”: “pieces error: lazyfilewalker: exit status 1”, “errorVerbose”: “pieces error: lazyfilewalker: exit status 1\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:85\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkCleanupTrash:187\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:422\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1.1:84\n\tstorj.io/common/sync2.(*Workplace).Start.func1:89”}
2024-07-19T18:37:17+03:00 WARN services service takes long to shutdown {“name”: “retain”}
2024-07-19T18:37:17+03:00 WARN servers service takes long to shutdown {“name”: “server”}
2024-07-19T18:37:17+03:00 INFO servers slow shutdown {“stack”: "goroutine 1077 [running]:\nstorj.io/storj/private/lifecycle.(*Group).logStackTrace.func1()\n\t/go/src/storj.io/storj/private/lifecycle/group.go:107 +0x78\nsync.(*Once).doSlow(0xc004910270?, 0xc004c2ca40?)\n\t/usr/local/go/src/sync/once.go:74 +0xc2\nsync.(*Once).Do(…)\n\t/usr/local/go/src/sync/once.go:65\nstorj.io/storj/private/lifecycle.(*Group).logStackTrace(0xc0007d9ee8?)\n\t/go/src/storj.io/storj/private/lifecycle/group.go:104 +0x3c\nstorj.io/storj/private/lifecycle.(*Group).Run.func1({0x1ad3230?, 0xc000918990?})\n\t/go/src/storj.io/storj/private/lifecycle/group.go:77 +0x2b2\nruntime/pprof.Do({0x1ad3498?, 0xc000878140?}, {{0xc000860140?, 0xc0007d9fd0?, 0xf65f65?}}, 0xc00087e200)\n\t/usr/local/go/src/runtime/pprof/runtime.go:51 +0x9d\ncreated by storj.io/storj/private/lifecycle.(*Group).Run in goroutine 35\n\t/go/src/storj.io/storj/private/lifecycle/group.go:64 +0x4cb\n\ngoroutine 1 [syscall, locked to thread]:\nsyscall.SyscallN(0x7ff93c1a7e20?, {0xc0000b9c40?, 0x3?, 0x175

continue the log here

2024-07-19T18:37:02+03:00 INFO Stop/Shutdown request received.2024-07-19T18:37...

[pangolin]

This seems to be windows, so you can forcibly close it in task manager.

[olympios]

that i do, maybe its related with bad/faulty hdd?

[Balage76]

I think it is related to this, but on Windows nodes:

High cpu load caused by storj containers

This happens because one of the threads of GC actually hangs (enters an infinite non-productive loop). And since it’s only one thread, it can only take up one core max. But do it at 100% load non-stop. In my case, it did not even respond to requests to restart the node - all other threads shut down correctly, but there was one that did not respond to commands. /mon/ps has shown that this thread is related to the garbage collector. So I even had to kill the node process to restart - because I…

I noticed that I can stop the node, but can not delete the log file, becouse it is locked by the process of the storagenode, which also use 100% of the CPU thread/core. As soon as I force stop the process (of the previously stopped node), I can also delete the log file.

[olympios]

thank you for answer!

[Alexey]

The failed filewalker is a consequence of the shutdown request

2024-07-19T18:37:02+03:00	INFO	Stop/Shutdown request received.

But the fact that the node is hanging after that is concerning.
@olympios Is it still true on v1.108.3 or higher?

[Julio]

I had a node hang, when auto updating from 1.108.3 to v1.109.2 last night. This morning, I killed that process, restarted… new version showed up - good times, working now.

1/2 cent

[Alexey]

And do you have a lazy mode enabled?
Could you please restart the container/service one more time? Would it hang?
I’m asking, because my docker and Windows nodes doesn’t behave like this…