INFO waiting for storagenode, processes-exit-eventlistener to die

Node Operators troubleshooting
1

Hello everyone,
Is somebody familiar with the following logs?

2022-06-30T15:59:09.010653274Z 2022-06-30 15:59:09,010 INFO waiting for storagenode, processes-exit-eventlistener to die
2022-06-30T15:59:12.015277495Z 2022-06-30 15:59:12,014 INFO waiting for storagenode, processes-exit-eventlistener to die
2022-06-30T15:59:14.730748782Z 2022-06-30 15:59:14,729 INFO stopped: storagenode (terminated by SIGKILL)
2022-06-30T15:59:14.733872085Z 2022-06-30 15:59:14,733 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)
2022-06-30T15:59:20.734262339Z 2022-06-30 15:59:20,732 INFO Set uid to user 0 succeeded
2022-06-30T15:59:20.773158439Z 2022-06-30 15:59:20,772 INFO RPC interface 'supervisor' initialized
2022-06-30T15:59:20.774328038Z 2022-06-30 15:59:20,773 INFO supervisord started with pid 1
2022-06-30T15:59:21.780149114Z 2022-06-30 15:59:21,778 INFO spawned: 'processes-exit-eventlistener' with pid 11
2022-06-30T15:59:21.787631540Z 2022-06-30 15:59:21,786 INFO spawned: 'storagenode' with pid 12
2022-06-30T15:59:21.795093189Z 2022-06-30 15:59:21,793 INFO spawned: 'storagenode-updater' with pid 13
2022-06-30T15:59:22.805025391Z 2022-06-30 15:59:22,802 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-06-30T15:59:22.805134557Z 2022-06-30 15:59:22,803 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-06-30T15:59:22.805158668Z 2022-06-30 15:59:22,804 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-06-30T16:02:22.376122053Z 2022/06/30 16:02:22 failed to check for file existence: stat config/config.yaml: input/output error
2022-06-30T16:02:22.377624723Z 2022-06-30 16:02:22,376 INFO exited: storagenode-updater (exit status 1; not expected)
2022-06-30T16:02:22.384631487Z 2022-06-30 16:02:22,383 INFO spawned: 'storagenode-updater' with pid 28
2022-06-30T16:02:22.385933697Z 2022-06-30 16:02:22,385 WARN received SIGQUIT indicating exit request
2022-06-30T16:02:22.387514144Z 2022-06-30 16:02:22,386 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2022-06-30T16:02:25.393450166Z 2022-06-30 16:02:25,392 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2022-06-30T16:02:28.397581003Z 2022-06-30 16:02:28,397 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2022-06-30T16:02:31.402495943Z 2022-06-30 16:02:31,401 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
`
2

maybe this one worth to check?

2 Likes
3

@krusty77 learned anything? One of my nodes just quit this Saturday with this error.

4

can you figure out, if you have some access difficulties to the config.yaml?

5

Not really, I just re-imaged my node.

6

That may work in the case where the image has also been updated. This might not work if your node is getting an early release, and the image has not been updated yet. That, or the restart will just happen to you again later when the update is pushed again.

Update to 1.64.1 required two manual starts, without –restart unless-stopped set.

See also, Node repeatly stop every few hrs after did a update yesterday - #4 by Pascal51882