Serial error while moving

Node Operators troubleshooting
, ,
1

Hello!
I would like to ask for help.
I have been struggling for over a month with moving my node from shr raid to ext4 separate disk.
The server runs under Synology docker, I perform the migration with rsync.

It’s 11TB of data, I reduced the free space in the configuration so that I don’t receive new traffic.

My problem is that during the transfer the docker container stopped 6 times in the last two hours. The hard drive is a Seagate x16 type.

I’m attaching a log to see if that’s not the problem. CPU and memory utilization below 6% I/O waiting 60%-70%.

Rsync is what I use
screen rsync -aP /volume2/docker/storj/storage/blobs/ /volume3/shared/storj/storage/blobs/

How can I solve the process to survive the migration?
Thanks

LOG

2

I cannot access the log file, so I’m guessing that your node probably stops due to insufficient I/O operations left. The check that the node performs tests whether it is possible to read or write a small file within one minute—and failing to do so will shut down the node. You can tune parameters responsible for this check.

Some other ideas to reduce the interaction of the copy operation and node’s are:

  • running your rsync under ionice (though not sure if this will help on Synology) with sth like ionice -c Idle rsync …,
  • if you have a lot of RAM in your Synology unit (probably at least 12 GB in your case), it will help to use the rsync’s --no-inc-recursive flag; but if you don’t, this flag will slow down the copy operation even more,
  • temporarily disabling uploads by reducing your disk space allocation,
  • if you estimate the time left for the copy to run to be below few days, you can probably turn off the node altogether—downtime of few a days is recoverable, though the satellite might migrate some small amounts of data to other nodes.
3

Thanks
It may not be caused by the copying, because I stopped it, but the node periodically stops and restarts. Based on the log, it seems as if the process is fired due to the update.

2023-08-23T15:41:52.559455909Z stdout 2023-08-23T15:41:52Z INFO New version is being rolled out but hasn’t made it to this node yet {Process: storagenode-updater, Service: storagenode-updater}
2023-08-23T15:41:52.559397602Z stdout 2023-08-23T15:41:52Z INFO Current binary version {Process: storagenode-updater, Service: storagenode-updater, Version: v1.85.1}
2023-08-23T15:41:52.559283098Z stdout 2023-08-23T15:41:52Z INFO New version is being rolled out but hasn’t made it to this node yet {Process: storagenode-updater, Service: storagenode}
2023-08-23T15:41:52.511719448Z stdout 2023-08-23T15:41:52Z INFO Current binary version {Process: storagenode-updater, Service: storagenode, Version: v1.85.1}
2023-08-23T15:41:51.687292394Z stdout 2023-08-23 15:41:51,686 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-08-23T15:41:51.687082006Z stdout 2023-08-23 15:41:51,686 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-08-23T15:41:51.686806941Z stdout 2023-08-23 15:41:51,686 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-08-23T15:41:51.001578853Z stdout 2023-08-23T15:41:50Z INFO Downloading versions. {Process: storagenode-updater, Server Address: https://version.storj.io}
2023-08-23T15:41:50.994649312Z stdout 2023-08-23T15:41:50Z INFO Running on version {Process: storagenode-updater, Service: storagenode-updater, Version: v1.85.1}
2023-08-23T15:41:50.994590560Z stdout 2023-08-23T15:41:50Z INFO Anonymized tracing enabled {Process: storagenode-updater}
2023-08-23T15:41:50.994537793Z stdout 2023-08-23T15:41:50Z INFO Invalid configuration file value for key {Process: storagenode-updater, Key: log.stack}
2023-08-23T15:41:50.994484596Z stdout 2023-08-23T15:41:50Z INFO Invalid configuration file value for key {Process: storagenode-updater, Key: log.output}
2023-08-23T15:41:50.994423759Z stdout 2023-08-23T15:41:50Z INFO Invalid configuration file value for key {Process: storagenode-updater, Key: log.level}
2023-08-23T15:41:50.994369309Z stdout 2023-08-23T15:41:50Z INFO Invalid configuration file value for key {Process: storagenode-updater, Key: log.encoding}
2023-08-23T15:41:50.994311628Z stdout 2023-08-23T15:41:50Z INFO Invalid configuration file value for key {Process: storagenode-updater, Key: log.development}
2023-08-23T15:41:50.994257608Z stdout 2023-08-23T15:41:50Z INFO Invalid configuration file key {Process: storagenode-updater, Key: operator.email}
2023-08-23T15:41:50.994191855Z stdout 2023-08-23T15:41:50Z INFO Invalid configuration file key {Process: storagenode-updater, Key: console.address}
2023-08-23T15:41:50.994134531Z stdout 2023-08-23T15:41:50Z INFO Invalid configuration file key {Process: storagenode-updater, Key: storage.allocated-disk-space}
2023-08-23T15:41:50.994081244Z stdout 2023-08-23T15:41:50Z INFO Invalid configuration file key {Process: storagenode-updater, Key: server.private-address}
2023-08-23T15:41:50.994028700Z stdout 2023-08-23T15:41:50Z INFO Invalid configuration file key {Process: storagenode-updater, Key: server.address}
2023-08-23T15:41:50.993975104Z stdout 2023-08-23T15:41:50Z INFO Invalid configuration file key {Process: storagenode-updater, Key: healthcheck.enabled}
2023-08-23T15:41:50.993920717Z stdout 2023-08-23T15:41:50Z INFO Invalid configuration file key {Process: storagenode-updater, Key: contact.external-address}
2023-08-23T15:41:50.993863279Z stdout 2023-08-23T15:41:50Z INFO Invalid configuration file key {Process: storagenode-updater, Key: operator.wallet-features}
2023-08-23T15:41:50.993807667Z stdout 2023-08-23T15:41:50Z INFO Invalid configuration file key {Process: storagenode-updater, Key: filestore.write-buffer-size}
2023-08-23T15:41:50.993750772Z stdout 2023-08-23T15:41:50Z INFO Invalid configuration file key {Process: storagenode-updater, Key: pieces.write-prealloc-size}
2023-08-23T15:41:50.993695812Z stdout 2023-08-23T15:41:50Z INFO Invalid configuration file key {Process: storagenode-updater, Key: healthcheck.details}
2023-08-23T15:41:50.993635405Z stdout 2023-08-23T15:41:50Z INFO Invalid configuration file key {Process: storagenode-updater, Key: storage.allocated-bandwidth}
2023-08-23T15:41:50.993572739Z stdout 2023-08-23T15:41:50Z INFO Invalid configuration file key {Process: storagenode-updater, Key: operator.wallet}
2023-08-23T15:41:50.993239229Z stdout 2023-08-23T15:41:50Z INFO Configuration loaded {Process: storagenode-updater, Location: /app/config/config.yaml}
2023-08-23T15:41:50.993124245Z stdout 2023-08-23 15:41:50,571 INFO spawned: ‘storagenode-updater’ with pid 13
2023-08-23T15:41:50.993037524Z stdout 2023-08-23 15:41:50,569 INFO spawned: ‘storagenode’ with pid 12
2023-08-23T15:41:50.572460477Z stdout 2023-08-23 15:41:50,565 INFO spawned: ‘processes-exit-eventlistener’ with pid 11
2023-08-23T15:41:49.675877109Z stdout 2023-08-23 15:41:49,562 INFO supervisord started with pid 1
2023-08-23T15:41:49.667890568Z stdout 2023-08-23 15:41:49,549 INFO RPC interface ‘supervisor’ initialized
2023-08-23T15:41:49.463287678Z stdout 2023-08-23 15:41:49,463 INFO Set uid to user 0 succeeded
2023-08-23T15:41:36.511183687Z stdout 2023-08-23 15:41:36,511 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)
2023-08-23T15:41:36.510276633Z stdout 2023-08-23 15:41:36,510 INFO stopped: storagenode (terminated by SIGTERM)
2023-08-23T15:41:36.508117684Z stdout 2023-08-23 15:41:36,507 INFO stopped: storagenode-updater (exit status 0)
2023-08-23T15:41:36.504211083Z stdout 2023-08-23T15:41:36Z INFO Got a signal from the OS: terminated {Process: storagenode-updater}
2023-08-23T15:41:36.476579210Z stdout 2023-08-23 15:41:36,476 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2023-08-23T15:41:36.475681381Z stdout 2023-08-23 15:41:36,475 WARN received SIGQUIT indicating exit request
2023-08-23T15:41:36.098191344Z stdout 2023-08-23 15:41:36,097 INFO spawned: ‘storagenode’ with pid 118
2023-08-23T15:41:36.053954470Z stdout 2023-08-23 15:41:35,910 INFO exited: storagenode (exit status 1; not expected)
2023-08-23T15:41:31.415453395Z stdout 2023-08-23T15:34:51Z INFO New version is being rolled out but hasn’t made it to this node yet {Process: storagenode-updater, Service: storagenode-updater}
2023-08-23T15:34:51.247748430Z stdout 2023-08-23T15:34:51Z INFO Current binary version {Process: storagenode-updater, Service: storagenode-updater, Version: v1.85.1}
2023-08-23T15:34:50.206401541Z stdout 2023-08-23T15:34:50Z INFO New version is being rolled out but hasn’t made it to this node yet {Process: storagenode-updater, Service: storagenode}
2023-08-23T15:34:50.206198883Z stdout 2023-08-23T15:34:50Z INFO Current binary version {Process: storagenode-updater, Service: storagenode, Version: v1.85.1}
2023-08-23T15:34:49.149081650Z stdout 2023-08-23T15:34:49Z INFO Downloading versions. {Process: storagenode-updater, Server Address: https://version.storj.io}
2023-08-23T15:24:42.532272388Z stdout 2023-08-23T15:19:52Z INFO New version is being rolled out but hasn’t made it to this node yet {Process: storagenode-updater, Service: storagenode-updater}
2023-08-23T15:24:42.532198998Z stdout 2023-08-23T15:19:52Z INFO Current binary version {Process: storagenode-updater, Service: storagenode-updater, Version: v1.85.1}
2023-08-23T15:24:42.532090813Z stdout 2023-08-23T15:19:51Z INFO New version is being rolled out but hasn’t made it to this node yet {Process: storagenode-updater, Service: storagenode}
2023-08-23T15:19:51.291276019Z stdout 2023-08-23T15:19:51Z INFO Current binary version {Process: storagenode-updater, Service: storagenode, Version: v1.85.1}
2023-08-23T15:19:49.148511008Z stdout 2023-08-23T15:19:49Z INFO Downloading versions. {Process: storagenode-updater, Server Address: https://version.storj.io}
2023-08-23T15:04:50.557531759Z stdout 2023-08-23 15:04:50,557 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-08-23T15:04:50.557453504Z stdout 2023-08-23 15:04:50,557 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-08-23T15:04:50.557255880Z stdout 2023-08-23 15:04:50,556 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-08-23T15:04:49.555912559Z stdout 2023-08-23T15:04:49Z INFO New version is being rolled out but hasn’t made it to this node yet {Process: storagenode-updater, Service: storagenode-updater}
2023-08-23T15:04:49.555721327Z stdout 2023-08-23T15:04:49Z INFO Current binary version {Process: storagenode-updater, Service: storagenode-updater, Version: v1.85.1}
2023-08-23T15:04:49.546080186Z stdout 2023-08-23T15:04:49Z INFO New version is being rolled out but hasn’t made it to this node yet {Process: storagenode-updater, Service: storagenode}
2023-08-23T15:04:49.545904936Z stdout 2023-08-23T15:04:49Z INFO Current binary version {Process: storagenode-updater, Service: storagenode, Version: v1.85.1}
2023-08-23T15:04:49.170427867Z stdout 2023-08-23T15:04:49Z INFO Downloading versions. {Process: storagenode-updater, Server Address: https://version.storj.io}
2023-08-23T15:04:49.170355465Z stdout 2023-08-23T15:04:49Z INFO Running on version {Process: storagenode-updater, Service: storagenode-updater, Version: v1.85.1}
2023-08-23T15:04:49.170291887Z stdout 2023-08-23T15:04:49Z INFO Anonymized tracing enabled {Process: storagenode-updater}
2023-08-23T15:04:49.170215853Z stdout 2023-08-23T15:04:49Z INFO Invalid configuration file value for key {Process: storagenode-updater, Key: log.level}
2023-08-23T15:04:49.170163364Z stdout 2023-08-23T15:04:49Z INFO Invalid configuration file key {Process: storagenode-updater, Key: server.address}
2023-08-23T15:04:49.170100373Z stdout 2023-08-23T15:04:49Z INFO Invalid configuration file key {Process: storagenode-updater, Key: operator.wallet-features}
2023-08-23T15:04:49.170035017Z stdout 2023-08-23T15:04:49Z INFO Invalid configuration file key {Process: storagenode-updater, Key: healthcheck.details}
2023-08-23T15:04:49.169969105Z stdout 2023-08-23T15:04:49Z INFO Invalid configuration file key {Process: storagenode-updater, Key: filestore.write-buffer-size}
2023-08-23T15:04:49.169906048Z stdout 2023-08-23T15:04:49Z INFO Invalid configuration file key {Process: storagenode-updater, Key: operator.email}
2023-08-23T15:04:49.169842818Z stdout 2023-08-23T15:04:49Z INFO Invalid configuration file key {Process: storagenode-updater, Key: console.address}
2023-08-23T15:04:49.169785949Z stdout 2023-08-23T15:04:49Z INFO Invalid configuration file key {Process: storagenode-updater, Key: pieces.write-prealloc-size}
2023-08-23T15:04:49.169719587Z stdout 2023-08-23T15:04:49Z INFO Invalid configuration file key {Process: storagenode-updater, Key: contact.external-address}
2023-08-23T15:04:49.169652654Z stdout 2023-08-23T15:04:49Z INFO Invalid configuration file key {Process: storagenode-updater, Key: storage.allocated-disk-space}
2023-08-23T15:04:49.169587356Z stdout 2023-08-23T15:04:49Z INFO Invalid configuration file key {Process: storagenode-updater, Key: operator.wallet}
2023-08-23T15:04:49.169517713Z stdout 2023-08-23T15:04:49Z INFO Invalid configuration file key {Process: storagenode-updater, Key: storage.allocated-bandwidth}
2023-08-23T15:04:49.169450608Z stdout 2023-08-23T15:04:49Z INFO Invalid configuration file key {Process: storagenode-updater, Key: healthcheck.enabled}
2023-08-23T15:04:49.169377117Z stdout 2023-08-23T15:04:49Z INFO Invalid configuration file key {Process: storagenode-updater, Key: server.private-address}
2023-08-23T15:04:49.169055256Z stdout 2023-08-23T15:04:49Z INFO Configuration loaded {Process: storagenode-updater, Location: /app/config/config.yaml}
2023-08-23T15:04:49.168921443Z stdout 2023-08-23 15:04:49,064 INFO spawned: ‘storagenode-updater’ with pid 13
2023-08-23T15:04:49.168827423Z stdout 2023-08-23 15:04:49,062 INFO spawned: ‘storagenode’ with pid 12
2023-08-23T15:04:49.065790511Z stdout 2023-08-23 15:04:49,060 INFO spawned: ‘processes-exit-eventlistener’ with pid 11
2023-08-23T15:04:48.057780200Z stdout 2023-08-23 15:04:48,057 INFO supervisord started with pid 1
2023-08-23T15:04:48.018238453Z stdout 2023-08-23 15:04:48,017 INFO RPC interface ‘supervisor’ initialized
2023-08-23T15:04:47.897110604Z stdout 2023-08-23 15:04:47,896 INFO Set uid to user 0 succeeded
2023-08-23T15:04:36.599894979Z stdout 2023-08-23 15:04:36,599 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)
2023-08-23T15:04:36.598718065Z stdout 2023-08-23 15:04:36,598 INFO stopped: storagenode (terminated by SIGKILL)
2023-08-23T15:04:36.568191115Z stdout 2023-08-23 15:04:36,567 WARN killing ‘storagenode’ (128) with SIGKILL
2023-08-23T15:04:35.566770338Z stdout 2023-08-23 15:04:35,566 INFO waiting for storagenode, processes-exit-eventlistener to die
2023-08-23T15:04:32.563301730Z stdout 2023-08-23 15:04:32,562 INFO waiting for storagenode, processes-exit-eventlistener to die
2023-08-23T15:04:29.560047320Z stdout 2023-08-23 15:04:29,559 INFO waiting for storagenode, processes-exit-eventlistener to die
2023-08-23T15:04:26.692081554Z stdout 2023-08-23 15:04:26,557 INFO stopped: storagenode-updater (exit status 0)
2023-08-23T15:04:26.692005106Z stdout 2023-08-23T15:04:26Z INFO Got a signal from the OS: terminated {Process: storagenode-updater}
2023-08-23T15:04:26.691932290Z stdout 2023-08-23 15:04:26,556 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2023-08-23T15:04:26.691833049Z stdout 2023-08-23 15:04:26,306 WARN received SIGQUIT indicating exit request
2023-08-23T15:04:25.896570870Z stdout 2023-08-23 15:04:25,896 INFO spawned: ‘storagenode’ with pid 128
2023-08-23T15:04:24.786201421Z stdout 2023-08-23 15:04:24,688 INFO exited: storagenode (exit status 1; not expected)
2023-08-23T15:04:03.605606286Z stdout 2023-08-23T15:02:40Z INFO New version is being rolled out but hasn’t made it to this node yet {Process: storagenode-updater, Service: storagenode-updater}
2023-08-23T15:04:03.605551686Z stdout 2023-08-23T15:02:40Z INFO Current binary version {Process: storagenode-updater, Service: storagenode-updater, Version: v1.85.1}
2023-08-23T15:04:03.605476793Z stdout 2023-08-23T15:02:40Z INFO New version is being rolled out but hasn’t made it to this node yet {Process: storagenode-updater, Service: storagenode}
2023-08-23T15:04:03.605380268Z stdout 2023-08-23T15:02:40Z INFO Current binary version {Process: storagenode-updater, Service: storagenode, Version: v1.85.1}
2023-08-23T15:02:38.532023711Z stdout 2023-08-23T15:02:37Z INFO Downloading versions. {Process: storagenode-updater, Server Address: https://version.storj.io}
2023-08-23T14:47:39.295125286Z stdout 2023-08-23T14:47:39Z INFO New version is being rolled out but hasn’t made it to this node yet {Process: storagenode-updater, Service: storagenode-updater}
2023-08-23T14:47:39.294960631Z stdout 2023-08-23T14:47:39Z INFO Current binary version {Process: storagenode-updater, Service: storagenode-updater, Version: v1.85.1}
2023-08-23T14:47:39.285722811Z stdout 2023-08-23T14:47:39Z INFO New version is being rolled out but hasn’t made it to this node yet {Process: storagenode-updater, Service: storagenode}
2023-08-23T14:47:39.285503431Z stdout 2023-08-23T14:47:39Z INFO Current binary version {Process: storagenode-updater, Service: storagenode, Version: v1.85.1}
2023-08-23T14:47:37.904881621Z stdout 2023-08-23T14:47:37Z INFO Downloading versions. {Process: storagenode-updater, Server Address: https://version.storj.io}
2023-08-23T14:32:39.225016822Z stdout 2023-08-23 14:32:39,224 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-08-23T14:32:39.224939120Z stdout 2023-08-23 14:32:39,224 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-08-23T14:32:39.224714512Z stdout 2023-08-23 14:32:39,224 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-08-23T14:32:38.241587969Z stdout 2023-08-23T14:32:38Z INFO New version is being rolled out but hasn’t made it to this node yet {Process: storagenode-updater, Service: storagenode-updater}
2023-08-23T14:32:38.223299751Z stdout 2023-08-23T14:32:38Z INFO Current binary version {Process: storagenode-updater, Service: storagenode-updater, Version: v1.85.1}
2023-08-23T14:32:38.214268421Z stdout 2023-08-23T14:32:38Z INFO New version is being rolled out but hasn’t made it to this node yet {Process: storagenode-updater, Service: storagenode}
2023-08-23T14:32:38.214083865Z stdout 2023-08-23T14:32:38Z INFO Current binary version {Process: storagenode-updater, Service: storagenode, Version: v1.85.1}
2023-08-23T14:32:37.862468072Z stdout 2023-08-23T14:32:37Z INFO Downloading versions. {Process: storagenode-updater, Server Address: https://version.storj.io}
2023-08-23T14:32:37.862407496Z stdout 2023-08-23T14:32:37Z INFO Running on version {Process: storagenode-updater, Service: storagenode-updater, Version: v1.85.1}
2023-08-23T14:32:37.862350846Z stdout 2023-08-23T14:32:37Z INFO Anonymized tracing enabled {Process: storagenode-updater}
2023-08-23T14:32:37.862296978Z stdout 2023-08-23T14:32:37Z INFO Invalid configuration file value for key {Process: storagenode-updater, Key: log.output}
2023-08-23T14:32:37.862241381Z stdout 2023-08-23T14:32:37Z INFO Invalid configuration file value for key {Process: storagenode-updater, Key: log.level}
2023-08-23T14:32:37.862172875Z stdout 2023-08-23T14:32:37Z INFO Invalid configuration file value for key {Process: storagenode-updater, Key: log.development}
2023-08-23T14:32:37.862116579Z stdout 2023-08-23T14:32:37Z INFO Invalid configuration file value for key {Process: storagenode-updater, Key: log.caller}
2023-08-23T14:32:37.862063027Z stdout 2023-08-23T14:32:37Z INFO Invalid configuration file key {Process: storagenode-updater, Key: server.private-address}
2023-08-23T14:32:37.862008171Z stdout 2023-08-23T14:32:37Z INFO Invalid configuration file key {Process: storagenode-updater, Key: filestore.write-buffer-size}
2023-08-23T14:32:37.861956609Z stdout 2023-08-23T14:32:37Z INFO Invalid configuration file key {Process: storagenode-updater, Key: operator.wallet}
2023-08-23T14:32:37.861903656Z stdout 2023-08-23T14:32:37Z INFO Invalid configuration file key {Process: storagenode-updater, Key: healthcheck.enabled}
2023-08-23T14:32:37.861849942Z stdout 2023-08-23T14:32:37Z INFO Invalid configuration file key {Process: storagenode-updater, Key: healthcheck.details}
2023-08-23T14:32:37.861795628Z stdout 2023-08-23T14:32:37Z INFO Invalid configuration file key {Process: storagenode-updater, Key: storage.allocated-bandwidth}
2023-08-23T14:32:37.861741891Z stdout 2023-08-23T14:32:37Z INFO Invalid configuration file key {Process: storagenode-updater, Key: server.address}
2023-08-23T14:32:37.861685858Z stdout 2023-08-23T14:32:37Z INFO Invalid configuration file key {Process: storagenode-updater, Key: operator.wallet-features}
2023-08-23T14:32:37.861560408Z stdout 2023-08-23T14:32:37Z INFO Invalid configuration file key {Process: storagenode-updater, Key: contact.external-address}
2023-08-23T14:32:37.861506327Z stdout 2023-08-23T14:32:37Z INFO Invalid configuration file key {Process: storagenode-updater, Key: operator.email}
2023-08-23T14:32:37.861448653Z stdout 2023-08-23T14:32:37Z INFO Invalid configuration file key {Process: storagenode-updater, Key: pieces.write-prealloc-size}
2023-08-23T14:32:37.861394466Z stdout 2023-08-23T14:32:37Z INFO Invalid configuration file key {Process: storagenode-updater, Key: console.address}
2023-08-23T14:32:37.861331888Z stdout 2023-08-23T14:32:37Z INFO Invalid configuration file key {Process: storagenode-updater, Key: storage.allocated-disk-space}
2023-08-23T14:32:37.861269148Z stdout 2023-08-23T14:32:37Z INFO Configuration loaded {Process: storagenode-updater, Location: /app/config/config.yaml}
2023-08-23T14:32:37.861200378Z stdout 2023-08-23 14:32:37,726 INFO spawned: ‘storagenode-updater’ with pid 13
2023-08-23T14:32:37.861115008Z stdout 2023-08-23 14:32:37,722 INFO spawned: ‘storagenode’ with pid 12
2023-08-23T14:32:37.722956818Z stdout 2023-08-23 14:32:37,719 INFO spawned: ‘processes-exit-eventlistener’ with pid 11
2023-08-23T14:32:36.716669246Z stdout 2023-08-23 14:32:36,716 INFO supervisord started with pid 1
2023-08-23T14:32:36.701318833Z stdout 2023-08-23 14:32:36,701 INFO RPC interface ‘supervisor’ initialized
2023-08-23T14:32:36.563443574Z stdout 2023-08-23 14:32:36,563 INFO Set uid to user 0 succeeded
2023-08-23T14:32:25.588295716Z stdout 2023-08-23 14:32:25,588 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)
2023-08-23T14:32:25.586576637Z stdout 2023-08-23 14:32:25,586 INFO stopped: storagenode (terminated by SIGKILL)
2023-08-23T14:32:25.557995797Z stdout 2023-08-23 14:32:25,557 WARN killing ‘storagenode’ (88) with SIGKILL
2023-08-23T14:32:24.556694335Z stdout 2023-08-23 14:32:24,556 INFO waiting for storagenode, processes-exit-eventlistener to die
2023-08-23T14:32:21.554316975Z stdout 2023-08-23 14:32:21,554 INFO waiting for storagenode, processes-exit-eventlistener to die
2023-08-23T14:32:18.550326591Z stdout 2023-08-23 14:32:18,550 INFO waiting for storagenode, processes-exit-eventlistener to die
2023-08-23T14:32:15.545939853Z stdout 2023-08-23 14:32:15,545 INFO stopped: storagenode-updater (exit status 0)
2023-08-23T14:32:15.544081964Z stdout 2023-08-23T14:32:15Z INFO Got a signal from the OS: terminated {Process: storagenode-updater}
2023-08-23T14:32:15.536515806Z stdout 2023-08-23 14:32:15,536 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2023-08-23T14:32:15.522199612Z stdout 2023-08-23 14:32:15,379 WARN received SIGQUIT indicating exit request
2023-08-23T14:32:14.745123540Z stdout 2023-08-23 14:32:14,744 INFO spawned: ‘storagenode’ with pid 88
2023-08-23T14:32:13.675035956Z stdout 2023-08-23 14:32:13,646 INFO exited: storagenode (exit status 1; not expected)
2023-08-23T14:32:05.004147723Z stdout 2023-08-23T14:25:02Z INFO New version is being rolled out but hasn’t made it to this node yet {Process: storagenode-updater, Service: storagenode-updater}
2023-08-23T14:32:05.004086460Z stdout 2023-08-23T14:25:02Z INFO Current binary version {Process: storagenode-updater, Service: storagenode-updater, Version: v1.85.1}
2023-08-23T14:32:05.003968597Z stdout 2023-08-23T14:25:01Z INFO New version is being rolled out but hasn’t made it to this node yet {Process: storagenode-updater, Service: storagenode}
2023-08-23T14:25:01.628262616Z stdout 2023-08-23T14:25:01Z INFO Current binary version {Process: storagenode-updater, Service: storagenode, Version: v1.85.1}
2023-08-23T14:24:58.161257409Z stdout 2023-08-23T14:24:57Z INFO Downloading versions. {Process: storagenode-updater, Server Address: https://version.storj.io}
2023-08-23T14:09:59.342586831Z stdout 2023-08-23T14:09:59Z INFO Version is up to date {Process: storagenode-updater, Service: storagenode-updater}
2023-08-23T14:09:59.342394032Z stdout 2023-08-23T14:09:59Z INFO Current binary version {Process: storagenode-updater, Service: storagenode-updater, Version: v1.85.1}
2023-08-23T14:09:59.333026381Z stdout 2023-08-23T14:09:59Z INFO Version is up to date {Process: storagenode-updater, Service: storagenode}
2023-08-23T14:09:59.332853662Z stdout 2023-08-23T14:09:59Z INFO Current binary version {Process: storagenode-updater, Service: storagenode, Version: v1.85.1}
2023-08-23T14:09:58.972098181Z stdout 2023-08-23 14:09:58,971 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-08-23T14:09:58.971903899Z stdout 2023-08-23 14:09:58,971 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-08-23T14:09:58.971185236Z stdout 2023-08-23 14:09:58,970 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-08-23T14:09:57.970513788Z stdout 2023-08-23T14:09:57Z INFO Downloading versions. {Process: storagenode-updater, Server Address: https://version.storj.io}
2023-08-23T14:09:57.970444725Z stdout 2023-08-23T14:09:57Z INFO Running on version {Process: storagenode-updater, Service: storagenode-updater, Version: v1.85.1}
2023-08-23T14:09:57.970275266Z stdout 2023-08-23T14:09:57Z INFO Anonymized tracing enabled {Process: storagenode-updater}
2023-08-23T14:09:57.969913542Z stdout 2023-08-23T14:09:57Z INFO Invalid configuration file key {Process: storagenode-updater, Key: server.address}
2023-08-23T14:09:57.969857596Z stdout 2023-08-23T14:09:57Z INFO Invalid configuration file key {Process: storagenode-updater, Key: pieces.write-prealloc-size}
2023-08-23T14:09:57.969799678Z stdout 2023-08-23T14:09:57Z INFO Invalid configuration file key {Process: storagenode-updater, Key: server.private-address}
2023-08-23T14:09:57.969743027Z stdout 2023-08-23T14:09:57Z INFO Invalid configuration file key {Process: storagenode-updater, Key: console.address}
2023-08-23T14:09:57.969680574Z stdout 2023-08-23T14:09:57Z INFO Invalid configuration file key {Process: storagenode-updater, Key: healthcheck.details}
2023-08-23T14:09:57.969625493Z stdout 2023-08-23T14:09:57Z INFO Invalid configuration file key {Process: storagenode-updater, Key: storage.allocated-bandwidth}
2023-08-23T14:09:57.969564142Z stdout 2023-08-23T14:09:57Z INFO Invalid configuration file key {Process: storagenode-updater, Key: operator.wallet-features}
2023-08-23T14:09:57.969510278Z stdout 2023-08-23T14:09:57Z INFO Invalid configuration file key {Process: storagenode-updater, Key: operator.wallet}
2023-08-23T14:09:57.969452228Z stdout 2023-08-23T14:09:57Z INFO Invalid configuration file key {Process: storagenode-updater, Key: storage.allocated-disk-space}
2023-08-23T14:09:57.969396745Z stdout 2023-08-23T14:09:57Z INFO Invalid configuration file key {Process: storagenode-updater, Key: healthcheck.enabled}
2023-08-23T14:09:57.969336095Z stdout 2023-08-23T14:09:57Z INFO Invalid configuration file key {Process: storagenode-updater, Key: filestore.write-buffer-size}
2023-08-23T14:09:57.969275874Z stdout 2023-08-23T14:09:57Z INFO Invalid configuration file key {Process: storagenode-updater, Key: operator.email}
2023-08-23T14:09:57.969119128Z stdout 2023-08-23T14:09:57Z INFO Invalid configuration file key {Process: storagenode-updater, Key: contact.external-address}
2023-08-23T14:09:57.968940849Z stdout 2023-08-23T14:09:57Z INFO Configuration loaded {Process: storagenode-updater, Location: /app/config/config.yaml}
2023-08-23T14:09:57.854944002Z stdout 2023-08-23 14:09:57,854 INFO spawned: ‘storagenode-updater’ with pid 14
2023-08-23T14:09:57.854644596Z stdout 2023-08-23 14:09:57,851 INFO spawned: ‘storagenode’ with pid 13
2023-08-23T14:09:57.850608311Z stdout 2023-08-23 14:09:57,848 INFO spawned: ‘processes-exit-eventlistener’ with pid 12
2023-08-23T14:09:56.845991170Z stdout 2023-08-23 14:09:56,845 INFO supervisord started with pid 1
2023-08-23T14:09:56.814176133Z stdout 2023-08-23 14:09:56,813 INFO RPC interface ‘supervisor’ initialized
2023-08-23T14:09:56.728920426Z stdout 2023-08-23 14:09:56,728 INFO Set uid to user 0 succeeded
2023-08-23T14:09:41.596893578Z stdout 2023-08-23 14:09:41,596 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)
2023-08-23T14:09:41.577898896Z stdout 2023-08-23 14:09:41,577 INFO stopped: storagenode (terminated by SIGTERM)
2023-08-23T14:09:41.564976042Z stdout 2023-08-23 14:09:41,564 INFO stopped: storagenode-updater (exit status 0)
2023-08-23T14:09:41.562461435Z stdout 2023-08-23T14:09:41Z INFO Got a signal from the OS: terminated {Process: storagenode-updater}
2023-08-23T14:09:41.430178818Z stdout 2023-08-23 14:09:41,429 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2023-08-23T14:09:41.428874417Z stdout 2023-08-23 14:09:41,428 WARN received SIGQUIT indicating exit request
2023-08-23T14:09:41.262589907Z stdout 2023-08-23 14:09:41,262 INFO spawned: ‘storagenode’ with pid 92
5

Unlikely. Seems you redirected logs to the file, but checking logs with docker logs instead of

tail /volume2/docker/storj/storagenode.log

(or what is your path to the logs)
When you redirect logs to the file, docker logs storagenode will show only logs from the storagenode-updater service, the node’s logs will be added to the specified file.

However, I believe that @Toyoo is right and your node is suffering from a slow disk subsystem during copy.

6

Ok, it doesn’t look like what I was thinking about. Sorry, can’t help more.

7

I don’t understand, but this week rsync is running without any problems or stops and in a few days it will complete the first big sync.
Will the other syncs be this slow? If I read correctly, I should run it next to the storj container stopped with the -delete option during the last run. Then the container must be created again with the parameters of the new volume.
I am very worried that it will be successful, my node has been running for almost 3 years :slight_smile:

earlier I tried to increase the file size in the config. When I moved from windows to docker, I forgot this and it remained on default… The synchronization progressed significantly faster on those folders than on folders with many small files.

8

That’s right, you need to stop and remove the container, then run rsync one more time but with an additional --delete option, it will remove files from the destination location, which were removed from the source location, the operational temporary files of the databases in particular.
Without that option you may end with corrupted databases, so please run it before starting a new container with the changed storage location.

9

thank you very much
the last sync before --delete is running, I hope it will be faster with a stopped node, so that there is no DQ due to a lot of offline time.

I noticed something interesting while checking the folders:
The owners and permissions on the files are different. Would it be worth running a chown for the entire share as well?

storj
storj799×809 204 KB

10

Of course you need to change permissions to the user who running docker especially if you use --user $(id -u):$(id -g) in the docker run command.
The root one can be here if you run you docker with sudo. It’s recommended to configure it to do not use sudo: Linux post-installation steps for Docker Engine | Docker Docs

11

Thanks to everyone for the advice, the migration was successful.
The last synchronization took a few days, then this morning I recreated the container with the new mount parameters and it works. Of course, the online score has dropped, but I don’t see anything wrong. Until now, the IO wait under btrfs with ssd cache was 30%, now under ext4 without cache it is 6%!
I only have to reinstall the docker application and I can get rid of the remaining btrfs volume.
I’m not touching the file permissions right now, I want to see how it works for a few days.

1 Like