Today I woke up and my server was kind of frozen. I checked the logs and saw that my nodes were all offline and doing this all the time:
2023-10-17T04:41:06Z INFO Current binary version {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.88.3"}
2023-10-17T04:41:06Z INFO New version is being rolled out but hasn't made it to this node yet {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2023-10-17T04:56:05Z INFO Downloading versions. {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2023-10-17T04:56:07Z INFO Current binary version {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.88.3"}
2023-10-17T04:56:07Z INFO New version is being rolled out but hasn't made it to this node yet {"Process": "storagenode-updater", "Service": "storagenode"}
2023-10-17T04:56:07Z INFO Current binary version {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.88.3"}
2023-10-17T04:56:07Z INFO New version is being rolled out but hasn't made it to this node yet {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2023-10-17T05:11:05Z INFO Downloading versions. {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2023-10-17T05:11:06Z INFO Current binary version {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.88.3"}
2023-10-17T05:11:06Z INFO New version is being rolled out but hasn't made it to this node yet {"Process": "storagenode-updater", "Service": "storagenode"}
2023-10-17T05:11:06Z INFO Current binary version {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.88.3"}
2023-10-17T05:11:06Z INFO New version is being rolled out but hasn't made it to this node yet {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2023-10-17T05:26:05Z INFO Downloading versions. {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2023-10-17T05:26:06Z INFO Current binary version {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.88.3"}
2023-10-17T05:26:06Z INFO New version is being rolled out but hasn't made it to this node yet {"Process": "storagenode-updater", "Service": "storagenode"}
2023-10-17T05:26:06Z INFO Current binary version {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.88.3"}
2023-10-17T05:26:06Z INFO New version is being rolled out but hasn't made it to this node yet {"Process": "storagenode-updater", "Service": "storagenode-updater"}
Because all nodes were doing this, I think my server got overloaded:
Strangely, I am not able to stop the docker containers. I am also not able to shutdown the whole system. It just doesn’t do anything. Even kill -9 PID
didn’t work.