Bandwidthdb: table bandwidth_usage has no column named action

Node Operators troubleshooting
, , ,
1

My node upgraded to 1.104.5 and I now get this error in a loop:

"errorVerbose": "bandwidthdb: table bandwidth_usage has no column named action\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:916\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:521\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:167\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:109\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:157\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-05-17T13:42:42Z    INFO    piecestore      uploaded        {"Process": "storagenode", "Piece ID": "IRVKXLNL3JEWWUCRAZ6H27C5OH6IKSFM3UVHQLPXMGCD6AW727GA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "10.218.0.1:38262", "Size": 512}
2024-05-17T13:42:42Z    ERROR   piecestore      failed to add bandwidth usage   {"Process": "storagenode", "error": "bandwidthdb: table bandwidth_usage has no column named action", "errorVerbose": "bandwidthdb: table bandwidth_usage has no column named action\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:916\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:546\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:167\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:109\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:157\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}

It’s a docker node that was previously on v. 1.102.3. Does someone else also have this issue?

2

I am not able to restart the node:

2024-05-17 14:37:12 (53.6 MB/s) - '/tmp/storagenode.zip' saved [18624672/18624672]

2024-05-17 14:37:13,320 INFO Set uid to user 0 succeeded
2024-05-17 14:37:13,327 INFO RPC interface 'supervisor' initialized
2024-05-17 14:37:13,328 INFO supervisord started with pid 1
2024-05-17 14:37:14,330 INFO spawned: 'processes-exit-eventlistener' with pid 54
2024-05-17 14:37:14,331 INFO spawned: 'storagenode' with pid 55
2024-05-17 14:37:14,332 INFO spawned: 'storagenode-updater' with pid 56
2024-05-17T14:37:14Z    INFO    Configuration loaded    {"Process": "storagenode-updater", "Location": "/app/config/config.yaml"}
2024-05-17T14:37:14Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-bandwidth"}
2024-05-17T14:37:14Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.address"}
2024-05-17T14:37:14Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-disk-space"}
2024-05-17T14:37:14Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage2.database-dir"}
2024-05-17T14:37:14Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.email"}
2024-05-17T14:37:14Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.wallet-features"}
2024-05-17T14:37:14Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.wallet"}
2024-05-17T14:37:14Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.private-address"}
2024-05-17T14:37:14Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "contact.external-address"}
2024-05-17T14:37:14Z    INFO    Anonymized tracing enabled      {"Process": "storagenode-updater"}
2024-05-17T14:37:14Z    INFO    Running on version      {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.102.3"}
2024-05-17T14:37:14Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-05-17T14:37:14Z    INFO    Configuration loaded    {"Process": "storagenode", "Location": "/app/config/config.yaml"}
2024-05-17T14:37:14Z    INFO    Anonymized tracing enabled      {"Process": "storagenode"}
2024-05-17T14:37:14Z    INFO    Operator email  {"Process": "storagenode", "Address": "******"}
2024-05-17T14:37:14Z    INFO    Operator wallet {"Process": "storagenode", "Address": "********"}
2024-05-17T14:37:14Z    INFO    server  existing kernel support for server-side tcp fast open detected  {"Process": "storagenode"}
2024-05-17T14:37:14Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.102.3"}
2024-05-17T14:37:14Z    INFO    New version is being rolled out but hasn't made it to this node yet     {"Process": "storagenode-updater", "Service": "storagenode"}
2024-05-17T14:37:14Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.102.3"}
2024-05-17T14:37:14Z    INFO    New version is being rolled out but hasn't made it to this node yet     {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-05-17T14:37:15Z    INFO    Telemetry enabled       {"Process": "storagenode", "instance ID": "1sb77XbBuEiGorcyK5XbtxNuQZKSy3zR3z337ENrv8XcrMCUCf"}
2024-05-17T14:37:15Z    INFO    Event collection enabled        {"Process": "storagenode", "instance ID": "1sb77XbBuEiGorcyK5XbtxNuQZKSy3zR3z337ENrv8XcrMCUCf"}
2024-05-17T14:37:15Z    INFO    db.migration    Database Version        {"Process": "storagenode", "version": 56}
2024-05-17 14:37:16,163 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-05-17 14:37:16,163 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-05-17 14:37:16,163 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-05-17T14:37:41Z    ERROR   failure during run      {"Process": "storagenode", "error": "Error during preflight check for storagenode databases: preflight: database \"bandwidth\": expected schema does not match actual:   &dbschema.Schema{\n  \tTables: []*dbschema.Table{\n  \t\t&{\n  \t\t\tName: \"bandwidth_usage\",\n  \t\t\tColumns: []*dbschema.Column{\n  \t\t\t\t&{\n- \t\t\t\t\tName:       \"action\",\n+ \t\t\t\t\tName:       \"delete_total\",\n- \t\t\t\t\tType:       \"INTEGER\",\n+ \t\t\t\t\tType:       \"BIGINT\",\n- \t\t\t\t\tIsNullable: false,\n+ \t\t\t\t\tIsNullable: true,\n  \t\t\t\t\tDefault:    \"\",\n  \t\t\t\t\tReference:  nil,\n  \t\t\t\t},\n  \t\t\t\t&{\n- \t\t\t\t\tName:       \"amount\",\n+ \t\t\t\t\tName:       \"get_audit_total\",\n  \t\t\t\t\tType:       \"BIGINT\",\n- \t\t\t\t\tIsNullable: false,\n+ \t\t\t\t\tIsNullable: true,\n  \t\t\t\t\tDefault:    \"\",\n  \t\t\t\t\tReference:  nil,\n  \t\t\t\t},\n  \t\t\t\t&{\n- \t\t\t\t\tName:       \"created_at\",\n+ \t\t\t\t\tName:       \"get_repair_total\",\n- \t\t\t\t\tType:       \"TIMESTAMP\",\n+ \t\t\t\t\tType:       \"BIGINT\",\n- \t\t\t\t\tIsNullable: false,\n+ \t\t\t\t\tIsNullable: true,\n  \t\t\t\t\tDefault:    \"\",\n  \t\t\t\t\tReference:  nil,\n  \t\t\t\t},\n+ \t\t\t\t(\n+ \t\t\t\t\ts\"\"\"\n+ \t\t\t\t\tName: get_total\n+ \t\t\t\t\tType: BIGINT\n+ \t\t\t\t\tNullable: true\n+ \t\t\t\t\tDefault: \"\"\n+ \t\t\t\t\tReference: nil\n+ \t\t\t\t\ts\"\"\"\n+ \t\t\t\t),\n+ \t\t\t\t(\n+ \t\t\t\t\ts\"\"\"\n+ \t\t\t\t\tName: interval_start\n+ \t\t\t\t\tType: TIMESTAMP\n+ \t\t\t\t\tNullable: false\n+ \t\t\t\t\tDefault: \"\"\n+ \t\t\t\t\tReference: nil\n+ \t\t\t\t\ts\"\"\"\n+ \t\t\t\t),\n+ \t\t\t\t(\n+ \t\t\t\t\ts\"\"\"\n+ \t\t\t\t\tName: put_repair_total\n+ \t\t\t\t\tType: BIGINT\n+ \t\t\t\t\tNullable: true\n+ \t\t\t\t\tDefault: \"\"\n+ \t\t\t\t\tReference: nil\n+ \t\t\t\t\ts\"\"\"\n+ \t\t\t\t),\n+ \t\t\t\t(\n+ \t\t\t\t\ts\"\"\"\n+ \t\t\t\t\tName: put_total\n+ \t\t\t\t\tType: BIGINT\n+ \t\t\t\t\tNullable: true\n+ \t\t\t\t\tDefault: \"\"\n+ \t\t\t\t\tReference: nil\n+ \t\t\t\t\ts\"\"\"\n+ \t\t\t\t),\n  \t\t\t\t&{Name: \"satellite_id\", Type: \"BLOB\"},\n  \t\t\t},\n- \t\t\tPrimaryKey: nil,\n+ \t\t\tPrimaryKey: []string{\"interval_start\", \"satellite_id\"},\n  \t\t\tUnique:     nil,\n  \t\t\tChecks:     nil,\n  \t\t},\n- \t\t(\n- \t\t\ts\"\"\"\n- \t\t\tName: bandwidth_usage_rollups\n- \t\t\tColumns:\n- \t\t\t\tName: action\n- \t\t\t\tType: INTEGER\n- \t\t\t\tNullable: false\n- \t\t\t\tDefault: \"\"\n- \t\t\t\tReference: nil\n- \t\t\t\tName: amount\n- \t\t\t\tType: BIGINT\n- \t\t\t\tNullable: false\n- \t\t\t\tDefault: \"\"\n- \t\t\t\tReference: nil\n- \t\t\t\tName: interval_start\n- \t\t\t\tType: TIMESTAMP\n- \t\t\t\tNullable: false\n- \t\t\t\tDefault: \"\"\n- \t\t\t\tReference: nil\n- \t\t\t\tName: satellite_id\n- \t\t\t\tType: BLOB\n- \t\t\t\tNullable: false\n- \t\t\t\tDefault: \"\"\n- \t\t\t\tReference: nil\n- \t\t\tPrimaryKey: action interval_start satellite_id\n- \t\t\tUniques:\n- \t\t\t\t\n- \t\t\tChecks:\n- \t\t\t\t\t\n- \t\t\ts\"\"\"\n- \t\t),\n  \t},\n- \tIndexes: []*dbschema.Index{\n- \t\ts`Index<Table: bandwidth_usage, Name: idx_bandwidth_usage_created, Columns: created_at, Unique: false, Partial: \"\">`,\n- \t\ts`Index<Table: bandwidth_usage, Name: idx_bandwidth_usage_satellite, Columns: satellite_id, Unique: false, Partial: \"\">`,\n- \t},\n+ \tIndexes:   nil,\n  \tSequences: nil,\n  }\n\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).preflight:446\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).Preflight:393\n\tmain.cmdRun:113\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:267", "errorVerbose": "Error during preflight check for storagenode databases: preflight: database \"bandwidth\": expected schema does not match actual:   &dbschema.Schema{\n  \tTables: []*dbschema.Table{\n  \t\t&{\n  \t\t\tName: \"bandwidth_usage\",\n  \t\t\tColumns: []*dbschema.Column{\n  \t\t\t\t&{\n- \t\t\t\t\tName:       \"action\",\n+ \t\t\t\t\tName:       \"delete_total\",\n- \t\t\t\t\tType:       \"INTEGER\",\n+ \t\t\t\t\tType:       \"BIGINT\",\n- \t\t\t\t\tIsNullable: false,\n+ \t\t\t\t\tIsNullable: true,\n  \t\t\t\t\tDefault:    \"\",\n  \t\t\t\t\tReference:  nil,\n  \t\t\t\t},\n  \t\t\t\t&{\n- \t\t\t\t\tName:       \"amount\",\n+ \t\t\t\t\tName:       \"get_audit_total\",\n  \t\t\t\t\tType:       \"BIGINT\",\n- \t\t\t\t\tIsNullable: false,\n+ \t\t\t\t\tIsNullable: true,\n  \t\t\t\t\tDefault:    \"\",\n  \t\t\t\t\tReference:  nil,\n  \t\t\t\t},\n  \t\t\t\t&{\n- \t\t\t\t\tName:       \"created_at\",\n+ \t\t\t\t\tName:       \"get_repair_total\",\n- \t\t\t\t\tType:       \"TIMESTAMP\",\n+ \t\t\t\t\tType:       \"BIGINT\",\n- \t\t\t\t\tIsNullable: false,\n+ \t\t\t\t\tIsNullable: true,\n  \t\t\t\t\tDefault:    \"\",\n  \t\t\t\t\tReference:  nil,\n  \t\t\t\t},\n+ \t\t\t\t(\n+ \t\t\t\t\ts\"\"\"\n+ \t\t\t\t\tName: get_total\n+ \t\t\t\t\tType: BIGINT\n+ \t\t\t\t\tNullable: true\n+ \t\t\t\t\tDefault: \"\"\n+ \t\t\t\t\tReference: nil\n+ \t\t\t\t\ts\"\"\"\n+ \t\t\t\t),\n+ \t\t\t\t(\n+ \t\t\t\t\ts\"\"\"\n+ \t\t\t\t\tName: interval_start\n+ \t\t\t\t\tType: TIMESTAMP\n+ \t\t\t\t\tNullable: false\n+ \t\t\t\t\tDefault: \"\"\n+ \t\t\t\t\tReference: nil\n+ \t\t\t\t\ts\"\"\"\n+ \t\t\t\t),\n+ \t\t\t\t(\n+ \t\t\t\t\ts\"\"\"\n+ \t\t\t\t\tName: put_repair_total\n+ \t\t\t\t\tType: BIGINT\n+ \t\t\t\t\tNullable: true\n+ \t\t\t\t\tDefault: \"\"\n+ \t\t\t\t\tReference: nil\n+ \t\t\t\t\ts\"\"\"\n+ \t\t\t\t),\n+ \t\t\t\t(\n+ \t\t\t\t\ts\"\"\"\n+ \t\t\t\t\tName: put_total\n+ \t\t\t\t\tType: BIGINT\n+ \t\t\t\t\tNullable: true\n+ \t\t\t\t\tDefault: \"\"\n+ \t\t\t\t\tReference: nil\n+ \t\t\t\t\ts\"\"\"\n+ \t\t\t\t),\n  \t\t\t\t&{Name: \"satellite_id\", Type: \"BLOB\"},\n  \t\t\t},\n- \t\t\tPrimaryKey: nil,\n+ \t\t\tPrimaryKey: []string{\"interval_start\", \"satellite_id\"},\n  \t\t\tUnique:     nil,\n  \t\t\tChecks:     nil,\n  \t\t},\n- \t\t(\n- \t\t\ts\"\"\"\n- \t\t\tName: bandwidth_usage_rollups\n- \t\t\tColumns:\n- \t\t\t\tName: action\n- \t\t\t\tType: INTEGER\n- \t\t\t\tNullable: false\n- \t\t\t\tDefault: \"\"\n- \t\t\t\tReference: nil\n- \t\t\t\tName: amount\n- \t\t\t\tType: BIGINT\n- \t\t\t\tNullable: false\n- \t\t\t\tDefault: \"\"\n- \t\t\t\tReference: nil\n- \t\t\t\tName: interval_start\n- \t\t\t\tType: TIMESTAMP\n- \t\t\t\tNullable: false\n- \t\t\t\tDefault: \"\"\n- \t\t\t\tReference: nil\n- \t\t\t\tName: satellite_id\n- \t\t\t\tType: BLOB\n- \t\t\t\tNullable: false\n- \t\t\t\tDefault: \"\"\n- \t\t\t\tReference: nil\n- \t\t\tPrimaryKey: action interval_start satellite_id\n- \t\t\tUniques:\n- \t\t\t\t\n- \t\t\tChecks:\n- \t\t\t\t\t\n- \t\t\ts\"\"\"\n- \t\t),\n  \t},\n- \tIndexes: []*dbschema.Index{\n- \t\ts`Index<Table: bandwidth_usage, Name: idx_bandwidth_usage_created, Columns: created_at, Unique: false, Partial: \"\">`,\n- \t\ts`Index<Table: bandwidth_usage, Name: idx_bandwidth_usage_satellite, Columns: satellite_id, Unique: false, Partial: \"\">`,\n- \t},\n+ \tIndexes:   nil,\n  \tSequences: nil,\n  }\n\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).preflight:446\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).Preflight:393\n\tmain.cmdRun:113\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:267\n\tmain.cmdRun:115\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:267"}
Error: Error during preflight check for storagenode databases: preflight: database "bandwidth": expected schema does not match actual:   &dbschema.Schema{
        Tables: []*dbschema.Table{
                &{
                        Name: "bandwidth_usage",
                        Columns: []*dbschema.Column{
                                &{
-                                       Name:       "action",
+                                       Name:       "delete_total",
-                                       Type:       "INTEGER",
+                                       Type:       "BIGINT",
-                                       IsNullable: false,
+                                       IsNullable: true,
                                        Default:    "",
                                        Reference:  nil,
                                },
                                &{
-                                       Name:       "amount",
+                                       Name:       "get_audit_total",
                                        Type:       "BIGINT",
-                                       IsNullable: false,
+                                       IsNullable: true,
                                        Default:    "",
                                        Reference:  nil,
                                },
                                &{
-                                       Name:       "created_at",
+                                       Name:       "get_repair_total",
-                                       Type:       "TIMESTAMP",
+                                       Type:       "BIGINT",
-                                       IsNullable: false,
+                                       IsNullable: true,
                                        Default:    "",
                                        Reference:  nil,
                                },
+                               (
+                                       s"""
+                                       Name: get_total
+                                       Type: BIGINT
+                                       Nullable: true
+                                       Default: ""
+                                       Reference: nil
+                                       s"""
+                               ),
+                               (
+                                       s"""
+                                       Name: interval_start
+                                       Type: TIMESTAMP
+                                       Nullable: false
+                                       Default: ""
+                                       Reference: nil
+                                       s"""
+                               ),
+                               (
+                                       s"""
+                                       Name: put_repair_total
+                                       Type: BIGINT
+                                       Nullable: true
+                                       Default: ""
+                                       Reference: nil
+                                       s"""
+                               ),
+                               (
+                                       s"""
+                                       Name: put_total
+                                       Type: BIGINT
+                                       Nullable: true
+                                       Default: ""
+                                       Reference: nil
+                                       s"""
+                               ),
                                &{Name: "satellite_id", Type: "BLOB"},
                        },
-                       PrimaryKey: nil,
+                       PrimaryKey: []string{"interval_start", "satellite_id"},
                        Unique:     nil,
                        Checks:     nil,
                },
-               (
-                       s"""
-                       Name: bandwidth_usage_rollups
-                       Columns:
-                               Name: action
-                               Type: INTEGER
-                               Nullable: false
-                               Default: ""
-                               Reference: nil
-                               Name: amount
-                               Type: BIGINT
-                               Nullable: false
-                               Default: ""
-                               Reference: nil
-                               Name: interval_start
-                               Type: TIMESTAMP
-                               Nullable: false
-                               Default: ""
-                               Reference: nil
-                               Name: satellite_id
-                               Type: BLOB
-                               Nullable: false
-                               Default: ""
-                               Reference: nil
-                       PrimaryKey: action interval_start satellite_id
-                       Uniques:
- 
-                       Checks:
- 
-                       s"""
-               ),
        },
-       Indexes: []*dbschema.Index{
-               s`Index<Table: bandwidth_usage, Name: idx_bandwidth_usage_created, Columns: created_at, Unique: false, Partial: "">`,
-               s`Index<Table: bandwidth_usage, Name: idx_bandwidth_usage_satellite, Columns: satellite_id, Unique: false, Partial: "">`,
-       },
+       Indexes:   nil,
        Sequences: nil,
  }

        storj.io/storj/storagenode/storagenodedb.(*DB).preflight:446
        storj.io/storj/storagenode/storagenodedb.(*DB).Preflight:393
        main.cmdRun:113
        main.newRunCmd.func1:33
        storj.io/common/process.cleanup.func1.4:393
        storj.io/common/process.cleanup.func1:411
        github.com/spf13/cobra.(*Command).execute:983
        github.com/spf13/cobra.(*Command).ExecuteC:1115
        github.com/spf13/cobra.(*Command).Execute:1039
        storj.io/common/process.ExecWithCustomOptions:112
        main.main:34
        runtime.main:267
2024-05-17 14:37:41,583 INFO exited: storagenode (exit status 1; not expected)
2024-05-17 14:37:42,587 INFO spawned: 'storagenode' with pid 94
2024-05-17 14:37:42,589 WARN received SIGQUIT indicating exit request
2024-05-17 14:37:42,589 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2024-05-17T14:37:42Z    INFO    Got a signal from the OS: "terminated"  {"Process": "storagenode-updater"}
2024-05-17 14:37:42,597 INFO stopped: storagenode-updater (exit status 0)
2024-05-17 14:37:42,598 INFO stopped: storagenode (terminated by SIGTERM)
2024-05-17 14:37:42,599 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)

This looks like it was downgraded back again to v1.102.3…

3

I believe you can now only re-create it:

4

Yes I‘ve already done that and it‘s online again.
I created this post in case many nodes have this kind of issue and we need to find some hints what is causing this. But all my other nodes upgraded without issues, so I guess it was just some strange combination of factors that caused this.

But the fact that the node was downgraded again made me a little worried.

5

It shouldn’t be many. When the rollout is stopped and started the next one, the previously updated nodes should be updated first.
Just unfortunate moment to reboot or re-create the container.