Note keeps restarting because of dbs

Xade · December 18, 2023, 10:44pm

Hi, my node keeps restarting. Apparently it has something to do with the dbs. I already tried to delete all of them, so that the node can create new ones. And it even does recreate all dbs, but I still get the same errors. Log below:

2023-12-18 22:29:20,233 INFO Set uid to user 0 succeeded
2023-12-18 22:29:20,256 INFO RPC interface 'supervisor' initialized
2023-12-18 22:29:20,257 INFO supervisord started with pid 1
2023-12-18 22:29:21,264 INFO spawned: 'processes-exit-eventlistener' with pid 54
2023-12-18 22:29:21,284 INFO spawned: 'storagenode' with pid 55
2023-12-18 22:29:21,294 INFO spawned: 'storagenode-updater' with pid 56
2023-12-18T22:29:21Z    INFO    Configuration loaded    {"Process": "storagenode-updater", "Location": "/app/config/config.yaml"}
2023-12-18T22:29:21Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-bandwidth"}
2023-12-18T22:29:21Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.debug-log-traffic"}
2023-12-18T22:29:21Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "kademlia.operator.email"}
2023-12-18T22:29:21Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-disk-space"}
2023-12-18T22:29:21Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "kademlia.external-address"}
2023-12-18T22:29:21Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.address"}
2023-12-18T22:29:21Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.private-address"}
2023-12-18T22:29:21Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "kademlia.operator.wallet"}
2023-12-18T22:29:21Z    INFO    Invalid configuration file value for key        {"Process": "storagenode-updater", "Key": "log.development"}
2023-12-18T22:29:21Z    INFO    Anonymized tracing enabled      {"Process": "storagenode-updater"}
2023-12-18T22:29:21Z    INFO    Running on version      {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.93.2"}
2023-12-18T22:29:21Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2023-12-18T22:29:21Z    INFO    Configuration loaded    {"process": "storagenode", "Location": "/app/config/config.yaml"}
2023-12-18T22:29:21Z    INFO    Anonymized tracing enabled      {"process": "storagenode"}
2023-12-18T22:29:21Z    INFO    Operator email  {"process": "storagenode", "Address": "weddingCrasher@yandex.com"}
2023-12-18T22:29:21Z    INFO    Operator wallet {"process": "storagenode", "Address": "0xB915d96D7a36559645b30c19ceFFD97A260eE734"}
2023-12-18T22:29:21Z    INFO    db      database does not exist {"process": "storagenode", "database": "info"}
2023-12-18T22:29:21Z    INFO    db      database does not exist {"process": "storagenode", "database": "bandwidth"}
2023-12-18T22:29:21Z    INFO    db      database does not exist {"process": "storagenode", "database": "orders"}
2023-12-18T22:29:21Z    INFO    db      database does not exist {"process": "storagenode", "database": "piece_expiration"}
2023-12-18T22:29:21Z    INFO    db      database does not exist {"process": "storagenode", "database": "pieceinfo"}
2023-12-18T22:29:21Z    INFO    db      database does not exist {"process": "storagenode", "database": "piece_spaced_used"}
2023-12-18T22:29:21Z    INFO    db      database does not exist {"process": "storagenode", "database": "reputation"}
2023-12-18T22:29:21Z    INFO    db      database does not exist {"process": "storagenode", "database": "storage_usage"}
2023-12-18T22:29:21Z    INFO    db      database does not exist {"process": "storagenode", "database": "used_serial"}
2023-12-18T22:29:21Z    INFO    db      database does not exist {"process": "storagenode", "database": "satellites"}
2023-12-18T22:29:21Z    INFO    db      database does not exist {"process": "storagenode", "database": "notifications"}
2023-12-18T22:29:21Z    INFO    db      database does not exist {"process": "storagenode", "database": "heldamount"}
2023-12-18T22:29:21Z    INFO    db      database does not exist {"process": "storagenode", "database": "pricing"}
2023-12-18T22:29:21Z    INFO    db      database does not exist {"process": "storagenode", "database": "secret"}
2023-12-18T22:29:21Z    INFO    server  kernel support for server-side tcp fast open enabled.   {"process": "storagenode"}
2023-12-18T22:29:22Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.93.2"}
2023-12-18T22:29:22Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2023-12-18T22:29:22Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.93.2"}
2023-12-18T22:29:22Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2023-12-18 22:29:22,261 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-12-18T22:29:22Z    INFO    Telemetry enabled       {"process": "storagenode", "instance ID": "123jk3dKL8USr66nnRQPZp9zczABvX9as9mv5QRtxwfGZHx9VoV"}
2023-12-18T22:29:22Z    INFO    Event collection enabled        {"process": "storagenode", "instance ID": "123jk3dKL8USr66nnRQPZp9zczABvX9as9mv5QRtxwfGZHx9VoV"}
2023-12-18 22:29:22,428 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-12-18 22:29:22,429 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Error: Error creating tables for master database on storagenode: migrate: v23: database: piece_spaced_used migrate tables to database: migrate tables: unable to open database file
        storj.io/storj/private/migrate.(*Migration).Run:209
        storj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:370
        main.cmdRun:95
        main.newRunCmd.func1:32
        storj.io/private/process.cleanup.func1.4:393
        storj.io/private/process.cleanup.func1:411
        github.com/spf13/cobra.(*Command).execute:852
        github.com/spf13/cobra.(*Command).ExecuteC:960
        github.com/spf13/cobra.(*Command).Execute:897
        storj.io/private/process.ExecWithCustomOptions:112
        main.main:30
        runtime.main:267
2023-12-18 22:29:23,909 INFO exited: storagenode (exit status 1; not expected)

JWvdV · December 18, 2023, 11:05pm

Are you sure, you deleted the right databases? Besides, the filesystem is still okay? No other errors in system log?

Knowledge · December 19, 2023, 1:16am

Your config file may be malformed. You may have the wrong kind of quotes or have some other data in there that shouldn’t be. Often this happens if someone edits the file in a word processor rather than a simple text editor.

Alexey · December 19, 2023, 3:12am

Try to pull the latest image, but you need to remove all databases again before run.

docker stop storagenode
docker rm storagenode
docker rmi storjlabs/storagenode:latest
docker pull storjlabs/storagenode:latest

Remove/rename/move all databases and execute your full docker run command, it should re-create all databases and start normally.
You may then try to stop and remove the container and copy back all your previous databases except piece_spaced_used.db and run the node back.

Xade · December 20, 2023, 11:27am

I have tried your approach, also checked the file system for errors. Still the same problem it seems:

2023-12-20 11:22:00,613 INFO Set uid to user 0 succeeded
2023-12-20 11:22:00,640 INFO RPC interface 'supervisor' initialized
2023-12-20 11:22:00,640 INFO supervisord started with pid 1
2023-12-20 11:22:01,648 INFO spawned: 'processes-exit-eventlistener' with pid 53
2023-12-20 11:22:01,661 INFO spawned: 'storagenode' with pid 54
2023-12-20 11:22:01,673 INFO spawned: 'storagenode-updater' with pid 55
2023-12-20T11:22:01Z    INFO    Configuration loaded    {"Process": "storagenode-updater", "Location": "/app/config/config.yaml"}
2023-12-20T11:22:01Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.debug-log-traffic"}
2023-12-20T11:22:01Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "kademlia.operator.email"}
2023-12-20T11:22:01Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.private-address"}
2023-12-20T11:22:01Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-bandwidth"}
2023-12-20T11:22:01Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "kademlia.external-address"}
2023-12-20T11:22:01Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.address"}
2023-12-20T11:22:01Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "kademlia.operator.wallet"}
2023-12-20T11:22:01Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-disk-space"}
2023-12-20T11:22:01Z    INFO    Invalid configuration file value for key        {"Process": "storagenode-updater", "Key": "log.output"}
2023-12-20T11:22:01Z    INFO    Anonymized tracing enabled      {"Process": "storagenode-updater"}
2023-12-20T11:22:01Z    INFO    Running on version      {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.93.2"}
2023-12-20T11:22:01Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2023-12-20T11:22:01Z    INFO    Configuration loaded    {"process": "storagenode", "Location": "/app/config/config.yaml"}
2023-12-20T11:22:01Z    INFO    Anonymized tracing enabled      {"process": "storagenode"}
2023-12-20T11:22:01Z    INFO    Operator email  {"process": "storagenode", "Address": "weddingCrasher@yandex.com"}
2023-12-20T11:22:01Z    INFO    Operator wallet {"process": "storagenode", "Address": "0xB915d96D7a36559645b30c19ceFFD97A260eE734"}
2023-12-20T11:22:01Z    INFO    db      database does not exist {"process": "storagenode", "database": "info"}
2023-12-20T11:22:01Z    INFO    db      database does not exist {"process": "storagenode", "database": "bandwidth"}
2023-12-20T11:22:01Z    INFO    db      database does not exist {"process": "storagenode", "database": "orders"}
2023-12-20T11:22:01Z    INFO    db      database does not exist {"process": "storagenode", "database": "piece_expiration"}
2023-12-20T11:22:01Z    INFO    db      database does not exist {"process": "storagenode", "database": "pieceinfo"}
2023-12-20T11:22:01Z    INFO    db      database does not exist {"process": "storagenode", "database": "piece_spaced_used"}
2023-12-20T11:22:01Z    INFO    db      database does not exist {"process": "storagenode", "database": "reputation"}
2023-12-20T11:22:01Z    INFO    db      database does not exist {"process": "storagenode", "database": "storage_usage"}
2023-12-20T11:22:01Z    INFO    db      database does not exist {"process": "storagenode", "database": "used_serial"}
2023-12-20T11:22:01Z    INFO    db      database does not exist {"process": "storagenode", "database": "satellites"}
2023-12-20T11:22:01Z    INFO    db      database does not exist {"process": "storagenode", "database": "notifications"}
2023-12-20T11:22:01Z    INFO    db      database does not exist {"process": "storagenode", "database": "heldamount"}
2023-12-20T11:22:01Z    INFO    db      database does not exist {"process": "storagenode", "database": "pricing"}
2023-12-20T11:22:01Z    INFO    db      database does not exist {"process": "storagenode", "database": "secret"}
2023-12-20T11:22:01Z    INFO    server  kernel support for server-side tcp fast open remains disabled.  {"process": "storagenode"}
2023-12-20T11:22:01Z    INFO    server  enable with: sysctl -w net.ipv4.tcp_fastopen=3  {"process": "storagenode"}
2023-12-20T11:22:02Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.93.2"}
2023-12-20T11:22:02Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2023-12-20T11:22:02Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.93.2"}
2023-12-20T11:22:02Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2023-12-20T11:22:02Z    INFO    Telemetry enabled       {"process": "storagenode", "instance ID": "123jk3dKL8USr66nnRQPZp9zczABvX9as9mv5QRtxwfGZHx9VoV"}
2023-12-20T11:22:02Z    INFO    Event collection enabled        {"process": "storagenode", "instance ID": "123jk3dKL8USr66nnRQPZp9zczABvX9as9mv5QRtxwfGZHx9VoV"}
2023-12-20 11:22:03,588 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-12-20 11:22:03,588 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-12-20 11:22:03,589 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Error: Error creating tables for master database on storagenode: migrate: v23: database: piece_spaced_used migrate tables to database: migrate tables: unable to open database file
        storj.io/storj/private/migrate.(*Migration).Run:209
        storj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:370
        main.cmdRun:95
        main.newRunCmd.func1:32
        storj.io/private/process.cleanup.func1.4:393
        storj.io/private/process.cleanup.func1:411
        github.com/spf13/cobra.(*Command).execute:852
        github.com/spf13/cobra.(*Command).ExecuteC:960
        github.com/spf13/cobra.(*Command).Execute:897
        storj.io/private/process.ExecWithCustomOptions:112
        main.main:30
        runtime.main:267
2023-12-20 11:22:04,205 INFO exited: storagenode (exit status 1; not expected)
2023-12-20 11:22:05,217 INFO spawned: 'storagenode' with pid 86
2023-12-20 11:22:05,219 WARN received SIGQUIT indicating exit request
2023-12-20 11:22:05,221 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2023-12-20T11:22:05Z    INFO    Got a signal from the OS: "terminated"  {"Process": "storagenode-updater"}
2023-12-20 11:22:05,228 INFO stopped: storagenode-updater (exit status 0)
2023-12-20T11:22:05Z    INFO    Configuration loaded    {"process": "storagenode", "Location": "/app/config/config.yaml"}
2023-12-20T11:22:05Z    INFO    Anonymized tracing enabled      {"process": "storagenode"}
2023-12-20T11:22:05Z    INFO    Operator email  {"process": "storagenode", "Address": "weddingCrasher@yandex.com"}
2023-12-20T11:22:05Z    INFO    Operator wallet {"process": "storagenode", "Address": "0xB915d96D7a36559645b30c19ceFFD97A260eE734"}
Error: Error starting master database on storagenode: database: piece_spaced_used opening file "config/storage/piece_spaced_used.db" failed: unable to open database file: no such file or directory
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:347
        storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:324
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:299
        storj.io/storj/storagenode/storagenodedb.OpenExisting:266
        main.cmdRun:62
        main.newRunCmd.func1:32
        storj.io/private/process.cleanup.func1.4:393
        storj.io/private/process.cleanup.func1:411
        github.com/spf13/cobra.(*Command).execute:852
        github.com/spf13/cobra.(*Command).ExecuteC:960
        github.com/spf13/cobra.(*Command).Execute:897
        storj.io/private/process.ExecWithCustomOptions:112
        main.main:30
        runtime.main:267
2023-12-20 11:22:05,749 INFO stopped: storagenode (exit status 1)
2023-12-20 11:22:05,750 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)
2023-12-20 11:22:07,787 INFO Set uid to user 0 succeeded
2023-12-20 11:22:07,808 INFO RPC interface 'supervisor' initialized
2023-12-20 11:22:07,810 INFO supervisord started with pid 1
2023-12-20 11:22:08,816 INFO spawned: 'processes-exit-eventlistener' with pid 11
2023-12-20 11:22:08,827 INFO spawned: 'storagenode' with pid 12
2023-12-20 11:22:08,838 INFO spawned: 'storagenode-updater' with pid 13
2023-12-20T11:22:08Z    INFO    Configuration loaded    {"Process": "storagenode-updater", "Location": "/app/config/config.yaml"}
2023-12-20T11:22:08Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.address"}
2023-12-20T11:22:08Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "kademlia.external-address"}
2023-12-20T11:22:08Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-bandwidth"}
2023-12-20T11:22:08Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "kademlia.operator.email"}
2023-12-20T11:22:08Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.private-address"}
2023-12-20T11:22:08Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-disk-space"}
2023-12-20T11:22:08Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.debug-log-traffic"}
2023-12-20T11:22:08Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "kademlia.operator.wallet"}
2023-12-20T11:22:08Z    INFO    Invalid configuration file value for key        {"Process": "storagenode-updater", "Key": "log.level"}
2023-12-20T11:22:08Z    INFO    Anonymized tracing enabled      {"Process": "storagenode-updater"}
2023-12-20T11:22:08Z    INFO    Running on version      {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.93.2"}
2023-12-20T11:22:08Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2023-12-20T11:22:08Z    INFO    Configuration loaded    {"process": "storagenode", "Location": "/app/config/config.yaml"}
2023-12-20T11:22:08Z    INFO    Anonymized tracing enabled      {"process": "storagenode"}
2023-12-20T11:22:09Z    INFO    Operator email  {"process": "storagenode", "Address": "weddingCrasher@yandex.com"}
2023-12-20T11:22:09Z    INFO    Operator wallet {"process": "storagenode", "Address": "0xB915d96D7a36559645b30c19ceFFD97A260eE734"}
Error: Error starting master database on storagenode: database: piece_spaced_used opening file "config/storage/piece_spaced_used.db" failed: unable to open database file: no such file or directory
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:347
        storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:324
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:299
        storj.io/storj/storagenode/storagenodedb.OpenExisting:266
        main.cmdRun:62
        main.newRunCmd.func1:32
        storj.io/private/process.cleanup.func1.4:393
        storj.io/private/process.cleanup.func1:411
        github.com/spf13/cobra.(*Command).execute:852
        github.com/spf13/cobra.(*Command).ExecuteC:960
        github.com/spf13/cobra.(*Command).Execute:897
        storj.io/private/process.ExecWithCustomOptions:112
        main.main:30
        runtime.main:267
2023-12-20 11:22:09,239 INFO exited: storagenode (exit status 1; not expected)
2023-12-20T11:22:09Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.93.2"}
2023-12-20T11:22:09Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2023-12-20T11:22:09Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.93.2"}
2023-12-20T11:22:09Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2023-12-20 11:22:10,488 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-12-20 11:22:10,497 INFO spawned: 'storagenode' with pid 43
2023-12-20 11:22:10,500 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-12-20T11:22:10Z    INFO    Configuration loaded    {"process": "storagenode", "Location": "/app/config/config.yaml"}
2023-12-20T11:22:10Z    INFO    Anonymized tracing enabled      {"process": "storagenode"}
2023-12-20T11:22:10Z    INFO    Operator email  {"process": "storagenode", "Address": "weddingCrasher@yandex.com"}
2023-12-20T11:22:10Z    INFO    Operator wallet {"process": "storagenode", "Address": "0xB915d96D7a36559645b30c19ceFFD97A260eE734"}
Error: Error starting master database on storagenode: database: piece_spaced_used opening file "config/storage/piece_spaced_used.db" failed: unable to open database file: no such file or directory
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:347
        storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:324
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:299
        storj.io/storj/storagenode/storagenodedb.OpenExisting:266
        main.cmdRun:62
        main.newRunCmd.func1:32
        storj.io/private/process.cleanup.func1.4:393
        storj.io/private/process.cleanup.func1:411
        github.com/spf13/cobra.(*Command).execute:852
        github.com/spf13/cobra.(*Command).ExecuteC:960
        github.com/spf13/cobra.(*Command).Execute:897
        storj.io/private/process.ExecWithCustomOptions:112
        main.main:30
        runtime.main:267
2023-12-20 11:22:10,929 INFO exited: storagenode (exit status 1; not expected)
2023-12-20 11:22:12,940 INFO spawned: 'storagenode' with pid 52
2023-12-20T11:22:13Z    INFO    Configuration loaded    {"process": "storagenode", "Location": "/app/config/config.yaml"}
2023-12-20T11:22:13Z    INFO    Anonymized tracing enabled      {"process": "storagenode"}
2023-12-20T11:22:13Z    INFO    Operator email  {"process": "storagenode", "Address": "weddingCrasher@yandex.com"}
2023-12-20T11:22:13Z    INFO    Operator wallet {"process": "storagenode", "Address": "0xB915d96D7a36559645b30c19ceFFD97A260eE734"}
Error: Error starting master database on storagenode: database: piece_spaced_used opening file "config/storage/piece_spaced_used.db" failed: unable to open database file: no such file or directory
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:347
        storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:324
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:299
        storj.io/storj/storagenode/storagenodedb.OpenExisting:266
        main.cmdRun:62
        main.newRunCmd.func1:32
        storj.io/private/process.cleanup.func1.4:393
        storj.io/private/process.cleanup.func1:411
        github.com/spf13/cobra.(*Command).execute:852
        github.com/spf13/cobra.(*Command).ExecuteC:960
        github.com/spf13/cobra.(*Command).Execute:897
        storj.io/private/process.ExecWithCustomOptions:112
        main.main:30
        runtime.main:267
2023-12-20 11:22:13,371 INFO exited: storagenode (exit status 1; not expected)
2023-12-20 11:22:16,384 INFO spawned: 'storagenode' with pid 61
2023-12-20T11:22:16Z    INFO    Configuration loaded    {"process": "storagenode", "Location": "/app/config/config.yaml"}
2023-12-20T11:22:16Z    INFO    Anonymized tracing enabled      {"process": "storagenode"}
2023-12-20T11:22:16Z    INFO    Operator email  {"process": "storagenode", "Address": "weddingCrasher@yandex.com"}
2023-12-20T11:22:16Z    INFO    Operator wallet {"process": "storagenode", "Address": "0xB915d96D7a36559645b30c19ceFFD97A260eE734"}
Error: Error starting master database on storagenode: database: piece_spaced_used opening file "config/storage/piece_spaced_used.db" failed: unable to open database file: no such file or directory
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:347
        storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:324
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:299
        storj.io/storj/storagenode/storagenodedb.OpenExisting:266
        main.cmdRun:62
        main.newRunCmd.func1:32
        storj.io/private/process.cleanup.func1.4:393
        storj.io/private/process.cleanup.func1:411
        github.com/spf13/cobra.(*Command).execute:852
        github.com/spf13/cobra.(*Command).ExecuteC:960
        github.com/spf13/cobra.(*Command).Execute:897
        storj.io/private/process.ExecWithCustomOptions:112
        main.main:30
        runtime.main:267
2023-12-20 11:22:16,813 INFO exited: storagenode (exit status 1; not expected)
2023-12-20 11:22:17,815 INFO gave up: storagenode entered FATAL state, too many start retries too quickly
2023-12-20 11:22:18,819 WARN received SIGQUIT indicating exit request
2023-12-20 11:22:18,820 INFO waiting for processes-exit-eventlistener, storagenode-updater to die
2023-12-20T11:22:18Z    INFO    Got a signal from the OS: "terminated"  {"Process": "storagenode-updater"}
2023-12-20 11:22:18,827 INFO stopped: storagenode-updater (exit status 0)
2023-12-20 11:22:19,831 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)
2023-12-20 11:22:21,521 INFO Set uid to user 0 succeeded
2023-12-20 11:22:21,543 INFO RPC interface 'supervisor' initialized
2023-12-20 11:22:21,543 INFO supervisord started with pid 1
2023-12-20 11:22:22,551 INFO spawned: 'processes-exit-eventlistener' with pid 11
2023-12-20 11:22:22,565 INFO spawned: 'storagenode' with pid 12
2023-12-20 11:22:22,574 INFO spawned: 'storagenode-updater' with pid 13
2023-12-20T11:22:22Z    INFO    Configuration loaded    {"Process": "storagenode-updater", "Location": "/app/config/config.yaml"}
2023-12-20T11:22:22Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "kademlia.external-address"}
2023-12-20T11:22:22Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "kademlia.operator.email"}
2023-12-20T11:22:22Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "kademlia.operator.wallet"}
2023-12-20T11:22:22Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-bandwidth"}
2023-12-20T11:22:22Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.private-address"}
2023-12-20T11:22:22Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-disk-space"}
2023-12-20T11:22:22Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.debug-log-traffic"}
2023-12-20T11:22:22Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.address"}
2023-12-20T11:22:22Z    INFO    Invalid configuration file value for key        {"Process": "storagenode-updater", "Key": "log.level"}
2023-12-20T11:22:22Z    INFO    Anonymized tracing enabled      {"Process": "storagenode-updater"}
2023-12-20T11:22:22Z    INFO    Running on version      {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.93.2"}
2023-12-20T11:22:22Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2023-12-20T11:22:22Z    INFO    Configuration loaded    {"process": "storagenode", "Location": "/app/config/config.yaml"}
2023-12-20T11:22:22Z    INFO    Anonymized tracing enabled      {"process": "storagenode"}
2023-12-20T11:22:22Z    INFO    Operator email  {"process": "storagenode", "Address": "weddingCrasher@yandex.com"}
2023-12-20T11:22:22Z    INFO    Operator wallet {"process": "storagenode", "Address": "0xB915d96D7a36559645b30c19ceFFD97A260eE734"}
Error: Error starting master database on storagenode: database: piece_spaced_used opening file "config/storage/piece_spaced_used.db" failed: unable to open database file: no such file or directory
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:347
        storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:324
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:299
        storj.io/storj/storagenode/storagenodedb.OpenExisting:266
        main.cmdRun:62
        main.newRunCmd.func1:32
        storj.io/private/process.cleanup.func1.4:393
        storj.io/private/process.cleanup.func1:411
        github.com/spf13/cobra.(*Command).execute:852
        github.com/spf13/cobra.(*Command).ExecuteC:960
        github.com/spf13/cobra.(*Command).Execute:897
        storj.io/private/process.ExecWithCustomOptions:112
        main.main:30
        runtime.main:267
2023-12-20 11:22:22,999 INFO exited: storagenode (exit status 1; not expected)
2023-12-20T11:22:23Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.93.2"}
2023-12-20T11:22:23Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2023-12-20T11:22:23Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.93.2"}
2023-12-20T11:22:23Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2023-12-20 11:22:24,238 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-12-20 11:22:24,248 INFO spawned: 'storagenode' with pid 43
2023-12-20 11:22:24,251 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-12-20T11:22:24Z    INFO    Configuration loaded    {"process": "storagenode", "Location": "/app/config/config.yaml"}
2023-12-20T11:22:24Z    INFO    Anonymized tracing enabled      {"process": "storagenode"}
2023-12-20T11:22:24Z    INFO    Operator email  {"process": "storagenode", "Address": "weddingCrasher@yandex.com"}
2023-12-20T11:22:24Z    INFO    Operator wallet {"process": "storagenode", "Address": "0xB915d96D7a36559645b30c19ceFFD97A260eE734"}
Error: Error starting master database on storagenode: database: piece_spaced_used opening file "config/storage/piece_spaced_used.db" failed: unable to open database file: no such file or directory
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:347
        storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:324
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:299
        storj.io/storj/storagenode/storagenodedb.OpenExisting:266
        main.cmdRun:62
        main.newRunCmd.func1:32
        storj.io/private/process.cleanup.func1.4:393
        storj.io/private/process.cleanup.func1:411
        github.com/spf13/cobra.(*Command).execute:852
        github.com/spf13/cobra.(*Command).ExecuteC:960
        github.com/spf13/cobra.(*Command).Execute:897
        storj.io/private/process.ExecWithCustomOptions:112
        main.main:30
        runtime.main:267

My run command which always worked:

docker run -d --restart unless-stopped --stop-timeout 300 --privileged -p 28967:28967/tcp -p 28967:28967/udp -p 14002:14002     -e WALLET="0xB915d96D7a36559645b30c19ceFFD97A260eE734"     -e EMAIL="weddingCrasher@yandex.com"     -e ADDRESS="sjcs.homeunix.org:28967"     -e BANDWIDTH="25TB"     -e STORAGE="10.5TB"    --user $(id -u):$(id -g)    --mount type=bind,source="/mnt/Elements12TB/identity/storagenode",destination=/app/identity     --mount type=bind,source="/mnt/Elements12TB/storJ",destination=/app/config     --name storagenode storjlabs/storagenode:latest

Alexey · December 21, 2023, 2:54am

Is it a network filesystem?
This is very weird, because the node should create a database here:

but then it cannot open a file because it is absent:

Please try to change the owner to

sudo chown $(id -u):$(id -g) -R /mnt/Elements12TB/storJ

then remove all databases and run the node. It must create all databases back, after that you may stop and remove the container and restore all databases except piece_spaced_used.db

Xade · December 22, 2023, 11:17pm

It’s a direct USB 3.0 HDD. Tried what you suggested, new error log:

2023-12-22 23:11:20,709 INFO Set uid to user 0 succeeded
2023-12-22 23:11:20,737 INFO RPC interface 'supervisor' initialized
2023-12-22 23:11:20,738 INFO supervisord started with pid 1
2023-12-22 23:11:21,744 INFO spawned: 'processes-exit-eventlistener' with pid 54
2023-12-22 23:11:21,754 INFO spawned: 'storagenode' with pid 55
2023-12-22 23:11:21,764 INFO spawned: 'storagenode-updater' with pid 56
2023-12-22T23:11:21Z    INFO    Configuration loaded    {"Process": "storagenode-updater", "Location": "/app/config/confi                                                 g.yaml"}
2023-12-22T23:11:21Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "kademlia.opera                                                 tor.email"}
2023-12-22T23:11:21Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.private                                                 -address"}
2023-12-22T23:11:21Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "kademlia.exter                                                 nal-address"}
2023-12-22T23:11:21Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.debug-l                                                 og-traffic"}
2023-12-22T23:11:21Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.alloca                                                 ted-disk-space"}
2023-12-22T23:11:21Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "kademlia.opera                                                 tor.wallet"}
2023-12-22T23:11:21Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.alloca                                                 ted-bandwidth"}
2023-12-22T23:11:21Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.address                                                 "}
2023-12-22T23:11:21Z    INFO    Invalid configuration file value for key        {"Process": "storagenode-updater", "Key":                                                  "log.level"}
2023-12-22T23:11:21Z    INFO    Invalid configuration file value for key        {"Process": "storagenode-updater", "Key":                                                  "log.stack"}
2023-12-22T23:11:21Z    INFO    Anonymized tracing enabled      {"Process": "storagenode-updater"}
2023-12-22T23:11:21Z    INFO    Running on version      {"Process": "storagenode-updater", "Service": "storagenode-update                                                 r", "Version": "v1.93.2"}
2023-12-22T23:11:21Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://ver                                                 sion.storj.io"}
2023-12-22T23:11:21Z    INFO    Configuration loaded    {"process": "storagenode", "Location": "/app/config/config.yaml"}
2023-12-22T23:11:21Z    INFO    Anonymized tracing enabled      {"process": "storagenode"}
2023-12-22T23:11:21Z    INFO    Operator email  {"process": "storagenode", "Address": "weddingCrasher@yandex.com"}
2023-12-22T23:11:21Z    INFO    Operator wallet {"process": "storagenode", "Address": "0xB915d96D7a36559645b30c19ceFFD97A                                                 260eE734"}
2023-12-22T23:11:22Z    INFO    db      database does not exist {"process": "storagenode", "database": "info"}
2023-12-22T23:11:22Z    INFO    db      database does not exist {"process": "storagenode", "database": "bandwidth"}
2023-12-22T23:11:22Z    INFO    db      database does not exist {"process": "storagenode", "database": "orders"}
2023-12-22T23:11:22Z    INFO    db      database does not exist {"process": "storagenode", "database": "piece_expiration"                                                 }
2023-12-22T23:11:22Z    INFO    db      database does not exist {"process": "storagenode", "database": "pieceinfo"}
2023-12-22T23:11:22Z    INFO    db      database does not exist {"process": "storagenode", "database": "piece_spaced_used                                                 "}
2023-12-22T23:11:22Z    INFO    db      database does not exist {"process": "storagenode", "database": "reputation"}
2023-12-22T23:11:22Z    INFO    db      database does not exist {"process": "storagenode", "database": "storage_usage"}
2023-12-22T23:11:22Z    INFO    db      database does not exist {"process": "storagenode", "database": "used_serial"}
2023-12-22T23:11:22Z    INFO    db      database does not exist {"process": "storagenode", "database": "satellites"}
2023-12-22T23:11:22Z    INFO    db      database does not exist {"process": "storagenode", "database": "notifications"}
2023-12-22T23:11:22Z    INFO    db      database does not exist {"process": "storagenode", "database": "heldamount"}
2023-12-22T23:11:22Z    INFO    db      database does not exist {"process": "storagenode", "database": "pricing"}
2023-12-22T23:11:22Z    INFO    db      database does not exist {"process": "storagenode", "database": "secret"}
2023-12-22T23:11:22Z    INFO    server  kernel support for server-side tcp fast open remains disabled.  {"process": "stor                                                 agenode"}
2023-12-22T23:11:22Z    INFO    server  enable with: sysctl -w net.ipv4.tcp_fastopen=3  {"process": "storagenode"}
2023-12-22T23:11:22Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Ver                                                 sion": "v1.93.2"}
2023-12-22T23:11:22Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2023-12-22T23:11:22Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-update                                                 r", "Version": "v1.93.2"}
2023-12-22T23:11:22Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-update                                                 r"}
2023-12-22T23:11:22Z    INFO    Telemetry enabled       {"process": "storagenode", "instance ID": "123jk3dKL8USr66nnRQPZp                                                 9zczABvX9as9mv5QRtxwfGZHx9VoV"}
2023-12-22T23:11:22Z    INFO    Event collection enabled        {"process": "storagenode", "instance ID": "123jk3dKL8USr6                                                 6nnRQPZp9zczABvX9as9mv5QRtxwfGZHx9VoV"}
2023-12-22 23:11:23,637 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > tha                                                 n 1 seconds (startsecs)
2023-12-22 23:11:23,638 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (star                                                 tsecs)
2023-12-22 23:11:23,645 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 secon                                                 ds (startsecs)
Error: Error creating tables for master database on storagenode: migrate: v23: database: piece_spaced_used migrate tables                                                  to database: migrate tables: unable to open database file
        storj.io/storj/private/migrate.(*Migration).Run:209
        storj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:370
        main.cmdRun:95
        main.newRunCmd.func1:32
        storj.io/private/process.cleanup.func1.4:393
        storj.io/private/process.cleanup.func1:411
        github.com/spf13/cobra.(*Command).execute:852
        github.com/spf13/cobra.(*Command).ExecuteC:960
        github.com/spf13/cobra.(*Command).Execute:897
        storj.io/private/process.ExecWithCustomOptions:112
        main.main:30
        runtime.main:267
2023-12-22 23:11:25,076 INFO exited: storagenode (exit status 1; not expected)
2023-12-22 23:11:26,086 INFO spawned: 'storagenode' with pid 86
2023-12-22 23:11:26,088 WARN received SIGQUIT indicating exit request
2023-12-22 23:11:26,090 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2023-12-22T23:11:26Z    INFO    Got a signal from the OS: "terminated"  {"Process": "storagenode-updater"}
2023-12-22 23:11:26,097 INFO stopped: storagenode-updater (exit status 0)
2023-12-22T23:11:26Z    INFO    Configuration loaded    {"process": "storagenode", "Location": "/app/config/config.yaml"}
2023-12-22T23:11:26Z    INFO    Anonymized tracing enabled      {"process": "storagenode"}
2023-12-22T23:11:26Z    INFO    Operator email  {"process": "storagenode", "Address": "weddingCrasher@yandex.com"}
2023-12-22T23:11:26Z    INFO    Operator wallet {"process": "storagenode", "Address": "0xB915d96D7a36559645b30c19ceFFD97A                                                 260eE734"}
Error: Error starting master database on storagenode: database: piece_spaced_used opening file "config/storage/piece_spac                                                 ed_used.db" failed: unable to open database file: no such file or directory
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:347
        storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:324
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:299
        storj.io/storj/storagenode/storagenodedb.OpenExisting:266
        main.cmdRun:62
        main.newRunCmd.func1:32
        storj.io/private/process.cleanup.func1.4:393
        storj.io/private/process.cleanup.func1:411
        github.com/spf13/cobra.(*Command).execute:852
        github.com/spf13/cobra.(*Command).ExecuteC:960
        github.com/spf13/cobra.(*Command).Execute:897
        storj.io/private/process.ExecWithCustomOptions:112
        main.main:30
        runtime.main:267
2023-12-22 23:11:26,510 INFO stopped: storagenode (exit status 1)
2023-12-22 23:11:26,511 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)
2023-12-22 23:11:28,230 INFO Set uid to user 0 succeeded
2023-12-22 23:11:28,250 INFO RPC interface 'supervisor' initialized
2023-12-22 23:11:28,250 INFO supervisord started with pid 1
2023-12-22 23:11:29,257 INFO spawned: 'processes-exit-eventlistener' with pid 11
2023-12-22 23:11:29,267 INFO spawned: 'storagenode' with pid 12
2023-12-22 23:11:29,276 INFO spawned: 'storagenode-updater' with pid 13
2023-12-22T23:11:29Z    INFO    Configuration loaded    {"Process": "storagenode-updater", "Location": "/app/config/confi                                                 g.yaml"}
2023-12-22T23:11:29Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.debug-l                                                 og-traffic"}
2023-12-22T23:11:29Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.private                                                 -address"}
2023-12-22T23:11:29Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "kademlia.opera                                                 tor.wallet"}
2023-12-22T23:11:29Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.address                                                 "}
2023-12-22T23:11:29Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "kademlia.exter                                                 nal-address"}
2023-12-22T23:11:29Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.alloca                                                 ted-disk-space"}
2023-12-22T23:11:29Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "kademlia.opera                                                 tor.email"}
2023-12-22T23:11:29Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.alloca                                                 ted-bandwidth"}
2023-12-22T23:11:29Z    INFO    Invalid configuration file value for key        {"Process": "storagenode-updater", "Key":                                                  "log.caller"}
2023-12-22T23:11:29Z    INFO    Invalid configuration file value for key        {"Process": "storagenode-updater", "Key":                                                  "log.development"}
2023-12-22T23:11:29Z    INFO    Invalid configuration file value for key        {"Process": "storagenode-updater", "Key":                                                  "log.level"}
2023-12-22T23:11:29Z    INFO    Anonymized tracing enabled      {"Process": "storagenode-updater"}
2023-12-22T23:11:29Z    INFO    Running on version      {"Process": "storagenode-updater", "Service": "storagenode-update                                                 r", "Version": "v1.93.2"}
2023-12-22T23:11:29Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://ver                                                 sion.storj.io"}
2023-12-22T23:11:29Z    INFO    Configuration loaded    {"process": "storagenode", "Location": "/app/config/config.yaml"}
2023-12-22T23:11:29Z    INFO    Anonymized tracing enabled      {"process": "storagenode"}
2023-12-22T23:11:29Z    INFO    Operator email  {"process": "storagenode", "Address": "weddingCrasher@yandex.com"}
2023-12-22T23:11:29Z    INFO    Operator wallet {"process": "storagenode", "Address": "0xB915d96D7a36559645b30c19ceFFD97A                                                 260eE734"}
Error: Error starting master database on storagenode: database: piece_spaced_used opening file "config/storage/piece_spac                                                 ed_used.db" failed: unable to open database file: no such file or directory
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:347
        storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:324
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:299
        storj.io/storj/storagenode/storagenodedb.OpenExisting:266
        main.cmdRun:62
        main.newRunCmd.func1:32
        storj.io/private/process.cleanup.func1.4:393
        storj.io/private/process.cleanup.func1:411
        github.com/spf13/cobra.(*Command).execute:852
        github.com/spf13/cobra.(*Command).ExecuteC:960
        github.com/spf13/cobra.(*Command).Execute:897
        storj.io/private/process.ExecWithCustomOptions:112
        main.main:30
        runtime.main:267
2023-12-22 23:11:29,663 INFO exited: storagenode (exit status 1; not expected)
2023-12-22T23:11:29Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Ver                                                 sion": "v1.93.2"}
2023-12-22T23:11:29Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2023-12-22T23:11:29Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-update                                                 r", "Version": "v1.93.2"}
2023-12-22T23:11:29Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-update                                                 r"}
2023-12-22 23:11:30,924 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > tha                                                 n 1 seconds (startsecs)
2023-12-22 23:11:30,931 INFO spawned: 'storagenode' with pid 42
2023-12-22 23:11:30,933 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 secon                                                 ds (startsecs)
2023-12-22T23:11:31Z    INFO    Configuration loaded    {"process": "storagenode", "Location": "/app/config/config.yaml"}
2023-12-22T23:11:31Z    INFO    Anonymized tracing enabled      {"process": "storagenode"}
2023-12-22T23:11:31Z    INFO    Operator email  {"process": "storagenode", "Address": "weddingCrasher@yandex.com"}
2023-12-22T23:11:31Z    INFO    Operator wallet {"process": "storagenode", "Address": "0xB915d96D7a36559645b30c19ceFFD97A                                                 260eE734"}
Error: Error starting master database on storagenode: database: piece_spaced_used opening file "config/storage/piece_spac                                                 ed_used.db" failed: unable to open database file: no such file or directory
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:347
        storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:324
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:299
        storj.io/storj/storagenode/storagenodedb.OpenExisting:266
        main.cmdRun:62
        main.newRunCmd.func1:32
        storj.io/private/process.cleanup.func1.4:393
        storj.io/private/process.cleanup.func1:411
        github.com/spf13/cobra.(*Command).execute:852
        github.com/spf13/cobra.(*Command).ExecuteC:960
        github.com/spf13/cobra.(*Command).Execute:897
        storj.io/private/process.ExecWithCustomOptions:112
        main.main:30
        runtime.main:267
2023-12-22 23:11:31,338 INFO exited: storagenode (exit status 1; not expected)
2023-12-22 23:11:33,349 INFO spawned: 'storagenode' with pid 51
2023-12-22T23:11:33Z    INFO    Configuration loaded    {"process": "storagenode", "Location": "/app/config/config.yaml"}
2023-12-22T23:11:33Z    INFO    Anonymized tracing enabled      {"process": "storagenode"}
2023-12-22T23:11:33Z    INFO    Operator email  {"process": "storagenode", "Address": "weddingCrasher@yandex.com"}
2023-12-22T23:11:33Z    INFO    Operator wallet {"process": "storagenode", "Address": "0xB915d96D7a36559645b30c19ceFFD97A                                                 260eE734"}
Error: Error starting master database on storagenode: database: piece_spaced_used opening file "config/storage/piece_spac                                                 ed_used.db" failed: unable to open database file: no such file or directory
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:347
        storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:324
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:299
        storj.io/storj/storagenode/storagenodedb.OpenExisting:266
        main.cmdRun:62
        main.newRunCmd.func1:32
        storj.io/private/process.cleanup.func1.4:393
        storj.io/private/process.cleanup.func1:411
        github.com/spf13/cobra.(*Command).execute:852
        github.com/spf13/cobra.(*Command).ExecuteC:960
        github.com/spf13/cobra.(*Command).Execute:897
        storj.io/private/process.ExecWithCustomOptions:112
        main.main:30
        runtime.main:267
2023-12-22 23:11:33,751 INFO exited: storagenode (exit status 1; not expected)
2023-12-22 23:11:36,763 INFO spawned: 'storagenode' with pid 60
2023-12-22T23:11:36Z    INFO    Configuration loaded    {"process": "storagenode", "Location": "/app/config/config.yaml"}
2023-12-22T23:11:36Z    INFO    Anonymized tracing enabled      {"process": "storagenode"}
2023-12-22T23:11:36Z    INFO    Operator email  {"process": "storagenode", "Address": "weddingCrasher@yandex.com"}
2023-12-22T23:11:36Z    INFO    Operator wallet {"process": "storagenode", "Address": "0xB915d96D7a36559645b30c19ceFFD97A                                                 260eE734"}
Error: Error starting master database on storagenode: database: piece_spaced_used opening file "config/storage/piece_spac                                                 ed_used.db" failed: unable to open database file: no such file or directory
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:347
        storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:324
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:299
        storj.io/storj/storagenode/storagenodedb.OpenExisting:266
        main.cmdRun:62
        main.newRunCmd.func1:32
        storj.io/private/process.cleanup.func1.4:393
        storj.io/private/process.cleanup.func1:411
        github.com/spf13/cobra.(*Command).execute:852
        github.com/spf13/cobra.(*Command).ExecuteC:960
        github.com/spf13/cobra.(*Command).Execute:897
        storj.io/private/process.ExecWithCustomOptions:112
        main.main:30
        runtime.main:267
2023-12-22 23:11:37,142 INFO exited: storagenode (exit status 1; not expected)
2023-12-22 23:11:38,144 INFO gave up: storagenode entered FATAL state, too many start retries too quickly
2023-12-22 23:11:39,148 WARN received SIGQUIT indicating exit request
2023-12-22 23:11:39,148 INFO waiting for processes-exit-eventlistener, storagenode-updater to die
2023-12-22T23:11:39Z    INFO    Got a signal from the OS: "terminated"  {"Process": "storagenode-updater"}
2023-12-22 23:11:39,155 INFO stopped: storagenode-updater (exit status 0)
2023-12-22 23:11:40,159 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)
2023-12-22 23:11:41,821 INFO Set uid to user 0 succeeded
2023-12-22 23:11:41,841 INFO RPC interface 'supervisor' initialized
2023-12-22 23:11:41,842 INFO supervisord started with pid 1
2023-12-22 23:11:42,848 INFO spawned: 'processes-exit-eventlistener' with pid 11
2023-12-22 23:11:42,858 INFO spawned: 'storagenode' with pid 12
2023-12-22 23:11:42,867 INFO spawned: 'storagenode-updater' with pid 13
2023-12-22T23:11:42Z    INFO    Configuration loaded    {"Process": "storagenode-updater", "Location": "/app/config/confi                                                 g.yaml"}
2023-12-22T23:11:42Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "kademlia.exter                                                 nal-address"}
2023-12-22T23:11:42Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.alloca                                                 ted-bandwidth"}
2023-12-22T23:11:42Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.debug-l                                                 og-traffic"}
2023-12-22T23:11:42Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.private                                                 -address"}
2023-12-22T23:11:42Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.address                                                 "}
2023-12-22T23:11:42Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "kademlia.opera                                                 tor.email"}
2023-12-22T23:11:42Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "kademlia.opera                                                 tor.wallet"}
2023-12-22T23:11:42Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.alloca                                                 ted-disk-space"}
2023-12-22T23:11:42Z    INFO    Invalid configuration file value for key        {"Process": "storagenode-updater", "Key":                                                  "log.output"}
2023-12-22T23:11:42Z    INFO    Anonymized tracing enabled      {"Process": "storagenode-updater"}
2023-12-22T23:11:42Z    INFO    Running on version      {"Process": "storagenode-updater", "Service": "storagenode-update                                                 r", "Version": "v1.93.2"}
2023-12-22T23:11:42Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://ver                                                 sion.storj.io"}
2023-12-22T23:11:43Z    INFO    Configuration loaded    {"process": "storagenode", "Location": "/app/config/config.yaml"}
2023-12-22T23:11:43Z    INFO    Anonymized tracing enabled      {"process": "storagenode"}
2023-12-22T23:11:43Z    INFO    Operator email  {"process": "storagenode", "Address": "weddingCrasher@yandex.com"}
2023-12-22T23:11:43Z    INFO    Operator wallet {"process": "storagenode", "Address": "0xB915d96D7a36559645b30c19ceFFD97A                                                 260eE734"}
Error: Error starting master database on storagenode: database: piece_spaced_used opening file "config/storage/piece_spac                                                 ed_used.db" failed: unable to open database file: no such file or directory
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:347
        storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:324
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:299
        storj.io/storj/storagenode/storagenodedb.OpenExisting:266
        main.cmdRun:62
        main.newRunCmd.func1:32
        storj.io/private/process.cleanup.func1.4:393
        storj.io/private/process.cleanup.func1:411
        github.com/spf13/cobra.(*Command).execute:852
        github.com/spf13/cobra.(*Command).ExecuteC:960
        github.com/spf13/cobra.(*Command).Execute:897
        storj.io/private/process.ExecWithCustomOptions:112
        main.main:30
        runtime.main:267
2023-12-22 23:11:43,277 INFO exited: storagenode (exit status 1; not expected)
2023-12-22T23:11:43Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Ver                                                 sion": "v1.93.2"}
2023-12-22T23:11:43Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2023-12-22T23:11:43Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-update                                                 r", "Version": "v1.93.2"}
2023-12-22T23:11:43Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-update                                                 r"}
2023-12-22 23:11:44,515 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > tha                                                 n 1 seconds (startsecs)
2023-12-22 23:11:44,523 INFO spawned: 'storagenode' with pid 43
2023-12-22 23:11:44,525 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 secon                                                 ds (startsecs)
2023-12-22T23:11:44Z    INFO    Configuration loaded    {"process": "storagenode", "Location": "/app/config/config.yaml"}
2023-12-22T23:11:44Z    INFO    Anonymized tracing enabled      {"process": "storagenode"}
2023-12-22T23:11:44Z    INFO    Operator email  {"process": "storagenode", "Address": "weddingCrasher@yandex.com"}
2023-12-22T23:11:44Z    INFO    Operator wallet {"process": "storagenode", "Address": "0xB915d96D7a36559645b30c19ceFFD97A                                                 260eE734"}
Error: Error starting master database on storagenode: database: piece_spaced_used opening file "config/storage/piece_spac                                                 ed_used.db" failed: unable to open database file: no such file or directory
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:347
        storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:324
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:299
        storj.io/storj/storagenode/storagenodedb.OpenExisting:266
        main.cmdRun:62
        main.newRunCmd.func1:32
        storj.io/private/process.cleanup.func1.4:393
        storj.io/private/process.cleanup.func1:411
        github.com/spf13/cobra.(*Command).execute:852
        github.com/spf13/cobra.(*Command).ExecuteC:960
        github.com/spf13/cobra.(*Command).Execute:897
        storj.io/private/process.ExecWithCustomOptions:112
        main.main:30
        runtime.main:267
2023-12-22 23:11:44,910 INFO exited: storagenode (exit status 1; not expected)

Alexey · December 23, 2023, 7:47am

It again either didn’t create a database piece_spaced_used.db or it was not saved to the disk.
Please show result of the command:

ls -l /mnt/Elements12TB/storJ
ls -l /mnt/Elements12TB/storJ/storage

Xade · December 24, 2023, 1:06pm

Here are the results:

root@raspberrypi4:/mnt/Elements# ls -l /mnt/Elements12TB/storJ
insgesamt 324
-rwxrwxrwx 1 root root    829 Jun 11  2020 config.yaml
-rwxrwxrwx 1 root root 262144 Okt  3  2019 kademlia
drwxrwxrwx 1 root root      0 Sep  4  2020 orders
-rwxrwxrwx 1 root root  32768 Dez 23 00:11 revocations.db
drwxrwxrwx 1 root root  24576 Dez 23 00:12 storage
-rwxrwxrwx 1 root root   2289 Dez 17 12:32 trust-cache.json
-rwxrwxrwx 1 root root   2730 Sep 24  2022 trust-cache.json1149288118

and

root@raspberrypi4:/mnt/Elements# ls -l /mnt/Elements12TB/storJ/storage
insgesamt 197777
-rwxrwxrwx 1 root root     24576 Dez 23 00:11 bandwidth.db
drwxrwxrwx 1 root root         0 Jul 16  2019 blob
drwxrwxrwx 1 root root      4096 Jan  7  2021 blobs
drwxrwxrwx 1 root root      4096 Dez 18 23:15 DB_Backups
drwxrwxrwx 1 root root     24576 Dez 17 12:32 garbage
-rwxrwxrwx 1 root root    135168 Dez 23 00:11 info.db
-rwxrwxrwx 1 root root     16384 Dez 23 00:11 orders.db
-rwxrwxrwx 1 root root     20480 Dez 23 00:11 piece_expiration.db
-rwxrwxrwx 1 root root     16384 Dez 23 00:11 pieceinfo.db
-rwxrwxrwx 1 root root      4096 Dez 23 00:11 piece_spaced_used.db
-rwxrwxrwx 1 root root         0 Dez 23 00:11 piece_spaced_used.db-wal
-rwxrwxrwx 1 root root      4096 Dez 23 00:11 reputation.db
-rwxrwxrwx 1 root root      4096 Dez 23 00:11 satellites.db
-rwxrwxrwx 1 root root        32 Dez 18 18:10 storage-dir-verification
-rwxrwxrwx 1 root root      4096 Dez 23 00:11 storage_usage.db
drwxrwxrwx 1 root root 202252288 Dez 17 12:33 temp
drwxrwxrwx 1 root root      4096 Jan 12  2021 trash
-rwxrwxrwx 1 root root      4096 Dez 23 00:11 used_serial.db
-rwxrwxrwx 1 root root         0 Sep 22  2022 write-test1265540363
-rwxrwxrwx 1 root root         0 Mär  7  2023 write-test1324701481
-rwxrwxrwx 1 root root         0 Mär 17  2023 write-test1477702234
-rwxrwxrwx 1 root root         0 Sep 22  2022 write-test1839237374
-rwxrwxrwx 1 root root         0 Sep 21  2022 write-test1867707981
-rwxrwxrwx 1 root root         0 Mär 18  2023 write-test1913174638
-rwxrwxrwx 1 root root         0 Mär 18  2023 write-test192220591
-rwxrwxrwx 1 root root         0 Mär 16  2023 write-test2284224945
-rwxrwxrwx 1 root root         0 Mär  9  2023 write-test233384524
-rwxrwxrwx 1 root root         0 Sep 19  2022 write-test2354714472
-rwxrwxrwx 1 root root         0 Sep 16  2022 write-test243715213
-rwxrwxrwx 1 root root         0 Mär 18  2023 write-test2457967941
-rwxrwxrwx 1 root root         0 Sep 24  2022 write-test2729741602
-rwxrwxrwx 1 root root         0 Sep 15  2022 write-test2804473926
-rwxrwxrwx 1 root root         0 Sep 19  2022 write-test3087336888
-rwxrwxrwx 1 root root         0 Mär 17  2023 write-test3267077572
-rwxrwxrwx 1 root root         0 Sep 22  2022 write-test3297119974
-rwxrwxrwx 1 root root         0 Mär  8  2023 write-test3598281214
-rwxrwxrwx 1 root root         0 Mär  8  2023 write-test3847121906
-rwxrwxrwx 1 root root         0 Mär 18  2023 write-test4026102791
-rwxrwxrwx 1 root root         0 Mär  9  2023 write-test4146183310
-rwxrwxrwx 1 root root         0 Sep 19  2022 write-test423293173
-rwxrwxrwx 1 root root         0 Sep 21  2022 write-test471064762
-rwxrwxrwx 1 root root         0 Sep 14  2022 write-test530076394
-rwxrwxrwx 1 root root         0 Mär  9  2023 write-test543405008
-rwxrwxrwx 1 root root         0 Sep 19  2022 write-test594408963
-rwxrwxrwx 1 root root         0 Mär 17  2023 write-test669119877
-rwxrwxrwx 1 root root         0 Mär 18  2023 write-test786796008
-rwxrwxrwx 1 root root         0 Sep 19  2022 write-test872397235
-rwxrwxrwx 1 root root         0 Sep 21  2022 write-test91682543
-rwxrwxrwx 1 root root         0 Sep 24  2022 write-test917766394

Alexey · December 25, 2023, 2:59am

Seems you didn’t replace the owner for the data. Then you need to remove --user $(id -u):$(id -g) from your docker run command.

Xade · December 26, 2023, 12:54am

Mhh, owner is root, so it should be correct as storJ is running on root? Anyway ran it without --user. Does not seem to matter much:

2023-12-26 00:49:24,472 INFO Set uid to user 0 succeeded
2023-12-26 00:49:24,500 INFO RPC interface 'supervisor' initialized
2023-12-26 00:49:24,500 INFO supervisord started with pid 1
2023-12-26 00:49:25,508 INFO spawned: 'processes-exit-eventlistener' with pid 53
2023-12-26 00:49:25,520 INFO spawned: 'storagenode' with pid 54
2023-12-26 00:49:25,532 INFO spawned: 'storagenode-updater' with pid 55
2023-12-26T00:49:25Z    INFO    Configuration loaded    {"Process": "storagenode-updater", "Location": "/app/config/config.yaml"}
2023-12-26T00:49:25Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "kademlia.external-address"}
2023-12-26T00:49:25Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "kademlia.operator.email"}
2023-12-26T00:49:25Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-bandwidth"}
2023-12-26T00:49:25Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "kademlia.operator.wallet"}
2023-12-26T00:49:25Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.address"}
2023-12-26T00:49:25Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.private-address"}
2023-12-26T00:49:25Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-disk-space"}
2023-12-26T00:49:25Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.debug-log-traffic"}
2023-12-26T00:49:25Z    INFO    Anonymized tracing enabled      {"Process": "storagenode-updater"}
2023-12-26T00:49:25Z    INFO    Running on version      {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.93.2"}
2023-12-26T00:49:25Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2023-12-26T00:49:25Z    INFO    Configuration loaded    {"process": "storagenode", "Location": "/app/config/config.yaml"}
2023-12-26T00:49:25Z    INFO    Anonymized tracing enabled      {"process": "storagenode"}
2023-12-26T00:49:25Z    INFO    Operator email  {"process": "storagenode", "Address": "weddingCrasher@yandex.com"}
2023-12-26T00:49:25Z    INFO    Operator wallet {"process": "storagenode", "Address": "0xB915d96D7a36559645b30c19ceFFD97A260eE734"}
2023-12-26T00:49:25Z    INFO    db      database does not exist {"process": "storagenode", "database": "info"}
2023-12-26T00:49:25Z    INFO    db      database does not exist {"process": "storagenode", "database": "bandwidth"}
2023-12-26T00:49:25Z    INFO    db      database does not exist {"process": "storagenode", "database": "orders"}
2023-12-26T00:49:25Z    INFO    db      database does not exist {"process": "storagenode", "database": "piece_expiration"}
2023-12-26T00:49:25Z    INFO    db      database does not exist {"process": "storagenode", "database": "pieceinfo"}
2023-12-26T00:49:25Z    INFO    db      database does not exist {"process": "storagenode", "database": "piece_spaced_used"}
2023-12-26T00:49:25Z    INFO    db      database does not exist {"process": "storagenode", "database": "reputation"}
2023-12-26T00:49:25Z    INFO    db      database does not exist {"process": "storagenode", "database": "storage_usage"}
2023-12-26T00:49:25Z    INFO    db      database does not exist {"process": "storagenode", "database": "used_serial"}
2023-12-26T00:49:25Z    INFO    db      database does not exist {"process": "storagenode", "database": "satellites"}
2023-12-26T00:49:25Z    INFO    db      database does not exist {"process": "storagenode", "database": "notifications"}
2023-12-26T00:49:25Z    INFO    db      database does not exist {"process": "storagenode", "database": "heldamount"}
2023-12-26T00:49:25Z    INFO    db      database does not exist {"process": "storagenode", "database": "pricing"}
2023-12-26T00:49:25Z    INFO    db      database does not exist {"process": "storagenode", "database": "secret"}
2023-12-26T00:49:25Z    INFO    server  kernel support for server-side tcp fast open enabled.   {"process": "storagenode"}
2023-12-26T00:49:26Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.93.2"}
2023-12-26T00:49:26Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2023-12-26T00:49:26Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.93.2"}
2023-12-26T00:49:26Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2023-12-26T00:49:26Z    INFO    Telemetry enabled       {"process": "storagenode", "instance ID": "123jk3dKL8USr66nnRQPZp9zczABvX9as9mv5QRtxwfGZHx9VoV"}
2023-12-26T00:49:26Z    INFO    Event collection enabled        {"process": "storagenode", "instance ID": "123jk3dKL8USr66nnRQPZp9zczABvX9as9mv5QRtxwfGZHx9VoV"}
2023-12-26 00:49:27,386 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-12-26 00:49:27,386 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-12-26 00:49:27,387 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Error: Error creating tables for master database on storagenode: migrate: v23: database: piece_spaced_used migrate tables to database: migrate tables: unable to open database file
        storj.io/storj/private/migrate.(*Migration).Run:209
        storj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:370
        main.cmdRun:95
        main.newRunCmd.func1:32
        storj.io/private/process.cleanup.func1.4:393
        storj.io/private/process.cleanup.func1:411
        github.com/spf13/cobra.(*Command).execute:852
        github.com/spf13/cobra.(*Command).ExecuteC:960
        github.com/spf13/cobra.(*Command).Execute:897
        storj.io/private/process.ExecWithCustomOptions:112
        main.main:30
        runtime.main:267
2023-12-26 00:49:27,748 INFO exited: storagenode (exit status 1; not expected)
2023-12-26 00:49:28,759 INFO spawned: 'storagenode' with pid 86
2023-12-26 00:49:28,762 WARN received SIGQUIT indicating exit request
2023-12-26 00:49:28,769 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2023-12-26T00:49:28Z    INFO    Got a signal from the OS: "terminated"  {"Process": "storagenode-updater"}
2023-12-26 00:49:28,777 INFO stopped: storagenode-updater (exit status 0)
2023-12-26T00:49:28Z    INFO    Configuration loaded    {"process": "storagenode", "Location": "/app/config/config.yaml"}
2023-12-26T00:49:28Z    INFO    Anonymized tracing enabled      {"process": "storagenode"}
2023-12-26T00:49:28Z    INFO    Operator email  {"process": "storagenode", "Address": "weddingCrasher@yandex.com"}
2023-12-26T00:49:28Z    INFO    Operator wallet {"process": "storagenode", "Address": "0xB915d96D7a36559645b30c19ceFFD97A260eE734"}
Error: Error starting master database on storagenode: database: piece_spaced_used opening file "config/storage/piece_spaced_used.db" failed: unable to open database file: no such file or directory
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:347
        storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:324
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:299
        storj.io/storj/storagenode/storagenodedb.OpenExisting:266
        main.cmdRun:62
        main.newRunCmd.func1:32
        storj.io/private/process.cleanup.func1.4:393
        storj.io/private/process.cleanup.func1:411
        github.com/spf13/cobra.(*Command).execute:852
        github.com/spf13/cobra.(*Command).ExecuteC:960
        github.com/spf13/cobra.(*Command).Execute:897
        storj.io/private/process.ExecWithCustomOptions:112
        main.main:30
        runtime.main:267
2023-12-26 00:49:29,165 INFO stopped: storagenode (exit status 1)
2023-12-26 00:49:29,166 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)
2023-12-26 00:49:33,163 INFO Set uid to user 0 succeeded
2023-12-26 00:49:33,184 INFO RPC interface 'supervisor' initialized
2023-12-26 00:49:33,184 INFO supervisord started with pid 1
2023-12-26 00:49:34,192 INFO spawned: 'processes-exit-eventlistener' with pid 11
2023-12-26 00:49:34,204 INFO spawned: 'storagenode' with pid 12
2023-12-26 00:49:34,216 INFO spawned: 'storagenode-updater' with pid 13
2023-12-26T00:49:34Z    INFO    Configuration loaded    {"Process": "storagenode-updater", "Location": "/app/config/config.yaml"}
2023-12-26T00:49:34Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "kademlia.external-address"}
2023-12-26T00:49:34Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.private-address"}
2023-12-26T00:49:34Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.debug-log-traffic"}
2023-12-26T00:49:34Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "kademlia.operator.email"}
2023-12-26T00:49:34Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-disk-space"}
2023-12-26T00:49:34Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.address"}
2023-12-26T00:49:34Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "kademlia.operator.wallet"}
2023-12-26T00:49:34Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-bandwidth"}
2023-12-26T00:49:34Z    INFO    Anonymized tracing enabled      {"Process": "storagenode-updater"}
2023-12-26T00:49:34Z    INFO    Running on version      {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.93.2"}
2023-12-26T00:49:34Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2023-12-26T00:49:34Z    INFO    Configuration loaded    {"process": "storagenode", "Location": "/app/config/config.yaml"}
2023-12-26T00:49:34Z    INFO    Anonymized tracing enabled      {"process": "storagenode"}
2023-12-26T00:49:34Z    INFO    Operator email  {"process": "storagenode", "Address": "weddingCrasher@yandex.com"}
2023-12-26T00:49:34Z    INFO    Operator wallet {"process": "storagenode", "Address": "0xB915d96D7a36559645b30c19ceFFD97A260eE734"}
Error: Error starting master database on storagenode: database: piece_spaced_used opening file "config/storage/piece_spaced_used.db" failed: unable to open database file: no such file or directory
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:347
        storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:324
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:299
        storj.io/storj/storagenode/storagenodedb.OpenExisting:266
        main.cmdRun:62
        main.newRunCmd.func1:32
        storj.io/private/process.cleanup.func1.4:393
        storj.io/private/process.cleanup.func1:411
        github.com/spf13/cobra.(*Command).execute:852
        github.com/spf13/cobra.(*Command).ExecuteC:960
        github.com/spf13/cobra.(*Command).Execute:897
        storj.io/private/process.ExecWithCustomOptions:112
        main.main:30
        runtime.main:267
2023-12-26 00:49:34,548 INFO exited: storagenode (exit status 1; not expected)
2023-12-26T00:49:34Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.93.2"}
2023-12-26T00:49:34Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2023-12-26T00:49:34Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.93.2"}
2023-12-26T00:49:34Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2023-12-26 00:49:35,867 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-12-26 00:49:35,876 INFO spawned: 'storagenode' with pid 42
2023-12-26 00:49:35,878 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-12-26T00:49:36Z    INFO    Configuration loaded    {"process": "storagenode", "Location": "/app/config/config.yaml"}
2023-12-26T00:49:36Z    INFO    Anonymized tracing enabled      {"process": "storagenode"}
2023-12-26T00:49:36Z    INFO    Operator email  {"process": "storagenode", "Address": "weddingCrasher@yandex.com"}
2023-12-26T00:49:36Z    INFO    Operator wallet {"process": "storagenode", "Address": "0xB915d96D7a36559645b30c19ceFFD97A260eE734"}
Error: Error starting master database on storagenode: database: piece_spaced_used opening file "config/storage/piece_spaced_used.db" failed: unable to open database file: no such file or directory
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:347
        storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:324
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:299
        storj.io/storj/storagenode/storagenodedb.OpenExisting:266
        main.cmdRun:62
        main.newRunCmd.func1:32
        storj.io/private/process.cleanup.func1.4:393
        storj.io/private/process.cleanup.func1:411
        github.com/spf13/cobra.(*Command).execute:852
        github.com/spf13/cobra.(*Command).ExecuteC:960
        github.com/spf13/cobra.(*Command).Execute:897
        storj.io/private/process.ExecWithCustomOptions:112
        main.main:30
        runtime.main:267
2023-12-26 00:49:36,235 INFO exited: storagenode (exit status 1; not expected)
2023-12-26 00:49:38,247 INFO spawned: 'storagenode' with pid 51
2023-12-26T00:49:38Z    INFO    Configuration loaded    {"process": "storagenode", "Location": "/app/config/config.yaml"}
2023-12-26T00:49:38Z    INFO    Anonymized tracing enabled      {"process": "storagenode"}
2023-12-26T00:49:38Z    INFO    Operator email  {"process": "storagenode", "Address": "weddingCrasher@yandex.com"}
2023-12-26T00:49:38Z    INFO    Operator wallet {"process": "storagenode", "Address": "0xB915d96D7a36559645b30c19ceFFD97A260eE734"}
Error: Error starting master database on storagenode: database: piece_spaced_used opening file "config/storage/piece_spaced_used.db" failed: unable to open database file: no such file or directory
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:347
        storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:324
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:299
        storj.io/storj/storagenode/storagenodedb.OpenExisting:266
        main.cmdRun:62
        main.newRunCmd.func1:32
        storj.io/private/process.cleanup.func1.4:393
        storj.io/private/process.cleanup.func1:411
        github.com/spf13/cobra.(*Command).execute:852
        github.com/spf13/cobra.(*Command).ExecuteC:960
        github.com/spf13/cobra.(*Command).Execute:897
        storj.io/private/process.ExecWithCustomOptions:112
        main.main:30
        runtime.main:267
2023-12-26 00:49:38,614 INFO exited: storagenode (exit status 1; not expected)
2023-12-26 00:49:41,627 INFO spawned: 'storagenode' with pid 60
2023-12-26T00:49:41Z    INFO    Configuration loaded    {"process": "storagenode", "Location": "/app/config/config.yaml"}
2023-12-26T00:49:41Z    INFO    Anonymized tracing enabled      {"process": "storagenode"}
2023-12-26T00:49:41Z    INFO    Operator email  {"process": "storagenode", "Address": "weddingCrasher@yandex.com"}
2023-12-26T00:49:41Z    INFO    Operator wallet {"process": "storagenode", "Address": "0xB915d96D7a36559645b30c19ceFFD97A260eE734"}
Error: Error starting master database on storagenode: database: piece_spaced_used opening file "config/storage/piece_spaced_used.db" failed: unable to open database file: no such file or directory
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:347
        storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:324
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:299
        storj.io/storj/storagenode/storagenodedb.OpenExisting:266
        main.cmdRun:62
        main.newRunCmd.func1:32
        storj.io/private/process.cleanup.func1.4:393
        storj.io/private/process.cleanup.func1:411
        github.com/spf13/cobra.(*Command).execute:852
        github.com/spf13/cobra.(*Command).ExecuteC:960
        github.com/spf13/cobra.(*Command).Execute:897
        storj.io/private/process.ExecWithCustomOptions:112
        main.main:30
        runtime.main:267
2023-12-26 00:49:42,013 INFO exited: storagenode (exit status 1; not expected)
2023-12-26 00:49:43,016 INFO gave up: storagenode entered FATAL state, too many start retries too quickly
2023-12-26 00:49:44,019 WARN received SIGQUIT indicating exit request
2023-12-26 00:49:44,021 INFO waiting for processes-exit-eventlistener, storagenode-updater to die
2023-12-26T00:49:44Z    INFO    Got a signal from the OS: "terminated"  {"Process": "storagenode-updater"}
2023-12-26 00:49:44,027 INFO stopped: storagenode-updater (exit status 0)
2023-12-26 00:49:45,031 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)
2023-12-26 00:49:46,780 INFO Set uid to user 0 succeeded
2023-12-26 00:49:46,801 INFO RPC interface 'supervisor' initialized
2023-12-26 00:49:46,802 INFO supervisord started with pid 1
2023-12-26 00:49:47,808 INFO spawned: 'processes-exit-eventlistener' with pid 11
2023-12-26 00:49:47,819 INFO spawned: 'storagenode' with pid 12
2023-12-26 00:49:47,830 INFO spawned: 'storagenode-updater' with pid 13
2023-12-26T00:49:47Z    INFO    Configuration loaded    {"Process": "storagenode-updater", "Location": "/app/config/config.yaml"}
2023-12-26T00:49:47Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-disk-space"}
2023-12-26T00:49:47Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "kademlia.external-address"}
2023-12-26T00:49:47Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "kademlia.operator.email"}
2023-12-26T00:49:47Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.address"}
2023-12-26T00:49:47Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.private-address"}
2023-12-26T00:49:47Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "kademlia.operator.wallet"}
2023-12-26T00:49:47Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.debug-log-traffic"}
2023-12-26T00:49:47Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-bandwidth"}
2023-12-26T00:49:47Z    INFO    Anonymized tracing enabled      {"Process": "storagenode-updater"}
2023-12-26T00:49:47Z    INFO    Running on version      {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.93.2"}
2023-12-26T00:49:47Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2023-12-26T00:49:48Z    INFO    Configuration loaded    {"process": "storagenode", "Location": "/app/config/config.yaml"}
2023-12-26T00:49:48Z    INFO    Anonymized tracing enabled      {"process": "storagenode"}
2023-12-26T00:49:48Z    INFO    Operator email  {"process": "storagenode", "Address": "weddingCrasher@yandex.com"}
2023-12-26T00:49:48Z    INFO    Operator wallet {"process": "storagenode", "Address": "0xB915d96D7a36559645b30c19ceFFD97A260eE734"}
Error: Error starting master database on storagenode: database: piece_spaced_used opening file "config/storage/piece_spaced_used.db" failed: unable to open database file: no such file or directory
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:347
        storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:324
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:299
        storj.io/storj/storagenode/storagenodedb.OpenExisting:266
        main.cmdRun:62
        main.newRunCmd.func1:32
        storj.io/private/process.cleanup.func1.4:393
        storj.io/private/process.cleanup.func1:411
        github.com/spf13/cobra.(*Command).execute:852
        github.com/spf13/cobra.(*Command).ExecuteC:960
        github.com/spf13/cobra.(*Command).Execute:897
        storj.io/private/process.ExecWithCustomOptions:112
        main.main:30
        runtime.main:267
2023-12-26 00:49:48,192 INFO exited: storagenode (exit status 1; not expected)
2023-12-26T00:49:48Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.93.2"}
2023-12-26T00:49:48Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2023-12-26T00:49:48Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.93.2"}
2023-12-26T00:49:48Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2023-12-26 00:49:49,484 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-12-26 00:49:49,494 INFO spawned: 'storagenode' with pid 42
2023-12-26 00:49:49,497 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-12-26T00:49:49Z    INFO    Configuration loaded    {"process": "storagenode", "Location": "/app/config/config.yaml"}
2023-12-26T00:49:49Z    INFO    Anonymized tracing enabled      {"process": "storagenode"}
2023-12-26T00:49:49Z    INFO    Operator email  {"process": "storagenode", "Address": "weddingCrasher@yandex.com"}
2023-12-26T00:49:49Z    INFO    Operator wallet {"process": "storagenode", "Address": "0xB915d96D7a36559645b30c19ceFFD97A260eE734"}
Error: Error starting master database on storagenode: database: piece_spaced_used opening file "config/storage/piece_spaced_used.db" failed: unable to open database file: no such file or directory
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:347
        storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:324
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:299
        storj.io/storj/storagenode/storagenodedb.OpenExisting:266
        main.cmdRun:62
        main.newRunCmd.func1:32
        storj.io/private/process.cleanup.func1.4:393
        storj.io/private/process.cleanup.func1:411
        github.com/spf13/cobra.(*Command).execute:852
        github.com/spf13/cobra.(*Command).ExecuteC:960
        github.com/spf13/cobra.(*Command).Execute:897
        storj.io/private/process.ExecWithCustomOptions:112
        main.main:30
        runtime.main:267
2023-12-26 00:49:49,855 INFO exited: storagenode (exit status 1; not expected)
2023-12-26 00:49:51,866 INFO spawned: 'storagenode' with pid 51
2023-12-26T00:49:52Z    INFO    Configuration loaded    {"process": "storagenode", "Location": "/app/config/config.yaml"}
2023-12-26T00:49:52Z    INFO    Anonymized tracing enabled      {"process": "storagenode"}
2023-12-26T00:49:52Z    INFO    Operator email  {"process": "storagenode", "Address": "weddingCrasher@yandex.com"}
2023-12-26T00:49:52Z    INFO    Operator wallet {"process": "storagenode", "Address": "0xB915d96D7a36559645b30c19ceFFD97A260eE734"}
Error: Error starting master database on storagenode: database: piece_spaced_used opening file "config/storage/piece_spaced_used.db" failed: unable to open database file: no such file or directory
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:347
        storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:324
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:299
        storj.io/storj/storagenode/storagenodedb.OpenExisting:266
        main.cmdRun:62
        main.newRunCmd.func1:32
        storj.io/private/process.cleanup.func1.4:393
        storj.io/private/process.cleanup.func1:411
        github.com/spf13/cobra.(*Command).execute:852
        github.com/spf13/cobra.(*Command).ExecuteC:960
        github.com/spf13/cobra.(*Command).Execute:897
        storj.io/private/process.ExecWithCustomOptions:112
        main.main:30
        runtime.main:267
2023-12-26 00:49:52,249 INFO exited: storagenode (exit status 1; not expected)
2023-12-26 00:49:55,263 INFO spawned: 'storagenode' with pid 60
2023-12-26T00:49:55Z    INFO    Configuration loaded    {"process": "storagenode", "Location": "/app/config/config.yaml"}
2023-12-26T00:49:55Z    INFO    Anonymized tracing enabled      {"process": "storagenode"}
2023-12-26T00:49:55Z    INFO    Operator email  {"process": "storagenode", "Address": "weddingCrasher@yandex.com"}
2023-12-26T00:49:55Z    INFO    Operator wallet {"process": "storagenode", "Address": "0xB915d96D7a36559645b30c19ceFFD97A260eE734"}
Error: Error starting master database on storagenode: database: piece_spaced_used opening file "config/storage/piece_spaced_used.db" failed: unable to open database file: no such file or directory
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:347
        storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:324
        storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:299
        storj.io/storj/storagenode/storagenodedb.OpenExisting:266
        main.cmdRun:62
        main.newRunCmd.func1:32
        storj.io/private/process.cleanup.func1.4:393
        storj.io/private/process.cleanup.func1:411
        github.com/spf13/cobra.(*Command).execute:852
        github.com/spf13/cobra.(*Command).ExecuteC:960
        github.com/spf13/cobra.(*Command).Execute:897
        storj.io/private/process.ExecWithCustomOptions:112
        main.main:30
        runtime.main:267
2023-12-26 00:49:55,660 INFO exited: storagenode (exit status 1; not expected)
2023-12-26 00:49:56,662 INFO gave up: storagenode entered FATAL state, too many start retries too quickly
2023-12-26 00:49:57,665 WARN received SIGQUIT indicating exit request
2023-12-26 00:49:57,666 INFO waiting for processes-exit-eventlistener, storagenode-updater to die
2023-12-26T00:49:57Z    INFO    Got a signal from the OS: "terminated"  {"Process": "storagenode-updater"}

and

root@raspberrypi4:/mnt/Elements# ls -l /mnt/Elements12TB/storJ/storage
insgesamt 197777
-rwxrwxrwx 1 root root     24576 Dez 26 01:49 bandwidth.db
drwxrwxrwx 1 root root         0 Jul 16  2019 blob
drwxrwxrwx 1 root root      4096 Jan  7  2021 blobs
drwxrwxrwx 1 root root      4096 Dez 18 23:15 DB_Backups
drwxrwxrwx 1 root root     24576 Dez 17 12:32 garbage
-rwxrwxrwx 1 root root    135168 Dez 26 01:49 info.db
-rwxrwxrwx 1 root root     16384 Dez 26 01:49 orders.db
-rwxrwxrwx 1 root root     20480 Dez 26 01:49 piece_expiration.db
-rwxrwxrwx 1 root root     16384 Dez 26 01:49 pieceinfo.db
-rwxrwxrwx 1 root root      4096 Dez 26 01:49 piece_spaced_used.db
-rwxrwxrwx 1 root root         0 Dez 26 01:49 piece_spaced_used.db-wal
-rwxrwxrwx 1 root root      4096 Dez 26 01:49 reputation.db
-rwxrwxrwx 1 root root      4096 Dez 26 01:49 satellites.db
-rwxrwxrwx 1 root root        32 Dez 18 18:10 storage-dir-verification
-rwxrwxrwx 1 root root      4096 Dez 26 01:49 storage_usage.db
drwxrwxrwx 1 root root 202252288 Dez 17 12:33 temp
drwxrwxrwx 1 root root      4096 Jan 12  2021 trash
-rwxrwxrwx 1 root root      4096 Dez 26 01:49 used_serial.db
-rwxrwxrwx 1 root root         0 Sep 22  2022 write-test1265540363
-rwxrwxrwx 1 root root         0 Mär  7  2023 write-test1324701481
-rwxrwxrwx 1 root root         0 Mär 17  2023 write-test1477702234
-rwxrwxrwx 1 root root         0 Sep 22  2022 write-test1839237374
-rwxrwxrwx 1 root root         0 Sep 21  2022 write-test1867707981
-rwxrwxrwx 1 root root         0 Mär 18  2023 write-test1913174638
-rwxrwxrwx 1 root root         0 Mär 18  2023 write-test192220591
-rwxrwxrwx 1 root root         0 Mär 16  2023 write-test2284224945
-rwxrwxrwx 1 root root         0 Mär  9  2023 write-test233384524
-rwxrwxrwx 1 root root         0 Sep 19  2022 write-test2354714472
-rwxrwxrwx 1 root root         0 Sep 16  2022 write-test243715213
-rwxrwxrwx 1 root root         0 Mär 18  2023 write-test2457967941
-rwxrwxrwx 1 root root         0 Sep 24  2022 write-test2729741602
-rwxrwxrwx 1 root root         0 Sep 15  2022 write-test2804473926
-rwxrwxrwx 1 root root         0 Sep 19  2022 write-test3087336888
-rwxrwxrwx 1 root root         0 Mär 17  2023 write-test3267077572
-rwxrwxrwx 1 root root         0 Sep 22  2022 write-test3297119974
-rwxrwxrwx 1 root root         0 Mär  8  2023 write-test3598281214
-rwxrwxrwx 1 root root         0 Mär  8  2023 write-test3847121906
-rwxrwxrwx 1 root root         0 Mär 18  2023 write-test4026102791
-rwxrwxrwx 1 root root         0 Mär  9  2023 write-test4146183310
-rwxrwxrwx 1 root root         0 Sep 19  2022 write-test423293173
-rwxrwxrwx 1 root root         0 Sep 21  2022 write-test471064762
-rwxrwxrwx 1 root root         0 Sep 14  2022 write-test530076394
-rwxrwxrwx 1 root root         0 Mär  9  2023 write-test543405008
-rwxrwxrwx 1 root root         0 Sep 19  2022 write-test594408963
-rwxrwxrwx 1 root root         0 Mär 17  2023 write-test669119877
-rwxrwxrwx 1 root root         0 Mär 18  2023 write-test786796008
-rwxrwxrwx 1 root root         0 Sep 19  2022 write-test872397235
-rwxrwxrwx 1 root root         0 Sep 21  2022 write-test91682543
-rwxrwxrwx 1 root root         0 Sep 24  2022 write-test917766394

Alexey · December 26, 2023, 1:13am

Please give me result of the command:

grep -E "path|database" /mnt/Elements12TB/storJ/config.yaml

Please do not remove databases anymore and remove only piece_spaced_used.db-wal, then try to run the node.

Xade · December 26, 2023, 11:18am

Removing the db-wal had no effect.

root@raspberrypi4:/mnt/Elements# grep -E "path|database" /mnt/Elements12TB/storJ/config.yaml
# path to the certificate chain for this identity
identity.cert-path: "identity/identity.cert"
# path to the private key for this identity
identity.key-path: "identity/identity.key"

Alexey · December 27, 2023, 4:59am

could you please try to execute:

docker exec -it storagenode ls -l /app/config/storage

also test the write and read:

docker exec -it storagenode bash
echo "test" > config/storage/test.file
cat config/storage/test.file
rm config/storage/test.file
exit

Xade · December 27, 2023, 11:07pm

Results:

root@raspberrypi4:~# docker exec -it storagenode ls -l /app/config/storage
total 197777
drwxrwxrwx 1 root root      4096 Dec 18 22:15 DB_Backups
-rwxrwxrwx 1 root root     24576 Dec 26 00:49 bandwidth.db
drwxrwxrwx 1 root root         0 Jul 16  2019 blob
drwxrwxrwx 1 root root      4096 Jan  7  2021 blobs
drwxrwxrwx 1 root root     24576 Dec 17 11:32 garbage
-rwxrwxrwx 1 root root    135168 Dec 26 00:49 info.db
-rwxrwxrwx 1 root root     16384 Dec 26 00:49 orders.db
-rwxrwxrwx 1 root root     20480 Dec 26 00:49 piece_expiration.db
-rwxrwxrwx 1 root root      4096 Dec 26 00:49 piece_spaced_used.db
-rwxrwxrwx 1 root root         0 Dec 26 11:15 piece_spaced_used.db-wal
-rwxrwxrwx 1 root root     16384 Dec 26 00:49 pieceinfo.db
-rwxrwxrwx 1 root root      4096 Dec 26 00:49 reputation.db
-rwxrwxrwx 1 root root      4096 Dec 26 00:49 satellites.db
-rwxrwxrwx 1 root root        32 Dec 18 17:10 storage-dir-verification
-rwxrwxrwx 1 root root      4096 Dec 26 00:49 storage_usage.db
drwxrwxrwx 1 root root 202252288 Dec 17 11:33 temp
drwxrwxrwx 1 root root      4096 Jan 12  2021 trash
-rwxrwxrwx 1 root root      4096 Dec 26 00:49 used_serial.db
-rwxrwxrwx 1 root root         0 Sep 22  2022 write-test1265540363
-rwxrwxrwx 1 root root         0 Mar  7  2023 write-test1324701481
-rwxrwxrwx 1 root root         0 Mar 17  2023 write-test1477702234
-rwxrwxrwx 1 root root         0 Sep 21  2022 write-test1839237374
-rwxrwxrwx 1 root root         0 Sep 21  2022 write-test1867707981
-rwxrwxrwx 1 root root         0 Mar 18  2023 write-test1913174638
-rwxrwxrwx 1 root root         0 Mar 18  2023 write-test192220591
-rwxrwxrwx 1 root root         0 Mar 16  2023 write-test2284224945
-rwxrwxrwx 1 root root         0 Mar  9  2023 write-test233384524
-rwxrwxrwx 1 root root         0 Sep 19  2022 write-test2354714472
-rwxrwxrwx 1 root root         0 Sep 16  2022 write-test243715213
-rwxrwxrwx 1 root root         0 Mar 18  2023 write-test2457967941
-rwxrwxrwx 1 root root         0 Sep 23  2022 write-test2729741602
-rwxrwxrwx 1 root root         0 Sep 15  2022 write-test2804473926
-rwxrwxrwx 1 root root         0 Sep 19  2022 write-test3087336888
-rwxrwxrwx 1 root root         0 Mar 17  2023 write-test3267077572
-rwxrwxrwx 1 root root         0 Sep 22  2022 write-test3297119974
-rwxrwxrwx 1 root root         0 Mar  8  2023 write-test3598281214
-rwxrwxrwx 1 root root         0 Mar  8  2023 write-test3847121906
-rwxrwxrwx 1 root root         0 Mar 18  2023 write-test4026102791
-rwxrwxrwx 1 root root         0 Mar  9  2023 write-test4146183310
-rwxrwxrwx 1 root root         0 Sep 19  2022 write-test423293173
-rwxrwxrwx 1 root root         0 Sep 21  2022 write-test471064762
-rwxrwxrwx 1 root root         0 Sep 14  2022 write-test530076394
-rwxrwxrwx 1 root root         0 Mar  9  2023 write-test543405008
-rwxrwxrwx 1 root root         0 Sep 19  2022 write-test594408963
-rwxrwxrwx 1 root root         0 Mar 17  2023 write-test669119877
-rwxrwxrwx 1 root root         0 Mar 18  2023 write-test786796008
-rwxrwxrwx 1 root root         0 Sep 19  2022 write-test872397235
-rwxrwxrwx 1 root root         0 Sep 21  2022 write-test91682543
-rwxrwxrwx 1 root root         0 Sep 24  2022 write-test917766394

and

root@raspberrypi4:~# docker exec -it storagenode bash
root@14794887d934:/app# echo "test" > config/storage/test.file
root@14794887d934:/app# cat config/storage/test.file
test

and

root@14794887d934:/app/config/storage# ls
DB_Backups                satellites.db             write-test192220591   write-test4026102791
bandwidth.db              storage-dir-verification  write-test2284224945  write-test4146183310
blob                      storage_usage.db          write-test233384524   write-test423293173
blobs                     temp                      write-test2354714472  write-test471064762
garbage                   test.file                 write-test243715213   write-test530076394
info.db                   trash                     write-test2457967941  write-test543405008
orders.db                 used_serial.db            write-test2729741602  write-test594408963
piece_expiration.db       write-test1265540363      write-test2804473926  write-test669119877
piece_spaced_used.db      write-test1324701481      write-test3087336888  write-test786796008
piece_spaced_used.db-wal  write-test1477702234      write-test3267077572  write-test872397235
pieceinfo.db              write-test1839237374      write-test3297119974  write-test91682543
pieceinfo.db-wal          write-test1867707981      write-test3598281214  write-test917766394
reputation.db             write-test1913174638      write-test3847121906

Alexey · December 28, 2023, 7:03am

I cannot reproduce this issue, so it’s not related to the used version, it must be a configuration issue or the issue with the disk.

What is filesystem on this disk?

df --si -T

Xade · December 29, 2023, 1:28pm

Results:

root@raspberrypi4:~# df --si -T
Dateisystem    Typ      Größe Benutzt Verf. Verw% Eingehängt auf
/dev/root      ext4      124G     24G   95G   21% /
devtmpfs       devtmpfs  1,9G       0  1,9G    0% /dev
tmpfs          tmpfs     2,1G    8,2k  2,1G    1% /dev/shm
tmpfs          tmpfs     2,1G     18M  2,0G    1% /run
tmpfs          tmpfs     5,3M    4,1k  5,3M    1% /run/lock
tmpfs          tmpfs     2,1G       0  2,1G    0% /sys/fs/cgroup
/dev/mmcblk0p6 vfat      265M     51M  214M   20% /boot
tmpfs          tmpfs     403M    4,1k  403M    1% /run/user/1000
tmpfs          tmpfs     403M       0  403M    0% /run/user/0
/dev/sda1      fuseblk    13T     11T  2,0T   85% /mnt/Elements12TB

It’s NTFS

daki82 · December 29, 2023, 1:50pm

yep thats one problem, ext4 would be better, can you backup the node and reformat?? preformated external disk i see.
defragmentation not done since years i guess. I have the same drive. At 7TB storj-ride got a bit bumpy…
this could be the timeout error all the time. not the dbs. are the logs redirected? if not you never see the fatal causing the restart.

Xade · December 30, 2023, 1:23am

I guess they are not redirected. How can I do it?

Alexey · December 30, 2023, 2:15am

You need to stop and remove the container, unmount the drive, connect it to the Windows PC and check and fix errors there, then safely eject it on Windows and connect back to Raspberry, mount the drive back, after that try to run the node.

As @daki82 said you need to reformat this drive, the NTFS support is not great under Linux and will always cause issues, first of all - it’s working slower in 2-3 times, the second - it will be always corrupted during an extensive usage and you need to check and fix a filesystem errors under Windows (under Linux the check and fix doesn’t work normally).
You need to backup data, reformat the drive to ext4 and restore data back, then run the node.
You may use this guide for backup: How do I migrate my node to a new device? - Storj Docs

but it’s not necessary.