Database issue (No such table)

[ianllew600]

Hi,

I was wondering if someone could help me with an issues please?

One of my oldest nodes is failing to start after rebooting my server. I know its a problem with storage_usage.db but i don’t know how to fix it.

2023-04-28 17:09:55,795 INFO stopped: storagenode (exit status 1)
2023-04-28 17:09:55,796 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)
2023-04-28 17:09:57,205 INFO Set uid to user 0 succeeded
2023-04-28 17:09:57,214 INFO RPC interface ‘supervisor’ initialized
2023-04-28 17:09:57,214 INFO supervisord started with pid 1
2023-04-28 17:09:57,622 INFO spawned: ‘processes-exit-eventlistener’ with pid 11
2023-04-28 17:09:57,626 INFO spawned: ‘storagenode’ with pid 12
2023-04-28 17:09:57,629 INFO spawned: ‘storagenode-updater’ with pid 13
2023-04-28 17:09:57,630 WARN received SIGTERM indicating exit request
2023-04-28 17:09:57,630 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2023-04-28T17:09:57.657+0100 INFO Configuration loaded {“Process”: “storagenode-updater”, “Location”: “/app/config/config.yaml”}
2023-04-28T17:09:57.657+0100 INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “console.address”}
2023-04-28T17:09:57.657+0100 INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “operator.wallet-features”}
2023-04-28T17:09:57.657+0100 INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “contact.external-address”}
2023-04-28T17:09:57.658+0100 INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “operator.wallet”}
2023-04-28T17:09:57.658+0100 INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “storage.allocated-bandwidth”}
2023-04-28T17:09:57.658+0100 INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “server.address”}
2023-04-28T17:09:57.658+0100 INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “storage.allocated-disk-space”}
2023-04-28T17:09:57.659+0100 INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “healthcheck.details”}
2023-04-28T17:09:57.659+0100 INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “server.private-address”}
2023-04-28T17:09:57.659+0100 INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “healthcheck.enabled”}
2023-04-28T17:09:57.659+0100 INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “operator.email”}
2023-04-28T17:09:57.660+0100 INFO Invalid configuration file value for key {“Process”: “storagenode-updater”, “Key”: “log.output”}
2023-04-28T17:09:57.660+0100 INFO Invalid configuration file value for key {“Process”: “storagenode-updater”, “Key”: “log.stack”}
2023-04-28T17:09:57.660+0100 INFO Invalid configuration file value for key {“Process”: “storagenode-updater”, “Key”: “log.encoding”}
2023-04-28T17:09:57.661+0100 INFO Anonymized tracing enabled {“Process”: “storagenode-updater”}
2023-04-28T17:09:57.664+0100 INFO Running on version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.76.2”}
2023-04-28T17:09:57.665+0100 INFO Downloading versions. {“Process”: “storagenode-updater”, “Server Address”: “https://version.storj.io”}
2023-04-28T17:09:57.696+0100 INFO Configuration loaded {“Process”: “storagenode”, “Location”: “/app/config/config.yaml”}
2023-04-28T17:09:57.697+0100 INFO Anonymized tracing enabled {“Process”: “storagenode”}
2023-04-28T17:09:57.698+0100 INFO Operator email {“Process”: “storagenode”, “Address”: “ianllew600@gmail.com”}
2023-04-28T17:09:57.699+0100 INFO Operator wallet {“Process”: “storagenode”, “Address”: “0xE7FDF9f92cBCA329094FC5a21E4568cE4249435A”}
2023-04-28T17:09:58.179+0100 INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode”, “Version”: “v1.76.2”}
2023-04-28T17:09:58.179+0100 INFO New version is being rolled out but hasn’t made it to this node yet {“Process”: “storagenode-updater”, “Service”: “storagenode”}
2023-04-28T17:09:58.189+0100 INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.76.2”}
2023-04-28T17:09:58.189+0100 INFO New version is being rolled out but hasn’t made it to this node yet {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”}
2023-04-28T17:09:58.290+0100 INFO server kernel support for server-side tcp fast open remains disabled. {“Process”: “storagenode”}
2023-04-28T17:09:58.290+0100 INFO server enable with: sysctl -w net.ipv4.tcp_fastopen=3 {“Process”: “storagenode”}
2023-04-28T17:09:58.761+0100 INFO Telemetry enabled {“Process”: “storagenode”, “instance ID”: “12qRPrHfCBYb3i6pgLsSBKKqFkbrxFg6ZaH2E9iqjQr2FucHeNa”}
2023-04-28T17:09:58.761+0100 INFO Event collection enabled {“Process”: “storagenode”, “instance ID”: “12qRPrHfCBYb3i6pgLsSBKKqFkbrxFg6ZaH2E9iqjQr2FucHeNa”}
2023-04-28 17:09:58,761 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-04-28 17:09:58,761 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-04-28T17:09:58.866+0100 INFO db.migration.54 Add interval_end_time field to storage_usage db, backfill interval_end_time with interval_start, rename interval_start to timestamp {“Process”: “storagenode”}
Error: Error creating tables for master database on storagenode: migrate: v54: no such table: storage_usage
storj.io/storj/private/migrate.(*Migration).Run:209
storj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:354
main.cmdRun:226
storj.io/private/process.cleanup.func1.4:377
storj.io/private/process.cleanup.func1:395
github.com/spf13/cobra.(*Command).execute:852
github.com/spf13/cobra.(*Command).ExecuteC:960
github.com/spf13/cobra.(*Command).Execute:897
storj.io/private/process.ExecWithCustomConfigAndLogger:92
main.main:478
runtime.main:250
2023-04-28 17:09:58,894 INFO exited: storagenode (exit status 1; not expected)
2023-04-28 17:09:59,895 WARN received SIGQUIT indicating exit request
2023-04-28 17:10:00,896 INFO waiting for processes-exit-eventlistener, storagenode-updater to die
2023-04-28 17:10:03,900 INFO waiting for processes-exit-eventlistener, storagenode-updater to die
2023-04-28 17:10:06,905 INFO waiting for processes-exit-eventlistener, storagenode-updater to die

Thats all i have in the logs and then the node restarts again.

Cheers,

[jammerdan]

Wrong used & remaining disk space

Yes I think so. From my experience the software does not recreate single database but only whole set. So what you want to do is to shutdown the node, then move all databases into a new directory that you create. Then start the node. It should create all new databases now, which you can delete except piece_spaced_used.db. This is the one you want to keep. So you shutdown the node again, delete the other databases and move the old databases from the folder you had created back to their original …

[ianllew600]

Hey,

Thanks for the reply, i tried your advice but now its throwing the below errors.

c1.1:83\n\tstorj.io/common/sync2.(*Workplace).Start.func1:89"}
2023-04-28T18:37:52.222+0100 ERROR piecestore:cache error getting current used space: {“Process”: “storagenode”, “error”: “context canceled; context canceled; context canceled; context canceled; context canceled; context canceled”, “errorVerbose”: “group:\n— context canceled\n— context canceled\n— context canceled\n— context canceled\n— context canceled\n— context canceled”}
Error: piecestore monitor: error verifying location and/or readability of storage directory: open config/storage/storage-dir-verification: no such file or directory
2023-04-28 18:37:52,340 INFO exited: storagenode (exit status 1; not expected)
2023-04-28 18:37:53,344 INFO spawned: ‘storagenode’ with pid 47
2023-04-28T18:37:53.399+0100 INFO Configuration loaded {“Process”: “storagenode”, “Location”: “/app/config/config.yaml”}
2023-04-28 18:37:53,399 WARN received SIGQUIT indicating exit request
2023-04-28 18:37:53,400 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2023-04-28T18:37:53.400+0100 INFO Got a signal from the OS: “terminated” {“Process”: “storagenode-updater”}
2023-04-28T18:37:53.401+0100 INFO Anonymized tracing enabled {“Process”: “storagenode”}
2023-04-28T18:37:53.403+0100 INFO Operator email {“Process”: “storagenode”, “Address”: “ianllew600@gmail.com”}
2023-04-28T18:37:53.403+0100 INFO Operator wallet {“Process”: “storagenode”, “Address”: “0xE7FDF9f92cBCA329094FC5a21E4568cE4249435A”}
2023-04-28 18:37:53,404 INFO stopped: storagenode-updater (exit status 0)
2023-04-28T18:37:53.404+0100 INFO Got a signal from the OS: “terminated” {“Process”: “storagenode”}
Error: Error starting master database on storagenode: database: info opening file “config/storage/info.db” failed: context canceled
storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:331
storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:308
storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:283
storj.io/storj/storagenode/storagenodedb.OpenExisting:250
main.cmdRun:193
storj.io/private/process.cleanup.func1.4:377
storj.io/private/process.cleanup.func1:395
github.com/spf13/cobra.(*Command).execute:852
github.com/spf13/cobra.(*Command).ExecuteC:960
github.com/spf13/cobra.(*Command).Execute:897
storj.io/private/process.ExecWithCustomConfigAndLogger:92
main.main:478
runtime.main:250
2023-04-28 18:37:53,417 INFO stopped: storagenode (exit status 1)
2023-04-28 18:37:53,418 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)
2023-04-28 18:37:54,459 INFO Set uid to user 0 succeeded
2023-04-28 18:37:54,468 INFO RPC interface ‘supervisor’ initialized
2023-04-28 18:37:54,469 INFO supervisord started with pid 1
2023-04-28 18:37:55,471 INFO spawned: ‘processes-exit-eventlistener’ with pid 12
2023-04-28 18:37:55,475 INFO spawned: ‘storagenode’ with pid 13
2023-04-28 18:37:55,478 INFO spawned: ‘storagenode-updater’ with pid 14
2023-04-28T18:37:55.508+0100 INFO Configuration loaded {“Process”: “storagenode-updater”, “Location”: “/app/config/config.yaml”}
2023-04-28T18:37:55.509+0100 INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “contact.external-address”}
2023-04-28T18:37:55.509+0100 INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “healthcheck.enabled”}
2023-04-28T18:37:55.509+0100 INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “operator.wallet-features”}
2023-04-28T18:37:55.509+0100 INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “operator.wallet”}
2023-04-28T18:37:55.510+0100 INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “server.private-address”}
2023-04-28T18:37:55.510+0100 INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “healthcheck.details”}
2023-04-28T18:37:55.510+0100 INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “storage.allocated-disk-space”}
2023-04-28T18:37:55.510+0100 INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “operator.email”}
2023-04-28T18:37:55.510+0100 INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “console.address”}
2023-04-28T18:37:55.511+0100 INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “server.address”}
2023-04-28T18:37:55.511+0100 INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “storage.allocated-bandwidth”}
2023-04-28T18:37:55.512+0100 INFO Anonymized tracing enabled {“Process”: “storagenode-updater”}
2023-04-28T18:37:55.514+0100 INFO Running on version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.76.2”}
2023-04-28T18:37:55.514+0100 INFO Downloading versions. {“Process”: “storagenode-updater”, “Server Address”: “https://version.storj.io”}
2023-04-28T18:37:55.545+0100 INFO Configuration loaded {“Process”: “storagenode”, “Location”: “/app/config/config.yaml”}
2023-04-28T18:37:55.546+0100 INFO Anonymized tracing enabled {“Process”: “storagenode”}
2023-04-28T18:37:55.548+0100 INFO Operator email {“Process”: “storagenode”, “Address”: “ianllew600@gmail.com”}
2023-04-28T18:37:55.548+0100 INFO Operator wallet {“Process”: “storagenode”, “Address”: “0xE7FDF9f92cBCA329094FC5a21E4568cE4249435A”}
2023-04-28T18:37:55.905+0100 INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode”, “Version”: “v1.76.2”}
2023-04-28T18:37:55.905+0100 INFO New version is being rolled out but hasn’t made it to this node yet {“Process”: “storagenode-updater”, “Service”: “storagenode”}
2023-04-28T18:37:55.922+0100 INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.76.2”}
2023-04-28T18:37:55.922+0100 INFO New version is being rolled out but hasn’t made it to this node yet {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”}
2023-04-28T18:37:55.981+0100 INFO server kernel support for server-side tcp fast open remains disabled. {“Process”: “storagenode”}
2023-04-28T18:37:55.981+0100 INFO server enable with: sysctl -w net.ipv4.tcp_fastopen=3 {“Process”: “storagenode”}
2023-04-28 18:37:56,209 WARN received SIGTERM indicating exit request
2023-04-28 18:37:56,209 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2023-04-28T18:37:56.209+0100 INFO Got a signal from the OS: “terminated” {“Process”: “storagenode-updater”}
2023-04-28 18:37:56,212 INFO stopped: storagenode-updater (exit status 0)
2023-04-28T18:37:56.212+0100 INFO Got a signal from the OS: “terminated” {“Process”: “storagenode”}
2023-04-28T18:37:56.214+0100 INFO Telemetry enabled {“Process”: “storagenode”, “instance ID”: “12qRPrHfCBYb3i6pgLsSBKKqFkbrxFg6ZaH2E9iqjQr2FucHeNa”}
2023-04-28T18:37:56.214+0100 INFO Event collection enabled {“Process”: “storagenode”, “instance ID”: “12qRPrHfCBYb3i6pgLsSBKKqFkbrxFg6ZaH2E9iqjQr2FucHeNa”}
Error: Error creating tables for master database on storagenode: migrate: database: info opening file “config/storage/info.db” failed: context canceled
storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:331
storj.io/storj/storagenode/storagenodedb.(*DB).Migration.func1:636
storj.io/storj/private/migrate.(*Migration).Run:164
storj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:354
main.cmdRun:226
storj.io/private/process.cleanup.func1.4:377
storj.io/private/process.cleanup.func1:395
github.com/spf13/cobra.(*Command).execute:852
github.com/spf13/cobra.(*Command).ExecuteC:960
github.com/spf13/cobra.(*Command).Execute:897
storj.io/private/process.ExecWithCustomConfigAndLogger:92
main.main:478
runtime.main:250
2023-04-28 18:37:56,254 INFO stopped: storagenode (exit status 1)
2023-04-28 18:37:56,255 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)

Cheers,

[jammerdan]

You need to check if your node starts without problems with completely new databases.
For that move your existing databases into a temp folder and start the node.
It should re-create all databases.
Now you keep only those the node has been complaining over and take the rest from your old databases from the temp folder.
Then start your node again and check if it still complaining.

[ianllew600]

Thanks for the replies, its really appriciated.

I followed your instructions and it looks like the node is back up. According to the logs uploads and downloads are running again. I can’t see any errors flagging in there.

When I open the Web interface, most things are reporting the correct values, except this months bandwidth and average disk space used, but i guess that would be the case as those are the two db’s that corrupted.

[Alexey]

@ianllew600

except this months bandwidth and average disk space used, but i guess that would be the case as those are the two db’s that corrupted.

Yes, and since you re-created them, your historic data is lost unfortunately.