I can screenshot it if need be, the above looks like a chicken got slash happy over the weekend.
It looks alright. Are you sure it isn’t already running in Docker?
I’m not sure.
I am unable to access the web or CLI dashboards. What other method is there to see if the node is running?
Type
docker ps
It should list the containers that are running.
I notice some differences between node1 and node2. What could cause this?
What differences do you mean? They are both running so you can check the logs to see their status.
Node1 is showing:
0.0.0.0:14002->14002/tcp, :::14002->14002/tcp
Node2 is only showing:
127.0.0.1:14003->14002/tcp
I understand the “14003->14002” in node2.
I need help understanding why node1 uses “0.0.0.0” and node2 uses “127.0.0.1”
I need help understanding why node2 does not have the second set of numbers “:::14003->14002/tcp” could the cause in variation prevent the node from communicating with the dashboards?
logs… where do I find those?
In your run command you specified 127.0.0.1 on the one node and the other you removed the IP. Typically you remove the IP so you can access the dashboard from a different computer than the host.
You are running two nodes and you have never checked your logs and don’t have access to the dashboards? Is your first node new? Why setup a second node? You won’t get data twice as fast and may actually get slightly less data. On full nodes you can maybe maximize egress by having a big pipe and multiple nodes. But if you are just starting out, a single node should serve you fine.
Anyway… You can do a ‘docker logs storagenode’ but you’ll maybe want to grep the logs or use the tail command to view the end of the logs. You can search the forum for details on this. I think we may have log checking info on the Storj Node setup instructions as well. I’m not in front of the PC at the moment or I’d provide a link.
Great points, the first node is an 8Tb drive that is pushing 80% full. I’ve had the node for about 2 years now. I would prefer to set up a second node with a second cpu. But that is not an option at the moment. I would like to have a second node at a different location, but I have only one home. so, one machine and two containers it is.
I’ll remove the IP for the second node and see what happens. I’ll take a look at the logs too and see if that points me in any direction.
1 Like
in this case you need to run the dashboard with providing a port
docker exec -it storagenode2 ./dashboard.sh --server.private-address=127.0.0.1:7779
But usually you should not change anything in the config.yaml file if you use a docker version, unless you are using --network host.
So, you may return the private address back to default,
server.private-address: 127.0.0.1:7778
Then docker exec -it storagenode2 ./dashboard.sh should work.
if you changed the internal listening address (which usually is not needed for docker nodes)
then your port mapping should be -p 28968:28968/tcp -p 28968:28968/udp
However, if you revert your changes, it should just work with your current docker run command for the second node.
I really appreciate your time and level of detail, thank you.
Here is an update, apparently the hard drive with the OS became corrupted. So, I have been in the process of getting a new drive and all the things that go along with that. I successfully got node1 up and running. Node2 is running, but its currently offline and the QUIC is misconfigured. As previously shown in the last screenshot.
I will pick up where we left off and make those changes promptly and let you know where I stand. again, thank you.
The recommendations above allowed me to see the node’s dashboard via CLI.
It was showing offline. I went to my router, changed some port forwarding stuff. Now, I am REALLY confused. The docker run command uses ports 28968 and 28967 while the port forwarding rule is for 28968 only. No need to explain this. Its green and green. I’m showing ingress and egress, so its good and working, I don’t need to know how. Unless you just want to point me towards some resources so I can read up on it.
But, thank you so much @Alexey I literally could not have done this without your help. I really appreciate it!
1 Like
You may take a look on:
and
1 Like
Ok, i got two nodes 2x 10TB, both mostly filled up to capacity. One of them failed today - seemingly - for no reason. Uptime robot is reporting it went down cca 9.40 AM. Both nodes seem updated to the latest version (watchtower) & ports are open.
What I tried so far:
stopped the node, remove, restart → upon starting CLI dash it says: Error: rpc: dial tcp 127.0.0.1:7778: connect: connection refused
“docker ps” shows node as restarting (never actually completes restart), if i try to stop & rm node, its giving me connection refused again. What gives? Should i give up, given its aready 9 hours of OFFLINE or is it possible to fix it. There is 9TB of data on it after all, which took a while, …
Also, ports are open, connectivity is fine, second node is ok&online
oh WOW! Yea, respect on the 2 x 10T almost full. It has taken me almost 3 years to fill an 8TB to about 80% full. so, to have 9TB of data to poof away… I have not seen a solution to this. The only thing I can contribute is to not give up on a node after being down for 9 hours. Just a couple weeks ago, my node went offline for over 3 days!!! I’m still in the red and trying to recover
Here is a screen shot. Yesterday I was in the 80’s fyi. So, if you can somehow manage to get the drive and node back online within 3 days, you should be okay.
1 Like
Hi, as I stated in “connection refused” post, one of my nodes went offline. There is 9TB of data on it & it would suck if i just let it die.
There is something about not being able to access DB due to a configuration error or something. My gut tells me something got corrupted, maybe failing drive. Anyway, i need some help with diagnosing this problem. Thank you.
Im pasting excerpt of my logs:
Error: Error starting master database on storagenode: database: bandwidth opening file "config/storage/bandwidth.db" fai led: disk I/O error: input/output error
storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:331
storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:308
storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:283
storj.io/storj/storagenode/storagenodedb.OpenExisting:250
main.cmdRun:193
storj.io/private/process.cleanup.func1.4:377
storj.io/private/process.cleanup.func1:395
github.com/spf13/cobra.(*Command).execute:852
github.com/spf13/cobra.(*Command).ExecuteC:960
github.com/spf13/cobra.(*Command).Execute:897
storj.io/private/process.ExecWithCustomConfigAndLogger:92
main.main:478
runtime.main:250
2023-03-11 04:56:37,888 INFO exited: storagenode (exit status 1; not expected)
2023-03-11 04:56:37,890 INFO spawned: 'storagenode' with pid 39
2023-03-11 04:56:37,891 WARN received SIGQUIT indicating exit request
2023-03-11 04:56:37,891 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2023-03-11T04:56:37.891Z INFO Got a signal from the OS: "terminated" {"Process": "storagenode-updater"}
2023-03-11 04:56:37,892 INFO stopped: storagenode-updater (exit status 0)
2023-03-11T04:56:37.924Z INFO Configuration loaded {"Process": "storagenode", "Location": "/app/config/conf ig.yaml"}
2023-03-11T04:56:37.924Z INFO Anonymized tracing enabled {"Process": "storagenode"}
2023-03-11T04:56:37.925Z INFO Operator email {"Process": "storagenode", "Address": "EMAIL"}
2023-03-11T04:56:37.925Z INFO Operator wallet {"Process": "storagenode", "Address": "0xWALLET"}
2023-03-11 04:56:40,929 INFO waiting for storagenode, processes-exit-eventlistener to die
2023-03-11 04:56:43,934 INFO waiting for storagenode, processes-exit-eventlistener to die
2023-03-11 04:56:46,937 INFO waiting for storagenode, processes-exit-eventlistener to die
2023-03-11 04:56:47,939 WARN killing 'storagenode' (39) with SIGKILL
2023-03-11 04:56:47,943 INFO stopped: storagenode (terminated by SIGKILL)
2023-03-11 04:56:47,944 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)
2023-03-11 04:56:48,895 INFO Set uid to user 0 succeeded
2023-03-11 04:56:48,903 INFO RPC interface 'supervisor' initialized
2023-03-11 04:56:48,903 INFO supervisord started with pid 1
2023-03-11 04:56:49,905 INFO spawned: 'processes-exit-eventlistener' with pid 11
2023-03-11 04:56:49,907 INFO spawned: 'storagenode' with pid 12
2023-03-11 04:56:49,908 INFO spawned: 'storagenode-updater' with pid 13
2023-03-11T04:56:49.920Z INFO Configuration loaded {"Process": "storagenode-updater", "Location": "/app/con fig/config.yaml"}
2023-03-11T04:56:49.920Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "stora ge.allocated-disk-space"}
2023-03-11T04:56:49.920Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "opera tor.wallet"}
2023-03-11T04:56:49.920Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "opera tor.email"}
2023-03-11T04:56:49.920Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "stora ge.allocated-bandwidth"}
2023-03-11T04:56:49.920Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "serve r.address"}
2023-03-11T04:56:49.920Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "opera tor.wallet-features.0"}
2023-03-11T04:56:49.920Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "serve r.private-address"}
2023-03-11T04:56:49.920Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "conta ct.external-address"}
2023-03-11T04:56:49.920Z INFO Invalid configuration file value for key {"Process": "storagenode-updater ", "Key": "log.level"}
2023-03-11T04:56:49.920Z INFO Invalid configuration file value for key {"Process": "storagenode-updater ", "Key": "log.stack"}
2023-03-11T04:56:49.920Z INFO Anonymized tracing enabled {"Process": "storagenode-updater"}
2023-03-11T04:56:49.922Z INFO Running on version {"Process": "storagenode-updater", "Service": "storageno de-updater", "Version": "v1.74.1"}
2023-03-11T04:56:49.922Z INFO Downloading versions. {"Process": "storagenode-updater", "Server Address": "ht tps://version.storj.io"}
2023-03-11T04:56:49.948Z INFO Configuration loaded {"Process": "storagenode", "Location": "/app/config/conf ig.yaml"}
2023-03-11T04:56:49.948Z INFO Anonymized tracing enabled {"Process": "storagenode"}
2023-03-11T04:56:49.949Z INFO Operator email {"Process": "storagenode", "Address": "EMAIL"}
2023-03-11T04:56:49.949Z INFO Operator wallet {"Process": "storagenode", "Address": "0xWALLET"}
2023-03-11T04:56:50.410Z INFO Current binary version {"Process": "storagenode-updater", "Service": "storageno de", "Version": "v1.74.1"}
2023-03-11T04:56:50.410Z INFO Version is up to date {"Process": "storagenode-updater", "Service": "storageno de"}
2023-03-11T04:56:50.420Z INFO Current binary version {"Process": "storagenode-updater", "Service": "storageno de-updater", "Version": "v1.74.1"}
2023-03-11T04:56:50.420Z INFO Version is up to date {"Process": "storagenode-updater", "Service": "storageno de-updater"}
2023-03-11 04:56:51,421 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for th an 1 seconds (startsecs)
2023-03-11 04:56:51,422 INFO success: storagenode entered RUNNING state, process has stayed up for than 1 seconds (sta rtsecs)
2023-03-11 04:56:51,422 INFO success: storagenode-updater entered RUNNING state, process has stayed up for than 1 seco nds (startsecs)
Error: Error starting master database on storagenode: database: bandwidth opening file "config/storage/bandwidth.db" fai led: disk I/O error: input/output error
storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:331
storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:308
storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:283
storj.io/storj/storagenode/storagenodedb.OpenExisting:250
main.cmdRun:193
storj.io/private/process.cleanup.func1.4:377
storj.io/private/process.cleanup.func1:395
github.com/spf13/cobra.(*Command).execute:852
github.com/spf13/cobra.(*Command).ExecuteC:960
github.com/spf13/cobra.(*Command).Execute:897
storj.io/private/process.ExecWithCustomConfigAndLogger:92
main.main:478
runtime.main:250
2023-03-11 04:57:36,887 INFO exited: storagenode (exit status 1; not expected)
2023-03-11 04:57:37,893 INFO spawned: 'storagenode' with pid 42
2023-03-11 04:57:37,894 WARN received SIGQUIT indicating exit request
2023-03-11 04:57:37,894 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2023-03-11T04:57:37.894Z INFO Got a signal from the OS: "terminated" {"Process": "storagenode-updater"}
2023-03-11 04:57:37,896 INFO stopped: storagenode-updater (exit status 0)
2023-03-11T04:57:37.930Z INFO Configuration loaded {"Process": "storagenode", "Location": "/app/config/conf ig.yaml"}
2023-03-11T04:57:37.930Z INFO Anonymized tracing enabled {"Process": "storagenode"}
2023-03-11T04:57:37.931Z INFO Operator email {"Process": "storagenode", "Address": "EMAIL"}
2023-03-11T04:57:37.931Z INFO Operator wallet {"Process": "storagenode", "Address": "0xWALLET"}
2023-03-11 04:57:40,936 INFO waiting for storagenode, processes-exit-eventlistener to die
2023-03-11 04:57:43,940 INFO waiting for storagenode, processes-exit-eventlistener to die
2023-03-11 04:57:46,944 INFO waiting for storagenode, processes-exit-eventlistener to die
2023-03-11 04:57:47,945 WARN killing 'storagenode' (42) with SIGKILL
2023-03-11 04:57:47,949 INFO stopped: storagenode (terminated by SIGKILL)
2023-03-11 04:57:47,951 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)
2023-03-11 04:57:48,793 INFO Set uid to user 0 succeeded
2023-03-11 04:57:48,801 INFO RPC interface 'supervisor' initialized
2023-03-11 04:57:48,802 INFO supervisord started with pid 1
2023-03-11 04:57:49,805 INFO spawned: 'processes-exit-eventlistener' with pid 11
2023-03-11 04:57:49,811 INFO spawned: 'storagenode' with pid 12
2023-03-11 04:57:49,812 INFO spawned: 'storagenode-updater' with pid 13
2023-03-11T04:57:49.823Z INFO Configuration loaded {"Process": "storagenode-updater", "Location": "/app/con fig/config.yaml"}
2023-03-11T04:57:49.824Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "opera tor.email"}
2023-03-11T04:57:49.824Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "stora ge.allocated-bandwidth"}
2023-03-11T04:57:49.824Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "opera tor.wallet-features.0"}
2023-03-11T04:57:49.824Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "conta ct.external-address"}
2023-03-11T04:57:49.825Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "serve r.private-address"}
2023-03-11T04:57:49.825Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "opera tor.wallet"}
2023-03-11T04:57:49.825Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "stora ge.allocated-disk-space"}
2023-03-11T04:57:49.825Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "serve r.address"}
2023-03-11T04:57:49.826Z INFO Invalid configuration file value for key {"Process": "storagenode-updater ", "Key": "log.output"}
2023-03-11T04:57:49.826Z INFO Invalid configuration file value for key {"Process": "storagenode-updater ", "Key": "log.stack"}
2023-03-11T04:57:49.826Z INFO Anonymized tracing enabled {"Process": "storagenode-updater"}
2023-03-11T04:57:49.827Z INFO Running on version {"Process": "storagenode-updater", "Service": "storageno de-updater", "Version": "v1.74.1"}
2023-03-11T04:57:49.828Z INFO Downloading versions. {"Process": "storagenode-updater", "Server Address": "ht tps://version.storj.io"}
2023-03-11T04:57:49.849Z INFO Configuration loaded {"Process": "storagenode", "Location": "/app/config/conf ig.yaml"}
2023-03-11T04:57:49.849Z INFO Anonymized tracing enabled {"Process": "storagenode"}
2023-03-11T04:57:49.850Z INFO Operator email {"Process": "storagenode", "Address": "EMAIL"}
2023-03-11T04:57:49.850Z INFO Operator wallet {"Process": "storagenode", "Address": "0xWALLET"}
2023-03-11T04:57:50.349Z INFO Current binary version {"Process": "storagenode-updater", "Service": "storageno de", "Version": "v1.74.1"}
2023-03-11T04:57:50.350Z INFO Version is up to date {"Process": "storagenode-updater", "Service": "storageno de"}
2023-03-11T04:57:50.361Z INFO Current binary version {"Process": "storagenode-updater", "Service": "storageno de-updater", "Version": "v1.74.1"}
2023-03-11T04:57:50.361Z INFO Version is up to date {"Process": "storagenode-updater", "Service": "storageno de-updater"}
2023-03-11 04:57:51,363 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for th an 1 seconds (startsecs)
2023-03-11 04:57:51,363 INFO success: storagenode entered RUNNING state, process has stayed up for than 1 seconds (sta rtsecs)
2023-03-11 04:57:51,363 INFO success: storagenode-updater entered RUNNING state, process has stayed up for than 1 seco nds (startsecs)
Error: Error starting master database on storagenode: database: bandwidth opening file "config/storage/bandwidth.db" fai led: disk I/O error: input/output error
storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:331
storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:308
storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:283
storj.io/storj/storagenode/storagenodedb.OpenExisting:250
main.cmdRun:193
storj.io/private/process.cleanup.func1.4:377
storj.io/private/process.cleanup.func1:395
github.com/spf13/cobra.(*Command).execute:852
github.com/spf13/cobra.(*Command).ExecuteC:960
github.com/spf13/cobra.(*Command).Execute:897
storj.io/private/process.ExecWithCustomConfigAndLogger:92
main.main:478
runtime.main:250
2023-03-11 04:58:42,692 INFO exited: storagenode (exit status 1; not expected)
2023-03-11 04:58:43,698 INFO spawned: 'storagenode' with pid 41
2023-03-11 04:58:43,699 WARN received SIGQUIT indicating exit request
2023-03-11 04:58:43,701 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2023-03-11T04:58:43.700Z INFO Got a signal from the OS: "terminated" {"Process": "storagenode-updater"}
2023-03-11 04:58:43,701 INFO stopped: storagenode-updater (exit status 0)
2023-03-11 04:58:43,702 INFO stopped: storagenode (terminated by SIGTERM)
2023-03-11 04:58:43,703 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)
2023-03-11 04:58:44,596 INFO Set uid to user 0 succeeded
2023-03-11 04:58:44,605 INFO RPC interface 'supervisor' initialized
2023-03-11 04:58:44,605 INFO supervisord started with pid 1
2023-03-11 04:58:45,608 INFO spawned: 'processes-exit-eventlistener' with pid 11
2023-03-11 04:58:45,612 INFO spawned: 'storagenode' with pid 12
2023-03-11 04:58:45,613 INFO spawned: 'storagenode-updater' with pid 13
2023-03-11T04:58:45.624Z INFO Configuration loaded {"Process": "storagenode-updater", "Location": "/app/con fig/config.yaml"}
2023-03-11T04:58:45.624Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "stora ge.allocated-disk-space"}
2023-03-11T04:58:45.625Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "opera tor.wallet"}
2023-03-11T04:58:45.625Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "serve r.address"}
2023-03-11T04:58:45.625Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "opera tor.wallet-features.0"}
2023-03-11T04:58:45.625Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "conta ct.external-address"}
2023-03-11T04:58:45.626Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "stora ge.allocated-bandwidth"}
2023-03-11T04:58:45.626Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "opera tor.email"}
2023-03-11T04:58:45.626Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "serve r.private-address"}
2023-03-11T04:58:45.626Z INFO Invalid configuration file value for key {"Process": "storagenode-updater ", "Key": "log.encoding"}
2023-03-11T04:58:45.627Z INFO Invalid configuration file value for key {"Process": "storagenode-updater ", "Key": "log.level"}
2023-03-11T04:58:45.627Z INFO Invalid configuration file value for key {"Process": "storagenode-updater ", "Key": "log.output"}
2023-03-11T04:58:45.627Z INFO Invalid configuration file value for key {"Process": "storagenode-updater ", "Key": "log.stack"}
2023-03-11T04:58:45.627Z INFO Invalid configuration file value for key {"Process": "storagenode-updater ", "Key": "log.caller"}
2023-03-11T04:58:45.627Z INFO Invalid configuration file value for key {"Process": "storagenode-updater ", "Key": "log.development"}
2023-03-11T04:58:45.627Z INFO Anonymized tracing enabled {"Process": "storagenode-updater"}
2023-03-11T04:58:45.629Z INFO Running on version {"Process": "storagenode-updater", "Service": "storageno de-updater", "Version": "v1.74.1"}
2023-03-11T04:58:45.630Z INFO Downloading versions. {"Process": "storagenode-updater", "Server Address": "ht tps://version.storj.io"}
2023-03-11T04:58:45.652Z INFO Configuration loaded {"Process": "storagenode", "Location": "/app/config/conf ig.yaml"}
2023-03-11T04:58:45.652Z INFO Anonymized tracing enabled {"Process": "storagenode"}
2023-03-11T04:58:45.653Z INFO Operator email {"Process": "storagenode", "Address": "EMAIL"}
2023-03-11T04:58:45.654Z INFO Operator wallet {"Process": "storagenode", "Address": "0xWALLET"}
2023-03-11T04:58:46.127Z INFO Current binary version {"Process": "storagenode-updater", "Service": "storageno de", "Version": "v1.74.1"}
2023-03-11T04:58:46.127Z INFO Version is up to date {"Process": "storagenode-updater", "Service": "storageno de"}
2023-03-11T04:58:46.138Z INFO Current binary version {"Process": "storagenode-updater", "Service": "storageno de-updater", "Version": "v1.74.1"}
2023-03-11T04:58:46.138Z INFO Version is up to date {"Process": "storagenode-updater", "Service": "storageno de-updater"}
2023-03-11 04:58:47,139 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for th an 1 seconds (startsecs)
2023-03-11 04:58:47,140 INFO success: storagenode entered RUNNING state, process has stayed up for than 1 seconds (sta rtsecs)
2023-03-11 04:58:47,140 INFO success: storagenode-updater entered RUNNING state, process has stayed up for than 1 seco nds (startsecs)
Error: Error starting master database on storagenode: database: bandwidth opening file "config/storage/bandwidth.db" fai led: disk I/O error: input/output error
storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:331
storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:308
storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:283
storj.io/storj/storagenode/storagenodedb.OpenExisting:250
main.cmdRun:193
storj.io/private/process.cleanup.func1.4:377
storj.io/private/process.cleanup.func1:395
github.com/spf13/cobra.(*Command).execute:852
github.com/spf13/cobra.(*Command).ExecuteC:960
github.com/spf13/cobra.(*Command).Execute:897
storj.io/private/process.ExecWithCustomConfigAndLogger:92
main.main:478
runtime.main:250
2023-03-11 04:59:24,559 INFO exited: storagenode (exit status 1; not expected)
2023-03-11 04:59:25,565 INFO spawned: 'storagenode' with pid 43
2023-03-11 04:59:25,566 WARN received SIGQUIT indicating exit request
2023-03-11 04:59:25,570 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2023-03-11T04:59:25.567Z INFO Got a signal from the OS: "terminated" {"Process": "storagenode-updater"}
2023-03-11 04:59:25,572 INFO stopped: storagenode-updater (exit status 0)
2023-03-11T04:59:25.602Z INFO Configuration loaded {"Process": "storagenode", "Location": "/app/config/conf ig.yaml"}
2023-03-11T04:59:25.602Z INFO Anonymized tracing enabled {"Process": "storagenode"}
2023-03-11T04:59:25.603Z INFO Operator email {"Process": "storagenode", "Address": "EMAIL"}
2023-03-11T04:59:25.603Z INFO Operator wallet {"Process": "storagenode", "Address": "0xWALLET"}
2023-03-11 04:59:28,607 INFO waiting for storagenode, processes-exit-eventlistener to die
2023-03-11 04:59:31,611 INFO waiting for storagenode, processes-exit-eventlistener to die
2023-03-11 04:59:34,615 INFO waiting for storagenode, processes-exit-eventlistener to die
2023-03-11 04:59:35,616 WARN killing 'storagenode' (43) with SIGKILL
2023-03-11 04:59:35,620 INFO stopped: storagenode (terminated by SIGKILL)
2023-03-11 04:59:35,622 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)
2023-03-11 04:59:36,508 INFO Set uid to user 0 succeeded
2023-03-11 04:59:36,517 INFO RPC interface 'supervisor' initialized
2023-03-11 04:59:36,517 INFO supervisord started with pid 1
2023-03-11 04:59:37,518 INFO spawned: 'processes-exit-eventlistener' with pid 11
2023-03-11 04:59:37,519 INFO spawned: 'storagenode' with pid 12
2023-03-11 04:59:37,521 INFO spawned: 'storagenode-updater' with pid 13
2023-03-11T04:59:37.531Z INFO Configuration loaded {"Process": "storagenode-updater", "Location": "/app/con fig/config.yaml"}
2023-03-11T04:59:37.531Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "stora ge.allocated-disk-space"}
2023-03-11T04:59:37.532Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "stora ge.allocated-bandwidth"}
2023-03-11T04:59:37.532Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "conta ct.external-address"}
2023-03-11T04:59:37.532Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "opera tor.email"}
2023-03-11T04:59:37.532Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "opera tor.wallet-features.0"}
2023-03-11T04:59:37.533Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "serve r.private-address"}
2023-03-11T04:59:37.533Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "opera tor.wallet"}
2023-03-11T04:59:37.533Z INFO Invalid configuration file key {"Process": "storagenode-updater", "Key": "serve r.address"}
2023-03-11T04:59:37.534Z INFO Anonymized tracing enabled {"Process": "storagenode-updater"}
2023-03-11T04:59:37.534Z INFO Running on version {"Process": "storagenode-updater", "Service": "storageno de-updater", "Version": "v1.74.1"}
2023-03-11T04:59:37.535Z INFO Downloading versions. {"Process": "storagenode-updater", "Server Address": "ht tps://version.storj.io"}
2023-03-11T04:59:37.561Z INFO Configuration loaded {"Process": "storagenode", "Location": "/app/config/conf ig.yaml"}
2023-03-11T04:59:37.562Z INFO Anonymized tracing enabled {"Process": "storagenode"}
2023-03-11T04:59:37.563Z INFO Operator email {"Process": "storagenode", "Address": "EMAIL"}
2023-03-11T04:59:37.563Z INFO Operator wallet {"Process": "storagenode", "Address": "0xWALLET"}
2023-03-11T04:59:38.014Z INFO Current binary version {"Process": "storagenode-updater", "Service": "storageno de", "Version": "v1.74.1"}
2023-03-11T04:59:38.014Z INFO Version is up to date {"Process": "storagenode-updater", "Service": "storageno de"}
2023-03-11T04:59:38.025Z INFO Current binary version {"Process": "storagenode-updater", "Service": "storageno de-updater", "Version": "v1.74.1"}
2023-03-11T04:59:38.025Z INFO Version is up to date {"Process": "storagenode-updater", "Service": "storageno de-updater"}
2023-03-11 04:59:39,027 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for th an 1 seconds (startsecs)
2023-03-11 04:59:39,027 INFO success: storagenode entered RUNNING state, process has stayed up for than 1 seconds (sta rtsecs)
2023-03-11 04:59:39,027 INFO success: storagenode-updater entered RUNNING state, process has stayed up for than 1 seco nds (startsecs)
Error: Error starting master database on storagenode: database: bandwidth opening file "config/storage/bandwidth.db" fai led: disk I/O error: input/output error
storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:331
storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:308
storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:283
storj.io/storj/storagenode/storagenodedb.OpenExisting:250
main.cmdRun:193
storj.io/private/process.cleanup.func1.4:377
storj.io/private/process.cleanup.func1:395
github.com/spf13/cobra.(*Command).execute:852
github.com/spf13/cobra.(*Command).ExecuteC:960
github.com/spf13/cobra.(*Command).Execute:897
storj.io/private/process.ExecWithCustomConfigAndLogger:92
main.main:478
runtime.main:250
You need to fix the disk errors.