Your node is not disqualified (the audit score is 100%), so it just was offline for a while and only the online score is affected.
if you fixed an offline issue, you need to keep it online for the next 30 days to fully recover.
Each downtime event requires another 30 days to recover, so keep it online.
im running linux (ubuntu). Any idea how to go about it?
1 Like
while trying to make a backup of db files, i encontered this problem:
Seems bandwith.db is corrupted…
Run fsck on your drive to check for disk errors first.
1 Like
Thanks for the help, im running fsck now, might take a while - will report when done.
Show the output of this command df -h
fsck works now & its running. Problem was that HDD was in use and locked somehow. I restarted server & rm storagenode, unmounted & now it says it repairing filesystem. top process -->gnome-discs, so it is doing something.
df -h:
Filesystem Size Used Avail Use% Mounted on
udev 7,7G 0 7,7G 0% /dev
tmpfs 1,6G 4,0M 1,6G 1% /run
/dev/sdf5 109G 86G 18G 83% /
tmpfs 7,7G 8,0K 7,7G 1% /dev/shm
tmpfs 5,0M 4,0K 5,0M 1% /run/lock
tmpfs 7,7G 0 7,7G 0% /sys/fs/cgroup
/dev/loop2 56M 56M 0 100% /snap/core18/2697
/dev/loop3 64M 64M 0 100% /snap/core20/1822
/dev/loop0 56M 56M 0 100% /snap/core18/2708
/dev/loop4 64M 64M 0 100% /snap/core20/1828
/dev/loop5 128K 128K 0 100% /snap/bare/5
/dev/loop6 73M 73M 0 100% /snap/core22/509
/dev/loop1 117M 117M 0 100% /snap/core/14447
/dev/loop7 73M 73M 0 100% /snap/core22/522
/dev/loop8 141M 141M 0 100% /snap/gnome-3-26-1604/104
/dev/loop9 453M 453M 0 100% /snap/gnome-42-2204/56
/dev/loop11 461M 461M 0 100% /snap/gnome-42-2204/65
/dev/loop10 347M 347M 0 100% /snap/gnome-3-38-2004/115
/dev/loop12 163M 163M 0 100% /snap/gnome-3-28-1804/145
/dev/loop14 141M 141M 0 100% /snap/gnome-3-26-1604/102
/dev/loop13 2,7M 2,7M 0 100% /snap/gnome-system-monitor/178
/dev/loop15 9,4M 9,4M 0 100% /snap/sensors-unity/257
/dev/loop16 12M 12M 0 100% /snap/nmap/2944
/dev/loop17 219M 219M 0 100% /snap/gnome-3-34-1804/72
/dev/loop19 6,3M 6,3M 0 100% /snap/notepad-plus-plus/385
/dev/loop18 92M 92M 0 100% /snap/gtk-common-themes/1535
/dev/loop21 1,5M 1,5M 0 100% /snap/gnome-system-monitor/181
/dev/loop20 219M 219M 0 100% /snap/gnome-3-34-1804/77
/dev/loop23 117M 117M 0 100% /snap/core/14784
/dev/loop25 100M 100M 0 100% /snap/wine-platform-3-stable/14
/dev/loop22 46M 46M 0 100% /snap/snap-store/638
/dev/loop24 3,5M 3,5M 0 100% /snap/mjpg-streamer/42
/dev/loop26 9,4M 9,4M 0 100% /snap/sensors-unity/253
/dev/loop27 88M 88M 0 100% /snap/sqlitebrowser/3233
/dev/sdf1 511M 6,1M 505M 2% /boot/efi
/dev/loop28 304M 304M 0 100% /snap/wine-platform-5-stable/18
/dev/loop29 100M 100M 0 100% /snap/wine-platform-3-stable/11
/dev/loop30 12M 12M 0 100% /snap/nmap/2864
/dev/loop31 323M 323M 0 100% /snap/wine-platform-6-stable/19
/dev/loop32 348M 348M 0 100% /snap/wine-platform-runtime/336
/dev/loop33 304M 304M 0 100% /snap/wine-platform-5-stable/16
/dev/loop35 46M 46M 0 100% /snap/snap-store/599
/dev/loop34 165M 165M 0 100% /snap/gnome-3-28-1804/161
/dev/sda1 916G 830G 40G 96% /home/cynicaldrink/STORJ_1TB
/dev/loop37 6,2M 6,2M 0 100% /snap/notepad-plus-plus/386
/dev/loop36 454M 454M 0 100% /snap/wine-platform-7-devel-core20/24
/dev/loop39 348M 348M 0 100% /snap/wine-platform-runtime/338
/dev/loop40 466M 466M 0 100% /snap/wine-platform-runtime-core20/55
/dev/loop38 466M 466M 0 100% /snap/wine-platform-runtime-core20/56
/dev/loop41 347M 347M 0 100% /snap/gnome-3-38-2004/119
/dev/loop42 323M 323M 0 100% /snap/wine-platform-6-stable/14
/dev/loop43 452M 452M 0 100% /snap/wine-platform-7-devel-core20/23
/dev/loop44 82M 82M 0 100% /snap/gtk-common-themes/1534
/dev/sdd2 9,1T 8,4T 185G 98% /home/cynicaldrink/STORJ_10TB
tmpfs 1,6G 40K 1,6G 1% /run/user/1000
/dev/sde1 932G 607G 325G 66% /home/cynicaldrink/PRENOSI
overlay 109G 86G 18G 83% /var/lib/docker/overlay2/42aeee47d0f71b8588c614d43e5726da294113412e0654bd43cd93b2ecdfed5c/merged
This is your drive that needs testing. Try fsck /dev/sdd2
actually is /dev/sdb1 (mounted on STORJ2_10TB–> now unmounted.). /home/cynicaldrink/STORJ_10TB is a node thats online & working. I have two 10TB nodes.
edit:
fsck still running
2 Likes
quick report: ubuntu disc utility (previous post) did nothing. updated fsck to latest version and ran it manually with -f -y flags. It completed the test, fixed a bunch of errors. I then proceeded to fix databases - ran pragma integrity checks. result:
ok, previously i got a bunch of I/O errors, now its fine. I mounted node container, start node & dash threw this error again:
so i guess nothing is solved. I am close to giving up on this node. 
This error may indicate that storagenode is busy.
However, did you change the server.private-address: option in the config.yaml file or with an --server.private-address argument of your docker run command?
And do you see any other errors in your logs after the node start?
Please post the last 20 lines from your logs between two new lines with three backticks, like this:
```
log lines here
```
no backticks in my log …these are few last lines from my log, pattern repeats
2023-03-13T16:39:08.156Z Error:: command not found
cynicaldrink@SERVER-PC:~$ storj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:331
-bash: syntax error near unexpected token*DB' cynicaldrink@SERVER-PC:~$ storj.io/storj/storagenode/storagenodedb.(*DB).openExistingDatabase:308 -bash: syntax error near unexpected token*DB’
cynicaldrink@SERVER-PC:~$ storj.io/storj/storagenode/storagenodedb.(*DB).openDatabases:283
-bash: syntax error near unexpected token*DB' cynicaldrink@SERVER-PC:~$ storj.io/storj/storagenode/storagenodedb.OpenExisting:250 -bash: storj.io/storj/storagenode/storagenodedb.OpenExisting:250: No such file or directory cynicaldrink@SERVER-PC:~$ main.cmdRun:193 main.cmdRun:193: command not found cynicaldrink@SERVER-PC:~$ storj.io/private/process.cleanup.func1.4:377 -bash: storj.io/private/process.cleanup.func1.4:377: No such file or directory cynicaldrink@SERVER-PC:~$ storj.io/private/process.cleanup.func1:395 -bash: storj.io/private/process.cleanup.func1:395: No such file or directory cynicaldrink@SERVER-PC:~$ github.com/spf13/cobra.(*Command).execute:852 -bash: syntax error near unexpected token*Command’
cynicaldrink@SERVER-PC:~$ github.com/spf13/cobra.(*Command).ExecuteC:960
-bash: syntax error near unexpected token*Command' cynicaldrink@SERVER-PC:~$ github.com/spf13/cobra.(*Command).Execute:897 -bash: syntax error near unexpected token*Command’
cynicaldrink@SERVER-PC:~$ storj.io/private/process.ExecWithCustomConfigAndLogger:92
-bash: storj.io/private/process.ExecWithCustomConfigAndLogger:92: No such file or directory
cynicaldrink@SERVER-PC:~$ main.main:478
main.main:478: command not found
cynicaldrink@SERVER-PC:~$ runtime.main:250
runtime.main:250: command not found
cynicaldrink@SERVER-PC:~$ 2023-03-13 16:39:04,726 INFO exited: storagenode (exit status 1; not expected)
-bash: syntax error near unexpected token(' cynicaldrink@SERVER-PC:~$ 2023-03-13 16:39:05,732 INFO spawned: 'storagenode' with pid 44 2023-03-13: command not found cynicaldrink@SERVER-PC:~$ 2023-03-13 16:39:05,732 WARN received SIGQUIT indicating exit request 2023-03-13: command not found cynicaldrink@SERVER-PC:~$ 2023-03-13 16:39:05,733 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die 2023-03-13: command not found cynicaldrink@SERVER-PC:~$ 2023-03-13T16:39:05.733Z INFO Got a signal from the OS: "terminated" {"Process": "storagenode-updater"} 2023-03-13T16:39:05.733Z: command not found cynicaldrink@SERVER-PC:~$ 2023-03-13 16:39:05,735 INFO stopped: storagenode-updater (exit status 0) -bash: syntax error near unexpected token(’
cynicaldrink@SERVER-PC:~$ 2023-03-13 16:39:05,736 INFO stopped: storagenode (terminated by SIGTERM)
-bash: syntax error near unexpected token(' cynicaldrink@SERVER-PC:~$ 2023-03-13 16:39:05,736 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM) -bash: syntax error near unexpected token(’
cynicaldrink@SERVER-PC:~$ 2023-03-13 16:39:06,623 INFO Set uid to user 0 succeeded
2023-03-13: command not found
cynicaldrink@SERVER-PC:~$ 2023-03-13 16:39:06,631 INFO RPC interface ‘supervisor’ initialized
2023-03-13: command not found
cynicaldrink@SERVER-PC:~$ 2023-03-13 16:39:06,631 INFO supervisord started with pid 1
2023-03-13: command not found
cynicaldrink@SERVER-PC:~$ 2023-03-13 16:39:07,635 INFO spawned: ‘processes-exit-eventlistener’ with pid 11
2023-03-13: command not found
cynicaldrink@SERVER-PC:~$ 2023-03-13 16:39:07,641 INFO spawned: ‘storagenode’ with pid 12
2023-03-13: command not found
cynicaldrink@SERVER-PC:~$ 2023-03-13 16:39:07,650 INFO spawned: ‘storagenode-updater’ with pid 13
2023-03-13: command not found
cynicaldrink@SERVER-PC:~$ 2023-03-13T16:39:07.675Z INFO Configuration loaded {“Process”: “storagenode-updater”, “Location”: “/app/config/config.yaml”}
2023-03-13T16:39:07.675Z: command not found
cynicaldrink@SERVER-PC:~$ 2023-03-13T16:39:07.675Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “server.private-address”}
2023-03-13T16:39:07.675Z: command not found
cynicaldrink@SERVER-PC:~$ 2023-03-13T16:39:07.675Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “contact.external-address”}
2023-03-13T16:39:07.675Z: command not found
cynicaldrink@SERVER-PC:~$ 2023-03-13T16:39:07.676Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “operator.wallet”}
2023-03-13T16:39:07.676Z: command not found
cynicaldrink@SERVER-PC:~$ 2023-03-13T16:39:07.676Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “operator.email”}
2023-03-13T16:39:07.676Z: command not found
cynicaldrink@SERVER-PC:~$ 2023-03-13T16:39:07.676Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “storage.allocated-bandwidth”}
2023-03-13T16:39:07.676Z: command not found
cynicaldrink@SERVER-PC:~$ 2023-03-13T16:39:07.677Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “operator.wallet-features.0”}
2023-03-13T16:39:07.677Z: command not found
cynicaldrink@SERVER-PC:~$ 2023-03-13T16:39:07.677Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “server.address”}
2023-03-13T16:39:07.677Z: command not found
cynicaldrink@SERVER-PC:~$ 2023-03-13T16:39:07.677Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “storage.allocated-disk-space”}
2023-03-13T16:39:07.677Z: command not found
cynicaldrink@SERVER-PC:~$ 2023-03-13T16:39:07.678Z INFO Anonymized tracing enabled {“Process”: “storagenode-updater”}
2023-03-13T16:39:07.678Z: command not found
cynicaldrink@SERVER-PC:~$ 2023-03-13T16:39:07.679Z INFO Running on version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.74.1”}
2023-03-13T16:39:07.679Z: command not found
cynicaldrink@SERVER-PC:~$ 2023-03-13T16:39:07.680Z INFO Downloading versions. {“Process”: “storagenode-updater”, “Server Address”: “https://version.storj.io”}
2023-03-13T16:39:07.680Z: command not found
cynicaldrink@SERVER-PC:~$ 2023-03-13T16:39:07.701Z INFO Configuration loaded {“Process”: “storagenode”, “Location”: “/app/config/config.yaml”}
2023-03-13T16:39:07.701Z: command not found
cynicaldrink@SERVER-PC:~$ 2023-03-13T16:39:07.702Z INFO Anonymized tracing enabled {“Process”: “storagenode”}
2023-03-13T16:39:07.702Z: command not found
cynicaldrink@SERVER-PC:~$ 2023-03-13T16:39:07.702Z INFO Operator email {“Process”: “storagenode”, “Address”: “cynicaldump@gmail.com”}
2023-03-13T16:39:07.702Z: command not found
cynicaldrink@SERVER-PC:~$ 2023-03-13T16:39:07.703Z INFO Operator wallet {“Process”: “storagenode”, “Address”: “0x7eA0B136fc8b693a00104e808Bd9514c3552e5aA”}
2023-03-13T16:39:07.703Z: command not found
cynicaldrink@SERVER-PC:~$ 2023-03-13T16:39:08.156Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode”, “Version”: “v1.74.1”}
2023-03-13T16:39:08.156Z: command not found
cynicaldrink@SERVER-PC:~$ 2023-03-13T16:39:08.156Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode”}
2023-03-13T16:39:08.156Z: command not found
cynicaldrink@SERVER-PC:~$ 2023-03-13T16:39:08.167Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.74.1”}
2023-03-13T16:39:08.167Z: command not found
cynicaldrink@SERVER-PC:~$ 2023-03-13T16:39:08.167Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”}
2023-03-13T16:39:08.167Z: command not found
cynicaldrink@SERVER-PC:~$ 2023-03-13 16:39:09,168 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
-bash: syntax error near unexpected token(' cynicaldrink@SERVER-PC:~$ 2023-03-13 16:39:09,169 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) -bash: syntax error near unexpected token(’
cynicaldrink@SERVER-PC:~$ 2023-03-13 16:39:09,169 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
-bash: syntax error near unexpected token `(’
i see in some other post i could just delete db files & generate new ones upon restart? Would that be viable option? never mind the loss of stats…
I mean - post your logs here, on the forum, between two new text lines with three backticks in the message window, not to search them in your logs or paste to your terminal.
Just last 20 lines from your logs, copied from the terminal and posted here.
These lines with backticks (also called a code block) are used to distinguish the text between such lines from the normal text, i.e. formatting, it will make your post pretty and will not convert characters to their fancy analogues, as it do, when you use a citation instead of the code block.
2 Likes
oh, sorry. Btw yesterday i made an backup of all db files (except bandwith.db and piece_expiration.db, these two i could only delete). Restarted storagenode & logs are now stuck. From this point on, storagenode stop command doesnt work anymore & i cant kill the process manually. I can only reboot. Also accessing folders becomes sluggish, storage folder is practically inaccessible. When i delete all db files i get better results: uptime robot reports node as alive. dash still inaccessible tho & logs seem to be stuck with these line below.
Here are the logs(all db deleted):
2023-03-14T05:32:31.746Z INFO db database does not exist {“Process”: “storagenode”, “database”: “orders”}
2023-03-14T05:32:31.746Z INFO db database does not exist {“Process”: “storagenode”, “database”: “piece_expiration”}
2023-03-14T05:32:31.746Z INFO db database does not exist {“Process”: “storagenode”, “database”: “pieceinfo”}
2023-03-14T05:32:31.746Z INFO db database does not exist {“Process”: “storagenode”, “database”: “piece_spaced_used”}
2023-03-14T05:32:31.746Z INFO db database does not exist {“Process”: “storagenode”, “database”: “reputation”}
2023-03-14T05:32:31.746Z INFO db database does not exist {“Process”: “storagenode”, “database”: “storage_usage”}
2023-03-14T05:32:31.746Z INFO db database does not exist {“Process”: “storagenode”, “database”: “used_serial”}
2023-03-14T05:32:31.746Z INFO db database does not exist {“Process”: “storagenode”, “database”: “satellites”}
2023-03-14T05:32:31.746Z INFO db database does not exist {“Process”: “storagenode”, “database”: “notifications”}
2023-03-14T05:32:31.746Z INFO db database does not exist {“Process”: “storagenode”, “database”: “heldamount”}
2023-03-14T05:32:31.746Z INFO db database does not exist {“Process”: “storagenode”, “database”: “pricing”}
2023-03-14T05:32:31.746Z INFO db database does not exist {“Process”: “storagenode”, “database”: “secret”}
2023-03-14T05:32:32.209Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode”, “Version”: “v1.74.1”}
2023-03-14T05:32:32.209Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode”}
2023-03-14T05:32:32.217Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.74.1”}
2023-03-14T05:32:32.217Z INFO Version is up to date {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”}
2023-03-14T05:32:42.886Z INFO server kernel support for server-side tcp fast open remains disabled. {“Process”: “storagenode”}
2023-03-14T05:32:42.886Z INFO server enable with: sysctl -w net.ipv4.tcp_fastopen=3 {“Process”: “storagenode”}
2023-03-14T05:32:49.127Z INFO Telemetry enabled {“Process”: “storagenode”, “instance ID”: “1s8zza6YTsjrLeu7kPy27G9JvjWpkk9WthtMdvExX2vfiLH6rD”}
2023-03-14T05:32:49.127Z INFO Event collection enabled {“Process”: “storagenode”, “instance ID”: “1s8zza6YTsjrLeu7kPy27G9JvjWpkk9WthtMdvExX2vfiLH6rD”}
If it hangs, then there is a hardware problem - either your disk is slow or dying.
How is your disk connected to this device? Is it an USB? if so, you need to have an external power supply for your disk and you may be need to try to replace a cable (or at least unplug/plug again) while your node is stopped.
And you need to check the health of your drive in S.M.A.R.T. and perhaps a surface scan.
Yeah, its a hardware problem. Got my first “reallocate sectors” warning about a month ago. And from then on i kept close watch, if the bad sector count was increasing - it didnt, so I figured I could keep my second (newer) node online for a while longer. I guess it didnt work out. No problem. I will spin up new node, when my schedule frees a bit. All HDD are connected directly to the board btw. Thx for all the help.