Node generates an error after update

[kofiko]

Hi.
I updated the node today and after the update, it did not recognize the path to the Storj DB. I rebooted the machine and then it mapped the right folder, but it shows the following error:
Error response from daemon: Container a485af9284a0f8a512f22439a0ac2bc70b1476b0b16636cb978dc217f5c013d7 is restarting, wait until the container is running

It shows this error for a few hours now…
Any suggestions on how to fix this?
Thanks
Ilan

[Cross91]

Hi kofiko,

to troubleshoot a problem, we’d need (always) some more information.
For example:

• if docker, your docker run command
• static mount
  …

something like in that template:

Template for troubleshooting storage node issues

Hi everyone, nearly everytime someone opens a post asking for help (mostly because their node is offline), they tend to not include details like software version etc, which leads to lots of questions makes remote troubleshooting more tedious for our helpful forum members. So to save everyone some time, let’s create a “Node Troubleshooting Template” to pin in the SNO subforum! So my first draft would be: Node software type: Win GUI | Docker Linux | Docker Windows (deprecated) N…

Please provide those basic information.

And of course I presuppose, that you already used the search function of this forum:
https://forum.storj.io/search?q=container%20restarting

Thank you.

[kofiko]

Hi, Cross.
Thank you for taking the time to look at the post. Sorry that I did not have the correct information before.
This is the run command data:
Using new storj version via docker
sudo docker run -d --restart unless-stopped -p 34567:28967
-e WALLET=“0x99969Df2dA9BF780Cfd62D7cC22f77E5BdB332df”
-e EMAIL="ilan.odroidstorj@gmail.com"
-e ADDRESS=“danidin.hopto.org:34567”
-e BANDWIDTH=“61.5TB”
-e STORAGE=“7TB”
–mount type=bind,source="/media/Seagate_8T/storj_identity/identity/storagenode",destination
=/app/identity
–mount type=bind,source="/media/Seagate_8T/Node_1",destination=/app/config
–name storagenode storjlabs/storagenode:beta

Node software type: Docker Linux
Node software version: v1.0.1
OS details: Raspberry PI Buster
Virtualization: no virtualization
Location (optional): USA
ISP type: Fiber
external IP address (please DON’T paste your public IP address here!): dynamic with DDNS
DDNS provider: No-IP
DDNS set up via router or via DUC software?: Router | DUC
Type of hardware: RPi
How is your HDD connected: USB
File system : Not sure
Satellite specific problem?: all Sats

Thanks again for taking a look.
Ilan

[Cross91]

Hi Ilan,

looks like you havn’t mounted fia FSTAB.
https://documentation.storj.io/resources/faq/linux-static-mount
Please mount fia FSTAB, update your docker run command and try again.

[kofiko]

Hi.
I believe I have FSTAB configured. Here is the output to show that:
more /etc/fstab
proc /proc proc defaults 0 0
PARTUUID=ea7d04d6-01 /boot vfat defaults 0 2
PARTUUID=ea7d04d6-02 / ext4 defaults,noatime 0 1

a swapfile is not a swap partition, no line here

use dphys-swapfile swap[on|off] for that

UUID=1CE24871E2485166 /media/Seagate_8T ntfs rw,auto 0 0

Could it be something else?
Thanks
Ilan

[BrightSilence]

I would recommend avoiding using /media in fstab as that is used for automatic mounting by the OS. It’s better to mount it to /mnt/Seagate_8T. Don’t forget to adjust the paths in the run command accordingly if you change this.

But, we’re shooting in the dark right now. This may not be causing your issue. Please post the last 30 lines of your log.

docker logs --tail 30 storagenode

[kofiko]

@BrightSilence

docker logs --tail 30 storagenode

$ sudo docker logs --tail 30 storagenode
storj.io/storj/storagenode/storagenodedb.(*DB).Preflight:293
main.cmdRun:198
storj.io/storj/pkg/process.cleanup.func1.2:312
storj.io/storj/pkg/process.cleanup.func1:330
github.com/spf13/cobra.(*Command).execute:826
github.com/spf13/cobra.(*Command).ExecuteC:914
github.com/spf13/cobra.(*Command).Execute:864
storj.io/storj/pkg/process.ExecWithCustomConfig:84
storj.io/storj/pkg/process.ExecCustomDebug:66
main.main:328
runtime.main:203
2020-04-01T21:58:35.431Z INFO Configuration loaded from: /app/config/config.yaml
2020-04-01T21:58:35.434Z INFO Operator email: ilan.odroidstorj@gmail.com
2020-04-01T21:58:35.434Z INFO operator wallet: 0x99969Df2dA9BF780Cfd62D7cC22f77E5BdB332df
2020-04-01T21:58:35.676Z INFO version running on version v1.0.1
2020-04-01T21:58:35.865Z INFO db.migration Database Version {“version”: 33}
Error: Error during preflight check for storagenode databases: storage node preflight database error: disk I/O error: input/output error
storj.io/storj/private/dbutil/sqliteutil.QuerySchema.func1:35
storj.io/storj/private/dbutil/sqliteutil.QuerySchema:53
storj.io/storj/storagenode/storagenodedb.(*DB).Preflight:293
main.cmdRun:198
storj.io/storj/pkg/process.cleanup.func1.2:312
storj.io/storj/pkg/process.cleanup.func1:330
github.com/spf13/cobra.(*Command).execute:826
github.com/spf13/cobra.(*Command).ExecuteC:914
github.com/spf13/cobra.(*Command).Execute:864
storj.io/storj/pkg/process.ExecWithCustomConfig:84
storj.io/storj/pkg/process.ExecCustomDebug:66
main.main:328
runtime.main:203

Again, I really appreciate your help.
Ilan

[Alexey]

@kofiko

disk I/O error: input/output error

Please, give me result of the command:

df -h

[kofiko]

@Alexey

df -h

df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 15G 3.5G 11G 26% /
devtmpfs 1.8G 0 1.8G 0% /dev
tmpfs 2.0G 84K 2.0G 1% /dev/shm
tmpfs 2.0G 185M 1.8G 10% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
/dev/mmcblk0p1 253M 54M 199M 22% /boot
/dev/sda2 7.3T 376G 7.0T 6% /media/Seagate_8T
tmpfs 391M 0 391M 0% /run/user/1000

Thanks Alexey.
Ilan

[Alexey]

sudo lsblk

Then

ls -l /media/Seagate_8T
ls -l /media/Seagate_8T/storage

[kofiko]

@Alexey

ls -l /media/Seagate_8T ls -l /media/Seagate_8T/storage

sudo lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 7.3T 0 disk
|-sda1 8:1 0 128M 0 part
-sda2 8:2 0 7.3T 0 part /media/Seagate_8T mmcblk0 179:0 0 14.9G 0 disk |-mmcblk0p1 179:1 0 256M 0 part /boot -mmcblk0p2 179:2 0 14.6G 0 part /
pi@pi:~$ ls -l /media/Seagate_8T
total 19537
drwxrwxrwx 1 root root 184320 Mar 27 12:13 Node_1
drwxrwxrwx 1 root root 0 Oct 25 13:03 otherFiles
-rwxrwxrwx 1 root root 474 Feb 17 18:59 README
drwxrwxrwx 1 root root 0 Oct 25 13:02 scripts
drwxrwxrwx 1 root root 0 Nov 25 2017 Seagate
-rwxrwxrwx 2 root root 562718 Jan 8 2015 SeagateExpansion.ico
drwxrwxrwx 1 root root 0 Nov 25 2017 Start_Here_Mac.app
-rwxrwxrwx 2 root root 18224504 May 4 2017 Start_Here_Win.exe
drwxrwxrwx 1 root root 0 Oct 25 12:57 storj_identity
-rwxrwxrwx 1 root root 1027404 Jun 28 2017 Warranty.pdf
pi@pi:~$ ls -l /media/Seagate_8T/storage

Thanks
Ilan

[Alexey]

Please, stop the storagenode and check your disk for errors, fix them all, then try to start the storagenode.

[kofiko]

@kofiko

sudo lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 7.3T 0 disk
|-sda1 8:1 0 128M 0 part
-sda2 8:2 0 7.3T 0 part /media/Seagate_8T mmcblk0 179:0 0 14.9G 0 disk |-mmcblk0p1 179:1 0 256M 0 part /boot -mmcblk0p2 179:2 0 14.6G 0 part /
pi@pi:~ ls -l /media/Seagate_8T total 19537 drwxrwxrwx 1 root root 184320 Mar 27 12:13 Node_1 drwxrwxrwx 1 root root 0 Oct 25 13:03 otherFiles -rwxrwxrwx 1 root root 474 Feb 17 18:59 README drwxrwxrwx 1 root root 0 Oct 25 13:02 scripts drwxrwxrwx 1 root root 0 Nov 25 2017 Seagate -rwxrwxrwx 2 root root 562718 Jan 8 2015 SeagateExpansion.ico drwxrwxrwx 1 root root 0 Nov 25 2017 Start_Here_Mac.app -rwxrwxrwx 2 root root 18224504 May 4 2017 Start_Here_Win.exe drwxrwxrwx 1 root root 0 Oct 25 12:57 storj_identity -rwxrwxrwx 1 root root 1027404 Jun 28 2017 Warranty.pdf pi@pi:~ ls -l /media/Seagate_8T/storage

Hi.
I did that. This is the response I get for the last 24 hours when I try to look at the docker container:
./showDockerDash.sh

Error response from daemon: Container 51ff4f8bea816e0a50bf8c0c63e067e4a7358bf233670155187f5ac7cdc23e58 is restarting, wait until the container is running

This is what I used to fix the hard drive.
sudo fsck /dev/sda2
fsck from util-linux 2.33.1
pi@pi:~$ sudo blkid /dev/sda2
/dev/sda2: LABEL=“Seagate Expansion Drive” UUID=“1CE24871E2485166” TYPE=“ntfs” PTTYPE=“atari” PARTLABEL=“Basic data partition” PARTUUID=“93e5f72f-db9e-4da9-a99c-bb64fcc78342”
pi@pi:~$ sudo mount /media/Seagate_8T/
pi@pi:~$ ls /media/Seagate_8T/
Node_1 README Seagate Start_Here_Mac.app storj_identity
otherFiles scripts SeagateExpansion.ico Start_Here_Win.exe Warranty.pdf

Is there a way to fix this, or should I just dump the node and try to install a new one on this machine?
Thanks
Ilan

[Alexey]

You should not use the NTFS on Linux in the first place.
Please, shutdown your pi, disconnect drive and connect it to Windows PC, check for errors and fix them, then safely remove the drive and connect it back to the pi. After that it should work some time until the next unexpected shutdown or reboot.

Better to move your data on ext4 filesystem.
To do that, you should copy your data to some temporary place: https://documentation.storj.io/resources/faq/migrate-my-node, stop and remove the container, format this drive to ext4 and return back your data and identity, then update /etc/fstab with new parameters (UUID and filesystem will be different), make sure that it’s mounted after reboot, then run the storagenode.

[kofiko]

I was able to restore the node to the network.
Thanks for your help.

Ilan