Unexpected shutdown of a runner

Node Operators troubleshooting
22

but i can unmount it:

lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 5.5T 0 disk
└─sda1 8:1 0 5.5T 0 part /home/user/storj/8
sdb 8:16 0 2.7T 0 disk
└─sdb1 8:17 0 2.7T 0 part
sdc 8:32 0 14.6T 0 disk
└─sdc1 8:33 0 14.6T 0 part /home/user/storj/3
mtdblock0 31:0 0 16M 0 disk
zram0 254:0 0 7.7G 0 disk [SWAP]
zram1 254:1 0 200M 0 disk /var/log
nvme0n1 259:0 0 476.9G 0 disk
├─nvme0n1p1 259:1 0 1G 0 part /boot
└─nvme0n1p2 259:2 0 475.9G 0 part /var/log.hdd

23

Then please try to unmount all mount points before use of sudo fsck.
It should not be blocked.

By the way, is it a VM?

24

Before doing this:

sudo e2fsck -f /dev/sdb1
[sudo] password for node:
e2fsck 1.46.5 (30-Dec-2021)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/sdb1: 9811707/183148544 files (2.2% non-contiguous), 563077932/732566129 blocks

It was unmounted.

It’s orange PI, Ubuntu 22.04.4 LTS

25

As i thought, and just did test, it is not able to run e2fsck while disk is mounted:

sudo e2fsck -f /dev/sdb1
e2fsck 1.46.5 (30-Dec-2021)
/dev/sdb1 is mounted.
e2fsck: Cannot continue, aborting.

26

db’s, are OK:

/storage $ find -maxdepth 1 -iname “*.db” -print0 -exec sqlite3 ‘{}’ ‘PRAGMA integrity_check;’ ‘;’
./secret.dbok
./used_space_per_prefix.dbok
./garbage_collection_filewalker_progress.dbok
./reputation.dbok
./heldamount.dbok
./used_serial.dbok
./piece_spaced_used.dbok
./orders.dbok
./piece_expiration.dbok
./bandwidth.dbok
./storage_usage.dbok
./pricing.dbok
./info.dbok
./notifications.dbok
./satellites.dbok
./pieceinfo.dbok

27

just tried to run once again, but log show this:

2024-05-17 12:13:26,095 INFO spawned: ‘storagenode’ with pid 151
2024-05-17 12:13:26,096 WARN received SIGQUIT indicating exit request
2024-05-17 12:13:26,098 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2024-05-17T12:13:26Z INFO Got a signal from the OS: “terminated” {“Process”: “storagenode-updater”}
2024-05-17 12:13:26,106 INFO stopped: storagenode-updater (exit status 0)
2024-05-17 12:13:26,108 INFO stopped: storagenode (terminated by SIGTERM)
2024-05-17 12:13:26,110 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)
2024-05-17 12:13:27,893 INFO RPC interface ‘supervisor’ initialized
2024-05-17 12:13:27,893 INFO supervisord started with pid 1
2024-05-17 12:13:28,899 INFO spawned: ‘processes-exit-eventlistener’ with pid 12
2024-05-17 12:13:28,905 INFO spawned: ‘storagenode’ with pid 13
2024-05-17 12:13:28,910 INFO spawned: ‘storagenode-updater’ with pid 14
2024/05/17 12:13:28 failed to check for file existence: stat config/config.yaml: input/output error
2024/05/17 12:13:28 failed to check for file existence: stat config/config.yaml: input/output error
2024-05-17 12:13:28,947 INFO exited: storagenode-updater (exit status 1; not expected)
2024-05-17 12:13:28,948 INFO exited: storagenode (exit status 1; not expected)
2024-05-17 12:13:29,950 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-05-17 12:13:29,955 INFO spawned: ‘storagenode’ with pid 25
2024-05-17 12:13:29,962 INFO spawned: ‘storagenode-updater’ with pid 26
2024/05/17 12:13:29 failed to check for file existence: stat config/config.yaml: input/output error
2024/05/17 12:13:29 failed to check for file existence: stat config/config.yaml: input/output error
2024-05-17 12:13:29,993 INFO exited: storagenode-updater (exit status 1; not expected)
2024-05-17 12:13:29,997 INFO exited: storagenode (exit status 1; not expected)
2024-05-17 12:13:32,007 INFO spawned: ‘storagenode’ with pid 37
2024-05-17 12:13:32,013 INFO spawned: ‘storagenode-updater’ with pid 38
2024/05/17 12:13:32 failed to check for file existence: stat config/config.yaml: input/output error
2024-05-17 12:13:32,045 INFO exited: storagenode-updater (exit status 1; not expected)
2024/05/17 12:13:32 failed to check for file existence: stat config/config.yaml: input/output error
2024-05-17 12:13:32,048 INFO exited: storagenode (exit status 1; not expected)
2024-05-17 12:13:35,059 INFO spawned: ‘storagenode’ with pid 49
2024-05-17 12:13:35,065 INFO spawned: ‘storagenode-updater’ with pid 50
2024/05/17 12:13:35 failed to check for file existence: stat config/config.yaml: input/output error
2024-05-17 12:13:35,093 INFO exited: storagenode-updater (exit status 1; not expected)
2024/05/17 12:13:35 failed to check for file existence: stat config/config.yaml: input/output error
2024-05-17 12:13:35,098 INFO gave up: storagenode-updater entered FATAL state, too many start retries too quickly
2024-05-17 12:13:35,104 INFO exited: storagenode (exit status 1; not expected)
2024-05-17 12:13:35,105 INFO gave up: storagenode entered FATAL state, too many start retries too quickly
2024-05-17 12:13:35,105 WARN received SIGQUIT indicating exit request
2024-05-17 12:13:35,105 INFO waiting for processes-exit-eventlistener to die
2024-05-17 12:13:37,111 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)

28

Then it should work. Could please try to run a node?

29

i just tried, and posted log. look on my post:

just tried to run once again, but log show this:

2024-05-17 12:13:26,095 INFO spawned: ‘storagenode’ with pid 151
2024-05-17 12:13:26,096 WARN received SIGQUIT indicating exit request
2024-05-17 12:13:26,098 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2024-05-17T12:13:26Z INFO Got a signal from the OS: “terminated” {“Process”: “storagenode-updater”}
2024-05-17 12:13:26,106 INFO stopped: storagenode-updater (exit status 0)
2024-05-17 12:13:26,108 INFO stopped: storagenode (terminated by SIGTERM)
2024-05-17 12:13:26,110 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)
2024-05-17 12:13:27,893 INFO RPC interface ‘supervisor’ initialized
2024-05-17 12:13:27,893 INFO supervisord started with pid 1
2024-05-17 12:13:28,899 INFO spawned: ‘processes-exit-eventlistener’ with pid 12
2024-05-17 12:13:28,905 INFO spawned: ‘storagenode’ with pid 13
2024-05-17 12:13:28,910 INFO spawned: ‘storagenode-updater’ with pid 14
2024/05/17 12:13:28 failed to check for file existence: stat config/config.yaml: input/output error
2024/05/17 12:13:28 failed to check for file existence: stat config/config.yaml: input/output error
2024-05-17 12:13:28,947 INFO exited: storagenode-updater (exit status 1; not expected)
2024-05-17 12:13:28,948 INFO exited: storagenode (exit status 1; not expected)
2024-05-17 12:13:29,950 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-05-17 12:13:29,955 INFO spawned: ‘storagenode’ with pid 25
2024-05-17 12:13:29,962 INFO spawned: ‘storagenode-updater’ with pid 26
2024/05/17 12:13:29 failed to check for file existence: stat config/config.yaml: input/output error
2024/05/17 12:13:29 failed to check for file existence: stat config/config.yaml: input/output error
2024-05-17 12:13:29,993 INFO exited: storagenode-updater (exit status 1; not expected)
2024-05-17 12:13:29,997 INFO exited: storagenode (exit status 1; not expected)
2024-05-17 12:13:32,007 INFO spawned: ‘storagenode’ with pid 37
2024-05-17 12:13:32,013 INFO spawned: ‘storagenode-updater’ with pid 38
2024/05/17 12:13:32 failed to check for file existence: stat config/config.yaml: input/output error
2024-05-17 12:13:32,045 INFO exited: storagenode-updater (exit status 1; not expected)
2024/05/17 12:13:32 failed to check for file existence: stat config/config.yaml: input/output error
2024-05-17 12:13:32,048 INFO exited: storagenode (exit status 1; not expected)
2024-05-17 12:13:35,059 INFO spawned: ‘storagenode’ with pid 49
2024-05-17 12:13:35,065 INFO spawned: ‘storagenode-updater’ with pid 50
2024/05/17 12:13:35 failed to check for file existence: stat config/config.yaml: input/output error
2024-05-17 12:13:35,093 INFO exited: storagenode-updater (exit status 1; not expected)
2024/05/17 12:13:35 failed to check for file existence: stat config/config.yaml: input/output error
2024-05-17 12:13:35,098 INFO gave up: storagenode-updater entered FATAL state, too many start retries too quickly
2024-05-17 12:13:35,104 INFO exited: storagenode (exit status 1; not expected)
2024-05-17 12:13:35,105 INFO gave up: storagenode entered FATAL state, too many start retries too quickly
2024-05-17 12:13:35,105 WARN received SIGQUIT indicating exit request
2024-05-17 12:13:35,105 INFO waiting for processes-exit-eventlistener to die
2024-05-17 12:13:37,111 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)

30

Please try today and post your current logs not expired two days ago.

31

OMG !!! these log’s are 15min old :expressionless: I just copied 15min, ago. and yes, i can see that DATE is incorrect for some reason :expressionless:

I just restarted computer and now i can see correct time ! And the node started to work!!! ?NICE… !!

How this is possible? Two another nodes was running fine even with the wrong date. Only this one wasn’t

32

The date is important in the crypto algorithms, so you need to have a correct date. Please make sure that you have a synced time everywhere.

1 Like
33

Thank you for your help!

1 Like