Hi 
Today (around 2023-02-23T07:30:00Z), my ISP decided to do workings wherever which caused the box to be offline for a few hours. However, when Internet got back, my nodes did not reconnect.
So I ssh’ed to my RPi hosting them only to find out that none of them were running anymore (only the watchtower docker process was running).
I have been having nodes stopping randomly for the past few months, for no apparent reason, not sure why, but this time they all stopped. I’m wondering if it could be related
I cannot figure out why this is happening. Here is what I checked (example on one node - I checked a few of them, and everything seems OK):
Node’s logs
pac@raspberrypi:~ $ tail -f storj/mounts/disk_2/storj_logs/storj_node_1/node.log -n 20
2023-02-23T07:26:46.014Z INFO piecestore upload started {"Process": "storagenode", "Piece ID": "EE4MSMVXKMDIE25KBQPMBZKI4CLFDXHTAVDHGZ5GXRXDYQCA7E3Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 130577496320, "Remote Address": "72.52.83.202:28538"}
2023-02-23T07:26:46.657Z INFO piecestore uploaded {"Process": "storagenode", "Piece ID": "EE4MSMVXKMDIE25KBQPMBZKI4CLFDXHTAVDHGZ5GXRXDYQCA7E3Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 181504, "Remote Address": "72.52.83.202:28538"}
2023-02-23T07:26:47.113Z INFO piecestore upload started {"Process": "storagenode", "Piece ID": "TS4NY5GRD2HG767LK43FVU7ZIRGCUCZNMSAR66JHZWJB7FPUYZRQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 130577314304, "Remote Address": "207.6.195.94:1591"}
2023-02-23T07:26:47.684Z INFO piecestore uploaded {"Process": "storagenode", "Piece ID": "TS4NY5GRD2HG767LK43FVU7ZIRGCUCZNMSAR66JHZWJB7FPUYZRQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 98304, "Remote Address": "207.6.195.94:1591"}
2023-02-23T07:26:50.343Z INFO piecestore upload started {"Process": "storagenode", "Piece ID": "T3P5SAG27TS26JHML665AYQEYYJBAGMM35TNGHIHL6ASLOTD4D5A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT_REPAIR", "Available Space": 130577215488, "Remote Address": "78.43.172.69:57418"}
2023-02-23T07:26:51.392Z INFO piecestore upload started {"Process": "storagenode", "Piece ID": "Z4GJRELWCHBP6PA3BXULYD64AQCGIQWOS2Y6TSELGT7KNKDOWIKA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 130575837952, "Remote Address": "5.161.129.80:23078"}
2023-02-23T07:26:51.405Z INFO piecestore uploaded {"Process": "storagenode", "Piece ID": "T3P5SAG27TS26JHML665AYQEYYJBAGMM35TNGHIHL6ASLOTD4D5A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT_REPAIR", "Size": 1377024, "Remote Address": "78.43.172.69:57418"}
2023-02-23T07:26:54.071Z INFO piecestore uploaded {"Process": "storagenode", "Piece ID": "LKXX7JDLOGNHEROKLY7B7MTDJJ6YGYWBANZ2NNTSA6IJIRRIACZQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 35072, "Remote Address": "5.161.128.79:43388"}
2023-02-23T07:26:54.106Z INFO piecestore upload started {"Process": "storagenode", "Piece ID": "NLWWJDHV6U2DB2OL5ZRHHSAUCSP4S6MPPCEF44OUYU2K65HHU2XQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 130575802368, "Remote Address": "38.104.84.243:60456"}
2023-02-23T07:26:54.217Z INFO piecestore uploaded {"Process": "storagenode", "Piece ID": "Z4GJRELWCHBP6PA3BXULYD64AQCGIQWOS2Y6TSELGT7KNKDOWIKA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 36864, "Remote Address": "5.161.129.80:23078"}
2023-02-23T07:26:54.791Z INFO piecestore uploaded {"Process": "storagenode", "Piece ID": "NLWWJDHV6U2DB2OL5ZRHHSAUCSP4S6MPPCEF44OUYU2K65HHU2XQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 181504, "Remote Address": "38.104.84.243:60456"}
2023-02-23T07:26:55.554Z INFO piecedeleter delete piece sent to trash {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "CA5H2ME7CMM7ACE5JYXDACRJGLRHQ4VB7IHVGSUCM3F2VX2XT6NQ"}
2023-02-23T07:26:59.613Z INFO piecestore upload started {"Process": "storagenode", "Piece ID": "HAI3WZT6OCUCNE2NDHWAFSBMX5KTOYUAAXMFQRYOWQTTIQXHI6EQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "Available Space": 130575582976, "Remote Address": "38.104.84.243:60534"}
2023-02-23T07:26:59.887Z INFO piecestore uploaded {"Process": "storagenode", "Piece ID": "HAI3WZT6OCUCNE2NDHWAFSBMX5KTOYUAAXMFQRYOWQTTIQXHI6EQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "Size": 14336, "Remote Address": "38.104.84.243:60534"}
2023-02-23T07:27:01.460Z INFO piecestore upload started {"Process": "storagenode", "Piece ID": "5ZPE7OXJXNRMSGSO4UX432U2L6DJO6YJJLV3NRTMP65IESABFAOA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 130575568128, "Remote Address": "34.106.184.147:42640"}
2023-02-23T07:27:01.549Z INFO piecestore uploaded {"Process": "storagenode", "Piece ID": "5ZPE7OXJXNRMSGSO4UX432U2L6DJO6YJJLV3NRTMP65IESABFAOA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 768, "Remote Address": "34.106.184.147:42640"}
2023-02-23T07:27:05.527Z INFO piecestore upload started {"Process": "storagenode", "Piece ID": "IFBFYLFF5A6VEY2NK4RLUKILOJ5HSVWO6WBGAOJ5C4UB7JC3S32Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 130575566848, "Remote Address": "38.88.241.43:37674"}
2023-02-23T07:27:05.658Z INFO piecestore uploaded {"Process": "storagenode", "Piece ID": "IFBFYLFF5A6VEY2NK4RLUKILOJ5HSVWO6WBGAOJ5C4UB7JC3S32Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 8960, "Remote Address": "38.88.241.43:37674"}
2023-02-23T07:27:05.944Z INFO piecestore upload started {"Process": "storagenode", "Piece ID": "2ZBMXDPBG7PUT2TVQDUO4LA65Q7BTVPUKFDWKSTAUPZMOUFBETWA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 130575557376, "Remote Address": "5.161.180.29:43868"}
2023-02-23T07:28:06.224Z INFO piecestore upload canceled {"Process": "storagenode", "Piece ID": "2ZBMXDPBG7PUT2TVQDUO4LA65Q7BTVPUKFDWKSTAUPZMOUFBETWA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 0, "Remote Address": "5.161.180.29:43868"}
Container’s logs
pac@raspberrypi:~ $ sudo docker logs -f storj_node_1 -n 20
2023-02-23T06:34:02.005Z INFO Downloading versions. {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2023-02-23T06:34:02.496Z INFO Current binary version {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.72.5"}
2023-02-23T06:34:02.496Z INFO Version is up to date {"Process": "storagenode-updater", "Service": "storagenode"}
2023-02-23T06:34:02.601Z INFO Current binary version {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.72.5"}
2023-02-23T06:34:02.601Z INFO Version is up to date {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2023-02-23T06:49:02.016Z INFO Downloading versions. {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2023-02-23T06:49:02.562Z INFO Current binary version {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.72.5"}
2023-02-23T06:49:02.562Z INFO Version is up to date {"Process": "storagenode-updater", "Service": "storagenode"}
2023-02-23T06:49:02.671Z INFO Current binary version {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.72.5"}
2023-02-23T06:49:02.672Z INFO Version is up to date {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2023-02-23T07:04:02.011Z INFO Downloading versions. {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2023-02-23T07:04:02.618Z INFO Current binary version {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.72.5"}
2023-02-23T07:04:02.618Z INFO Version is up to date {"Process": "storagenode-updater", "Service": "storagenode"}
2023-02-23T07:04:02.718Z INFO Current binary version {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.72.5"}
2023-02-23T07:04:02.718Z INFO Version is up to date {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2023-02-23T07:19:02.019Z INFO Downloading versions. {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2023-02-23T07:19:02.596Z INFO Current binary version {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.72.5"}
2023-02-23T07:19:02.596Z INFO Version is up to date {"Process": "storagenode-updater", "Service": "storagenode"}
2023-02-23T07:19:02.700Z INFO Current binary version {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.72.5"}
2023-02-23T07:19:02.700Z INFO Version is up to date {"Process": "storagenode-updater", "Service": "storagenode-updater"}
Sys logs
pac@raspberrypi:~/storj $ grep -i "kill" /var/log/syslog
Feb 23 08:20:47 raspberrypi systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch.
Feb 23 08:20:47 raspberrypi systemd[1]: Starting Load/Save RF Kill Switch Status...
Feb 23 08:20:47 raspberrypi systemd[1]: Started Load/Save RF Kill Switch Status.
Feb 23 08:20:48 raspberrypi dhcpcd[436]: dhcpcd_prestartinterface: wlan0: Operation not possible due to RF-kill
Feb 23 08:20:50 raspberrypi systemd[1]: systemd-rfkill.service: Succeeded.
Feb 23 08:20:54 raspberrypi systemd[1]: Starting Load/Save RF Kill Switch Status...
Feb 23 08:20:54 raspberrypi systemd[1]: Started Load/Save RF Kill Switch Status.
Feb 23 08:20:58 raspberrypi systemd[1]: dhcpcd.service: Main process exited, code=killed, status=11/SEGV
Feb 23 08:20:59 raspberrypi dhcpcd[1900]: dhcpcd_prestartinterface: wlan0: Operation not possible due to RF-kill
Feb 23 08:20:59 raspberrypi systemd[1]: systemd-rfkill.service: Succeeded.
The above is weird but I think it’s because I never configured Wifi on this Pi. I don’t think it’s related to my nodes stopping, although it did happen a few minutes before the Internet connection stopped (as this timestamp is probably my local time which is UTC+1 right now).
pac@raspberrypi:~/storj $ grep -i 'kill' /var/log/messages*
pac@raspberrypi:~/storj $
So, hum… 
Is there anything in the node software that would make it exit without saying anything in any logs?
Is there anything else I’m missing that would be worth checking?
Thx
As a side note, I restarted all of them manually, and they seem happy and back in business.
