Latest Update (v.108.3) killed node

Node Operators troubleshooting
, ,
1

After the latest update, my Node just died. It would not come back online after restarting the docker container. I decided to stop and remove the old docker container and image, and performed a fresh install in accordance with: Storage Node - Storj Docs
However, doing this has the node stuck at:

pi@raspberrypi:~ $ sudo docker logs -f --tail 100 20e36483de73
 18800K .......... .......... .......... .......... .......... 94%  260K 2s
 18850K .......... .......... .......... .......... .......... 94%  266K 2s
 18900K .......... .......... .......... .......... .......... 94%  260K 2s
 18950K .......... .......... .......... .......... .......... 95%  228K 2s
 19000K .......... .......... .......... .......... .......... 95%  278K 2s
 19050K .......... .......... .......... .......... .......... 95%  255K 2s
 19100K .......... .......... .......... .......... .......... 95%  347K 2s
 19150K .......... .......... .......... .......... .......... 96%  271K 2s
 19200K .......... .......... .......... .......... .......... 96%  520K 1s
 19250K .......... .......... .......... .......... .......... 96%  569K 1s
 19300K .......... .......... .......... .......... .......... 96%  544K 1s
 19350K .......... .......... .......... .......... .......... 97%  232K 1s
 19400K .......... .......... .......... .......... .......... 97%  520K 1s
 19450K .......... .......... .......... .......... .......... 97%  569K 1s
 19500K .......... .......... .......... .......... .......... 97%  568K 1s
 19550K .......... .......... .......... .......... .......... 98%  541K 1s
 19600K .......... .......... .......... .......... .......... 98%  595K 1s
 19650K .......... .......... .......... .......... .......... 98%  567K 1s
 19700K .......... .......... .......... .......... .......... 98%  569K 0s
 19750K .......... .......... .......... .......... .......... 99%  464K 0s
 19800K .......... .......... .......... .......... .......... 99%  544K 0s
 19850K .......... .......... .......... .......... .......... 99%  329K 0s
 19900K .......... .......... .......... .......... .......... 99%  446K 0s
 19950K .......... .......... .......... .......... ........  100%  437K=38s

2024-07-15 16:26:59 (525 KB/s) - '/tmp/storagenode.zip' saved [20478889/20478889]

2024-07-15 16:27:42,881 INFO RPC interface 'supervisor' initialized
2024-07-15 16:27:42,882 INFO supervisord started with pid 1
2024-07-15 16:27:43,886 INFO spawned: 'processes-exit-eventlistener' with pid 57
2024-07-15 16:27:43,891 INFO spawned: 'storagenode' with pid 58
2024-07-15 16:27:43,896 INFO spawned: 'storagenode-updater' with pid 59
2024-07-15T16:27:43Z    INFO    Configuration loaded    {"Process": "storagenode-updater", "Location": "/app/config/config.yaml"}
2024-07-15T16:27:43Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "healthcheck.details"}
2024-07-15T16:27:43Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "console.address"}
2024-07-15T16:27:43Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.address"}
2024-07-15T16:27:43Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.private-address"}
2024-07-15T16:27:43Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "contact.external-address"}
2024-07-15T16:27:43Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.wallet-features"}
2024-07-15T16:27:43Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-bandwidth"}
2024-07-15T16:27:43Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.email"}
2024-07-15T16:27:43Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-disk-space"}
2024-07-15T16:27:43Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.wallet"}
2024-07-15T16:27:43Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "healthcheck.enabled"}
2024-07-15T16:27:43Z    INFO    Anonymized tracing enabled      {"Process": "storagenode-updater"}
2024-07-15T16:27:43Z    INFO    Running on version      {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.108.3"}
2024-07-15T16:27:43Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-07-15T16:27:43Z    INFO    Configuration loaded    {"Process": "storagenode", "Location": "/app/config/config.yaml"}
2024-07-15T16:27:43Z    INFO    Anonymized tracing enabled      {"Process": "storagenode"}
2024-07-15T16:27:43Z    INFO    Operator email  {"Process": "storagenode", "Address": "myEmailAddress@mail.com"}
2024-07-15T16:27:43Z    INFO    Operator wallet {"Process": "storagenode", "Address": "0x5093D60D6FaBBD11A2583e902e79E00CE0Fd378e"}
2024-07-15T16:27:44Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.108.3"}
2024-07-15T16:27:44Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-07-15T16:27:44Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.108.3"}
2024-07-15T16:27:44Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-07-15 16:27:45,344 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-07-15 16:27:45,345 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-07-15 16:27:45,345 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-07-15 16:32:13,809 WARN received SIGTERM indicating exit request
2024-07-15 16:32:13,810 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2024-07-15T16:32:13Z    INFO    Got a signal from the OS: "terminated"  {"Process": "storagenode-updater"}
2024-07-15 16:32:13,815 INFO stopped: storagenode-updater (exit status 0)
2024-07-15T16:32:13Z    INFO    Got a signal from the OS: "terminated"  {"Process": "storagenode"}
2024-07-15 16:32:16,819 INFO waiting for storagenode, processes-exit-eventlistener to die
2024-07-15 16:32:19,821 INFO waiting for storagenode, processes-exit-eventlistener to die
2024-07-15 16:32:22,822 INFO waiting for storagenode, processes-exit-eventlistener to die
2024-07-15 16:32:23,823 WARN killing 'storagenode' (58) with SIGKILL
2024-07-15 16:32:23,829 INFO stopped: storagenode (terminated by SIGKILL)
2024-07-15 16:32:23,831 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)
2024-07-15 16:32:27,848 INFO RPC interface 'supervisor' initialized
2024-07-15 16:32:27,849 INFO supervisord started with pid 1
2024-07-15 16:32:28,853 INFO spawned: 'processes-exit-eventlistener' with pid 12
2024-07-15 16:32:28,857 INFO spawned: 'storagenode' with pid 13
2024-07-15 16:32:28,862 INFO spawned: 'storagenode-updater' with pid 14
2024-07-15T16:32:28Z    INFO    Configuration loaded    {"Process": "storagenode-updater", "Location": "/app/config/config.yaml"}
2024-07-15T16:32:28Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.email"}
2024-07-15T16:32:28Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-disk-space"}
2024-07-15T16:32:28Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "console.address"}
2024-07-15T16:32:28Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.wallet-features"}
2024-07-15T16:32:28Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.wallet"}
2024-07-15T16:32:28Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-bandwidth"}
2024-07-15T16:32:28Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "healthcheck.details"}
2024-07-15T16:32:28Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "healthcheck.enabled"}
2024-07-15T16:32:28Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.private-address"}
2024-07-15T16:32:28Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.address"}
2024-07-15T16:32:28Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "contact.external-address"}
2024-07-15T16:32:28Z    INFO    Anonymized tracing enabled      {"Process": "storagenode-updater"}
2024-07-15T16:32:28Z    INFO    Running on version      {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.108.3"}
2024-07-15T16:32:28Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-07-15T16:32:28Z    INFO    Configuration loaded    {"Process": "storagenode", "Location": "/app/config/config.yaml"}
2024-07-15T16:32:28Z    INFO    Anonymized tracing enabled      {"Process": "storagenode"}
2024-07-15T16:32:28Z    INFO    Operator email  {"Process": "storagenode", "Address": "myemail@mail.com"}
2024-07-15T16:32:28Z    INFO    Operator wallet {"Process": "storagenode", "Address": "0x5093D60D6FaBBD11A2583e902e79E00CE0Fd378e"}
2024-07-15T16:32:29Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.108.3"}
2024-07-15T16:32:29Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-07-15T16:32:29Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.108.3"}
2024-07-15T16:32:29Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-07-15 16:32:30,196 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-07-15 16:32:30,196 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-07-15 16:32:30,197 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

I wouldn’t be upset if operators weren’t penalized for being offline for reasons that were not our fault. What do I do. I haven’t changed anything.

System information: Raspberry Pi 4b+, running headless Debian OS. No changes have been made to my system; the update seems to have broken the Node.

2

It appears your config is missing, incorrect, or not accessible. Did you change or remove it?

3

I have not altered permissions, potentially rendering it inaccessible; I have not moved any folder or file around besides deleting the old docker container and image, and replacing it with a fresh copy via

docker pull storjlabs/storagenode:latest
4

Setting log.level to debug might give some more information.

5

Are you running Watchtower? It may be that it is killing it to perform an update.

6

changing the .yaml log.level from info → debug gives no further information when restarting the docker container. I can paste the same thing again if someone would like to verify, but it is still getting “stuck” at storagenode-updater

7

Short answer: I am not running Watchtower.

Long answer: I tried running Watchtower, but was not able to get it running. The second command does not work on my system when following: Software Updates - Storj Docs

pi@raspberrypi:~ $ sudo docker pull storjlabs/watchtower
Using default tag: latest
latest: Pulling from storjlabs/watchtower
3a807ee5a0f8: Pull complete
18bdfcdb74a4: Pull complete
047a83254312: Pull complete
Digest: sha256:31c0c558ec8adad5cec6cb86320c58018289ca22dafa019c31cd5d6f65f03655
Status: Downloaded newer image for storjlabs/watchtower:latest
docker.io/storjlabs/watchtower:latest
pi@raspberrypi:~ $ sudo docker run -d --restart=always --name watchtower -v /var/run/docker.sock:/var/run/docker.sock storjlabs/watchtower storagenode watchtower --stop-timeout 300s
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
d26ca80c40d074457bdc6e722c5a6cc931e7af30186540cae8b4bbdc5f32d4b3
pi@raspberrypi:~ $
8

Why does it spit out an architecture mismatch?

9

Could you show your docker run command ? Hide your personal details like email, wallet, ddns.

10

No idea. For reference, this exactly what OS version I am using:

pi@raspberrypi:~ $ uname -a
Linux raspberrypi 6.6.20+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.20-1+rpt1 (2024-03-07) aarch64 GNU/Linux```
11

This is it; the only thing I’ve changed from the docs command is -p 127.0.0.1:14002:14002 \ → -p 14002:14002
(Previously, I’ve been unable to see my dashboard when using the one on the docs page):

docker run -d --restart unless-stopped --stop-timeout 300 \
    -p 28967:28967/tcp \
    -p 28967:28967/udp \
    -p 14002:14002 \
    -e WALLET="myWallet" \
    -e EMAIL="myEmail" \
    -e ADDRESS="public_ip_addr:28967" \
    -e STORAGE="10TB" \
    --user $(id -u):$(id -g) \
    --mount type=bind,source="/home/pi/Identity/storagenode",destination=/app/identity \
    --mount type=bind,source="/mnt/NAS/Storj/Storage",destination=/app/config \
    --name storagenode storjlabs/storagenode:latest```
12

How large is your node? My ~ 8Tb node was stuck for ~45min at exactly the same point after update to v.108.3. During all this period it made intensive read and write to it’s DB (about ~20Gb on separate SSD).
I guess it converted or updated something in DB and when it finished - it continued working as usual.
So may be you just need to give it some more time to finish what it’s doing. Especially if node is large and DB is on HDD.

13

10tb; I restarted the container before going to bed, and woke up to the dashboard just loading and loading. After waiting ~ 1 hour since posting, I saw that the dashboard came up and the node was online. A little bit after that, however, the dashboard loads up after a while, but shows as Offline. Log snippet:

2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 249856}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "5P35GFHRGILR5MRNCEANSFYCV6UFVKKVLE7IPU2ZK3UE4NLQVGUA"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 3328}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "L33MTTEA6VGCZPQZDM3QT4ZRZRUBLVLXQEXVFBVSNYWA3LB4A3RA"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 249856}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "UU6ZKLEK52CNALR3JGOHUSMBQ75EP2EUNHVTDERZRPUJD47QMSAA"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 249856}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "ALEUTWWYU6P2Z3X6X5NJMAJ7DBI7ASDY7SAUBMO4X7TPCAWLEUCA"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 249856}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "YXDUVJXEZ4L5BGHWZLC3ACXIZET72WFE5MZ2RU4EDSHATTHOQILA"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 3328}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "4FQNRTB2TIHVOVJCCRONY6MOOULMGT2FYCNWCC6RDMBWRG5P5CQQ"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 3328}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "PFFOR4BIJBZ6DFMMBYQK7LYXB5AGJCPBPZKGWITCXWD5YSSR53OA"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 249856}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "5BSM4NRG37HK2QXMUWACQKDPGARQTQHDX2DRDAWBN5KJIEEAXU2A"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 249856}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "YNBRFFZJAGL6TQG3YBM3NT7GKMZLXWOH2UIMFDR5EOIZYYR5GZHA"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 3328}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "3CD5UZFBQ3QAMG6S3OL5FOS7DA5NXUDN7UJBDKEBJKCCLODSQHZQ"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 249856}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "65QDUJ5G7IAFSVA5GEJFZRZXLV3A4FZG5JGQ5OBZON3UAZIIMBZA"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 249856}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "RRGGYHIHYHDTZQEDQVHYAF6OU6KMSG26SD5R3DIPI7VQT5OO5CJQ"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 3328}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "JBP27FGRM4I7CQU7L7BWKISG2AURPGDD52UESXBJ6YQFXD4Z67HQ"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 249856}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "EJI6H4TQ2VYMBVEWE2223OC5P5HO3ATAQONMXHULWDOXV4F456LA"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 3328}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "3NODELZWLYUC65FV5DLDPX4GPE7CP5MAZUWN3BML7WH5ZYFOOTTQ"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 3328}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "PWDYTONVWEEC6VXEUZZBX2SN4MREYMFJFNAPASN5FBRH2XWWTARQ"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 3328}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "MSG7TOEBPXKOS3BWOHSMSEP7ADQVPZCQOZSZE22EM7DNELLDNU7A"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 3328}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "XDNHJ7WDEQTJB3VOOTHAQHCAPKS57JDU2PS6MWIS7SQGBFQQZJ5A"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 249856}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "USLO7G3XBYB5JZSIM6SZ6YPKY7RRAAAEWCXFETZ7SSLM6S4T6EYA"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 3328}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "OX6LTYLLYVWGXGXXSG5SKDA2XGL6R2IJWRJNW5K43GNHF5XMOEJQ"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 3328}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "EAGIKUZF2SLB4RIV5Q6QJCVQYBAUEGW4BZTHDH72D5VRGXNNAFSA"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 3328}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "EYJJ27D3HLRY3I7WHMZWZYAGOYI4BWJ54GQM6XS3UFZVSJRPHX3Q"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 249856}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "7VIMAIR2GSBZILTTFO6Z5VPM7E7FDALNPG2DW3DLUJVU3ZVPXJHQ"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 3328}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "PXUPH345LZYNVC2T6G7AT4EIIR5ZTK5XE3NMGYPDGU5WJA2EE24Q"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 249856}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "3OC3VZ6YZMTBQPNSFNMX6WY4ZMYOEVGDLFMEI5YUOGMZL7KASA7Q"}
2024-07-15T19:31:31Z    WARN    collector       unable to delete piece  {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "YDEKSMIZBDDZ6QYSO47JSIUSPYWSOOM6ZUED3KYY5FIE4BBUG2HA", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).Stat:122\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:290\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).DeleteWithStorageFormat:270\n\tstorj.io/storj/storagenode/pieces.(*Store).DeleteSkipV0:356\n\tstorj.io/storj/storagenode/collector.(*Service).Collect.func1:88\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).GetExpired:71\n\tstorj.io/storj/storagenode/pieces.(*Store).GetExpired:576\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:87\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:65\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/storj/storagenode/collector.(*Service).Run:61\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 249856}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "NY3DJV6BWWODN7CAU2L5IWPHQXIZ47B7RVYBDX4PVTI4R7EWYWDQ"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 249856}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "QG67DD2PVKYSRAVKUHZGXEKYFDWTIEKXC5PFLWB7JQ47ZUGQIK2A"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 3328}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "KEBGDSERJ54G63TGPXYXJPLZOD4JARCA7DSAR7SNDCUZKCDPWI4A"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 249856}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "C2VCTIAVKGFO5CYPYJSA6PGI4JBF3SQ2C54UGGNCOQTL4ZRT46NA"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 3328}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "XLOPMFQ5IJYVAODYUZNKRLH2SNAFOWATWVMXZT4ULN4RK34CHGXQ"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 3328}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "OKFQNFCE4F5MJ3PRGM4NANUDZW3OJHKAQFEOHPF5E6253WWLWVCQ"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 3328}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "UYI3T7SDRD677G7OKH2N42YT3HCCCZBODKFDF44F7X4A6QG6Z4RQ"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 3328}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "G6EHGD5RWHCVALJPVQM7N6IDTFJ54F5MJBVEPCO74VBN45PX3NBQ"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 249856}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "F6IFIMHP2JHM3ZOQHPV4ALFMKHYBAMBWUY4LOVXWYPIFY7AKCIOQ"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 249856}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "TMVGYGDM4AVLOAX32DWPMNSXNFCQNGH3LL7UARB5QD3IZGNF4FXA"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 249856}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "VPPW2652IUBFXKYW2PMJO36XKKYU35OVQEM74OBGRFFPJOXR3WIQ"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 249856}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "ZU3XHU2LGUREIPS2V4XFE52MVHBLYH4RXAC3CTFGZRLGJ4D7S7KA"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 249856}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "KR2647NG7GHSSR3YYQIO5L5YAFKSA2MGE4TVQ5IL2RH3FBIO3NIA"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 3328}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "WCDQ4BHKBL6K7NDWKQO3D4EHRR5JTWPTDVYR35DIPI7GVMRZ24QQ"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 3328}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "UGR4EI7IBKXJ5YT2F7GQEGXJBYBYMMJIYJS7BA4ZS64ADUU6E46A"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 249856}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "577SICGAHDY3PR3N4OXM4THSDDGQELP5225YDHNLVBZ33ESB72PQ"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 3328}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "P74DHDB3UHEPZA5OKZL7R7BBL33FT7U5OYRHRNEUSP4ASJMTFPNQ"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 249856}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "JBKJBRC4PQPQFM5MX3V6EEANSMZEMAFK6KLBE5NB2IJ76BH5DYVQ"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 249856}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "34TIY7K3DM5CN3DS6DVBIBWDWYRMOJBFTQRFJBTWUTTXRZTJ227A"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 3328}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "EVVVAEK6ERMVJBQ45PT3F7P6EILT6TTX5547EIFMG32G2SWQNK6Q"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 3328}
2024-07-15T19:31:31Z    DEBUG   collector       deleted expired piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Piece ID": "767GX2YAE5VMEMJSMD3FBQECUO3HPPGT32ODRNFRF6HRKW7UQAJA"}
2024-07-15T19:31:31Z    DEBUG   blobscache      deleted piece   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Version": 1, "disk space freed in bytes": 3328}

Additionally, I also see that the CPU usage is insanely high. All 4 cores are consistently loaded at 100%, and yes this is only the case when storj container is up and running.

14

Try restarting your Pi

15

I’m gonna give it an hour, maybe 2 tops to see if it sorts itself out somehow, then I will restart and give an update.

16

Still malfunctioning after restarting. If anyone at all is interested in helping me troubleshoot the new buggy version of Storj, I have a cute little snippet from the logs:

2024-07-15T20:05:04Z    DEBUG   piecestore      upload failed   {"Process": "storagenode", "Piece ID": "HX4UWHPWIXANNXTHD7C7FGUPF6O7I5GIH34IM34WCIHBLJ7CXJ6Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.219.41:39548", "Size": 14080, "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func6:526\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:535\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:167\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:109\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:157\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-07-15T20:05:04Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"Process": "storagenode", "Piece ID": "X5HDUANUC2J2LV6NNAE4HQUAOCQTZ5ALJEO66CHOEPOQB7LLO4YQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "109.61.92.73:46874"}
2024-07-15T20:05:04Z    DEBUG   piecestore      upload failed   {"Process": "storagenode", "Piece ID": "X5HDUANUC2J2LV6NNAE4HQUAOCQTZ5ALJEO66CHOEPOQB7LLO4YQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "109.61.92.73:46874", "Size": 2816, "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func6:526\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:535\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:167\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:109\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:157\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-07-15T20:05:04Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"Process": "storagenode", "Piece ID": "FCMDBFEYOHGV6PDCP6AXPMO6NSVXO7JPG6EU6BUSFNX4ULXV5Y5Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.201.210:36774"}
2024-07-15T20:05:04Z    DEBUG   piecestore      upload failed   {"Process": "storagenode", "Piece ID": "FCMDBFEYOHGV6PDCP6AXPMO6NSVXO7JPG6EU6BUSFNX4ULXV5Y5Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.201.210:36774", "Size": 24064, "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func6:526\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:535\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:167\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:109\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:157\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-07-15T20:05:04Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"Process": "storagenode", "Piece ID": "JOPUC4IPLT2LKESJCDLHHMJTANKCQGEI4UU5FWPLOJUYZB6RDKQQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.219.43:59398"}
2024-07-15T20:05:04Z    DEBUG   piecestore      upload failed   {"Process": "storagenode", "Piece ID": "JOPUC4IPLT2LKESJCDLHHMJTANKCQGEI4UU5FWPLOJUYZB6RDKQQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.219.43:59398", "Size": 9728, "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func6:526\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:535\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:167\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:109\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:157\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-07-15T20:05:04Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"Process": "storagenode", "Piece ID": "YIOOVNQHFLN6S6AQHFB45NCMH5KXTF2PY7D7FOXLZBD6TDC7GF7A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.226.102:38972"}
2024-07-15T20:05:04Z    DEBUG   piecestore      upload failed   {"Process": "storagenode", "Piece ID": "YIOOVNQHFLN6S6AQHFB45NCMH5KXTF2PY7D7FOXLZBD6TDC7GF7A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.226.102:38972", "Size": 2048, "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func6:526\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:535\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:167\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:109\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:157\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-07-15T20:05:04Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"Process": "storagenode", "Piece ID": "FSRVEYRZAASZVPDX5IOIDQWHCURPZRHD6CEEX2JEZ2GHJ2EBVZCQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.219.44:42022"}
2024-07-15T20:05:04Z    DEBUG   piecestore      upload failed   {"Process": "storagenode", "Piece ID": "FSRVEYRZAASZVPDX5IOIDQWHCURPZRHD6CEEX2JEZ2GHJ2EBVZCQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.219.44:42022", "Size": 24064, "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func6:526\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:535\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:167\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:109\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:157\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-07-15T20:05:04Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"Process": "storagenode", "Piece ID": "N627JAS4OIXDXHFCRFRQSFOMGOXNUFGY55L3F27BDXKKON4W24NA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.219.37:56020"}
2024-07-15T20:05:04Z    DEBUG   piecestore      upload failed   {"Process": "storagenode", "Piece ID": "N627JAS4OIXDXHFCRFRQSFOMGOXNUFGY55L3F27BDXKKON4W24NA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.219.37:56020", "Size": 12288, "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func6:526\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:535\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:167\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:109\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:157\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-07-15T20:05:04Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"Process": "storagenode", "Piece ID": "ISMI4HSZ5DF5DXEXJ3OINXPRGOICSTQ67IV6U7X4Z4EVV7DZRZWQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.205.244:48044"}
2024-07-15T20:05:04Z    DEBUG   piecestore      upload failed   {"Process": "storagenode", "Piece ID": "ISMI4HSZ5DF5DXEXJ3OINXPRGOICSTQ67IV6U7X4Z4EVV7DZRZWQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.205.244:48044", "Size": 56320, "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func6:526\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:535\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:167\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:109\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:157\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-07-15T20:05:04Z    INFO    piecestore      upload canceled (race lost or node shutdown)    {"Process": "storagenode", "Piece ID": "E2UUALCMJXK442STYPBQ4CXMSYM2Y763GMYPG32IYBTIKZ3V6G7A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.226.97:38248"}
2024-07-15T20:05:04Z    DEBUG   piecestore      upload failed   {"Process": "storagenode", "Piece ID": "E2UUALCMJXK442STYPBQ4CXMSYM2Y763GMYPG32IYBTIKZ3V6G7A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.226.97:38248", "Size": 4096, "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func6:526\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:535\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:294\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:167\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:109\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:157\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2024-07-15T20:05:04Z    INFO    piecestore      upload canceled {"Process": "storagenode", "Piece ID": "ZW7IAUEHZVIQ3A6YVNH4FJYYGZLBWAWZRI2TCSQJP4GFXFLAOQDQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT_REPAIR", "Remote Address": "199.102.71.65:34768", "Size": 0}
2024-07-15T20:16:19Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-07-15T20:16:19Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.108.3"}
2024-07-15T20:16:19Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-07-15T20:16:19Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.108.3"}
2024-07-15T20:16:19Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-07-15T20:31:19Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-07-15T20:31:19Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.108.3"}
2024-07-15T20:31:19Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-07-15T20:31:19Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.108.3"}
2024-07-15T20:31:19Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-07-15T20:46:19Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-07-15T20:46:19Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.108.3"}
2024-07-15T20:46:19Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-07-15T20:46:19Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.108.3"}
2024-07-15T20:46:19Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-07-15T21:01:19Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-07-15T21:01:19Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.108.3"}
2024-07-15T21:01:19Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-07-15T21:01:19Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.108.3"}
2024-07-15T21:01:19Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-07-15T21:16:19Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-07-15T21:16:19Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.108.3"}
2024-07-15T21:16:19Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-07-15T21:16:19Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.108.3"}
2024-07-15T21:16:19Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-07-15T21:31:19Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-07-15T21:31:19Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.108.3"}
2024-07-15T21:31:19Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-07-15T21:31:19Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.108.3"}
2024-07-15T21:31:19Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-07-15T21:46:19Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-07-15T21:46:19Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.108.3"}
2024-07-15T21:46:19Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-07-15T21:46:19Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.108.3"}
2024-07-15T21:46:19Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-07-15T22:01:19Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-07-15T22:01:19Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.108.3"}
2024-07-15T22:01:19Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-07-15T22:01:19Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.108.3"}
2024-07-15T22:01:19Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-07-15T22:16:19Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-07-15T22:16:19Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.108.3"}
2024-07-15T22:16:19Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-07-15T22:16:19Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.108.3"}
2024-07-15T22:16:19Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-07-15 22:19:50,008 WARN received SIGTERM indicating exit request
2024-07-15 22:19:50,163 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2024-07-15T22:19:50Z    INFO    Got a signal from the OS: "terminated"  {"Process": "storagenode-updater"}
2024-07-15 22:19:50,179 INFO stopped: storagenode-updater (exit status 0)
2024-07-15T22:19:50Z    INFO    Got a signal from the OS: "terminated"  {"Process": "storagenode"}
2024-07-15 22:19:53,187 INFO waiting for storagenode, processes-exit-eventlistener to die
2024-07-15 22:19:56,191 INFO waiting for storagenode, processes-exit-eventlistener to die
2024-07-15 22:19:59,195 INFO waiting for storagenode, processes-exit-eventlistener to die
2024-07-15 22:20:00,196 WARN killing 'storagenode' (55) with SIGKILL
2024-07-15 22:20:00,257 INFO stopped: storagenode (terminated by SIGKILL)
2024-07-15 22:20:00,259 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)
2024-07-15 22:23:43,752 INFO RPC interface 'supervisor' initialized
2024-07-15 22:23:43,754 INFO supervisord started with pid 1
2024-07-15 22:23:44,759 INFO spawned: 'processes-exit-eventlistener' with pid 12
2024-07-15 22:23:44,772 INFO spawned: 'storagenode' with pid 13
2024-07-15 22:23:44,778 INFO spawned: 'storagenode-updater' with pid 14
2024-07-15T22:23:45Z    INFO    Configuration loaded    {"Process": "storagenode-updater", "Location": "/app/config/config.yaml"}
2024-07-15T22:23:45Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.wallet-features"}
2024-07-15T22:23:45Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-disk-space"}
2024-07-15T22:23:45Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.email"}
2024-07-15T22:23:45Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-bandwidth"}
2024-07-15T22:23:45Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "console.address"}
2024-07-15T22:23:45Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "contact.external-address"}
2024-07-15T22:23:45Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.private-address"}
2024-07-15T22:23:45Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "healthcheck.details"}
2024-07-15T22:23:45Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.address"}
2024-07-15T22:23:45Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "healthcheck.enabled"}
2024-07-15T22:23:45Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.wallet"}
2024-07-15T22:23:45Z    INFO    Anonymized tracing enabled      {"Process": "storagenode-updater"}
2024-07-15T22:23:45Z    INFO    Running on version      {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.108.3"}
2024-07-15T22:23:45Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-07-15T22:23:45Z    INFO    Configuration loaded    {"Process": "storagenode", "Location": "/app/config/config.yaml"}
2024-07-15 22:23:45,963 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-07-15 22:23:45,963 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-07-15 22:23:45,964 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-07-15T22:23:45Z    INFO    Anonymized tracing enabled      {"Process": "storagenode"}
2024-07-15T22:23:45Z    DEBUG   tracing collector       started {"Process": "storagenode"}
2024-07-15T22:23:45Z    DEBUG   debug server listening on 127.0.0.1:39461       {"Process": "storagenode"}
2024-07-15T22:23:45Z    INFO    Operator email  {"Process": "storagenode", "Address": "myemailaddress@mail.com"}
2024-07-15T22:23:45Z    INFO    Operator wallet {"Process": "storagenode", "Address": "mywalletaddr"}
2024-07-15T22:23:46Z    DEBUG   Version info    {"Process": "storagenode", "Version": "1.108.3", "Commit Hash": "318a54287f2ebd59a5cd9ce3320cee798007a51c", "Build Timestamp": "2024-07-09 18:27:39 +0000 UTC", "Release Build": true}
2024-07-15T22:23:46Z    INFO    server  kernel support for server-side tcp fast open remains disabled.{"Process": "storagenode"}
2024-07-15T22:23:46Z    INFO    server  enable with: sysctl -w net.ipv4.tcp_fastopen=3  {"Process": "storagenode"}
2024-07-15T22:23:46Z    DEBUG   version Allowed minimum version from control server.    {"Process": "storagenode", "Minimum Version": "1.105.4"}
2024-07-15T22:23:46Z    DEBUG   version Running on allowed version.     {"Process": "storagenode", "Version": "1.108.3"}
2024-07-15T22:23:46Z    INFO    Telemetry enabled       {"Process": "storagenode", "instance ID": "12sz6NY5JinWKoGUBQ6Az5Cu3nhnVCo1zbmLiXjqwTRDz9G4mLt"}
2024-07-15T22:23:46Z    INFO    Event collection enabled        {"Process": "storagenode", "instance ID": "12sz6NY5JinWKoGUBQ6Az5Cu3nhnVCo1zbmLiXjqwTRDz9G4mLt"}
2024-07-15T22:23:46Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.108.3"}
2024-07-15T22:23:46Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2024-07-15T22:23:46Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.108.3"}
2024-07-15T22:23:46Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-07-15T22:23:46Z    INFO    db.migration    Database Version        {"Process": "storagenode", "version": 60}
2024-07-15T22:23:46Z    DEBUG   db      Database version is up to date  {"Process": "storagenode", "version": 60}```
17

Maybe one or more of your databases got corrupted during the migration? Perhaps you can backup your databases (or just move them to another folder) and try starting the node with no databases at all and see if that makes any difference. If the node starts up it should re-create the missing databases. Just be careful not to remove or delete the storage-dir-verification file which is usually in the same folder as all the databases. If that doesn’t solve the issue you can just move your original databases back to their original location.

18

How much Ram does this PI have? It could be running out of memory (OOM).

19

Will give this a shot tomorrow, thanks for the suggestion.

20

I’ve got the 8GB RAM model; earlier when I was monitoring it, I saw it was utilizing >7 gb, which is unusual, even for Storj (usually sits around 3gb). Maybe this is so. But I don’t understand how RAM usage could’ve (more than) doubled via an update.