Version 1.52.2 : Error: The directory named as part of the path /var/log/supervisor/supervisord.log does not exist

[Pac]

So, my nodes running on a poor RPi 4B (32bit OS) stopped receiving data because, well as you may have guessed I’m too many versions behind (still on v1.49.5) and there’s no official ARM 32bit docker image released yet (!)

I followed @baker’s good post below to update one of my nodes to version storjlabs/storagenode:bfc3d4c9d-v1.52.2-go1.17.5 to see how it goes:

No ingress in the last 16 hours

@RuMbOiDe and @gs.gs At the moment, the latest tag will not update for for arm32. You will need to manually update your nodes with a specific image, otherwise your nodes will not get any more ingress. I know Storj is working on updating the latest tag, but for now you will have to use a specific tag. Stop your node, remove the container, and run your docker run command again, except instead of using storjlabs/storagenode:latest change the image to storjlabs/storagenode:bfc3d4c9d-v1.52.2-go1.17…

The result isn’t great, here are the docker logs:

downloading storagenode-updater
--2022-04-20 20:17:58--  https://version.storj.io/processes/storagenode-updater/minimum/url?os=linux&arch=arm
Resolving version.storj.io (version.storj.io)... 35.224.88.204
Connecting to version.storj.io (version.storj.io)|35.224.88.204|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 90 [text/plain]
Saving to: 'STDOUT'

     0K                                                       100% 17.8M=0s

2022-04-20 20:17:58 (17.8 MB/s) - written to stdout [90/90]

--2022-04-20 20:17:58--  https://github.com/storj/storj/releases/download/v1.52.2/storagenode-updater_linux_arm.zip
Resolving github.com (github.com)... 140.82.121.3
Connecting to github.com (github.com)|140.82.121.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/f0f96f9f-3f1d-4fb0-a949-448181cfa573?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220420%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220420T201759Z&X-Amz-Expires=300&X-Amz-Signature=2276dc24bbfab4b3f9e5a45991b276b8857c9a5d0fe4f15d86012232ff970335&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_arm.zip&response-content-type=application%2Foctet-stream [following]
--2022-04-20 20:17:59--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/f0f96f9f-3f1d-4fb0-a949-448181cfa573?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220420%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220420T201759Z&X-Amz-Expires=300&X-Amz-Signature=2276dc24bbfab4b3f9e5a45991b276b8857c9a5d0fe4f15d86012232ff970335&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_arm.zip&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.108.133, 185.199.111.133, 185.199.110.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 7445387 (7.1M) [application/octet-stream]
Saving to: '/tmp/storagenode-updater.zip'

     0K .......... .......... .......... .......... ..........  0% 2.42M 3s
[...]
  7250K .......... ..........                                 100% 25.1M=0.6s

2022-04-20 20:18:00 (11.6 MB/s) - '/tmp/storagenode-updater.zip' saved [7445387/7445387]

downloading storagenode
--2022-04-20 20:18:00--  https://version.storj.io/processes/storagenode/minimum/url?os=linux&arch=arm
Resolving version.storj.io (version.storj.io)... 35.224.88.204
Connecting to version.storj.io (version.storj.io)|35.224.88.204|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 82 [text/plain]
Saving to: 'STDOUT'

     0K                                                       100% 8.11M=0s

2022-04-20 20:18:01 (8.11 MB/s) - written to stdout [82/82]

--2022-04-20 20:18:01--  https://github.com/storj/storj/releases/download/v1.52.2/storagenode_linux_arm.zip
Resolving github.com (github.com)... 140.82.121.3
Connecting to github.com (github.com)|140.82.121.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/d9c2ef87-1c86-4d78-a305-9133483c5239?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220420%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220420T201801Z&X-Amz-Expires=300&X-Amz-Signature=324099c1fc326425ffc9bea1fa0a3cd144476e2a8b8d039504296331e96cb725&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode_linux_arm.zip&response-content-type=application%2Foctet-stream [following]
--2022-04-20 20:18:01--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/d9c2ef87-1c86-4d78-a305-9133483c5239?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220420%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220420T201801Z&X-Amz-Expires=300&X-Amz-Signature=324099c1fc326425ffc9bea1fa0a3cd144476e2a8b8d039504296331e96cb725&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode_linux_arm.zip&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.109.133, 185.199.110.133, 185.199.111.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.109.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 14489454 (14M) [application/octet-stream]
Saving to: '/tmp/storagenode.zip'

     0K .......... .......... .......... .......... ..........  0% 2.52M 5s
[...]
 14100K .......... .......... .......... .......... ......... 100% 17.9M=1.0s

2022-04-20 20:18:03 (13.2 MB/s) - '/tmp/storagenode.zip' saved [14489454/14489454]

Error: The directory named as part of the path /var/log/supervisor/supervisord.log does not exist
For help, use /usr/bin/supervisord -h
Error: The directory named as part of the path /var/log/supervisor/supervisord.log does not exist
For help, use /usr/bin/supervisord -h
Error: The directory named as part of the path /var/log/supervisor/supervisord.log does not exist
For help, use /usr/bin/supervisord -h
Error: The directory named as part of the path /var/log/supervisor/supervisord.log does not exist
For help, use /usr/bin/supervisord -h
[...]

The node stays stuck on this error.
I’m not sure what to do

[baker]

That’s one I haven’t seen yet. Myself, I would try the removing and recreating the container again in case it’s a weird one time error. I suppose you could try creating the directory in the container manually. You would have to do it while the container is running (assuming it isn’t in a restart loop).

docker exec -it storagenode mkdir /var/log/supervisor

In my container, the /var/log/supervisor folder is owned by root with rwx permissions for all

$ docker exec -it storagenode ls -la /var/log/ 
...
drwxrwxrwx 1 root root  4096 Apr  6 23:13 supervisor
...

[Pac]

Thanks a lot.

Removing the container and recreating it just in case did not solve the issue.
However, creating the folder manually did work.

I’m surprised I’m the only one facing this though? Shouldn’t everyone running this image have faced the same issue?

[baker]

Very strange, glad it worked though. I am running this image and didn’t see that problem, although on an arm64 platform. And you’re definitely not the only one doing this for an arm32 RPi . Hopefully a storjling can chime in on this one.

[Pac]

@baker

Hopefully a storjling can chime in on this one

That’d be nice.

For anyone facing the same issue, be aware that containers that are stuck this way keep restarting:

pi@raspberrypi:~ $ sudo docker ps
CONTAINER ID        IMAGE                                              COMMAND                  CREATED             STATUS                         PORTS                                                                       NAMES
[...]
6e728b2712ef        storjlabs/storagenode:bfc3d4c9d-v1.52.2-go1.17.5   "/entrypoint --log.o…"   14 seconds ago      Restarting (2) 2 seconds ago                                                                               storj_node_2
[...]
                                                                                           storj_watchtower

So unless you manually create the folder just after starting the node while it is still setting itself up (downloading binaries), you may have to issue the command several times before it succeeds.

Let me illustrate:

pi@raspberrypi:~ $ sudo docker exec -it storj_node_2 mkdir /var/log/supervisor
Error response from daemon: Container 66309508bb99470998ce2c4b1162a20b61d369c865bd3ff03c316105c2fadff4 is restarting, wait until the container is running
pi@raspberrypi:~ $ sudo docker exec -it storj_node_2 mkdir /var/log/supervisor
Error response from daemon: Container 66309508bb99470998ce2c4b1162a20b61d369c865bd3ff03c316105c2fadff4 is restarting, wait until the container is running
[...]
pi@raspberrypi:~ $ sudo docker exec -it storj_node_2 mkdir /var/log/supervisor
Error response from daemon: Container 66309508bb99470998ce2c4b1162a20b61d369c865bd3ff03c316105c2fadff4 is restarting, wait until the container is running
pi@raspberrypi:~ $ sudo docker exec -it storj_node_2 mkdir /var/log/supervisor
pi@raspberrypi:~ $ 

---

Thanks again @baker for your quick and effective reply