Version 1.52.2 : Error: The directory named as part of the path /var/log/supervisor/supervisord.log does not exist

So, my nodes running on a poor RPi 4B (32bit OS) stopped receiving data because, well as you may have guessed I’m too many versions behind (still on v1.49.5) and there’s no official ARM 32bit docker image released yet (!)

I followed @baker’s good post below to update one of my nodes to version storjlabs/storagenode:bfc3d4c9d-v1.52.2-go1.17.5 to see how it goes:

The result isn’t great, here are the docker logs:

downloading storagenode-updater
--2022-04-20 20:17:58--  https://version.storj.io/processes/storagenode-updater/minimum/url?os=linux&arch=arm
Resolving version.storj.io (version.storj.io)... 35.224.88.204
Connecting to version.storj.io (version.storj.io)|35.224.88.204|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 90 [text/plain]
Saving to: 'STDOUT'

     0K                                                       100% 17.8M=0s

2022-04-20 20:17:58 (17.8 MB/s) - written to stdout [90/90]

--2022-04-20 20:17:58--  https://github.com/storj/storj/releases/download/v1.52.2/storagenode-updater_linux_arm.zip
Resolving github.com (github.com)... 140.82.121.3
Connecting to github.com (github.com)|140.82.121.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/f0f96f9f-3f1d-4fb0-a949-448181cfa573?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220420%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220420T201759Z&X-Amz-Expires=300&X-Amz-Signature=2276dc24bbfab4b3f9e5a45991b276b8857c9a5d0fe4f15d86012232ff970335&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_arm.zip&response-content-type=application%2Foctet-stream [following]
--2022-04-20 20:17:59--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/f0f96f9f-3f1d-4fb0-a949-448181cfa573?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220420%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220420T201759Z&X-Amz-Expires=300&X-Amz-Signature=2276dc24bbfab4b3f9e5a45991b276b8857c9a5d0fe4f15d86012232ff970335&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_arm.zip&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.108.133, 185.199.111.133, 185.199.110.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 7445387 (7.1M) [application/octet-stream]
Saving to: '/tmp/storagenode-updater.zip'

     0K .......... .......... .......... .......... ..........  0% 2.42M 3s
[...]
  7250K .......... ..........                                 100% 25.1M=0.6s

2022-04-20 20:18:00 (11.6 MB/s) - '/tmp/storagenode-updater.zip' saved [7445387/7445387]

downloading storagenode
--2022-04-20 20:18:00--  https://version.storj.io/processes/storagenode/minimum/url?os=linux&arch=arm
Resolving version.storj.io (version.storj.io)... 35.224.88.204
Connecting to version.storj.io (version.storj.io)|35.224.88.204|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 82 [text/plain]
Saving to: 'STDOUT'

     0K                                                       100% 8.11M=0s

2022-04-20 20:18:01 (8.11 MB/s) - written to stdout [82/82]

--2022-04-20 20:18:01--  https://github.com/storj/storj/releases/download/v1.52.2/storagenode_linux_arm.zip
Resolving github.com (github.com)... 140.82.121.3
Connecting to github.com (github.com)|140.82.121.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/d9c2ef87-1c86-4d78-a305-9133483c5239?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220420%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220420T201801Z&X-Amz-Expires=300&X-Amz-Signature=324099c1fc326425ffc9bea1fa0a3cd144476e2a8b8d039504296331e96cb725&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode_linux_arm.zip&response-content-type=application%2Foctet-stream [following]
--2022-04-20 20:18:01--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/d9c2ef87-1c86-4d78-a305-9133483c5239?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220420%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220420T201801Z&X-Amz-Expires=300&X-Amz-Signature=324099c1fc326425ffc9bea1fa0a3cd144476e2a8b8d039504296331e96cb725&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode_linux_arm.zip&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.109.133, 185.199.110.133, 185.199.111.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.109.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 14489454 (14M) [application/octet-stream]
Saving to: '/tmp/storagenode.zip'

     0K .......... .......... .......... .......... ..........  0% 2.52M 5s
[...]
 14100K .......... .......... .......... .......... ......... 100% 17.9M=1.0s

2022-04-20 20:18:03 (13.2 MB/s) - '/tmp/storagenode.zip' saved [14489454/14489454]

Error: The directory named as part of the path /var/log/supervisor/supervisord.log does not exist
For help, use /usr/bin/supervisord -h
Error: The directory named as part of the path /var/log/supervisor/supervisord.log does not exist
For help, use /usr/bin/supervisord -h
Error: The directory named as part of the path /var/log/supervisor/supervisord.log does not exist
For help, use /usr/bin/supervisord -h
Error: The directory named as part of the path /var/log/supervisor/supervisord.log does not exist
For help, use /usr/bin/supervisord -h
[...]

The node stays stuck on this error.
I’m not sure what to do :confused:

That’s one I haven’t seen yet. Myself, I would try the removing and recreating the container again in case it’s a weird one time error. I suppose you could try creating the directory in the container manually. You would have to do it while the container is running (assuming it isn’t in a restart loop).

docker exec -it storagenode mkdir /var/log/supervisor

In my container, the /var/log/supervisor folder is owned by root with rwx permissions for all

$ docker exec -it storagenode ls -la /var/log/ 
...
drwxrwxrwx 1 root root  4096 Apr  6 23:13 supervisor
...
2 Likes

Thanks a lot.

Removing the container and recreating it just in case did not solve the issue.
However, creating the folder manually did work.

I’m surprised I’m the only one facing this though? Shouldn’t everyone running this image have faced the same issue?

2 Likes

Very strange, glad it worked though. I am running this image and didn’t see that problem, although on an arm64 platform. And you’re definitely not the only one doing this for an arm32 RPi . Hopefully a storjling can chime in on this one.

1 Like

That’d be nice.

For anyone facing the same issue, be aware that containers that are stuck this way keep restarting:

pi@raspberrypi:~ $ sudo docker ps
CONTAINER ID        IMAGE                                              COMMAND                  CREATED             STATUS                         PORTS                                                                       NAMES
[...]
6e728b2712ef        storjlabs/storagenode:bfc3d4c9d-v1.52.2-go1.17.5   "/entrypoint --log.o…"   14 seconds ago      Restarting (2) 2 seconds ago                                                                               storj_node_2
[...]
                                                                                           storj_watchtower

So unless you manually create the folder just after starting the node while it is still setting itself up (downloading binaries), you may have to issue the command several times before it succeeds.

Let me illustrate:

pi@raspberrypi:~ $ sudo docker exec -it storj_node_2 mkdir /var/log/supervisor
Error response from daemon: Container 66309508bb99470998ce2c4b1162a20b61d369c865bd3ff03c316105c2fadff4 is restarting, wait until the container is running
pi@raspberrypi:~ $ sudo docker exec -it storj_node_2 mkdir /var/log/supervisor
Error response from daemon: Container 66309508bb99470998ce2c4b1162a20b61d369c865bd3ff03c316105c2fadff4 is restarting, wait until the container is running
[...]
pi@raspberrypi:~ $ sudo docker exec -it storj_node_2 mkdir /var/log/supervisor
Error response from daemon: Container 66309508bb99470998ce2c4b1162a20b61d369c865bd3ff03c316105c2fadff4 is restarting, wait until the container is running
pi@raspberrypi:~ $ sudo docker exec -it storj_node_2 mkdir /var/log/supervisor
pi@raspberrypi:~ $ 

Thanks again @baker for your quick and effective reply :+1: :slight_smile:

2 Likes