Node stopped working as docker container on qnap-container-station

The reason likely that it doesn’t allow exec even in the docker volume to my surprise.
You may see an error during the run of the container, if you would temporary replace -d option with -it, it should print everything to the console.

Could you please show an output of the command:

docker run -it --rm -v storagenode-binaries:/app/config/bin ubuntu ls -l /app/config/bin/

Another workaround would be to use a BINARY_DIR to provide a different path inside the container.
Please try to add -e BINARY_DIR=/app/bin, this will disable the feature to persist binaries between restarts.

The command outputs nothing - the directory is empty.

[~] # docker run -it --rm -v storagenode-binaries:/app/config/bin ubuntu ls -l /app/config/bin/
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
eb7e9efa9f9d: Pull complete
Digest: sha256:dfc10878be8d8fc9c61cbff33166cb1d1fe44391539243703c72766894fa834a
Status: Downloaded newer image for ubuntu:latest
[~] #
[~] #
[~] # cd /share/CACHEDEV1_DATA/Container/container-station-data/lib/docker/volumes/storagenode-binaries/_data
[/share/CACHEDEV1_DATA/Container/container-station-data/lib/docker/volumes/storagenode-binaries/_data] # ll
total 8
drwxr-xr-x    2 admin    administ    4.0K Sep 22 19:00 ./
drwx-----x    3 admin    administ    4.0K Sep 22 19:00 ../

I tried starting the container with -it instead of -d and with the “binary-dir-e” - no success and not one line of output.

PS:
i started the 0c21bd6-node with '-e BINARY_DIR=/app/config/bin ' - the node is running.

but:

[/share/Container/STORJ/config/bin] # ll
total 8
drwxr-xr-x    2 admin    administ    4.0K Sep 22 19:09 ./
drwxrwxrwx    6 admin    administ    4.0K Sep 24 08:08 ../

If you started it with -e BINARY_DIR=/app/bin, then the binary would be inside the container, they will not be exposed to the host. So, yes, it would continue to be empty.
If you started it with -e BINARY_DIR=/app/config/bin, then you need to check logs, to see where binaries are downloaded, or check it inside the container:

docker exec -it storagenode bash

then

ls -l /app
ls -l /app/config
ls -l /app/config/bin

The weird thing that it didn’t download binaries there. Even to the docker volume. And a most weird thing, that it didn’t start the node and doesn’t report anything. Is it just exiting? Or doing something?

Command from Container running with “-e BINARY_DIR=/app/config/bin”:

[/etc] # docker exec -it storagenode bash
root@862ac1c48ee9:/app# ls -l /app
total 100424
drwxrwxrwx 6 root root     4096 Sep 24 06:08 config
-rwxrwxrwx 1 root root       91 Feb 24  2023 dashboard.sh
drwxrwxrwx 2 root root     4096 Aug 17  2019 identity
-rwxr--r-- 1 root root 72589728 Sep 24 06:08 storagenode
-rwxr--r-- 1 root root 30221788 Sep 24 06:08 storagenode-updater
root@862ac1c48ee9:/app# ls -l /app/config
total 172
drwxr-xr-x 2 root root   4096 Sep 22 17:09 bin
-rw------- 1 root root   5782 Jun 28  2020 config.yaml
-rw-r--r-- 1 root root  17518 Aug  8  2020 earnings.py
-rw------- 1 root root 131072 Jun 27  2020 kademlia
drwx------ 4 root root   4096 Sep  5  2020 orders
drwxr-xr-x 2 root root   4096 Sep 24 00:13 retain
-rw------- 1 root root  32768 Sep 24 06:08 revocations.db
drwx------ 6 root root   4096 Sep 24 07:28 storage
-rw------- 1 root root   2289 Sep 24 06:08 trust-cache.json
root@862ac1c48ee9:/app# ls -l /app/config/bin
total 0

It it switching between running and other(restarting) the whole time - creating no output.

Yes, since it cannot download binaries to the /app/config/bin location.
This is weird, that it also cannot download them to the /app/bin, but perhaps it could be related to inability to create this folder due to a lack of permissions regardless of running under root (QNAP has messed the whole docker functionality in this case).
Please try to use -e BINARY_DIR=/tmp.

Please try to use -e BINARY_DIR=/tmp

same as everyday… restarting after errorlevel 139 and no logs

But this must work, this path inside the container and it’s exist (so no need to create), the Linux wouldn’t work normally without that path. What they broke?

Please try to add an exec permission to the mount options in the /etc/fstab for that drive directly (/dev/mapper/cachedev1), then try to check:

sudo mount -a

if no errors would be thrown, then try to remount this drive (if possible):

sudo mount -o remount /dev/mapper/cachedev1

Make sure that it now has the exec permission in /etc/mtab.

I tested that on Ubuntu 20.04, the rw permission also has a exec seems, so everything is working as expected, and I do not see a noexec in your mount options in

but it’s worth to try anyway.

I do not see any reason why it doesn’t produce any single log line when you use a new image. And especially, when binaries must be downloaded to /app/bin, as specified with a BINARY_DIR variable.
Please try with the new image again, without the BINARY_DIR variable and without the docker volume, replace -d with -it and add an option --log.output=stderr after the image name. It must say something, what’s wrong with the container.

first problem:
no cachedev1 in fstab

# /etc/fstab: static file system information.
#
# <file system> <mount pt>     <type>   <options>         <dump> <pass>
/dev/ram       /              ext2      defaults         1      1
proc            /proc          proc     defaults          0      0
none            /dev/pts        devpts  gid=5,mode=620  0       0

i can feel - you begin to like qnap more and more :wink:

i will look after this after noon.
many thanks for your support!

I see, they reimplemented the own wheel.

I still cannot understand, why the docker run with -it and without -d doesn’t print anything to the console. It must at least print that’s tried to start the container.

Could you please try after run the container to execute this command:

docker debug storagenode

You should get a shell in the failing container and may try to see, what is it see in the /app/, /app/config, /app/config/bin.

We prepared a new image with the fix:

Could someone with QNAP try it before we would made it latest?

sorry - had some private issues.
i tested already a bunch of things, without any clues.

Here is the new image 9109fd2

[/etc] # docker debug storagenode
docker: 'debug' is not a docker command.

and ran it with -it

[/etc] # docker logs storagenode
[/etc] #

got crazy with this boxes :frowning:

Hi folks

i tried some stuff with following releases
storjlabs/storagenode:9109fd2
storjlabs/storagenode:2159415
storjlabs/storagenode:0c21bd6

the 0c21bd6 tag is the ONLY ON generating output to -it or docker logs.
i couldn’t figure out which change is responsible for this issue :confused:

The only thing which I can think of - the new OS inside the container has changed something, but why is it cannot run it?

Is it on ARM processor?
If so, could you please try to add --platform "linux/arm64" to your docker run command before the image name?

[/etc] # docker logs storagenode --tail 500 --follow
exec /entrypoint: exec format error
exec /entrypoint: exec format error
exec /entrypoint: exec format error

something is moving :laughing:

Could you please provide the platform for your device?

uname --hardware-platform
[/etc] # uname --hardware-platform
unknown

:sob:

[/proc] # cat cpuinfo
processor       : 0
model name      : Annapurna Labs Alpine AL314 Quad-core ARM Cortex-A15 CPU @ 1.70GHz

32 bit ARM CPU, ok, please try with the --plarform "linux/arm/v5" option.

okay - with the v5-parameter we have the “old” behavior - creating NO logs “inside” the container.

[/var/log/container-station] # docker logs storagenode
[/var/log/container-station] #

I want to apologize for driving you crazy with our qnap-boxes…

No need to apologize. Just some very uncommon behavior.
And sorry, that I use your box for experimenting :slight_smile:

I do not have a QNAP box, so used the emulator. Both ARM64 and ARM32 are working perfectly, no issues, no root and no --priviledged.

By the way, could you please try to remove a --priviledged option?

1 Like