Always restart on arm64 docker image since recent days

zxz_27 · April 8, 2021, 4:17pm

Hi help,
My node has been working well for one and half years, but it seems not working- always restarting in recent days.

Platform: arm64v8, Docker version: 18.09.7

I believe docker image for arm64v8 is incorrect (out of dated) after using commands “docker pull storjlabs/storagenode:latest”
storjlabs/storagenode latest efb7528d1df3 4 months ago 29.5MB

Please can you point out correct docker image tag (":latest" seems incorrect) for arm64v8 platform.

Thanks
Larry Zheng

nerdatwork · April 8, 2021, 4:47pm

Welcome to the forum @zxz_27 !

Show last 20 lines from the log.

zxz_27 · April 9, 2021, 2:09am

“validate db version mismatch error: expected 45 != 48”.

I believe the root cause that is storjlabs/watchtower is not working properly on latest arm64v8 image update, did you test latest release on arm64v8 HW platform?
Please point out a correct docker image tag (not latest) for arm64v8 for manual update.

nerdatwork · April 9, 2021, 2:37am

Does this help?

zxz_27 · April 9, 2021, 3:09am

Your attached info is what I requested years ago and got support for both arm64v8 and arm32v7 releases.

I missed my previous account password and then created new one for this support request.

I believe your team has done Jenkins daily auto build for AMD64, arm64v8 and arm32v7 images , please ask your release QA and just point out which release (docker image tag) is suitable for arm64v8 platform.

Thanks
Larry Zheng

zxz_27 · April 9, 2021, 3:50am

AMD64 release
REPOSITORY TAG IMAGE ID CREATED SIZE
storjlabs/storagenode latest d296e0336b05 4 weeks ago 37.7MB

arm64 release
REPOSITORY TAG IMAGE ID CREATED SIZE
storjlabs/storagenode latest efb7528d1df3 4 months ago 29.5MB

I compare two releases for x86 and arm platform, so it sounds the arm64 release is 3-months out-of-dated, please tentatively point out an image tag for arm64 (or arm32 is fine) for my node recovery, meanwhile help chase release issue on arm platform.

@ifraixedes, @nerdatwork tag of latest for arm64 is 3-months out-of-dated, while tag of arm is not retrievable, could you help?

Alexey · April 9, 2021, 4:50am

You need to pull and use storjlabs/storagenode:latest
Please, try it and re-create a container

https://hub.docker.com/r/storjlabs/storagenode/tags?page=1&ordering=last_updated&name=latest

zxz_27 · April 9, 2021, 5:04am

No.
tag :latest is surly not working on arm64, I eventually downloaded all tags, and the issue got resolved on tag: arm release.

You can see all releases from above snapshot for arm64 platform

Alexey · April 9, 2021, 5:08am

Ok. Please, prune all images and pull the only :latest

zxz_27 · April 9, 2021, 5:10am

yes, I will manually create tag :latest with tag:arm image ID, I assume you will help relay this arm64 issue to your release team

Alexey · April 9, 2021, 5:14am

You are the only one who have the problem since the switch to the latest alpine for arm64 a few months ago.
So, please, help to troubleshoot the problem.
I kindly ask you to stop and remove the container:

docker stop -t 300 storagenode
docker rm storagenode

Remove all stopped containers

docker container prune

Confirm remove. And prune all images:

docker image prune

Confirm removing.
It should remove all images and layers, include dangling ones.
Then try to pull the storjlabs/storagenode:latest and show results.

zxz_27 · April 9, 2021, 6:20am

I did “docker image rm” all storjlabs/storagenode images after stop and rm container when the issue surfaced on arm64.
However pulling latest tag for arm64 which is 4-months aged seems not working (restarting every 30s), until I pull all tags and notice tag: arm is most recently updated.

Did your QA verify on arm64 platform upon tag latest?

zxz_27 · April 9, 2021, 6:34am

AMD64 images are all the same
REPOSITORY TAG IMAGE ID CREATED SIZE
storjlabs/storagenode alpha d296e0336b05 4 weeks ago 37.7MB
storjlabs/storagenode arm d296e0336b05 4 weeks ago 37.7MB
storjlabs/storagenode beta d296e0336b05 4 weeks ago 37.7MB
storjlabs/storagenode latest d296e0336b05 4 weeks ago 37.7MB

ARM64 images are different
REPOSITORY TAG IMAGE ID CREATED SIZE
storjlabs/storagenode arm 5e64bc3c759a 4 weeks ago 33MB
storjlabs/storagenode latest 5e64bc3c759a 4 weeks ago (manually synched to tag :arm, otherwise 4 months ago) 33MB
storjlabs/storagenode beta 66ea0edabd14 3 months ago 30.8MB
storjlabs/storagenode alpha fdf8af35684c 16 months ago 22.1MB

Alexey · April 10, 2021, 2:30am

How you still have images like alpha and beta, if you did pruning today?

Please, understand, we have a lot of users with ARM64, but only your system has this problem, otherwise we would have other users with the same problem here on the forum and have similar support tickets.

I would try to help to fix the problem in your specific case. Maybe there is a bug in our publish pipeline, but first we need to exclude problems on your system.
For that I need that you follow recommendations, even if you think they are useless or stupid.
If we able to fix your configuration, you would not have this problem anymore.
Please, do:

Stop and remove the storagenode container
docker system prune
docker pull storjlabs/storagenode:latest
Try to run your storagenode back
Post results

zxz_27 · April 10, 2021, 7:37am

Hi Alexey,

If you take a look at the snapshot in my 1st message, there is no Alpha/Beta/arm tags but only latest tag with 4 months aged for arm64.

I was doing your QA’s job by removing all tags, then pull and compare all AMD64 and arm64 (snapshot in my last reply), why not check your release engineer or QA to pull latest tag for arm64 to confirm the bug?

BR//Larry Zheng

Alexey · April 10, 2021, 12:04pm

Because we do not have your setup, which does not work as expected.

All available arm64 systems pull the right image.
I would try to check it on emulator though.

Could you please copy info here?

docker info

zxz_27 · April 12, 2021, 2:22pm

root@cc0105f1:~# docker info
Containers: 6
Running: 6
Paused: 0
Stopped: 0
Images: 10
Server Version: 18.09.7
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.4.139-mvebu64
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: aarch64
CPUs: 2
Total Memory: 1.886GiB
Name: cc0105f1
ID: FSGH:ZY2N:J5JC:DAU2:PZOH:XCID:MQPR:6OYU:MZ2Y:GFPV:7EBK:YJM7
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
101.89.159.2:5000
61.172.47.198:5000
127.0.0.0/8
Registry Mirrors:
https://yot4f98p.mirror.aliyuncs.com/
Live Restore Enabled: false
Product License: Community Engine

andrew2.hart · April 13, 2021, 5:35pm

root@raspberrypi3:~# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
storjlabs/storagenode latest 5e64bc3c759a 4 weeks ago 33MB
root@raspberrypi3:~# uname -a
Linux raspberrypi3 5.3.0-1018-raspi2 #20~18.04.1-Ubuntu SMP Wed Feb 5 04:56:34 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux
docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 1
Server Version: 18.09.7
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version:
runc version: N/A
init version: v0.18.0 (expected: fec3683b971d9c3ef73f284f176672c44b448662)
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.3.0-1018-raspi2
Operating System: Ubuntu 18.04.4 LTS
OSType: linux
Architecture: aarch64
CPUs: 4
Total Memory: 3.703GiB
Name: raspberrypi3
ID: RP5D:AX5F:QQRJ:EPB7:C57N:WISI:4K7Q:TC6U:YGDM:RXFY:I5WY:3USP
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

Perhaps it is these other registries interfering? I don’t know, just asking

Alexey · April 24, 2021, 12:23pm

Sorry for a long delay. I finally configured my emulator for different plaforms

$ docker pull --platform aarch64 storjlabs/storagenode
Using default tag: latest
latest: Pulling from storjlabs/storagenode
912815139b61: Pull complete
4d02e358b77f: Pull complete
cee291b58af9: Pull complete
2db525139b8a: Pull complete
50112edebc6f: Pull complete
d55283d75638: Pull complete
Digest: sha256:147a226e8a20eb99837592c0b792bc2c15eb3447d517353c79907865a7f8f6fc
Status: Downloaded newer image for storjlabs/storagenode:latest
docker.io/storjlabs/storagenode:latest

$ docker image ls storjlabs/storagenode
REPOSITORY              TAG                                  IMAGE ID            CREATED             SIZE
storjlabs/storagenode   latest                               2e3edc89250f        2 weeks ago         34.4MB
storjlabs/storagenode   d14f2e416-go1.15.6-arm64v8           0d59e5c0f02f        4 months ago        30.8MB
storjlabs/storagenode   d14f2e416-go1.15.6-arm32v6           7e69a611c33d        4 months ago        30.3MB
storjlabs/storagenode   d14f2e416-go1.15.6-amd64             09b2597eb71f        4 months ago        35.1MB
storjlabs/storagenode   <none>                               efb7528d1df3        4 months ago        29.5MB
storjlabs/storagenode   2413a9a0d-v1.18.1-go1.15.5-arm32v6   a0b54dc73d0d        4 months ago        30.4MB
storjlabs/storagenode   arm                                  bb347b79869d        4 months ago        35.4MB
storjlabs/storagenode   beta                                 927cebfc3923        14 months ago       28.2MB
storjlabs/storagenode   alpha                                72d228f5ed07        19 months ago       35.3MB

amd64
https://hub.docker.com/layers/storjlabs/storagenode/latest/images/sha256-38f07c8d48cef611ae3353e2985f367fd14928f31ab49409c8acfd9aba6a2007?context=explore

$ docker pull storjlabs/storagenode@sha256:38f07c8d48cef611ae3353e2985f367fd14928f31ab49409c8acfd9aba6a2007
sha256:38f07c8d48cef611ae3353e2985f367fd14928f31ab49409c8acfd9aba6a2007: Pulling from storjlabs/storagenode
ca3cd42a7c95: Pull complete
4cad969f73ac: Pull complete
ce2391e241ea: Pull complete
6b7c0eba9db2: Pull complete
50112edebc6f: Pull complete
8a4ba33b7ce6: Pull complete
Digest: sha256:38f07c8d48cef611ae3353e2985f367fd14928f31ab49409c8acfd9aba6a2007
Status: Downloaded newer image for storjlabs/storagenode@sha256:38f07c8d48cef611ae3353e2985f367fd14928f31ab49409c8acfd9aba6a2007
docker.io/storjlabs/storagenode@sha256:38f07c8d48cef611ae3353e2985f367fd14928f31ab49409c8acfd9aba6a2007

arm64/v8
https://hub.docker.com/layers/storjlabs/storagenode/latest/images/sha256-1ccbcb844d1ea5c3e2d59c51f3aafc5122380b050989713cbcef9fda171a20c1?context=explore

$ docker pull storjlabs/storagenode@sha256:1ccbcb844d1ea5c3e2d59c51f3aafc5122380b050989713cbcef9fda171a20c1
sha256:1ccbcb844d1ea5c3e2d59c51f3aafc5122380b050989713cbcef9fda171a20c1: Pulling from storjlabs/storagenode
Digest: sha256:1ccbcb844d1ea5c3e2d59c51f3aafc5122380b050989713cbcef9fda171a20c1
Status: Downloaded newer image for storjlabs/storagenode@sha256:1ccbcb844d1ea5c3e2d59c51f3aafc5122380b050989713cbcef9fda171a20c1
docker.io/storjlabs/storagenode@sha256:1ccbcb844d1ea5c3e2d59c51f3aafc5122380b050989713cbcef9fda171a20c1

arm/v6
https://hub.docker.com/layers/storjlabs/storagenode/latest/images/sha256-2dc1f64694bae64c642230c97ebe3fe172fe878a91b1210dfb2975c8b85ad13d?context=explore

$ docker pull storjlabs/storagenode@sha256:2dc1f64694bae64c642230c97ebe3fe172fe878a91b1210dfb2975c8b85ad13d
sha256:2dc1f64694bae64c642230c97ebe3fe172fe878a91b1210dfb2975c8b85ad13d: Pulling from storjlabs/storagenode
bb87125c6ee1: Pull complete
af418c858467: Pull complete
13ce0856c28d: Pull complete
8edd74dced91: Pull complete
50112edebc6f: Pull complete
5165a019078d: Pull complete
Digest: sha256:2dc1f64694bae64c642230c97ebe3fe172fe878a91b1210dfb2975c8b85ad13d
Status: Downloaded newer image for storjlabs/storagenode@sha256:2dc1f64694bae64c642230c97ebe3fe172fe878a91b1210dfb2975c8b85ad13d
docker.io/storjlabs/storagenode@sha256:2dc1f64694bae64c642230c97ebe3fe172fe878a91b1210dfb2975c8b85ad13d

$ docker image ls storjlabs/storagenode --digests
REPOSITORY              TAG                                  DIGEST                                                                    IMAGE ID            CREATED             SIZE
storjlabs/storagenode   latest                               sha256:147a226e8a20eb99837592c0b792bc2c15eb3447d517353c79907865a7f8f6fc   2e3edc89250f        2 weeks ago         34.4MB
storjlabs/storagenode   latest                               sha256:1ccbcb844d1ea5c3e2d59c51f3aafc5122380b050989713cbcef9fda171a20c1   2e3edc89250f        2 weeks ago         34.4MB
storjlabs/storagenode   <none>                               sha256:2dc1f64694bae64c642230c97ebe3fe172fe878a91b1210dfb2975c8b85ad13d   3dedbe26fc08        2 weeks ago         33.9MB
storjlabs/storagenode   <none>                               sha256:38f07c8d48cef611ae3353e2985f367fd14928f31ab49409c8acfd9aba6a2007   2f2cc0228ec1        2 weeks ago         39.4MB

As you can see the latest for aarch64 platform has the same Image ID (but different digests, even if size and data are the same)

$ docker pull storjlabs/storagenode:a760b8bc4-v1.26.3-go1.15.7-arm64v8
a760b8bc4-v1.26.3-go1.15.7-arm64v8: Pulling from storjlabs/storagenode
Digest: sha256:1ccbcb844d1ea5c3e2d59c51f3aafc5122380b050989713cbcef9fda171a20c1
Status: Downloaded newer image for storjlabs/storagenode:a760b8bc4-v1.26.3-go1.15.7-arm64v8
docker.io/storjlabs/storagenode:a760b8bc4-v1.26.3-go1.15.7-arm64v8

$ docker image ls storjlabs/storagenode --digests
REPOSITORY              TAG                                  DIGEST                                                                    IMAGE ID            CREATED             SIZE
storjlabs/storagenode   a760b8bc4-v1.26.3-go1.15.7-arm64v8   sha256:147a226e8a20eb99837592c0b792bc2c15eb3447d517353c79907865a7f8f6fc   2e3edc89250f        2 weeks ago         34.4MB
storjlabs/storagenode   a760b8bc4-v1.26.3-go1.15.7-arm64v8   sha256:1ccbcb844d1ea5c3e2d59c51f3aafc5122380b050989713cbcef9fda171a20c1   2e3edc89250f        2 weeks ago         34.4MB
storjlabs/storagenode   latest                               sha256:147a226e8a20eb99837592c0b792bc2c15eb3447d517353c79907865a7f8f6fc   2e3edc89250f        2 weeks ago         34.4MB
storjlabs/storagenode   latest                               sha256:1ccbcb844d1ea5c3e2d59c51f3aafc5122380b050989713cbcef9fda171a20c1   2e3edc89250f        2 weeks ago         34.4MB
storjlabs/storagenode   <none>                               sha256:2dc1f64694bae64c642230c97ebe3fe172fe878a91b1210dfb2975c8b85ad13d   3dedbe26fc08        2 weeks ago         33.9MB
storjlabs/storagenode   <none>                               sha256:38f07c8d48cef611ae3353e2985f367fd14928f31ab49409c8acfd9aba6a2007   2f2cc0228ec1        2 weeks ago         39.4MB