Always restart on arm64 docker image since recent days

Hi help,
My node has been working well for one and half years, but it seems not working- always restarting in recent days.

Platform: arm64v8, Docker version: 18.09.7

I believe docker image for arm64v8 is incorrect (out of dated) after using commands “docker pull storjlabs/storagenode:latest”
storjlabs/storagenode latest efb7528d1df3 4 months ago 29.5MB

Please can you point out correct docker image tag (":latest" seems incorrect) for arm64v8 platform.

Larry Zheng

Welcome to the forum @zxz_27 !

Show last 20 lines from the log.

“validate db version mismatch error: expected 45 != 48”.

I believe the root cause that is storjlabs/watchtower is not working properly on latest arm64v8 image update, did you test latest release on arm64v8 HW platform?
Please point out a correct docker image tag (not latest) for arm64v8 for manual update.

Does this help?

Your attached info is what I requested years ago and got support for both arm64v8 and arm32v7 releases.

I missed my previous account password and then created new one for this support request.

I believe your team has done Jenkins daily auto build for AMD64, arm64v8 and arm32v7 images , please ask your release QA and just point out which release (docker image tag) is suitable for arm64v8 platform.

Larry Zheng

AMD64 release
storjlabs/storagenode latest d296e0336b05 4 weeks ago 37.7MB

arm64 release
storjlabs/storagenode latest efb7528d1df3 4 months ago 29.5MB

I compare two releases for x86 and arm platform, so it sounds the arm64 release is 3-months out-of-dated, please tentatively point out an image tag for arm64 (or arm32 is fine) for my node recovery, meanwhile help chase release issue on arm platform.

@ifraixedes, @nerdatwork tag of latest for arm64 is 3-months out-of-dated, while tag of arm is not retrievable, could you help?

You need to pull and use storjlabs/storagenode:latest
Please, try it and re-create a container

tag :latest is surly not working on arm64, I eventually downloaded all tags, and the issue got resolved on tag: arm release.

You can see all releases from above snapshot for arm64 platform

Ok. Please, prune all images and pull the only :latest

yes, I will manually create tag :latest with tag:arm image ID, I assume you will help relay this arm64 issue to your release team

You are the only one who have the problem since the switch to the latest alpine for arm64 a few months ago.
So, please, help to troubleshoot the problem.
I kindly ask you to stop and remove the container:

docker stop -t 300 storagenode
docker rm storagenode

Remove all stopped containers

docker container prune

Confirm remove. And prune all images:

docker image prune

Confirm removing.
It should remove all images and layers, include dangling ones.
Then try to pull the storjlabs/storagenode:latest and show results.

I did “docker image rm” all storjlabs/storagenode images after stop and rm container when the issue surfaced on arm64.
However pulling latest tag for arm64 which is 4-months aged seems not working (restarting every 30s), until I pull all tags and notice tag: arm is most recently updated.

Did your QA verify on arm64 platform upon tag latest?

AMD64 images are all the same
storjlabs/storagenode alpha d296e0336b05 4 weeks ago 37.7MB
storjlabs/storagenode arm d296e0336b05 4 weeks ago 37.7MB
storjlabs/storagenode beta d296e0336b05 4 weeks ago 37.7MB
storjlabs/storagenode latest d296e0336b05 4 weeks ago 37.7MB

ARM64 images are different
storjlabs/storagenode arm 5e64bc3c759a 4 weeks ago 33MB
storjlabs/storagenode latest 5e64bc3c759a 4 weeks ago (manually synched to tag :arm, otherwise 4 months ago) 33MB
storjlabs/storagenode beta 66ea0edabd14 3 months ago 30.8MB
storjlabs/storagenode alpha fdf8af35684c 16 months ago 22.1MB

How you still have images like alpha and beta, if you did pruning today?

Please, understand, we have a lot of users with ARM64, but only your system has this problem, otherwise we would have other users with the same problem here on the forum and have similar support tickets.

I would try to help to fix the problem in your specific case. Maybe there is a bug in our publish pipeline, but first we need to exclude problems on your system.
For that I need that you follow recommendations, even if you think they are useless or stupid.
If we able to fix your configuration, you would not have this problem anymore.
Please, do:

  1. Stop and remove the storagenode container
  2. docker system prune
  3. docker pull storjlabs/storagenode:latest
  4. Try to run your storagenode back
  5. Post results

Hi Alexey,

If you take a look at the snapshot in my 1st message, there is no Alpha/Beta/arm tags but only latest tag with 4 months aged for arm64.

I was doing your QA’s job by removing all tags, then pull and compare all AMD64 and arm64 (snapshot in my last reply), why not check your release engineer or QA to pull latest tag for arm64 to confirm the bug?

BR//Larry Zheng

Because we do not have your setup, which does not work as expected.

All available arm64 systems pull the right image.
I would try to check it on emulator though.

Could you please copy info here?

docker info