Is it normal watchtower to autoupdate my docker nodes to pre-release versions? I know that in the run command of storagenode and watchtower there is a “latest” tag. I see on github that ver. 1.66.1 is latest. Than why part of my nodes are at ver. 1.67.1 and 1.67.3, that are tagged “pre-release”? Is there a way to opt-out from pre-releases?
Those versions are just in regular rollout to production at the moment. Version 1.67.3 is a bugfix version that contains a fix for wrong disk usage displayed on the payout page. It started rolling out to nodes that got 1.67.1 first. Once all nodes are updated, the latest version will update as well.
These aren’t pre-release versions. Don’t worry.
2 Likes
The current recommended version is reported via https://version.storj.io and right now it indicates 1.67.3 is the suggested current release.
1 Like
The same question came up to me recently. I confirm that is misleading.
If a newer version is in rollout to production, it should be labled as “latest”. No? “latest” is not the same as (100%) “released”.
[sn1] : hdd 58.28%
.. new version : 1.67.3 > 1.66.1
My script is checking the latest releases, too - and the corresponding alert looked strange. That’s why I became aware of that “strange behavior”. For me, it’s not obvious. 
Yes, the mathematical expression is correct above, but it should be interpreted as:
.. new version : [currently installed] > [available on github]
… just to avoid comments on that. haha
Is it normal watchtower to autoupdate my docker nodes to pre-release versions? I know that in the run command of storagenode and watchtower there is a “latest” tag
There are 2 updates, the Storagenode application, and the Docker image.
Watchtower updates the Docker image and, as you mention, it updates when the latest label changes.
A while back, we decided to integrate our auto-updater application inside the Docker image to follow our release process which is controlled by what it’s published in https://version.storj.io. So the auto-updater inside the Docker container updates the Storagenode binary when your Node ID matches with the mask and cursor.
Once we publish the 100% cursor (which involves setting the minimum version field to the same one as the suggested version field), we update the latest tag to the Docker image that matches the release that we have rolled out. From that point, watchtower will do the Docker image update.
The only changes that we do in the Storagenode Docker image are updating the latest Debian packages unless we find some bug in the Docker image itself, however, it has been quite stable since we provided them with the auto-update and we debugged (and fixed things) during the first weeks.
If a newer version is in rollout to production, it should be labeled as “latest”. No? “latest” is not the same as (100%) “released”.
Ideally yes, but if we do that then Watchtower would update all the Storagenode that run on Docker.
The main issue is that we cannot control when Watchtower will update them, so it screws up our rollout process.
Why do we need to rollout the version over a period of time and we don’t do it one shot?
Because of 2 main reasons:
- We can see if the new Storagenode version is causing any issues and if it’s we can go and fix it but without having impacted all the Storagenodes. Our community, you, help a lot in spotting them if they have a few hours since they get an update, THANK YOU to all of you!
- We can control the amount of Storagenodes that are going to be offline at a certain point in time because they are updating.
3 Likes
I’m not following some of the things you mentioned @ifraixedes . As far as I’m aware the docker image only contains a script that first downloads the most recent updater and then the updater downloads the correct storagenode version based on the cursor. So neither of these binaries are included in the image and it doesn’t really matter when the image is updated, because if you update it early, it would still download the old storagenode version for nodes that don’t match the cursor yet. (although currently it might downgrade some nodes from 1.67.1 to 1.66.1 due to the bugfix version starting a new rollout.)
Watchtower is not working since months I guess. There are a couple of posts on that in the forum. It stopped working well when these light release came alive.
Yes, that’s how it works. Saying that the auto-updater was inside of the Docker image was not the best way to concretely say how it works, which is how you described. Mostly I was referring to that the container cares itself to have the last version of the binaries following our update rules.
It doesn’t matter in terms of the Storj binaries, the only difference is that it may contain updates on the Debian packages.
Not sure, if you read that in the forum, then it’s probably what you read. I’m using Docker, but I removed Watchtower since we released this new Docker image that they get the binaries when the container runs.
I’m going to ask internally about why we are not tagging to latest the images since we start the rollout if Watchtower doesn’t work anymore because as far as I know that was the only reason.
1 Like
After checking it internally, we don’t publish the latest tag when we start the rollout of a new release is because of tooling like Watchtower.
It’s what I said, but not just because of “Watchtower” alone.
Anyway, we followed what we did before we had this Docker images that update the binaries themselves, and only the non Docker storagenodes were receiving updates through the auto-updater, and we haven’t thought that much if we should update or not (event remove) the latest tag since we got these new Docker images.
I’m going to comment about this in our next general engineering meeting.
3 Likes
If possible I would refrain from updating the Docker image unless it’s necessary. It triggers the file walker unnecessarily which causes a lot of unneeded IO, especially for people who run multiple nodes, in which case it triggers it for all nodes at the same time. Really not ideal.
2 Likes
I guess that SNOs that know how the Docker image works and have issues with the file walker, they are not updating the Docker image on each release.
However, they shouldn’t know how it internally works and publishing on each release is misleading.
Yeah, but what if there is an important update to the image and half of the node operators have watchtower off to prevent running the file walker. I like to stay up to date, so I’m leaving it on as likely most SNOs will. Seeing how it works now it seems logical to remove the Docker image from the regular release schedule and just do individual updates as needed.
1 Like
I’m very confused… I run storagenodes (2 per machine) in Docker, on Synology Diskstations. I also run the watchtower (see my other posts) on all machines, like the storj docs sais. You say that the storagenode image in Docker is updating itsef and we don’t need the watchtower anymore? We can “docker rm watchtower” on all Docker nodes?
Given that recently there aren’t many changes to storage nodes, maybe you’d be satisfied with Storj skipping some node updates and instead scheduling docker image updates?
Docker image update is not the same thing as storagenode update? 
I realy don’t get this linux way of doing things…
The proper Linux way would be to provide apt/rpm package. 
But these mechanisms do not provide rolling updates.
1 Like
I said no such thing. I’m pretty sure I said I would keep using watchtower. And i pointed out a problem if people stopped using watchtower. I’m not sure where you got this conclusion from.
That’s the exact opposite of what I’m advocating for. Storage node updates contain meaningful changes and they have a staggered rollout so the file walker hits one node at a time. The Docker image updates do not have any meaningful changes for storage nodes (if any at all). I want my nodes to stay up to date, I just want to get rid of the needless restarts because there is a new Docker image with effectively no changes. Simply because it’s part of the release pipeline.
On the contrary, they may come with security bugfixes for the container itself. Remember it’s a full Debian container.
Sorry, I realy don’t understand how it works. I will keep watchtower on and l will get my hands off it. When it updates, it updates…
Funny fact: one node was running 1.67.1. I stopped and rm it. I ran the manual update commands. Restarted and now I have 1.66.1 
I’m so confused… but I won’t bother with this anymore.
you can be suspended if you will not update in next version, there is some restrictions as i remember