Current rollout may lead to unexpected downgrades


I have been observing the rollouts of the latest versions, and I believe the current pattern may result in unexpected downgrades for some nodes in some scenarios (at the very least using docker).

Note that while this topic is similar to “Are automatic node downgrades expected?”, the purpose is different therefore I thought it might be beneficial to make a new topic.

The pattern that I have observed is the following:

  • Last rollout was for versions 1.99.3 (min) to 1.101.3 (suggested), which completed without any issues and the cursor reached FFFFF… .

  • Current rollout has a minimum of 1.99.3 and suggested of 1.102.3, and currently has a cursor of 19999… .

  • If a container running the storagenode software is recreated (not restarted) for any reason, then by default the docker image will fetch the appropriate version. As the rollout for 1.102.3 has not been completed, there is a high chance that recreating any nodes will fetch 1.99.3 instead of 1.102.3, therefore resulting in a downgrade from 1.101.3 → 1.99.3.

  • This is not an issue for restarting nodes, as they should not explicitly downgrade if the cursor hasn’t reached them, this is an issue for recreating/reinstalling the node software (at least in docker as you do not select any specific version) as they might fetch and run the minimum version resulting in a downgrade from the version the node was running on previously.

This behaviour is something that I have experienced previously, as due to my current setup having to delete and recreate containers might be necessary during maintenance.
Likewise this issue can be avoided by SNOs by not recreating nodes mid rollout.

I wanted to bring this up as from what I understand downgrades of storagenode versions can be problematic as it is something that is not usually tested, and possibly even more so with 1.99.3 and 1.101.3 as they brought changes to the trash and garbage mechanisms.

To avoid this kind of situation, I think it might be beneficial to ensure that the minimum version of any rollout is the suggested (or target) version of the previous rollout. While Im not sure if this is something that could be changed right now, it could be useful for future rollouts (with the exception of planned downgrades due to issues found with new versions).

Apologies if I made incorrect assumptions in some part of my explanation, or if the whole premise might be wrong due to something I missed, Im simply trying to collaborate with the team with something I believe could be helpful.

Thank you for reading.

1 Like

I have read this topic as I thoguht it might be useful to create a new topic talking specifically about one cause of these unexpected downgrades caused by the min/suggested versions used in rollouts.

The topic you mention was already marked as solved and discusses other issues, specifically downgrades caused due to missing cursors in the rollouts.

If it would be better, Im happy to delete this one and post it in the one you link.

Hello @pasatmalo,
Welcome to the forum!

Passed this info to the team.

1 Like

Hello, two of my nodes downgraded after recreation from 1.10x.3 to 1.99.3. Need any action from my side?

Same here.

When will the minimum version be upgraded?

This has been resolved and min version is now 1.101.3


After downgrade my node successfully processed gc for US1 and make trash folder in older format.

Need action from my side to resolve issue?

can’t be,
i see one of my node is still at 1.99.3
and getting ingress., please restart your node if you would like to expedite the upgrade. Min version and what can receive ingress are different things.

1 Like

It appears that the new rollout faces the same issue once again.
Current minimum version is 1.101.3. Last rollout finished with 1.102.3.

Nodes recreated in docker will lead to a downgrade from 1.102.3 → 1.101.3.

Sadly as I had to do some server maintenance and didnt notice the new rollout, all my nodes have downgraded already (my fault for not checking I guess), but at least (so far) they seem to be running fine.

Would it be possible to update the minimum version to 1.102.3 to avoid these downgrades from happening?

Update: The Storj team has taken care of this issue and update the minimum version to 1.102.3.