Issues downloading from GitHub

Hi,

This post is not really an argument, rather than for letting the Storj team know about some issues faced from time to time.

In my case, some of my nodes are suffering from time to time of issues connecting to GitHub to fetch the latest release.

In happens less than 5% of the times (I’d say) but when it happens, it freezes like this:

--2022-07-23 12:40:35--  https://github.com/storj/storj/releases/download/v1.58.2/storagenode-updater_linux_amd64.zip
Resolving github.com (github.com)... 140.82.121.4
Connecting to github.com (github.com)|140.82.121.4|:443... connected.

It is always solved by manually restarting the docker container, but it requires manual intervention which is not nice when the autoupdater is executed automatically and I also have the docker container to auto restart itself in case of a crash/kill (which never happened - yet).

Other time, I experience very slow downloads speeds from GitHub (~30KB/s using a 1Gbps connection), which is also not the optimal situation.

Some related issues in the forum:

Could it be changed to use a more reliable source? Like downloading it from Storj DCS for example :sunglasses:

Another cool option could be to allow the Docker to configure when (and how? maybe manully) we want to run the autoupdater! It would be nice to allow us to stop the cron task and just manually run it at certain time.

it is just a suggestion, I’m more than happy with the current state and the project.
Storj is awesome!

2 Likes

There is no cron, we rolling out updates by waves, so you actually cannot choose the time when your node will be updated. It will be updated, when its time to come (waves allowing update nodes based on their NodeID).

Regarding the freeze you may submit an issue on our GitHub: Issues · storj/storj · GitHub
I think there should be a timeout

Yes and no. Nodes are updated by waves, but the process in charge of that (storagenode-updater) is executed periodically on the Docker container (see here).

My point is: why not letting us choose if we want to execute it automatically or not (by default, automatically) and let us manage update checks and updates by our own?

Imagine that my server has some load during the day, and the filewalker smashes it when the node restarts. It may be sensible to schedule the execution of the updater everyday at midnight, so I can check if there is an available update for my node and apply it in a “safe” hour.

I can always update the Docker image myself, but it felt like something useful for SNOs with IT knowledge.

Regards!

While this is not something officially supported by Storj, you don’t need to run Storj updater. You can just fetch the storage node binary and run it explicitly, even outside of Docker if you want. This is the script that currently manages the updating process.

Storj wants uncorrelated restarts. By eye-balling the map it seems that a significant chunk of nodes is hosted in a single time zone. If suddenly most of them were to disappear for a few minutes because of an upgrade, integrity of the whole network would be at risk.

There are ways to lessen the impact of the file walker process, for example by choosing the right file system. I wish this was documented in the official manual…

1 Like

To do not shutdown the part of the network from the same time zone in the same time, @Toyoo is right.
Using NodeID to determine nodes allows us to have uncorrelated restarts during upgrade.

5 posts were merged into an existing topic: Disk fragmentation is inevitable… Do we need to prepare?

A post was merged into an existing topic: Disk fragmentation is inevitable… Do we need to prepare?