Stop the deployment of update 1.97.2 immediately

Does running storagenode help or storagenode info from command-line work?

I rolled back the updated nodes to version 1.96.6 so that they work and do not crash online. Also prohibited updating on them.
Currently, when the update is enabled, they are not updated.
Thank you, apparently the server has paused the update.
So I don’t have version 1.97.2 right now and can’t check if it works storagenode help or storagenode info from command-line.

__github.com/storj/storj/releases/download/v1.97.2/storagenode_windows_amd64.zip
I manually downloaded the file and tried to run it

Apparently the new version did not read the paths from the settings file config.yaml ?
Do you need additional keys to execute the command?

storagenode help - runs fine

No, that’s sufficient for that test. I wanted to make the first check for whether there’s any init time, DLL related or anti-virus related issues – in which case the binary wouldn’t start at all.

I’m still digging into all the differences between the versions, but I haven’t yet found anything that would explain the issue.

Storagenode version 1.98.2 (Something wrong in my algorythm or satellite. Node version greater than latest 1.97.2)

I have the version 1.98.2 installed on one node and it works

There is v1.98.2 Release v1.98.2 Ā· storj/storj Ā· GitHub, the ā€œlatestā€ in github is somewhat misleading.

7 posts were split to a new topic: Virtualization Discussion

A post was merged into an existing topic: Virtualization Discussion

So I’ve been digging in the code and have a question:

Do you happen to have any custom scripting or monitoring that calls Storage Node API-s directly? If yes, what and how are you calling them?

Additionally, did you look into Windows Event Viewer logs, whether there was any mention about storagenode.exe?

PS: the only thing I currently found at the moment was that calling API ā€œ/payout-history/[date]ā€, with an invalid [date] could cause a failure.

1 Like

I did not see any entries in the log; as I understand it, the application crashed before any logs could be written.
There is a simple option - I can watch with you via anydesk how the node crashes.
You just need to write to Telegram and make an appointment.
Don’t waste your time playing with a broken phone.

Yes, that’s sounds like a good plan.

I was initially hoping it was something more obvious, because there hadn’t been that many changes between those versions.

yes, @AiS1972 , if node crashes there is a record there in Event Viewer, under ā€œWindows Logs>Applicationsā€ tab. Every time my storagenode.exe crashes there is a record there, might be some hints for us why!

v1.95.1 running great!

Unfortunately the windows event viewer doesn’t have that much information at the moment. However, I’ve been spending time improving that yesterday – so some upcoming release should log information there when the logging system isn’t yet setup.

4 Likes

7 posts were split to a new topic: Node offline after update

I would recommend using the last release that you were using, for the other nodes that didn’t start properly.

We’re in the progress of making a new v1.97.3 release that should give more information in the log or windows event viewer. But, it’ll take some time to build, run the full testsuite and make the release.

3 Likes

There finally exists version v1.97.3 that should be able to capture the startup failure.

If the failure happens before the service is fully operational, it will try to log to event viewer. If that also fails it outputs it into stdout – which will be only visible when starting the service from command line as the administrator with the same arguments as the service.

Hopefully this will help us figure out what’s causing the issue.

2 Likes

@AiS1972 have you had a chance to test v1.97.3, or does the issue not happen with that version any more?

Most nodes running on windows encounter the same problem.
The node that was automatically updated to version 1.96.6 failed to start. When it was returned to 1.95.1, everything was normal.
Part of the upgrade to 1.96.6 was running normally, but it stopped working when it was updated to 1.97.3 again. Manually roll back to the previous version and it will run normally.
More than ten of my nodes have the same problem, and I have deleted the upgrade program.
Because the program stops immediately after being started manually, no new error information is written to the log file.

May it be the problem with accidentialy doubled config entrys in the yaml, causing nodes not to start?

1 Like