I am running windows and having problems with my node uptime getting reset - it provides an error in the event logs:
“The Storj V3 Storage Node service terminated unexpectedly.”
I am getting no response when connecting ot http://localhost:14002/ - which was working fine before I updated to a new storagenode version - I tried restoring back to older version - but still nothning.
No connection very likely means that the service was unable to start properly.
The cause of that can be multiple different things, but you should check what the logs says. The most recent error-messages will be at the bottom of the file.
For Windows, its located here:
“C:/Program Files/Storj/Storage Node/storagenode.log”
\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
That is the last line in the file. I have no idea what that means.
This seems to be the entire error entry from the log after looking a bit closer…
I am getting the following log errpr - is this a fixable issue? How to do so?
024-07-12T15:42:40-04:00 FATAL Unrecoverable error {“error”: “Error checking version for storagenode database: validate db version mismatch: expected 26, but current version is 59\n\tstorj.io/storj/private/migrate.(*Migration).ValidateVersions:141\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).CheckVersion:671\n\tmain.cmdRun:103\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”, “errorVerbose”: “Error checking version for storagenode database: validate db version mismatch: expected 26, but current version is 59\n\tstorj.io/storj/private/migrate.(*Migration).ValidateVersions:141\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).CheckVersion:671\n\tmain.cmdRun:103\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78\n\tmain.cmdRun:105\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}
This means that because you tried using an older version, it can no longer use the db (if I recall correctly). You’ll have to use the most recent version, try to start it and then view the log on the type of error you recieve.
1 Like
An older version of what? The most recent version of what?
This. You need to reverse that. You’ll find the most recent version - as downloaded by the updater - in the same folder where your stoagenode.log is.
1 Like
The most recent version is not in that folder - only version 1.104.5.
Do you see something like “old” “1.104.5”? The most recent version will not have a version in its name, thats the one you are looking for.
I do, yes (I see all the old and the storagenode.exe - which is the current version already. Still no access to the dashboard and log error
I have no experience in manully installing a particular version, since the updater always took care of that. So, someone else has to help out here.
Anyway, I think that you reverting to an old version created a second problem (the db version issue). That needs to be resolved first, before the real issue can be seen or even addressed.
EDIT: one final thought (untested), you might be able to download the latest version manullay from: below URL. Copy it to the folder where hte log is located and restart the service (or simply reboot your computer). Then check the log again for any errors you might find.
URL: https://github.com/storj/storj/releases/download/v1.108.3/storagenode_windows_amd64.zip
I have done that (that is the way to manually get to a version).
Same issue persists.
Running now 1.105.4 and also same issue when I had 1.108.3
I am also getting (not sure if I was before) the following:
The Storj V3 Storage Node service terminated unexpectedly. It has done this 1 time(s). The following corrective action will be taken in 0 milliseconds: Restart the service. It happens repeatedly / continually
From log -
| 2024-07-12T17:26:28-04:00 | ERROR | failure during run | {error: Error checking version for storagenode database: validate db version mismatch: expected 26, but current version is 59\n\tstorj.io/storj/private/migrate.(*Migration).ValidateVersions:141\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).CheckVersion:671\n\tmain.cmdRun:103\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78, errorVerbose: Error checking version for storagenode database: validate db version mismatch: expected 26, but current version is 59\n\tstorj.io/storj/private/migrate.(*Migration).ValidateVersions:141\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).CheckVersion:671\n\tmain.cmdRun:103\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78\n\tmain.cmdRun:105\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78} |
|---|---|---|---|
| 2024-07-12T17:26:28-04:00 | FATAL | Unrecoverable error | {error: Error checking version for storagenode database: validate db version mismatch: expected 26, but current version is 59\n\tstorj.io/storj/private/migrate.(*Migration).ValidateVersions:141\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).CheckVersion:671\n\tmain.cmdRun:103\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78, errorVerbose: Error checking version for storagenode database: validate db version mismatch: expected 26, but current version is 59\n\tstorj.io/storj/private/migrate.(*Migration).ValidateVersions:141\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).CheckVersion:671\n\tmain.cmdRun:103\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78\n\tmain.cmdRun:105\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78} |
Seems to keep reference version 59 vs 26 - no clue what that means.
And to the mod that restored my post - thank you - it would be nice to have my restriction of posts removed as a new user - especially when I am trying to get help with a non functioning node.
It seems like your database was converted to a newer version. So now you can not go back to older version of storagenode.exe. This is how I would cleanup that situation:
- Stop storagenode and storagenode updater in services panel.
- Replace both executables with newest versions (1.108.3) from github zip-files.
- Delete all databases (*.db files). Storagenode will create new databases on next start. You will lose some history in dashboard but who cares…
- Start storagenode and storagenode updater in services panel.
1 Like
Still same issue after your suggestions.
Following happening with the service…
The Storj V3 Storage Node service terminated unexpectedly. It has done this 1 time(s). The following corrective action will be taken in 0 milliseconds: Restart the service.
If your node recreated the deleted databases using V1.108.3 and then the auto updater brings the node back down to a lower version such as 1.105.4(auto updater doesn’t like it when people upgrade ahead of schedule and will sometimes downgrade people, at least on docker installs. I’m not sure how storj on windows handles updates), I think you’ll be back where you started with databases that don’t match the version you are currently running.
I’d try removing 1.108.3, deleting the databases and then install and run 1.105.4. Report back and post the log output if there are issues.
It did not recreate the databases.
It is currently with 1.108.3.
Right now, it also has now log file (I had deleted it as it was filling up with the error posted earlier).
Restarting system now.
I had a similar issue a few days ago when I set up a node on the test network and it kept downgrading itself to an incompatible version(it ignores the custom version url in my config file and uses the default url instead for some reason, but this was on linux/docker) Mine also did not re-create the databases which I deleted. I don’t know why. The log complained that they were all missing but it did not regenerate them. In the end,I basically backed up my blobs folder, reinstalled the storagnode software from scratch(of course I did not create a new identity, I reused my existing one. keep a backup of that some place safe), and then moved the blobs folder back to where it belongs. I’m not saying you should do this though. If your node starts up and goes online somehow while the blobs folder (customer data pieces) is missing, you can get disqualified in a matter of minutes so you need to be careful.
I just keep getting the storage node stopping unexpectedly and then trying to restart… the cycle continues.
not creating the .db files
no recreation of the log file
Not sure I would consider doing that - as I’m into substantial TB for that.
I can’t even tell my status of dq at this point.
You may need to wait for expert advise. Assuming you started with an online status of near 100%, I think you can be offline for well over a week before you get DQ for simply being offline, so you’ve got time to figure out this issue.
Edit. I’m playing around with the windows version of storagenode to see what I can learn.