So in New Zealand here, we just had daylight savings time adjustment over the weekend, was NZ Daylight Time, now back to Standard time (1 hour back).
One of my nodes PC failed to update the time via windows.
Why is this such a cataclysmic event for Storj?
I awaken to multiple emails saying my node is suspended for “Bad Data”, wasted 15 minutes of my life thinking something serious has gone wrong with Storj, Storj would not start, executable kept dying when restarting the storj service.
Inspect the log to find the error message preflight:localtime system clock is out of sync with satellite
Check the time on the PC, find its still on NZDT, update the time, all good.
It’s not a particularly user friendly way of dealing with such a trivial issue…
The correct error was reported in the log, why not simply stick a note on the dashboard, something like…
"Your node is currently suspended: It appears your system time is incorrect. Please correct the system time immediately. "
I think if you look at the messages around that one…you’ll see it couldn’t contact the satellite at all, about anything. I’ve never seen an actual time issue: it was always a connectivity issue.
The extended messages usually make it clear that it couldn’t connect: but if it could have connected it would have asked about the time.
I have four nodes, this is the only one where the executable had died and continued to die on startup, presumably because the system time had not updated overnight.
After I corrected the time, storj would start and has been running ever since with no more errors of the kind.
So you are correct it couldn’t contact the server but that was because it wasn’t running at all.
Having said that it must be reporting something to the satellite otherwise it wouldn’t have information to suspend the node…
I just feel like the error reported by the dashboard could be a bit more helpful under the circumstances
The other errors
2025-04-07T09:25:15+12:00 ERROR preflight:localtime system clock is out of sync with satellite {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "error": "system clock is out of sync: clock off by 59.985382 minutes", "errorVerbose": "system clock is out of sync: clock off by 59.985382 minutes\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).checkSatelliteTime:132\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check.func1:74\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2025-04-07T09:25:15+12:00 ERROR Failed preflight check. {"error": "system clock is out of sync: system clock is out of sync with all trusted satellites", "errorVerbose": "system clock is out of sync: system clock is out of sync with all trusted satellites\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check:96\n\tstorj.io/storj/storagenode.(*Peer).Run:1089\n\tmain.cmdRun:127\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2025-04-07T09:25:15+12:00 ERROR failure during run {"error": "system clock is out of sync: system clock is out of sync with all trusted satellites", "errorVerbose": "system clock is out of sync: system clock is out of sync with all trusted satellites\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check:96\n\tstorj.io/storj/storagenode.(*Peer).Run:1089\n\tmain.cmdRun:127\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2025-04-07T09:25:15+12:00 FATAL Unrecoverable error {"error": "system clock is out of sync: system clock is out of sync with all trusted satellites", "errorVerbose": "system clock is out of sync: system clock is out of sync with all trusted satellites\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check:96\n\tstorj.io/storj/storagenode.(*Peer).Run:1089\n\tmain.cmdRun:127\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
Its generalized error reporting. So the idea is, you see the error then look at the logs to diagnose the problem and come up with a solution.
In order to elaborate a little further on generalized error reporting, please look at the example below.
Enter your name
– User enters 123. Error message shown is, “This is not a valid name. Please try again.”
A helpful error message is, “Please enter alphabets instead of numbers.”
– User enters !@#$%^. Error message shown is, “This is not a valid name. Please try again.”
A helpful error message is, “Special characters are not allowed. Please try again.”
Technically, these user inputs are special cases where a specific error message needs to be shown. In order to incorporate all such inputs, a generalized message is shown which then takes care of any special cases the devs haven’t come across during testing or future bugs.
^ This. To be more precise it is the payment for download traffic that is at risk. The node needs to batch all the paychecks aka orders and can submit only one batch per hour. Once a batch is submitted it can’t reopen the case. So if the uplink sends an order late the node has to decide to reject it or provide the service for free. → Systemclock needs to get corrected to make sure the node can identify and reject bad uplinks.
That sounds like a misconfiguration. Are you running docker? Depending on how it is configured docker might not be aware of your timezone. Try to pass the host timezone to docker and the problem should go away.
not docker just a windows install. It is configured correctly date/time wise, for some reason though it didn’t update the time for end of daylight savings.
I understand that correct system time is required for correct operation. What I don’t understand is why I have to go visit a forum and pour through system logs for such a stupidly basic error.
The node understands what happened, it was reported in the log - why not report the error on the dashboard and save node operator 10 minutes of figuring out why the stupid thing suddenly stopped working and got itself suspended?
Even a link on the dashboard to open the log or show the last 10 entries in the log on the dashboard would have been been far more useful. Instead the shown error message sends you to this forum…
I can repeat it one more time. Something is wrong with your configuration. In general windows does support timezones and it doesn’t need a timesync just for that. My windows clock switches between UTC+1 and UTC+2 just fine without me having to touch the system clock.
it just not possible to make a message on dashboard on every error. we will have a madness there. if you have problem on node, you have to be capable to search for problem as you as SNO is responsible for your side of work of node. SNO are not buy software from storj to run node, so it is made as it is. We can make offers for changing something, but then you need to make looks like it should be, like feature request with some description and voting, that storj see that it wanted feature or not.
not offensive at all, I believe we are all here trying to make storj work.
The node doesn’t have to catch all errors to the point of ‘madness’.
Conceivably it could be the top 10 recurring errors (of which an errant timezone change may or may not be one) - most software systems usually have some form of error messaging with a catchall of sorts for everything else. I guess i’m just wondering why storj does not?
Surely more meaningful error messages are helpful and make the day to day running of a storage node just a little bit better and easier for node operators? or is there some big picture reason for a lack of meaningful error messages that eludes me…
You may set the loglevel to error or fatal, then you will have only errors or fatal errors.
Showing errors from logs on the dashboard is not planned. However, we are glad to accept a Community contribution - please submit a PR on our GitHub and our developers would be happy to review and accept it.
