Clock error on frenass

[Chelle]

Hi, new here. I’m trying to install storj in frenass/trunass jail.
but getting clock error.
The clock seems right though?

         /usr/local/go/src/runtime/proc.go:225
2021-05-05T19:05:37.976+0100    INFO    operator/operator.go:56 Operator wallet {"Address": "0x1Ea6C5981C1Dd6090d7A0D1c099B17b47B27affB"}
2021-05-05T19:05:38.578+0100    INFO    db.migration    migrate/versions.go:218 Database Version        {"version": 51}
2021-05-05T19:05:38.631+0100    INFO    preflight:localtime     preflight/localtime.go:54       start checking local system clock with trusted satellites' system clock.
2021-05-05T19:05:38.631+0100    ERROR   storagenode/peer.go:838 Failed preflight check. {"error": "system clock is out of sync: system clock is out of sync with all trusted satellites", "errorVerbose": "system clock is out of sync: system clock is out of sync with all trusted satellites\n\tstorj.io/storj/storagenode/preflight.(*LocalTime).Check:96\n\tstorj.io/storj/storagenode.(*Peer).Run:837\n\tmain.cmdRun:218\n\tstorj.io/private/process.cleanup.func1.4:363\n\tstorj.io/private/process.cleanup.func1:381\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfig:88\n\tstorj.io/private/process.ExecCustomDebug:70\n\tmain.main:385\n\truntime.main:225"}
storj.io/storj/storagenode.(*Peer).Run
        /home/storj/storj/storagenode/peer.go:838
main.cmdRun
        /home/storj/storj/cmd/storagenode/main.go:218
storj.io/private/process.cleanup.func1.4
        /home/storj/go/pkg/mod/storj.io/private@v0.0.0-20210423085237-5af81f2a2b21/process/exec_conf.go:363
storj.io/private/process.cleanup.func1
        /home/storj/go/pkg/mod/storj.io/private@v0.0.0-20210423085237-5af81f2a2b21/process/exec_conf.go:381
github.com/spf13/cobra.(*Command).execute
        /home/storj/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:852
github.com/spf13/cobra.(*Command).ExecuteC
        /home/storj/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:960
github.com/spf13/cobra.(*Command).Execute
        /home/storj/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:897
storj.io/private/process.ExecWithCustomConfig
        /home/storj/go/pkg/mod/storj.io/private@v0.0.0-20210423085237-5af81f2a2b21/process/exec_conf.go:88
storj.io/private/process.ExecCustomDebug
        /home/storj/go/pkg/mod/storj.io/private@v0.0.0-20210423085237-5af81f2a2b21/process/exec_conf.go:70
main.main
        /home/storj/storj/cmd/storagenode/main.go:385
runtime.main
        /usr/local/go/src/runtime/proc.go:225
Error: system clock is out of sync: system clock is out of sync with all trusted satellite

This was the message I got
Anyone help me?

[Alexey]

Hello @Chelle ,
Welcome to the forum!

What is the first error after start?

[Chelle]

It doesn’t really start, it’s the result of what I get in ssh terminal after I do ‘storagenode run’
The first few lines are:
Yes I realise it’s not got an e-mail in config, but that’s optional I gather.

2021-05-05T19:05:37.961+0100    INFO    process/exec_conf.go:284        
Configuration loaded    {"Location": "/home/storj/.local/share/storj/storagenode/config.yaml"}
2021-05-05T19:05:37.976+0100    WARN    operator/operator.go:40 Operator 
email address isn't specified.
storj.io/storj/storagenode/operator.isOperatorEmailValid

Do you think everything installed properly ?

[waistcoat]

I run storagenodes inside truenas jails - is your server syncing it’s time correctly?
On your truenas server try checking it’s time offset with something like
root@freenas:~ # ntpdate -d time.google.com and checking ntpd with
root@freenas:~ # ntpq -p

[Chelle]

I 've no idea if this is good or not?

server 216.239.35.12, port 123
stratum 1, precision -20, leap 00, trust 000
refid [GOOG], root delay 0.000000, root dispersion 0.000107
reference time:      e43d831e.82dc1e96  Wed, May  5 2021 21:48:30.511
originate timestamp: e43d831e.82dc1e99  Wed, May  5 2021 21:48:30.511
transmit timestamp:  e43d831e.813e56d3  Wed, May  5 2021 21:48:30.504
filter delay:  0.04906    0.04872    0.04926    0.04831
           ----       ----       ----       ----
filter offset: -0.004946  -0.005065  -0.004752  -0.005032
           ----       ----       ----       ----
delay 0.04831, dispersion 0.00006, offset -0.005032

server 216.239.35.0, port 123
stratum 1, precision -20, leap 00, trust 000
refid [GOOG], root delay 0.000000, root dispersion 0.000092
reference time:      e43d831e.b5bd0416  Wed, May  5 2021 21:48:30.709
originate timestamp: e43d831e.b5bd0419  Wed, May  5 2021 21:48:30.709
transmit timestamp:  e43d831e.b4702cdb  Wed, May  5 2021 21:48:30.704
filter delay:  0.04535    0.04472    0.04466    0.04485
           ----       ----       ----       ----
filter offset: -0.004604  -0.004532  -0.004669  -0.004539
           ----       ----       ----       ----
delay 0.04466, dispersion 0.00011, offset -0.004669

server 216.239.35.8, port 123
stratum 1, precision -20, leap 00, trust 000
refid [GOOG], root delay 0.000000, root dispersion 0.000092
reference time:      e43d831e.e91f94af  Wed, May  5 2021 21:48:30.910
originate timestamp: e43d831e.e91f94b2  Wed, May  5 2021 21:48:30.910
transmit timestamp:  e43d831e.e7a450cf  Wed, May  5 2021 21:48:30.904
filter delay:  0.04605    0.04524    0.04524    0.04581
           ----       ----       ----       ----
filter offset: -0.004196  -0.004328  -0.004318  -0.004310
           ----       ----       ----       ----
delay 0.04524, dispersion 0.00002, offset -0.004328

server 216.239.35.4, port 123
stratum 1, precision -20, leap 00, trust 000
refid [GOOG], root delay 0.000000, root dispersion 0.000092
reference time:      e43d831f.1c218a64  Wed, May  5 2021 21:48:31.109
originate timestamp: e43d831f.1c218a67  Wed, May  5 2021 21:48:31.109
transmit timestamp:  e43d831f.1ad72edd  Wed, May  5 2021 21:48:31.104
filter delay:  0.04652    0.04430    0.04477    0.04488
           ----       ----       ----       ----
filter offset: -0.004046  -0.004596  -0.004565  -0.004592
           ----       ----       ----       ----
delay 0.04430, dispersion 0.00008, offset -0.004596

There could be an offset, but how small does it need to be?

[Chelle]

Ah when I do the second command I get:

ntpq -p
localhost: timed out, nothing received
***Request timed out
EDIT
    root@truenas[~]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 slardar.parseq. .INIT.          16 u    -  128    0    0.000   +0.000   0.000
 85.199.214.100  .INIT.          16 u    -  128    0    0.000   +0.000   0.000
 tor.nocabal.de  .INIT.          16 u    -  128    0    0.000   +0.000   0.000
*ns3.turbodns.co 85.199.214.99    2 u   63   64  377   18.466   -0.959   0.628

I thought the jail gets it time from the main frenass os ?
is there something I need to add?

EDIT
Do I run ntpq -p in the jail or the frenass terminal?
I think you run it in frenass

[waistcoat]

I was doing ntpq -p on the truenas server itself (yes you can use the shell). That should show you that your server is in sync and with which servers.

The output of ntpdate -q is a little more concise:
root@freenas:~ # ntpdate -q time.google.com
server 216.239.35.0, stratum 1, offset -0.000841, delay 0.03847
server 216.239.35.12, stratum 1, offset -0.001845, delay 0.04324
server 216.239.35.8, stratum 1, offset -0.001164, delay 0.03918
server 216.239.35.4, stratum 1, offset -0.001225, delay 0.03912
5 May 22:49:47 ntpdate[39287]: adjust time server 216.239.35.0 offset -0.000841 sec

I may well have better offsets than yours (I have a local GPS clock), However I’d have thought that an offset of -0.005032 from your server would be just fine.

Do you have host_time set in your custom jail properties? This should make your jail synchronise time with the freenas server.

[Chelle]

Yes the box host_time is ticked in custom properties.
I tried switching different time servers, but didn’t make any difference.

Jail is 12.2 release p6.

EDIT
just tried re-downloading and installing but same error with clock.

I’m using the command

go install -v ./cmd/...

to build bin and pkg is this correct ?
Do I need anything else installed ?
Have already generated identity, and set up storage for jail, edited config file.

[Alexey]

The clock check is a last check in the preflight check list. I would like to see the first errors before clock check.
Something like “ping satellite failed”

[waistcoat]

Looking at the following thread it seems that the clock error could be a red-herring and the error might actually mean your jail cannot access the satellites to check the time, rather than your time is wrong.

System clock is out of sync: system clock is out of sync with all trusted satellites

You do not need it for Debian. Install ntp and it will be synced automatically But I think your problem is not clock out of sync, this is a last error after all checks. You need a first one, where your node is likely is unable to contact satellites, either because your IP is not reachable or network is not configured properly (some firewalls or network interface issue). Please, post last 20 lines from your logs: How do I check my logs? - Storj

Try to manually connect to a satellite from inside your jail. i.e.
$ telnet us2.storj.io 7777
Trying 35.192.11.148…
Connected to us2.storj.io.
Escape character is ‘^]’.

If DNS doesn’t resolve or you don’t get a connection, you have a networking issue.

[Chelle]

Think its connecting?

[storj@Stor-jl ~/go/bin]$ telnet us2.storj.io 7777
Trying 34.86.70.247...
Connected to us2.storj.io.
Escape character is '^]'.

Cant seem to see any other errors that come first?

[Alexey]

Then please, perform usual checks:

My Node is OFFLINE for 2 hours

It should connect in seconds. There are a few things you should check. Start with checking the port. On https://www.yougetsignal.com/tools/open-ports/?port=28967 you can fill in the domain or IP you are using in the run command. Then check the port. If it’s closed, it could still be several things. Make sure the WAN IP you see in your router settings matches the IP shown on the above page. Make sure your port forward is set up correctly. That means no filter on incoming IPs, forwarding to th…

[Chelle]

yougetsignal says the port closed.
Have opened them in router(s) (network is double nat ip4) but not sure if it works

But storj wont run as it exits with clock error.
Does BSD jail / frenass have firewall ports I need to open as well ?

Edit
just checking form the other thread.
should the identity folder have 4 or 6 files ?
Have 4
ca.cert.
identity.key
identity.cert
ca.key

Maybe another problem

[waistcoat]

I’m running my jails with their own IPv4 addresses using a VNET. I couldn’t figure out how to get inbound connections when using NAT.

I do have issues with my jails loosing connectivity when I change network config on the TrueNAS server, but that happens so rarely I haven’t looked into it properly. Bouncing the jails fixes the issue anyway.

[Chelle]

I’m going to generate another identity file, as i could have a problem with the first one, I may not have done it right.
I’ll have to come back tomorrow and see if that one works

[Chelle]

I 've been trying to follow the guide here : Getting started with Storj Testnet on FreeNAS (freeBSD) – Storj DCS

but can not get it to work, cant even get the test/sim network to work

Error: some executables cannot be found: [gateway]

Cant help think it’s something simple, but not familiar with go
and don’t think everything’s installed right.

[Alexey]

This is to install storj-sim. It will not install storagenode for production.

[Chelle]

I didn’t know if it would test the installation of the production storagenode.

I seem to have built the storagenode binary, its in the bin folder and is 57,043Kb.
Every time I try to run it

Its still exiting with the clock error, and don’t know what to troubleshoot.
or what else to try.

[Alexey]

The clock out of sync is not necessarily mean that it’s a reason and not a consequence. This is the last check and more like the error was somewhere earlier.
Let’s eliminate each point separately:

Node stopped responding overnight. Still shows OFFLINE after restart

ok, let check it together. WAN IP on the main page of the router. Please, compare it with IP there: https://www.yougetsignal.com/tools/open-ports/ , they should match Local IP in the port forwarding rule, it should be the same as IP of your PC with docker nslookup your.ddns.address 8.8.8.8 should return the same IP as p.1 Firewall. It should have an incoming rule to allow any traffic from the any source to the TCP 28967 port and your PC as a destination. If you have any outgoing rule, then y…

The identity could be checked locally as well: Step 5. Create an Identity - Storj Docs