X509: certificate signed by unknown authority for https://tardigrade.io/trusted-satellites

Toyoo · December 5, 2020, 3:38pm

Hi, it looks like the newest docker image (IMAGE ID = ba7e12155ec9) does not have the CA required for https://tardigrade.io/trusted-satellites. Symptoms:

From a log file of a restarted node:

2020-12-05T15:31:40.798Z        FATAL   Unrecoverable error     {"error": "trust: failed to fetch from source \"https://tardigrade.io/trusted-satellites\": HTTP source: Get \"https://tardigrade.io/trusted-satellites\": x509: certificate signed by unknown authority", "errorVerbose": "trust: failed to fetch from source \"https://tardigrade.io/trusted-satellites\": HTTP source: Get \"https://tardigrade.io/trusted-satellites\": x509: certificate signed by unknown authority\n\tstorj.io/storj/storagenode/trust.(*List).fetchEntries:96\n\tstorj.io/storj/storagenode/trust.(*List).FetchURLs:49\n\tstorj.io/storj/storagenode/trust.(*Pool).fetchURLs:240\n\tstorj.io/storj/storagenode/trust.(*Pool).Refresh:177\n\tstorj.io/storj/storagenode.(*Peer).Run:781\n\tmain.cmdRun:210\n\tstorj.io/private/process.cleanup.func1.4:362\n\tstorj.io/private/process.cleanup.func1:380\n\tgithub.com/spf13/cobra.(*Command).execute:842\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:950\n\tgithub.com/spf13/cobra.(*Command).Execute:887\n\tstorj.io/private/process.ExecWithCustomConfig:88\n\tstorj.io/private/process.ExecCustomDebug:70\n\tmain.main:338\n\truntime.main:204"}

From a container of a node I did not restart yet:

root@x:~# docker exec -it d0653897873a sh                                                                                                                                                                                                                                   
/app # wget https://tardigrade.io/trusted-satellites                                                                                                                                                                                                                             
Connecting to tardigrade.io (104.198.14.52:443)                                                                                                                                                                                                                                  
ssl_client: tardigrade.io: certificate verification failed: unable to get local issuer certificate                                                                                                                                                                               
wget: error getting response: Connection reset by peer                                                                                                                                                                                                                           
/app #

jocelyn · December 5, 2020, 3:45pm

I’ll let engineering know, thank you

stefanbenten · December 5, 2020, 3:46pm

@ToyooCan you please let me know which tag this corresponds to and additionally which arch?

Alexey · December 5, 2020, 3:55pm

latest x86 at least

PS C:\Users\aaleo> docker run -it --rm --entrypoint /bin/sh storjlabs/storagenode
/app # wget https://tardigrade.io/trusted-satellites
Connecting to tardigrade.io (104.198.14.52:443)
ssl_client: tardigrade.io: certificate verification failed: unable to get local issuer certificate
wget: error getting response: Connection reset by peer

The arm is affected too

pi@raspberrypi:~ $ docker run -it --rm --entrypoint /bin/sh storjlabs/storagenode
/app # wget https://tardigrade.io/trusted-satellites
Connecting to tardigrade.io (104.198.14.52:443)
ssl_client: tardigrade.io: certificate verification failed: unable to get local issuer certificate
wget: error getting response: Connection reset by peer
/app #

alpine:latest working fine

$ docker run -it --rm alpine wget https://tardigrade.io/trusted-satellites
Connecting to tardigrade.io (104.198.14.52:443)
saving to 'trusted-satellites'
trusted-satellites   100% |************************************************************************|   417  0:00:00 ETA
'trusted-satellites' saved

Toyoo · December 5, 2020, 3:50pm

storjlabs/storagenode:latest, amd64

stefanbenten · December 5, 2020, 3:55pm

I just downloaded it, installed curl and checked it. That seems to find the correct CA.
I’ll see what the alpine container is missing.

stefanbenten · December 5, 2020, 4:45pm

@Toyoo I pushed images that should be fixed here:

Can you test them please?

Toyoo · December 5, 2020, 4:53pm

Seems fine, the node works now. Thank you!

stefanbenten · December 24, 2020, 12:46am

I just pushed the updated sets to our main repo. This hopefully fixes it.
Thanks for reporting and keeping an eye out!

jocelyn · December 5, 2020, 7:49pm

amazing fast time to resolution! thank you @stefanbenten for the quick action, and thanks to @Toyoo and everyone else onthread for helping us improve

Toyoo · December 5, 2020, 8:25pm

I agree, I couldn’t imagine a better response from Storj!

andrew2.hart · December 6, 2020, 7:47am

Did my docker nodes restart 12h ago because of this or do I need to check?

Edit: I checked

time=“2020-12-05T17:19:09Z” level=info msg=“Found new storjlabs/storagenode:latest image (sha256:cbeccde9e35acbe3ec35a21e3a68529c96fe01627a1982dd4921ce9674473ab2)”

I am not sure I like getting updates without a version number update

Alexey · December 6, 2020, 3:30pm

Yes it’s updated to the same version at the moment with a new certificates to do not fail during any restart.

ACarneiro · December 6, 2020, 3:46pm

Yes, my nodes just restarted and kept the same version number.
Very off-putting

Alexey · December 6, 2020, 4:04pm

So you prefer that it will be in infinity restart cycle on any restart without this update?
I don’t think so

ACarneiro · December 6, 2020, 4:07pm

Ideally I would have preferred a different version number, but really it makes no difference as such.
I was just puzzled by all my nodes restarting for no obvious reason, that’s all.

dseah · December 7, 2020, 2:49pm

Hi I am new here, 2 months old as a storagenode operator. I am using Rock64 running ubuntu 18.04 bionic minimal 64bit. Everything was working well until few days ago my OS crashed. I am switching to armbian focal now, re-installed docker and copied my backup storagenode identity to my config path. However, when I start the docker, these errors show up in my log.

How should I troubleshoot this?

Alexey · December 7, 2020, 8:56pm

Hello @dseah,
Welcome to the forum!

Stop the storagenode, remove the container, then execute

docker pull storjlabs/storagenode

And run your node with all your parameters again.

majklsee · December 8, 2020, 4:47am

Hello, I tried to pull latest image, but doesn’t help…

2020-12-07T21:56:42.908Z INFO Configuration loaded {“Location”: “/app/config/config.yaml”}
2020-12-07T21:56:42.914Z INFO Operator email {“Address”: “XXXXXXX”}
2020-12-07T21:56:42.914Z INFO Operator wallet {“Address”: “XXXXXXX”}
2020-12-07T21:56:43.943Z INFO Telemetry enabled {“instance ID”: “1iPV9F3GHWqyXBmaUVfhnADNBpYJ9UmxYrckstmXhdNCjyz9pH”}
2020-12-07T21:56:44.351Z INFO db.migration Database Version {“version”: 46}
2020-12-07T21:56:45.545Z ERROR trust Failed to fetch URLs from source {“source”: “https://tardigrade.io/trusted-satellites”, “error”: “HTTP source: Get “https://tardigrade.io/trusted-satellites”: x509: certificate signed by unknown authority”, “errorVerbose”: “HTTP source: Get “https://tardigrade.io/trusted-satellites”: x509: certificate signed by unknown authority\n\tstorj.io/storj/storagenode/trust.(*HTTPSource).FetchEntries:63\n\tstorj.io/storj/storagenode/trust.(*List).fetchEntries:90\n\tstorj.io/storj/storagenode/trust.(*List).FetchURLs:49\n\tstorj.io/storj/storagenode/trust.(*Pool).fetchURLs:240\n\tstorj.io/storj/storagenode/trust.(*Pool).Refresh:177\n\tstorj.io/storj/storagenode.(*Peer).Run:781\n\tmain.cmdRun:210\n\tstorj.io/private/process.cleanup.func1.4:362\n\tstorj.io/private/process.cleanup.func1:380\n\tgithub.com/spf13/cobra.(*Command).execute:842\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:950\n\tgithub.com/spf13/cobra.(*Command).Execute:887\n\tstorj.io/private/process.ExecWithCustomConfig:88\n\tstorj.io/private/process.ExecCustomDebug:70\n\tmain.main:338\n\truntime.main:204”}
Error: trust: failed to fetch from source “https://tardigrade.io/trusted-satellites”: HTTP source: Get “https://tardigrade.io/trusted-satellites”: x509: certificate signed by unknown authority

any ideas what could be wrong?

Thank you

Alexey · December 7, 2020, 10:08pm

Please, show the output of the

docker pull storjlabs/storagenode