ERROR trust Failed to fetch URLs from source {"source": "https://tardigrade.io/trusted-satellites",

user2 · April 8, 2021, 4:38pm

Hi there,

I have just re-validated and attempted to re-deploy a Ubuntu/docker instance and am facing a frustrating error.

The docker logs show the following:

2021-04-08T16:15:48.886Z        ERROR   trust   Failed to fetch URLs from source        {"source": "https://tardigrade.io/trusted-satellites", "error": "HTTP source: Get \"https://tardigrade.io/trusted-satellites\": dial tcp: lookup tardigrade.io: Try again", "errorVerbose": "HTTP source: Get \"https://tardigrade.io/trusted-satellites\": dial tcp: lookup tardigrade.io: Try again\n\tstorj.io/storj/storagenode/trust.(*HTTPSource).FetchEntries:63\n\tstorj.io/storj/storagenode/trust.(*List).fetchEntries:90\n\tstorj.io/storj/storagenode/trust.(*List).FetchURLs:49\n\tstorj.io/storj/storagenode/trust.(*Pool).fetchURLs:240\n\tstorj.io/storj/storagenode/trust.(*Pool).Refresh:177\n\tstorj.io/storj/storagenode.(*Peer).Run:831\n\tmain.cmdRun:218\n\tstorj.io/private/process.cleanup.func1.4:363\n\tstorj.io/private/process.cleanup.func1:381\n\tgithub.com/spf13/cobra.(*Command).execute:842\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:950\n\tgithub.com/spf13/cobra.(*Command).Execute:887\n\tstorj.io/private/process.ExecWithCustomConfig:88\n\tstorj.io/private/process.ExecCustomDebug:70\n\tmain.main:385\n\truntime.main:204"}
Error: trust: failed to fetch from source "https://tardigrade.io/trusted-satellites": HTTP source: Get "https://tardigrade.io/trusted-satellites": dial tcp: lookup tardigrade.io: Try again

Port 28967 is enabled via ufw and the docker config is set up as outlined in the docs.

The previous node worked flawlessly until the end of the previous month. Then the docker container stopped due to a time-sync error for upwards of a few days (based on the docker log). After restarting, it subsequently went offline in the past week. Watchtower is enabled for the respective docker container.

Any insight is appreciated.
Cheers

twl · April 8, 2021, 4:59pm

This could be some kind of DNS error or anything else preventing your container from reaching the shown URL to fetch the trusted satellite list.

user2 · April 8, 2021, 5:31pm

Thanks for the quick reply.

I tried to manually set the docker container dns server to one of Cloudflare’s using the following additional parameter in the bash docker run command:

--dns=1.1.1.1

This did not change anything unfortunately.

Are there any additional firewall rules required beyond allowing traffic to port 28967?

I tried a method posted before which involved pulling the docker container, opening the docker container shell and attempting to download (using wget) the same ‘trusted-satellites’ html page. This worked fine.

Not sure what else to try at the moment.

twl · April 8, 2021, 5:36pm

The documentation for Windows only refers to an inbound TCP rule on said port - Linux should require the same, but nothing else. I don’t run any Linux/Docker Storj nodes tho, so probably someone else might have something more useful to add

baker · April 8, 2021, 10:00pm

Have you tried restarting everything between your node and the internet? The only time I had problems with the docker container and DNS lookups, it ended up being a restart of my router that fixed it. Can’t tell you why, but that was the problem.

Alexey · April 9, 2021, 5:07am

Please, make sure that you uses storjlabs/storagenode:latest image, not :beta or :alpha

user2 · April 9, 2021, 6:55am

@baker : The server is hosted in a data center. I did consider a reboot but was a little reluctant due to other services running. I did however restart the docker service/socket and, to my surprise, the instance started without any hiccups.

sudo systemctl stop docker.socket
sudo systemctl stop docker.service

sudo systemctl start docker.service
sudo systemctl start docker.socket

@Alexey : Thanks for the suggestion. I checked through the parameters as well and the correct docker image was listed.

To be honest I am a bit concerned about any future watchtower updates of the docker image resulting in similar DNS issues…

Alexey · April 10, 2021, 4:43am

This is likely related to the updates or installation of some services.
The watchtower is a usual docker container, which is rarely checks the docker hub and updates running containers.

user2 · April 10, 2021, 9:33am

@Alexey : Thanks for the clarification. I should have been more clear: I’m concerned about the dependability of my setup in general

I’ll report back should this event ever happen again. Hopefully I’ll have a bit more information from which to draw any definitive conclusions.

Salman · September 16, 2022, 4:35pm

hello，Alexey，i have a big problem.
I ran the first node in July this year. Today, I plan to run the second node (with a different IP address) in another home, but I found that the node (win GUI) cannot be run during the configuration process. After checking the log and trying different methods, I found that this is because the node will communicate with the satellite and generate (trust-cache.json) files when it runs for the first time. However, it should be that my country (China) blocked the relevant links not long ago, so the file could not be generated, resulting in the node being unable to run.

After I try to copy the file of another node, the second node can work normally.

Here are my questions:

If I do this, will it affect the operation of the first node.
Does the (trust cache. json) file update frequently? Whether this will lead to node operation problems in the long run.
Will this problem affect the normal update of the node?
Whether it can be considered to solve the problem later.

We look forward to your answer:

The following is my log (when the first run terminates).

2022-09-17T00:23:49.925+0800	INFO	Configuration loaded	{"Location": "C:\\Program Files\\Storj\\Storage Node\\config.yaml"}
2022-09-17T00:23:49.926+0800	INFO	Anonymized tracing enabled
2022-09-17T00:23:49.946+0800	WARN	Operator email address isn't specified.
2022-09-17T00:23:49.946+0800	INFO	Operator wallet	{"Address": "0x912f882AA826d714D52b06e462D4D335EA1d86AB"}
2022-09-17T00:23:50.283+0800	INFO	Telemetry enabled	{"instance ID": "12nA3FUYGpw58vKGkDUjxUeGsqwgWRTK7c69Y5UecguPtgp3vhb"}
2022-09-17T00:23:50.372+0800	INFO	db.migration	Database Version	{"version": 54}
2022-09-17T00:24:11.754+0800	ERROR	trust	Failed to fetch URLs from source	{"source": "https://www.storj.io/dcs-satellites", "error": "HTTP source: Get \"https://www.storj.io/dcs-satellites\": dial tcp 104.244.43.167:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.", "errorVerbose": "HTTP source: Get \"https://www.storj.io/dcs-satellites\": dial tcp 104.244.43.167:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.\n\tstorj.io/storj/storagenode/trust.(*HTTPSource).FetchEntries:68\n\tstorj.io/storj/storagenode/trust.(*List).fetchEntries:90\n\tstorj.io/storj/storagenode/trust.(*List).FetchURLs:49\n\tstorj.io/storj/storagenode/trust.(*Pool).fetchURLs:250\n\tstorj.io/storj/storagenode/trust.(*Pool).Refresh:187\n\tstorj.io/storj/storagenode.(*Peer).Run:894\n\tmain.cmdRun:251\n\tstorj.io/private/process.cleanup.func1.4:378\n\tstorj.io/private/process.cleanup.func1:396\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:93\n\tstorj.io/private/process.ExecWithCustomConfig:75\n\tstorj.io/private/process.Exec:65\n\tmain.(*service).Execute.func1:61\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2022-09-17T00:24:12.114+0800	FATAL	Unrecoverable error	{"error": "trust: failed to fetch from source \"https://www.storj.io/dcs-satellites\": HTTP source: Get \"https://www.storj.io/dcs-satellites\": dial tcp 104.244.43.167:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.", "errorVerbose": "trust: failed to fetch from source \"https://www.storj.io/dcs-satellites\": HTTP source: Get \"https://www.storj.io/dcs-satellites\": dial tcp 104.244.43.167:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.\n\tstorj.io/storj/storagenode/trust.(*List).fetchEntries:96\n\tstorj.io/storj/storagenode/trust.(*List).FetchURLs:49\n\tstorj.io/storj/storagenode/trust.(*Pool).fetchURLs:250\n\tstorj.io/storj/storagenode/trust.(*Pool).Refresh:187\n\tstorj.io/storj/storagenode.(*Peer).Run:894\n\tmain.cmdRun:251\n\tstorj.io/private/process.cleanup.func1.4:378\n\tstorj.io/private/process.cleanup.func1:396\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:93\n\tstorj.io/private/process.ExecWithCustomConfig:75\n\tstorj.io/private/process.Exec:65\n\tmain.(*service).Execute.func1:61\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

Alexey · September 16, 2022, 5:38pm

You can use this file, but it would not be updated if we add or remove the satellite into/from the trusted list.
However, if your node cannot communicate with satellites it likely will not work. So make sure that it can communicate with the satellites and only this site is blocked for you.