Node could not start since yesterday

Fred · April 22, 2022, 12:23pm

Hey!
One of my nodes, which runs smooth since 2 years, stoped working yesterday evening. I removed the container to start from scratch, but it never starts. Nodename is hp-mic1-n1, it runs in docker on a debian10 server. DNS, time, internetconnection is checked twice.
Here is the log:

root@hp-mic1:/# docker start hp-mic1-n1
hp-mic1-n1
root@hp-mic1:/# docker logs -t hp-mic1-n1 --tail 50
2022-04-22T11:54:57.179040682Z downloading storagenode-updater
2022-04-22T11:54:57.186106509Z --2022-04-22 11:54:57--  https://version.storj.io/processes/storagenode-updater/minimum/url?os=linux&arch=amd64
2022-04-22T11:54:57.235691353Z Resolving version.storj.io (version.storj.io)... 35.224.88.204
2022-04-22T11:54:57.369159947Z Connecting to version.storj.io (version.storj.io)|35.224.88.204|:443... connected.
2022-04-22T11:54:57.910249375Z HTTP request sent, awaiting response... 200 OK
2022-04-22T11:54:57.910320088Z Length: 92 [text/plain]
2022-04-22T11:54:57.910337398Z Saving to: 'STDOUT'
2022-04-22T11:54:57.910442753Z
2022-04-22T11:54:57.910543935Z      0K                                                       100% 65.1M=0s
2022-04-22T11:54:57.910574655Z
2022-04-22T11:54:57.910591423Z 2022-04-22 11:54:57 (65.1 MB/s) - written to stdout [92/92]
2022-04-22T11:54:57.910607108Z
2022-04-22T11:54:57.922362557Z --2022-04-22 11:54:57--  https://github.com/storj/storj/releases/download/v1.52.2/storagenode-updater_linux_amd64.zip
2022-04-22T11:54:57.964106460Z Resolving github.com (github.com)... 140.82.121.4
2022-04-22T11:54:58.001143840Z Connecting to github.com (github.com)|140.82.121.4|:443... connected.
2022-04-22T11:54:58.179430018Z HTTP request sent, awaiting response... 302 Found
2022-04-22T11:54:58.179557744Z Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/7cc525c9-8b78-4199-98e6-861239e6b4f4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220422%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220422T115313Z&X-Amz-Expires=300&X-Amz-Signature=592cb04ec04ed7c2011e46972a814b43bb50e1a77e5e0c6d3137b3bc7fe9f5b7&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_amd64.zip&response-content-type=application%2Foctet-stream [following]
2022-04-22T11:54:58.179816723Z --2022-04-22 11:54:58--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/7cc525c9-8b78-4199-98e6-861239e6b4f4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220422%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220422T115313Z&X-Amz-Expires=300&X-Amz-Signature=592cb04ec04ed7c2011e46972a814b43bb50e1a77e5e0c6d3137b3bc7fe9f5b7&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_amd64.zip&response-content-type=application%2Foctet-stream
2022-04-22T11:54:58.203408669Z Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...
2022-04-22T11:54:58.225748929Z Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443... connected.

Any idea? Thank you!

Pac · April 22, 2022, 9:08pm

Hello @Fred and welcome to the forum

Is this the complete log?
Doesn’t it contain any warning or error that would help investigate?

Fred · April 22, 2022, 9:41pm

Yes, this is the complete log. It stops exactly everytime at this point, and nothing happens then.

serger001 · April 23, 2022, 5:04am

Restart docker image. Works for me

Alexey · April 23, 2022, 6:14am

Looks like it stuck in downloading from GitHub. Could you please restart the container?

docker restart hp-mic1-n1

Fred · April 23, 2022, 9:38am

Good morning. Restart did not help, here is the whole log.

root@hp-mic1:/# docker logs -t hp-mic1-n1
2022-04-22T18:44:44.145067463Z downloading storagenode-updater
2022-04-22T18:44:44.150657176Z --2022-04-22 18:44:44--  https://version.storj.io/processes/storagenode-updater/minimum/url?os=linux&arch=amd64
2022-04-22T18:44:44.198430746Z Resolving version.storj.io (version.storj.io)... 35.224.88.204
2022-04-22T18:44:44.328250369Z Connecting to version.storj.io (version.storj.io)|35.224.88.204|:443... connected.
2022-04-22T18:44:44.952735991Z HTTP request sent, awaiting response... 200 OK
2022-04-22T18:44:44.952806172Z Length: 92 [text/plain]
2022-04-22T18:44:44.952823158Z Saving to: 'STDOUT'
2022-04-22T18:44:44.953340183Z
2022-04-22T18:44:44.953410516Z      0K                                                       100% 64.5M=0s
2022-04-22T18:44:44.953428775Z
2022-04-22T18:44:44.953583267Z 2022-04-22 18:44:44 (64.5 MB/s) - written to stdout [92/92]
2022-04-22T18:44:44.953614716Z
2022-04-22T18:44:44.965830965Z --2022-04-22 18:44:44--  https://github.com/storj/storj/releases/download/v1.52.2/storagenode-updater_linux_amd64.zip
2022-04-22T18:44:45.067498183Z Resolving github.com (github.com)... 140.82.121.4
2022-04-22T18:44:45.161763820Z Connecting to github.com (github.com)|140.82.121.4|:443... connected.
2022-04-22T18:44:45.636710131Z HTTP request sent, awaiting response... 302 Found
2022-04-22T18:44:45.636790111Z Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/7cc525c9-8b78-4199-98e6-861239e6b4f4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220422%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220422T184255Z&X-Amz-Expires=300&X-Amz-Signature=19071bded3c2ef30621aa9371d1a9b54e0266397835938e2a3d092ec5000596e&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_amd64.zip&response-content-type=application%2Foctet-stream [following]
2022-04-22T18:44:45.636822349Z --2022-04-22 18:44:45--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/7cc525c9-8b78-4199-98e6-861239e6b4f4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220422%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220422T184255Z&X-Amz-Expires=300&X-Amz-Signature=19071bded3c2ef30621aa9371d1a9b54e0266397835938e2a3d092ec5000596e&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_amd64.zip&response-content-type=application%2Foctet-stream
2022-04-22T18:44:45.703289089Z Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.108.133, 185.199.110.133, 185.199.111.133, ...
2022-04-22T18:44:45.791421256Z Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443... connected.
2022-04-23T07:35:12.880073686Z downloading storagenode-updater
2022-04-23T07:35:12.888381136Z --2022-04-23 07:35:12--  https://version.storj.io/processes/storagenode-updater/minimum/url?os=linux&arch=amd64
2022-04-23T07:35:12.957030764Z Resolving version.storj.io (version.storj.io)... 35.224.88.204
2022-04-23T07:35:13.117654456Z Connecting to version.storj.io (version.storj.io)|35.224.88.204|:443... connected.
2022-04-23T07:35:14.114659985Z HTTP request sent, awaiting response... 200 OK
2022-04-23T07:35:14.114759022Z Length: 92 [text/plain]
2022-04-23T07:35:14.114777202Z Saving to: 'STDOUT'
2022-04-23T07:35:14.114881518Z
2022-04-23T07:35:14.114911573Z      0K                                                       100% 62.0M=0s
2022-04-23T07:35:14.114927999Z
2022-04-23T07:35:14.114942724Z 2022-04-23 07:35:14 (62.0 MB/s) - written to stdout [92/92]
2022-04-23T07:35:14.114958532Z
2022-04-23T07:35:14.125402615Z --2022-04-23 07:35:14--  https://github.com/storj/storj/releases/download/v1.52.2/storagenode-updater_linux_amd64.zip
2022-04-23T07:35:14.187943421Z Resolving github.com (github.com)... 140.82.121.4
2022-04-23T07:35:14.234814852Z Connecting to github.com (github.com)|140.82.121.4|:443... connected.
2022-04-23T07:35:14.624190883Z HTTP request sent, awaiting response... 302 Found
2022-04-23T07:35:14.624425658Z Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/7cc525c9-8b78-4199-98e6-861239e6b4f4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220423%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220423T073514Z&X-Amz-Expires=300&X-Amz-Signature=823bc1d95f25d8fb8bd28cc37653f174c70b5db60332caffe415518769df414e&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_amd64.zip&response-content-type=application%2Foctet-stream [following]
2022-04-23T07:35:14.624594115Z --2022-04-23 07:35:14--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/7cc525c9-8b78-4199-98e6-861239e6b4f4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220423%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220423T073514Z&X-Amz-Expires=300&X-Amz-Signature=823bc1d95f25d8fb8bd28cc37653f174c70b5db60332caffe415518769df414e&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_amd64.zip&response-content-type=application%2Foctet-stream
2022-04-23T07:35:14.675060894Z Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...
2022-04-23T07:35:14.717799853Z Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443... connected.
root@hp-mic1:/#

I have several physical servers with two nodes each. On this server are also two nodes and the problem is only on this host on both storj nodes. It occurs at the same time on both nodes in the night, nothing was changed on the server or docker installation. Now i reinstalled docker and did a apt upgrade and a reboot - but its the same behaviour.

Alexey · April 23, 2022, 2:15pm

Something is blocking connection to githubusercontent.com, this is weird.
I just checked - it’s working normally
Please try this command:

docker exec -it storagenode wget -O /tmp/storagenode-updater.zip https://github.com/storj/storj/releases/download/v1.52.2/storagenode-updater_linux_amd64.zip

serger001 · April 23, 2022, 2:42pm

Алексей, я воспроизвел дефект. Иногда почему-то не докачивается обновление. При этом модуль обновления считает что все ок, и пытается его запустить. Не плохо бы сделать проверку того, что архивом скачалось, и если crc не совпадает, то пробовать перекачать. У меня эта проблема появлялась на узле с высокой нагрузкой на сеть и диски.
У меня вот это скачивание обрывалось на 7 мегабайтах и пыталось обновиться битым архивом
15200K … … … … … 99% 5.96M 0s
15250K … … … … … 99% 3.36M 0s
15300K … 100% 50.8M=12s
2022-04-23 14:48:43 (1.26 MB/s) - ‘/tmp/storagenode.zip’ saved [15675435/15675435]

Иногда при перезапуске вот так выглядит лог:

4100K … … … … … 52% 36.9M 1s
4150K … … … … … 53% 42.5M 1s
4200K … … … … … 53% 36.5M 1s
4250K … … … … … 54% 60.0M 0s
4300K … … … … … 54% 44.9M 0s
4350K … … … … … 55% 51.8M 0s
4400K … … … … … 56% 47.8M 0s
4450K … … … … … 56% 46.6M 0s
4500K … … … … … 57% 46.4M 0s
4550K … … … … … 58% 45.9M 0s
4600K … … … … … 58% 35.2M 0s
4650K … … … … … 59% 45.6M 0s
4700K … … … … … 60% 45.4M 0s
4750K … … … … … 60% 43.1M 0s
4800K … … … … … 61% 46.1M 0s
4850K … … … … … 61% 43.3M 0s
4900K … … … … … 62% 47.1M 0s
4950K … … … … … 63% 45.8M 0s
5000K … … … … … 63% 35.6M 0s
5050K … … … … … 64% 46.1M 0s
5100K … … … … … 65% 47.3M 0s
5150K … … … … … 65% 46.9M 0s
5200K … … … … … 66% 45.1M 0s
5250K … … … … … 66% 48.8M 0s
5300K … … … … … 67% 43.1M 0s
5350K … … … … … 68% 50.5M 0s
5400K … … … … … 68% 38.2M 0s
5450K … … … … … 69% 44.7M 0s
5500K … … … … … 70% 46.8M 0s
5550K … … … … … 70% 59.3M 0s
5600K … … … … … 71% 53.2M 0s
5650K … … … … … 72% 58.3M 0s
5700K … … … … … 72% 64.9M 0s
5750K … … … … … 73% 49.6M 0s
5800K … … … … … 73% 51.3M 0s
5850K … … … … … 74% 62.2M 0s
5900K … … … … … 75% 52.0M 0s
5950K … … … … … 75% 53.0M 0s
6000K … … … … … 76% 63.5M 0s
6050K … … … … … 77% 55.7M 0s
6100K … … … … … 77% 55.7M 0s
6150K … … … … … 78% 63.7M 0s
6200K … … … … … 78% 47.3M 0s
6250K … … … … … 79% 66.5M 0s
6300K … … … … … 80% 66.6M 0s
6350K … … … … … 80% 37.4M 0s
6400K … … … … … 81% 36.7M 0s
6450K … … … … … 82% 12.9M 0s
6500K … … … … … 82% 43.2M 0s
6550K … … … … … 83% 10.6M 0s
6600K … … … … … 84% 5.19M 0s
6650K … … … … … 84% 30.3M 0s
6700K … … … … … 85% 10.1M 0s
6750K … … … … … 85% 31.4M 0s
6800K … … … … … 86% 48.1M 0s
6850K … … … … … 87% 56.2M 0s
6900K … … … … … 87% 53.6M 0s
6950K … … … … … 88% 55.2M 0s
7000K … … … … … 89% 43.0M 0s
7050K … … … … … 89% 64.0M 0s
7100K … … … … … 90% 40.3M 0s
7150K … … … … … 90% 23.4M 0s
7200K … … … … … 91% 47.6M 0s
7250K … … … … … 92% 60.2M 0s
7300K … … … … … 92% 37.5M 0s
7350K … … … … … 93% 47.5M 0s
7400K … … … … … 94% 30.8M 0s
7450K … … … … … 94% 47.6M 0s
7500K … … … … … 95% 56.0M 0s
7550K … … … … … 96% 53.5M 0s
7600K … … … … … 96% 50.6M 0s
7650K … … … … … 97% 60.1M 0s
7700K … … … … … 97% 54.0M 0s
7750K … … … … … 98% 34.4M 0s
7800K … … … … … 99% 29.1M 0s
7850K … … … … … 99% 59.7M 0s
7900K … … 100% 128M=0.7s
2022-04-23 15:08:54 (11.3 MB/s) - ‘/tmp/storagenode-updater.zip’ saved [8106329/8106329]
downloading storagenode
–2022-04-23 15:08:54-- https://version.storj.io/processes/storagenode/minimum/url?os=linux&arch=amd64
Resolving version.storj.io (version.storj.io)… 35.224.88.204
Connecting to version.storj.io (version.storj.io)|35.224.88.204|:443… connected.
HTTP request sent, awaiting response… 200 OK
Length: 84 [text/plain]
Saving to: ‘STDOUT’
0K 100% 38.5M=0s
2022-04-23 15:08:55 (38.5 MB/s) - written to stdout [84/84]
–2022-04-23 15:08:55-- https://github.com/storj/storj/releases/download/v1.52.2/storagenode_linux_amd64.zip
Resolving github.com (github.com)… 140.82.121.3
Connecting to github.com (github.com)|140.82.121.3|:443… connected.
HTTP request sent, awaiting response… 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/1bd10c0f-982c-482a-a490-6e5860e789f2?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220423%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220423T150855Z&X-Amz-Expires=300&X-Amz-Signature=56cae633525247992b018eecd033d1c318254be7a78aebe7fcd102c288e274d4&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode_linux_amd64.zip&response-content-type=application%2Foctet-stream [following]
–2022-04-23 15:08:55-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/1bd10c0f-982c-482a-a490-6e5860e789f2?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220423%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220423T150855Z&X-Amz-Expires=300&X-Amz-Signature=56cae633525247992b018eecd033d1c318254be7a78aebe7fcd102c288e274d4&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode_linux_amd64.zip&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)… 185.199.111.133, 185.199.110.133, 185.199.108.133, …
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.111.133|:443… connected.

и все намертво зависает

Alexey · April 23, 2022, 4:35pm

There is a little bit different problem - the download doesn’t start at all.
But this is a good catch - I would suggest to submit an issue on GitHub to allow our engineers take a look into problem.

На русском

Здесь немного другая проблема - скачивание вообще не начинается. Я не могу воспроизвести проблему даже на сервере в России, где действует масса ограничений доступа к внешним ресурсам.
Но я допускаю, что где-то что-то может не скачиваться нормально, GitHub теперь принадлежит MS…
Это хорошее замечание и я попробую привлечь внимание команды к этой проблеме. Очень помогло бы наличие issue на GitHub

ChristianDresel · April 23, 2022, 5:23pm

same probleme here at some nodes on some machines. Not all but 2 or 3 from 15 Nodes on 2 different hosts, some nodes come back after many restarts, other nodes hang permanently. I don’t know what’s up

root@raspberrypi:/home/pi# docker logs storagenode3
downloading storagenode-updater
–2022-04-23 12:20:53-- https://version.storj.io/processes/storagenode-updater/minimum/url?os=linux&arch=arm
Resolving version.storj.io (version.storj.io)… 35.224.88.204
Connecting to version.storj.io (version.storj.io)|35.224.88.204|:443… connected.
HTTP request sent, awaiting response… 200 OK
Length: 90 [text/plain]
Saving to: ‘STDOUT’

 0K                                                       100% 19.9M=0s

2022-04-23 12:20:54 (19.9 MB/s) - written to stdout [90/90]

–2022-04-23 12:20:54-- https://github.com/storj/storj/releases/download/v1.52.2/storagenode-updater_linux_arm.zip
Resolving github.com (github.com)… 140.82.121.3
Connecting to github.com (github.com)|140.82.121.3|:443… connected.
HTTP request sent, awaiting response… 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/f0f96f9f-3f1d-4fb0-a949-448181cfa573?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220423%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220423T122054Z&X-Amz-Expires=300&X-Amz-Signature=e2b2a0571273e8726a8633f849b2447378aaa26a6d0332d7dfcd0e000c6e90e1&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_arm.zip&response-content-type=application%2Foctet-stream [following]
–2022-04-23 12:20:54-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/f0f96f9f-3f1d-4fb0-a949-448181cfa573?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220423%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220423T122054Z&X-Amz-Expires=300&X-Amz-Signature=e2b2a0571273e8726a8633f849b2447378aaa26a6d0332d7dfcd0e000c6e90e1&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_arm.zip&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)… 185.199.108.133, 185.199.109.133, 185.199.110.133, …
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443… connected.
root@raspberrypi:/home/pi# ^C
root@raspberrypi:/home/pi# docker logs storagenode3
downloading storagenode-updater
–2022-04-23 12:20:53-- https://version.storj.io/processes/storagenode-updater/minimum/url?os=linux&arch=arm
Resolving version.storj.io (version.storj.io)… 35.224.88.204
Connecting to version.storj.io (version.storj.io)|35.224.88.204|:443… connected.
HTTP request sent, awaiting response… 200 OK
Length: 90 [text/plain]
Saving to: ‘STDOUT’

 0K                                                       100% 19.9M=0s

2022-04-23 12:20:54 (19.9 MB/s) - written to stdout [90/90]

–2022-04-23 12:20:54-- https://github.com/storj/storj/releases/download/v1.52.2/storagenode-updater_linux_arm.zip
Resolving github.com (github.com)… 140.82.121.3
Connecting to github.com (github.com)|140.82.121.3|:443… connected.
HTTP request sent, awaiting response… 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/f0f96f9f-3f1d-4fb0-a949-448181cfa573?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220423%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220423T122054Z&X-Amz-Expires=300&X-Amz-Signature=e2b2a0571273e8726a8633f849b2447378aaa26a6d0332d7dfcd0e000c6e90e1&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_arm.zip&response-content-type=application%2Foctet-stream [following]
–2022-04-23 12:20:54-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/f0f96f9f-3f1d-4fb0-a949-448181cfa573?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220423%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220423T122054Z&X-Amz-Expires=300&X-Amz-Signature=e2b2a0571273e8726a8633f849b2447378aaa26a6d0332d7dfcd0e000c6e90e1&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_arm.zip&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)… 185.199.108.133, 185.199.109.133, 185.199.110.133, …
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443… connected.

serger001 · April 23, 2022, 7:29pm

Так нет у нас проблем доступа к гитхабу. Все работает. Ссылка спокойно открывается в браузере. Просто не качается. У ChristianDresel аналогичная проблема/
В моих логах выше он пишет, что пытается скачать обновление. И не скачивает его до конца. Либо вообще не скачивает. Но при этом пытается его распаковать и зависает

Fred · April 23, 2022, 9:41pm

Tried this before and it worked. Here is the output. There is no problem with router, routing, DNS or firewall. I have two identical servers in the same subnet which are working so far. Now i disabled watchtower on my other nodes not to run in the same problem. Wired thing…

root@hp-mic1:/# docker exec -it hp-mic1-n1 wget -O /tmp/storagenode-updater.zip https://github.com/storj/storj/releases/download/v1.52.2/storagenode-updater_linux_amd64.zip
--2022-04-23 21:37:19--  https://github.com/storj/storj/releases/download/v1.52.2/storagenode-updater_linux_amd64.zip
Resolving github.com (github.com)... 140.82.121.3
Connecting to github.com (github.com)|140.82.121.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/7cc525c9-8b78-4199-98e6-861239e6b4f4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220423%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220423T213556Z&X-Amz-Expires=300&X-Amz-Signature=8b626c789ecfe1a36d2c6a544f60e2faf4fb7c9322c8d1fcffd09c49210a70cd&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_amd64.zip&response-content-type=application%2Foctet-stream [following]
--2022-04-23 21:37:19--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/7cc525c9-8b78-4199-98e6-861239e6b4f4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220423%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220423T213556Z&X-Amz-Expires=300&X-Amz-Signature=8b626c789ecfe1a36d2c6a544f60e2faf4fb7c9322c8d1fcffd09c49210a70cd&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_amd64.zip&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 8106329 (7.7M) [application/octet-stream]
Saving to: '/tmp/storagenode-updater.zip'

/tmp/storagenode-updater.zip                        100%[===================================================================================================================>]   7.73M  7.30MB/s    in 1.1s

2022-04-23 21:37:20 (7.30 MB/s) - '/tmp/storagenode-updater.zip' saved [8106329/8106329]

root@hp-mic1:/#

ChristianDresel · April 24, 2022, 4:58am

looks like there are problems with MTU. I have a connection with lower then 1500byte MTU. I don’t know why this probleme only some times and not all the time. I have fix it with

iptables -t mangle -A POSTROUTING -p tcp --tcp-flags SYN,RST SYN -o $IFACE -j TCPMSS --clamp-mss-to-pmtu

on both site from the lower MTU connection

looks like that work now

Alexey · April 24, 2022, 7:04am

@serger001 @Fred
Could you please check your MTU on your connection?
Perhaps solution from @ChristianDresel could help you too?

serger001 · April 24, 2022, 7:28am

MTU в контейнере:

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.17.0.27 netmask 255.255.0.0 broadcast 172.17.255.255
ether 02:42:ac:11:00:1b txqueuelen 0 (Ethernet)
RX packets 150770875 bytes 178108192136 (165.8 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 101176505 bytes 20677294726 (19.2 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1000 (Local Loopback)
RX packets 1741355 bytes 122948945 (117.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1741355 bytes 122948945 (117.2 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

tun0: flags=209<UP,POINTOPOINT,RUNNING,NOARP> mtu 1420
inet 10.0.0.48 netmask 255.255.255.255 destination 10.0.0.48
unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 txqueuelen 0 (UNSPEC)
RX packets 24667508 bytes 27867734792 (25.9 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 16868972 bytes 2540620964 (2.3 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

В хосте MTU 1500.
Я попробовал несколько раз сделать различные манипуляции изнутри контейнера storj.
Команда apt install net-tools выполнилась только со второго раза. В первый раз она зависла аналогично модулю обновления

root@7b9b05485ac4:/app# apt install net-tools
Reading package lists… Done
Building dependency tree
Reading state information… Done
The following NEW packages will be installed:
net-tools
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 248 kB of archives.
After this operation, 1002 kB of additional disk space will be used.
Get:1 Index of /debian buster/main amd64 net-tools amd64 1.60+git20180626.aebd88e-1 [248 kB]
Get:1 Index of /debian buster/main amd64 net-tools amd64 1.60+git20180626.aebd88e-1 [248 kB]
0% [1 net-tools 0 B/248 kB 0%]^C
root@7b9b05485ac4:/app# ^C

Еще я попробовал напрямую скачать архив из контейнера storj
wget https://github.com/storj/storj/releases/download/v1.52.2/storagenode-updater_linux_amd64.zip
и он тоже завис в первый раз. Во второй сработал отлично.

При этом с хостовой машины wget сработал 10 раз из 10, с рабочего компьютера 10 из 10.

Alexey · April 24, 2022, 7:34am

You need to make changes on your host, not in the container.

на русском

Вам нужно внести предложенные изменения на хосте, не в контейнере

serger001 · April 24, 2022, 7:40am

На хосте MTU 1500

docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255
inet6 fe80::42:d1ff:fe59:d011 prefixlen 64 scopeid 0x20
ether 02:42:d1:59:d0:11 txqueuelen 0 (Ethernet)
RX packets 11465690537 bytes 5883542258943 (5.3 TiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 13707002266 bytes 14587920999967 (13.2 TiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

enp7s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 00:25:90:03:17:15 txqueuelen 1000 (Ethernet)
RX packets 40211931444 bytes 42884808398977 (39.0 TiB)
RX errors 9 dropped 242313 overruns 0 frame 7
TX packets 46792136055 bytes 35972426085646 (32.7 TiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 17 memory 0xfafe0000-fb000000

fwbr146i0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 66:f8:96:19:74:a2 txqueuelen 1000 (Ethernet)
RX packets 36100571 bytes 2729648772 (2.5 GiB)
RX errors 0 dropped 4955 overruns 0 frame 0
TX packets 2 bytes 108 (108.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

На хосте несколько докеров, на трафик не смотрите. MTU у всех одинаковое

serger001 · April 24, 2022, 7:45am

Воспроизвел проблему на другом докере. Просто качал один и тот же файл. 3 раза подряд скачалось, на 4й зависло

Alexey · April 24, 2022, 7:45am

Could you please try to apply these changes on your host?

serger001 · April 24, 2022, 7:48am

root@pvest01:~# iptables -t mangle -A POSTROUTING -p tcp --tcp-flags SYN,RST SYN -o $IFACE -j TCPMSS --clamp-mss-to-pmtu
Bad argument TCPMSS' Try iptables -h’ or ‘iptables --help’ for more information.