Since I had such good luck getting HashBackup to work with uplink, I thought I’d try the S3 MT Gateway. My testing is on a very small VM (512M), so I figured using the MT Gateway would keep my memory usage down, even with a bunch of threads. Poor gateway has to handle all the erasure coding!
I’ve run into a couple of incompatibilities with S3. The first is that the gateway doesn’t support regular http connections. I know there can be replay attacks, but it seems a pretty low risk because each S3 request is timestamped and will be rejected after 15 minutes if replayed. Instead of honoring http requests, MTG redirects to https://…
The 2nd problem is:
dest storjs3: unable to start: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:726)
Traceback (most recent call last):
File "/dest.py", line 211, in startdest
File "/s3dest.py", line 362, in init1
File "/opt/lib/python2.7/site-packages/boto/s3/connection.py", line 506, in get_bucket
return self.head_bucket(bucket_name, headers=headers)
File "/opt/lib/python2.7/site-packages/boto/s3/connection.py", line 525, in head_bucket
response = self.make_request('HEAD', bucket_name, headers=headers)
File "/opt/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
retry_handler=retry_handler
File "/opt/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
retry_handler=retry_handler)
File "/opt/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
raise ex
SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:726)
I’m using Python 2.7.15 custom built with openssl 1.02r. No trouble connecting over SSL to AWS, Backblaze, Google Storage, and others. HB uses boto2 with a CA root file from the Requests project. I tried updating the CA file to the latest, but that didn’t help. I also checked the SSLLabs report for the MT Gateway and they can connect okay with openssl 1.0.2s, 1 rev higher.
I did some debugging output with the openssl command. This is without -servername (no SNI):
[jim@mb openssl-1.0.2r]$ ./apps/openssl s_client -connect gateway.us1.storjshare.io:443 -CAfile ~jim/hbrel/cacerts.crt
WARNING: can't open config file: /usr/local/ssl/openssl.cnf
CONNECTED(00000003)
depth=0 CN = TRAEFIK DEFAULT CERT
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = TRAEFIK DEFAULT CERT
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
0 s:/CN=TRAEFIK DEFAULT CERT
i:/CN=TRAEFIK DEFAULT CERT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDXjCCAkagAwIBAgIRAL4c/zsOHFhtOI4tbEZBLZUwDQYJKoZIhvcNAQELBQAw
HzEdMBsGA1UEAxMUVFJBRUZJSyBERUZBVUxUIENFUlQwHhcNMjEwODEyMDMwMDE3
WhcNMjIwODEyMDMwMDE3WjAfMR0wGwYDVQQDExRUUkFFRklLIERFRkFVTFQgQ0VS
VDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAN6Q4JdUb4+wPZRyoK9H
a6S+74tVrfWDHu5wc+NDpXqojIPS2vcc/9UhqJjtBYICVnQ+JGtJmhOs6jYwCUTW
idQAIwTKjaAM8nRD8Wl6SU+47zA1sY2cYMH3cNtE5XAq6DNblww06HQnF40aij7g
V1tqEwLxaksyeobSTZm+fsTb093RieHzzSw5vSVC8qzyeqAAIeOm+Hs953wGHXo6
WGzEwVBqTfmAv46eYnX7oBVa1ujI+d7D/JsdpS001bMbf5f/erqmf77jMn/UnBdT
TJqpCvEJkMqnn7v2iydQnoQMPITpifYS6BqEANRf4fkxg+jELUcrlFsPrEDbuxrv
PcUCAwEAAaOBlDCBkTAOBgNVHQ8BAf8EBAMCA7gwEwYDVR0lBAwwCgYIKwYBBQUH
AwEwDAYDVR0TAQH/BAIwADBcBgNVHREEVTBTglFkYjQwOTNmMmUyYWNiODg2NDhl
ZjA0NjNmZGRkNDBlMy5hYjBmZTE4NjZkYTk0MGJmMzU3MjlhYzVlNGRhOWI0NC50
cmFlZmlrLmRlZmF1bHQwDQYJKoZIhvcNAQELBQADggEBACTNHuSZUn+e7Qqu20BM
MPXPMGw4E0TTZUeMNc8qeyKJeH+Bsqe7F6otXRtnwOyaVn4jf0KwKFeXk30EnyMC
1rSfs5q3wgEvcNn1/V1w/FPI3OL2k4zPiGaoAgvv7D65T2pSYsxarMJ2E9Fyjtzv
LDRL128c2AaCmbbeQe0ZOj8V5KjLapEhwaNSAS4adi047h3gwrh46SmP0nKmdZeK
RGUtRtjJqEnHOOfwqsDPnzxYi263Gk3aGuBF7HYMe5JBRyTgtdH/3GsDyR3zhg5E
VChBpamdEiAGo1TY0ma4TJfQ8VZoy9HGYtedcMARmIPOf8zZIPSBo1WNK4UBpTi5
tbU=
-----END CERTIFICATE-----
subject=/CN=TRAEFIK DEFAULT CERT
issuer=/CN=TRAEFIK DEFAULT CERT
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1487 bytes and written 437 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 7D4AFE385D7D667D0E642C94344F0C8BC4E1C7469403480616BCB541B8B3CA33
Session-ID-ctx:
Master-Key: DB14B955F28E5449B6A8B34FCA24CD7A4D2D7FFEDFE315AD83B321B4EA4ACBB4EA1E2B7A3444C175B945420BC94C9EAC
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket:
0000 - 81 3c d6 47 12 d6 a0 c1-08 b8 6d f5 b0 54 41 b4 .<.G......m..TA.
0010 - 03 05 ef 95 f5 ce b3 fc-45 37 ba a9 ff a7 5c 15 ........E7....\.
0020 - 52 ab 09 f1 2b b2 5b 09-00 3b a6 bc 93 98 0b 32 R...+.[..;.....2
0030 - 2a 7b 84 8b c4 ab 45 74-3e f8 16 ae 42 1d 19 18 *{....Et>...B...
0040 - 19 2f 89 c9 00 ed 87 6b-0d be 9b af 6c 59 b0 c1 ./.....k....lY..
0050 - 37 bc 0d 4e 1c d6 6f 37-3c ba 78 3d 32 0d 6f 76 7..N..o7<.x=2.ov
0060 - 61 5f bd 9f b2 6e 6b 59-f9 4c 40 1d 55 0c fa 14 a_...nkY.L@.U...
0070 - c9 4b 8f ef ac 22 b7 68-d4 fb e5 7d 57 e6 ed c3 .K...".h...}W...
0080 - de .
Start Time: 1632173533
Timeout : 300 (sec)
Verify return code: 21 (unable to verify the first certificate)
---
^C
Here’s with -servername:
[jim@mb openssl-1.0.2r]$ ./apps/openssl s_client -connect gateway.us1.storjshare.io:443 -servername gateway.us1.storjshare.io -CAfile ~jim/hbrel/cacerts.crt
WARNING: can't open config file: /usr/local/ssl/openssl.cnf
CONNECTED(00000003)
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = gateway.us1.storjshare.io
verify return:1
---
Certificate chain
0 s:/CN=gateway.us1.storjshare.io
i:/C=US/O=Let's Encrypt/CN=R3
1 s:/C=US/O=Let's Encrypt/CN=R3
i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGVzCCBT+gAwIBAgISBD8bIMrOeA7Zzqyq7fclqaDcMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTA4MTIwMTU5NDlaFw0yMTExMTAwMTU5NDdaMCQxIjAgBgNVBAMT
GWdhdGV3YXkudXMxLnN0b3Jqc2hhcmUuaW8wggIiMA0GCSqGSIb3DQEBAQUAA4IC
DwAwggIKAoICAQDmlUmwthVA4r709du7ZhWtBmfe7u7jrSmTOeygLy1TGJWJpecN
RM5ETPiv3qJvWmKPfZY/Uk1aLLpUhyd1tumEbFkracRZzq4ZGzdgH7QA9Q/O/aVN
PPLpGxgyZnDjAofDja4r72b1Mo+aLPujfJaAfxZQ9UJ5gVR4Qkwhd5FxSzZZZIjD
3wS41KmCRRl6nXaAzygRcGKMDPUQs26y4r1mAggWtULXwveUwpvlZXYh5wYzLg1y
0PiLmlUIwZgroATg3VTa4I6CDUNETa8iKd6LGMNhMWYol5i//7zARnhwUNkh7amu
+ztXPVd7dB9cYVMmjqi/w6MLJZYSCL+Hxbpxr4RGrmvOCZNtSOE3+aAjPSDKw9cd
gmTRL2bNuTiq45n1/i6df7zQx0xCdemAgIYjV6Qe3Rv5pZbTmgVTRfVHZsF9Kyov
qInkhioz6mv2UIlC23nimj0JFvOxplwInbQ5tFhl8h2DJVz5UwSzd1rQglMytBga
JQneEMvzUw/PJR+sI5NC85NoOh6T4stL2bujqJmXR0Nb2x5ULQM3SUkkwIaooalr
SK1PIXzAnqXIITQRR6/QinEgh7d4Edrov8+3bgpJ5F/mbxi4itvbThKITGPQvUJy
/VyftM0sQ0bWO7tr55eKD5gyWeRZQOyQck5Rp7PQo/mnszBeHUJow+tpCwIDAQAB
o4ICczCCAm8wDgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggr
BgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBS+6EaZ5gcFqPrjpGEQfglz
w+5dizAfBgNVHSMEGDAWgBQULrMXt1hWy65QCUDmH6+dixTCxjBVBggrBgEFBQcB
AQRJMEcwIQYIKwYBBQUHMAGGFWh0dHA6Ly9yMy5vLmxlbmNyLm9yZzAiBggrBgEF
BQcwAoYWaHR0cDovL3IzLmkubGVuY3Iub3JnLzBBBgNVHREEOjA4ghsqLmdhdGV3
YXkudXMxLnN0b3Jqc2hhcmUuaW+CGWdhdGV3YXkudXMxLnN0b3Jqc2hhcmUuaW8w
TAYDVR0gBEUwQzAIBgZngQwBAgEwNwYLKwYBBAGC3xMBAQEwKDAmBggrBgEFBQcC
ARYaaHR0cDovL2Nwcy5sZXRzZW5jcnlwdC5vcmcwggEGBgorBgEEAdZ5AgQCBIH3
BIH0APIAdwBc3EOS/uarRUSxXprUVuYQN/vV+kfcoXOUsl7m9scOygAAAXs4TTZD
AAAEAwBIMEYCIQD08vuK2HLE+rzHS5G3oawEWJ9/6QvC1OBxg9RgE4qyDQIhAJSc
AfQxmZwD0Xl3b887hrd+eqcCwmEy3Mrbmiwy5GpyAHcAfT7y+I//iFVoJMLAyp5S
iXkrxQ54CX8uapdomX4i8NcAAAF7OE02YAAABAMASDBGAiEAkMs3//wTznY30Gqo
SeKhFpQbrtEhq16O95zQdyQOXEUCIQC/+om7mZlfAKGmazaqFvUZXaYzrDCflhuA
TOehid4kXTANBgkqhkiG9w0BAQsFAAOCAQEAe8CnYsaUYajX9ghF9PR2vLwKamxg
NeCzqdPwaRImwu5diKg3oc8p4494WhsS3fGDWDToMS/UoSN+0xfOCxtJk1eyY2Cf
P+Pf8cLbyeHrqqN4fChY1enVKwn+d3pGBy04rbWSfkaF67VlHU0m73LDvBZBzbPf
hrmxKEh7hmMuVQHCbvTZs56aix2VpS7g3oRvIz3mrszSANqu3PtadQYM0E+OPFtH
40fEnzwDP0yW8zv0oJ/z4HBNuOiq3b1k+vsqSB+3yzLnRp27iUgtLGsS7A9c6Szq
Rclvd9vmrIYKhPMzBZuvO5GJUq1WHX9yp/+Vq3Mjrg63y87DR+to4cfz4g==
-----END CERTIFICATE-----
subject=/CN=gateway.us1.storjshare.io
issuer=/C=US/O=Let's Encrypt/CN=R3
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 5196 bytes and written 471 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: B767870DA912E14DBB6A12933CDD49C0CA815DA003CBB54E868B0B5249ACDF8B
Session-ID-ctx:
Master-Key: 60EE0C205D2C6C7AA326B3658904333893F06E9144DA32547E566648DA553950CBA32E101E126A69D361A6F35652C3E9
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket:
0000 - 3a a9 96 7e 13 40 b7 d1-45 8a b4 aa 47 80 c8 ff :..~.@..E...G...
0010 - ae 6d 56 30 a3 0e dd e8-ff be a0 1c 61 da ad b4 .mV0........a...
0020 - d7 ab 43 d2 b3 d1 92 9d-b2 88 8e 20 2f 2d e4 1e ..C........ /-..
0030 - 30 2c ea 15 22 32 ad 50-e2 95 02 ff b9 20 35 48 0,.."2.P..... 5H
0040 - ca 4c 23 ff 48 13 25 ff-e2 c9 ff 9e e9 73 4f d1 .L#.H.%......sO.
0050 - d1 b6 a1 ee ce d1 4d dc-d6 f1 91 49 dc 88 01 43 ......M....I...C
0060 - a6 00 13 6a ea d8 d4 6d-88 af d1 76 5b c8 20 28 ...j...m...v[. (
0070 - dd bc 0d ce 29 44 e3 1c-dc 0b a3 43 a0 d4 f3 76 ....)D.....C...v
0080 - 42 B
Start Time: 1632173400
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
^C
So it looks like Python clients using boto2 can’t connect with the MT Gateway because boto2 doesn’t support SNI, though AWS connections work fine.
Footnotes:
Why still using Python 2? It’s faster than Python 3 in my tests and I don’t want to look through 100K+ lines of code for incompatibilies.
Why using boto2 instead of boto3? With boto3, Amazon uses a JSON file to define the API, then builds classes at runtime. This doesn’t play well with a static executable like HB.
Update: if there was only 1 server configured at gateway.us1.storjshare.io, I don’t think SNI would be required. This would let boto2 applications work on the gateway. Enabling non-SSL http on port 80 would also probably work.
Jim
