summaryrefslogtreecommitdiff
path: root/posts/curl-and-the-tls-sni-extension.rst
blob: 7c2950a7ba484a463e86d0db2d6ec4d90f7b1f60 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
.. title: cURL and the TLS SNI extension
.. slug: curl-and-the-tls-sni-extension
.. date: 2016-08-01 12:18:13 UTC+02:00
.. tags:
.. link:
.. description: How to customize the Host header with TLS SNI extension.
.. type: text

Let's say that you are hosting multiple websites on the same port of a single
machine which IPv4 address is 176.31.99.217. Both websites must be accessible
via HTTP and HTTPS.

You own the NS domain *example.org* and it points to your machine.

You are using virtualhosts to manage your differents websites. i.e. the *Host*
HTTP header is analysed by your webserver so it can make the decision of which
website it is going to serve.

However, these virtualhosts are not pointing to your machine yet. Only
*example.org* does.

Here is an excerpt of a nginx configuration using three virtualhosts:

* example.org (it will be served by default because of the **default_server** directive),
* vhost1.example.org,
* vhost2.example.org

.. code-block:: nginx

    server {
        listen 0.0.0.0:80   default_server;
        listen 0.0.0.0:443  ssl default_server;

        server_name         example.org;

        root                /srv/www/example.org/www;

        ssl_certificate     ssl/example.org.crt;
        ssl_certificate_key ssl/example.org.key;

        try_files $uri $uri/ =404;
    }

    server {
        listen 0.0.0.0:80;
        listen 0.0.0.0:443  ssl;

        server_name         vhost1.example.org;

        root                /srv/www/vhost1.example.org/www;

        ssl_certificate     ssl/vhost1.example.org.crt;
        ssl_certificate_key ssl/vhost1.example.org.key;

        try_files $uri $uri/ =404;
    }

    server {
        listen 0.0.0.0:80;
        listen 0.0.0.0:443  ssl;

        server_name         vhost2.example.org;

        root                /srv/www/vhost2.example.org/www;

        ssl_certificate     ssl/vhost2.example.org.crt;
        ssl_certificate_key ssl/vhost2.example.org.key;

        try_files $uri $uri/ =404;
    }

For testing purposes:

* the file */srv/www/example.org/www/index.html* contains the string **default**.
* the file */srv/www/vhost1.example.org/www/index.html* contains the string **vhost1**.
* the file */srv/www/vhost2.example.org/www/index.html* contains the string **vhost2**.

HTTP
----

Let's try that out with HTTP::

    $ curl http://example.org
    default

    $ curl http://176.31.99.217
    default

The default website is served because 172.31.99.217 does not match any
virtualhost *server_name* directive::

    $ curl http://vhost1.example.org
    curl: (6) Could not resolve host: vhost1.example.org
    $ curl http://vhost2.example.org
    curl: (6) Could not resolve host: vhost2.example.org

Since the DNS does not know about vhost1.example.org and vhost2.example.org, we
can't test our websites this way.
We are offered (at least) two possibilities:

* adding **vhost1.example.org** and **vhost2.example.org** to the file */etc/hosts*.
  Example::

        $ cat /etc/hosts
        127.0.0.1       localhost
        176.31.99.217   vhost1.example.org
        176.31.99.217   vhost2.example.org

        $ curl http://vhost1.example.org
        vhost1
        $ curl http://vhost2.example.org
        vhost2

* specifying the **Host** header manually using the option **-H** of cURL. This
  is my favorite way in most cases.
  Example::

        $ curl http://176.31.99.217 -H 'Host: vhost1.example.org'
        vhost1
        $ curl http://176.31.99.217 -H 'Host: vhost2.example.org'
        vhost2

  The **Host** header has been placed in the HTTP request. We can verify this way::

        $ curl -v http://176.31.99.217 -H 'Host: vhost1.example.org'
        * Rebuilt URL to: http://176.31.99.217/
        * Hostname was NOT found in DNS cache
        *   Trying 176.31.99.217...
        * Connected to 176.31.99.217 (176.31.99.217) port 80 (#0)
        > GET / HTTP/1.1
        > User-Agent: curl/7.38.0
        > Accept: */*
        > Host: vhost1.example.org


HTTPS
-----

Now what happens if we use the HTTPS version ?::

    $ curl https://example.org
    default

    $ curl https://176.31.99.217
    curl: (51) SSL: certificate subject name 'example.org' does not match target host name '176.31.99.217'

The second command fails because the default webserver answered with its
certificate, but it is not valid for *176.31.99.217*, only *example.org*.

This is a normal behaviour. We can solve this issue by creating a certificate
valid for *example.org* and *176.31.99.217* for example.

But what about requesting our virtualhosts ? Well, pretty much the same happens
as it did before (unless you didn't clear your */etc/hosts* file).::

    $ curl https://vhost1.example.org
    curl: (6) Could not resolve host: vhost1.example.org
    $ curl https://vhost2.example.org
    curl: (6) Could not resolve host: vhost2.example.org

The good news is: it will still work if you modify your */etc/hosts* file (huray)::

    $ cat /etc/hosts
    127.0.0.1       localhost
    176.31.99.217   vhost1.example.org
    176.31.99.217   vhost2.example.org

    $ curl https://vhost1.example.org
    vhost1
    $ curl https://vhost2.example.org
    vhost2

And the bad news is::

    $ curl https://176.31.99.217 -H 'Host: vhost1.example.org'
    curl: (51) SSL: certificate subject name 'example.org' does not match target host name '176.31.99.217'
    $ curl https://176.31.99.217 -H 'Host: vhost2.example.org'
    curl: (51) SSL: certificate subject name 'example.org' does not match target host name '176.31.99.217'

So why does it fail ?

Well, as we showed previously, the *Host* header is analysed to figure out which
website to serve. Nevertheless, when HTTPS is being used, the first thing that
the user agent and the server have to do is negociate the certificate. At this
point, the HTTP headers have just NOT been sent yet.

So how does the webserver decide which certificate to send ?

.. note:: 10 years ago, that was simply not possible. The webservers had to send the same certificate for every virtualhost behind the same address and port.

Nowadays, there is a TLS extension named **SNI**, which stands for Server
Name Indication, which has been created to address this issue. SNI is supported
by most browsers and operating systems. However, the TLS stack present in
Windows XP does not support it. As a consequence, Internet Explorer under
Windows XP cannot use SNI. But most other browsers use their own
TLS stack and can thus use SNI even under XP.

.. note:: See more on Wikipedia: https://en.wikipedia.org/wiki/Server_Name_Indication

Briefly, the extension works by sending the server name along with the first
TLS packet. This way, the remote server knows which certificate to reply with.
Only then, after the certificate has been negociated, the server can receives
the HTTP headers and is able to analyse the *Host* header.

.. note:: The host header does not require to match with the server name that has been sent in the TLS packet.

So what can we do ?

* We can still use the */etc/hosts* file as previously stated.
* We can also use the **--resolve** option of cURL (which does pretty much the
  same thing as modyfiying the */etc/hosts* file). The 443 is the port being
  used: example::

        $ curl https://vhost1.example.org --resolve vhost1.example.org:443:176.31.99.217
        > vhost1
        $ curl https://vhost2.example.org --resolve vhost2.example.org:443:176.31.99.217
        > vhost2

* Or we can negociate one certificate that we know is good and then use the
  *Host* header to get the content of another website::

        $ curl https://example.org -H 'Host: vhost1.example.org'
        > vhost1
        $ curl https://example.org -H 'Host: vhost2.example.org'
        > vhost2

  We just negociated the certificate of *example.org* and got the content of
  another virtualhost ! We can verify::

        $ curl -v https://example.org -H 'Host: vhost1.example.org'
        ...
        > * Server certificate:
        ...
        > *     common name: example.org (matched)
        ...
        > *     SSL certificate verify ok.
        > ...
        > ...
        vhost1

That's all folks.