path: root/posts/curl-and-the-tls-sni-extension.rst

.. title: cURL and the TLS SNI extension
.. slug: curl-and-the-tls-sni-extension
.. date: 2016-08-01 12:18:13 UTC+02:00
.. tags:
.. link:
.. description: How to customize the Host header with TLS SNI extension.
.. type: text

Let's say that you are hosting multiple websites on the same port of a single
machine which IPv4 address is 176.31.99.217. Both websites must be accessible
via HTTP and HTTPS.

You possess the NS domain *example.org* and it points to your machine.

You are using virtualhosts to manage your differents websites. i.e. the *Host*
HTTP header is analysed by your webserver so it can make the decision of which
website it is going to serve.

However, these virtualhosts are not pointing to your machine yet. Only
*example.org* does.

Here is an excerpt of a nginx configuration using three virtualhosts:

* example.org (it will be served by default because of the **default_server** directive),
* vhost1.example.org,
* vhost2.example.org

.. code-block:: nginx

    server {
        listen 0.0.0.0:80   default_server;
        listen 0.0.0.0:443  ssl default_server;

        server_name         example.org;

        root                /srv/www/example.org/www;

        ssl_certificate     ssl/example.org.crt;
        ssl_certificate_key ssl/example.org.key;

        try_files $uri $uri/ =404;
    }

    server {
        listen 0.0.0.0:80;
        listen 0.0.0.0:443  ssl;

        server_name         vhost1.example.org;

        root                /srv/www/vhost1.example.org/www;

        ssl_certificate     ssl/vhost1.example.org.crt;
        ssl_certificate_key ssl/vhost1.example.org.key;

        try_files $uri $uri/ =404;
    }

    server {
        listen 0.0.0.0:80;
        listen 0.0.0.0:443  ssl;

        server_name         vhost2.example.org;

        root                /srv/www/vhost2.example.org/www;

        ssl_certificate     ssl/vhost2.example.org.crt;
        ssl_certificate_key ssl/vhost2.example.org.key;

        try_files $uri $uri/ =404;
    }

For testing purposes:

* the file */srv/www/example.org/www/index.html* contains the string **default**.
* the file */srv/www/vhost1.example.org/www/index.html* contains the string **vhost1**.
* the file */srv/www/vhost2.example.org/www/index.html* contains the string **vhost2**.

HTTP
----

Let's try that out with HTTP::

    $ curl http://example.org
    default

    $ curl http://176.31.99.217
    default

The default server is served because 172.31.99.217 does not match any
virtualhost *server_name* directive::

    $ curl http://vhost1.example.org
    curl: (6) Could not resolve host: vhost1.example.org
    $ curl http://vhost2.example.org
    curl: (6) Could not resolve host: vhost2.example.org

Since the DNS does not know about vhost1.example.org and vhost2.example.org, we
can't test our websites this way.
We are offered (at least) two possibilities:

* adding **vhost1.example.org** and **vhost2.example.org** to the file */etc/hosts*.
  Example::

        $ cat /etc/hosts
        127.0.0.1       localhost
        176.31.99.217   vhost1.example.org
        176.31.99.217   vhost2.example.org

        $ curl http://vhost1.example.org
        vhost1
        $ curl http://vhost2.example.org
        vhost2

* specifying the **Host** header manually using the option **-H** of cURL. This
  is the recommended way in most cases.
  Example::

        $ curl http://176.31.99.217 -H 'Host: vhost1.example.org'
        vhost1
        $ curl http://176.31.99.217 -H 'Host: vhost2.example.org'
        vhost2

  The **Host** header has been placed in the HTTP request. We can verify this way::

        $ curl -v http://176.31.99.217 -H 'Host: vhost1.example.org'
        * Rebuilt URL to: http://176.31.99.217/
        * Hostname was NOT found in DNS cache
        *   Trying 176.31.99.217...
        * Connected to 176.31.99.217 (176.31.99.217) port 80 (#0)
        > GET / HTTP/1.1
        > User-Agent: curl/7.38.0
        > Accept: */*
        > Host: vhost1.example.org


HTTPS
-----

Now what happens if we use the HTTPS version ?::

    $ curl https://example.org
    default

    $ curl https://176.31.99.217
    curl: (51) SSL: certificate subject name 'example.org' does not match target host name '176.31.99.217'

The second command fails because the default webserver answered with its
certificate, but it is not valid for *176.31.99.217*, only *example.org*.

This is a normal behaviour. We can solve this issue by creating a certificate
valid for *example.org* and *176.31.99.217* for example.

But what about requesting our virtualhosts ? Well, pretty much the same as
before (unless you didn't clear your */etc/hosts* file.::

    $ curl https://vhost1.example.org
    curl: (6) Could not resolve host: vhost1.example.org
    $ curl https://vhost2.example.org
    curl: (6) Could not resolve host: vhost2.example.org

The good news is: it will still work if you modify your */etc/hosts* file (huray)::

        $ cat /etc/hosts
        127.0.0.1       localhost
        176.31.99.217   vhost1.example.org
        176.31.99.217   vhost2.example.org

        $ curl https://vhost1.example.org
        vhost1
        $ curl https://vhost2.example.org
        vhost2

And the bad news is::

        $ curl https://176.31.99.217 -H 'Host: vhost1.example.org'
        curl: (51) SSL: certificate subject name 'example.org' does not match target host name '176.31.99.217'
        $ curl https://176.31.99.217 -H 'Host: vhost2.example.org'
        curl: (51) SSL: certificate subject name 'example.org' does not match target host name '176.31.99.217'

So why does it fail ?

Well, as we showed previously, the *Host* header is analysed to figure out which
website to serve. Nevertheless, when HTTPS is being used, the first thing that
the user agent and the server have to do is negociate the certificate. At this
point, the HTTP headers have just NOT been sent yet.

So how does the webserver decides which certificate to send ?

.. note:: 10 years ago, it was simply not possible. The webservers had to send the same certificate for every virtualhost behind the same address and port.

Nowadays, there is a TLS extension, SNI (which stands for Server Name
Indication) that is supported by most browsers and operating systems. The TLS
stack present in Windows XP does not support it. As a consequence, Internet
Explorer under Windows XP cannot use SNI. But most other browsers use their own
TLS stack and can thus use SNI even under XP.

.. note:: See more on Wikipedia: https://en.wikipedia.org/wiki/Server_Name_Indication

Briefly, the extension works by sending the server name along with the first
TLS packet. This way, the remote server knows which certificate to reply with.
Only then, after the certificate has been negociated, that the server can
analyse the *Host* header.

.. note:: The host header does not require to match with the server name that has been sent in the TLS packet.

So what can we do ?

* We can still use the */etc/hosts* file as previously stated.
* We can also use the **--resolve** option of cURL (which does pretty much the
  same thing as modyfiying the */etc/hosts* file (the 443 is the port being
  used): example::

        $ curl https://vhost1.example.org --resolve vhost1.example.org:443:176.31.99.217
        > vhost1
        $ curl https://vhost2.example.org --resolve vhost2.example.org:443:176.31.99.217
        > vhost2

* Or we can negociate one certificate that we know is good and then use the
  *Host* header to get the content of another website::

        $ curl https://example.org -H 'Host: vhost1.example.org'
        > vhost1
        $ curl https://example.org -H 'Host: vhost2.example.org'
        > vhost2

  We just negociated the certificate of *example.org* and got the content of
  another virtualhost ! We can verify::

        $ curl -v https://example.org -H 'Host: vhost1.example.org'
        ...
        > * Server certificate:
        ...
        > *     common name: example.org (matched)
        ...
        > *     SSL certificate verify ok.
        > ...
        > ...
        vhost1

That's all folks.