OCSP Stapling with HAProxy

OCSP stapling was introduced in RFC 2560 back in 1999.  In 2006 RFC 4366 introduced TLS extensions, among which was included the ability to allow the server to send certificate status information as part of the TLS extensions during a TLS handshake.  In July 2013 Mozilla introduced OCSP stapling support in Firefox.

OCSP stapling provides the client with the certificate status immediately and specifically, reducing the latency for the page load by avoiding a separate request to an OCSP service hosted by the issuing CA.  Anyone who's turned on strict OCSP checking in their browser will have observed higher latency while the OCSP check blocks the connection to the site, and depending on the fallback preference the site may not load if the CA's OCSP service isn't responding.  Additionally, using OCSP via the CA's service may be undesirable as it leaks information to the CA about what site the user is visiting.

A lesser known feature of the recent HAProxy 1.5 release, in which SSL/TLS support was introduced, is to support OCSP stapling.  To make use of this feature we need to periodically retrieve the certificate status and provide this information to HAProxy.

HAProxy offers two ways to achieve this, either via static files or by way of the unix socket commands.

For each certificate provided to HAProxy it checks for the presence of another file at the same path suffixed by .ocsp.  It will then serve the content of this file via the TLS extensions when a new client connects.

For example, the following configuration provides a single PEM file that contains the signed certificate, key and intermediate certificates.

frontend https
    bind 9.8.7.6:443 ssl crt /etc/pki/pems/www.example.com.pem no-sslv3

Therefore to allow HAProxy to serve up the certificate status information we expect to see the following files:

/etc/pki/pems/www.example.com.pem
/etc/pki/pems/www.example.com.pem.ocsp

There are three conditions the .ocsp file must satisfy in order to be used by HAProxy:

1. it has to indicate a good status
2. it has to be a single response for the certificate of the PEM file
3. it has to be valid at the moment of addition

It's important to note that this last point requires us to update our .ocsp file regularly because a signed OCSP response will often only be valid for anything from a few hours to a few days.

We can easily automate updating our .ocsp file with the openssl ocsp command.  Before doing this we'll first need to store the issuer certificate for the openssl-ocsp command.  HAProxy is aware of this and will ignore any files with the suffix .issuer, so we'll use this as part of our naming, which means we'll have the following files before we begin.

/etc/pki/pems/www.example.com.pem
/etc/pki/pems/www.example.com.pem.issuer

If the certificate is signed by an intermediate certificate we will have received this with the certificate and that is the only certificate that should be in the .issuer file.

We can then use the following script to automate retrieval of the OCSP response.

#!/bin/sh -e

# Get an OSCP response from the certificates OCSP issuer for use
# with HAProxy, then reload HAProxy if there have been updates.

# Path to certificates
PEMSDIR=/etc/pki/pems

# Path to log output to
LOGDIR=/var/log/haproxy

# Create the log path if it doesn't already exist
[ -d ${LOGDIR} ] || mkdir ${LOGDIR}
UPDATED=0

cd ${PEMSDIR}
for pem in *.pem; do
    echo "= $(date)" >> ${LOGDIR}/${pem}.log

    # Get the OCSP URL from the certificate
    ocsp_url=$(openssl x509 -noout -ocsp_uri -in $pem)

    # Extract the hostname from the OCSP URL
    ocsp_host=$(echo $ocsp_url | cut -d/ -f3)

    # Only process the certificate if we have a .issuer file
    if [ -r ${pem}.issuer ]; then

        # Request the OCSP response from the issuer and store it
        openssl ocsp \
            -issuer ${pem}.issuer \
            -cert ${pem} \
            -url ${ocsp_url} \
            -header Host ${ocsp_host} \
            -respout ${pem}.ocsp >> ${LOGDIR}/${pem}.log 2>&1
    fi
    UPDATED=$(( $UPDATED + 1 ))
done

if [ $UPDATED -gt 0 ]; then
    echo "= $(date) - Updated $UPDATED OCSP responses" >> ${LOGDIR}/${pem}.log
    service haproxy reload > ${LOGDIR}/service-reload.log 2>&1
else
    echo "= $(date) - No updates" >> ${LOGDIR}/${pem}.log
fi 

This script does all we need to make use of static file OCSP stapling.  You can then cron this script to pull in your updates as often as you want.  It's a good idea to also logrotate the output files.

You can test the OCSP response using the openssl s_client:

$ openssl s_client -connect www.example.com:443 -tlsextdebug -status

CONNECTED(00000003)
TLS server extension "renegotiate" (id=65281), len=1
0001 - <SPACES/NULS>
TLS server extension "server ticket" (id=35), len=0
TLS server extension "status request" (id=5), len=0
depth=1 /O=CAcert Inc./OU=http://www.CAcert.org/CN=CAcert Class 3 Root
verify error:num=20:unable to get local issuer certificate
verify return:0
OCSP response:
======================================
OCSP Response Data:
    OCSP Response Status: successful (0x0)
    Response Type: Basic OCSP Response
    Version: 1 (0x0)
    Responder Id: C = AU, ST = NSW, L = Sydney, O = CAcert Inc., OU = Server Administration, CN = ocsp.cacert.org
    Produced At: Mar 21 18:35:14 2015 GMT
    Responses:
    Certificate ID:
      Hash Algorithm: sha1
      Issuer Name Hash: A11F312582B6DA5AD0B98D3A135E35D1EB183661
      Issuer Key Hash: 64C782514C8813F078D98977B56DC589DFBCB17A
      Serial Number: 03FB93
    Cert Status: good
    This Update: Mar 21 18:02:21 2015 GMT
    Next Update: Mar 23 18:35:14 2015 GMT
    ...

The key parts of this output are in bold and give us an indication of how often we'll need to cron our updates.  For example if an update were to fail due to the CA's OCSP responder being offline we'll want at least one or two retries before our OCSP staple expires.

Making use of HAProxy's OCSP stapling support via the command socket improves on this static file approach by avoiding the need for reloading HAProxy.  This method will be covered in a subsequent post.