Let's Encrypt for Kubernetes

A while ago, I blogged about automatic Let's Encrypt certificate renewal with nginx. Since then, I've also set up renewal in our Kubernetes cluster.

Like with nginx, I'm using acme-tiny to do the renewals. For Kubernetes, I created a Docker image. It reads the Let's Encrypt secret key from /etc/acme-tiny/secrets/account.key, and CSR files from /etc/acme-tiny/csrs/{name}.csr. In Kubernetes, these can be set up by mounting a Secrets store and a ConfigMap, respectively. It also reads the current certificates from /etc/acme-tiny/certs/{name}, which should also be set up by mounting a ConfigMap (called certificates), since that is where the container will put the new certificates.

Starting an acme-tiny pod will start an nginx server to store the .well-known directory for the Acme challenge. Running /opt/acme-tiny-utils/renew in the pod will renew the certificate if it will expire within 20 days (running it with the -f option will disable the check). Of course, we want the renewal to be automated, so we want to set up a sort of cron task. Kubernetes has cron jobs since 1.4, but at the time I was setting this up, we were still on 1.3. Kubernetes also does cron jobs by creating a new pod, whereas the way I want this to work is to run a program in an existing pod (though it could be set up to work the other way too). So I have another cron Docker image, which I have set up to run

kubectl exec `kubectl get pods --namespace=lb -l role=acme -o name | cut -d / -f 2` --namespace=lb ./renew sbscalculus.com

every day. That command finds the acme-tiny pod and runs the renew command, telling it to renew the sbscalculus.com certificate.

Now in order for the Acme challenge to work, HTTP requests to /.well-known/acme-challenge/ get redirected to acme-tiny rather than to the regular pods serving those services. Our services are behind our HAProxy image. So I have a 0acmetiny entry (the 0 causes it to be sorted before all other entries) in the services ConfigMap for HAProxy that reads:

    {
      "namespace": "lb",
      "selector": {
        "role": "acme"
      },
      "hostnames": ["^.*"],
      "path": "^/\\.well-known/acme-challenge/.*$",
      "ports": [80]
    }

This causes HAProxy to all the Acme challeges to the acme-tiny pod, while leaving all the other requests alone.

And that's how we have our certificates automatically renewed from Let's Encrypt.