Dynamic SSL certificates using LetsEncrypt on Openshift

Managing SSL certificates in Openshift can be a bit of a chore, especially when you have more than a few routes. Using LetsEncrypt helps to simplify this.

Dynamic SSL certificates using LetsEncrypt on Openshift

Managing SSL certificates in OpenShift can be a bit of a chore, especially when you have more than a few routes to manage. Having an automated mechanism to manage this helps with the operational overhead, and in this example LetsEncrypt is the weapon of choice.

You could quite conveniently use a wild card certificate to cover most of your routes but that doesn't cover every use case that you might have, especially when you manage multiple domains. Consider also that wildcard certificates are deprecated[1] in favour of tooling that can provide programmatic access to easily create and renew SSL certificates on demand. There are a bunch of advantages (and disadvantages) to this and a tonne of articles out there, already covering the nuts and bolts of that topic, so I'm going to skip over that and instead share my experience deploying and using LetsEncrypt on OpenShift.

LetsEncrypt has been around for a while now and has been adopted into many environments so I thought it is about time that I shared how I have applied Lets Encrypt to solve my problem managing certificates across multiple domains on my OpenShift cluster.

Automated SSL Certification Authority (LetsEncrypt)

Lets Encrypt is a free, automated and open certification authority based on the ACME standard and is a service operated by the Internet Security Research Group and provides free, secure SSL certificates to the world. Lets Encrypt uses the ACME protocol to distribute certificates using tooling to manage the request, issuance, renewal and expiration of certificates.

Initially I thought about using a similar approach to Mak in his blog post covering the usage of the ACME client, and I have for another use case.  He describes using an ACME client to request a wildcard certificate using the tool at the cli, and then to update the certificate for the default route certificate on the OpenShift router. This is very convenient for use cases where you predominantly serve routes that are children of the same subdomain - for example *.apps.example.com. This is useful for tools like Red Hat CodeReady Workspaces/Eclipse Che which produce routes based on this pattern. With this method you do still need to craft a process (or remember) to renew the certificate and inject your freshly minted renewed certificate into the router. This led me on to looking at one of the suggestions in the comments of that article, the openshift-acme project on GitHub by Tomáš Nožička.

The OpenShift ACME Operator

The project that I used to automate the issuance of my certificates, provides an operator which once deployed into your cluster, watches for annotations on routes, and issues certificates according to those annotates across all namespaces. The installation of the Operator was very easy, and I followed the process for a 'live cluster-wide' implementation.

We will be deploying 2 components to make this happen:
- an Operator
- an annotation on the route

To deploy the operator into your cluster, run:

oc apply -fhttps://raw.githubusercontent.com/tnozicka/openshift-acme/master/deploy/cluster-wide/{clusterrole,serviceaccount,issuer-letsencrypt-live,deployment}.yaml
oc create clusterrolebinding openshift-acme --clusterrole=openshift-acme --serviceaccount="$( oc project -q ):openshift-acme" --dry-run -o yaml | oc apply -f -
Running the deployment gives us an operator deployed into the prod-letsencrypt namespace

Great, so you now have a new pod running in your cluster which creates SSL certificates on demand, how do you use it? Easy!

Issuing a certificate

Start off by deploying an application. If you haven't got something already, the CakePHP application template is a good place to start.

oc new-project example-project
oc new-app cakephp-ephemeral
Create a secure route

To issue a certificate simply add this annotation to your route:

    kubernetes.io/tls-acme: "true"
Adding the annotation to the route to trigger certificate generation

What happens next is that a path-based route is created along side the route which you are requesting a certificate for. Something like /.well-known/ib78by87oby78y8788 which LetsEncrypt validates using a HTTP-01 challenge.

Note: DNS-01 challenges are required for wildcard certificates, which are currently not supported by this operator.

The validation route is created and removed again, once validation passes

The freshly generated validation route will be removed once the CA has processed the request and the certificate has been successfully installed.


There are currently a couple of limitations with this Operator, but these wouldn't stop me from using it in anger.

1) Validation routes are created on the default router which means that if you run your routers in a sharded mode, for example where you have a separate router to handle internal traffic (think management and development workloads).

Sharding in this way means that Lets Encrypt is unable to validate your ownership and issue the certificate. It's worth noting that at the time of writing there is a PR in progress to resolve this issue: (https://github.com/tnozicka/openshift-acme/pull/69).

2) There is also not yet support for wildcard SSL certificates, but you can also follow development of that feature here (https://github.com/tnozicka/openshift-acme/issues/61). This is a limitation of the validation mechanism that is being used for routes.


So there you have it, it's actually really simple to deploy the LetsEncrypt operator and use it t0 generate SSL certificates on demand. If you're wondering, the certificates are automatically re-issued when they get close to their expiry too! It's always a treat when you can reduce some of your operational overhead.

Mak's post about using the ACME client: https://blog.openshift.com/create-https-based-encrypted-urls-using-routes/
More information about LetsEncrypt: https://letsencrypt.org/how-it-works/
The foundation behind LetsEncrypt: https://www.abetterinternet.org/

[1] RFC 6125: https://tools.ietf.org/html/rfc6125#section-7.2
[2] Wildcard SSL certificates with FreeIPA: https://frasertweedale.github.io/blog-redhat/posts/2017-06-26-freeipa-wildcard-san.html
[3] Wildcard certificates with FreeIPA: https://www.freeipa.org/page/Howto/Wildcard_certificates
[4] Lets Encrypt ACME challenge mechanisms: https://letsencrypt.readthedocs.io/en/latest/challenges.html#dns-01-challenge