3. SSL/Cert-manager
Last updated
Last updated
All content on this page by eGov Foundation is licensed under a Creative Commons Attribution 4.0 International License.
To get free and Automatic SSL certificates using Cert manager and Let’s Encrypt
In today's scenario, SSL certificates are the most important part of Deploying an application to the Internet. It is only the most important attributes that determine whether your websites are safe or not.
This paddle lock symbol conveys to your customer that the website they are visiting is safe, secured, and verified. So how do you actually achieve HTTPS on your website?
HTTP + SSL = HTTPS.
Getting an SSL certificate is not that easy. Moreover, it is expensive too. In today's world where Kubernetes is Predominantly everywhere. And with tonnes of Ingress resources in Kubernetes, it becomes really hard to get such a huge number of certificates, monitor them, and rotate them every time. This would be a nightmare for the DevOps Engineers. What if I tell you that there a tool that could get you free SSL certificates and rotate them automatically when they expire? Here comes Cert-Manager. Cert-manager was created by Jetstack, and a lot of the development is still sponsored by them. As per the cert-manager’s official guide, the cert-manager is a native Kubernetes certificate management controller. It can help with issuing certificates from a variety of sources, such as Let’s Encrypt, HashiCorp Vault, Venafi, a simple signing key pair, or self-signed. It will ensure certificates are valid and up to date, and attempt to renew certificates at a configured time before expiry.
The HTTP protocol developed in the early 1990s has become an integral part of our daily life. Today, we cannot live a single day without it. However, it does not even provide a basic level of security when exchanging information between the user and the web server. That is when HTTPS (“S” means “secure” here) comes to the rescue. In HTTPS, the exchanged data is encrypted using SSL/TLS — that family of protocols has proven itself well in protecting privacy and data integrity and is being actively promoted by the industry.
For example, Google has been calling for “HTTPS everywhere” since 2014. When prioritizing search results, it takes into account whether sites use secure, encrypted connections. All this propaganda affects ordinary users as well: modern browsers warn their users about insecure connections and invalid SSL certificates.
A certificate for a personal website might cost tens of dollars. However, buying it is not always justified. Fortunately, since late 2015, there is a free alternative in the form of Let’s Encrypt (LE) certificates. This nonprofit authority was created by Mozilla enthusiasts to make internet-wide hassle-free encryption a reality.
The certificate authority issues domain-validated certificates (the most basic ones available on the market) valid for 90 days, and it is also possible to obtain a so-called wildcard certificate for several subdomains.
The algorithms described in the Automated Certification Management Environment (ACME) protocol (designed specifically for Let’s Encrypt) are used to obtain a certificate. With it, the agent can prove control of the domain either by provisioning an HTTP resource (the so-called “HTTP-01 challenge”) or DNS records (“DNS-01 challenge”) — more information about them you may find below.
Cert-manager is a Kubernetes-native certificate management controller consisting of a set of CustomResourceDefinitions (hence the restriction on the minimum supported version of K8s, v1.12) for configuring CA (certificate authorities) and obtaining certificates. The installation of CRDs in a cluster is straightforward and boils down to applying a single YAML file:
(You can also install it with Helm).
Before being able to request certificates, you must create CA resources: Issuer or ClusterIssuer. They are used for signing CSRs (certificate requests). The difference between these resources is that ClusterIssuer is non-namespaced and can be used in multiple namespaces:
Issuer is used within a single namespace only;
ClusterIssuer is a global cluster object.
Let’s start with the simplest case — requesting a self-signed certificate. It is quite common. For example, you can use it in your K8s cluster for testing environments dynamically created for developers needs. It can be also useful in the case of the external load balancer that terminates SSL traffic.
The Issuer resource would look like this:
To issue a certificate, you have to define the Certificate resource that determines the issuance (see the issuerRef section below) and the location of the private key (the secretName field). Then you need to invoke that key in the Ingress (note the tls section in the spec field):
The certificate will be issued in a few seconds after these resources are added to the cluster. You can see the confirmation in the output of the command:
Looking at the secret resource itself, you will see:
the tls.key private key,
the ca.crt root certificate,
and the self-signed tls.crt certificate.
You can browse the contents of these files using the openssl utility:
It is worth noting that clients consuming certificates issued in such a way (i.e., using the self-signed issuer) will not trust them. The reason is simple: this issuer type does not have a CA (see the note on the project website).
To avoid this, specify the path to the secret file containing ca.crt in the Certificate resource. For example, you can use the corporate CA. This way, you will be able to sign certificates issued for your Ingress with a key that is already in use by other server services/information systems.
As I mentioned before, there are two types of “challenges” available to prove the control of the domain, HTTP-01 and DNS-01.
The first approach (HTTP-01) involves deploying a tiny web server as a separate deployment. It will be serving some information at the http://<YOUR_DOMAIN>/.well-known/acme-challenge/<TOKEN> URL per request of the certification server. Therefore, this method implies the accessibility of Ingress from the outer world via port 80 and the publicity of the domain’s DNS record.
The second challenge (DNS-01) makes sense if there is an API you can use to change the DNS records of your domain. The Issuer uses these tokens to create TXT records for your domain. Then, ACME server gets these records during confirmation. Let’s Encrypt easily integrates with various DNS providers, including CloudFlare, AWS Route53, Google CloudDNS, and others (as well as with the LE’s own DNS implementation, acme-dns).
Note: Let’s Encrypt imposes fairly strict limits on requests to ACME servers. To avoid unnecessary load on LE’s production environment, we recommend using the letsencrypt-staging certificate for testing (the difference is in the ACME server only).
So, let’s describe the resources:
Note that we use the staging server in acme’s server field for our Issuer. You can replace it with the production one later.
Let’s apply this configuration and trace the entire process of obtaining the certificate:
1. The creation of a Certificate leads to the emergence of a new CertificateRequest resource:
2. In its description there is a notification about an Order creation:
3. The Order contains the description of parameters of the validation and its current status. The validation is performed by the Challenge resource:
4. And finally, there is information about the status of the validation itself in resource details:
The certificate will be issued in less than a minute if all the prerequisites are met: the domain is accessible from the outer world, the rate limits on the LE side are respected, and so on. If the issuance was successful, you should see the message “Certificate issued successfully” in the output of the describe certificate le-tls command.
Now you can safely change the ACME server address to the production one (https://acme-v02.api.letsencrypt.org/directory) and re-issue valid certificates signed by Let's Encrypt Authority X3 instead of Fake LE Intermediate X1.
But first, you have to delete the Certificate resource. Otherwise, the issuance procedure will not start because the certificate already exists, and it is valid. Deleting a secret would immediately result in invalidation of the certificate with the following message in the output of the describe certificate command:
Now it is time to apply the production manifest for the Issuer with the Certificate described above (it has not changed):
After receiving the Certificate issued successfully confirmation, let us check it out:
To make one step further we will issue a certificate for all subdomains of the site using another method of validation — the DNS one. We will use CloudFlare as our DNS provider to change the domain records we need.
First, let’s create a token to use the CloudFlare API:
1. Profile → API Tokens → Create Token.
2. Set access rights as follows:
Permissions:
→ Zone — DNS — Edit
→ Zone — Zone — Read
Zone Resources:
→ Include — All Zones
3. Copy the token generated (for example, y_JNkgQwkroIsflbbYqYmBooyspN6BskXZpsiH4M).
Create a Secret resource containing the token and describe it in your Issuer:
(Do not forget to use a staging environment for testing!)
It is time to go through the domain ownership confirmation procedure:
The TXT record will appear at your DNS dashboard:
… and after a while, the status will change to:
Let’s make sure that the certificate is valid for all subdomains:
The validation over DNS is usually slow since most DNS providers have a so-called “propagation time” — a period showing how long it takes for an updated DNS record to become available on all DNS servers of the provider.
The ACME standard also supports a combination of both types of validation. You can use it to speed up obtaining a certificate for the main domain. In this case, the description of the Issuer will have the following form:
If you apply this configuration, two Challenge resources will be created:
Besides creating certificates directly, you can use cert-manager’s ingress-shim component. It relieves you of the need to create Certificate resources explicitly. The idea is to obtain a certificate automatically using the Issuer specified in the special annotations of Ingress. Here is an example of the respective Ingress resource:
In this case, the availability of Issuer is enough to get the work done, meaning that we have to create a lesser number of entities.
Also, there is an obsolete kube-lego annotation — kubernetes.io/tls-acme: "true". The peculiarity about it is that when deploying cert-manager, the default Issuer must be specified via Helm arguments (or by appending arguments of cert-manager’s deployment container).
We at Flant do not use these approaches (mentioned in Method #4) and cannot recommend them due to their opacity (and various associated problems). However, it’s good to mention them in the article to provide a fuller picture.
We have learned how to obtain auto-renewable, self-signed, and free SSL certificates from Let’s Encrypt for website domains that are managed by Ingresses in the Kubernetes clusters.
The article provides example solutions for the most common problems we face. However, cert-manager features are not limited to those described above. On the project’s website, you can find examples of using it with other services. For example, you can use Vault as a certificate authority or set up external Issuers.
PLEASE NOTE: our blog has MOVED to https://blog.flant.com/! New articles from Flant’s engineers will be posted there only. Check it out and subscribe to keep updated!
This article has been written by our engineer Oleg Saprykin.
