I recently upgraded ingress-nginx to version 1.0.3.
As a result, I removed the kubernetes.io/ingress.class annotation from my ingress, and put .spec.ingressClassName instead.
I am running cert-manager-v1.4.0.
This morning I had an email saying that my Let's Encrypt certificate will expire in 10 days. I tried to figure out what was wrong with it - not positive that it was entirely due to the ingress-nginx upgrade.
I deleted the CertificateRequest to see if it would fix itself. I got a new Ingress with the challenge, but:
The challenge ingress had the kubernetes.io/ingress.class annotation set correctly, even though my ingress has .spec.ingressClassName instead - don't know how or why, but it seems like it should be OK.
However, the challenge ingress wasn't picked up by the ingress controller, it said:
ingress class annotation is not equal to the expected by Ingress Controller
I guess it wants only the .spec.ingressClassName even though I thought the annotation was supposed to work as well.
So I manually set .spec.ingressClassName on the challenge ingress. It was immediately seen by the ingress controller, and the rest of the process ran smoothly, and I got a new cert - yay.
It seems to me like this will happen again, so I need to know how to either:
Convince cert-manager to create the challenge ingress with .spec.ingressClassName instead of kubernetes.io/ingress.class. Maybe this is fixed in 1.5 or 1.6?
Convince ingress-nginx to respect the kubernetes.io/ingress.class annotation for the challenge ingress. I don't know why this doesn't work.
The issue was fixed by certificate renewal, it works fine without manually set spec.ingressClassName in challenge ingress (I saw it with older version), issue was somewhere else.
Also with last available (at the writing moment) cert-manager v1.5.4 challenge ingress has the right setup "out of the box":
spec:
ingressClassName: nginx
---
$ kubectl get ing
NAME CLASS HOSTS ADDRESS PORTS AGE
cm-acme-http-solver-szxfg nginx dummy-host ip_address 80 11s
I'll describe main steps how this process works so troubleshooting will be straight-forward in almost all cases. I'll take a letsencypt staging as an issuer.
There's a chain when certificate is requested to be created which issuer follows to complete (all resources have owners - previous resource in chain):
main ingress resource -> certificate -> certificaterequest -> order -> challenge -> challenge ingress.
Knowing this, if something failed, you can go down by the chain and using kubectl describe command find where the issue appeared.
I intentionally added a wrong domain in ingress to .spec.tls.hosts and applied it. Below how the chain will look like (all names will be unique!):
See certificates:
$ kubectl get cert
NAME READY SECRET AGE
lets-secret-test-2 False lets-secret-test-2 15m
Describe certificate we are interested in (you can notice I changed domain, there was already secret):
$ kubectl describe cert lets-secret-test-2
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Issuing 16m cert-manager Existing issued Secret is not up to date for spec: [spec.commonName spec.dnsNames]
Normal Reused 16m cert-manager Reusing private key stored in existing Secret resource "lets-secret-test-2"
Normal Requested 16m cert-manager Created new CertificateRequest resource "lets-secret-test-2-pvb25"
Nothing suspicious here, moving forward.
$ kubectl get certificaterequest
NAME APPROVED DENIED READY ISSUER REQUESTOR AGE
lets-secret-test-2-pvb25 True False letsencrypt-staging system:serviceaccount:cert-manager:cert-manager 19m
Describing certificaterequest:
$ kubectl describe certificaterequest lets-secret-test-2-pvb25
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal cert-manager.io 19m cert-manager Certificate request has been approved by cert-manager.io
Normal OrderCreated 19m cert-manager Created Order resource default/lets-secret-test-2-pvb25-2336849393
Again, everything looks fine, no errors, moving forward to order:
$ kubectl get order
NAME STATE AGE
lets-secret-test-2-pvb25-2336849393 pending 21m
It says pending, that's closer:
$ kubectl describe order lets-secret-test-2-pvb25-2336849393
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Created 21m cert-manager Created Challenge resource "lets-secret-test-2-pvb25-2336849393-3788447910" for domain "dummy-domain"
Challenge may shed some light, moving forward:
$ kubectl get challenge
NAME STATE DOMAIN AGE
lets-secret-test-2-pvb25-2336849393-3788447910 pending dummy-domain 23m
Describing it:
$ kubectl describe challenge lets-secret-test-2-pvb25-2336849393-3788447910
Checking status:
Status:
Presented: true
Processing: true
Reason: Waiting for HTTP-01 challenge propagation: failed to perform self check GET request 'http://dummy-domain/.well-known/acme-challenge/xxxxyyyyzzzzz': Get "http://dummy-domain/.well-known/acme-challenge/xxxxyyyyzzzzz": dial tcp: lookup dummy-domain on xx.yy.zz.ww:53: no such host
State: pending
Now it's clear that something is wrong with domain, worth checking it:
Found and fixed the "mistake":
$ kubectl apply -f ingress.yaml
ingress.networking.k8s.io/ingress configured
Certificate is ready!
$ kubectl get cert
NAME READY SECRET AGE
lets-secret-test-2 True lets-secret-test-2 26m
It's possible to renew a certificate by deleting corresponding secret, however documentation says it's not recommended:
Deleting the Secret resource associated with a Certificate resource is not a recommended solution for manually rotating the private key. The recommended way to manually rotate the private key is to trigger the reissuance of the Certificate resource with the following command (requires the kubectl cert-manager plugin):
kubectl cert-manager renew cert-1
Kubectl cert-manager command installation process is described here as well as other commands and examples.