kubernetes azure-aks cloudflare cert-manager

cert-manager does not seem to work with AKS and Cloudflare

I've been trying for a couple of days now to get my AKS to issue a certificate request to Cloudflare via its API key. From what I can see, the API key has all the right permissions, but the certificate can't seem to finish its round-robin and complete.

I've tried various things and different versions of cert-manager, but nothing seems to work.

It just comes back with the error: Issuing certificate as Secret does not exist

I've also followed these links to try and resolve this issue:

https://cert-manager.io/docs/troubleshooting/ Issuing certificate as Secret does not exist

Basically no matter what I do I am left with this pending state:

  Normal  OrderCreated     <invalid>  cert-manager  Created Order resource ingress-nginx/jc-aks-testing-cert-5z48g-3941378753
  Normal  cert-manager.io  <invalid>  cert-manager  Certificate request has been approved by cert-manager.io
  Normal  OrderPending     <invalid>  cert-manager  Waiting on certificate issuance from order ingress-nginx/jc-aks-testing-cert-5z48g-3941378753: ""

Here is my long script to make all this:

#!/bin/bash
rg="jc-testing5-aks-rg"
location="francecentral"
cluster="jc-aks-testing5-cluster"
keyvaultname="jc-aks-testing5-kv"

## Create RG
echo "Creating Resource Group $rg"
az group create --name $rg --location $location

## Create AKS Cluster
echo "Creating AKS Cluster $cluster"

az aks create -g $rg -n $cluster --load-balancer-managed-outbound-ip-count 1 --enable-managed-identity --node-vm-size Standard_B2s --node-count 1 --generate-ssh-keys

## Create KeyVault
echo "Creating KeyVault $keyvaultname"
az key vault create --resource-group $rg --name $keyvaultname

## Connect to Cluster
echo "Connecting to AKS Cluster."
az aks get-credentials --resource-group $rg --name $cluster --overwrite-existing

## Install Nginx
echo "Installing Nginx into the cluster"
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

helm install nginx-ingress ingress-nginx/ingress-nginx \
    --namespace ingress-nginx --create-namespace\
    --set controller.replicaCount=2 \
    --set controller.nodeSelector."kubernetes\.io/os"=linux \
    --set controller.admissionWebhooks.patch.nodeSelector."kubernetes\.io/os"=linux \
    --set defaultBackend.nodeSelector."kubernetes\.io/os"=linux

#CERT_MANAGER_TAG=v1.3.1
CERT_MANAGER_TAG=v1.13.6

# Label the ingress-basic namespace to disable resource validation
kubectl label namespace ingress-nginx cert-manager.io/disable-validation=true

# Add the Jetstack Helm repository in preparation to install Cert-Manager
echo "Installing Cert-Manager"
helm repo add jetstack https://charts.jetstack.io --force-update

# Update your local Helm chart repository cache
helm repo update

# Install the cert-manager Helm chart
helm install cert-manager jetstack/cert-manager \
  --namespace ingress-nginx \
  --version $CERT_MANAGER_TAG \
  --set installCRDs=true \
  --set nodeSelector."kubernetes\.io/os"=linux

## Create a Cert-Cluster Issuer.
echo "Creating Certmanger Cluster Issuer for ArgoCD"

cat << EOF | kubectl apply -f - 
apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-key-secret
  namespace: ingress-nginx
type: Opaque
Data:
  api-key: MYVALUE
EOF

cat << EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt
  namespace: ingress-nginx
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: jason@mydomain.com
    privateKeySecretRef:
      name: letsencrypt
    solvers:
    solvers:
    - dns01:
        cloudflare:
          apiKeySecretRef:
            key: api-key
            name: cloudflare-api-key-secret
          email: jason@mydomain.com
EOF


cat << EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: jc-aks-testing-cert
  namespace: ingress-nginx
spec:
  secretName: mydomain.com-tls
  issuerRef:
    name: letsencrypt

  duration: 2160h # 90d
  renewBefore: 720h # 30d before SSL will expire, renew it
  dnsNames:
    - "mydomain.com"
    - "mydomain.com"
EOF

## Install Argo CD
echo "Installing Argo CD"
kubectl create namespace argocd

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

## Configure Argo CD to Look at Custom Domain
echo "Configuring Argo CD to Look at Custom Domain"
cat << EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-server-ingress
  namespace: argocd
  annotations:
    cert-manager.io/issuer: letsencrypt
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    # If you encounter a redirect loop or are getting a 307 response code
    # then you need to force the nginx ingress to connect to the backend using HTTPS.
    #
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  rules:
  - host: mydomain.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: argocd-server
            port:
              name: https
  tls:
  - hosts:
    - mydomain.com
    secretName: argocd-secret # do not change, this is provided by Argo CD
EOF


## Get the Password for Argo CD Login
echo "Getting the Password to login into Argo-CD"
argo_cd_pass=$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d)
echo "$argo_cd_pass"

Solution

The way I got this to work in the end was from these three sources:

The first item led me down a rabbit hole of figuring out if something was wrong with the API key. I believe Cloudflare doesn't want you to use the Global API key and the General API key.

I then found out on another blog post, but I don't have this to hand that rights for the API key need to be Zone > Zone > Read, Zone > DNS > Edit and Zone > DNS > Read for some reason and I can't for the life of me figure out why, if you have just Zone > DNS > Edit the API Key can not see the DNS records. That was a weird issue.

Once the API key was set correctly, I followed the Cloudflare blog post to put the origin ca issuer repo tech into the cluster. I found I had to do this to get an issuer communicating correctly with the API key into Cloudflare. Without the Cloudflare Origin Issuer tech, you get some weird errors, and the communication does not work correctly. Also, note that when you install this onto your cluster, your cluster needs to have three nodes minimum.

Once that was done, I followed the third resource on how to make an Issuer, Certificate, and Nginx load balancer. This works, but there are a couple of things to note.

First, ensure your issuer is a ClusterIssuer and not just an Issuer. A cluster issuer works for all namespaces, but an issuer only works for the namespace you put it in. In my case, it was cert-manager, which is not great if your application is in its own namespace with an Nginx service. The whole thing will not work.

Also, the creation of the secret is like this:

kubectl create secret generic jasons-api-key \
    -n cert-manager\
    --from-literal api-token='api-key-value'

If you notice, before I specify the API-key-value, there is an = and then a name associated with that called API-token. This is important because when you come to your issuer code under apiTokenSecretRef:, you have to make sure that the key: value is the same as that association.

So your cluster issuer code will look as follows:

cat  << EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: lets-encrypt-jasons-cert
  namespace: cert-manager
spec:
  acme:
    email: <email address>
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource that will be used to store the account's private key.
      name: lets-encrypt-jasons-cert
    solvers:
    - dns01:
        cloudflare:
          email: <email address>
          apiTokenSecretRef:
            name: jasons-api-key
            key: api-token
      selector:
        dnsZones:
        - <central domain name of the account, so the top-level domain>
EOF

Also note under solvers -dns01: the DNS zones: needs to be the domain name, for example , example.com it's NOT the DNS record that you are pointing Cloudflare to. This caught me out for a while

I hope this helps someone that might come across this issue.