Certmanager tries to use HTTPS inside the cluster and fails when it gets self signed certificates

Here is minimum code to reproduce this. I am using GKE.

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash 

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx  --force-update
helm upgrade --install nginx-ingress ingress-nginx/ingress-nginx \
    --namespace nginx-ingress --create-namespace \
    --set controller.service.loadBalancerIP="35.238.217.163" \
    --set controller.service.type=LoadBalancer \
    --set controller.admissionWebhooks.patch.nodeSelector."kubernetes\.io/os"=linux


helm repo add jetstack https://charts.jetstack.io --force-update
helm upgrade --install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.16.2 \
  --set crds.enabled=true

Getting the logs of cert manager kubectl logs cert-manager-startupapicheck-qzrqw -n cert-manager it seems to try to do a HTTPS call to https://cert-manager-webhook.cert-manager.svc:443, which would fail because it doesn't have a signed certificate yet from anywhere. Is there a way to switch this to an HTTP call? It makes no sense to me.

I0108 00:52:35.104988       1 api.go:106] "Not ready" logger="cert-manager.startupapicheck.checkAPI" err="Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=30s\": tls: failed to verify certificate: x509: certificate signed by unknown authority"
I0108 00:52:40.327920       1 api.go:106] "Not ready" logger="cert-manager.startupapicheck.checkAPI" err="Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=30s\": tls: failed to verify certificate: x509: certificate signed by unknown authority"
I0108 00:52:45.066769       1 api.go:106] "Not ready" logger="cert-manager.startupapicheck.checkAPI" err="Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=30s\": tls: failed to verify certificate: x509: certificate signed by unknown authority"

If you want to switch to HTTP call edit the deployment of cert-manager and cert-manager-webhook and configure the webhook settings as mentioned below:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cert-manager
  namespace: cert-manager
spec:
  template:
       spec: 
         containers:
name: cert - manager
env:
- name:  WEBHOOK_HTTP
  value:  “true”
- name:   WEBHOOK_TLS_DISABLE
  value:  “true” 

Here, setting WEBHOOK_TLS_DISABLE to true will allow the webhook to communicate over HTTP instead of HTTPS.Once this is done apply the changes and restart the cert-manager pod so the changes can take effect.

$Kubectl apply -f <your-modified-cert-manager-deployment>.yaml
$Kubectl rollout restart deployment
$Cert-manager -n cert-manager

This will allow cert-Manager to start and initialize the webhook without requiring valid HTTPS certificates immediately.

Note: If you want to stick to HTTPS and avoid switching to HTTP, you could configure Cert-manager to use a self-signed certificate for the webhook by generating a certificate for the webhook , by this way cert-manager would trust its own self signed cert and avoid errors during initial startup.

Self-signed certificates fall short of offering every security feature that a CA-signed certificate is supposed to offer.