HTTP-01 Challenge Fails on Bare-Metal Kubernetes with cert-manager and Let’s Encrypt

I am running a bare-metal Kubernetes cluster (no cloud provider) with cert-manager and Let’s Encrypt to issue TLS certificates. I bought a domain from Namecheap and configured a basic Nginx ingress controller. However, my certificates are stuck in a pending/errored state.

Setup Kubernetes cluster: 1 control plane + 3 workers CI/CD: GitHub Runner with ArgoCD Ingress: Nginx LoadBalancer: MetalLB with a local IP assigned to the Nginx controller Domain: A record points to my public IP

Here is what my conf is (I have remove my domain and email info.):

K8s

• Ingress YAML

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: private-test-deployment-api
  namespace: default
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - 'my-domain'
      secretName: my-domain-tls
  rules:
    - host: 'my-domain'
      http:
        paths:
          - path: /grades
            pathType: Prefix
            backend:
              service:
                name: private-test-deployment-api
                port:
                  number: 80

• ClusterIssuer YAML

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: 'my-email'
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
      - http01:
          ingress:
            class: nginx
            ingressTemplate:
              metadata:
                annotations:
                  nginx.ingress.kubernetes.io/ssl-redirect: "false"
                  nginx.ingress.kubernetes.io/force-ssl-redirect: "false"

• MikroTik NAT Rules

 0    ;;; PPPoer Connection !
      chain=srcnat action=masquerade out-interface=pppoe-out1 log=no log-prefix="" 

 2    ;;; Forward HTTP to worker0 ingress
      chain=dstnat action=dst-nat to-addresses=192.168.88.39 to-ports=80 protocol=tcp in-interface=pppoe-out1 dst-port=80 
      log=no log-prefix="" 

 3    ;;; Forward HTTPS to worker0 ingress
      chain=dstnat action=dst-nat to-addresses=192.168.88.39 to-ports=443 protocol=tcp in-interface=pppoe-out1 dst-port=443 
      log=no log-prefix=""

• Mikrotik Firewall Rules

0    ;;; Allow HTTP to ingress
      chain=forward action=accept protocol=tcp dst-address=192.168.88.39 in-interface=pppoe-out1 dst-port=80 

 1    ;;; Allow HTTPS to ingress
      chain=forward action=accept protocol=tcp dst-address=192.168.88.39 in-interface=pppoe-out1 dst-port=443 

 2    ;;; Allow Access From Trusted IPs
      chain=input action=accept src-address-list=Trusted-IPs log=no log-prefix="" 

 3    ;;; Accept Established & Related Connections
      chain=input action=accept connection-state=established,related log=no log-prefix="" 

 4    ;;; Accept ICMP Request From LAN
      chain=input action=accept tcp-flags="" protocol=icmp in-interface=ether2 icmp-options=8:0-255 log=no log-prefix="" 

 5    ;;; Accept DNS-UDP
      chain=input action=accept protocol=udp in-interface=ether2 dst-port=53 log=no log-prefix="" 

 6    ;;; Accept DNS-TCP
      chain=input action=accept protocol=tcp in-interface=ether2 dst-port=53 log=no log-prefix="" 

 7    ;;; Drop Everything Else
      chain=input action=drop log=no log-prefix=""

Tests

• Inside Cluster

kubectl -n default run -it --rm --image=curlimages/curl curltest -- curl -sv http://private-test-deployment-api.default/.well-known/acme-challenge/test

Returns: error: timed out waiting for the condition

• Outside Cluster

curl -v http://test.icptokens.net/.well-known/acme-challenge/test

Returns: 404 (Not Found)

Observation and Question I suspect the issue is external HTTP traffic not reaching the solver pod, but I’m not sure if it’s my MikroTik NAT rules, firewall, or ingress configuration.

Has anyone successfully run HTTP-01 on a bare-metal cluster behind a MikroTik router? I would appreciate guidance on:

• Correct NAT and firewall setup for HTTP-01
• Any additional ingress annotations or tweaks needed
• Ways to debug why Let’s Encrypt cannot reach the challenge path

Thanks in advance for any advice!

I know that Namecheap is not a natively supported issuer for cert-manager but in your situation I would drop firewall/NAT rules in favor of webhook issuer.

This method will allow you to generate certificate via placing txt record in your DNS provider and generate any certificate without need to expose your internal cluster.

Example webhook for your DNS.