I am trying to use cert-manager with Let's Encrypt to issue certificates using HTTP-01 challenge.
It looks like routing problem - certificate is stuck on domain control validation with error: Waiting for HTTP-01 challenge propagation: wrong status code '304', expected '200'. Route to application (oauth2-proxy) is working so it responds with 403 - standard for oauth2-proxy.
Software stack:
helm version 1.19.3 (base + istiod in namespace istio-system, gateway in separate one istio-ingress-public - just like in Istio docs)helm version 1.13.2 in namespace cert-manageroauth2-proxyhello-kubernetes for testing purposesFor both oauth2-proxy and hello-kubernetes I have native Istio Gateway and VirtualService for each one of them in their namespaces and it works.
For cert-manager I have created Issuer.
I have created Certificate in namespace oauth2-proxy and cert-manager created CertificateRequest, Order, Challenge and Ingress, Service and Pod to handle verification.
I have created IngressClass for Istio:
kubectl get IngressClass -o yaml
apiVersion: v1
items:
- apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
creationTimestamp: "2023-11-03T12:12:09Z"
generation: 1
name: istio
resourceVersion: "4756892"
uid: cd6205b9-e818-48dd-8882-59c6d038e28e
spec:
controller: istio.io/ingress-controller
kind: List
metadata:
resourceVersion: ""
Ingress has .spec.ingressClassName set to istio:
kubectl get Ingress -o yaml -n oauth2-proxy
apiVersion: v1
items:
- apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0,::/0
creationTimestamp: "2023-11-03T14:03:42Z"
generateName: cm-acme-http-solver-
generation: 1
labels:
acme.cert-manager.io/http-domain: "1390741491"
acme.cert-manager.io/http-token: "1315376821"
acme.cert-manager.io/http01-solver: "true"
name: cm-acme-http-solver-4z6kk
namespace: oauth2-proxy
ownerReferences:
- apiVersion: acme.cert-manager.io/v1
blockOwnerDeletion: true
controller: true
kind: Challenge
name: oauth2-proxy-cert-staging-1-2829224423-1571648249
uid: 95daf075-ee7e-465e-b129-9d99218addf1
resourceVersion: "4774848"
uid: e6a15d62-e010-4b16-8ee0-86bf809e5365
spec:
ingressClassName: istio
rules:
- host: apps.mydomain.com
http:
paths:
- backend:
service:
name: cm-acme-http-solver-spkrv
port:
number: 8089
path: /.well-known/acme-challenge/************
pathType: ImplementationSpecific
status:
loadBalancer: {}
kind: List
metadata:
resourceVersion: ""
I followed docs for integrating Istio with cert-manager: Istio / cert-manager and how to deal with k8s Ingress: Istio / Kubernetes Ingress.
I've also tried to use kubernetes.io/ingress.class annotation with same result - no new routes.
What am I doing wrong? From docs it seems that Ingress with .spec.ingressClassName set to istio should be handled without any other configuration. Is there any flag to be set like in previous Istio versions? I didn't find it in helm values...
What should happen after creating Ingress? I assume that new route should be created and visible using istioctl. Should Istio create new VirtualService?
I figured that out.
This issue helped me: Certificate Challenges failing with Istio + Istio Gateway when installed via Helm · Issue #37329 · istio/istio (github.com)
Istio configuration depends on how you installed it. And if installed using helm you need to configure few values to make it work. Of course values may change depending on your environment - but without setting them Istio does not recognize Ingress resources.
set {
name = "meshConfig.ingressService"
value = "istio-ingress"
}
set {
name = "meshConfig.ingressSelector"
value = "ingress"
}
When properly configured, Istio will create VirtualService to handle challenge and you should be able to debug Istio config using istioctl:
// list ingress gateways
$ istioctl proxy-status
// show routes
$ istioctl proxy-config routes -n <ingress_gateway_namespace> <ingress_gateway_name_from_previous_command>