digital-oceankubernetes-ingress

error broken header: "GET /.well-known/acme-challeng with LetsEncrypt on Kubernates


I've created the LetsEncrypt production ClusterIssuers in Digital Ocean Kubernaties DO kubernaties ver - 1.17.5 My cert-manager version is v0.15.0

I used this howto

kubectl describe clusterissuer letsencrypt-prod

Name:         letsencrypt-prod
Namespace:    
Labels:       <none>
Annotations:  API Version:  cert-manager.io/v1alpha3
Kind:         ClusterIssuer
Metadata:
  Creation Timestamp:  2020-05-13T12:08:52Z
  Generation:          1
  Resource Version:    16757
  Self Link:           /apis/cert-manager.io/v1alpha3/clusterissuers/letsencrypt-prod
  UID:                 2bbd1ca6-9c85-45e3-ad6e-7b85d9e93657
Spec:
  Acme:
    Email:  cert@example.com
    Private Key Secret Ref:
      Name:  letsencrypt-prod
    Server:  https://acme-v02.api.letsencrypt.org/directory
    Solvers:
      http01:
        Ingress:
          Class:  nginx
Status:
  Acme:
    Last Registered Email:  cert@example.com
    Uri:                    https://acme-v02.api.letsencrypt.org/acme/acct/86033097
  Conditions:
    Last Transition Time:  2020-05-13T12:08:53Z
    Message:               The ACME account was registered with the ACME server
    Reason:                ACMEAccountRegistered
    Status:                True
    Type:                  Ready
Events:                    <none>

kubectl describe ingress

Name:             bb-ingress
Namespace:        default
Address:          167.99.17.96
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
  bb-cloud-tls terminates example.com
Rules:
  Host                 Path  Backends
  ----                 ----  --------
  example.com  
                       /   bb-web-service:80 (10.244.0.166:3000,10.244.0.31:3000)
Annotations:           cert-manager.io/cluster-issuer: letsencrypt-prod
                       kubernetes.io/ingress.class: nginx
Events:
  Type     Reason     Age                   From                      Message
  ----     ------     ----                  ----                      -------
  Warning  BadConfig  8m17s                 cert-manager              TLS entry 0 for hosts [example.com] must specify a secretName
  Normal   UPDATE     7m24s (x11 over 24h)  nginx-ingress-controller  Ingress default/bb-ingress


Name:             cm-acme-http-solver-kbnn6
Namespace:        default
Address:          167.99.17.96
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host                 Path  Backends
  ----                 ----  --------
  example.com  
                       /.well-known/acme-challenge/i5J8QI4XwJZVnS4xC_nSbK-8QFYlUJkmmOnETFXltdE   cm-acme-http-solver-kgbd8:8089 (10.244.0.188:8089)
Annotations:           kubernetes.io/ingress.class: nginx
                       nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0,::/0
Events:                <none>

kubectl describe certificate

Name:         bb-cloud-tls
Namespace:    default
Labels:       <none>
Annotations:  API Version:  cert-manager.io/v1alpha3
Kind:         Certificate
Metadata:
  Creation Timestamp:  2020-05-13T11:06:34Z
  Generation:          1
  Resource Version:    13723
  Self Link:           /apis/cert-manager.io/v1alpha3/namespaces/default/certificates/bb-cloud-tls
  UID:                 11e6d711-56a9-4711-a6c4-cca516b96c41
Spec:
  Common Name:  example.com
  Dns Names:
    example.com
  Duration:  24h0m0s
  Issuer Ref:
    Kind:        ClusterIssuer
    Name:        letsencrypt-prod
  Renew Before:  12h0m0s
  Secret Name:   bb-cloud-tls
Status:
  Conditions:
    Last Transition Time:  2020-05-13T11:46:24Z
    Message:               Waiting for CertificateRequest "bb-cloud-tls-1534494017" to complete
    Reason:                InProgress
    Status:                False
    Type:                  Ready
Events:                    <none>

kubectl describe order

Name:         bb-cloud-tls-1534494017-2165728012
Namespace:    default
Labels:       <none>
Annotations:  cert-manager.io/certificate-name: bb-cloud-tls
              cert-manager.io/private-key-secret-name: bb-cloud-tls
API Version:  acme.cert-manager.io/v1alpha3
Kind:         Order
Metadata:
  Creation Timestamp:  2020-05-13T11:46:24Z
  Generation:          1
  Owner References:
    API Version:           cert-manager.io/v1alpha2
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  CertificateRequest
    Name:                  bb-cloud-tls-1534494017
    UID:                   5b2972ba-bfe5-4149-a53b-13764a1a8269
  Resource Version:        13730
  Self Link:               /apis/acme.cert-manager.io/v1alpha3/namespaces/default/orders/bb-cloud-tls-1534494017-2165728012
  UID:                     1dd81160-c700-4d29-88c1-0c5a5dee5774
Spec:
  Common Name:  example.com
  Csr:          LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNU**************************
  Dns Names:
    example.com
  Issuer Ref:
    Kind:  ClusterIssuer
    Name:  letsencrypt-prod
Status:
  Authorizations:
    Challenges:
      Token:        i5J8QI4XwJZVnS4*********
      Type:         http-01
      URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/4557349440/4vbwhw
      Token:        i5J8QI4XwJZVnS******
      Type:         dns-01
      URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/4557349440/yILvmw
      Token:        i5J8QI4Xw*****
      Type:         tls-alpn-01
      URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/4557349440/iPGc-Q
    Identifier:     example.com
    Initial State:  pending
    URL:            https://acme-v02.api.letsencrypt.org/acme/authz-v3/4557349440
    Wildcard:       false
  Finalize URL:     https://acme-v02.api.letsencrypt.org/acme/finalize/86033097/3348998322
  State:            pending
  URL:              https://acme-v02.api.letsencrypt.org/acme/order/86033097/3348998322
Events:             <none>

Also I have such logs for ingress pod devspace logs -n ingress-nginx --pod ingress-nginx-controller-5cc4589cc8-z5hb4 -c controller

" while reading PROXY protocol, client: 10.244.0.178, server: 0.0.0.0:80
2020/05/14 11:59:02 [error] 163#163: *388536 broken header: "GET /.well-known/acme-challenge/i5J8QI4XwJZVnS4xC_nSbK-8QFYlUJkmmOnETFXltdE HTTP/1.1
Host: example.com
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip
Connection: close

I have incorrect certificate: "Kubernetes Ingress Controller Fake Certificate"

How I can fix this issue?

PS. also I found simular issue on githib but it is closed and I have new version of cert-manager


Solution

  • I change ACME from http01 to dns01

    before:

    apiVersion: cert-manager.io/v1alpha2
    kind: ClusterIssuer
    metadata:
      name: letsencrypt-prod
      namespace: cert-manager
    spec:
      acme:
        # The ACME server URL
        server: https://acme-v02.api.letsencrypt.org/directory
        # Email address used for ACME registration
        email: my@example.com
        # Name of a secret used to store the ACME account private key
        privateKeySecretRef:
          name: letsencrypt-prod
        # Enable the HTTP-01 challenge provider
        solvers:
        - http01:
            ingress:
              class: nginx
    

    after:

    apiVersion: cert-manager.io/v1alpha2
    kind: ClusterIssuer
    metadata:
      name: letsencrypt-prod
      namespace: cert-manager
    spec:
      acme:
        # The ACME server URL
        server: https://acme-v02.api.letsencrypt.org/directory
        # Email address used for ACME registration
        email: my@example.com
        # Name of a secret used to store the ACME account private key
        privateKeySecretRef:
          name: letsencrypt-prod
        # Enable the DNS-01 challenge provider
        solvers:
        - dns01:
            digitalocean:
              tokenSecretRef:
                name: digitalocean-dns
                key: access-token
    

    Also I add Secret - see https://cert-manager.io/docs/configuration/acme/dns01/digitalocean/ for details

    Now it is works