gitopsfluxcd

flux deployment error X509 certificate signed by unknown authority

My aim is to deploy a container-labelling-webhook solution onto my AKS cluster using flux CD v2. Once I have it operational, I want to rollout to multiple clusters.

Command used to bootstrap AKS Cluster(Flux Installation I mean)

flux bootstrap git --url=https://github.xxxxxx.com/user1/test-repo.git --username=$GITHUB_USER --password=$GITHUB_TOKEN --token-auth=true --path=clusters/my-cluster

✔ Kustomization reconciled successfully
► confirming components are healthy
✔ helm-controller: deployment ready
✔ kustomize-controller: deployment ready
✔ notification-controller: deployment ready
✔ source-controller: deployment ready
✔ all components are healthy

Now, I am trying to deploy my helm charts, note, helm chart deployment by itself works fine, not via Flux though.

flux create source helm label-webhook --url https://github.xxxxxx.com/user1/test-repo/tree/main/chart --namespace label-webhook --cert-file=./tls/label-webhook.pem --key-file=./tls/label-webhook-key.pem --ca-file=./tls/ca.pem --verbose
✚ generating HelmRepository source
► applying secret with repository credentials
✔ authentication configured
► applying HelmRepository source
✔ source created
◎ waiting for HelmRepository source reconciliation
✗ failed to fetch Helm repository index: failed to cache index to temporary file: Get "https://github.xxxxxx.com/user1/test-repo/tree/main/chart/index.yaml": x509: certificate signed by unknown authority

I am generating certs with the process below:

cat << EOF > ca-config.json
{
  "signing": {
    "default": {
      "expiry": "43830h"
    },
    "profiles": {
      "default": {
        "usages": ["signing", "key encipherment", "server auth", "client auth"],
        "expiry": "43830h"
      }
     }
   }
}
EOF
cat << EOF > ca-csr.json
{
    "hosts": [
      "cluster.local"
    ],
    "key": {
      "algo": "rsa",
      "size": 4096
    },
    "names": [
      {
        "C": "AU",
        "L": "Melbourne",
        "O": "xxxxxx",
        "OU": "Container Team",
        "ST": "aks1-dev"
       }
      ]
}
EOF

docker run -it --rm -v ${PWD}:/work -w /work debian bash

apt-get update && apt-get install -y curl &&
curl -L https://github.com/cloudflare/cfssl/releases/download/v1.5.0/cfssl_1.5.0_linux_amd64 -o /usr/local/bin/cfssl && \
curl -L https://github.com/cloudflare/cfssl/releases/download/v1.5.0/cfssljson_1.5.0_linux_amd64 -o /usr/local/bin/cfssljson && \
chmod +x /usr/local/bin/cfssl && \
chmod +x /usr/local/bin/cfssljson

cfssl gencert -initca ca-csr.json | cfssljson -bare /tmp/ca

cfssl gencert \
  -ca=/tmp/ca.pem \
  -ca-key=/tmp/ca-key.pem \
  -config=ca-config.json \
  -hostname="mutation-label-webhook,mutation-label-webhook.label-webhook.svc.cluster.local,mutation-label-webhook.label-webhook.svc,localhost,127.0.0.1" \
  -profile=default \
  ca-csr.json | cfssljson -bare /tmp/label-webhook

root@91bc7986cb94:/work# ls -alrth /tmp/
total 32K
drwxr-xr-x 1 root root 4.0K Jul 29 04:42 ..
-rw-r--r-- 1 root root 2.0K Jul 29 04:43 ca.pem
-rw-r--r-- 1 root root 1.8K Jul 29 04:43 ca.csr
-rw------- 1 root root 3.2K Jul 29 04:43 ca-key.pem
-rw-r--r-- 1 root root 2.2K Jul 29 04:43 label-webhook.pem
-rw-r--r-- 1 root root 1.9K Jul 29 04:43 label-webhook.csr
-rw------- 1 root root 3.2K Jul 29 04:43 label-webhook-key.pem
drwxrwxrwt 1 root root 4.0K Jul 29 04:43 .
root@91bc7986cb94:/work#


root@83faa77cd5f6:/work# cp -apvf /tmp/* .
'/tmp/ca-key.pem' -> './ca-key.pem'
'/tmp/ca.csr' -> './ca.csr'
'/tmp/ca.pem' -> './ca.pem'
'/tmp/label-webhook-key.pem' -> './label-webhook-key.pem'
'/tmp/label-webhook.csr' -> './label-webhook.csr'
'/tmp/label-webhook.pem' -> './label-webhook.pem'
root@83faa77cd5f6:/work# pwd
/work

chmod -R 777 tls/

helm upgrade --install mutation chart --namespace label-webhook --create-namespace --set secret.cert=$(cat tls/label-webhook.pem | base64 | tr -d '\n') --set secret.key=$(cat tls/label-webhook-key.pem | base64 | tr -d '\n') --set secret.cabundle=$(openssl base64 -A <"tls/ca.pem")

I am totally confused as to how to get flux working?

Solution

  • Flux doesn't trust the certificate presented by your git server github.xxxxxx.com

    Quick workaround is to use --insecure-skip-tls-verify flag as described here: https://fluxcd.io/docs/cmd/flux_bootstrap_git/

    Full command:

    flux create source helm label-webhook --url https://github.xxxxxx.com/user1/test-repo/tree/main/chart --namespace label-webhook --cert-file=./tls/label-webhook.pem --key-file=./tls/label-webhook-key.pem --ca-file=./tls/ca.pem --verbose --insecure-skip-tls-verify

    It's interesting there wasn't problem with flux bootstrap git step but it probably just create configuration for repository in this step and not establish connection to it.

    Whatever certificates you are generating don't have anything to do with your GIT server TLS certificate. Seems you're doing some admission webhook magic but the certs you generate there have nothing in common with github.xxxxxx.com certificate so there is no need to specify if in --ca-file flag.

    Permanent solution is to get the CA certificate that signed the github.xxxxxx.com so you need to ask the administrators of the GIT server to provide you CA file and specify that one in --ca-file flag. Not the one you created for your webhook experiments.