How to take input from users in an argo gitops repo

I am struggling with how to take input for my gitops repo. I am providing a managed kubernetes configuration, that provides a developer experience.

Now the teams that need to utilize it, will enable the config with something like this:

# Consumer managed yaml, not part of my repo
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: asterix
  namespace: argo
spec:
  project: default
  source:
    repoURL: 'git@github.com:repo/devex-argo.git'
    targetRevision: feat/feast
    path: 'environments/devex-stable'
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: argo
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

The team will not be capable of adding patches themselves, or maintaining overlays. Also, the scope is too big for me to add an overlay per cluster, and also in my mind defeats the purpose.

The environemnts/devex-stable looks like this:

# environemnts/devex-stable
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - ../../services/feast/

and my feast sample looks like this:

# services/feast/kustomize
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace-feast.yaml
- argocd-feast.yaml
- ingress-feast.yaml

the argocd-feast is simply an argo/application that deploys the helm chart. Additionally i deploy an ingressroute like this:

# services/feast/ingress-feast.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: feast.DOMAIN
  namespace: feast
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  entryPoints:
    - websecure
    - web
  routes:
    - kind: Rule
      match: Host(feast.DOMAIN) && PathPrefix(/)
      services:
        - name: feast-feature-server
          port: 6566
  tls:
    certResolver: le
    domains:
      - main: feast.DOMAIN

and replace the DOMAIN string with the user supplied value.

I tried using kustomize, but seems to get stuck on it. My most promising soloution was a patch like this:

# services/feast/kustomize
...
replacements:
- source:
    kind: IngressRoute
    fieldPath: metadata.annotations.domain
  targets:
    - select:
        kind: IngressRoute
      fieldPaths:
        - metadata.name
        - spec.routes.0.match
        - spec.tls.domains.0.main
      options:
        delimiter: '.'
        index: 1

and adding this to the consumer yaml:

# consumer managed yaml
...
    kustomize:
      commonAnnotations:
        domain: "MyCustomDomain.com"  # Substitute this with the dynamic value as needed
      commonAnnotationsEnvsubst: true

But it seems the patch from my own repo runs before the annotation is changed to the customer override.

I know i can achieve this with helm, but i would prefer not to create helm charts with only 1 ressource in them for the sake of templating (unless that is the only way of course).

Does anyone have any ideas on this?

Solution

So I finally made a solution that works, it took a while so I will document my findings in this answer

The solution

Kustomize itself does not support environment variables, so I made a simple argocd plugin that runs envsubst for all variables prefixed "ARGOCD_ENV" before it builds the kustomize projects.

For reference, when you set a env var as input from the user side, it will automatically get the ARGOCD_ENV prefix.

Argocd was deployed with these overrides:

#values.yaml
repoServer:
  volumes:
   - name: argocd-cmp-cm
     configMap:
       name: argocd-cmp-cm
   - name: cmp-tmp
     emptyDir: {}
  extraContainers:
    - name: envsubst
      command:
        - "/var/run/argocd/argocd-cmp-server"
      image: bitnami/kubectl
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
      volumeMounts:
        - mountPath: /var/run/argocd
          name: var-files
        - mountPath: /home/argocd/cmp-server/plugins
          name: plugins
        # Remove this volumeMount if you've chosen to bake the config file into the sidecar image.
        - mountPath: /home/argocd/cmp-server/config/plugin.yaml
          subPath: envsubst.yaml
          name: argocd-cmp-cm
        # Starting with v2.4, do NOT mount the same tmp volume as the repo-server container. The filesystem separation helps
        # mitigate path traversal attacks.
        - mountPath: /tmp
          name: cmp-tmp

configs:
  cmp:
    create: true
    plugins:
      envsubst:
        init:
          command: ["/bin/sh", "-c"]
          args: [
             "for file in $(find $(git rev-parse --show-toplevel) -type f); do envsubst $(compgen -v | grep ARGOCD_ENV | sed 's/^/\$/') < $file > $file.tmp && mv $file.tmp $file; done"
          ]
        generate:
          command: ["/bin/sh", "-c"]
          args: [
            "kubectl kustomize ."
            ]

Now in my kustomize files, i add my ingressroute as:

#ingress.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: feast.${ARGOCD_ENV_DOMAIN}
  namespace: feast
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  entryPoints:
    - websecure
    - web
  routes:
    - kind: Rule
      match: Host(feast.${ARGOCD_ENV_DOMAIN}) && PathPrefix(/)
      services:
        - name: feast-feature-server
          port: 6566
  tls:
    certResolver: le
    domains:
      - main: feast.${ARGOCD_ENV_DOMAIN}

Finally, I deploy the repo to my cluster like this:

#argoApplication.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: asterix
  namespace: argo
spec:
  project: default
  source:
    repoURL: 'git@github.com:repo/devex-argo.git'
    targetRevision: feat/feast
    path: 'environments/devex-stable'
    plugin:
      name: envsubst
      # my custom variables
      env:
        - name: DOMAIN
          value: mydomain.com
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: argo
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Some details as to why I did not just use helm or Kustomize

I tried to do this natively with both helm and Kustomize, but hit a wall in both cases.

Kustomize

Kustomize is not meant for templating in its native form, though possible with plugins it seems: https://github.com/kubernetes-sigs/kustomize/issues/4120#issuecomment-912037164

Helm

Another approach is to deploy the cluster setup as a helm charts with sub charts. The issue I ran into here has to do with multi namespaces, as helm deploys all sub charts into the same namespace as the main chart. Some charts have the option to override the namespace, but there are no guarantees all helm charts has been written with that option. In my case, this is a dealbreaker as I deploy around 15-20 services, that should live in separate namespaces