kubernetescontinuous-deliveryargocdgitopsfluxcd

GitOps: Roll out changes to k8s resources across multiple namespaces


We have a multi-tenant kubernetes cluster, that hosts various customer environments. Kubernetes namespaces are used to isolate these customer environments. Each of these namespaces have a similar set of k8s resources (deployments, configmap etc) configured. Some of these resources are identical across namespaces, while some are not identical.

From a continuous delivery perspective, I am exploring options to roll out changes to these identical components (across namespaces) in a seamless manner. Git-Ops/Pull-based-Continuous-Delivery seems to be a good approach, which would enable us to seamlessly manage 100s of namespaces across various clusters.

Explored a couple of Git-Ops tools such as ArgoCD, Fluxv2 etc... but couldn't figure out whether these tools would let you roll out changes to multiple namespaces simultaneously (or within a predictable time window). It will be helpful, if you can guide/advice me in terms of choosing the right tool/approach to perform roll out to multiple namespaces. It will also be good to understand whether these Git-Ops tools can be customised to handle such scenarios.

An illustration of how namespaces are setup in our k8s cluster.

Namespace: customer1
Deployments: app1, common-app, common-service
Configmaps: cm1, common-cm

Namespace: customer2
Deployments: app2, common-app, common-service
Configmaps: cm2, common-cm

common-app, common-service, common-cm are identical across environments/namespaces.

app1, cm1, app2, cm2 are not identical. They have different image tags, labels etc.


Solution

  • I'm using Fluxv2. My approach will be to create charts for common-app, common-service, etc with same values and define the namespace on separate HelmReleases

    So you can put the common-app chart on clusters/production/charts/common-apps and refer the chart path on multiple HelmReleases which can look like this

    ---
    apiVersion: helm.toolkit.fluxcd.io/v2beta1
    kind: HelmRelease
    metadata:
      name: common-apps-customer1
      namespace: customer1
      labels:
        chart: common-apps
    spec:
      interval: 1m
      releaseName: common-apps
      chart:
        spec:
          chart: clusters/production/charts/common-apps
          sourceRef:
            kind: GitRepository
            name: manifests
            namespace: flux-system
      values:
        some_key: some_value
    ---
    apiVersion: helm.toolkit.fluxcd.io/v2beta1
    kind: HelmRelease
    metadata:
      name: common-apps-customer2
      namespace: customer2
      labels:
        chart: common-apps
    spec:
      interval: 1m
      releaseName: common-apps
      chart:
        spec:
          chart: clusters/production/charts/common-apps
          sourceRef:
            kind: GitRepository
            name: manifests
            namespace: flux-system
      values:
        some_key: some_value
    

    enter image description here

    Update

    for ns in $(kubectl get ns --no-headers | cut -d " " -f1); do    
        flux create helmrelease common-apps-$ns \
        --source=GitRepository/manifests \
        --chart=common-apps \
        --release-name=common-apps \
        --target-namespace=$ns \
        --interval=1m0s \
        --export > ./clusters/production/common-apps-$ns.yaml     
    done;
    

    This will generate the necessary manifests that should be anyway synced with your repo.

    enter image description here