We have a multi-tenant kubernetes cluster, that hosts various customer environments. Kubernetes namespaces are used to isolate these customer environments. Each of these namespaces have a similar set of k8s resources (deployments, configmap etc) configured. Some of these resources are identical across namespaces, while some are not identical.
From a continuous delivery perspective, I am exploring options to roll out changes to these identical components (across namespaces) in a seamless manner. Git-Ops/Pull-based-Continuous-Delivery seems to be a good approach, which would enable us to seamlessly manage 100s of namespaces across various clusters.
Explored a couple of Git-Ops tools such as ArgoCD, Fluxv2 etc... but couldn't figure out whether these tools would let you roll out changes to multiple namespaces simultaneously (or within a predictable time window). It will be helpful, if you can guide/advice me in terms of choosing the right tool/approach to perform roll out to multiple namespaces. It will also be good to understand whether these Git-Ops tools can be customised to handle such scenarios.
An illustration of how namespaces are setup in our k8s cluster.
Namespace: customer1
Deployments: app1, common-app, common-service
Configmaps: cm1, common-cm
Namespace: customer2
Deployments: app2, common-app, common-service
Configmaps: cm2, common-cm
common-app
, common-service
, common-cm
are identical across environments/namespaces.
app1
, cm1
, app2
, cm2
are not identical. They have different image tags, labels etc.
I'm using Fluxv2. My approach will be to create charts for common-app
, common-service
, etc with same values and define the namespace on separate HelmReleases
So you can put the common-app
chart on clusters/production/charts/common-apps
and refer the chart path on multiple HelmReleases
which can look like this
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: common-apps-customer1
namespace: customer1
labels:
chart: common-apps
spec:
interval: 1m
releaseName: common-apps
chart:
spec:
chart: clusters/production/charts/common-apps
sourceRef:
kind: GitRepository
name: manifests
namespace: flux-system
values:
some_key: some_value
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: common-apps-customer2
namespace: customer2
labels:
chart: common-apps
spec:
interval: 1m
releaseName: common-apps
chart:
spec:
chart: clusters/production/charts/common-apps
sourceRef:
kind: GitRepository
name: manifests
namespace: flux-system
values:
some_key: some_value
Update
for ns in $(kubectl get ns --no-headers | cut -d " " -f1); do
flux create helmrelease common-apps-$ns \
--source=GitRepository/manifests \
--chart=common-apps \
--release-name=common-apps \
--target-namespace=$ns \
--interval=1m0s \
--export > ./clusters/production/common-apps-$ns.yaml
done;
This will generate the necessary manifests that should be anyway synced with your repo.