gitgithubkubernetesairflowgit-sync

Is there a different way to mount DAGs from private repo to Airflow on Kubernetes with GitSync without using SSH keys?


I have an Airflow environment (v2.4.3) on Kubernetes and I want to sync it with a private git repo so that any changes I make to DAGs in my master branch get automatically picked up by my Airflow environment.

According to Airflow documentation, I can use Git-sync sidecar along with an SSH key added to my private git repo and Airflow env to make it work.

However, given that I am constantly creating new private repos and Airflow environments, I am wondering if there is a more simple way of connecting my private git repos to their respective Airflow environment.

If I have a webapp managing my Airflow environments and have access to an OAuth token from Github after signing into my account (or any other git service), could I use that to connect my an Airflow environement and sync changes to any git repo of my choice under my account?


Solution

  • I was able to figure it out.

    One can use personal access tokens as passwords provided by whatever git service the private repo is in along with the repo's username.

    I just stored the personal access token as an Opaque secret in my Airflow K8s cluster and referenced that in my git-sync sidecar container yaml definition which I included in my Airflow yaml deployment definition.

          containers:
            - name: git-sync
              image: registry.k8s.io/git-sync/git-sync:v3.6.5
              args:
                - "-wait=60"
                - "-repo=<repo>"
                - "-branch=master"
                - "-root=/opt/airflow/dags"
                - "-username=<username>"
                - "-password-file=/etc/git-secret/token"
              volumeMounts:
                - name: git-secret
                  mountPath: /etc/git-secret
                  readOnly: true
                - name: dags-data
                  mountPath: /opt/airflow/dags
          volumes:
            - name: dags-data
              emptyDir: {}
            - name: git-secret
              secret:
                secretName: github-token