google-cloud-platformairflowdirected-acyclic-graphsgoogle-cloud-composergoogle-cloud-shell

Check to see if a specific DAG is present or not in Composer using Cloud Shell?

I am trying to use a Cloud Shell command to check if a specific DAG is present or not (if it exists), but I am unsure what command to write in order to accomplish this.

For example, if I wanted to check to see if my_dag.py exists, I would like Cloud Shell to indicate whether it is present or not.

So far, I have used this command, but it lists all of the DAGS:

gcloud composer environments storage dags list --environment=ENVIRONMENT --location=LOCATION

I want to specifically check to see if a certain DAG exists or not using Cloud Shell, but I do not know how what command to use.

Solution

  • The airflow dags list command (which is what this Cloud Shell command is using) has a --subdir option where you can specify the file path to a DAG file. The output will be the DAGs tied to that fileloc.

    gcloud composer environments storage dags list --environment=ENVIRONMENT --location=LOCATION --subdir dags/path/to/file.py