airflowairflow-2.xairflow-webserverairflow-apiairflow-taskflow

How can i get all dags details from airflow rest api


i have more dags (say 150dags) in my airflow account. My airflow version = 2.4.0

im using url and query parameter http://localhost:8081/api/v1/dags?limit=1000&offset=0

But my response consists only 100 records(dags). i think query parameters are not working here.

How can i fetch all dags available inside my airflow.


Solution

  • there is a PR that explain that the limit can not be passed and if you put more then the limit the fallback would be the limit

    if you want to change the limit in the api, you can change "maximum_page_limit" in airflow.cfg to other number (default = 100)

    Another option is to play with offset until you do not get dags in the list. for example : first call limit=100, offset=0, second call limit=100, offset=101 and so on until empty response.

    http://localhost:8081/api/v1/dags?limit=100&offset=0
    http://localhost:8081/api/v1/dags?limit=100&offset=101
    

    also, an option not in the api is to create a dag with a task and using DagBag to get all dags details.

    in this example I print all the dag ids

    from datetime import datetime
    
    from airflow import DAG, settings
    from airflow.decorators import task
    from airflow.models import DagBag
    
    with DAG(
            dag_id="test_dag",
            schedule_interval=None,
            default_args={
                "start_date": datetime(2022, 1, 1),
                "retries": 0,
                "catchup": False,
            },
            render_template_as_native_obj=True,
            tags=["test"],
    ) as dag:
        dag.doc_md = __doc__
    
    
        @task
        def print_dags():
            dagbag = DagBag(settings.DAGS_FOLDER)
            print(dagbag.dags.keys())
    
    
        (print_dags())