pythonairflowetlorchestration

How can I access @dag decorator parameters within tasks


I try to use Apache Airflow's @dag decorator with parameters (params argument) to be able to run same instructions for different configurations, but can't find information on how to access these params' values within the code.

Here is the sample:

@dag(
        dag_id='DAG_ID',
        start_date=datetime(2023, 6, 1),
        params={
            'id': 'ID_A'
        }
)
def main(**kwargs):
    print(...) # want to print 'ID_A' here

I want to print 'ID_A' value in the function and later pass it as an argument to one of the DAG tasks, how can I do it?


Solution

  • You can use Jinja Template,

    also, you can get the dag_run and find it inside "conf" property.

    from datetime import datetime
    
    from airflow.decorators import task, dag
    
    id_param = "{{ params.id }}"
    
    
    @dag(
        dag_id='DAG_ID',
        start_date=datetime(2023, 6, 1),
        params={
            'id': 'ID_A'
        }
    )
    def my_dag():
    
        @task
        def print_value(value):
            print(value)
    
        @task
        def get_conf():
            context = get_current_context()
            dag_run: DagRun = context["dag_run"]
            print(dag_run.conf.get("id"))
    
        print_value(id_param) >> get_conf()
    
    
    my_dag()