airflowdbtastronomer

Cosmos DbtDag running model can not find environment variables used in macros


I am running a DBT model in Airflow using the Astronomer-Cosmos SDK. I have run demos successfully using this environment. I can run my models from the DBT command line successfully also. However when I tried to run using a DAG running in Airflow that contains macros it fails to resolve the statements:

 database_name = env_var('DB_PREFIX')+"_"+env_var('DB_ENV')+database_subfix -%}

I have added the following to the Dockerimage

ENV DB_PREFIX=ZZ_ID

I also tried to override the file at /etc/environment in the docker image. I can see the variables when I logon to the running container issuing a 'set' command. They are also defined in the Airflow Admin Environment Variables. Can anybody please give me a hint here?


Solution

  • In DBT Cosmos the environment variables need to be passed in a ProjectConfig. That makes it available to all calls of env_var(). In the example below the OS environment variables are captured to a local variable and then passed to the ProfileConfig which it will be pass to the DbtDag class.

    ENV_DB_PREFIX = os.getenv("DB_PREFIX", "DV")
    ENV_DBT_USER_ID = os.getenv("DBT_USER_ID", "my_user_id")
    project_config = ProjectConfig(
        "/usr/local/airflow/dbt",
        env_vars={"DBT_USER_ID": ENV_DBT_USER_ID, "DB_PREFIX": ENV_DB_PREFIX},
    )
    
    dbt_snowflake_dag = DbtDag(project_config=project_config,
                               operator_args={"install_deps": True},
                               profile_config=profile_config,
                               execution_config=execution_config,
                               schedule_interval="@hourly",
                               start_date=datetime(2024, 1, 30),
                               catchup=False,
                               dag_id="sample_dag", )