[SOLVED] Cosmos DbtDag running model can not find environment variables used in macros

Cosmos DbtDag running model can not find environment variables used in macros

I am running a DBT model in Airflow using the Astronomer-Cosmos SDK. I have run demos successfully using this environment. I can run my models from the DBT command line successfully also. However when I tried to run using a DAG running in Airflow that contains macros it fails to resolve the statements:

 database_name = env_var('DB_PREFIX')+"_"+env_var('DB_ENV')+database_subfix -%}

I have added the following to the Dockerimage

ENV DB_PREFIX=ZZ_ID

I also tried to override the file at /etc/environment in the docker image. I can see the variables when I logon to the running container issuing a 'set' command. They are also defined in the Airflow Admin Environment Variables. Can anybody please give me a hint here?

Solution

In DBT Cosmos the environment variables need to be passed in a ProjectConfig. That makes it available to all calls of env_var(). In the example below the OS environment variables are captured to a local variable and then passed to the ProfileConfig which it will be pass to the DbtDag class.

ENV_DB_PREFIX = os.getenv("DB_PREFIX", "DV")
ENV_DBT_USER_ID = os.getenv("DBT_USER_ID", "my_user_id")
project_config = ProjectConfig(
    "/usr/local/airflow/dbt",
    env_vars={"DBT_USER_ID": ENV_DBT_USER_ID, "DB_PREFIX": ENV_DB_PREFIX},
)

dbt_snowflake_dag = DbtDag(project_config=project_config,
                           operator_args={"install_deps": True},
                           profile_config=profile_config,
                           execution_config=execution_config,
                           schedule_interval="@hourly",
                           start_date=datetime(2024, 1, 30),
                           catchup=False,
                           dag_id="sample_dag", )