dbt

Where are the default database and schema defined in dbt Cloud?


So, I'm completely new to dbt and Snowflake, and I'm following a tutorial here to get my feet wet: https://docs.getdbt.com/guides/snowflake. I set up my Snowflake account and my dbt account. I linked the two using Partner Connect Trial.

I can build models and get them deployed into my database/schema (PC_DBT_DB.DBT_ZTRUELOVE) as expected, no problems. However, I then went down the rabbit hole of, what if I want to deploy a model to a different database in my Snowflake warehouse.

I first attempted to just use an inline config statement in my model

{{ config(
    database='RAW',
    schema='JAFFLE_SHOP'
) }}

This materialized the model in the correct database, but the schema ended up being DBT_ZTRUELOVE_JAFFLE_SHOP, not JAFFLE_SHOP as I expected.

Did some googling and ended up

  1. creating a subdirectory under my models folder called raw-jaffle_shop
  2. modifying dbt_project.yml with the configuration below
models:
  jaffle_shop:
      +materialized: table
      # Applies to all files under models/example/
      example:
        +materialized: view
      raw-jaffle_shop:
          +database: RAW
          +schema: JAFFLE_SHOP
          +materialized: view
  1. creating a generate_schema_name.sql file in my macros folder
{% macro generate_schema_name(custom_schema_name, node) -%}
  {%- if custom_schema_name %}
    {{ custom_schema_name | trim }}
  {%- else %}
    {{ target.schema }}
  {%- endif -%}
{%- endmacro %}

  1. moved my model to the new subdirectory
  2. ran dbt run

The model now builds as expected with my view materialized in the correct schema.

My question is...where the heck in dbt Cloud are these default database/schema values configured? I've looked all over and can't seem to find where they're defined, and it's irritating me to no end. lol

And as a bonus...why does dbt want to prefix the default schema value to what I originally specified in my config statement?


Solution

  • where the heck in dbt Cloud are these default database/schema values configured?

    By default, all dbt models are built in the schema specified in your environment (dbt cloud) or profiles.yml file (dbt Core). source


    why does dbt want to prefix the default schema value to what I originally specified in my config statement?

    By default, all dbt models are built in the schema specified in your environment (dbt cloud) or profiles file (dbt Core). If you define a custom schema in dbt_projects.yml file, config() macro, or properties.yml file etc. , dbt concatenates the default schema with the custom one. detailed explanation

    When first using custom schemas, it's a common misunderstanding to assume that a model only uses the new schema configuration; for example, a model that has the configuration schema: marketing would be built in the marketing schema. However, dbt puts it in a schema like <target_schema>_marketing.

    There's a good reason for this deviation. Each dbt user has their own target schema for development (refer to Managing Environments). If dbt ignored the target schema and only used the model's custom schema, every dbt user would create models in the same schema and would overwrite each other's work.

    By combining the target schema and the custom schema, dbt ensures that objects it creates in your data warehouse don't collide with one another. source

    If you don't like this default behavior, you can override it using generate_schema_name macro.