continuous-integrationdbtincremental-load

Is there a way to make the dbt_cloud_pr_xxxx_xxx a clone of an existing data?

so using dbt cloud, and having a run on every pull request, but my incremental models are fully refreshed since everything runs in a new db destination (dbt_cloud_pr_xxxxx_xxx) any way of solving this? perhaps creating the new destination as a clone of an old one?

Solution

  • dbt calls this "Slim CI". You can use their "deferral" and "state comparison" features -- they will check the manifest of the compiled project to the manifest from another run you specify (typically the last production run). Any models that are unchanged will have ref() compile to the prod target, and then you can use the --state:modified+ selector in your dbt Cloud job definition to only rebuild the models with changes.

    See the docs for CI in dbt Cloud.