[SOLVED] How can I run python code after a DBT run (or a specific model) is completed?

How can I run python code after a DBT run (or a specific model) is completed?

I would like to be able to run an ad-hoc python script that would access and run analytics on the model calculated by a dbt run, are there any best practices around this?

Solution

We recently built a tool that could that caters very much to this scenario. It leverages the ease of referencing tables from dbt in Python-land. It's called dbt-fal.

The idea is that you would define the python scripts you would like to run after your dbt models are run:

# schema.yml
models:
- name: iris
  meta:
    owner: "@matteo"
    fal:
      scripts:
        - "notify.py"

And then the file notify.py is called if the iris model was run in the last dbt run:

# notify.py
import os
from slack_sdk import WebClient
from slack_sdk.errors import SlackApiError

CHANNEL_ID = os.getenv("SLACK_BOT_CHANNEL")
SLACK_TOKEN = os.getenv("SLACK_BOT_TOKEN")

client = WebClient(token=SLACK_TOKEN)
message_text = f"""Model: {context.current_model.name}
Status: {context.current_model.status}
Owner: {context.current_model.meta['owner']}"""


try:
    response = client.chat_postMessage(
        channel=CHANNEL_ID,
        text=message_text
    )
except SlackApiError as e:
    assert e.response["error"]

Each script is ran with a reference to the current model for which it is running in a context variable.

To start using fal, just pip install fal and start writing your python scripts.