I am using dbt for the DWH automation. I have around 40 procedures which i need to deploy with dbt on my database. It is recommended to use dbt instead of stored procedures but I need to stay with those procedures. I tried to create a new materialization type. I also tried to put the procedure body into macros but I can run only one macro at a time. I need a macro which can run all macros in the path macros/procedures. The following code gives me compiler error 'list' is undefined
{% macro run_models_in_proc() %}
{# Get the list of models from the `macros/procedures` directory #}
{% set models_to_run = list() %}
{% for model in graph.nodes.values() %}
{% if model.path.startswith('macros/procedures/') %}
{% do models_to_run.append(model.name) %}
{% endif %}
{% endfor %}
{# Print the models that will be run #}
{% for model in models_to_run %}
{{ log('Running model: ' ~ model, info=True) }}
{% endfor %}
{% endmacro %}
How do I make the code run?
Depends on the database used, but the idea is to wrap procedure creation in a macro. Then using on-run-start it's possible to invoke the dbt macro that creates the procedure as soon as dbt run/build starts.
Here's an example for snowflake:
File: macros/parse_user_agent.sql
{% macro create_parse_user_agent_udf() %}
create or replace function {{ target.database }}.{{ target.schema }}.parse_user_agent(useragent text)
returns variant
language python
runtime_version = '3.10'
packages = ('ua-parser')
handler = 'parse_user_agent'
as
$$
from ua_parser import user_agent_parser
def parse_user_agent(useragent):
parsed_string = user_agent_parser.Parse(useragent)
return {
'browser_family': parsed_string.get('user_agent', {}).get('family'),
'browser_version_major': parsed_string.get('user_agent', {}).get('major'),
'browser_version_minor': parsed_string.get('user_agent', {}).get('minor'),
'browser_version_patch': parsed_string.get('user_agent', {}).get('patch'),
'os_family': parsed_string.get('os', {}).get('family'),
'os_version_major': parsed_string.get('os', {}).get('major'),
'os_version_minor': parsed_string.get('os', {}).get('minor'),
'os_version_patch': parsed_string.get('os', {}).get('patch'),
'os_version_patch_minor': parsed_string.get('os', {}).get('patch_minor'),
'device_family': parsed_string.get('device', {}).get('family'),
'device_brand': parsed_string.get('device', {}).get('brand'),
'device_model': parsed_string.get('device', {}).get('model')
}
$$
;
{% endmacro %}
dbt_project.yml
must contain the following:
on-run-start:
- '{{ create_parse_user_agent_udf()}}'
Another option is to use dbt run-operation to invoke the macro. This way the on-run-start
part in dbt_project.yml
is not needed.