sqldatabasestored-proceduresjinja2dbt

Create procedures in dbt


I am using dbt for the DWH automation. I have around 40 procedures which i need to deploy with dbt on my database. It is recommended to use dbt instead of stored procedures but I need to stay with those procedures. I tried to create a new materialization type. I also tried to put the procedure body into macros but I can run only one macro at a time. I need a macro which can run all macros in the path macros/procedures. The following code gives me compiler error 'list' is undefined

{% macro run_models_in_proc() %}
    {# Get the list of models from the `macros/procedures` directory #}
    {% set models_to_run = list() %}
    
    {% for model in graph.nodes.values() %}
        {% if model.path.startswith('macros/procedures/') %}
            {% do models_to_run.append(model.name) %}
        {% endif %}
    {% endfor %}

    {# Print the models that will be run #}
    {% for model in models_to_run %}
        {{ log('Running model: ' ~ model, info=True) }}
    {% endfor %}
{% endmacro %}

How do I make the code run?


Solution

  • Depends on the database used, but the idea is to wrap procedure creation in a macro. Then using on-run-start it's possible to invoke the dbt macro that creates the procedure as soon as dbt run/build starts.

    Here's an example for snowflake:

    File: macros/parse_user_agent.sql

    {% macro create_parse_user_agent_udf() %}
    
    create or replace function {{ target.database }}.{{ target.schema }}.parse_user_agent(useragent text)
    returns variant
    language python
    runtime_version = '3.10'
    packages = ('ua-parser')
    handler = 'parse_user_agent'
    as
    $$
    from ua_parser import user_agent_parser
    
    def parse_user_agent(useragent):
        parsed_string = user_agent_parser.Parse(useragent)
        return {
            'browser_family': parsed_string.get('user_agent', {}).get('family'),
            'browser_version_major': parsed_string.get('user_agent', {}).get('major'),
            'browser_version_minor': parsed_string.get('user_agent', {}).get('minor'),
            'browser_version_patch': parsed_string.get('user_agent', {}).get('patch'),
            'os_family': parsed_string.get('os', {}).get('family'),
            'os_version_major': parsed_string.get('os', {}).get('major'),
            'os_version_minor': parsed_string.get('os', {}).get('minor'),
            'os_version_patch': parsed_string.get('os', {}).get('patch'),
            'os_version_patch_minor': parsed_string.get('os', {}).get('patch_minor'),
            'device_family': parsed_string.get('device', {}).get('family'),
            'device_brand': parsed_string.get('device', {}).get('brand'),
            'device_model': parsed_string.get('device', {}).get('model')
        }
        $$
    ;
    
    {% endmacro %}
    

    dbt_project.yml must contain the following:

    on-run-start:
      -  '{{ create_parse_user_agent_udf()}}'
    

    Another option is to use dbt run-operation to invoke the macro. This way the on-run-start part in dbt_project.yml is not needed.