databricksazure-synapsedelta-live-tables

Trigger Databricks Delta Live Tables Pipeline from Synapse

We would like to trigger and run a Databricks Delta Live Tables Pipeline from an Azure Synapse pipeline which creates a couple of bronze and silver tables.

enter image description here

I cant find any info on that.

However, it is possible to run Databricks notebooks in Synapse pipelines.

enter image description here

So second best would be, to run a Datatbricks notebook from Synapse. But how can I triger a Delta Live Tables pipeline from a Databricks notebook then?

Solution

  • You cannot call dlt pipeline by using databricks notebook in synapse pipeline, but what you can do is call web activity which can be triggered manually.

    Follow below steps.

    First, add a schedule to your dlt pipeline.

    enter image description here

    After, clicking on Add a schedule you will get below options.

    enter image description here

    Add Job name, check Schedule as Manual and then click on create.

    Next, go these created job under Workflows.

    enter image description here

    There, copy the Job Id.

    Now, in synapse pipeline add Web activity and add below details as shown in image.

    1. URL = https://<Your_databricks_url>/api/2.1/jobs/run-now
    2. METHOD = POST
    3. Body = { "job_id": "Job_id" } Add job id you copied before.
    4. Headers = Authorization as Bearer <Your_databricks_token> and Content-Type as application/json

    enter image description here

    After adding all those details call the web activity.

    Output:

    enter image description here

    You will get return response having run_id.

    enter image description here