databricksazure-databrickspyodbc

Adding init .sh file for databricks job


I have a python script that's using a cluster that has an init script:

 /Volumes/projects/default/admin/odbc.sh

this helps me connect via odbc to a sql server table.

When I run the job and add the cluster that has this ^ I see the error I saw before adding this to my cluster which was:

Can't open lib 'ODBC Driver 17 for SQL Server' : file not found

My question is, how to add this .sh file to my job??


Solution

  • I have tried the below approach:

    The below code downloads the install MS SQL ODBC Driver on Azure Databricks cluster.

    First Approach:

    %sh
    curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
    curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
    sudo apt-get update
    sudo ACCEPT_EULA=Y apt-get -q -y install msodbcsql17
    

    Second Approach:

    if ! [[ "18.04 20.04 22.04 23.04 24.04" == *"$(lsb_release -rs)"* ]];
    then
        echo "Ubuntu $(lsb_release -rs) is not currently supported.";
        exit;
    fi
    
    curl https://packages.microsoft.com/keys/microsoft.asc | sudo tee /etc/apt/trusted.gpg.d/microsoft.asc
    
    curl https://packages.microsoft.com/config/ubuntu/$(lsb_release -rs)/prod.list | sudo tee /etc/apt/sources.list.d/mssql-release.list
    
    sudo apt-get update
    sudo ACCEPT_EULA=Y apt-get install -y msodbcsql18
    #optional: for bcp and sqlcmd
    sudo ACCEPT_EULA=Y apt-get install -y mssql-tools18
    echo 'export PATH="$PATH:/opt/mssql-tools18/bin"' >> ~/.bashrc
    source ~/.bashrc
    #optional: for unixODBC development headers
    sudo apt-get install -y unixodbc-dev
    

    Results: enter image description here

    enter image description here

    Reference: