Connecting to SQL on Databricks Using SQLAlchemy or pyodbc

On Databricks, when I try to connect to SQL using SQLAlchemy or pyodbc to run delete queries on a specific table, I get this error: (pyodbc.Error) ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib 'ODBC Driver 17 for SQL Server' : file not found (0) (SQLDriverConnect)".

import sqlalchemy
import urllib
server_ = "<server>"
username = "<usrname>"
password = "pwd"
SQLCred = f"UID="+username+";PWD="+password+";"
params = urllib.parse.quote_plus("DRIVER={ODBC Driver 17 for SQL Server};SERVER="+server_+";DATABASE=db_test;" + SQLCred + "Trusted_Connection=Yes;")
engine = sqlalchemy.create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
conn = engine.connect()

Does anyone have any idea why this error is generated and how it could be fixed?

Dileep Raj Narayan Thumula first of all, I would like to thank you for the explanation. The code with Scala works! I rewrote it in PySpark, and for the database connection part, I used the native Java API, and it worked!

jdbcUsername = "username"
jdbcPassword = "password"
driverClass = "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbcUrl = "jdbc:sqlserver://prtc.database.windows.net:1433;database=db02"
connectionProperties = {
    "user": jdbcUsername,
    "password": jdbcPassword,
    "driver": driverClass
}
connection = spark._jvm.java.sql.DriverManager.getConnection(jdbcUrl, jdbcUsername, jdbcPassword)

stmt = connection.createStatement()
sql = "delete from dbo.ExampleTable where ID = 2"

stmt.execute(sql)
connection.close()