We have our primary data in snowflake and we want to share it with some users over in databricks.
The ask is not to "copy the data into databricks" as it will add to the cost. Rather just access it using SQL queries from databricks.
I know you can use scala or python to create connections and put in something like a dataframe and read it.
But I wanted to see if we can create like an "external table" in databricks and point to data from snowflake.
I can into this link: https://docs.databricks.com/external-data/snowflake.html#language-sql
Under the "Query a Snowflake table in Databricks" link, click on "SQL" tab
and it kind of does that, but does anybody know that databricks create this table as external table OR copies the data over and they make it available?
Databricks recently announced "Lakehouse Federation" which allows you to connect and read from external data warehouses including Snowflake in a unified way with Unity Catalog.
There is documentation for Snowflake specifically here:
https://learn.microsoft.com/en-us/azure/databricks/query-federation/snowflake
And not to make my own plug, but I made a video on it using SQL Server; the steps are very similar for Snowflake:
https://www.youtube.com/watch?v=k1JvTEZY8c0&feature=youtu.be
Note: as of this writing, this feature is limited to read-only access.