azuredatabricksazure-databricksdatabricks-unity-catalog

Automatic creation of unity-catalog-access-connector

When deploying a simple databricks workspace on Azure it seems that databricks itself creates a unity access connector called 'unity-catalog-access-connector' inside the databricks managed resource group.

Our workspaces are not automatically enabled for unity catalog and if we enable a workspace we create our own access connector.

Is this access connector required? Can it be avoided that it is created automatically for every workspace?

enter image description here

enter image description here

Solution

  • This connector is part of the infrastructure that supports Unity Catalog, which is Databricks unified governance solution for all data assets across an organization.

    In Unity Catalog, all data can be referenced using a three-level namespace: catalog.schema.table. It is a unified governance solution for managing data and AI assets on the Lakehouse platform.

    Setting the default catalog for the workspace determines the catalog used for queries that do not reference a fully qualified three-level name. For example, if the default catalog is set to 'retail_prod', a query like 'SELECT * FROM myTable' would reference 'retail_prod.default.myTable', with 'default' assumed as the schema.

    Note:This setting requires a restart of clusters and SQL warehouses to take effect. It applies only to Unity Catalog compatible compute, meaning the workspace must have an assigned Unity Catalog metastore, and the cluster must be in 'Shared' or 'Single User' access mode, or be a SQL warehouse.

    As you asked Can it be avoided that it is created automatically for every workspace?

    I have tried the below approach : My workspace is enabled like below

    enter image description here

    I have created a Delta table and I did

    %sql
describe extended d01

    enter image description here

    From the above image you can see the Catalog used for the Table is my workspace name dileepdbx.

    If you do not want to use the catalog.

    Go to Catalog in databricks workspace. enter image description here

    Select the catalog workspace and REVOKE

    enter image description here

    Results:

    Users cannot access the selected catalog from this workspace.
Use the below screen to configure which workspaces are allowed to access the catalog.

    %sql
describe extended d01

    [[UNAUTHORIZED_ACCESS](https://docs.microsoft.com/azure/databricks/error-messages/error-classes#unauthorized_access)] Unauthorized access: PERMISSION_DENIED: Catalog 'dilepdbx' is not accessible in current workspace SQLSTATE: 42501