azure-cosmosdbazure-databricksfeature-storedatabricks-unity-catalog

Publish features to cosmos dB using Azure Databricks Feature Store Client fails on workspace with unity catalog enabled


we are trying to create an online feature store using cosmosdb following this documentation: https://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/publish-features .

But I get an error when I publish the table to cosmosdb: AnalysisException: Catalog 'cosmoscatalog' not found. The issue only happens when using unity-enabled workspaces. I can publish using a non-unity enabled workspace.

P.S. If I create the table using the non-unity enabled workspace, then the unity-enabled workspace can update the cosmosdb. But the unity-enabled worskpace cannot create the cosmos container/database using the fs.publish_table.

I tried the following code:

from databricks.feature_store.online_store_spec import AzureCosmosDBSpec
from databricks.feature_store.client import FeatureStoreClient

fs = FeatureStoreClient()

account_uri = "https://online-feature-store.documents.azure.com:443/"

# Specify the online store.
online_store_spec = AzureCosmosDBSpec(
  account_uri=account_uri,
  write_secret_prefix="secret/write-cosmos",
  read_secret_prefix="secret/read-cosmos",
  database_name="online_feature_store_example",
  container_name="feature_store_online_wine_features"
)

# Push the feature table to online store.
fs.publish_table("online_feature_store_example.wine_static_features", online_store_spec, mode='merge')

The following code works on workspaces without unity catalog enabled. However, on a unity-catalog enabled workspace, it trhows an error: AnalysisException: Catalog 'cosmoscatalog' not found


Solution

  • You need to create the database and container in CosmosDB with the name you are specifying in AzureCosmosDBSpec.