ADF Pipeline Scheduled Hourly Fails Occasionally with Lookup Activity Fetching Last Ingested Date from Databricks Catalog

We have scheduled an ADF (Azure Data Factory) pipeline that contains a Lookup activity, which is responsible for fetching the last ingested date from the Databricks catalog(Hive metastore). This pipeline is scheduled to run on an hourly basis, but sometimes it fails with the following error:

we increased timeout and cluster configuration. lookup activity configuration

ErrorCode=FailedToReadFromAzureDatabricksDeltaLake,Failed to read from azure databricks delta lake.
Error message : Failed to send request to Azure Databricks Cluster

The above error may occur during heavy load of data, also it can be network error or fluctuations. To resolve the error here are some steps:

• Ensure your network is configured to access data from databricks.

• Ensure that your Azure Databricks cluster is running. If the cluster is terminated or in an error state, the request will fail.

As @Ganesh Chandrasekaran said if it's an Enterprise account, please open an Azure support ticket with Databricks Engineering.

For more information you can refer to this.