Errors with df.display() and df.show() in Databricks

I am reaching out to you for assistance with some issues I'm encountering in my Databricks environment. I'm hoping the community can provide some guidance to help me resolve these problems.

1. Error with df.display(): When I try to use the df.display() function, I'm receiving the following error: "'Failed to store the result. Try rerunning the command. Failed to upload command result to DBFS. Error message: PUT request to create file error HttpResponse Proxy(HTTP/1.1 404 The specified filesystem does not exist. [Content-Length: 175. Content-Type: application/json;charset=utf-8, Server: Windows-Azure-HDFS/1.0 Microsoft-HTTPAPI/2.0, x-ms-error-code: FilesystemNotFound, x-ms-request-id: 33854442-8011- 0028-3da6-bc0285000000, x-ms-version: 2021-04-10, Date: Wed, 12 Jun 2024 08:59:45 GMT] Response Entity Proxy([Content-Type: application/json;charset=utf-8 Content-Length: 175, Chunked: false])) "

2. Error with df.show(): I'm also facing a similar issue when using df.show() for data frames with more than 10-20 rows. The same error as mentioned in point 1 appears.

Screenshots of errors

Debugging insights:

1. During my investigation, I found that the issue seems to be related to Databricks trying to access the root storage and failing to write the data.

error on cluster logs: ''* ERROR PresignedUrlClientUtils$: FS_OP_CREATE FILE[https://<root_storage-account-name>.dfs.core.windows.net/jobs/4079550013704479 /command-results/4056370604825597/b3041abc-2eeb-45da-90b7-23d98973d4d0] Presigned URL: Failed to upload stream using AzureAdl2SasUri*"

2. I tried to upload some files to DBFS (Databricks File System) from the UI, but it's throwing an error that says 'The Azure Container Does Not exist' and also a 500 error code but i can see managed storage is there in managed reource group.

3. Same error faced when I tried to import 3 to 4 notebook on shared location

4. I tried with different different types of clusters and databricks run time also but same issues

5. I have Admin access on databricks workspace and I am using Premium (+ Role-based access controls) databricks

6. Currently I am using another storage account for my DE Work using storage credentials I am able to access data easily in Notebooks

I'm not sure how to resolve this problem, as the root storage is inside the managed resource group of Databricks I don't have any control on that.

I wanted to share some updates regarding the issues I've been encountering in my Databricks environment.

After raising a ticket with Microsoft and collaborating with their team for approximately a week, we undertook several troubleshooting steps. Ultimately, both the storage team and the Databricks team informed us that when launching resources, the root storage account was created due to some issues, which resulted in the root container not being created. Consequently, this led to the problems we experienced. Since no one has control over the root storage, it should be created automatically. To resolve the issues, we need to re-provision the resources, after which the problems were resolved.