pythonazurehttpazure-blob-storage

Handle DNS timeout with call to blob_client.upload_blob


While using the Azure storage SDK in Python, I have been unable to override what appears to be a default 90-second timeout to catch a DNS exception occurring within a call to blob_client.upload_blob(). I am looking for a way to override this with a shorter time interval (i.e. 5 seconds).

The following code illustrates this issue using a fictitious account name which DNS cannot resolve. I am using a timeout argument in the call to upload_blob, and I understand from reviewing documentation this enforces a server-side threshold, not a client-side threshold. I have not been successful in getting a client-side threshold to be enforced.

This issue appears similar to this unanswered question: How to handle timeout for Uploading a blob in Azure Storage using Python SDK?. The one (not accepted) solution suggests using a timeout threshold within the call to upload_blob. As noted above (and shown within the code below), this is not producing the desired effect.

from azure.core.exceptions import AzureError
from azure.storage.blob import BlobServiceClient

# Define Azure Storage Blob connection details
connection_string = "DefaultEndpointsProtocol=https;AccountName=test;AccountKey=removed==;EndpointSuffix=core.windows.net"
container_name = "containername"
blob_name = "blobname"
local_file_path = "c:/temp/test.txt"
 
 
# Create the BlobServiceClient
blob_service_client = BlobServiceClient.from_connection_string(connection_string)
 
# Function to perform the blob upload
def upload_blob_process():
    try:
        with open(local_file_path, "rb") as data:
            blob_client = blob_service_client.get_blob_client(container_name, blob_name)
            blob_client.upload_blob(data, timeout=5)
        print("Blob uploaded successfully!")
    except AzureError as e:
        print(f"Azure error occurred: {e}")
    except Exception as e:
        print(f"An error occurred: {e}")
 
 
upload_blob_process()

Solution

  • While using the Azure storage SDK in Python, I have been unable to override what appears to be a default 90-second timeout to catch a DNS exception occurring within a call to blob_client.upload_blob(). I am looking for a way to override this with a shorter time interval (i.e. 5 seconds).

    As, I mentioned in comment, The timeout parameter in upload_blob() applies to each individual HTTP request to the Azure Blob service and it does not affect DNS resolution or connection establishment timeouts but instead limits the server-side operation duration for each request.

    thanks, I'm aware. Do you know of any way to apply a DNS timeout on top of this?

    Code:

    import dns.resolver
    from azure.storage.blob import BlobServiceClient
    import time
    
    
    connection_string = "DefaultEndpointsProtocol=https;AccountName=venkat326;AccountKey=xxxx=;EndpointSuffix=core.windows.net"
    
    container_name = "result"
    blob_name = "test.csv"
    local_file_path = "path to file"
    
    # DNS timeout in seconds
    DNS_TIMEOUT = 5
    
    
    # Function to resolve DNS with timeout
    def resolve_dns_with_timeout(connection_string, timeout=DNS_TIMEOUT):
     
        blob_service_client = BlobServiceClient.from_connection_string(connection_string)
        account_url = blob_service_client.url
        hostname = account_url.split("//")[1].split("/")[0]
    
        resolver = dns.resolver.Resolver()
        resolver.lifetime = timeout
        resolver.timeout = timeout
        try:
            resolver.resolve(hostname, "A")
            print(f"DNS resolution for {hostname} succeeded.")
        except dns.resolver.Timeout:
            raise Exception(f"DNS resolution for {hostname} timed out after {timeout} seconds.")
        except dns.resolver.NXDOMAIN:
            raise Exception(f"The domain {hostname} does not exist.")
        except Exception as e:
            raise Exception(f"DNS resolution failed for {hostname}: {e}")
    
    
    # Function to upload blob with DNS resolution
    def upload_blob_with_dns_check(connection_string, container_name, blob_name, local_file_path):
        start_time = time.time()
        try:
            resolve_dns_with_timeout(connection_string)
            blob_service_client = BlobServiceClient.from_connection_string(connection_string)
            blob_client = blob_service_client.get_blob_client(container_name, blob_name)
    
            with open(local_file_path, "rb") as data:
                blob_client.upload_blob(data)
            print("Blob uploaded successfully!")
        except Exception as e:
            elapsed_time = time.time() - start_time
            print(f"An error occurred: {e}")
            print(f"Elapsed time before timeout or failure: {elapsed_time:.2f} seconds")
    
    
    # Run the upload process
    upload_blob_with_dns_check(connection_string, container_name, blob_name, local_file_path)
    

    If the DNS is wrong, you'll get output like:.

    An error occurred: The domain venkat326.blob.core.windows.net does not exist.
    Elapsed time before timeout or failure: 0.13 seconds
    

    enter image description here

    If the DNS is correct, you'll get output like:

    DNS resolution for venkat326123.blob.core.windows.net succeeded.
    Blob uploaded successfully!
    

    enter image description here