While using the Azure storage SDK in Python, I have been unable to override what appears to be a default 90-second timeout to catch a DNS exception occurring within a call to blob_client.upload_blob()
. I am looking for a way to override this with a shorter time interval (i.e. 5 seconds).
The following code illustrates this issue using a fictitious account name which DNS cannot resolve. I am using a timeout
argument in the call to upload_blob
, and I understand from reviewing documentation this enforces a server-side threshold, not a client-side threshold. I have not been successful in getting a client-side threshold to be enforced.
This issue appears similar to this unanswered question: How to handle timeout for Uploading a blob in Azure Storage using Python SDK?. The one (not accepted) solution suggests using a timeout threshold within the call to upload_blob
. As noted above (and shown within the code below), this is not producing the desired effect.
from azure.core.exceptions import AzureError
from azure.storage.blob import BlobServiceClient
# Define Azure Storage Blob connection details
connection_string = "DefaultEndpointsProtocol=https;AccountName=test;AccountKey=removed==;EndpointSuffix=core.windows.net"
container_name = "containername"
blob_name = "blobname"
local_file_path = "c:/temp/test.txt"
# Create the BlobServiceClient
blob_service_client = BlobServiceClient.from_connection_string(connection_string)
# Function to perform the blob upload
def upload_blob_process():
try:
with open(local_file_path, "rb") as data:
blob_client = blob_service_client.get_blob_client(container_name, blob_name)
blob_client.upload_blob(data, timeout=5)
print("Blob uploaded successfully!")
except AzureError as e:
print(f"Azure error occurred: {e}")
except Exception as e:
print(f"An error occurred: {e}")
upload_blob_process()
While using the Azure storage SDK in Python, I have been unable to override what appears to be a default 90-second timeout to catch a DNS exception occurring within a call to
blob_client.upload_blob()
. I am looking for a way to override this with a shorter time interval (i.e. 5 seconds).
As, I mentioned in comment, The timeout
parameter in upload_blob() applies to each individual HTTP request to the Azure Blob service and it does not affect DNS resolution or connection establishment timeouts but instead limits the server-side operation duration for each request.
thanks, I'm aware. Do you know of any way to apply a DNS timeout on top of this?
dns.resolver
library to perform DNS resolution before initiating the upload, with a custom timeout for DNS resolution. This will help you achieve the same result.Code:
import dns.resolver
from azure.storage.blob import BlobServiceClient
import time
connection_string = "DefaultEndpointsProtocol=https;AccountName=venkat326;AccountKey=xxxx=;EndpointSuffix=core.windows.net"
container_name = "result"
blob_name = "test.csv"
local_file_path = "path to file"
# DNS timeout in seconds
DNS_TIMEOUT = 5
# Function to resolve DNS with timeout
def resolve_dns_with_timeout(connection_string, timeout=DNS_TIMEOUT):
blob_service_client = BlobServiceClient.from_connection_string(connection_string)
account_url = blob_service_client.url
hostname = account_url.split("//")[1].split("/")[0]
resolver = dns.resolver.Resolver()
resolver.lifetime = timeout
resolver.timeout = timeout
try:
resolver.resolve(hostname, "A")
print(f"DNS resolution for {hostname} succeeded.")
except dns.resolver.Timeout:
raise Exception(f"DNS resolution for {hostname} timed out after {timeout} seconds.")
except dns.resolver.NXDOMAIN:
raise Exception(f"The domain {hostname} does not exist.")
except Exception as e:
raise Exception(f"DNS resolution failed for {hostname}: {e}")
# Function to upload blob with DNS resolution
def upload_blob_with_dns_check(connection_string, container_name, blob_name, local_file_path):
start_time = time.time()
try:
resolve_dns_with_timeout(connection_string)
blob_service_client = BlobServiceClient.from_connection_string(connection_string)
blob_client = blob_service_client.get_blob_client(container_name, blob_name)
with open(local_file_path, "rb") as data:
blob_client.upload_blob(data)
print("Blob uploaded successfully!")
except Exception as e:
elapsed_time = time.time() - start_time
print(f"An error occurred: {e}")
print(f"Elapsed time before timeout or failure: {elapsed_time:.2f} seconds")
# Run the upload process
upload_blob_with_dns_check(connection_string, container_name, blob_name, local_file_path)
If the DNS
is wrong, you'll get output like:.
An error occurred: The domain venkat326.blob.core.windows.net does not exist.
Elapsed time before timeout or failure: 0.13 seconds
If the DNS
is correct, you'll get output like:
DNS resolution for venkat326123.blob.core.windows.net succeeded.
Blob uploaded successfully!