pythonsdktimeoutcouchbase

couchbase.exceptions.UnAmbiguousTimeoutException using Python SDK


I’m getting couchbase.exceptions.UnAmbiguousTimeoutException every time I try to connect to my Couchbase cluster using the python SDK. With the same naming and configuration, I have no issue connecting via .NET sdk.

This is my Python code:

class CouchbaseBase:
    def __init__(self, endpoint, username, password, bucket_name):

        logging.basicConfig(filename='example.log',
                            filemode='w',
                            level=logging.DEBUG,
                            format='%(levelname)s::%(asctime)s::%(message)s',
                            datefmt='%Y-%m-%d %H:%M:%S')
        logger = logging.getLogger()
        couchbase.configure_logging(logger.name, level=logger.level)

        # Connect options - authentication
        auth = PasswordAuthenticator(username, password)
        timeout_opts = ClusterTimeoutOptions(connect_timeout=timedelta(seconds=60),
                                             kv_timeout=timedelta(seconds=60))

        # get a reference to our cluster
        options = ClusterOptions(auth,
                                 timeout_options=timeout_opts,
                                 disable_mozilla_ca_certificates=True,
                                 enable_dns_srv=False,
                                 tls_verify=None)
        cluster = Cluster.connect(f"couchbase://{endpoint}", options)

        # Wait until the cluster is ready for use.
        cluster.wait_until_ready(timedelta(seconds=5))

And this is the logs ( named address and ip are replaced for security reasons):

DEBUG::2024-06-24 09:58:57::{"openssl_default_cert_dir": "/etc/ssl/certs", "openssl_default_cert_file": "/etc/ssl/cert.pem", "openssl_headers": "OpenSSL 1.1.1 (compatible; BoringSSL)", "openssl_runtime": "BoringSSL", "txns_forward_compat_extensions": "TI,MO,BM,QU,SD,BF3787,BF3705,BF3838,RC,UA,CO,BF3791,CM,SI,QC,IX,TS,PU", "txns_forward_compat_protocol_version": "2.0", "version": "1.0.0"}
DEBUG::2024-06-24 09:58:57::Found DNS Servers: [10.70.0.2, 10.2.127.254, 8.8.8.8], selected nameserver: "10.70.0.2"
DEBUG::2024-06-24 09:58:57::open cluster, id: "a7e0e7-ee62-6f4b-ab4b-67f599fe7e4337", core version: "1.0.0+", {"bootstrap_nodes":[{"hostname":"couchbase-named-address.com","port":"11210"}],"options":{"analytics_timeout":"75000ms","bootstrap_timeout":"10000ms","config_idle_redial_timeout":"300000ms","config_poll_floor":"50ms","config_poll_interval":"2500ms","connect_timeout":"60000ms","disable_mozilla_ca_certificates":false,"dns_config":{"nameserver":"10.70.0.2","port":53,"timeout":"500ms"},"dump_configuration":false,"enable_clustermap_notification":true,"enable_compression":true,"enable_dns_srv":true,"enable_metrics":true,"enable_mutation_tokens":true,"enable_tcp_keep_alive":true,"enable_tls":false,"enable_tracing":true,"enable_unordered_execution":true,"idle_http_connection_timeout":"1000ms","key_value_durable_timeout":"10000ms","key_value_timeout":"60000ms","management_timeout":"75000ms","max_http_connections":0,"metrics_options":{"emit_interval":"600000ms"},"network":"auto","query_timeout":"75000ms","resolve_timeout":"2000ms","search_timeout":"75000ms","show_queries":false,"tcp_keep_alive_interval":"60000ms","tls_verify":"peer","tracing_options":{"analytics_threshold":"1000ms","key_value_threshold":"500ms","management_threshold":"1000ms","orphaned_emit_interval":"10000ms","orphaned_sample_size":64,"query_threshold":"1000ms","search_threshold":"1000ms","threshold_emit_interval":"10000ms","threshold_sample_size":64,"view_threshold":"1000ms"},"transactions_options":{"cleanup_config":{"cleanup_client_attempts":false,"cleanup_lost_attempts":false,"cleanup_window":"0ms","collections":[]},"durability_level":"none","query_config":{"scan_consistency":"not_bounded"},"timeout":"0ns"},"trust_certificate":"","use_ip_protocol":"any","user_agent_extra":"pycbc/4.2.1 (python/3.9.7)","view_timeout":"75000ms"}}
DEBUG::2024-06-24 09:58:57::Query DNS-SRV: address="couchbase-named-address.com", service="_couchbase", nameserver="10.70.0.2:53"
DEBUG::2024-06-24 09:58:57::DNS UDP returned 0 records
WARNING::2024-06-24 09:58:57::DNS SRV query returned 0 records for "couchbase-named-address.com", assuming that cluster is listening this address
DEBUG::2024-06-24 09:58:57::[a7e0e7-ee62-6f4b-ab4b-67f599fe7e4337/b49bd8-2d09-e440-6ac3-0b7691c4cef19e/plain/-] <couchbase-named-address.com:11210> attempt to establish MCBP connection
DEBUG::2024-06-24 09:58:57::[a7e0e7-ee62-6f4b-ab4b-67f599fe7e4337/b49bd8-2d09-e440-6ac3-0b7691c4cef19e/plain/-] <couchbase-named-address.com:11210> connecting to 127.0.0.1:11210 ("couchbase-named-address.com:11210"), timeout=60000ms
DEBUG::2024-06-24 09:58:59::[a7e0e7-ee62-6f4b-ab4b-67f599fe7e4337/b49bd8-2d09-e440-6ac3-0b7691c4cef19e/plain/-] <couchbase-named-address.com:11210> reached the end of list of bootstrap nodes, waiting for 500ms before restart
DEBUG::2024-06-24 09:59:00::[a7e0e7-ee62-6f4b-ab4b-67f599fe7e4337/b49bd8-2d09-e440-6ac3-0b7691c4cef19e/plain/-] <couchbase-named-address.com:11210> attempt to establish MCBP connection
DEBUG::2024-06-24 09:59:00::[a7e0e7-ee62-6f4b-ab4b-67f599fe7e4337/b49bd8-2d09-e440-6ac3-0b7691c4cef19e/plain/-] <couchbase-named-address.com:11210> connecting to 127.0.0.1:11210 ("couchbase-named-address.com:11210"), timeout=60000ms
DEBUG::2024-06-24 09:59:02::[a7e0e7-ee62-6f4b-ab4b-67f599fe7e4337/b49bd8-2d09-e440-6ac3-0b7691c4cef19e/plain/-] <couchbase-named-address.com:11210> reached the end of list of bootstrap nodes, waiting for 500ms before restart
DEBUG::2024-06-24 09:59:02::[a7e0e7-ee62-6f4b-ab4b-67f599fe7e4337/b49bd8-2d09-e440-6ac3-0b7691c4cef19e/plain/-] <couchbase-named-address.com:11210> attempt to establish MCBP connection
DEBUG::2024-06-24 09:59:02::[a7e0e7-ee62-6f4b-ab4b-67f599fe7e4337/b49bd8-2d09-e440-6ac3-0b7691c4cef19e/plain/-] <couchbase-named-address.com:11210> connecting to 127.0.0.1:11210 ("couchbase-named-address.com:11210"), timeout=60000ms
DEBUG::2024-06-24 09:59:04::[a7e0e7-ee62-6f4b-ab4b-67f599fe7e4337/b49bd8-2d09-e440-6ac3-0b7691c4cef19e/plain/-] <couchbase-named-address.com:11210> reached the end of list of bootstrap nodes, waiting for 500ms before restart
DEBUG::2024-06-24 09:59:05::[a7e0e7-ee62-6f4b-ab4b-67f599fe7e4337/b49bd8-2d09-e440-6ac3-0b7691c4cef19e/plain/-] <couchbase-named-address.com:11210> attempt to establish MCBP connection
DEBUG::2024-06-24 09:59:05::[a7e0e7-ee62-6f4b-ab4b-67f599fe7e4337/b49bd8-2d09-e440-6ac3-0b7691c4cef19e/plain/-] <couchbase-named-address.com:11210> connecting to 127.0.0.1:11210 ("couchbase-named-address.com:11210"), timeout=60000ms
DEBUG::2024-06-24 09:59:07::[a7e0e7-ee62-6f4b-ab4b-67f599fe7e4337/b49bd8-2d09-e440-6ac3-0b7691c4cef19e/plain/-] <couchbase-named-address.com:11210> reached the end of list of bootstrap nodes, waiting for 500ms before restart
DEBUG::2024-06-24 09:59:07::all nodes failed to bootstrap, triggering DNS-SRV refresh, ec=unambiguous_timeout (14), last endpoint="couchbase-named-address.com:11210"
WARNING::2024-06-24 09:59:07::[a7e0e7-ee62-6f4b-ab4b-67f599fe7e4337/b49bd8-2d09-e440-6ac3-0b7691c4cef19e/plain/-] <couchbase-named-address.com:11210> unable to bootstrap in time
DEBUG::2024-06-24 09:59:07::[a7e0e7-ee62-6f4b-ab4b-67f599fe7e4337/b49bd8-2d09-e440-6ac3-0b7691c4cef19e/plain/-] <couchbase-named-address.com:11210> stop MCBP connection, reason=do_not_retry
DEBUG::2024-06-24 09:59:07::Query DNS-SRV: address="couchbase-named-address.com", service="_couchbase", nameserver="10.70.0.2:53"
DEBUG::2024-06-24 09:59:07::PYCBC: create conn callback completed
DEBUG::2024-06-24 09:59:07::[a7e0e7-ee62-6f4b-ab4b-67f599fe7e4337/b49bd8-2d09-e440-6ac3-0b7691c4cef19e/plain/-] <couchbase-named-address.com:11210> destroy MCBP connection

The only difference I’m seeing in my.NET configuration, is there are some options to HttpIgnoreRemoteCertificateMismatch and KvIgnoreRemoteCertificateNameMismatch which I set to true in my case, but I cannot find any for python sdk.

.NET equivalent code that is working without any issue:

_clusterOptions = new ClusterOptions();
config.GetCouchbaseClientDefinition().Bind(_clusterOptions);

_clusterOptions.HttpCertificateCallbackValidation += new RemoteCertificateValidationCallback((object sender, X509Certificate certificate, X509Chain chain, SslPolicyErrors errors) =>
{
    return true;
});
_clusterOptions.KvCertificateCallbackValidation += new RemoteCertificateValidationCallback((object sender, X509Certificate certificate, X509Chain chain, SslPolicyErrors errors) =>
{
    return true;
});

_clusterOptions.NetworkResolution = "external";
_clusterOptions.EnableDnsSrvResolution = false;

_clusterOptions.WithSerializer(new CouchbaseSerializer());
_clusterOptions.AddLinq();
_clusterOptions.KvTimeout = TimeSpan.FromSeconds(10);

_cluster = Cluster.ConnectAsync(_clusterOptions).Result;

Solution

  • After few rounds of back and forth conversation with Couchbase Support Team, I was finally able to connect via Python SDK by adding the Options:

            options = ClusterOptions(auth,
                                     timeout_options=timeout_opts,
                                     disable_mozilla_ca_certificates=True,
                                     enable_tls=enable_tls,
                                     enable_dns_srv=False,
                                     # This option was not documented in the python SDK at this time
                                     network='external',
                                     tls_verify=TLSVerifyMode.NO_VERIFY)
    

    Also, my external Couchbase cluster uses Encrypted connection. In .NET it's probably handled in handshaking, but in Python SDK we need to put and extra s (couchbase ----> couchbases) so that the connection is switched over from Port=11210 to encrypted Port=11207:

    cluster = Cluster.connect(f"couchbases://{endpoint}", options)
    

    And this resolved my issue.