Spring Cassandra time out query 'SELECT * FROM system_schema.tables' timed out after PT2S

I am using Spring Boot 2.4.4 and Spring Data Cassandra dependency to connect to the Cassandra database. During the application startup, I am getting a DriverTimeout error (I am using VPN).

I have gone through all the Stack Overflow questions similar to this and none of them worked for me. I have cross-posted the same question on the Spring Boot official page here.

I used below configuration properties below -

spring.data.cassandra.contact-points=xxxxxx
spring.data.cassandra.username=xxxx
spring.data.cassandra.password=xxxxx
spring.data.cassandra.keyspace-name=xxxx
spring.data.cassandra.port=9042
spring.data.cassandra.schema-action=NONE
spring.data.cassandra.local-datacenter=mydc
spring.data.cassandra.connection.connect-timeout=PT10S
spring.data.cassandra.connection.init-query-timeout=PT20S
spring.data.cassandra.request.timeout=PT10S

I also added DataStax properties in the application.properties to check if they can be picked up from there or not.

datastax-java-driver.basic.request.timeout = 10 seconds
datastax-java-driver.advanced.connection.init-query-timeout = 10 seconds
datastax-java-driver.advanced.control-connection.timeout = 10 seconds

Below is the configuration I used as suggested in the post here -

@EnableCassandraRepositories
public class CassandraConfig {
    @Bean
        DriverConfigLoaderBuilderCustomizer cassandraDriverCustomizer() {
            return (builder) -> builder.withDuration(DefaultDriverOption.CONTROL_CONNECTION_TIMEOUT,
                    Duration.ofSeconds(30));
        }
}

But I still get the same error

Caused by: com.datastax.oss.driver.api.core.DriverTimeoutException: query 'SELECT * FROM system_schema.tables' timed out after PT2S

I also tried different approached like creating custom CqlSessionFactoryBean and provide all the DataStax properties programmatically to override -

@EnableCassandraRepositories
public class CassandraConfig extends AbstractCassandraConfiguration { 

    @Bean(name = "session")
        @Primary
        public CqlSessionFactoryBean cassandraSession() {
            CqlSessionFactoryBean factory = new CqlSessionFactoryBean();
            factory.setUsername(userName);
            factory.setPassword(password);
            factory.setPort(port);
            factory.setKeyspaceName(keyspaceName);
            factory.setContactPoints(contactPoints);
            factory.setLocalDatacenter(dataCenter);
            factory.setSessionBuilderConfigurer(getSessionBuilderConfigurer()); // my session builder configurer 
            return factory;
        }
    // And provided my own SessionBuilder Configurer like below
    
    protected SessionBuilderConfigurer getSessionBuilderConfigurer() {
        return new SessionBuilderConfigurer() {
    
            @Override
            public CqlSessionBuilder configure(CqlSessionBuilder cqlSessionBuilder) {
                ProgrammaticDriverConfigLoaderBuilder config = DriverConfigLoader.programmaticBuilder()
                        .withDuration(DefaultDriverOption.CONNECTION_INIT_QUERY_TIMEOUT, Duration.ofSeconds(30))
                        .withBoolean(DefaultDriverOption.RECONNECT_ON_INIT, true)
                        .withDuration(DefaultDriverOption.REQUEST_TIMEOUT, Duration.ofSeconds(30))
                        .withDuration(DefaultDriverOption.CONTROL_CONNECTION_TIMEOUT, Duration.ofSeconds(20));
                return cqlSessionBuilder.withAuthCredentials(userName, password).withConfigLoader(config.build());
            }
        };
      }
}

It didn't work same error. Also, I excluded the Cassandra auto-configuration classes like suggested here on StackOverflow

I also tried to customize custom session builder like below to see if that can work -

@Bean
public CqlSessionBuilderCustomizer cqlSessionBuilderCustomizer() {
        return cqlSessionBuilder -> cqlSessionBuilder.withAuthCredentials(userName, password)
                .withConfigLoader(DriverConfigLoader.programmaticBuilder()
                        .withDuration(DefaultDriverOption.REQUEST_TIMEOUT, Duration.ofMillis(15000))
                        .withDuration(DefaultDriverOption.CONNECTION_INIT_QUERY_TIMEOUT, Duration.ofSeconds(30))
                        .withBoolean(DefaultDriverOption.RECONNECT_ON_INIT, true)
                        .withDuration(DefaultDriverOption.REQUEST_TIMEOUT, Duration.ofSeconds(30))
                        .withDuration(DefaultDriverOption.CONTROL_CONNECTION_TIMEOUT, Duration.ofSeconds(20)).build());
    }

Still no luck.

Not only that I also added the application.conf file as DataStax documentation suggested putting that on the classpath, even though that file is getting parsed (after making the syntactical mistake I got to know that it is being read). It didn't work.

application.conf-

datastax-java-driver {
 basic.request.timeout = 10 seconds
 advanced.connection.init-query-timeout = 10 seconds
 advanced.control-connection.timeout = 10 seconds
}

I also switched my Spring Boot version to 2.5.0.M3 to see property files works it does not. I have pushed my project to my GitHub account.

Update

As per the comment, I am pasting my whole stack trace. Also, this does not happen all the time sometimes it works sometimes it does not. I need to override the timeout from PT2S to PT10S or something.

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cassandraConverter' defined in class path resource [com/example/demo/CassandraConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.data.cassandra.core.convert.CassandraConverter]: Factory method 'cassandraConverter' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cassandraSession' defined in class path resource [com/example/demo/CassandraConfig.class]: Invocation of init method failed; nested exception is com.datastax.oss.driver.api.core.DriverTimeoutException: query 'SELECT * FROM system_schema.tables' timed out after PT2S
    at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:656) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:484) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1338) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1177) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:557) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:895) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:878) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:758) [spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:750) [spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397) [spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) [spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237) [spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) [spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
    at com.example.demo.SpringCassandraTestingApplication.main(SpringCassandraTestingApplication.java:13) [classes/:na]
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.data.cassandra.core.convert.CassandraConverter]: Factory method 'cassandraConverter' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cassandraSession' defined in class path resource [com/example/demo/CassandraConfig.class]: Invocation of init method failed; nested exception is com.datastax.oss.driver.api.core.DriverTimeoutException: query 'SELECT * FROM system_schema.tables' timed out after PT2S
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:651) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    ... 19 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cassandraSession' defined in class path resource [com/example/demo/CassandraConfig.class]: Invocation of init method failed; nested exception is com.datastax.oss.driver.api.core.DriverTimeoutException: query 'SELECT * FROM system_schema.tables' timed out after PT2S
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1796) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:595) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:227) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1174) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveBean(DefaultListableBeanFactory.java:422) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:352) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:345) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.data.cassandra.config.AbstractSessionConfiguration.requireBeanOfType(AbstractSessionConfiguration.java:100) ~[spring-data-cassandra-3.0.0.RELEASE.jar:3.0.0.RELEASE]
    at org.springframework.data.cassandra.config.AbstractSessionConfiguration.getRequiredSession(AbstractSessionConfiguration.java:200) ~[spring-data-cassandra-3.0.0.RELEASE.jar:3.0.0.RELEASE]
    at org.springframework.data.cassandra.config.AbstractCassandraConfiguration.cassandraConverter(AbstractCassandraConfiguration.java:73) ~[spring-data-cassandra-3.0.0.RELEASE.jar:3.0.0.RELEASE]
    at com.example.demo.CassandraConfig$$EnhancerBySpringCGLIB$$cec229ff.CGLIB$cassandraConverter$12(<generated>) ~[classes/:na]
    at com.example.demo.CassandraConfig$$EnhancerBySpringCGLIB$$cec229ff$$FastClassBySpringCGLIB$$faa9c2c1.invoke(<generated>) ~[classes/:na]
    at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244) ~[spring-core-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at com.example.demo.CassandraConfig$$EnhancerBySpringCGLIB$$cec229ff.cassandraConverter(<generated>) ~[classes/:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_275]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_275]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_275]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_275]
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    ... 20 common frames omitted
Caused by: com.datastax.oss.driver.api.core.DriverTimeoutException: query 'SELECT * FROM system_schema.tables' timed out after PT2S
    at com.datastax.oss.driver.api.core.DriverTimeoutException.copy(DriverTimeoutException.java:34) ~[java-driver-core-4.6.1.jar:na]
    at com.datastax.oss.driver.internal.core.util.concurrent.CompletableFutures.getUninterruptibly(CompletableFutures.java:149) ~[java-driver-core-4.6.1.jar:na]
    at com.datastax.oss.driver.api.core.session.Session.refreshSchema(Session.java:140) ~[java-driver-core-4.6.1.jar:na]
    at org.springframework.data.cassandra.config.CqlSessionFactoryBean.afterPropertiesSet(CqlSessionFactoryBean.java:437) ~[spring-data-cassandra-3.0.0.RELEASE.jar:3.0.0.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1855) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1792) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    ... 43 common frames omitted

I am answering my own question here to make this complete and let others know how I fixed this particular problem.

I am using Spring Boot 2.4.5. and I started facing this timeout issue when I upgraded to version 2.3+ onwards.

Based on my experience with this issue, below is what I found.

Irrespective of whatever timeout you provide in the application.properties or application.conf (DataStax notion), They all somehow getting overridden by the spring boot or maybe selecting the default value from the DataStax driver.

Even there is an issue created on the Spring Boot official project to handle this problem. Check here. Which got fixed later in the 2.5.0.M1 version.

My problem got fixed when I passed this as a VM argument.

 $java -Ddatastax-java-driver.basic.request.timeout="15 seconds" application.jar

I passed other params as well like advanced.control-connection.timeout as I was suggested to use on a different forum but that didn't work for me. Check reference manual here for other config params.

I am getting this error at my local only so I passed this in the Eclipse VM argument and then I didn't see that error any more.

Also if I reduce this time to 7-8 seconds sometimes I see that error again PT2S. Seems like that exception message is hardcoded somewhere irrespective of whatever timeout value you pass. (That is my observation).

Update: Solution 2 - Which I figured out later and I see many of the people have answered that too
The actual key that DataStax provides is given below and this works.

 @Bean
    public DriverConfigLoaderBuilderCustomizer defaultProfile(){
        return builder -> builder.withString(DefaultDriverOption.METADATA_SCHEMA_REQUEST_TIMEOUT, "3 seconds").build();
    }