Spring Cassandra time out query 'SELECT * FROM system_schema.tables' timed out after PT2S
I am using Spring Boot 2.4.4 and Spring Data Cassandra dependency to connect to the Cassandra database. During the application startup, I am getting a DriverTimeout error (I am using VPN).
I have gone through all the Stack Overflow questions similar to this and none of them worked for me. I have cross-posted the same question on the Spring Boot official page here.
I used below configuration properties below -
spring.data.cassandra.contact-points=xxxxxx
spring.data.cassandra.username=xxxx
spring.data.cassandra.password=xxxxx
spring.data.cassandra.keyspace-name=xxxx
spring.data.cassandra.port=9042
spring.data.cassandra.schema-action=NONE
spring.data.cassandra.local-datacenter=mydc
spring.data.cassandra.connection.connect-timeout=PT10S
spring.data.cassandra.connection.init-query-timeout=PT20S
spring.data.cassandra.request.timeout=PT10S
I also added DataStax properties in the application.properties to check if they can be picked up from there or not.
datastax-java-driver.basic.request.timeout = 10 seconds
datastax-java-driver.advanced.connection.init-query-timeout = 10 seconds
datastax-java-driver.advanced.control-connection.timeout = 10 seconds
Below is the configuration I used as suggested in the post here -
@EnableCassandraRepositories
public class CassandraConfig {
@Bean
DriverConfigLoaderBuilderCustomizer cassandraDriverCustomizer() {
return (builder) -> builder.withDuration(DefaultDriverOption.CONTROL_CONNECTION_TIMEOUT,
Duration.ofSeconds(30));
}
}
But I still get the same error
Caused by: com.datastax.oss.driver.api.core.DriverTimeoutException: query 'SELECT * FROM system_schema.tables' timed out after PT2S
I also tried different approached like creating custom CqlSessionFactoryBean and provide all the DataStax properties programmatically to override -
@EnableCassandraRepositories
public class CassandraConfig extends AbstractCassandraConfiguration {
@Bean(name = "session")
@Primary
public CqlSessionFactoryBean cassandraSession() {
CqlSessionFactoryBean factory = new CqlSessionFactoryBean();
factory.setUsername(userName);
factory.setPassword(password);
factory.setPort(port);
factory.setKeyspaceName(keyspaceName);
factory.setContactPoints(contactPoints);
factory.setLocalDatacenter(dataCenter);
factory.setSessionBuilderConfigurer(getSessionBuilderConfigurer()); // my session builder configurer
return factory;
}
// And provided my own SessionBuilder Configurer like below
protected SessionBuilderConfigurer getSessionBuilderConfigurer() {
return new SessionBuilderConfigurer() {
@Override
public CqlSessionBuilder configure(CqlSessionBuilder cqlSessionBuilder) {
ProgrammaticDriverConfigLoaderBuilder config = DriverConfigLoader.programmaticBuilder()
.withDuration(DefaultDriverOption.CONNECTION_INIT_QUERY_TIMEOUT, Duration.ofSeconds(30))
.withBoolean(DefaultDriverOption.RECONNECT_ON_INIT, true)
.withDuration(DefaultDriverOption.REQUEST_TIMEOUT, Duration.ofSeconds(30))
.withDuration(DefaultDriverOption.CONTROL_CONNECTION_TIMEOUT, Duration.ofSeconds(20));
return cqlSessionBuilder.withAuthCredentials(userName, password).withConfigLoader(config.build());
}
};
}
}
It didn't work same error. Also, I excluded the Cassandra auto-configuration classes like suggested here on StackOverflow
I also tried to customize custom session builder like below to see if that can work -
@Bean
public CqlSessionBuilderCustomizer cqlSessionBuilderCustomizer() {
return cqlSessionBuilder -> cqlSessionBuilder.withAuthCredentials(userName, password)
.withConfigLoader(DriverConfigLoader.programmaticBuilder()
.withDuration(DefaultDriverOption.REQUEST_TIMEOUT, Duration.ofMillis(15000))
.withDuration(DefaultDriverOption.CONNECTION_INIT_QUERY_TIMEOUT, Duration.ofSeconds(30))
.withBoolean(DefaultDriverOption.RECONNECT_ON_INIT, true)
.withDuration(DefaultDriverOption.REQUEST_TIMEOUT, Duration.ofSeconds(30))
.withDuration(DefaultDriverOption.CONTROL_CONNECTION_TIMEOUT, Duration.ofSeconds(20)).build());
}
Still no luck.
Not only that I also added the application.conf file as DataStax documentation suggested putting that on the classpath, even though that file is getting parsed (after making the syntactical mistake I got to know that it is being read). It didn't work.
application.conf-
datastax-java-driver {
basic.request.timeout = 10 seconds
advanced.connection.init-query-timeout = 10 seconds
advanced.control-connection.timeout = 10 seconds
}
I also switched my Spring Boot version to 2.5.0.M3 to see property files works it does not. I have pushed my project to my GitHub account.
Update
As per the comment, I am pasting my whole stack trace. Also, this does not happen all the time sometimes it works sometimes it does not. I need to override the timeout from PT2S to PT10S or something.
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cassandraConverter' defined in class path resource [com/example/demo/CassandraConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.data.cassandra.core.convert.CassandraConverter]: Factory method 'cassandraConverter' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cassandraSession' defined in class path resource [com/example/demo/CassandraConfig.class]: Invocation of init method failed; nested exception is com.datastax.oss.driver.api.core.DriverTimeoutException: query 'SELECT * FROM system_schema.tables' timed out after PT2S
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:656) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:484) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1338) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1177) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:557) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:895) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:878) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:758) [spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:750) [spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397) [spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) [spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237) [spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) [spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
at com.example.demo.SpringCassandraTestingApplication.main(SpringCassandraTestingApplication.java:13) [classes/:na]
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.data.cassandra.core.convert.CassandraConverter]: Factory method 'cassandraConverter' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cassandraSession' defined in class path resource [com/example/demo/CassandraConfig.class]: Invocation of init method failed; nested exception is com.datastax.oss.driver.api.core.DriverTimeoutException: query 'SELECT * FROM system_schema.tables' timed out after PT2S
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:651) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
... 19 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cassandraSession' defined in class path resource [com/example/demo/CassandraConfig.class]: Invocation of init method failed; nested exception is com.datastax.oss.driver.api.core.DriverTimeoutException: query 'SELECT * FROM system_schema.tables' timed out after PT2S
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1796) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:595) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:227) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1174) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveBean(DefaultListableBeanFactory.java:422) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:352) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:345) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.data.cassandra.config.AbstractSessionConfiguration.requireBeanOfType(AbstractSessionConfiguration.java:100) ~[spring-data-cassandra-3.0.0.RELEASE.jar:3.0.0.RELEASE]
at org.springframework.data.cassandra.config.AbstractSessionConfiguration.getRequiredSession(AbstractSessionConfiguration.java:200) ~[spring-data-cassandra-3.0.0.RELEASE.jar:3.0.0.RELEASE]
at org.springframework.data.cassandra.config.AbstractCassandraConfiguration.cassandraConverter(AbstractCassandraConfiguration.java:73) ~[spring-data-cassandra-3.0.0.RELEASE.jar:3.0.0.RELEASE]
at com.example.demo.CassandraConfig$$EnhancerBySpringCGLIB$$cec229ff.CGLIB$cassandraConverter$12(<generated>) ~[classes/:na]
at com.example.demo.CassandraConfig$$EnhancerBySpringCGLIB$$cec229ff$$FastClassBySpringCGLIB$$faa9c2c1.invoke(<generated>) ~[classes/:na]
at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244) ~[spring-core-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at com.example.demo.CassandraConfig$$EnhancerBySpringCGLIB$$cec229ff.cassandraConverter(<generated>) ~[classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_275]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_275]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_275]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_275]
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
... 20 common frames omitted
Caused by: com.datastax.oss.driver.api.core.DriverTimeoutException: query 'SELECT * FROM system_schema.tables' timed out after PT2S
at com.datastax.oss.driver.api.core.DriverTimeoutException.copy(DriverTimeoutException.java:34) ~[java-driver-core-4.6.1.jar:na]
at com.datastax.oss.driver.internal.core.util.concurrent.CompletableFutures.getUninterruptibly(CompletableFutures.java:149) ~[java-driver-core-4.6.1.jar:na]
at com.datastax.oss.driver.api.core.session.Session.refreshSchema(Session.java:140) ~[java-driver-core-4.6.1.jar:na]
at org.springframework.data.cassandra.config.CqlSessionFactoryBean.afterPropertiesSet(CqlSessionFactoryBean.java:437) ~[spring-data-cassandra-3.0.0.RELEASE.jar:3.0.0.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1855) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1792) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
... 43 common frames omitted
Solution
I am answering my own question here to make this complete and let others know how I fixed this particular problem.
I am using Spring Boot 2.4.5. and I started facing this timeout issue when I upgraded to version 2.3+ onwards.
Based on my experience with this issue, below is what I found.
Irrespective of whatever timeout you provide in the application.properties or application.conf (DataStax notion), They all somehow getting overridden by the spring boot or maybe selecting the default value from the DataStax driver.
Even there is an issue created on the Spring Boot official project to handle this problem. Check here. Which got fixed later in the 2.5.0.M1 version.
My problem got fixed when I passed this as a VM argument.
$java -Ddatastax-java-driver.basic.request.timeout="15 seconds" application.jar
I passed other params as well like advanced.control-connection.timeout as I was suggested to use on a different forum but that didn't work for me. Check reference manual here for other config params.
I am getting this error at my local only so I passed this in the Eclipse VM argument and then I didn't see that error any more.
Also if I reduce this time to 7-8 seconds sometimes I see that error again PT2S. Seems like that exception message is hardcoded somewhere irrespective of whatever timeout value you pass. (That is my observation).
Update: Solution 2 - Which I figured out later and I see many of the people have answered that too
The actual key that DataStax provides is given below and this works.
@Bean
public DriverConfigLoaderBuilderCustomizer defaultProfile(){
return builder -> builder.withString(DefaultDriverOption.METADATA_SCHEMA_REQUEST_TIMEOUT, "3 seconds").build();
}
- How do I use org.scribe.oauth to connect to the tumblr API?
- java stream to set the values to a list from another list
- Delete directories recursively in Java
- How can I call scikit-learn classifiers from Java?
- Java method declared to throw IOException not propagating the exception up the call stack
- How to get integer and fraction portions of a BigDecimal in Java
- How to auto add imports in IntelliJ?
- remove double quotes from csv while using open csv
- Java custom annotation for a behaviour like trim
- Cors Error when calling microservice from Vue frontend application
- Why isn't my class unloaded when I erase any link to them and to ClassLoader?
- Java UTF-8 filenames with IBM JVM (AIX)
- What would be a good use-case scenario for the Spliterator in Java 8?
- Caused by: java.lang.NoClassDefFoundError: io/jsonwebtoken/Jwts
- Are supertypes of a type just the ones that are directly extended or implemented?
- ModelAttribute returns null values to controller in Spring MVC
- Jaybird 3 and Firebird transaction information
- IntelliJ - Invalid source release: 17
- How to use JScrollPane with JTextArea in Swing?
- Java FFM - Obtaining a var handle to an array field of a structure
- Is there auto type inferring in Java?
- Springboot Keycloak Admin add role or reset password error
- Setting up IntelliJ IDEA with Java 1.8/1.7 and resolve "java: System Java Compiler was not found in classpath"
- How to add a simple horizontal line at Y value of JFreeChart TimeSeries
- java.sql.SQLException: Column Index out of range, 2 > 1
- Task app:compileDebugJavaWithJavac failed
- Column name as a parameter to Spring Data JPA Query
- Pervasive SQL Java database name
- XML to be validated against multiple xsd schemas
- How many sessions can be managed by an Java Application in Struts 2?