[SOLVED] Astyanax not respecting configured socket timeout

Astyanax not respecting configured socket timeout

This is my setup:

4 nodes Cassandra 1.2.19
Astyanax 1.56.49

I am setting configuration like

AstyanaxContext<Keyspace> context = new AstyanaxContext.Builder()
    .forCluster(service.getClusterName())
    .forKeyspace(service.getKeySpaceName())
    .withAstyanaxConfiguration(new AstyanaxConfigurationImpl()
         .setDiscoveryType(NodeDiscoveryType.NONE)
         .setCqlVersion("3.0.0")
         .setDefaultReadConsistencyLevel(consistencyLevel.getAstyanaxValue())
         .setDefaultWriteConsistencyLevel(consistencyLevel.getAstyanaxValue())
    )
    .withConnectionPoolConfiguration(new ConnectionPoolConfigurationImpl("b2bConnectionPool")
         .setPort(service.getPort())
         .setMaxConnsPerHost(5)
         .setSeeds(StringUtils.join(hosts, ","))
         // increase default timeout for heavy operations (milliseconds)
         .setSocketTimeout(15000)
         .setSSLConnectionContext(sslContext)
         .setAuthenticationCredentials(credentials)
    )
    .withConnectionPoolMonitor(new CountingConnectionPoolMonitor())
    .buildKeyspace(ThriftFamilyFactory.getInstance());

Now there is a reproducible query that takes a long time and finally throws a OperationTimeoutException:

com.netflix.astyanax.connectionpool.exceptions.OperationTimeoutException: OperationTimeoutException: [host=myhost(myip):13260, latency=10001(40007), attempts=4]TimedOutException()
    at com.netflix.astyanax.thrift.ThriftConverter.ToConnectionPoolException(ThriftConverter.java:171) ~[astyanax-thrift-1.56.49.jar:na]
    at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:65) ~[astyanax-thrift-1.56.49.jar:na]
    at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1$2.execute(ThriftColumnFamilyQueryImpl.java:190) ~[astyanax-thrift-1.56.49.jar:na]
    at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1$2.execute(ThriftColumnFamilyQueryImpl.java:182) ~[astyanax-thrift-1.56.49.jar:na]
    at com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection.execute(ThriftSyncConnectionFactoryImpl.java:151) ~[astyanax-thrift-1.56.49.jar:na]
    at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:119) ~[astyanax-core-1.56.49.jar:na]
    at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:338) ~[astyanax-core-1.56.49.jar:na]
    at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1.execute(ThriftColumnFamilyQueryImpl.java:180) ~[astyanax-thrift-1.56.49.jar:na]

The exception message says "latency=10001" and I thought this should be the socket timeout configured to 15000 ms but it's obviously not. How can I increase the timeout for a query operation in astyanax?

Solution

I was getting the similar timeout exception when trying to query nodes with bigger data. I modified these 4 values in cassandra.yaml and it resolved all the timeout errors.

How long the coordinator should wait for read operations to complete read_request_timeout_in_ms: 15000

How long the coordinator should wait for seq or index scans to complete range_request_timeout_in_ms: 30000

How long the coordinator should wait for writes to complete write_request_timeout_in_ms: 30000

How long the coordinator should wait for counter writes to complete counter_write_request_timeout_in_ms: 15000

Note: You need to do this in all the nodes in the cluster and also need to restart Cassandra in all of them.