cassandratombstone

How to avoid Cassandra read failure, due to large tombstone scans?

I use Apache Spark to read data from Cassandra, which under the hood does, token range based scan. But, recently I see a lot of failure during data read from my Cassandra table.

Read fails with below cause:

Caused by: com.datastax.driver.core.exceptions.ReadFailureException: Cassandra failure during read query at consistency LOCAL_ONE (1 responses were required but only 0 replica responded, 1 failed)    
at com.datastax.driver.core.exceptions.ReadFailureException.copy(ReadFailureException.java:85)  
at com.datastax.driver.core.exceptions.ReadFailureException.copy(ReadFailureException.java:27)  
at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)   
at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)  
at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:68)

When I checked the systemlogs I found out below issue:

ERROR [ReadStage-1] 2020-04-09 10:25:59,701 StorageProxy.java:1896 - Scanned over 100001 tombstones during query 'SELECT * FROM my_keyspace.my_table WHERE token(id) > -5868586221781016365 AND token(id) <= -5347313590141106513 LIMIT 5000' (last scanned row partion key was (my_key_1)); query aborted

The error log is straight forward, the reads are failing due to tombstone scans.

What I don't get is, I was running the same Spark job for over a year and never faced this problem. But, that's secondary. First I would like to know how can I fix this?

The tombstones could be created for the column values that are not passed, as I use Cassandra Insert Json feature to insert doc. Confirmed here, that this approach will create the tombstones.

How do I fix this? Creating multiple insert queries based on non-null values is a complicated option.

Even inserting dummy value is impractical in my opinion. All the clients that read data from these tables need to make the change.

I personally feel Cassandra should never create Tombstone for a column if the column doesn't exist in JSON.

What are the other options to solve this?

Solution

  • You have several possibilities:

    1. Don't insert data as JSON, but instead parse JSON in application, and insert data, using unset instead of missing values (done by default, look to docs) - you can simplify your work by mapping JSON into POJO, and using Object Mapper that also supports replacement of nulls with unset. This is best approach, as tombstones will affect all applications, and Cassandra itself. But real solution will depend on your use case - are you inserting only new data, or you're updating existing as well? In the later case, you'll need to make sure that you overwrite previous data
    2. Decrease the page size in Spark application to read less rows in one page. This is regulated by spark.cassandra.input.fetch.size_in_rows parameter, described in docs. If you're using DSE with DSE Analytics, you may need to disable continuous paging