We encountered a strange issue when altering a table. We use Cassandra python driver sync_table() method to sync from our model (defined in a py file) to Cassandra. The cluster is a 20 node being stressed decently (all nodes in range of 50-70% max usage).
When the schema is synced using the Cassandra python driver, internally it is executing the "ALTER TABLE ADD " commands. In a particular table, when we added seven new columns, we noticed this strange behavior
DESCRIBE TABLE command shows 3 or 4 out of the new 7 columns created. Once, it showed all 7 columns in the DESCRIBE TABLE output.
The behavior is inconsistent. We dropped the columns manually and then resynced the schema. Every time the issue appears with select command not showing few of the 7 columns.
Any pointers to debug this issue? Is it due to stress on Cassandra nodes?
The most probable issue is that you hit the schema agreement problem because of execution of many schema change commands.
Typically, you need to send the schema change commands only to one host, and get confirmation about schema agreement. The first thing is usually done by creating a session that uses white-list policy where list consists only of one node (as opposite to token-aware or round robin policies). The second thing is easy - you either check corresponding flag of the result set returned after execution of the command, or by checking corresponding field/method of the cluster's metadata.