In Cassandra Lunch #96, Rahul Singh, CEO of Anant, will discuss different ways to get change data into and out of Cassandra using a few different strategies which could work out for your platform. Accompanying Blog: Coming Soon! Accompanying YouTube: https://youtu.be/HjgqAhsCQ64 Sign Up For Our Newsletter: http://eepurl.com/grdMkn Join Cassandra Lunch Weekly at 12 PM EST Every Wednesday: https://www.meetup.com/Cassandra-DataStax-DC/events/ Cassandra.Link: https://cassandra.link/ Follow Us and Reach Us At: Anant: https://www.anant.us/ Awesome Cassandra: https://github.com/Anant/awesome-cassandra Cassandra.Lunch: https://github.com/Anant/Cassandra.Lunch Email: solutions@anant.us LinkedIn: https://www.linkedin.com/company/anant/ Twitter: https://twitter.com/anantcorp Eventbrite: https://www.eventbrite.com/o/anant-1072927283 Facebook: https://www.facebook.com/AnantCorp/ Join The Anant Team: https://www.careers.anant.us

1. Change Data Capture
Apache Cassandra
Strategies and tactics for getting data in and out
of Cassandra as it’s happening in realtime to
synchronize databases with Triggers, Kafka,
Pulsar, Debezium, SymmetricDS

2. There are only two hard
things in Computer
Science: Cache invalidation
and naming things.
Phil Karlton, Martin Fowler, etc. etc.. etc..

3. Batch In
● COPY
● DSBulk
● Spark
● ..
Batch Out CDC Out
● COPY
● DSBulk
● Spark
● ..
● Triggers
● CDC Log
● Debezium
● Kafka Connect
● Pulsar Io
Change Data Capture
CDC In
● SymmetricDS
● Debezium
● Kafka Connect
● Lenses
● Pulsar IO
● AWS DMS

4. Change Data Capture: Kafka Connect

5. Change Data Capture: Pulsar IO

6. Change Data Capture: SymmetricDS

7. Sources
● Kafka
○ Kafka Connect
■ Debezium
■ Open
● Pulsar
○ Pulsar IO
● SymmetricDS
○ Many
● AWS DMS
○ Many
Channel Sinks
● Kafka
○ Kafka Connect
■ Cassandra Sink for Kafka
○ Kafka Streams
● Pulsar
○ Pulsar IO
■ Cassandra Sink for Pulsar
○ Pulsar Functions
● Akka/Alpakka
● RYO
○ Serverless Functions
○ Traditional Code
○ SparkStreaming
○ Flink
● Kafka
● Pulsar
● RabbitMQ
● AWS SNS
● AWS SQS
● AWS Kinesis
Change Data Capture IN TO CASSANDRA

8. Preparing Change Data for Ingestion
Data Preparation
- Is the data coming in key value updates?
- Are the keys well designed?
- Do they need to replaced with surrogate keys?
- Is the data coming in have nulls?
- Do we want to append null/tombstones?
- Do we need empty strings instead?
- Do different tables have any referential integrity?
- Is it okay for some tables to sync at a later time?
- Do we need to transform the data?
- In flight Transformation?
- After flight Transformation?

9. Getting Data In to Cassandra
Which method makes sense?
- Simple sync?
- Kafka Connect
- Pulsar IO
- Medium complexity transformation?
- Kafka Connect SMT
- Kafka Streams + Kafka Connect
- Pulsar Streams + Pulsar IO
- High complexity transformation?
- Kafka Streams
- Spark Streaming
- Flink
- Alpakka/Akka

10. Sources
● Kafka
○ Connect
■ Debezium
● Pulsar
○ Pulsar IO
● RYO
○ Trigger
○ Custom CDC
Reader
Channel
● Kafka
○ Kafka Connect
■ Cassandra Sink for Kafka
○ Kafka Streams
● Pulsar
○ Pulsar IO
■ Cassandra Sink for Pulsar
○ Pulsar Functions
● Akka/Alpakka
○ Cassandra
● RYO
○ Serverless Functions
○ Traditional Code
○ SparkStreaming
○ Flink
Sinks
● Kafka
● Pulsar
● RabbitMQ
● AWS SNS
● AWS SQS
● AWS Kinesis
Change Data Capture OUT OF CASSANDRA

11. Getting Data Out of Cassandra : CDC

12. Getting Data Out of Cassandra : Trigger

13. Considerations for Change Data for Publishing
Considerations for CDC
- CDC can be turned on any table as long as
CDC is enabled.
- Need to manage the CDC raw logs otherwise
will fill up and CDC will reject writes.
- CDC log has lots of info so data has to be
filtered.
- Commit logs are replicated so each Commit (to
each replica) will produce a Message
- Better to use a tool rather than make your
own.
Considerations for Trigger
- Triggers can be configured per table.
- Trigger is sent before each inset, update or
delete, and passes the data to the JVM class.
- Trigger is called one for each row and it will
be ordered.
- It may add a little overhead for the insert
based on what action is being performed in
the class.
- As the trigger is called only, the application
has to handle handle the error scenarios.

14. Key Takeaways
- Netflix, Yelp, etc have had this problem.
- They tried a bunch of different things.
- Then Confluent/Debezium/ etc standardized it.
- Kafka Connect / Pulsar IO Ecosystems are rich.
- There are managed services available for Kafka/Pulsar
- Not many for Kafka Connect / Pulsar IO but can be found.
- Most things are available for free, but free is never free.
- Only pay for what you need, not what you don’t.
- Paying for Confluent, Debezium, SymmetricDS,
Striim can save you a lot of time.
Don’t reinvent the
wheel.
Use Kafka Connect
or Pulsar IO.
Pay for it if you can.

15. Resources
● https://engineeringblog.yelp.com/2019/12/cassandra-source-connector-part-1.html
● https://www.instaclustr.com/blog/apache-kafka-kongo-part-4-1-connecting-kafka-
cassandra-kafka-connect/
● https://medium.com/walmartglobaltech/getting-started-with-the-kafka-connect-
cassandra-source-e6e06ec72e97
● https://blog.knoldus.com/kafka-connect-fundamentals/
● https://doc.akka.io/docs/alpakka/0.17/cassandra.html
● https://www.striim.com/docs/en/cassandra-writer.html
● https://www.confluent.io/blog/new-confluent-cloud-connector-features-and-single-
message-transforms/
● https://blog.anant.us/alpakka-cassandra-and-twitter/
● https://github.com/timurt/cassandra-trigger
● https://medium.com/building-the-open-data-stack/combine-data-at-rest-and-data-in-
motion-with-cdc-for-apache-cassandra-f4e1cdf08901