Calle Wilund presented on Change Data Capture (CDC) in Scylla. CDC in Scylla captures changes made to tables in the database and makes them available asynchronously to consumers. It is enabled per table and generates a log of modifications including pre-image, delta, and post-image data. This log is stored as another table in the database and can be consumed through normal CQL queries. CDC provides an easy way to integrate data duplication, replication, and analytics use cases without external tools.

1. Change Data
Capture in Scylla
Calle Wilund, COW

2. Presenter
Calle Wilund, Software Developer at ScyllaDB
Co-founder of Appeal Virtual Machines and one of the principal
architects behind the JRockit JVM, Calle Wilund has an
extensive background in software development, specializing in
virtual machines, compiler technologies and high performance
computing as well as systems manageability.

3. What is Change
Data Capture?

4. Change Data Capture - CDC
Consumable modification record for one or more tables in the database
■ Capture changes (write/delete)
■ Asynchronously readable by a consumer
■ Key feature in Scylla 2020

5. Use cases
■ Transaction analysis
● Fraud detection
● Kafka pipeline
■ Direct integration without third-party adaptor
■ Data duplication
● Database mirroring
● Database replication
■ <Insert your use case>

6. CDC in Scylla

7. What does it do
■ Enabled per table
■ On modification of a row
● Read pre-image (current state of the row) - optional
■ If row exists
■ For affected columns
● Add a log write to the modification
■ Pre-image data
■ Changes per column (delta)
■ Post-image (current state of row) - optional

8. How does the log work
■ CDC log per enabled table
■ CDC log just is another table
● Stored distributed on nodes in cluster
● Rows ordered by operation timestamp and batch sequence
● Mirrored columns for preimage/delta records
■ Every column record contains information about modification operation and TTL
● Topology matches source table
■ CDC log is colocated with original data
● But can use different consistency level (more or less reliable)
■ Data is transient
● CDC data is TTL:ed to 24h (configurable)
● Less risk for uncontrolled metadata buildup

9. Downsides
■ Read before write
● Additional latency
■ CDC log is eventually consistent, like everything else
● Concept of change is based on client view
■ I.e. data as seen and written via coordinator
● Does not contain information on how availability etc impacts actually resolved (read)
values later
● Can get partial logs in case of node crashes

10. Consuming the data

11. It is up to you. And us.
■ CDC data is available through normal CQL
● Easy to read raw stream
● Already de-duplicated
● All delta and pre image values are normal CQL data
● Can consume without knowledge of server internals
■ Layered approach
● CDC core functionality relatively simple. Allows for more sophisticated adaptors
■ Push models etc.
■ Integrators
● Kafka
● Alternator (dynamo API)
● More...

12. Nitty gritty

13. Sort of what happens...
Assuming a simple base table...
> CREATE TABLE base_table (
pk text,
ck text,
val1 text,
val2 text,
PRIMARY KEY (pk, ck)
) WITH cdc = { ‘enabled’ = ‘true’, preimage = ‘true’ };

14. Sort of what happens...
And an insert:
> insert into base_table(pk, ck, val1, val2) values(“foo”, “bar”, “val1”, “val2”);
> insert into base_table(pk, ck, val1, val2) values(“foo”, “baz”, “vaz1”, “vaz2”);
We get an initial CDC stream:
Stream_id | time | batch_seq | operation | ttl | _pk | _ck | _val1(op, value, ttl)| _val2(...)
----------+--------+-----------+------------+-----+-------+--------+----------------------+---------------------
UUID1 | <time1>| 0 | UPDATE | | “foo” | “bar” | (ADD, “val1”, null) | (ADD, “val2”, null)
UUID1 | <time2>| 0 | UPDATE | | “foo” | “baz” | (ADD, “vaz1”, null) | (ADD, “vaz2”, null)

15. Sort of what happens...
Now a modification:
> update base_table set val1 = “val3” where pk = “foo” and ck = “bar”;
In the CDC stream:
Stream_id | time | batch_seq | operation | ttl | _pk | _ck | _val1 | _val2
----------+---------+-----------+------------+-----+-------+--------+----------------------+---------------------
UUID1 | <time3> | 0 | PREIMAGE | | “foo” | “bar” | (ADD, “val1”, null) |
UUID1 | <time3> | 1 | UPDATE | | “foo” | “bar” | (ADD, “val3”, null) |

16. Sort of what happens...
Now a delete:
> delete val2 from base_table where pk = “foo” and ck = “bar”;
In the CDC stream:
Stream_id | time | batch_seq | operation | ttl | _pk | _ck | _val1 | _val2
----------+---------+-----------+------------+-----+-------+--------+----------------------+---------------------
UUID1 | <time4> | 0 | PREIMAGE | | “foo” | “bar” | | (ADD, “val2”, null)
UUID1 | <time4> | 1 | UPDATE | | “foo” | “bar” | | (DEL, null, null)

17. Sort of what happens...
A row delete:
> delete from base_table where pk = “foo” and ck = “bar”;
In the CDC stream:
Stream_id | time | batch_seq | operation | ttl | _pk | _ck | _val1 | _val2
----------+---------+-----------+------------+-----+-------+--------+----------------------+---------------------
UUID1 | <time5> | 0 | PREIMAGE | | “foo” | “bar” | (ADD, “val3”, null) |
UUID1 | <time5> | 1 | ROW_DEL | | “foo” | “bar” | |

18. Sort of what happens...
A partition delete:
> delete from base_table where pk = “foo”;
In the CDC stream:
Stream_id | time | batch_seq | operation | ttl | _pk | _ck | _val1 | _val2
----------+---------+-----------+------------+-----+-------+--------+----------------------+---------------------
UUID1 | <time6> | 1 | PART_DEL | | | | |

19. Take away

20. CDC in Scylla
■ Easy to integrate and consume
● Plain CQL tables
■ Robust
● Replicated in same way as normal data
■ Reasonable overhead
● Coalesced writes to same replica ranges
■ Does not overflow if consumer fails to act
● Data is TTL:ed

21. Comparison chart
Cassandra DynamoDB MongoDB Scylla
Consumer location on-node off-node off-node off-node
Replication duplicated deduplicated deduplicated deduplicated
Deltas yes no partial yes
Pre-image no yes no optional
Post-image no yes yes optional
Slow consumer
reaction
Table stopped Consumer loses data Consumer loses data Consumer loses data
Ordering no yes yes yes

22. CDC roadmap
■ Experimental feature in master branch
■ Production feature in Q1 2020
■ Additional features in Q2 2020 and forward
● Performance improvements
● Integrators
● Scylla Cloud

23. Thank you Stay in touch