ScyllaDB, profile picture
Uploaded byScyllaDB
PDF, PPTX1,253 views

Tracking Data Updates in Real-time with Change Data Capture

The document discusses ScyllaDB's Change Data Capture (CDC) feature, which allows real-time tracking of database modifications such as inserts, updates, and deletes. It highlights the benefits of CDC, including its asynchronous data capturing, table level granularity, and multiple deployment options across various clusters. Additionally, it outlines the use cases for CDC, including feeding microservices and real-time analytics, as well as practical examples of implementing CDC in a database context.

Embed presentation

Download as PDF, PPTX
Tracking Data Updates in Real-time with Change Data Capture Tzach Livyatan, ScyllaDB VP of Product Piotr Jastrzębski, ScyllaDB Software Team Lead
2 + The Real-Time Big Data Database + Drop-in replacement for Apache Cassandra and Amazon DynamoDB + 10X the performance & low tail latency + Open Source, Enterprise and Cloud options + Founded by the creators of KVM hypervisor + HQs: Palo Alto, CA, USA; Herzelia, Israel; Warsaw, Poland About ScyllaDB
Presenters Tzach Livyatan, VP of Product Tzach has a 15 year career in development, system engineering and product management. He has worked in the Telecom domain, focusing on carrier grade systems, signalling, policy and charging applications for Oracle and others. 3 Piotr Jastrzębski, Software Development Team Lead Piotr is a software engineer with over 12 years of experience in software development. He worked for a hedge fund called Two Sigma on alpha/system models execution infrastructure and for Google on Android Java Runtime and Google Search on Android.
Agenda 4 + About ScyllaDB + Introducing Scylla Change Data Capture - what is it good for ? + Using CDC - how to use it ? + Roadmap - when can I get it? + QA
Introducing Scylla Change Data Capture (CDC) 5
Change Data Capture - CDC Consumable modification record for one or more tables in the database + Capture changes (write/update/delete) + Asynchronously readable by a consumer + Table level Granularity + Highly Available + Persistence + (Eventually) Consistence
Feeding Microservices Kafka CDC Stream Fraud Detection Data Lake Real Time Analysis Search
Low Coupling Replication Agent Agent Multi DC Deployment Multi Cluster Deployment CDC Stream
Low Coupling Replication Multi DC CDC base Sync Granularity Key Space Table Consistency Strong / Eventual Weak Scylla Cluster Same Cluster Two Distinct Clusters Data Same on all replicas Can be different, manipulate by the agent PowerGuarantees
Low Coupling Use Cases + Active / Replication deployment + Clusters with different Scylla versions / editions / HW + Clusters with different TTL for the data. + Hot cluster: TTL = 1 day + Cold cluster: TTL = 1 month + Weak Consistency requirements
Poll Question
Using Scylla CDC
Table for Examples CREATE TABLE company.employees ( department text, last_name text, first_name text, age int, level int, PRIMARY KEY (department, last_name, first_name) ) WITH cdc = {/* CDC parameters go here */}; department last_name first_name age level Production Brown Robert 55 4 Production Johnson Alice 33 3 Production Smith John 35 2 Marketing Green Elizabeth 50 5 Marketing Miller Paul 21 1
CDC Modes + Delta + Preimage + Postimage
CDC Modes - Delta INSERT INTO employees (department, last_name, first_name, age, level) VALUES ('Production', 'Smith', 'John', 35, 2); department last_name first_name age level Production Smith John 35 2 department last_name first_name age level Production Smith John 35 2
CDC Modes - Delta INSERT INTO employees (department, last_name, first_name, age) VALUES ('Marketing', 'Green', 'Elizabeth', 50); department last_name first_name age level Production Smith John 35 2 Marketing Green Elizabeth 50 department last_name first_name age level Production Smith John 35 2 Marketing Green Elizabeth 50
CDC Modes - Delta UPDATE employees SET level = 3 WHERE department = 'Production' AND last_name = 'Smith' AND first_name = 'John'; department last_name first_name age level Production Smith John 35 2 Marketing Green Elizabeth 50 Production Smith John 3 department last_name first_name age level Production Smith John 35 3 Marketing Green Elizabeth 50
CDC Modes - Delta DELETE age FROM employees WHERE department = 'Production' AND last_name = 'Smith' AND first_name = 'John'; department last_name first_name age level Production Smith John 35 3 Marketing Green Elizabeth 50 Production Smith John department last_name first_name age level Production Smith John 3 Marketing Green Elizabeth 50
CDC Modes - Delta DELETE FROM employees WHERE department = 'Marketing' AND last_name = 'Green' AND first_name = 'Elizabeth'; department last_name first_name age level Production Smith John 3 Marketing Green Elizabeth 50 Marketing Green Elizabeth
CDC Modes - Preimage INSERT INTO employees (department, last_name, first_name, age, level) VALUES ('Production', 'Smith', 'John', 35, 2); department last_name first_name age leveldepartment last_name first_name age level Production Smith John 35 2 Production Smith John 35 2
CDC Modes - Preimage UPDATE employees SET level = 3 WHERE department = 'Production' AND last_name = 'Smith' AND first_name = 'John'; department last_name first_name age level Production Smith John 35 2 Production Smith John 2 department last_name first_name age level Production Smith John 35 3 Production Smith John 3
CDC Modes - Preimage DELETE age FROM employees WHERE department = 'Production' AND last_name = 'Smith' AND first_name = 'John'; department last_name first_name age level Production Smith John 35 3 Production Smith John 35 department last_name first_name age level Production Smith John 3 Production Smith John
CDC Modes - Preimage DELETE FROM employees WHERE department = 'Production' AND last_name = 'Smith' AND first_name = 'John'; department last_name first_name age level Production Smith John 35 3 Production Smith John 35 3 Production Smith John
CDC Modes - Postimage UPDATE employees SET level = 3 WHERE department = 'Production' AND last_name = 'Smith' AND first_name = 'John'; department last_name first_name age level Production Smith John 35 2 Production Smith John 35 3 department last_name first_name age level Production Smith John 35 3 Production Smith John 3
CDC Modes - Postimage DELETE age FROM employees WHERE department = 'Production' AND last_name = 'Smith' AND first_name = 'John'; department last_name first_name age level Production Smith John 35 3 Production Smith John 3 department last_name first_name age level Production Smith John 3 Production Smith John
CDC Modes - Postimage DELETE FROM employees WHERE department = 'Production' AND last_name = 'Smith' AND first_name = 'John'; department last_name first_name age level Production Smith John 35 3 Production Smith John Production Smith John
CREATE TABLE company.employees ( department text, last_name text, first_name text, age int, level int, PRIMARY KEY (department, last_name, first_name) ) WITH cdc = {/* CDC parameters go here */}; CDC Log CREATE TABLE company.employees_scylla_cdc_log ( cdc$stream_id blob, cdc$time timeuuid, cdc$batch_seq_no int, cdc$operation tinyint, cdc$ttl bigint, department text, first_name text, last_name text, age int, cdc$deleted_age boolean, level int, cdc$deleted_level boolean, PRIMARY KEY (cdc$stream_id, cdc$time, cdc$batch_seq_no) )
CDC Log CREATE TABLE company.employees ( department text, last_name text, first_name text, age int, level int, PRIMARY KEY (department, last_name, first_name) ) WITH cdc = {/* CDC parameters go here */}; CREATE TABLE company.employees_scylla_cdc_log ( cdc$stream_id blob, cdc$time timeuuid, cdc$batch_seq_no int, cdc$operation tinyint, cdc$ttl bigint, department text, first_name text, last_name text, age int, cdc$deleted_age boolean, level int, cdc$deleted_level boolean, PRIMARY KEY (cdc$stream_id, cdc$time, cdc$batch_seq_no) )
CDC Log CREATE TABLE company.employees ( department text, last_name text, first_name text, age int, level int, PRIMARY KEY (department, last_name, first_name) ) WITH cdc = {/* CDC parameters go here */}; CREATE TABLE company.employees_scylla_cdc_log ( cdc$stream_id blob, cdc$time timeuuid, cdc$batch_seq_no int, cdc$operation tinyint, cdc$ttl bigint, department text, first_name text, last_name text, age int, cdc$deleted_age boolean, level int, cdc$deleted_level boolean, PRIMARY KEY (cdc$stream_id, cdc$time, cdc$batch_seq_no) )
CREATE TABLE company.employees ( department text, last_name text, first_name text, age int, level int, PRIMARY KEY (department, last_name, first_name) ) WITH cdc = {/* CDC parameters go here */}; CREATE TABLE company.employees_scylla_cdc_log ( cdc$stream_id blob, cdc$time timeuuid, cdc$batch_seq_no int, cdc$operation tinyint, cdc$ttl bigint, department text, first_name text, last_name text, age int, cdc$deleted_age boolean, level int, cdc$deleted_level boolean, PRIMARY KEY (cdc$stream_id, cdc$time, cdc$batch_seq_no) ) CDC Log
CDC Log CREATE TABLE company.employees_scylla_cdc_log ( cdc$stream_id blob, cdc$time timeuuid, cdc$batch_seq_no int, cdc$operation tinyint, cdc$ttl bigint, department text, first_name text, last_name text, age int, cdc$deleted_age boolean, level int, cdc$deleted_level boolean, PRIMARY KEY (cdc$stream_id, cdc$time, cdc$batch_seq_no) ) CREATE TABLE company.employees ( department text, last_name text, first_name text, age int, level int, PRIMARY KEY (department, last_name, first_name) ) WITH cdc = {/* CDC parameters go here */};
CREATE TABLE company.employees_scylla_cdc_log ( cdc$stream_id blob, cdc$time timeuuid, cdc$batch_seq_no int, cdc$operation tinyint, cdc$ttl bigint, department text, first_name text, last_name text, age int, cdc$deleted_age boolean, level int, cdc$deleted_level boolean, PRIMARY KEY (cdc$stream_id, cdc$time, cdc$batch_seq_no) ) CDC Log CREATE TABLE company.employees ( department text, last_name text, first_name text, age int, level int, PRIMARY KEY (department, last_name, first_name) ) WITH cdc = {/* CDC parameters go here */};
CREATE TABLE company.employees_scylla_cdc_log ( cdc$stream_id blob, cdc$time timeuuid, cdc$batch_seq_no int, cdc$operation tinyint, cdc$ttl bigint, department text, first_name text, last_name text, age int, cdc$deleted_age boolean, level int, cdc$deleted_level boolean, PRIMARY KEY (cdc$stream_id, cdc$time, cdc$batch_seq_no) ) CDC Log CREATE TABLE company.employees ( department text, last_name text, first_name text, age int, level int, PRIMARY KEY (department, last_name, first_name) ) WITH cdc = {/* CDC parameters go here */};
CDC Log CREATE TABLE company.employees_scylla_cdc_log ( cdc$stream_id blob, cdc$time timeuuid, cdc$batch_seq_no int, cdc$operation tinyint, cdc$ttl bigint, department text, first_name text, last_name text, age int, cdc$deleted_age boolean, level int, cdc$deleted_level boolean, PRIMARY KEY (cdc$stream_id, cdc$time, cdc$batch_seq_no) ) CREATE TABLE company.employees ( department text, last_name text, first_name text, age int, level int, PRIMARY KEY (department, last_name, first_name) ) WITH cdc = {/* CDC parameters go here */};
CREATE TABLE company.employees ( department text, last_name text, first_name text, age int, level int, PRIMARY KEY (department, last_name, first_name) ) WITH cdc = {/* CDC parameters go here */}; CDC log CREATE TABLE company.employees_scylla_cdc_log ( cdc$stream_id blob, cdc$time timeuuid, cdc$batch_seq_no int, cdc$operation tinyint, cdc$ttl bigint, department text, first_name text, last_name text, age int, cdc$deleted_age boolean, level int, cdc$deleted_level boolean, PRIMARY KEY (cdc$stream_id, cdc$time, cdc$batch_seq_no) ) 0 PREIMAGE 1 UPDATE 2 INSERT 3 ROW DELETE 4 PARTITION DELETE 5 RANGE DELETE INCLUSIVE LOWER BOUND 6 RANGE DELETE EXCLUSIVE LOWER BOUND 7 RANGE DELETE INCLUSIVE UPPER BOUND 8 RANGE DELETE EXCLUSIVE UPPER BOUND 9 POSTIMAGE
CREATE TABLE company.employees_scylla_cdc_log ( cdc$stream_id blob, cdc$time timeuuid, cdc$batch_seq_no int, cdc$operation tinyint, cdc$ttl bigint, department text, first_name text, last_name text, age int, cdc$deleted_age boolean, level int, cdc$deleted_level boolean, PRIMARY KEY (cdc$stream_id, cdc$time, cdc$batch_seq_no) ) CREATE TABLE company.employees ( department text, last_name text, first_name text, age int, level int, PRIMARY KEY (department, last_name, first_name) ) WITH cdc = {/* CDC parameters go here */}; CDC log0 PREIMAGE 1 UPDATE 2 INSERT 3 ROW DELETE 4 PARTITION DELETE 5 RANGE DELETE INCLUSIVE LOWER BOUND 6 RANGE DELETE EXCLUSIVE LOWER BOUND 7 RANGE DELETE INCLUSIVE UPPER BOUND 8 RANGE DELETE EXCLUSIVE UPPER BOUND 9 POSTIMAGE UPDATE employees SET level = 3 WHERE department = 'Production' AND last_name = 'Smith' AND first_name = 'John';
CREATE TABLE company.employees_scylla_cdc_log ( cdc$stream_id blob, cdc$time timeuuid, cdc$batch_seq_no int, cdc$operation tinyint, cdc$ttl bigint, department text, first_name text, last_name text, age int, cdc$deleted_age boolean, level int, cdc$deleted_level boolean, PRIMARY KEY (cdc$stream_id, cdc$time, cdc$batch_seq_no) ) CREATE TABLE company.employees ( department text, last_name text, first_name text, age int, level int, PRIMARY KEY (department, last_name, first_name) ) WITH cdc = {/* CDC parameters go here */}; CDC log0 PREIMAGE 1 UPDATE 2 INSERT 3 ROW DELETE 4 PARTITION DELETE 5 RANGE DELETE INCLUSIVE LOWER BOUND 6 RANGE DELETE EXCLUSIVE LOWER BOUND 7 RANGE DELETE INCLUSIVE UPPER BOUND 8 RANGE DELETE EXCLUSIVE UPPER BOUND 9 POSTIMAGE INSERT INTO employees (department, last_name, first_name, age, level) VALUES ('Production', 'Smith', 'John', 35, 2);
CREATE TABLE company.employees_scylla_cdc_log ( cdc$stream_id blob, cdc$time timeuuid, cdc$batch_seq_no int, cdc$operation tinyint, cdc$ttl bigint, department text, first_name text, last_name text, age int, cdc$deleted_age boolean, level int, cdc$deleted_level boolean, PRIMARY KEY (cdc$stream_id, cdc$time, cdc$batch_seq_no) ) CREATE TABLE company.employees ( department text, last_name text, first_name text, age int, level int, PRIMARY KEY (department, last_name, first_name) ) WITH cdc = {/* CDC parameters go here */}; CDC log0 PREIMAGE 1 UPDATE 2 INSERT 3 ROW DELETE 4 PARTITION DELETE 5 RANGE DELETE INCLUSIVE LOWER BOUND 6 RANGE DELETE EXCLUSIVE LOWER BOUND 7 RANGE DELETE INCLUSIVE UPPER BOUND 8 RANGE DELETE EXCLUSIVE UPPER BOUND 9 POSTIMAGE DELETE FROM employees WHERE department = 'Production' AND last_name = 'Smith' AND first_name = 'John';
CREATE TABLE company.employees_scylla_cdc_log ( cdc$stream_id blob, cdc$time timeuuid, cdc$batch_seq_no int, cdc$operation tinyint, cdc$ttl bigint, department text, first_name text, last_name text, age int, cdc$deleted_age boolean, level int, cdc$deleted_level boolean, PRIMARY KEY (cdc$stream_id, cdc$time, cdc$batch_seq_no) ) CREATE TABLE company.employees ( department text, last_name text, first_name text, age int, level int, PRIMARY KEY (department, last_name, first_name) ) WITH cdc = {/* CDC parameters go here */}; CDC log0 PREIMAGE 1 UPDATE 2 INSERT 3 ROW DELETE 4 PARTITION DELETE 5 RANGE DELETE INCLUSIVE LOWER BOUND 6 RANGE DELETE EXCLUSIVE LOWER BOUND 7 RANGE DELETE INCLUSIVE UPPER BOUND 8 RANGE DELETE EXCLUSIVE UPPER BOUND 9 POSTIMAGE DELETE FROM employees WHERE department = 'Production';
CREATE TABLE company.employees ( department text, last_name text, first_name text, age int, level int, PRIMARY KEY (department, last_name, first_name) ) WITH cdc = {/* CDC parameters go here */}; CDC log0 PREIMAGE 1 UPDATE 2 INSERT 3 ROW DELETE 4 PARTITION DELETE 5 RANGE DELETE INCLUSIVE LOWER BOUND 6 RANGE DELETE EXCLUSIVE LOWER BOUND 7 RANGE DELETE INCLUSIVE UPPER BOUND 8 RANGE DELETE EXCLUSIVE UPPER BOUND 9 POSTIMAGE CREATE TABLE company.employees_scylla_cdc_log ( cdc$stream_id blob, cdc$time timeuuid, cdc$batch_seq_no int, cdc$operation tinyint, cdc$ttl bigint, department text, first_name text, last_name text, age int, cdc$deleted_age boolean, level int, cdc$deleted_level boolean, PRIMARY KEY (cdc$stream_id, cdc$time, cdc$batch_seq_no) ) DELETE FROM employees WHERE department = 'Production' AND last_name >= 'A' AND last_name < 'C';
CDC Log CREATE TABLE company.employees_scylla_cdc_log ( cdc$stream_id blob, cdc$time timeuuid, cdc$batch_seq_no int, cdc$operation tinyint, cdc$ttl bigint, department text, first_name text, last_name text, age int, cdc$deleted_age boolean, level int, cdc$deleted_level boolean, PRIMARY KEY (cdc$stream_id, cdc$time, cdc$batch_seq_no) ) CREATE TABLE company.employees ( department text, last_name text, first_name text, age int, level int, PRIMARY KEY (department, last_name, first_name) ) WITH cdc = {/* CDC parameters go here */};
scylla.yaml: experimental_features: - cdc Command line: --experimental CREATE TABLE company.employees ( ... ) WITH cdc = {'enabled' : true}; ALTER TABLE company.employees WITH cdc = {'enabled' : true}; WITH cdc = { 'enabled' : true, 'preimage' : true, 'postimage' : true, 'ttl' : 1000 }; How to Enable CDC
INSERT INTO employees (department, last_name, first_name, level) VALUES ('Production', 'Smith', 'John', 2); cdc$stream_id 0x6f06e163e099df1b8545901cd23663e3 cdc$time 54b28d54-69cb-11ea-6541-6fa9f3f294ae cdc$batch_seq_no 0 cdc$operation 2 cdc$ttl null department Production last_name Smith first_name John age null level 2 cdc$deleted_age null cdc$deleted_level null
UPDATE employees SET age = 35 WHERE department = 'Production' AND last_name = 'Smith' AND first_name = 'John'; cdc$stream_id 0x6f06e163e099df1b8545901cd23663e3 cdc$time b82e2cb0-69cd-11ea-6a10-8c39c8f7dc3a cdc$batch_seq_no 0 cdc$operation 1 cdc$ttl null department Production last_name Smith first_name John age 35 level null cdc$deleted_age null cdc$deleted_level null
DELETE age FROM employees WHERE department = 'Production' AND last_name = 'Smith' AND first_name = 'John'; cdc$stream_id 0x6f06e163e099df1b8545901cd23663e3 cdc$time c8f5b55e-69cd-11ea-68c1-e036079d1ce4 cdc$batch_seq_no 0 cdc$operation 1 cdc$ttl null department Production last_name Smith first_name John age null level null cdc$deleted_age true cdc$deleted_level null
DELETE FROM employees WHERE department = 'Production' AND last_name = 'Smith' AND first_name = 'John'; cdc$stream_id 0x6f06e163e099df1b8545901cd23663e3 cdc$time e75e6612-69cd-11ea-dd3e-2e150dc8859c cdc$batch_seq_no 0 cdc$operation 3 cdc$ttl null department Production last_name Smith first_name John age null level null cdc$deleted_age null cdc$deleted_level null
DELETE FROM employees WHERE department = 'Production'; cdc$stream_id 0x6f06e163e099df1b8545901cd23663e3 cdc$time 0a454bbe-69ce-11ea-31f2-98b4c3f08688 cdc$batch_seq_no 0 cdc$operation 4 cdc$ttl null department Production last_name null first_name null age null level null cdc$deleted_age null cdc$deleted_level null
DELETE FROM employees WHERE department = 'Production' AND last_name >= 'A' AND last_name < 'C'; cdc$stream_id 0x6f06e163e099df1b8545901cd23663e3 cdc$time 447dd18e-69ce-11ea-6ff4-81708ddd348a cdc$batch_seq_no 0 1 cdc$operation 5 8 cdc$ttl null null department Production Production last_name A C first_name null null age null null level null null cdc$deleted_age null null cdc$deleted_level null null
CDC Write RF=3 CL = QUORUM (2)
CDC Streams CREATE TABLE company.employees_scylla_cdc_log ( cdc$stream_id blob, cdc$time timeuuid, cdc$batch_seq_no int, ... PRIMARY KEY (cdc$stream_id, cdc$time, cdc$batch_seq_no) ) CREATE TABLE company.employees ( ... PRIMARY KEY (department, last_name, first_name) ) WITH cdc = {/* CDC parameters go here */};
Benchmark Results CDC disabled Delta Delta + Preimage Throughput (ops) 78,847 61,851 ( -21.55%) 33,358 ( -57.69%) Mean latency (ms) 1.3 1.6 (+23.07%) 3.0 (+130.76%) 99th percentile latency (ms) 3.6 4.7 (+30.55%) 9.3 (+158.33%)
Take Away
CDC in Scylla + Easy to integrate and consume + Plain CQL tables + Robust + Replicated in same way as normal data + Reasonable overhead + Coalesced writes to same replica ranges + Does not overflow if consumer fails to act + Data is TTL:ed
Comparison Chart Cassandra DynamoDB MongoDB Scylla Consumer location on-node off-node off-node off-node Replication duplicated deduplicated deduplicated deduplicated Deltas yes no partial yes Pre-image no yes no optional Post-image no yes yes optional Slow consumer reaction Table stopped Consumer loses data Consumer loses data Consumer loses data Ordering no yes yes yes
Try Scylla CDC Now! + Download Scylla Nightly Docker and use --experimental https://hub.docker.com/r/scylladb/scylla-nightly + Use and follow the docs: https://docs.scylladb.com/using-scylla/cdc/ 55
Q&A Tzach Livyatan Piotr Jastrzębski
piotr@scylladb.com tzach@scylladb.com Stay in touch
United States 545 Faber Place Palo Alto, CA 94303 Israel 11 Galgalei Haplada Herzelia, Israel www.scylladb.com @scylladb Thank you

More Related Content

PDF
Tech Talk: Best Practices for Data Modeling
byScyllaDB
 
PDF
Quick Wins
byHighLoad2009
 
RTF
Seistech SQL code
bySimon Hoyle
 
PDF
Data handling in r
byAbhik Seal
 
PDF
Association Rule Mining with R
byYanchang Zhao
 
TXT
Oracle ORA Errors
byManish Mudhliyar
 
PDF
Analyzing the Performance Effects of Meltdown + Spectre on Apache Spark Workl...
byDatabricks
 
PDF
[Pgday.Seoul 2021] 2. Porting Oracle UDF and Optimization
byPgDay.Seoul
 
Tech Talk: Best Practices for Data Modeling
byScyllaDB
 
Quick Wins
byHighLoad2009
 
Seistech SQL code
bySimon Hoyle
 
Data handling in r
byAbhik Seal
 
Association Rule Mining with R
byYanchang Zhao
 
Oracle ORA Errors
byManish Mudhliyar
 
Analyzing the Performance Effects of Meltdown + Spectre on Apache Spark Workl...
byDatabricks
 
[Pgday.Seoul 2021] 2. Porting Oracle UDF and Optimization
byPgDay.Seoul
 

What's hot

PPTX
Database administration commands
byVarsha Ajith
 
PDF
Sangam 19 - PLSQL still the coolest
byConnor McDonald
 
PDF
MySQL Idiosyncrasies That Bite
byRonald Bradford
 
PPTX
Dun ddd
byLyuben Todorov
 
PDF
Connor McDonald Partitioning
byInSync Conference
 
PDF
EKON22 Introduction to Machinelearning
byMax Kleiner
 
DOCX
Materi my sql part 1
byAmar Senjaku Ofdetraisar
 
PDF
Visualization of Supervised Learning with {arules} + {arulesViz}
byTakashi J OZAKI
 
PDF
PLOTCON NYC: Behind Every Great Plot There's a Great Deal of Wrangling
byPlotly
 
PPT
Dbms plan - A swiss army knife for performance engineers
byRiyaj Shamsudeen
 
PPT
Oracle sql ppt1
byMadhavendra Dutt
 
PPTX
OTN TOUR 2016 - DBA Commands and Concepts That Every Developer Should Know
byAlex Zaballa
 
TXT
Quick reference for cql
byRajkumar Asohan
 
DOCX
Tugas praktikum smbd
byAmar Senjaku Ofdetraisar
 
PDF
15 MySQL Basics #burningkeyboards
byDenis Ristic
 
PPTX
Introduction To MySQL Lecture 1
byAjay Khatri
 
PDF
Robert Bernier - Recovering From A Damaged PostgreSQL Cluster @ Postgres Open
byPostgresOpen
 
PDF
Flashback ITOUG
byConnor McDonald
 
DOCX
TimesTen in memory database Creation
byMonowar Mukul
 
PDF
MySQL Idiosyncrasies That Bite 2010.07
byRonald Bradford
 
Database administration commands
byVarsha Ajith
 
Sangam 19 - PLSQL still the coolest
byConnor McDonald
 
MySQL Idiosyncrasies That Bite
byRonald Bradford
 
Dun ddd
byLyuben Todorov
 
Connor McDonald Partitioning
byInSync Conference
 
EKON22 Introduction to Machinelearning
byMax Kleiner
 
Materi my sql part 1
byAmar Senjaku Ofdetraisar
 
Visualization of Supervised Learning with {arules} + {arulesViz}
byTakashi J OZAKI
 
PLOTCON NYC: Behind Every Great Plot There's a Great Deal of Wrangling
byPlotly
 
Dbms plan - A swiss army knife for performance engineers
byRiyaj Shamsudeen
 
Oracle sql ppt1
byMadhavendra Dutt
 
OTN TOUR 2016 - DBA Commands and Concepts That Every Developer Should Know
byAlex Zaballa
 
Quick reference for cql
byRajkumar Asohan
 
Tugas praktikum smbd
byAmar Senjaku Ofdetraisar
 
15 MySQL Basics #burningkeyboards
byDenis Ristic
 
Introduction To MySQL Lecture 1
byAjay Khatri
 
Robert Bernier - Recovering From A Damaged PostgreSQL Cluster @ Postgres Open
byPostgresOpen
 
Flashback ITOUG
byConnor McDonald
 
TimesTen in memory database Creation
byMonowar Mukul
 
MySQL Idiosyncrasies That Bite 2010.07
byRonald Bradford
 

Similar to Tracking Data Updates in Real-time with Change Data Capture

PPTX
Change Data Capture in Scylla
byScyllaDB
 
PPTX
SQL and Stored Procedures question which are asked in Interviews
bypcmbinstitute87
 
PDF
Introduction to data modeling with apache cassandra
byPatrick McFadin
 
PDF
Cassandra Day Atlanta 2015: Data Modeling 101
byDataStax Academy
 
PDF
Cassandra Day Chicago 2015: Apache Cassandra Data Modeling 101
byDataStax Academy
 
PDF
Cassandra Day London 2015: Data Modeling 101
byDataStax Academy
 
PPT
Les09
bySudharsan S
 
PPT
Les09 (using ddl statements to create and manage tables)
byAchmad Solichin
 
PDF
Cdc
bySabyasachi Srimany
 
PPTX
Database Management System Review
byKaya Ota
 
PDF
4.Database Management System.pdf
byExport Promotion Bureau
 
PDF
Time series with apache cassandra strata
byPatrick McFadin
 
PPTX
ScyllaDB Real-Time Event Processing with CDC
byScyllaDB
 
PDF
MariaDB Temporal Tables
byFederico Razzoli
 
PDF
PHPDay 2019 - MySQL 8, not only good, great!
byGabriela Ferrara
 
PDF
Session 1 - Databases-JUNE 2023.pdf
bySwapnilSaurav7
 
PPTX
In memory databases presentation
byMichael Keane
 
PPT
Les09
byVijay Kumar
 
PPTX
Dbms sql-final
byNV Chandra Sekhar Nittala
 
PPTX
New SQL features in latest MySQL releases
byGeorgi Sotirov
 
Change Data Capture in Scylla
byScyllaDB
 
SQL and Stored Procedures question which are asked in Interviews
bypcmbinstitute87
 
Introduction to data modeling with apache cassandra
byPatrick McFadin
 
Cassandra Day Atlanta 2015: Data Modeling 101
byDataStax Academy
 
Cassandra Day Chicago 2015: Apache Cassandra Data Modeling 101
byDataStax Academy
 
Cassandra Day London 2015: Data Modeling 101
byDataStax Academy
 
Les09
bySudharsan S
 
Les09 (using ddl statements to create and manage tables)
byAchmad Solichin
 
Cdc
bySabyasachi Srimany
 
Database Management System Review
byKaya Ota
 
4.Database Management System.pdf
byExport Promotion Bureau
 
Time series with apache cassandra strata
byPatrick McFadin
 
ScyllaDB Real-Time Event Processing with CDC
byScyllaDB
 
MariaDB Temporal Tables
byFederico Razzoli
 
PHPDay 2019 - MySQL 8, not only good, great!
byGabriela Ferrara
 
Session 1 - Databases-JUNE 2023.pdf
bySwapnilSaurav7
 
In memory databases presentation
byMichael Keane
 
Les09
byVijay Kumar
 
Dbms sql-final
byNV Chandra Sekhar Nittala
 
New SQL features in latest MySQL releases
byGeorgi Sotirov
 

More from ScyllaDB

PDF
Real-Time AI Masterclass: Vector Search at Scale
byScyllaDB
 
PDF
Real-Time AI at Scale Masterclass: Feature Store at Scale
byScyllaDB
 
PDF
Real-Time AI at Scale Masterclass - Challenges of AI at Scale
byScyllaDB
 
PDF
How to Evaluate a High Performance Database
byScyllaDB
 
PDF
GPUS and How to Program Them by Manya Bansal
byScyllaDB
 
PDF
Database Performance at Scale: The TL;DR
byScyllaDB
 
PPTX
xCapture v3: Efficient, Always-On Thread Level Observability with eBPF by Tan...
byScyllaDB
 
PDF
ZGC: A Decade of Innovation by Stefan Johansson
byScyllaDB
 
PDF
ChatGPT Ain't Got $%@& On Me! by Andy Pavlo
byScyllaDB
 
PDF
Designing an Energy-efficient Architecture for Geo Databases by Yichen Wei
byScyllaDB
 
PDF
Bridging epoll and io_uring in Async Rust by Tzu Gwo
byScyllaDB
 
PDF
The Tale of Taming TigerBeetle's Tail Latency by Tobias Ziegler
byScyllaDB
 
PDF
From Gatekeeper to Kyverno : Kubernetes Policy Management with Performance by...
byScyllaDB
 
PDF
Design Considerations for P99-optimized Hash Tables by Steve Heller
byScyllaDB
 
PDF
Cassandra vs. ScyllaDB: Evolutionary Differences
byScyllaDB
 
PDF
40x Faster Binary Search by Ragnar Groot Koerkamp
byScyllaDB
 
PDF
The Gory Details of a Full-Featured Userspace CPU Scheduler by Avi Kivity
byScyllaDB
 
PDF
Turbocharging MCP: Speed, Smarts, and Scale by Viraj Sharma
byScyllaDB
 
PDF
What Would You Give for Speed: Trade-offs in Eventually Consistent Systems at...
byScyllaDB
 
PDF
Measuring Query Latency the Hard Way: An Adventure in Impractical Postgres Mo...
byScyllaDB
 
Real-Time AI Masterclass: Vector Search at Scale
byScyllaDB
 
Real-Time AI at Scale Masterclass: Feature Store at Scale
byScyllaDB
 
Real-Time AI at Scale Masterclass - Challenges of AI at Scale
byScyllaDB
 
How to Evaluate a High Performance Database
byScyllaDB
 
GPUS and How to Program Them by Manya Bansal
byScyllaDB
 
Database Performance at Scale: The TL;DR
byScyllaDB
 
xCapture v3: Efficient, Always-On Thread Level Observability with eBPF by Tan...
byScyllaDB
 
ZGC: A Decade of Innovation by Stefan Johansson
byScyllaDB
 
ChatGPT Ain't Got $%@& On Me! by Andy Pavlo
byScyllaDB
 
Designing an Energy-efficient Architecture for Geo Databases by Yichen Wei
byScyllaDB
 
Bridging epoll and io_uring in Async Rust by Tzu Gwo
byScyllaDB
 
The Tale of Taming TigerBeetle's Tail Latency by Tobias Ziegler
byScyllaDB
 
From Gatekeeper to Kyverno : Kubernetes Policy Management with Performance by...
byScyllaDB
 
Design Considerations for P99-optimized Hash Tables by Steve Heller
byScyllaDB
 
Cassandra vs. ScyllaDB: Evolutionary Differences
byScyllaDB
 
40x Faster Binary Search by Ragnar Groot Koerkamp
byScyllaDB
 
The Gory Details of a Full-Featured Userspace CPU Scheduler by Avi Kivity
byScyllaDB
 
Turbocharging MCP: Speed, Smarts, and Scale by Viraj Sharma
byScyllaDB
 
What Would You Give for Speed: Trade-offs in Eventually Consistent Systems at...
byScyllaDB
 
Measuring Query Latency the Hard Way: An Adventure in Impractical Postgres Mo...
byScyllaDB
 

Recently uploaded

PPTX
API Gateway Architecture - Technical Report 2026
byPowersoft2026
 
PPTX
Code Like Bro -A guide for better Coding
byMadan Panthi
 
PDF
Azure DevOps Managed Services: Driving Speed, Security, and Business Growth
byjohncarterjn
 
DOCX
Best Web to Learn About Buying Verified Skrill Accounts (Los Angeles).docx
byhttps://topsellerit.com/product/buy-verified-binance-accounts/
 
PDF
Deploying Windows Clients & Managing Identities
byVICTOR MAESTRE RAMIREZ
 
PDF
Chapter 3 Cryptography and encryption techniques.pdf
byGetnet Tigabie Askale -(GM)
 
PDF
What ONIX can do: Leveraging metadata to support the discoverability of First...
byBookNet Canada
 
PDF
Certified AI-Native Foundations Professional.pdf
byMarc Entsminger SPC6, RTE, SSM, SA, SDP
 
PPTX
SOFTWARE DEVELOPMENT PROCESS - INTRODUCTION
byParithi Thamizh
 
PPT
Database Management Systems: Basics, History, Data Models, etc.
byParithi Thamizh
 
PDF
Computer Science Conferences 2026 | Research by SAI Conference
byHealth Conference 2026
 
PPTX
apidays Australia 2025 | Building AI RAG Applications with No Code.pptx
byapidays
 
PPTX
Spacecraft Guidance Quick Research Guide by Arthur Morgan
byArthur Morgan
 
PDF
Why Are Cloud Migration Services Essential for Modern Business Growth?
byGeoPITS Global Pvt Ltd
 
PDF
Global Asset Tokenization Platforms – Summary (2026) - Garima Singh
byGarima Singh
 
PDF
How the EU Ecolabel's Record Growth Creates ESG Opportunities for CRE
byWastify AI
 
PDF
How to Build a Crypto Wallet that Excels in 2026.pdf
byimoliviabennett
 
PDF
Peculiar Findings Around CEX Activity and ETH Price
bytobbymnbvcxz
 
PPTX
AI Meets DBA Transforming Database Administration with Intelligence
byGeoPITS Global Pvt Ltd
 
PDF
Teaching Robots how to Read 1/2: AI Center & Classic Document Understanding (...
byanabulhac
 
API Gateway Architecture - Technical Report 2026
byPowersoft2026
 
Code Like Bro -A guide for better Coding
byMadan Panthi
 
Azure DevOps Managed Services: Driving Speed, Security, and Business Growth
byjohncarterjn
 
Best Web to Learn About Buying Verified Skrill Accounts (Los Angeles).docx
byhttps://topsellerit.com/product/buy-verified-binance-accounts/
 
Deploying Windows Clients & Managing Identities
byVICTOR MAESTRE RAMIREZ
 
Chapter 3 Cryptography and encryption techniques.pdf
byGetnet Tigabie Askale -(GM)
 
What ONIX can do: Leveraging metadata to support the discoverability of First...
byBookNet Canada
 
Certified AI-Native Foundations Professional.pdf
byMarc Entsminger SPC6, RTE, SSM, SA, SDP
 
SOFTWARE DEVELOPMENT PROCESS - INTRODUCTION
byParithi Thamizh
 
Database Management Systems: Basics, History, Data Models, etc.
byParithi Thamizh
 
Computer Science Conferences 2026 | Research by SAI Conference
byHealth Conference 2026
 
apidays Australia 2025 | Building AI RAG Applications with No Code.pptx
byapidays
 
Spacecraft Guidance Quick Research Guide by Arthur Morgan
byArthur Morgan
 
Why Are Cloud Migration Services Essential for Modern Business Growth?
byGeoPITS Global Pvt Ltd
 
Global Asset Tokenization Platforms – Summary (2026) - Garima Singh
byGarima Singh
 
How the EU Ecolabel's Record Growth Creates ESG Opportunities for CRE
byWastify AI
 
How to Build a Crypto Wallet that Excels in 2026.pdf
byimoliviabennett
 
Peculiar Findings Around CEX Activity and ETH Price
bytobbymnbvcxz
 
AI Meets DBA Transforming Database Administration with Intelligence
byGeoPITS Global Pvt Ltd
 
Teaching Robots how to Read 1/2: AI Center & Classic Document Understanding (...
byanabulhac
 

Tracking Data Updates in Real-time with Change Data Capture

  • 1.
    Tracking Data Updates inReal-time with Change Data Capture Tzach Livyatan, ScyllaDB VP of Product Piotr Jastrzębski, ScyllaDB Software Team Lead
  • 2.
    2 + The Real-TimeBig Data Database + Drop-in replacement for Apache Cassandra and Amazon DynamoDB + 10X the performance & low tail latency + Open Source, Enterprise and Cloud options + Founded by the creators of KVM hypervisor + HQs: Palo Alto, CA, USA; Herzelia, Israel; Warsaw, Poland About ScyllaDB
  • 3.
    Presenters Tzach Livyatan, VPof Product Tzach has a 15 year career in development, system engineering and product management. He has worked in the Telecom domain, focusing on carrier grade systems, signalling, policy and charging applications for Oracle and others. 3 Piotr Jastrzębski, Software Development Team Lead Piotr is a software engineer with over 12 years of experience in software development. He worked for a hedge fund called Two Sigma on alpha/system models execution infrastructure and for Google on Android Java Runtime and Google Search on Android.
  • 4.
    Agenda 4 + About ScyllaDB +Introducing Scylla Change Data Capture - what is it good for ? + Using CDC - how to use it ? + Roadmap - when can I get it? + QA
  • 5.
  • 6.
    Change Data Capture- CDC Consumable modification record for one or more tables in the database + Capture changes (write/update/delete) + Asynchronously readable by a consumer + Table level Granularity + Highly Available + Persistence + (Eventually) Consistence
  • 7.
  • 8.
    Low Coupling Replication Agent Agent MultiDC Deployment Multi Cluster Deployment CDC Stream
  • 9.
    Low Coupling Replication MultiDC CDC base Sync Granularity Key Space Table Consistency Strong / Eventual Weak Scylla Cluster Same Cluster Two Distinct Clusters Data Same on all replicas Can be different, manipulate by the agent PowerGuarantees
  • 10.
    Low Coupling UseCases + Active / Replication deployment + Clusters with different Scylla versions / editions / HW + Clusters with different TTL for the data. + Hot cluster: TTL = 1 day + Cold cluster: TTL = 1 month + Weak Consistency requirements
  • 11.
  • 12.
  • 13.
    Table for Examples CREATETABLE company.employees ( department text, last_name text, first_name text, age int, level int, PRIMARY KEY (department, last_name, first_name) ) WITH cdc = {/* CDC parameters go here */}; department last_name first_name age level Production Brown Robert 55 4 Production Johnson Alice 33 3 Production Smith John 35 2 Marketing Green Elizabeth 50 5 Marketing Miller Paul 21 1
  • 14.
    CDC Modes + Delta +Preimage + Postimage
  • 15.
    CDC Modes -Delta INSERT INTO employees (department, last_name, first_name, age, level) VALUES ('Production', 'Smith', 'John', 35, 2); department last_name first_name age level Production Smith John 35 2 department last_name first_name age level Production Smith John 35 2
  • 16.
    CDC Modes -Delta INSERT INTO employees (department, last_name, first_name, age) VALUES ('Marketing', 'Green', 'Elizabeth', 50); department last_name first_name age level Production Smith John 35 2 Marketing Green Elizabeth 50 department last_name first_name age level Production Smith John 35 2 Marketing Green Elizabeth 50
  • 17.
    CDC Modes -Delta UPDATE employees SET level = 3 WHERE department = 'Production' AND last_name = 'Smith' AND first_name = 'John'; department last_name first_name age level Production Smith John 35 2 Marketing Green Elizabeth 50 Production Smith John 3 department last_name first_name age level Production Smith John 35 3 Marketing Green Elizabeth 50
  • 18.
    CDC Modes -Delta DELETE age FROM employees WHERE department = 'Production' AND last_name = 'Smith' AND first_name = 'John'; department last_name first_name age level Production Smith John 35 3 Marketing Green Elizabeth 50 Production Smith John department last_name first_name age level Production Smith John 3 Marketing Green Elizabeth 50
  • 19.
    CDC Modes -Delta DELETE FROM employees WHERE department = 'Marketing' AND last_name = 'Green' AND first_name = 'Elizabeth'; department last_name first_name age level Production Smith John 3 Marketing Green Elizabeth 50 Marketing Green Elizabeth
  • 20.
    CDC Modes -Preimage INSERT INTO employees (department, last_name, first_name, age, level) VALUES ('Production', 'Smith', 'John', 35, 2); department last_name first_name age leveldepartment last_name first_name age level Production Smith John 35 2 Production Smith John 35 2
  • 21.
    CDC Modes -Preimage UPDATE employees SET level = 3 WHERE department = 'Production' AND last_name = 'Smith' AND first_name = 'John'; department last_name first_name age level Production Smith John 35 2 Production Smith John 2 department last_name first_name age level Production Smith John 35 3 Production Smith John 3
  • 22.
    CDC Modes -Preimage DELETE age FROM employees WHERE department = 'Production' AND last_name = 'Smith' AND first_name = 'John'; department last_name first_name age level Production Smith John 35 3 Production Smith John 35 department last_name first_name age level Production Smith John 3 Production Smith John
  • 23.
    CDC Modes -Preimage DELETE FROM employees WHERE department = 'Production' AND last_name = 'Smith' AND first_name = 'John'; department last_name first_name age level Production Smith John 35 3 Production Smith John 35 3 Production Smith John
  • 24.
    CDC Modes -Postimage UPDATE employees SET level = 3 WHERE department = 'Production' AND last_name = 'Smith' AND first_name = 'John'; department last_name first_name age level Production Smith John 35 2 Production Smith John 35 3 department last_name first_name age level Production Smith John 35 3 Production Smith John 3
  • 25.
    CDC Modes -Postimage DELETE age FROM employees WHERE department = 'Production' AND last_name = 'Smith' AND first_name = 'John'; department last_name first_name age level Production Smith John 35 3 Production Smith John 3 department last_name first_name age level Production Smith John 3 Production Smith John
  • 26.
    CDC Modes -Postimage DELETE FROM employees WHERE department = 'Production' AND last_name = 'Smith' AND first_name = 'John'; department last_name first_name age level Production Smith John 35 3 Production Smith John Production Smith John
  • 27.
    CREATE TABLE company.employees( department text, last_name text, first_name text, age int, level int, PRIMARY KEY (department, last_name, first_name) ) WITH cdc = {/* CDC parameters go here */}; CDC Log CREATE TABLE company.employees_scylla_cdc_log ( cdc$stream_id blob, cdc$time timeuuid, cdc$batch_seq_no int, cdc$operation tinyint, cdc$ttl bigint, department text, first_name text, last_name text, age int, cdc$deleted_age boolean, level int, cdc$deleted_level boolean, PRIMARY KEY (cdc$stream_id, cdc$time, cdc$batch_seq_no) )
  • 28.
    CDC Log CREATE TABLEcompany.employees ( department text, last_name text, first_name text, age int, level int, PRIMARY KEY (department, last_name, first_name) ) WITH cdc = {/* CDC parameters go here */}; CREATE TABLE company.employees_scylla_cdc_log ( cdc$stream_id blob, cdc$time timeuuid, cdc$batch_seq_no int, cdc$operation tinyint, cdc$ttl bigint, department text, first_name text, last_name text, age int, cdc$deleted_age boolean, level int, cdc$deleted_level boolean, PRIMARY KEY (cdc$stream_id, cdc$time, cdc$batch_seq_no) )
  • 29.
    CDC Log CREATE TABLEcompany.employees ( department text, last_name text, first_name text, age int, level int, PRIMARY KEY (department, last_name, first_name) ) WITH cdc = {/* CDC parameters go here */}; CREATE TABLE company.employees_scylla_cdc_log ( cdc$stream_id blob, cdc$time timeuuid, cdc$batch_seq_no int, cdc$operation tinyint, cdc$ttl bigint, department text, first_name text, last_name text, age int, cdc$deleted_age boolean, level int, cdc$deleted_level boolean, PRIMARY KEY (cdc$stream_id, cdc$time, cdc$batch_seq_no) )
  • 30.
    CREATE TABLE company.employees( department text, last_name text, first_name text, age int, level int, PRIMARY KEY (department, last_name, first_name) ) WITH cdc = {/* CDC parameters go here */}; CREATE TABLE company.employees_scylla_cdc_log ( cdc$stream_id blob, cdc$time timeuuid, cdc$batch_seq_no int, cdc$operation tinyint, cdc$ttl bigint, department text, first_name text, last_name text, age int, cdc$deleted_age boolean, level int, cdc$deleted_level boolean, PRIMARY KEY (cdc$stream_id, cdc$time, cdc$batch_seq_no) ) CDC Log
  • 31.
    CDC Log CREATE TABLEcompany.employees_scylla_cdc_log ( cdc$stream_id blob, cdc$time timeuuid, cdc$batch_seq_no int, cdc$operation tinyint, cdc$ttl bigint, department text, first_name text, last_name text, age int, cdc$deleted_age boolean, level int, cdc$deleted_level boolean, PRIMARY KEY (cdc$stream_id, cdc$time, cdc$batch_seq_no) ) CREATE TABLE company.employees ( department text, last_name text, first_name text, age int, level int, PRIMARY KEY (department, last_name, first_name) ) WITH cdc = {/* CDC parameters go here */};
  • 32.
    CREATE TABLE company.employees_scylla_cdc_log( cdc$stream_id blob, cdc$time timeuuid, cdc$batch_seq_no int, cdc$operation tinyint, cdc$ttl bigint, department text, first_name text, last_name text, age int, cdc$deleted_age boolean, level int, cdc$deleted_level boolean, PRIMARY KEY (cdc$stream_id, cdc$time, cdc$batch_seq_no) ) CDC Log CREATE TABLE company.employees ( department text, last_name text, first_name text, age int, level int, PRIMARY KEY (department, last_name, first_name) ) WITH cdc = {/* CDC parameters go here */};
  • 33.
    CREATE TABLE company.employees_scylla_cdc_log( cdc$stream_id blob, cdc$time timeuuid, cdc$batch_seq_no int, cdc$operation tinyint, cdc$ttl bigint, department text, first_name text, last_name text, age int, cdc$deleted_age boolean, level int, cdc$deleted_level boolean, PRIMARY KEY (cdc$stream_id, cdc$time, cdc$batch_seq_no) ) CDC Log CREATE TABLE company.employees ( department text, last_name text, first_name text, age int, level int, PRIMARY KEY (department, last_name, first_name) ) WITH cdc = {/* CDC parameters go here */};
  • 34.
    CDC Log CREATE TABLEcompany.employees_scylla_cdc_log ( cdc$stream_id blob, cdc$time timeuuid, cdc$batch_seq_no int, cdc$operation tinyint, cdc$ttl bigint, department text, first_name text, last_name text, age int, cdc$deleted_age boolean, level int, cdc$deleted_level boolean, PRIMARY KEY (cdc$stream_id, cdc$time, cdc$batch_seq_no) ) CREATE TABLE company.employees ( department text, last_name text, first_name text, age int, level int, PRIMARY KEY (department, last_name, first_name) ) WITH cdc = {/* CDC parameters go here */};
  • 35.
    CREATE TABLE company.employees( department text, last_name text, first_name text, age int, level int, PRIMARY KEY (department, last_name, first_name) ) WITH cdc = {/* CDC parameters go here */}; CDC log CREATE TABLE company.employees_scylla_cdc_log ( cdc$stream_id blob, cdc$time timeuuid, cdc$batch_seq_no int, cdc$operation tinyint, cdc$ttl bigint, department text, first_name text, last_name text, age int, cdc$deleted_age boolean, level int, cdc$deleted_level boolean, PRIMARY KEY (cdc$stream_id, cdc$time, cdc$batch_seq_no) ) 0 PREIMAGE 1 UPDATE 2 INSERT 3 ROW DELETE 4 PARTITION DELETE 5 RANGE DELETE INCLUSIVE LOWER BOUND 6 RANGE DELETE EXCLUSIVE LOWER BOUND 7 RANGE DELETE INCLUSIVE UPPER BOUND 8 RANGE DELETE EXCLUSIVE UPPER BOUND 9 POSTIMAGE
  • 36.
    CREATE TABLE company.employees_scylla_cdc_log( cdc$stream_id blob, cdc$time timeuuid, cdc$batch_seq_no int, cdc$operation tinyint, cdc$ttl bigint, department text, first_name text, last_name text, age int, cdc$deleted_age boolean, level int, cdc$deleted_level boolean, PRIMARY KEY (cdc$stream_id, cdc$time, cdc$batch_seq_no) ) CREATE TABLE company.employees ( department text, last_name text, first_name text, age int, level int, PRIMARY KEY (department, last_name, first_name) ) WITH cdc = {/* CDC parameters go here */}; CDC log0 PREIMAGE 1 UPDATE 2 INSERT 3 ROW DELETE 4 PARTITION DELETE 5 RANGE DELETE INCLUSIVE LOWER BOUND 6 RANGE DELETE EXCLUSIVE LOWER BOUND 7 RANGE DELETE INCLUSIVE UPPER BOUND 8 RANGE DELETE EXCLUSIVE UPPER BOUND 9 POSTIMAGE UPDATE employees SET level = 3 WHERE department = 'Production' AND last_name = 'Smith' AND first_name = 'John';
  • 37.
    CREATE TABLE company.employees_scylla_cdc_log( cdc$stream_id blob, cdc$time timeuuid, cdc$batch_seq_no int, cdc$operation tinyint, cdc$ttl bigint, department text, first_name text, last_name text, age int, cdc$deleted_age boolean, level int, cdc$deleted_level boolean, PRIMARY KEY (cdc$stream_id, cdc$time, cdc$batch_seq_no) ) CREATE TABLE company.employees ( department text, last_name text, first_name text, age int, level int, PRIMARY KEY (department, last_name, first_name) ) WITH cdc = {/* CDC parameters go here */}; CDC log0 PREIMAGE 1 UPDATE 2 INSERT 3 ROW DELETE 4 PARTITION DELETE 5 RANGE DELETE INCLUSIVE LOWER BOUND 6 RANGE DELETE EXCLUSIVE LOWER BOUND 7 RANGE DELETE INCLUSIVE UPPER BOUND 8 RANGE DELETE EXCLUSIVE UPPER BOUND 9 POSTIMAGE INSERT INTO employees (department, last_name, first_name, age, level) VALUES ('Production', 'Smith', 'John', 35, 2);
  • 38.
    CREATE TABLE company.employees_scylla_cdc_log( cdc$stream_id blob, cdc$time timeuuid, cdc$batch_seq_no int, cdc$operation tinyint, cdc$ttl bigint, department text, first_name text, last_name text, age int, cdc$deleted_age boolean, level int, cdc$deleted_level boolean, PRIMARY KEY (cdc$stream_id, cdc$time, cdc$batch_seq_no) ) CREATE TABLE company.employees ( department text, last_name text, first_name text, age int, level int, PRIMARY KEY (department, last_name, first_name) ) WITH cdc = {/* CDC parameters go here */}; CDC log0 PREIMAGE 1 UPDATE 2 INSERT 3 ROW DELETE 4 PARTITION DELETE 5 RANGE DELETE INCLUSIVE LOWER BOUND 6 RANGE DELETE EXCLUSIVE LOWER BOUND 7 RANGE DELETE INCLUSIVE UPPER BOUND 8 RANGE DELETE EXCLUSIVE UPPER BOUND 9 POSTIMAGE DELETE FROM employees WHERE department = 'Production' AND last_name = 'Smith' AND first_name = 'John';
  • 39.
    CREATE TABLE company.employees_scylla_cdc_log( cdc$stream_id blob, cdc$time timeuuid, cdc$batch_seq_no int, cdc$operation tinyint, cdc$ttl bigint, department text, first_name text, last_name text, age int, cdc$deleted_age boolean, level int, cdc$deleted_level boolean, PRIMARY KEY (cdc$stream_id, cdc$time, cdc$batch_seq_no) ) CREATE TABLE company.employees ( department text, last_name text, first_name text, age int, level int, PRIMARY KEY (department, last_name, first_name) ) WITH cdc = {/* CDC parameters go here */}; CDC log0 PREIMAGE 1 UPDATE 2 INSERT 3 ROW DELETE 4 PARTITION DELETE 5 RANGE DELETE INCLUSIVE LOWER BOUND 6 RANGE DELETE EXCLUSIVE LOWER BOUND 7 RANGE DELETE INCLUSIVE UPPER BOUND 8 RANGE DELETE EXCLUSIVE UPPER BOUND 9 POSTIMAGE DELETE FROM employees WHERE department = 'Production';
  • 40.
    CREATE TABLE company.employees( department text, last_name text, first_name text, age int, level int, PRIMARY KEY (department, last_name, first_name) ) WITH cdc = {/* CDC parameters go here */}; CDC log0 PREIMAGE 1 UPDATE 2 INSERT 3 ROW DELETE 4 PARTITION DELETE 5 RANGE DELETE INCLUSIVE LOWER BOUND 6 RANGE DELETE EXCLUSIVE LOWER BOUND 7 RANGE DELETE INCLUSIVE UPPER BOUND 8 RANGE DELETE EXCLUSIVE UPPER BOUND 9 POSTIMAGE CREATE TABLE company.employees_scylla_cdc_log ( cdc$stream_id blob, cdc$time timeuuid, cdc$batch_seq_no int, cdc$operation tinyint, cdc$ttl bigint, department text, first_name text, last_name text, age int, cdc$deleted_age boolean, level int, cdc$deleted_level boolean, PRIMARY KEY (cdc$stream_id, cdc$time, cdc$batch_seq_no) ) DELETE FROM employees WHERE department = 'Production' AND last_name >= 'A' AND last_name < 'C';
  • 41.
    CDC Log CREATE TABLEcompany.employees_scylla_cdc_log ( cdc$stream_id blob, cdc$time timeuuid, cdc$batch_seq_no int, cdc$operation tinyint, cdc$ttl bigint, department text, first_name text, last_name text, age int, cdc$deleted_age boolean, level int, cdc$deleted_level boolean, PRIMARY KEY (cdc$stream_id, cdc$time, cdc$batch_seq_no) ) CREATE TABLE company.employees ( department text, last_name text, first_name text, age int, level int, PRIMARY KEY (department, last_name, first_name) ) WITH cdc = {/* CDC parameters go here */};
  • 42.
    scylla.yaml: experimental_features: - cdc Command line:--experimental CREATE TABLE company.employees ( ... ) WITH cdc = {'enabled' : true}; ALTER TABLE company.employees WITH cdc = {'enabled' : true}; WITH cdc = { 'enabled' : true, 'preimage' : true, 'postimage' : true, 'ttl' : 1000 }; How to Enable CDC
  • 43.
    INSERT INTO employees(department, last_name, first_name, level) VALUES ('Production', 'Smith', 'John', 2); cdc$stream_id 0x6f06e163e099df1b8545901cd23663e3 cdc$time 54b28d54-69cb-11ea-6541-6fa9f3f294ae cdc$batch_seq_no 0 cdc$operation 2 cdc$ttl null department Production last_name Smith first_name John age null level 2 cdc$deleted_age null cdc$deleted_level null
  • 44.
    UPDATE employees SETage = 35 WHERE department = 'Production' AND last_name = 'Smith' AND first_name = 'John'; cdc$stream_id 0x6f06e163e099df1b8545901cd23663e3 cdc$time b82e2cb0-69cd-11ea-6a10-8c39c8f7dc3a cdc$batch_seq_no 0 cdc$operation 1 cdc$ttl null department Production last_name Smith first_name John age 35 level null cdc$deleted_age null cdc$deleted_level null
  • 45.
    DELETE age FROMemployees WHERE department = 'Production' AND last_name = 'Smith' AND first_name = 'John'; cdc$stream_id 0x6f06e163e099df1b8545901cd23663e3 cdc$time c8f5b55e-69cd-11ea-68c1-e036079d1ce4 cdc$batch_seq_no 0 cdc$operation 1 cdc$ttl null department Production last_name Smith first_name John age null level null cdc$deleted_age true cdc$deleted_level null
  • 46.
    DELETE FROM employeesWHERE department = 'Production' AND last_name = 'Smith' AND first_name = 'John'; cdc$stream_id 0x6f06e163e099df1b8545901cd23663e3 cdc$time e75e6612-69cd-11ea-dd3e-2e150dc8859c cdc$batch_seq_no 0 cdc$operation 3 cdc$ttl null department Production last_name Smith first_name John age null level null cdc$deleted_age null cdc$deleted_level null
  • 47.
    DELETE FROM employeesWHERE department = 'Production'; cdc$stream_id 0x6f06e163e099df1b8545901cd23663e3 cdc$time 0a454bbe-69ce-11ea-31f2-98b4c3f08688 cdc$batch_seq_no 0 cdc$operation 4 cdc$ttl null department Production last_name null first_name null age null level null cdc$deleted_age null cdc$deleted_level null
  • 48.
    DELETE FROM employeesWHERE department = 'Production' AND last_name >= 'A' AND last_name < 'C'; cdc$stream_id 0x6f06e163e099df1b8545901cd23663e3 cdc$time 447dd18e-69ce-11ea-6ff4-81708ddd348a cdc$batch_seq_no 0 1 cdc$operation 5 8 cdc$ttl null null department Production Production last_name A C first_name null null age null null level null null cdc$deleted_age null null cdc$deleted_level null null
  • 49.
  • 50.
    CDC Streams CREATE TABLEcompany.employees_scylla_cdc_log ( cdc$stream_id blob, cdc$time timeuuid, cdc$batch_seq_no int, ... PRIMARY KEY (cdc$stream_id, cdc$time, cdc$batch_seq_no) ) CREATE TABLE company.employees ( ... PRIMARY KEY (department, last_name, first_name) ) WITH cdc = {/* CDC parameters go here */};
  • 51.
    Benchmark Results CDC disabledDelta Delta + Preimage Throughput (ops) 78,847 61,851 ( -21.55%) 33,358 ( -57.69%) Mean latency (ms) 1.3 1.6 (+23.07%) 3.0 (+130.76%) 99th percentile latency (ms) 3.6 4.7 (+30.55%) 9.3 (+158.33%)
  • 52.
  • 53.
    CDC in Scylla +Easy to integrate and consume + Plain CQL tables + Robust + Replicated in same way as normal data + Reasonable overhead + Coalesced writes to same replica ranges + Does not overflow if consumer fails to act + Data is TTL:ed
  • 54.
    Comparison Chart Cassandra DynamoDBMongoDB Scylla Consumer location on-node off-node off-node off-node Replication duplicated deduplicated deduplicated deduplicated Deltas yes no partial yes Pre-image no yes no optional Post-image no yes yes optional Slow consumer reaction Table stopped Consumer loses data Consumer loses data Consumer loses data Ordering no yes yes yes
  • 55.
    Try Scylla CDCNow! + Download Scylla Nightly Docker and use --experimental https://hub.docker.com/r/scylladb/scylla-nightly + Use and follow the docs: https://docs.scylladb.com/using-scylla/cdc/ 55
  • 56.
  • 57.
  • 58.
    United States 545 FaberPlace Palo Alto, CA 94303 Israel 11 Galgalei Haplada Herzelia, Israel www.scylladb.com @scylladb Thank you