What is Change Data Capture (CDC) and Why is it Important?
•Download as PPTX, PDF•
3 likes•799 views
Check out what Change Data Capture (CDC) is and why it is becoming ever more important. Slides also include useful tips on how to design your CDC implementation.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to What is Change Data Capture (CDC) and Why is it Important?
Similar to What is Change Data Capture (CDC) and Why is it Important? (20)
More from FlyData Inc.
More from FlyData Inc. (11)
Recently uploaded
Recently uploaded (20)
What is Change Data Capture (CDC) and Why is it Important?
- 1. www.flydata.com What Change Data Capture (CDC) is and Why It’s Important
- 2. What is CDC? Copyright © 2015 FlyData Inc. All rights reserved. CDC captures just the change data so that replication can be done more efficiently. www.flydata.com
- 3. www.flydata.com Why do we need CDC? • Increase in data collected • Queries take longer because database size is growing Need for BI Databases (Data Warehousing) & Need for Data Replication growing Copyright © 2015 FlyData Inc. All rights reserved.
- 4. www.flydata.com Methods for Syncing Data ▪ Dumping the whole DB ▪ Change Data Capture (CDC) ▪ Other approaches Copyright © 2015 FlyData Inc. All rights reserved.
- 5. www.flydata.com Benefits of CDC ▪ No need for full export/import ▪ Near real-time replication is possible => data always available for queries Copyright © 2015 FlyData Inc. All rights reserved.
- 6. www.flydata.com Steps for CDC 1. Capture Change Data 2. Transform data to format supported on destination 3. Upload to destination 4. (Handle any errors as they occur) Copyright © 2015 FlyData Inc. All rights reserved.
- 7. www.flydata.com 1. Methods for Change Data Capture • Transaction Logs • Triggers • Custom Solution at Application Layer Copyright © 2015 FlyData Inc. All rights reserved.
- 8. www.flydata.com 1. Methods for Capturing Change Data Method Pros Cons Transaction Logs • Minimal performance impact • No change to tables or application necessary • Transaction logs are in binary format. • Lacking good documentation Triggers • Handle everything at the SQL level • Significant performance impact to the master DB • Need to define triggers on each table Custom Solutions • If INSERTs only, relatively easy to implement • With UPDATEs and DELETEs, keeping data integrity becomes much harder. Copyright © 2015 FlyData Inc. All rights reserved.
- 9. www.flydata.com 2. Transforming Change Data Destination DB may have certain data type restrictions: • Identify such differences in data type • Convert the data Example of differences between MySQL and Amazon Redshift: • Case sensitive table/column names: MySQL - yes, Redshift - no • TIME column value beyond 23:59:59: MySQL - yes, Redshift - no • VARCHAR length: MySQL - character length, Redshift - byte length • Date ‘0000-00-00’: MySQL - allowed, Redshift - not allowed • BINARY, VARBINARY: MySQL - supported, Redshift - not supported • ENUM, SET: MySQL - supported, Redshift - not supported Copyright © 2015 FlyData Inc. All rights reserved.
- 10. www.flydata.com 3. Upload to Destination • Uploading data to BI databases could be tricky. • Error Handling Crucial for CDC data integrity: • Recognize when errors occur • Buffer change data after the error (order important!) • Recognize when error is handled • Resume uploading • Do this in parallel to scale Copyright © 2015 FlyData Inc. All rights reserved.
- 11. www.flydata.com www.flydata.com Check us out! -> http://flydata.com sales@flydata.com Toll Free: 1-855-427-9787 http://flydata.com We are an official data integration partner of Amazon Redshift