A constant flow of fresh water will keep your lake alive and both fish and fishers happy. See how easy it is to feed changes from your operational databases directly to your Data Lake and/or Kafka to support real-/neartime analytics.

1. Fresh water for your data lake
Matthias Reiß
IBM – CTP Hybrid Data Management
matthias.reiss@de.ibm.com

2. Challenges in providing incremental data delivery
HDFS/Hive
2 © 2018 IBM Corporation
Cons:
• Bulk/Batch approach
• No change data capture
(append, lastmodified)
• Unable to capture physical delete
• Can consume significant resources
on the source
• Command line only
Pros:
• Open Source
• Included in Hadoop distribution
• Comprehensive RDBMS support
• Parallel offload from RDBMS

3. IBM Data Replication – WebHDFS or Kafka
3
Database Logs
(online/archive)
Capture Apply
TCP/IP
Transport
Central GUI
for Admin & Monitoring
Source
Application
Source Server CDC Target Server
Flatfiles
Avro JSON Binary
© 2018 IBM Corporation
HDFS/Hive
Pros:
• Realtime CDC
• DB Log based
• Constant Flow of data
• Very low resource
consumption
Cons:
• It‘s not for free ;-)

4. IBM Data Replication – Change Record Format
4
“When”
Timestamp of Change
Transaction Identifier
“What”
Type of change
I : Insert (After)
A : Update (After)
B : Update (Before)
D : Delete (Before)
“Who”
Who has changed the data
Complete Record
I/A : After image
B/D : Before image
2015-11-18 07:26:51,65886,I,CDCUSER ,576,90001,Alexis Bull,5000,…
2015-11-19 07:43:25,80623,A,CDCUSER ,302,33055,Harrison Bloom,50000,…
2015-11-19 07:43:25,80624,B,CDCUSER ,302,33055,Harrison Bloom,50000,…
2015-11-19 08:46:50,81575,D,CDCUSER ,790,93055,Martha Sullivan,25000,…
© 2018 IBM Corporation

5. Demo 1 – IBM Db2 -> HDFS/Hive/IBM Big SQL
5 © 2018 IBM Corporation
IDR
Capture
HDFS/Hive
IDR
Apply
IBM Data Replication
TCP/IP Transport
Source Table:
bikecdc.aggregated_daily_demographics
HDFS Path:
/apps/hive/warehouse/flatfile/aggregated_daily

6. Demo 2 – Oracle -> Kafka
6 © 2018 IBM Corporation
IDR
Capture
IDR
Apply
IBM Data Replication
TCP/IP Transport
Source Table:
cdc.aggregated_daily_demographics
Kafka Topic:
linux.orakafka.sourcedb.cdc.aggregated_daily
_demographics

7. Demo Screenshots
7 © 2018 IBM Corporation
.
.

8. Demo Screenshots
8 © 2018 IBM Corporation
kafka-avro-console-consumer --zookeeper localhost:2181 --topic linux.orakafka.sourcedb.cdc.aggregated_daily_demographics --property print.key=true
{"DDATE":{"string":"2018-02-28T00:00:00.000000000000"},"FROM_COMMUNITY":{"string":"NORTH PARK"}} {"DDATE":{"string":"2018-02-
28T00:00:00.000000000000"},"FROM_COMMUNITY":{"string":"NORTH
PARK"},"NUM_RIDES":{"string":"0"},"MAXIMUM_TEMPERATURE":{"string":"56.20"},"MINIMUM_TEMPERATURE":{"string":"41.70"},"AVERAGE_TEMPERATURE":{"string":
"48.95"},"TEMPERATURE_RANGE":{"string":"14.50"},"TOTAL_PRECIPITATION":{"string":"0.09"},"TOTAL_SNOW":{"string":"0.00"},"AVERAGE_HUMIDITY":{"string":
"81"},"MAXIMUM_WIND_SPEED":{"string":"27"},"PERCENT_CUSTOMERS":null,"PERCENT_SUBSCRIBERS":null,"PERCENT_DEPENDENTS":null,"PERCENT_MALES":null,"PERCE
NT_FEMALES":null,"AVERAGE_AGE":null,"DAY_OF_WEEK":{"string":"4"},"WEEKEND":{"string":"0"},"MONTH":{"string":"2"},"YEAR":{"string":"2018"},"CHGTIME":
"2018-05-29T19:54:47.000000000000","CCID":"36580232","TYPE":"PT","USERNAME":"CDC"}
{"DDATE":{"string":"2018-02-28T00:00:00.000000000000"},"FROM_COMMUNITY":{"string":"PORTAGE PARK"}} {"DDATE":{"string":"2018-02-
28T00:00:00.000000000000"},"FROM_COMMUNITY":{"string":"PORTAGE
PARK"},"NUM_RIDES":{"string":"0"},"MAXIMUM_TEMPERATURE":{"string":"56.20"},"MINIMUM_TEMPERATURE":{"string":"41.70"},"AVERAGE_TEMPERATURE":{"string":
"48.95"},"TEMPERATURE_RANGE":{"string":"14.50"},"TOTAL_PRECIPITATION":{"string":"0.09"},"TOTAL_SNOW":{"string":"0.00"},"AVERAGE_HUMIDITY":{"string":
"81"},"MAXIMUM_WIND_SPEED":{"string":"27"},"PERCENT_CUSTOMERS":null,"PERCENT_SUBSCRIBERS":null,"PERCENT_DEPENDENTS":null,"PERCENT_MALES":null,"PERCE
NT_FEMALES":null,"AVERAGE_AGE":null,"DAY_OF_WEEK":{"string":"4"},"WEEKEND":{"string":"0"},"MONTH":{"string":"2"},"YEAR":{"string":"2018"},"CHGTIME":
"2018-05-29T19:54:47.000000000000","CCID":"36580232","TYPE":"PT","USERNAME":"CDC"}

9. © 2018 IBM Corporation
9
Fresh Water for your datalake
Ressources
• IBM Data Replication Community https://ibm.biz/BdZk4U
• IBM Data Replication Hadoop https://ibm.biz/BdZk45
• IBM Data Replication Kafka https://ibm.biz/BdZk4N

10. 10
THANK YOU