Real-Time Data Replication to Hadoop using GoldenGate 12c Adaptors

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Real-Time Data Replication to Hadoop using
GoldenGate 12c Adaptors
Michael Rainey
Principal Consultant - Oracle ACE
Rittman Mead
February 11, 2015
Please Stand By. This session will begin promptly at the time
indicated on the agenda. Thank You.

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied
upon in making purchasing decisions. The development, release, and timing of any
features or functionality described for Oracle’s products remains at the sole discretion of
Oracle.
Oracle Confidential – Internal/Restricted/Highly Restricted 2

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or
+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : info@rittmanmead.com
W : www.rittmanmead.com
Introduction
•Michael Rainey – Principal Consultant – Rittman Mead
‣Oracle Data Integration expert and Oracle ACE
‣GoldenGate and Oracle Data Integrator
•Rittman Mead
‣Provide consulting, training, and managed
services worldwide
‣Focus on business intelligence, data
integration, and advanced analytics
‣Rittman Mead India recently named Oracle
Analytics Partner of the Year
@mRainey

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or
Agenda
•Why Oracle Data Integration for Big Data?
•Review the technologies
‣Oracle GoldenGate 12c and GoldenGate 12c Adapters
‣Hadoop, HDFS, and Hive
‣Sqoop and Flume
•Big Data Lite VM introduction
•Demonstrations
‣Initial load from MySQL to Hadoop
‣Real-time replication using GoldenGate 12c direct to
Hadoop
‣Real-time replication using GoldenGate 12c and Flume

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or
Oracle Data Integration 12c
•High performance data integration
•Real-time data replication
•End-to-end integrated with simplified
deployment
•Unified tooling for both structured data
sources and Hadoop / NoSQL
•Flexible deployment on-premise or in
the cloud for heterogeneous systems
•Data governance and full metadata
management
Real-time data integration, data management for Cloud and Big Data
Big Data
Cloud
Apps
Database
Oracle Data IntegratorOracle Data IntegratorOracle Data IntegratorOracle Data Integrator
Oracle GoldenGateOracle GoldenGateOracle GoldenGateOracle GoldenGate
Oracle Enterprise DataOracle Enterprise Data
QualityQuality
Oracle Enterprise DataOracle Enterprise Data
QualityQuality
Oracle Data ServicesOracle Data Services
IntegratorIntegrator
Oracle Data ServicesOracle Data Services
IntegratorIntegrator
Oracle EnterpriseOracle Enterprise
Metadata ManagementMetadata Management
Oracle EnterpriseOracle Enterprise
Metadata ManagementMetadata Management

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or
Oracle Data Integration with Big Data
•Big Data Adapters
‣Natively connect to Hadoop
‣Produce native code to execute
on big data source or target
•Utilize Oracle Data Integration
capabilities
‣“Design once, run anywhere”
‣High performance replication,
heterogeneous sources/targets
data quality checks, etc.
‣Easy to extend and customize

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or
Oracle GoldenGate 12c

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or
Hadoop, HDFS and Hive
•Hadoop is the framework for storing large amounts of data and processing it in an efficient
and faster manner
‣Storage: Hadoop Distributed File System (HDFS)
‣Processing: MapReduce
•HDFS – stores data as large files across multiple systems, using redundancy for reliability
‣Master/slave architecture – namenodes and datanodes
‣Data replicated to multiple datanodes, with namenode tracking location
•Hive – data warehouse infrastructure for analysis of large datasets in HDFS
‣HiveQL – SQL-like language for querying data
‣Transparently converts queries to MapReduce

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or
Loading Relational Data with Sqoop
•Sqoop - short for “SQL to Hadoop”
‣Import whole tables, or whole schemas, from relational databases into HDFS
‣Export data from HDFS back out to these databases – with the export and import being
performed through MapReduce jobs
‣Import using a SQL SELECT statement, rather than grabbing whole tables
‣Incremental loads, specifying a key column to determine what to exclude
‣Load directly into Hive tables, creating HDFS files in the background and the Hive
metadata automatically
•Sqoop User Guide: http://archive.cloudera.com/cdh4/cdh/4/sqoop/SqoopUserGuide.html

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or
Moving Data with Flume
•Reliable, distributed approach to collecting, aggregating, & moving large amounts of log data
•Installed as Java agents that run on source and target
‣Source – listens for and consumes incoming events (transactions)
‣Channel – where events are queued and staged
‣Sink – processes that write transactions to disk
•Transactions can be distributed to multiple targets or fed from many sources to one target

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or
Oracle GoldenGate 12c Application Adapters
•GoldenGate Hive Adapter
‣Integrating OGG Adapter with
Hive (Doc ID 1586188.1)
•GoldenGate Flume Adapter
‣Integrating OGG Adapter with
Flume (Doc ID 1926867.1)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or
Big Data Lite VM and Hands-On Labs
•Big Data Lite 4.0 VM
‣Fully integrated Oracle Big Data environment
‣Similar to Big Data Appliance setup
‣Technologies include ODI, GoldenGate, Big Data
Connectors, Hadoop, Flume, Sqoop, Hive, etc.
•Hands-on labs provide step-by-step instructions
‣Demo environment built-in to Big Data Lite VM
•Big Data Lite VM:
http://www.oracle.com/technetwork/database/
bigdata-appliance/oracle-bigdatalite-2104726.html
•Hands-on Labs:
http://www.oracle.com/webfolder/technetwork/tutorials/obe/fmw/odi/odi_12c/DI_BDL_Guide/
BigDataIntegration_Demo.html

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or
Getting Started with the Big Data Lite VM
•Scenario: Oracle MoviePlex is an online
movie streaming company with web logs
and MySQL database sources
•Goal: Move data into Hadoop, perform
integration, then distribute to Oracle Exadata
data warehouse for further processing

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or
Tame Big Data with Oracle Data Integration: Part 1
•Real-time data replication from MySQL
relational database to Hadoop
‣Initial load using Sqoop and Oracle Data
Integrator 12c
‣Change Data Capture using GoldenGate
12c and GoldenGate Adapters
(Hive and Flume)

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or
Initial Load using Sqoop and ODI 12c
•Mapping between source MySQL table “MOVIE” and target Hive table “movie”
‣Data server connections setup for each technology
‣Tables reverse engineered as Datastores in ODI Models
•Mapping uses “IKM SQL to Hive-HBase-File (SQOOP)” Knowledge Module to load the Hive
table via Sqoop
‣Creates Sqoop option file and launches Sqoop import
‣Optional loads into Hive staging table
‣Loads rows into target table using HiveQL insert overwrite

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or
Demonstration
Initial Load via Sqoop

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or
Real-time Load using Oracle GoldenGate 12c to Hive
•Adapter developed using Oracle GoldenGate’s Java API and Hadoop HDFS Java API
‣Writes trail data to target Hive tables
‣Example “SampleHandler.java” found on My Oracle Support
•Properties file must be in same directory as parameter files
‣Contains information about necessary JAR files, target Hive tables, logging parameters, etc
•Another example:
‣http://www.rittmanmead.com/2014/09/using-oracle-goldengate-for-trickle-feeding-rdbms-
transactions-into-hive-and-hdfs/

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or
Demonstration
Load MySQL to HDFS using
GoldenGate 12c

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or
Another Real-time Load Option – using GoldenGate and Flume
•Adapter developed using Oracle GoldenGate’s Java API and Apache Flume Client API
‣Transactions delivered to RpcClient API
‣Example “SampleHandlerFlume.java” found on My Oracle Support
•Flume configuration file (flume.conf)
‣Flume agent listens for RPC calls on the source, stages transactions in memory, and
delivers data to HDFS
‣Flume agent must be started
•Properties file must be in same directory as parameter files

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or
Demonstration
Load MySQL to HDFS using
GoldenGate 12c and Flume

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or
Summary
•Oracle Data Integration and Big Data are a great match
‣GoldenGate 12c Adapters
‣ODI 12c Big Data Connectors
‣“Design once, run anywhere” fits for Big Data sources and targets
•Big Data Lite VM and Hands On labs
‣Get started with Oracle Data Integration and Big Data
‣Learn by example
•GoldenGate 12c can load Hadoop effectively
‣Java API examples on MOS are great for getting started
•Further information:
‣http://www.rittmanmead.com/blog
‣https://blogs.oracle.com/dataintegration

Keep Learning with Training from Oracle University
Oracle Confidential – Internal/Restricted/Highly Restricted22
• Hands-on training delivered in-class or online by
tenured instructors around the world
• New subscription-based learning services to give you
any-time access to training
• Certification programs to validate your skills
education.oracle.com

Oracle Confidential – Internal/Restricted/Highly Restri25

Real-Time Data Replication to Hadoop using GoldenGate 12c Adaptors

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Real-Time Data Replication to Hadoop using GoldenGate 12c Adaptors

Similar to Real-Time Data Replication to Hadoop using GoldenGate 12c Adaptors (20)

More from Michael Rainey

More from Michael Rainey (11)

Recently uploaded

Recently uploaded (20)

Real-Time Data Replication to Hadoop using GoldenGate 12c Adaptors

Editor's Notes