0
©Continuent 2014
Replicate from Oracle to Oracle,
Oracle to MySQL
and Oracle to Analytics
MC Brown, Senior Information Arc...
©Continuent 2014
Introducing Continuent
2
• The leading provider of clustering and
replication for open source DBMS
• Our ...
©Continuent 2014
Quick Continuent Facts
• Largest Tungsten installation processes over
700 million transactions daily on 2...
©Continuent 2014©Continuent 2014
Continuent Tungsten Customers
4
1
©Continuent 2014
Tungsten Replicator
5
Tungsten Replicator is a fast,
open source, database
replication engine
!
Designed ...
©Continuent 2014
Tungsten Master/Slave in Action
6
Master
(Transactions + Metadata)
Slave
THL
DBMS	

Logs
Replicator
(Tran...
©Continuent 2014
Master Replication Service
7
Extract Filter Apply
Stage
Extract Filter Apply
Stage
Pipeline
MySQL	

Maste...
©Continuent 2014
Slave Replication Service
8
Extract Filter Apply
Stage
Extract Filter Apply
Stage
Extract Filter Apply
St...
©Continuent 2014
Multiple Services per Replicator
Service
frommysql
Service
fromoracle
frommysql
fromoracle
Replicator
Rep...
©Continuent 2014
star
master-slave Heterogeneous
fan-in slave all-masters
MySQL
Oracle
Oracle
MySQL Oracle
Oracle
MySQL My...
©Continuent 2014
Replicating from
Oracle to Oracle
©Continuent 2014©Continuent 2013
Steps to Homogeneous Replication
1. Restore backup on the slave
2. Set up replication
3. ...
©Continuent 2014
Preparing CDC
13
Tables
Source schema
setupCDC
Oracle CDC
(Synchronous or
Async Hotlog)
Publisher schema
...
©Continuent 2014
CDC in Action
14
Source table X
Source schema
Oracle CDC
(Synchronous or
Async Hotlog)
Publisher schema
C...
©Continuent 2014©Continuent 2013
Installation
15
./current/tools/tpm configure fromora 
--enable-heterogenous-service=true...
©Continuent 2014
Deployment
16
Service
fromora
Tungsten Master
Replicator
Service
fromora
Tungsten Slave
Replicator
demo	
...
©Continuent 2014©Continuent 2013
Heterogeneous
Replication
©Continuent 2014
Use Case: Web Content Publishing
Real-Time Publication
Backend Office Web-Based Catalog
©Continuent 2014©Continuent 2013
Steps to Heterogeneous Replication
1. Prepare (translate) schema for
the slave DBMS
2. Se...
©Continuent 2014©Continuent 2013
(Open Source)
©Continuent 2014©Continuent 2013
1.Translating schema
for the slave
©Continuent 2014©Continuent 2013
Translating Schema
22
• Beginning - how to convert tables?
empty
Tables
•Data types?	

•C...
©Continuent 2014©Continuent 2013
ddlscan
23
• Part of Tungsten Replicator, GPL v2
• Translates schema with replication in ...
©Continuent 2014©Continuent 2013
Usage (Oracle to MySQL Example)
24
$ cd tungsten-replicator/bin
!
$ ./ddlscan 
-db DEMO 
...
©Continuent 2014©Continuent 2013
Translating Schema
25
• ddlscan looks into source schema
Tables
empty
©Continuent 2014©Continuent 2013
Translating Schema
26
• ddlscan translates and renders DDL commands
mysql-ddl.sql
Tables
...
©Continuent 2014©Continuent 2013
Result of ddlscan
27
DROP TABLE IF EXISTS demo.test;
CREATE TABLE demo.test
(
id1 INT /* ...
©Continuent 2014©Continuent 2013
Translating Schema
28
• You run resulting SQL file on Oracle
mysql-ddl.sql
Tables
empty
©Continuent 2014©Continuent 2013
Translating Schema
29
• Tables are ready!
Translated tables

(no rows)
Tables
©Continuent 2014
Preparing CDC
30
Tables
Source schema
setupCDC
Oracle CDC
(Synchronous or
Async Hotlog)
Publisher schema
...
©Continuent 2014©Continuent 2013
Connecting the Dots
31
• Preparation completed
Translated tables

(no rows)
Tables Change...
©Continuent 2014©Continuent 2013
2. Set Up Replication
©Continuent 2014©Continuent 2013 33
./current/tools/tpm configure fromora 
--enable-heterogenous-service=true 
--user=orac...
©Continuent 2014©Continuent 2013
MySQL Slave Part
34
!
!
./current/tools/tpm configure fromora --hosts=beta 
--user=tungst...
©Continuent 2014
Deployment
35
Service
fromoracle
Tungsten Master
Replicator
OracleCDCExtractor
No Special Filters
Service...
©Continuent 2014©Continuent 2013
3. Provision &
Replication
©Continuent 2014©Continuent 2013
Parallel Extractor
37
• trepctl online -provision <setupCDC_SCN>
Tables
Service
fromoracl...
©Continuent 2014©Continuent 2013
Real-time Replication
38
• Automatically switches to real-time extraction
Service
fromora...
©Continuent 2014
Accelerate the Slave Provision
39
Extract Filter Apply
Stage
Extract Filter Apply
Stage
Stage
Pipeline
Re...
©Continuent 2014
Replication to
Vertica and
Hadoop
©Continuent 2014
The Data Warehouse Impedance
Mismatch
41
Replication
CSV	

Files
CSV	

Files
Buffered
Transactions
Dump/l...
©Continuent 2014
Column Store--Real-Time Batches
MySQL/Oracle Tungsten Master
Replicator
Service ora2vr
Special Filters	

...
©Continuent 2014
Batch Loading--The Gory Details
Replicator
Service ora2vr
Transactions
from master
CSV	

Files
CSV	

File...
©Continuent 2014
Basic Hadoop Loading
MySQL/Oracle Tungsten Master
Replicator
hadoop
Master-Side Filtering	

* pkey - Fill...
©Continuent 2012
Provisioning plus Replication
45
MySQL/Oracle
Tungsten 3.0 Master
hadoop
Tungsten 3.0 Slave
hadoop
CSV	

...
©Continuent 2014
Materialized Views
Transaction logs Snapshot
UNION ALL
Emit last row per key if not a delete
MAP
REDUCE
M...
©Continuent 2014
Tungsten Replicator 3.0 & Hadoop
47
• Extract from MySQL or Oracle
• Base Hadoop plus commercial distribu...
©Continuent 2014
Getting Started!
48
• Tungsten Replicator builds are available on
code.google.com
http://code.google.com/...
©Continuent 2014
We’re Hiring!
Continuent Web Page:	

http://www.continuent.com	

!
Tungsten Replicator:	

http://code.goo...
Upcoming SlideShare
Loading in...5
×

Set Up & Operate Open Source Oracle Replication

408

Published on

Oracle's expensive and complex replication makes it difficult to build cost-effective applications that move data in real-time to data warehouses (Oracle, Hadoop, Vertica) and popular databases like MySQL. Fortunately, Continuent Tungsten offers a solution. In this virtual course, you will learn how Continuent Tungsten solves problems with Oracle replication at a fraction of the cost of other solutions and with less management overhead too – think "Oracle GoldenGate without the price tag!”

Course Topics:
- Replicator installation and configuration
- Provisioning existing schema and tables from Oracle into another server
- Replicating between Oracle schemas
- Replicating from Oracle to MySQL and data warehouses
- Tips for solving common problems and improving performance.

The course includes live demos to illustrate operation. By the end of the course you should be ready move ahead with your own deployments.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
408
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
22
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Set Up & Operate Open Source Oracle Replication"

  1. 1. ©Continuent 2014 Replicate from Oracle to Oracle, Oracle to MySQL and Oracle to Analytics MC Brown, Senior Information Architect Linas Virbalas, Senior Software Engineer
  2. 2. ©Continuent 2014 Introducing Continuent 2 • The leading provider of clustering and replication for open source DBMS • Our Product: Continuent Tungsten • Clustering - Commercial-grade HA, performance scaling and data management for MySQL • Replication - Flexible, high-performance data movement
  3. 3. ©Continuent 2014 Quick Continuent Facts • Largest Tungsten installation processes over 700 million transactions daily on 225 terabytes of data • Tungsten Replicator was application of the year at the 2011 MySQL User Conference • Wide variety of topologies including MySQL, Oracle, Vertica, and MongoDB are in production now • MySQL to Hadoop deployments are now in progress with multiple customers 3
  4. 4. ©Continuent 2014©Continuent 2014 Continuent Tungsten Customers 4 1
  5. 5. ©Continuent 2014 Tungsten Replicator 5 Tungsten Replicator is a fast, open source, database replication engine ! Designed for speed and flexibility GPLV2 license 100% open source Annual support subscription available from Continuent
  6. 6. ©Continuent 2014 Tungsten Master/Slave in Action 6 Master (Transactions + Metadata) Slave THL DBMS Logs Replicator (Transactions + Metadata) THLReplicator Download transactions via network Apply using JDBC
  7. 7. ©Continuent 2014 Master Replication Service 7 Extract Filter Apply Stage Extract Filter Apply Stage Pipeline MySQL Master Transaction History Log In-Memory Queue Slave ReplicatorsBinlog tcp/ip
  8. 8. ©Continuent 2014 Slave Replication Service 8 Extract Filter Apply Stage Extract Filter Apply Stage Extract Filter Apply Stage Pipeline Transaction History Log In-Memory Queue Slave DBMS Master Replicator tcp/ip
  9. 9. ©Continuent 2014 Multiple Services per Replicator Service frommysql Service fromoracle frommysql fromoracle Replicator Replicator Replicator Aggregated
  10. 10. ©Continuent 2014 star master-slave Heterogeneous fan-in slave all-masters MySQL Oracle Oracle MySQL Oracle Oracle MySQL MySQL
  11. 11. ©Continuent 2014 Replicating from Oracle to Oracle
  12. 12. ©Continuent 2014©Continuent 2013 Steps to Homogeneous Replication 1. Restore backup on the slave 2. Set up replication 3. Continue real-time replication Tungsten Replicator tpmsetupCDC &
  13. 13. ©Continuent 2014 Preparing CDC 13 Tables Source schema setupCDC Oracle CDC (Synchronous or Async Hotlog) Publisher schema Change tables Capture started at <setupCDC_SCN>
  14. 14. ©Continuent 2014 CDC in Action 14 Source table X Source schema Oracle CDC (Synchronous or Async Hotlog) Publisher schema Change table X_CT INSERT UPDATE DELETEx SCN++
  15. 15. ©Continuent 2014©Continuent 2013 Installation 15 ./current/tools/tpm configure fromora --enable-heterogenous-service=true --user=oracle --install-directory=/opt/fromora/continuent --members=alhpa,beta --master=alpha ! ./current/tools/tpm configure fromora --hosts=alpha --datasource-type=oracle --user=oracle --datasource-oracle-service=ORCL --replication-user=DEMO_PUB --replication-password=DEMO_PUB --svc-table-engine=CDCASYNC --property=replicator.global.extract.db.user=tungsten --property=replicator.global.extract.db.password=secret --property=replicator.extractor.parallel-extractor. ChunkDefinitionFile=/opt/fromora/chunks.csv ! ./tools/tpm configure fromora --hosts=beta --datasource-type=oracle --datasource-oracle-service=ORCL --replication-user=DEMO --replication-password=DEMO
  16. 16. ©Continuent 2014 Deployment 16 Service fromora Tungsten Master Replicator Service fromora Tungsten Slave Replicator demo schema demo_pub schema Oracle CDC (Synchronous or Async Hotlog) Tungsten Slave Replicator OracleApplierOracleCDCExtractor
  17. 17. ©Continuent 2014©Continuent 2013 Heterogeneous Replication
  18. 18. ©Continuent 2014 Use Case: Web Content Publishing Real-Time Publication Backend Office Web-Based Catalog
  19. 19. ©Continuent 2014©Continuent 2013 Steps to Heterogeneous Replication 1. Prepare (translate) schema for the slave DBMS 2. Set up replication 3. Provision initial data
 
 4. Continue real-time replication ddlscan Tungsten Replicator Parallel ApplyParallel Extract & tpmsetupCDC &
  20. 20. ©Continuent 2014©Continuent 2013 (Open Source)
  21. 21. ©Continuent 2014©Continuent 2013 1.Translating schema for the slave
  22. 22. ©Continuent 2014©Continuent 2013 Translating Schema 22 • Beginning - how to convert tables? empty Tables •Data types? •Column lengths? •Naming conventions? •Reserved words? ddlscan
  23. 23. ©Continuent 2014©Continuent 2013 ddlscan 23 • Part of Tungsten Replicator, GPL v2 • Translates schema with replication in mind • Provides errors and warnings • Can rename schema/tables/columns
  24. 24. ©Continuent 2014©Continuent 2013 Usage (Oracle to MySQL Example) 24 $ cd tungsten-replicator/bin ! $ ./ddlscan -db DEMO -template ddl-oracle-mysql.vm
  25. 25. ©Continuent 2014©Continuent 2013 Translating Schema 25 • ddlscan looks into source schema Tables empty
  26. 26. ©Continuent 2014©Continuent 2013 Translating Schema 26 • ddlscan translates and renders DDL commands mysql-ddl.sql Tables empty
  27. 27. ©Continuent 2014©Continuent 2013 Result of ddlscan 27 DROP TABLE IF EXISTS demo.test; CREATE TABLE demo.test ( id1 INT /* NUMBER(10, ?) */ NOT NULL, id2 INT /* NUMBER(10, ?) */ NOT NULL, val TINYINT /* NUMBER(3, ?) */, owner VARCHAR(30) /* VARCHAR2(30) */ NOT NULL, created DATETIME /* DATE */ NOT NULL, PRIMARY KEY (id1, id2) /* WARN: no PK found, using suitable unique index instead: UQ_UK1 */ ) ENGINE=InnoDB; ! CREATE TABLE talks ...
  28. 28. ©Continuent 2014©Continuent 2013 Translating Schema 28 • You run resulting SQL file on Oracle mysql-ddl.sql Tables empty
  29. 29. ©Continuent 2014©Continuent 2013 Translating Schema 29 • Tables are ready! Translated tables
 (no rows) Tables
  30. 30. ©Continuent 2014 Preparing CDC 30 Tables Source schema setupCDC Oracle CDC (Synchronous or Async Hotlog) Publisher schema Change tables Capture started at <setupCDC_SCN> SCN++
  31. 31. ©Continuent 2014©Continuent 2013 Connecting the Dots 31 • Preparation completed Translated tables
 (no rows) Tables Change tables …
  32. 32. ©Continuent 2014©Continuent 2013 2. Set Up Replication
  33. 33. ©Continuent 2014©Continuent 2013 33 ./current/tools/tpm configure fromora --enable-heterogenous-service=true --user=oracle --install-directory=/opt/fromora/continuent --members=alhpa,beta --master=alpha ! ./current/tools/tpm configure fromora --hosts=alpha --datasource-type=oracle --user=oracle --datasource-oracle-service=ORCL --replication-user=DEMO_PUB --replication-password=DEMO_PUB --svc-table-engine=CDCASYNC --property=replicator.global.extract.db.user=tungsten --property=replicator.global.extract.db.password=secret --property=replicator.extractor.parallel-extractor. ChunkDefinitionFile=/opt/fromora/chunks.csv Oracle Master Part
  34. 34. ©Continuent 2014©Continuent 2013 MySQL Slave Part 34 ! ! ./current/tools/tpm configure fromora --hosts=beta --user=tungsten --replication-user=tungsten --replication-password=secret --svc-applier-filters=CDC,casetransform,rename --property=replicator.filter.CDC.from=DEMO_PUB.HEARTBEAT --property=replicator.filter.CDC.to=tungsten_fromora.heartbeat --property=replicator.filter.casetransform.to_upper_case=false --property=replicator.filter.rename.definitionsFile= /opt/frommysql/rename.csv
  35. 35. ©Continuent 2014 Deployment 35 Service fromoracle Tungsten Master Replicator OracleCDCExtractor No Special Filters Service fromoracle Tungsten Slave Replicator Special Filters •Map names to lower case •Ignore extra tables •Heartbeat table renaming demo schema demo_pub schema Oracle CDC (Synchronous or Async Hotlog) MySQLApplierOracleCDCExtractor
  36. 36. ©Continuent 2014©Continuent 2013 3. Provision & Replication
  37. 37. ©Continuent 2014©Continuent 2013 Parallel Extractor 37 • trepctl online -provision <setupCDC_SCN> Tables Service fromoracle Tungsten Master Replicator THL ParallelExtractor Thread 1 Thread 2 Thread 3 SELECT … AS OF <setupCDC_SCN>
  38. 38. ©Continuent 2014©Continuent 2013 Real-time Replication 38 • Automatically switches to real-time extraction Service fromoracle Tungsten Master Replicator THL Oracle CDC (Synchronous or Async Hotlog) Change tables >=$setupCDC_SCN OracleCDCExtractor Tables
  39. 39. ©Continuent 2014 Accelerate the Slave Provision 39 Extract Filter Apply Stage Extract Filter Apply Stage Stage Pipeline Remote Master Transaction History Log Parallel Queue Slave DBMS Extract Filter Apply Extract Filter Apply Extract Filter Apply (Assign Shard ID)
  40. 40. ©Continuent 2014 Replication to Vertica and Hadoop
  41. 41. ©Continuent 2014 The Data Warehouse Impedance Mismatch 41 Replication CSV Files CSV Files Buffered Transactions Dump/load Single Transactions Batches
  42. 42. ©Continuent 2014 Column Store--Real-Time Batches MySQL/Oracle Tungsten Master Replicator Service ora2vr Special Filters * pkey - Fill in pkey info * colnames - Fill in names * replicate - Ignore tables Tungsten Slave Replicator Service ora2vr CSV Files CSV Files CSV Files CSV Files CSV Files Large transaction batches to leverage load parallelization
  43. 43. ©Continuent 2014 Batch Loading--The Gory Details Replicator Service ora2vr Transactions from master CSV Files CSV Files CSV Files Staging Tables Staging Tables Staging Tables Base Tables Base Tables Base Tables Merge Script (or) COPY directly to base tables COPY to stage tables SELECT to base tables
  44. 44. ©Continuent 2014 Basic Hadoop Loading MySQL/Oracle Tungsten Master Replicator hadoop Master-Side Filtering * pkey - Fill in pkey info * colnames - Fill in names * replicate - Subset tables to be replicated Tungsten Slave Replicator hadoop CSV Files CSV Files CSV Files CSV Files CSV Files Hadoop Cluster Extract from source DBMS Load raw CSV to HDFS (e.g., via LOAD DATA to Hive) Access via Hive
  45. 45. ©Continuent 2012 Provisioning plus Replication 45 MySQL/Oracle Tungsten 3.0 Master hadoop Tungsten 3.0 Slave hadoop CSV Files CSV Files CSV Files CSV FilesCSV Apache Sqoop/ETL Fast data filtering Buffered CSV Programmable load scripts Parallel apply Parallel table dumps Low impact extraction
  46. 46. ©Continuent 2014 Materialized Views Transaction logs Snapshot UNION ALL Emit last row per key if not a delete MAP REDUCE Materialized view including all updates Sort by key(s), transaction orderSHUFFLE
  47. 47. ©Continuent 2014 Tungsten Replicator 3.0 & Hadoop 47 • Extract from MySQL or Oracle • Base Hadoop plus commercial distributions: Cloudera, HortonWorks, MapR, IBM, Apache • Provision using Sqoop or parallel extraction • Automatic replication of incremental changes • Transformation to preferred HDFS formats • Schema generation for Hive • Tools for generating materialized views
  48. 48. ©Continuent 2014 Getting Started! 48 • Tungsten Replicator builds are available on code.google.com http://code.google.com/p/tungsten-replicator/ • Replicator documentation is available on Continuent website http://docs.continuent.com/tungsten-replicator-3.0/ deployment-hadoop.html • Tungsten Hadoop tools are available on GitHub https://github.com/continuent/continuent-tools-hadoop Contact Continuent for support
  49. 49. ©Continuent 2014 We’re Hiring! Continuent Web Page: http://www.continuent.com ! Tungsten Replicator: http://code.google.com/p/tungsten-replicator
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×