Set Up & Operate Open Source Oracle Replication
 

Like this? Share it with your network

Share

Set Up & Operate Open Source Oracle Replication

on

  • 197 views

Oracle's expensive and complex replication makes it difficult to build cost-effective applications that move data in real-time to data warehouses (Oracle, Hadoop, Vertica) and popular databases like ...

Oracle's expensive and complex replication makes it difficult to build cost-effective applications that move data in real-time to data warehouses (Oracle, Hadoop, Vertica) and popular databases like MySQL. Fortunately, Continuent Tungsten offers a solution. In this virtual course, you will learn how Continuent Tungsten solves problems with Oracle replication at a fraction of the cost of other solutions and with less management overhead too – think "Oracle GoldenGate without the price tag!”

Course Topics:
- Replicator installation and configuration
- Provisioning existing schema and tables from Oracle into another server
- Replicating between Oracle schemas
- Replicating from Oracle to MySQL and data warehouses
- Tips for solving common problems and improving performance.

The course includes live demos to illustrate operation. By the end of the course you should be ready move ahead with your own deployments.

Statistics

Views

Total Views
197
Views on SlideShare
197
Embed Views
0

Actions

Likes
0
Downloads
9
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Set Up & Operate Open Source Oracle Replication Presentation Transcript

  • 1. ©Continuent 2014 Replicate from Oracle to Oracle, Oracle to MySQL and Oracle to Analytics MC Brown, Senior Information Architect Linas Virbalas, Senior Software Engineer
  • 2. ©Continuent 2014 Introducing Continuent 2 • The leading provider of clustering and replication for open source DBMS • Our Product: Continuent Tungsten • Clustering - Commercial-grade HA, performance scaling and data management for MySQL • Replication - Flexible, high-performance data movement
  • 3. ©Continuent 2014 Quick Continuent Facts • Largest Tungsten installation processes over 700 million transactions daily on 225 terabytes of data • Tungsten Replicator was application of the year at the 2011 MySQL User Conference • Wide variety of topologies including MySQL, Oracle, Vertica, and MongoDB are in production now • MySQL to Hadoop deployments are now in progress with multiple customers 3
  • 4. ©Continuent 2014©Continuent 2014 Continuent Tungsten Customers 4 1
  • 5. ©Continuent 2014 Tungsten Replicator 5 Tungsten Replicator is a fast, open source, database replication engine ! Designed for speed and flexibility GPLV2 license 100% open source Annual support subscription available from Continuent
  • 6. ©Continuent 2014 Tungsten Master/Slave in Action 6 Master (Transactions + Metadata) Slave THL DBMS Logs Replicator (Transactions + Metadata) THLReplicator Download transactions via network Apply using JDBC
  • 7. ©Continuent 2014 Master Replication Service 7 Extract Filter Apply Stage Extract Filter Apply Stage Pipeline MySQL Master Transaction History Log In-Memory Queue Slave ReplicatorsBinlog tcp/ip
  • 8. ©Continuent 2014 Slave Replication Service 8 Extract Filter Apply Stage Extract Filter Apply Stage Extract Filter Apply Stage Pipeline Transaction History Log In-Memory Queue Slave DBMS Master Replicator tcp/ip
  • 9. ©Continuent 2014 Multiple Services per Replicator Service frommysql Service fromoracle frommysql fromoracle Replicator Replicator Replicator Aggregated
  • 10. ©Continuent 2014 star master-slave Heterogeneous fan-in slave all-masters MySQL Oracle Oracle MySQL Oracle Oracle MySQL MySQL
  • 11. ©Continuent 2014 Replicating from Oracle to Oracle
  • 12. ©Continuent 2014©Continuent 2013 Steps to Homogeneous Replication 1. Restore backup on the slave 2. Set up replication 3. Continue real-time replication Tungsten Replicator tpmsetupCDC &
  • 13. ©Continuent 2014 Preparing CDC 13 Tables Source schema setupCDC Oracle CDC (Synchronous or Async Hotlog) Publisher schema Change tables Capture started at <setupCDC_SCN>
  • 14. ©Continuent 2014 CDC in Action 14 Source table X Source schema Oracle CDC (Synchronous or Async Hotlog) Publisher schema Change table X_CT INSERT UPDATE DELETEx SCN++
  • 15. ©Continuent 2014©Continuent 2013 Installation 15 ./current/tools/tpm configure fromora --enable-heterogenous-service=true --user=oracle --install-directory=/opt/fromora/continuent --members=alhpa,beta --master=alpha ! ./current/tools/tpm configure fromora --hosts=alpha --datasource-type=oracle --user=oracle --datasource-oracle-service=ORCL --replication-user=DEMO_PUB --replication-password=DEMO_PUB --svc-table-engine=CDCASYNC --property=replicator.global.extract.db.user=tungsten --property=replicator.global.extract.db.password=secret --property=replicator.extractor.parallel-extractor. ChunkDefinitionFile=/opt/fromora/chunks.csv ! ./tools/tpm configure fromora --hosts=beta --datasource-type=oracle --datasource-oracle-service=ORCL --replication-user=DEMO --replication-password=DEMO
  • 16. ©Continuent 2014 Deployment 16 Service fromora Tungsten Master Replicator Service fromora Tungsten Slave Replicator demo schema demo_pub schema Oracle CDC (Synchronous or Async Hotlog) Tungsten Slave Replicator OracleApplierOracleCDCExtractor
  • 17. ©Continuent 2014©Continuent 2013 Heterogeneous Replication
  • 18. ©Continuent 2014 Use Case: Web Content Publishing Real-Time Publication Backend Office Web-Based Catalog
  • 19. ©Continuent 2014©Continuent 2013 Steps to Heterogeneous Replication 1. Prepare (translate) schema for the slave DBMS 2. Set up replication 3. Provision initial data
 
 4. Continue real-time replication ddlscan Tungsten Replicator Parallel ApplyParallel Extract & tpmsetupCDC &
  • 20. ©Continuent 2014©Continuent 2013 (Open Source)
  • 21. ©Continuent 2014©Continuent 2013 1.Translating schema for the slave
  • 22. ©Continuent 2014©Continuent 2013 Translating Schema 22 • Beginning - how to convert tables? empty Tables •Data types? •Column lengths? •Naming conventions? •Reserved words? ddlscan
  • 23. ©Continuent 2014©Continuent 2013 ddlscan 23 • Part of Tungsten Replicator, GPL v2 • Translates schema with replication in mind • Provides errors and warnings • Can rename schema/tables/columns
  • 24. ©Continuent 2014©Continuent 2013 Usage (Oracle to MySQL Example) 24 $ cd tungsten-replicator/bin ! $ ./ddlscan -db DEMO -template ddl-oracle-mysql.vm
  • 25. ©Continuent 2014©Continuent 2013 Translating Schema 25 • ddlscan looks into source schema Tables empty
  • 26. ©Continuent 2014©Continuent 2013 Translating Schema 26 • ddlscan translates and renders DDL commands mysql-ddl.sql Tables empty
  • 27. ©Continuent 2014©Continuent 2013 Result of ddlscan 27 DROP TABLE IF EXISTS demo.test; CREATE TABLE demo.test ( id1 INT /* NUMBER(10, ?) */ NOT NULL, id2 INT /* NUMBER(10, ?) */ NOT NULL, val TINYINT /* NUMBER(3, ?) */, owner VARCHAR(30) /* VARCHAR2(30) */ NOT NULL, created DATETIME /* DATE */ NOT NULL, PRIMARY KEY (id1, id2) /* WARN: no PK found, using suitable unique index instead: UQ_UK1 */ ) ENGINE=InnoDB; ! CREATE TABLE talks ...
  • 28. ©Continuent 2014©Continuent 2013 Translating Schema 28 • You run resulting SQL file on Oracle mysql-ddl.sql Tables empty
  • 29. ©Continuent 2014©Continuent 2013 Translating Schema 29 • Tables are ready! Translated tables
 (no rows) Tables
  • 30. ©Continuent 2014 Preparing CDC 30 Tables Source schema setupCDC Oracle CDC (Synchronous or Async Hotlog) Publisher schema Change tables Capture started at <setupCDC_SCN> SCN++
  • 31. ©Continuent 2014©Continuent 2013 Connecting the Dots 31 • Preparation completed Translated tables
 (no rows) Tables Change tables …
  • 32. ©Continuent 2014©Continuent 2013 2. Set Up Replication
  • 33. ©Continuent 2014©Continuent 2013 33 ./current/tools/tpm configure fromora --enable-heterogenous-service=true --user=oracle --install-directory=/opt/fromora/continuent --members=alhpa,beta --master=alpha ! ./current/tools/tpm configure fromora --hosts=alpha --datasource-type=oracle --user=oracle --datasource-oracle-service=ORCL --replication-user=DEMO_PUB --replication-password=DEMO_PUB --svc-table-engine=CDCASYNC --property=replicator.global.extract.db.user=tungsten --property=replicator.global.extract.db.password=secret --property=replicator.extractor.parallel-extractor. ChunkDefinitionFile=/opt/fromora/chunks.csv Oracle Master Part
  • 34. ©Continuent 2014©Continuent 2013 MySQL Slave Part 34 ! ! ./current/tools/tpm configure fromora --hosts=beta --user=tungsten --replication-user=tungsten --replication-password=secret --svc-applier-filters=CDC,casetransform,rename --property=replicator.filter.CDC.from=DEMO_PUB.HEARTBEAT --property=replicator.filter.CDC.to=tungsten_fromora.heartbeat --property=replicator.filter.casetransform.to_upper_case=false --property=replicator.filter.rename.definitionsFile= /opt/frommysql/rename.csv
  • 35. ©Continuent 2014 Deployment 35 Service fromoracle Tungsten Master Replicator OracleCDCExtractor No Special Filters Service fromoracle Tungsten Slave Replicator Special Filters •Map names to lower case •Ignore extra tables •Heartbeat table renaming demo schema demo_pub schema Oracle CDC (Synchronous or Async Hotlog) MySQLApplierOracleCDCExtractor
  • 36. ©Continuent 2014©Continuent 2013 3. Provision & Replication
  • 37. ©Continuent 2014©Continuent 2013 Parallel Extractor 37 • trepctl online -provision <setupCDC_SCN> Tables Service fromoracle Tungsten Master Replicator THL ParallelExtractor Thread 1 Thread 2 Thread 3 SELECT … AS OF <setupCDC_SCN>
  • 38. ©Continuent 2014©Continuent 2013 Real-time Replication 38 • Automatically switches to real-time extraction Service fromoracle Tungsten Master Replicator THL Oracle CDC (Synchronous or Async Hotlog) Change tables >=$setupCDC_SCN OracleCDCExtractor Tables
  • 39. ©Continuent 2014 Accelerate the Slave Provision 39 Extract Filter Apply Stage Extract Filter Apply Stage Stage Pipeline Remote Master Transaction History Log Parallel Queue Slave DBMS Extract Filter Apply Extract Filter Apply Extract Filter Apply (Assign Shard ID)
  • 40. ©Continuent 2014 Replication to Vertica and Hadoop
  • 41. ©Continuent 2014 The Data Warehouse Impedance Mismatch 41 Replication CSV Files CSV Files Buffered Transactions Dump/load Single Transactions Batches
  • 42. ©Continuent 2014 Column Store--Real-Time Batches MySQL/Oracle Tungsten Master Replicator Service ora2vr Special Filters * pkey - Fill in pkey info * colnames - Fill in names * replicate - Ignore tables Tungsten Slave Replicator Service ora2vr CSV Files CSV Files CSV Files CSV Files CSV Files Large transaction batches to leverage load parallelization
  • 43. ©Continuent 2014 Batch Loading--The Gory Details Replicator Service ora2vr Transactions from master CSV Files CSV Files CSV Files Staging Tables Staging Tables Staging Tables Base Tables Base Tables Base Tables Merge Script (or) COPY directly to base tables COPY to stage tables SELECT to base tables
  • 44. ©Continuent 2014 Basic Hadoop Loading MySQL/Oracle Tungsten Master Replicator hadoop Master-Side Filtering * pkey - Fill in pkey info * colnames - Fill in names * replicate - Subset tables to be replicated Tungsten Slave Replicator hadoop CSV Files CSV Files CSV Files CSV Files CSV Files Hadoop Cluster Extract from source DBMS Load raw CSV to HDFS (e.g., via LOAD DATA to Hive) Access via Hive
  • 45. ©Continuent 2012 Provisioning plus Replication 45 MySQL/Oracle Tungsten 3.0 Master hadoop Tungsten 3.0 Slave hadoop CSV Files CSV Files CSV Files CSV FilesCSV Apache Sqoop/ETL Fast data filtering Buffered CSV Programmable load scripts Parallel apply Parallel table dumps Low impact extraction
  • 46. ©Continuent 2014 Materialized Views Transaction logs Snapshot UNION ALL Emit last row per key if not a delete MAP REDUCE Materialized view including all updates Sort by key(s), transaction orderSHUFFLE
  • 47. ©Continuent 2014 Tungsten Replicator 3.0 & Hadoop 47 • Extract from MySQL or Oracle • Base Hadoop plus commercial distributions: Cloudera, HortonWorks, MapR, IBM, Apache • Provision using Sqoop or parallel extraction • Automatic replication of incremental changes • Transformation to preferred HDFS formats • Schema generation for Hive • Tools for generating materialized views
  • 48. ©Continuent 2014 Getting Started! 48 • Tungsten Replicator builds are available on code.google.com http://code.google.com/p/tungsten-replicator/ • Replicator documentation is available on Continuent website http://docs.continuent.com/tungsten-replicator-3.0/ deployment-hadoop.html • Tungsten Hadoop tools are available on GitHub https://github.com/continuent/continuent-tools-hadoop Contact Continuent for support
  • 49. ©Continuent 2014 We’re Hiring! Continuent Web Page: http://www.continuent.com ! Tungsten Replicator: http://code.google.com/p/tungsten-replicator