Your SlideShare is downloading. ×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

869
views

Published on

Oracle is the most powerful DBMS in the world. However, Oracle's expensive and complex replication makes it difficult to build highly available applications or move data in real-time to data …

Oracle is the most powerful DBMS in the world. However, Oracle's expensive and complex replication makes it difficult to build highly available applications or move data in real-time to data warehouses and popular databases like MySQL.

In this webinar you will learn how Continuent Tungsten solves problems with Oracle replication at a fraction of the cost of other solutions and with less management overhead too – think "Oracle GoldenGate without the price tag!" We will demo constructing a highly available site using Oracle-to-Oracle replication. We will then show you how to replicate data in real time from Oracle to MySQL as well as load a data warehouse.

Published in: Technology

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
869
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
84
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Replicate Oracle to Oracle, Oracle to MySQL , and Oracle to Analytics Linas Virbalas, Senior Software Engineer Robert Hodges, CEO ©Continuent 2014
  • 2. Introducing Continuent • The leading provider of clustering and replication for open source DBMS • Our Product: Continuent Tungsten • Clustering - Commercial-grade HA, performance scaling and data management for MySQL • Replication - Flexible, high-performance data movement ©Continuent 2014 2
  • 3. What is Tungsten Replicator? Tungsten Replicator: a high-performance open source database replication engine ! GPL V2 license - 100% open source Download from https://code.google.com/p/tungsten-replicator/ Annual support subscription available from Continuent “GoldenGate ® without the Price Tag” ©Continuent 2014 3
  • 4. Tungsten Replicator Overview Master Download transactions via network DBMS Logs Slave Replicator (Transactions + Metadata) Replicator Apply using JDBC ©Continuent 2014 THL THL (Transactions + Metadata) 4
  • 5. Tungsten Replication Service Pipeline Stage Extract Filter Master DBMS ©Continuent 2014 Stage Apply Extract Filter Transaction History Log Stage Apply Extract Filter Queue 5 Apply Slave DBMS
  • 6. Filters and Parallel Apply Pipeline Stage Extract Filter Master DBMS ©Continuent 2014 Stage Stage Apply Extract Filter Transaction History Log Apply Extract Filter Extract Filter Extract Filter In-Memory Queue 6 Apply Apply Apply Slave DBMS
  • 7. Multiple Services per Replicator Replicator Service frommysql Replicator frommysql fromoracle Aggregated Service fromoracle Replicator ©Continuent 2014
  • 8. master-slave MySQL Oracle fan-in slave ©Continuent 2014 Oracle MySQL all-masters Heterogeneous Oracle MySQL Oracle star MySQL
  • 9. Heterogeneous Replication ©Continuent 2014
  • 10. Steps to Heterogeneous Replication 1. Prepare (translate) schema for the slave DBMS 2. Set up replication ddlscan tpm 3. Provision initial data
 
 MySQL Blackhole Parallel Apply Parallel Extract 4. Continue real-time replication Tungsten Replicator ©Continuent 2014
  • 11. Replicating from MySQL to Oracle ©Continuent 2014
  • 12. Use Case: Real-Time Sales/Order Fulfillment Web-based Sales ~1M Xacts/ Day ©Continuent 2014 Purchase Ordering Real-Time Replication 3M Order Items
  • 13. 1. Translating schema for the slave ©Continuent 2014
  • 14. Translating Schema • Goal - set up heterogeneous replication Replication ©Continuent 2014 14
  • 15. Translating Schema • Beginning - how to convert tables? empty Tables ©Continuent 2014 15
  • 16. Translating Schema • Beginning - how to convert tables? empty •Data types? •Column lengths? •Naming conventions? •Reserved words? Tables ©Continuent 2014 16
  • 17. Translating Schema • ddlscan automates these conversions ddlscan Tables ©Continuent 2014 17 empty
  • 18. ddlscan • • • • ©Continuent 2014 Part of Tungsten Replicator, GPL v2 Translates schema with replication in mind Provides errors and warnings Can rename schema/tables/columns 18
  • 19. Usage (MySQL to Oracle Example) $ cd tungsten-replicator/bin ! $ ./ddlscan -db test -template ddl-mysql-oracle.vm -user tungsten -pass secret ©Continuent 2014 19
  • 20. Translating Schema • ddlscan looks into source schema empty Tables ©Continuent 2014 20
  • 21. Translating Schema • ddlscan translates and renders DDL commands empty Tables oracle-ddl.sql ©Continuent 2014 21
  • 22. Result of ddlscan CREATE TABLE 3colors /* ERROR: table starts with a number, use rename option and a filter */ ( id NUMBER(10, 0), color VARCHAR2(1) /* ENUM('R','G','B') */, enabled NUMBER(1) /* BIT(1) - no constraints for 0 or 1 */, acolumnthatdoesntfittooracleat NUMBER(10, 0) /* WARN: truncated column name exceeding 30 characters (acolumnthatdoesntfittooracleatall) */ ! /* ERROR: table must have a primary key! */ ); ! CREATE TABLE talks ... ©Continuent 2014 22
  • 23. Translating Schema • You run resulting SQL file on Oracle empty Tables oracle-ddl.sql ©Continuent 2014 23
  • 24. Translating Schema • Tables are ready! Tables ©Continuent 2014 Translated tables (empty) 24
  • 25. 2. Set Up Replication ©Continuent 2014
  • 26. Replicator Installation ./tools/tpm configure my_ora --enable-heterogenous-service=true --install-directory=/opt/mysql_to_ora/continuent --members=alpha,bravo --master=alpha ! ./tools/tpm configure my_ora --hosts=alpha --replication-user=tungsten --replication-password=secret ! ./tools/tpm configure my_ora --hosts=bravo --replication-user=tungsten_frommysql --replication-password=secret --datasource-type=oracle --datasource-oracle-service=ORCL --svc-applier-filters=dropstatementdata ! ./tools/tpm install my_ora ! ./tools/tpm start my_ora ! ©Continuent 2014 26
  • 27. Replication Under the Hood MySQLExtractor OracleApplier Tungsten Master Replicator Service my_ora MySQL Binlog binlog_format=row ©Continuent 2014 Tungsten Slave Replicator Service my_ora MySQLExtractor Special Filters • Transform ENUM to string • Transform SET to string • Column names and signed flag Special Filters • Drop DDL • Map names to upper case • Rename too long or reserved objects • Optimize updates to remove unchanged columns • Ignore extra tables 27
  • 28. 3. Provisioning an Oracle slave ©Continuent 2014
  • 29. Provisioning (2) ddlscan table structure table structure table data table structure table data table data ©Continuent 2014 29 table structure table structure table structure
  • 30. Provisioning (3) MySQL Sandbox table structure table structure table data table structure table data table data table structure table structure table structure CONVERT to BLACKHOLE ©Continuent 2014 30
  • 31. Provisioning (4) not replicating MySQL Sandbox table structure table structure table data table structure table data table data backup ©Continuent 2014 31
  • 32. Provisioning (5) backup ©Continuent 2014 Master Tungsten Replicator THL 32 4 (ROW) 3 bin logs 1 2 MySQL Sandbox Slave Tungsten Replicator
  • 33. 4. Continue Real-Time Replication ©Continuent 2014
  • 34. Provisioning (6) MySQL Sandbox table structure table structure table data table structure table data table data Master Tungsten Replicator bin logs THL ©Continuent 2014 34 Slave Tungsten Replicator
  • 35. Replicating from Oracle to MySQL ©Continuent 2014
  • 36. Use Case: Web Content Publishing Backend Office Web-Based Catalog Real-Time Publication ©Continuent 2014
  • 37. How Does Tungsten Do That? OracleCDCExtractor MySQLApplier Tungsten Master Replicator Tungsten Slave Replicator Service ora_my Service ora_my OracleCDCExtractor No Special Filters demo schema demo_pub schema Oracle CDC (Synchronous or Async Hotlog) ©Continuent 2014 37 Special Filters • Map names to lower case • Ignore extra tables • Heartbeat table renaming
  • 38. Replicating from Oracle to Oracle ©Continuent 2014
  • 39. How Does Tungsten Do That? OracleCDCExtractor OracleApplier Tungsten Master Replicator Tungsten Slave Replicator Service ora_ora Service ora_ora OracleCDCExtractor No Special Filters demo schema demo_pub schema Oracle CDC (Synchronous or Async Hotlog) ©Continuent 2014 39 Special Filters No Special Filters
  • 40. DEMO DEMO.* DEMO2.* ©Continuent 2014 demo.* demo2.*
  • 41. Replicating from Oracle to Vertica ©Continuent 2014
  • 42. Use Case: Web Content Publishing Backend Office Analytic Reports Near Real-Time Publication Feeds from other data sources ©Continuent 2014
  • 43. How Does Tungsten Do That? OracleCDCExtractor SimpleBatchApplier Tungsten Master Replicator Tungsten Slave Replicator Service ora_ver Service ora_ver OracleCDCExtractor No Special Filters demo schema demo_pub schema Oracle CDC (Synchronous or Async Hotlog) ©Continuent 2014 43 Special Filters • Map names to lower case • Ignore extra tables • Rename data tables (if required)
  • 44. Batch Loading--The Gory Details Replicator Transactions from master Service ora_ver COPY to stage tables CSV CSV CSV Files Files Files Merge Script ©Continuent 2014 44 (or) COPY directly to base tables Staging Staging Staging Tables Tables Tables SELECT to base tables Base Base Base Tables Tables Tables
  • 45. DEMO ©Continuent 2014
  • 46. Coming Attractions ©Continuent 2014
  • 47. New Replication Features in 2014 • • ©Continuent 2014 Parallel extraction for fast provisioning • • • Select data as of specific SCN Pull data across in multiple streams Matches parallel apply on slave side ! Hadoop data loading • • • Direct loading from MySQL/Oracle to HDFS View data using Hive Generation of materialized views 47
  • 48. 560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel +1 (866) 998-3642 Fax +1 (408) 668-1009 e-mail: sales@continuent.com Our Blogs: http://flyingclusters.blogspot.com http://scale-out-blog.blogspot.com http://datacharmer.blogspot.com http://continuent-tungsten.blogspot.com Continuent Web Page: http://www.continuent.com ! Documentation: https://docs.continuent.com ! Tungsten Replicator: http://code.google.com/p/tungsten-replicator ©Continuent 2014 48