Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle
 

Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle

on

  • 346 views

Learn how Gittigidiyor (a subsidiary of eBay) replicates data in real time from MySQL to Oracle with Continuent Tungsten. Gittigidiyor consolidates reporting data in a single MySQL server. ...

Learn how Gittigidiyor (a subsidiary of eBay) replicates data in real time from MySQL to Oracle with Continuent Tungsten. Gittigidiyor consolidates reporting data in a single MySQL server. Transactions arriving on this server must be replicated in real time to an Oracle instance.

Levent Kurt, Site Operations Manager at Gittigidiyor, describes how Gittigidiyor solved their problems of handling large tables in MySQL by replicating 20 GB of binlog data/day from MySQL 5.5 Community Edition to Oracle Enterprise Edition 11G with Continuent Tungsten, in a hosted VMware ESX environment.

Statistics

Views

Total Views
346
Views on SlideShare
346
Embed Views
0

Actions

Likes
0
Downloads
3
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle Presentation Transcript

  • How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle Levent Kurt, Site Operations Manager, Gittigidiyor Giuseppe Maxia, Director of Quality Assurance, Continuent Linas Virbalas, Senior Software Engineer, Continuent ©Continuent 2013.
  • Introducing Continuent • The leading provider of clustering and replication for open source DBMS • Our Product: Continuent Tungsten • Clustering - Commercial-grade HA, performance scaling and data management for MySQL • Replication - Flexible, high-performance data movement ©Continuent 2013 2
  • A Review of Tungsten Replicator ©Continuent 2013 3 View slide
  • Tungsten Replicator Overview Master Download transactions via network DBMS Logs Slave Replicator (Transactions + Metadata) Replicator Apply using JDBC ©Continuent 2013 THL THL (Transactions + Metadata) 4 View slide
  • Master Replication Service Pipeline Stage Extract Filter Stage Apply Extract Filter Apply tcp/ip Binlog MySQL Master ©Continuent 2013 Transaction History Log In-Memory Queue 5 Slave Replicators
  • Slave Replication Service Pipeline Stage Apply Extract Filter Stage Apply Extract Filter Apply tcp/ip Extract Filter Stage Master Replicator Transaction History Log ©Continuent 2013 In-Memory Queue 6 Slave DBMS
  • master-slave MySQL Oracle fan-in slave ©Continuent 2013 Oracle MySQL all-masters 7 Heterogeneous Oracle MySQL Oracle star MySQL
  • Heterogeneous Replication ©Continuent 2013
  • Steps to Heterogeneous Replication 1. Prepare (translate) schema for the slave DBMS 2. Set up replication ddlscan tpm 3. Provision initial data
 
 MySQL Blackhole Parallel Apply 4. Continue real-time replication Tungsten Replicator ©Continuent 2013
  • Replicating from MySQL to Oracle ©Continuent 2013
  • Use Case: Real-Time Sales/Order Fulfillment Web-based Sales ~1M Xacts/ Day ©Continuent 2013 Purchase Ordering Real-Time Replication 3M Order Items
  • 1. Translating schema for the slave ©Continuent 2013
  • Translating Schema • Goal - set up heterogeneous replication Replication ©Continuent 2013 13
  • Translating Schema • Beginning - how to convert tables? empty Tables ©Continuent 2013 14
  • Translating Schema • Beginning - how to convert tables? empty •Data types? •Column lengths? •Naming conventions? •Reserved words? Tables ©Continuent 2013 15
  • Translating Schema • ddlscan automates these conversions ddlscan Tables ©Continuent 2013 16 empty
  • ddlscan • • • • ©Continuent 2013 Part of Tungsten Replicator, GPL v2 Translates schema with replication in mind Provides errors and warnings Can rename schema/tables/columns 17
  • Usage (MySQL to Oracle Example) $ cd tungsten-replicator/bin ! $ ./ddlscan -db test -template ddl-mysql-oracle.vm -user tungsten -pass secret ©Continuent 2013 18
  • Translating Schema • ddlscan looks into source schema empty Tables ©Continuent 2013 19
  • Translating Schema • ddlscan translates and renders DDL commands empty Tables oracle-ddl.sql ©Continuent 2013 20
  • Result of ddlscan CREATE TABLE 3colors /* ERROR: table starts with a number, use rename option and a filter */ ( id NUMBER(10, 0), color VARCHAR2(1) /* ENUM('R','G','B') */, enabled NUMBER(1) /* BIT(1) - no constraints for 0 or 1 */, acolumnthatdoesntfittooracleat NUMBER(10, 0) /* WARN: truncated column name exceeding 30 characters (acolumnthatdoesntfittooracleatall) */ ! /* ERROR: table must have a primary key! */ ); ! CREATE TABLE talks ... ©Continuent 2013 21
  • Translating Schema • You run resulting SQL file on Oracle empty Tables oracle-ddl.sql ©Continuent 2013 22
  • Translating Schema • Tables are ready! Tables ©Continuent 2013 Translated tables (empty) 23
  • 2. Set Up Replication ©Continuent 2013
  • Replicator Installation ./tools/tpm configure my_ora --enable-heterogenous-service=true --install-directory=/opt/mysql_to_ora/continuent --members=alpha,bravo --master=alpha ! ./tools/tpm configure my_ora --hosts=alpha --replication-user=tungsten --replication-password=secret ! ./tools/tpm configure my_ora --hosts=bravo --replication-user=tungsten_frommysql --replication-password=secret --datasource-type=oracle --datasource-oracle-service=ORCL --svc-applier-filters=dropstatementdata ! ./tools/tpm install my_ora ! ./tools/tpm start my_ora ! ©Continuent 2013 25
  • Replication Under the Hood MySQLExtractor OracleApplier Tungsten Master Replicator Service frommysql MySQL Binlog binlog_format=row ©Continuent 2013 Tungsten Slave Replicator Service frommysql MySQLExtractor Special Filters • Transform ENUM to string • Transform SET to string Special Filters • Drop DDL • Map names to upper case • Rename too long or reserved objects • Optimize updates to remove unchanged columns • Ignore extra tables 26
  • 3. Provisioning an Oracle slave with auditing trail ©Continuent 2013
  • The Requirements • Export data from MySQL master to the Oracle slave before starting replication • The Oracle slave should also take trace of the changes • Convert the MySQL tables to stream of raw data in THL format • Data is about 400 GB ©Continuent 2013 28
  • The Practical Steps 1. Export data from master 2. Keep track of replication position 3. Start the replicator from the sandbox 4. Load data into slave with parallel apply 5. Switch replicator to real master 6. Dismiss the sandbox 7. Replication from MySQL to Oracle live ©Continuent 2013 29
  • Provisioning (1) MySQL Oracle master slave table structure table structure table data table structure table data table data ©Continuent 2013 30
  • Provisioning (2) MySQL Oracle master slave ddlscan table structure table structure table data table structure table data table data ©Continuent 2013 31 table structure table structure table structure
  • Provisioning (3) MySQL master table structure table structure table data table structure table data table data Oracle MySQL sandbox table structure table structure table structure CONVERT to BLACKHOLE ©Continuent 2013 32 slave
  • Provisioning (4) not replicating MySQL master MySQL sandbox table structure table structure table data table structure table data table data backup ©Continuent 2013 33 Oracle slave
  • Provisioning (5) MySQL sandbox backup ©Continuent 2013 bin logs (ROW) Oracle slave tungsten replicator THL 34 tungsten replicator
  • How Parallel Apply Works Pipeline Stage Stage Filter Extract Apply (Shard) Stage Extract/Filter/Apply Extract/Filter/Apply Extract/Filter/Apply Extract/Filter/ Apply Parallel Queue Master DBMS ©Continuent 2013 Transaction History Log Provisioning Mutlitenant source Slave DBMS
  • 4. Continue Real-Time Replication ©Continuent 2013
  • Provisioning (6) MySQL MySQL sandbox master table structure table structure table data table structure table data table data Oracle slave tungsten replicator bin logs THL ©Continuent 2013 37 tungsten replicator
  • Gittigidiyor (subsidiary of eBay) architecture and needs ©Continuent 2013
  • • Gittigidiyor • • • • • • • • A subsidiary of eBay Founded in 2001, joined the eBay family in June 2011 Most widely-used e-shopping site in Turkey More than 9 million registered users Over 900 thousand daily visitors and 5 million listings at any given time. 700 thousand sellers all around Turkey Selling 1 item every 3 seconds ©Continuent 2013 1
  • Gittigidiyor Datawarehouse The Problem Replication ! • • • • • 400 GB of data 450 tables over 150 million rows 20GB binlog/day MySQL inefficiency for large data handling • Migration to Oracle • Need to keep auditing trail • Tried Golden Gate, but • technical problems • too expensive ©Continuent 2013 Master CDC ETL DWH 40
  • Tungsten Solution ! • DDL generation • Replication to Oracle master • Handling data correctly • CDC generation • Fail-safe Rep licat Master ion Tungsten Replication Replicator Tungsten Filters ETL ORA Master ORA CDC ©Continuent 2013 41 ORA DWH
  • The steps we took ! 1. Proof of Concept a. replicated 5 tables b. created the DDL Generator. 2. Set up the replication to Oracle Master 3. Set up filters for Oracle CDC generation. 4. Dumped all data in MySQL Master and MySQL CDC to Oracle equivalents. Rep licat Master ion Tungsten Replication Replicator Tungsten Filters ETL ORA Master ORA CDC ©Continuent 2013 42 ORA DWH
  • MySQL to Oracle Demo ©Continuent 2013
  • Support for Tungsten Replicator • Community support is available on project mailing list • Continuent offers implementation services and annual 24/7 support subscriptions • For high availability or disaster recovery needs, check out Continuent Tungsten clustering • Meanwhile, have fun and try it out! ©Continuent 2013 44
  • 560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel +1 (866) 998-3642 Fax +1 (408) 668-1009 e-mail: sales@continuent.com Our Blogs: http://flyingclusters.blogspot.com http://scale-out-blog.blogspot.com http://datacharmer.blogspot.com http://continuent-tungsten.blogspot.com Continuent Web Page: http://www.continuent.com ! Tungsten Replicator: http://tungsten-replicator.org ©Continuent 2013 45