Gittigidiyor, a subsidiary of eBay, needed to replicate data in real time from their MySQL database to an Oracle database to power their data warehouse. Continuent Tungsten was used to provide heterogeneous replication between the databases. The schema was translated from MySQL to Oracle using ddlscan. Initial data was exported and loaded into Oracle using parallel apply. Ongoing real-time replication now occurs between MySQL and Oracle using Tungsten Replicator with custom filters to handle data type translations.

1. How Gittigidiyor (a subsidiary of eBay)
Replicates Data In Real Time
From MySQL To Oracle
Levent Kurt, Site Operations Manager, Gittigidiyor
Giuseppe Maxia, Director of Quality Assurance, Continuent
Linas Virbalas, Senior Software Engineer, Continuent
©Continuent 2013.

2. Introducing Continuent
•
The leading provider of clustering and
replication for open source DBMS
•
Our Product: Continuent Tungsten
• Clustering - Commercial-grade HA, performance
scaling and data management for MySQL
• Replication - Flexible, high-performance data
movement
©Continuent 2013
2

3. A Review of Tungsten Replicator
©Continuent 2013
3

4. Tungsten Replicator Overview
Master
Download
transactions
via network
DBMS
Logs
Slave
Replicator
(Transactions + Metadata)
Replicator
Apply using JDBC
©Continuent 2013
THL
THL
(Transactions + Metadata)
4

5. Master Replication Service
Pipeline
Stage
Extract Filter
Stage
Apply
Extract Filter
Apply
tcp/ip
Binlog
MySQL
Master
©Continuent 2013
Transaction
History Log
In-Memory
Queue
5
Slave
Replicators

6. Slave Replication Service
Pipeline
Stage
Apply
Extract Filter
Stage
Apply
Extract Filter
Apply
tcp/ip
Extract Filter
Stage
Master
Replicator
Transaction
History Log
©Continuent 2013
In-Memory
Queue
6
Slave
DBMS

7. master-slave
MySQL
Oracle
fan-in slave
©Continuent 2013
Oracle
MySQL
all-masters
7
Heterogeneous
Oracle
MySQL
Oracle
star
MySQL

8. Heterogeneous
Replication
©Continuent 2013

9. Steps to Heterogeneous Replication
1. Prepare (translate) schema for
the slave DBMS
2. Set up replication
ddlscan
tpm
3. Provision initial data
MySQL Blackhole
Parallel Apply
4. Continue real-time replication
Tungsten Replicator
©Continuent 2013

10. Replicating from
MySQL to Oracle
©Continuent 2013

11. Use Case: Real-Time Sales/Order
Fulfillment
Web-based Sales
~1M
Xacts/
Day
©Continuent 2013
Purchase Ordering
Real-Time Replication
3M
Order
Items

12. 1. Translating schema
for the slave
©Continuent 2013

13. Translating Schema
•
Goal - set up heterogeneous replication
Replication
©Continuent 2013
13

14. Translating Schema
•
Beginning - how to convert tables?
empty
Tables
©Continuent 2013
14

15. Translating Schema
•
Beginning - how to convert tables?
empty
•Data types?
•Column lengths?
•Naming conventions?
•Reserved words?
Tables
©Continuent 2013
15

16. Translating Schema
•
ddlscan automates these conversions
ddlscan
Tables
©Continuent 2013
16
empty

17. ddlscan
•
•
•
•
©Continuent 2013
Part of Tungsten Replicator, GPL v2
Translates schema with replication in mind
Provides errors and warnings
Can rename schema/tables/columns
17

18. Usage (MySQL to Oracle Example)
$ cd tungsten-replicator/bin
!
$ ./ddlscan
-db test
-template ddl-mysql-oracle.vm
-user tungsten
-pass secret
©Continuent 2013
18

19. Translating Schema
•
ddlscan looks into source schema
empty
Tables
©Continuent 2013
19

20. Translating Schema
•
ddlscan translates and renders DDL commands
empty
Tables
oracle-ddl.sql
©Continuent 2013
20

21. Result of ddlscan
CREATE TABLE 3colors
/* ERROR: table starts with a number, use rename option and a
filter */
(
id NUMBER(10, 0),
color VARCHAR2(1) /* ENUM('R','G','B') */,
enabled NUMBER(1) /* BIT(1) - no constraints for 0 or 1 */,
acolumnthatdoesntfittooracleat NUMBER(10, 0)
/* WARN: truncated column name exceeding 30 characters
(acolumnthatdoesntfittooracleatall) */
!
/* ERROR: table must have a primary key! */
);
!
CREATE TABLE talks
...
©Continuent 2013
21

22. Translating Schema
•
You run resulting SQL file on Oracle
empty
Tables
oracle-ddl.sql
©Continuent 2013
22

23. Translating Schema
•
Tables are ready!
Tables
©Continuent 2013
Translated
tables (empty)
23

24. 2. Set Up Replication
©Continuent 2013

25. Replicator Installation
./tools/tpm configure my_ora
--enable-heterogenous-service=true
--install-directory=/opt/mysql_to_ora/continuent
--members=alpha,bravo
--master=alpha
!
./tools/tpm configure my_ora --hosts=alpha
--replication-user=tungsten
--replication-password=secret
!
./tools/tpm configure my_ora --hosts=bravo
--replication-user=tungsten_frommysql
--replication-password=secret
--datasource-type=oracle
--datasource-oracle-service=ORCL
--svc-applier-filters=dropstatementdata
!
./tools/tpm install my_ora
!
./tools/tpm start my_ora
!
©Continuent 2013
25

26. Replication Under the Hood
MySQLExtractor
OracleApplier
Tungsten Master
Replicator
Service
frommysql
MySQL
Binlog
binlog_format=row
©Continuent 2013
Tungsten Slave
Replicator
Service
frommysql
MySQLExtractor
Special Filters
• Transform
ENUM to string
• Transform SET
to string
Special Filters
• Drop DDL
• Map names to upper case
• Rename too long or reserved
objects
• Optimize updates to remove
unchanged columns
• Ignore extra tables
26

27. 3. Provisioning an
Oracle slave with
auditing trail
©Continuent 2013

28. The Requirements
•
Export data from MySQL master to the
Oracle slave before starting replication
•
The Oracle slave should also take trace of the
changes
•
Convert the MySQL tables to stream of raw
data in THL format
•
Data is about 400 GB
©Continuent 2013
28

29. The Practical Steps
1. Export data from master
2. Keep track of replication position
3. Start the replicator from the sandbox
4. Load data into slave with parallel apply
5. Switch replicator to real master
6. Dismiss the sandbox
7. Replication from MySQL to Oracle live
©Continuent 2013
29

30. Provisioning (1)
MySQL
Oracle
master
slave
table structure
table structure
table data
table structure
table data
table data
©Continuent 2013
30

31. Provisioning (2)
MySQL
Oracle
master
slave
ddlscan
table structure
table structure
table data
table structure
table data
table data
©Continuent 2013
31
table structure
table structure
table structure

32. Provisioning (3)
MySQL
master
table structure
table structure
table data
table structure
table data
table data
Oracle
MySQL
sandbox
table structure
table structure
table structure
CONVERT to
BLACKHOLE
©Continuent 2013
32
slave

33. Provisioning (4)
not replicating
MySQL
master
MySQL
sandbox
table structure
table structure
table data
table structure
table data
table data
backup
©Continuent 2013
33
Oracle
slave

34. Provisioning (5)
MySQL
sandbox
backup
©Continuent 2013
bin
logs
(ROW)
Oracle
slave
tungsten
replicator
THL
34
tungsten
replicator

35. How Parallel Apply Works
Pipeline
Stage
Stage
Filter
Extract
Apply
(Shard)
Stage
Extract/Filter/Apply
Extract/Filter/Apply
Extract/Filter/Apply
Extract/Filter/
Apply
Parallel
Queue
Master
DBMS
©Continuent 2013
Transaction
History Log
Provisioning
Mutlitenant
source
Slave
DBMS

36. 4. Continue Real-Time
Replication
©Continuent 2013

37. Provisioning (6)
MySQL
MySQL
sandbox
master
table structure
table structure
table data
table structure
table data
table data
Oracle
slave
tungsten
replicator
bin
logs
THL
©Continuent 2013
37
tungsten
replicator

38. Gittigidiyor
(subsidiary of eBay)
architecture and
needs
©Continuent 2013

39. • Gittigidiyor
•
•
•
•
•
•
•
•
A subsidiary of eBay
Founded in 2001, joined the eBay family in June 2011
Most widely-used e-shopping site in Turkey
More than 9 million registered users
Over 900 thousand daily visitors and
5 million listings at any given time.
700 thousand sellers all around Turkey
Selling 1 item every 3 seconds
©Continuent 2013
1

40. Gittigidiyor Datawarehouse
The Problem
Replication
!
•
•
•
•
•
400 GB of data
450 tables
over 150 million rows
20GB binlog/day
MySQL inefficiency for large
data handling
• Migration to Oracle
• Need to keep auditing trail
• Tried Golden Gate, but
• technical problems
• too expensive
©Continuent 2013
Master
CDC
ETL
DWH
40

41. Tungsten Solution
!
• DDL generation
• Replication to Oracle master
• Handling data correctly
• CDC generation
• Fail-safe
Rep
licat
Master
ion
Tungsten
Replication
Replicator
Tungsten
Filters
ETL
ORA Master
ORA CDC
©Continuent 2013
41
ORA DWH

42. The steps we took
!
1. Proof of Concept
a. replicated 5 tables
b. created the DDL
Generator.
2. Set up the replication to
Oracle Master
3. Set up filters for Oracle
CDC generation.
4. Dumped all data in MySQL
Master and MySQL CDC to
Oracle equivalents.
Rep
licat
Master
ion
Tungsten
Replication
Replicator
Tungsten
Filters
ETL
ORA Master
ORA CDC
©Continuent 2013
42
ORA DWH

43. MySQL to Oracle
Demo
©Continuent 2013

44. Support for Tungsten Replicator
•
Community support is available on project
mailing list
•
Continuent offers implementation services and
annual 24/7 support subscriptions
•
For high availability or disaster recovery needs,
check out Continuent Tungsten clustering
•
Meanwhile, have fun and try it out!
©Continuent 2013
44

45. 560 S. Winchester Blvd., Suite 500
San Jose, CA 95128
Tel +1 (866) 998-3642
Fax +1 (408) 668-1009
e-mail: sales@continuent.com
Our Blogs:
http://flyingclusters.blogspot.com
http://scale-out-blog.blogspot.com
http://datacharmer.blogspot.com
http://continuent-tungsten.blogspot.com
Continuent Web Page:
http://www.continuent.com
!
Tungsten Replicator:
http://tungsten-replicator.org
©Continuent 2013
45