The document discusses heterogeneous replication technology by Continuent, focusing on data integration and migration between different DBMS types, specifically MySQL and PostgreSQL. It highlights the features and benefits of the Tungsten Replicator, including advanced logical replication and the use of filters to manage data type differences. Additionally, it addresses challenges such as SQL dialect discrepancies and provides examples of replication methodologies.

1. Linas Virbalas
Continuent, Inc.
© Continuent 2010

2. / Definition & Motivation
/ Three Stories
/ How?
/ Questions and Answers
© Continuent 2010

3. © Continuent 2010

4. Heterogeneous Replication

Replication between different types of DBMS
© Continuent 2010

5. 1. Real-time integration of data between different DBMS
types
2. Seamless migration out of one DBMS type to another
3. Data warehousing (real-time) from different DBMS
types
© Continuent 2010

6. / Name: Linas Virbalas
/ Country: Lithuania
/ Implementing for Tungsten:
• MySQL -> PostgreSQL
• MySQL -> Greenplum
• MySQL -> Oracle
• PostgreSQL WAL
• PostgreSQL Streaming Replication
• PostgreSQL Logical Replication
via Slony logs
/ Blog:
http://flyingclusters.blogspot.com
© Continuent 2010

7. © Continuent 2010

8. 1. Story of MySQL -> …
• How to replicate from MySQL to PostgreSQL/Greenplum?
Oracle?
2. Story of PostgreSQL -> PostgreSQL
• Advanced Logical Replication
3. Story of PostgreSQL -> …
• For now: how to replicate from PostgreSQL to MySQL
© Continuent 2010

9. With Tungsten Replicator
Why?
Because It Can! :)
© Continuent 2010

10. / Interfaces to implement new:
• Extractors
• Filters
• Appliers
/ Multiple replication services per one process
/ JAVA
/ Open Source GPL v2
© Continuent 2010

11. Technology: Replication Pipelines
© Continuent 2010

12. © Continuent 2010

13. / Statement Based Replication
/ Row Based Replication
© Continuent 2010

14. © Continuent 2010

15. Tungsten Tungsten
Replicator Replicator
© Continuent 2010

16. Master Slave
Replicator Replicator
Transaction Transaction
History Log History Log
Filters Filters
MySQL PostgreSQL
Extractor Applier
© Continuent 2010

17. / Provisioning
/ Data Type Differences
/ Database vs. Schema
/ Default (Implicitly Defined) Schema Selection
/ SQL Dialect Differences
• Statement Replication vs. Row Replication
/ Character Sets and Binary Data
/ Old Versions of MySQL
© Continuent 2010

18. Provisioning
/ Harder way: Dump data explicitly
/ Easier way: Replicate a mysqldump backup
Replicator
© Continuent 2010

19. / Note the type differences between MySQL and PG
MySQL PostgreSQL
! TINYINT SMALLINT
SMALLINT SMALLINT
INTEGER INTEGER
BIGINT BIGINT
! CHAR(1) CHAR(5) = {‘true’, ‘false’}
CHAR(x) CHAR(x)
VARCHAR(x) VARCHAR(x)
DATE DATE
TIMESTAMP TIMESTAMP
! TEXT (diff. sizes) TEXT
! BLOB BYTEA
…
© Continuent 2010

20. Database vs. Schema
/ In MySQL these are the same:
! !CREATE DATABASE foo!
! !CREATE SCHEMA foo!
/ In PostgreSQL these are very different:
CREATE DATABASE foo!
! !CREATE SCHEMA foo!
/ Tungsten uses filters to rectify MySQL databases to
PostgreSQL schemas
© Continuent 2010

21. / MySQL: Trivial to use `USE`
/ MySQL: Going without `USE` generates different
events
MySQL Implicit MySQL Explicit
CREATE SCHEMA s; CREATE SCHEMA s;
USE s;
! CREATE TABLE t (i int); CREATE TABLE s.t (i int);
! INSERT INTO t (1); INSERT INTO s.t (1);
/ PG: Extract the default schema from the event
/ PG: Set it before applying
MySQL PostgreSQL
USE s; > SET search_path TO s, "$user”;
© Continuent 2010

22. / Differences between DDL and DML statement SQL
dialects
/ Row Replication resolves issues rising from
differences in DML, but still leaves DDL to handle
/ Tungsten Replicator Filters come to the rescue!
• Simple to develop Java or JavaScript extensions
• Event structure IN -> Filter -> Event structure OUT
MySQL PostgreSQL
CREATE TABLE complex (id CREATE TABLE complex (id
INTEGER AUTO_INCREMENT SERIAL PRIMARY KEY, i INT);
PRIMARY KEY, i INT);
CREATE TABLE dt (i TINYINT); CREATE TABLE dt (i SMALLINT);
…
© Continuent 2010

23. / Statement replication: MySQL syntax is “permissive”
/ Embedded binary / alternate charsets
/ Different charsets for different clients
/ Row replication: database/table/column charsets
may differ
/ Answer: Stick with one character set throughout; use
row replication to move binary data
MySQL PostgreSQL
INSERT INTO embedded_blob ARGH!!! (SQL statement fails)
(key, data) VALUES (1, ‘?0^Es
0^0’’)
create table xlate(id int, d1 ARGH!!! (no way to translate to
varchar(25) character set common charset)
latin1, d2 varchar(25)
character set utf8);
© Continuent 2010

24. © Continuent 2010

25. © Continuent 2010

26. Logical Physical
MySQL Statement Based x
MySQL Row Based x
MySQL Mixed x
PostgreSQL WAL Shipping x
PostgreSQL Streaming Replication x
Filters (data transformation) possible + -
Different data/structure on slave + -
possible
/ A transaction is not accessible to the replicator under
physical replication
/ Tungsten Replicator manages WAL/Streaming
Replication
© Continuent 2010

27. Logical Physical
MySQL Statement Based x
MySQL Row Based x
MySQL Mixed x
PostgreSQL WAL Shipping x
PostgreSQL Streaming Replication x
Tungsten Replicator w/ x
PostgreSQLSlonyExtractor
Filters (data transformation) possible + -
Different data/structure on slave + -
possible
/ With PostgreSQLSlonyExtractor transaction goes
through the Replicator pipeline
© Continuent 2010

28. Master Slave
Replicator Replicator
Transaction Transaction
History Log History Log
Filters Filters
PostgreSQL
SlonyExtractor

29. © Continuent 2010

30. © Continuent 2010

31. Master Slave
Replicator Replicator
Transaction Transaction
History Log History Log
Filters Filters
PostgreSQL MySQLApplier
SlonyExtractor

32. © Continuent 2010

33. © Continuent 2010

34. © Continuent 2010

35. Open Source Commercial
http://tungsten-replicator.org sales@continuent.com
#tungsten @ irc.freenode.net
My Blog:
http://flyingclusters.blogspot.com
Continuent Web Site:
http://www.continuent.com
© Continuent 2010