Robert Hodges and Linas Virbalas                    Continuent, Inc.© Continuent 2010
/  Introductions  /  What is Tungsten?  /  A Word About MySQL Replication  /  Tungsten’s MySQL to PostgreSQL Replication  ...
© Continuent 2010
/  Our Value: Make open source as robust as     commercial databases           •  Enterprise capability           •  Cloud...
What is Tungsten and How                    Does It Work?© Continuent 2010
What Is Tungsten?  /  Tungsten creates data services using off-     the-shelf open source databases to:           1.  Ensu...
Apache/Mod_PHP                             Apache/Mod_PHP                                 Connector                 Connec...
Technology: Replication Pipelines© Continuent 2010
So Why Are We Here?© Continuent 2010
Why Would You Want to Do That?  /  Feed MySQL web transactions to PostgreSQL     applications  /  Feed PG-based data wareh...
© Continuent 2010
Slave readable    Slave                                                                 writeable                     MySQ...
/  Statement Based Replication  /  Row Based Replication© Continuent 2010
Logical   Physical                           MySQL Statement Based           x                                 MySQL Row B...
© Continuent 2010
© Continuent 2010
Tungsten     Tungsten                    Replicator   Replicator© Continuent 2010
Master         Slave                    Replicator    Replicator                    Transaction   Transaction             ...
/  Provisioning  /  Data Type Differences  /  Database vs. Schema  /  Default (Implicitly Defined) Schema Selection  /  SQ...
Provisioning /  Harder way: Dump data explicitly /  Easier way: Replicate a mysqldump backup                          Repl...
/  Note the type differences between MySQL and PG                        MySQL                PostgreSQL                  ...
Database vs. Schema  /  In MySQL these are the same:    ! !CREATE DATABASE foo!    ! !CREATE SCHEMA foo!  /  In PostgreSQL...
/  MySQL: Trivial to use `USE`  /  MySQL: Going without `USE` generates different     events                        MySQL ...
/  Differences between DDL and DML statement SQL     dialects  /  Row Replication resolves issues rising from     differen...
/  Statement replication: MySQL syntax is “permissive”      /  Embedded binary / alternate charsets      /  Different char...
MySQL Versions  /  Problem: Data stored on hard-to-replicate MySQL     versions or configurations           •  Row replica...
What’s Left?  /  We have covered the basics but there is more…  /  Data transformation (more Javascript filtering)  /  Mak...
© Continuent 2010
© Continuent 2010
HQ and Americas                      EMEA and APAC   560 S. Winchester Blvd., Suite 500   Lars Sonckin kaari 16   San Jose...
Upcoming SlideShare
Loading in …5
×

Liberating Your Data From MySQL: Cross-Database Replication to the Rescue!

2,487 views

Published on

Countless petabytes of data are sitting in MySQL database where they are perfectly useless for PostgreSQL users. Fortunately there is a solution: Tungsten Replicator can move data from MySQL to PostgreSQL, and in real time, too. In this talk we'll describe how to design cross-database replication, then set it up using Tungsten Replicator. We will cover some of the pitfalls and corner cases like SQL dialect differences, data types, character sets, and MySQL bugs that make implementation both exciting and fun. We'll conclude with a demo of database updates moving in real time between databases.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,487
On SlideShare
0
From Embeds
0
Number of Embeds
23
Actions
Shares
0
Downloads
31
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Liberating Your Data From MySQL: Cross-Database Replication to the Rescue!

  1. 1. Robert Hodges and Linas Virbalas Continuent, Inc.© Continuent 2010
  2. 2. /  Introductions /  What is Tungsten? /  A Word About MySQL Replication /  Tungsten’s MySQL to PostgreSQL Replication /  Demo! /  Questions and Answers© Continuent 2010
  3. 3. © Continuent 2010
  4. 4. /  Our Value: Make open source as robust as commercial databases •  Enterprise capability •  Cloud flexibility •  Open source cost /  Our Solution: Tungsten Database Clustering /  Our Company: •  Founded in Finland, HQ in USA •  US/European presence •  Deep experience in replication, distributed systems, and RDBMS© Continuent 2010
  5. 5. What is Tungsten and How Does It Work?© Continuent 2010
  6. 6. What Is Tungsten? /  Tungsten creates data services using off- the-shelf open source databases to: 1.  Ensure reliable, fast access to data 2.  Simplify common management tasks 3.  Enable disaster recovery …Without changing applications© Continuent 2010
  7. 7. Apache/Mod_PHP Apache/Mod_PHP Connector Connector Replicator Replicator Replicator Manager Manager Manager© Continuent 2010
  8. 8. Technology: Replication Pipelines© Continuent 2010
  9. 9. So Why Are We Here?© Continuent 2010
  10. 10. Why Would You Want to Do That? /  Feed MySQL web transactions to PostgreSQL applications /  Feed PG-based data warehouse and reporting /  Migrate MySQL data to PG with minimum downtime /  Scott McNealy says we should share (a PG-West joke)© Continuent 2010
  11. 11. © Continuent 2010
  12. 12. Slave readable Slave writeable MySQL Statement Based + + MySQL Row Based + + MySQL Mixed + + PostgreSQL WAL Shipping - - PostgreSQL Streaming + - Replication (Hot Standby)/  Tungsten for MySQL •  You can both read and write to the DB operating in slave mode •  This, potentially, allows you to break consistency •  In some scenarios that is a plus/  Tungsten for PostgreSQL •  In Streaming Replication you can read from slaves •  Still, you cannot write to them •  In effect, you cannot have different data/structure on a slave© Continuent 2010
  13. 13. /  Statement Based Replication /  Row Based Replication© Continuent 2010
  14. 14. Logical Physical MySQL Statement Based x MySQL Row Based x MySQL Mixed x PostgreSQL WAL Shipping x PostgreSQL Streaming Replication x Filters (data transformation) possible + - Different data/structure on slave + - possible© Continuent 2010
  15. 15. © Continuent 2010
  16. 16. © Continuent 2010
  17. 17. Tungsten Tungsten Replicator Replicator© Continuent 2010
  18. 18. Master Slave Replicator Replicator Transaction Transaction History Log History Log Filters Filters MySQL PostgreSQL Extractor Applier© Continuent 2010
  19. 19. /  Provisioning /  Data Type Differences /  Database vs. Schema /  Default (Implicitly Defined) Schema Selection /  SQL Dialect Differences •  Statement Replication vs. Row Replication /  Character Sets and Binary Data /  Old Versions of MySQL© Continuent 2010
  20. 20. Provisioning /  Harder way: Dump data explicitly /  Easier way: Replicate a mysqldump backup Replicator© Continuent 2010
  21. 21. /  Note the type differences between MySQL and PG MySQL PostgreSQL ! TINYINT SMALLINT SMALLINT SMALLINT INTEGER INTEGER BIGINT BIGINT ! CHAR(1) CHAR(5) = {‘true’, ‘false’} CHAR(x) CHAR(x) VARCHAR(x) VARCHAR(x) DATE DATE TIMESTAMP TIMESTAMP ! TEXT (diff. sizes) TEXT ! BLOB BYTEA …© Continuent 2010
  22. 22. Database vs. Schema /  In MySQL these are the same: ! !CREATE DATABASE foo! ! !CREATE SCHEMA foo! /  In PostgreSQL these are very different: CREATE DATABASE foo! ! !CREATE SCHEMA foo! /  Tungsten uses filters to rectify MySQL databases to PostgreSQL schemas© Continuent 2010
  23. 23. /  MySQL: Trivial to use `USE` /  MySQL: Going without `USE` generates different events MySQL Implicit MySQL Explicit CREATE SCHEMA s; CREATE SCHEMA s; USE s; ! CREATE TABLE t (i int); CREATE TABLE s.t (i int); ! INSERT INTO t (1); INSERT INTO s.t (1); /  PG: Extract the default schema from the event /  PG: Set it before applying MySQL PostgreSQL USE s; > SET search_path TO s, "$user”;© Continuent 2010
  24. 24. /  Differences between DDL and DML statement SQL dialects /  Row Replication resolves issues rising from differences in DML, but still leaves DDL to handle /  Tungsten Replicator Filters come to the rescue! •  Simple to develop Java or JavaScript extensions •  Event structure IN -> Filter -> Event structure OUT MySQL PostgreSQL CREATE TABLE complex (id CREATE TABLE complex (id INTEGER AUTO_INCREMENT SERIAL PRIMARY KEY, i INT); PRIMARY KEY, i INT); CREATE TABLE dt (i TINYINT); CREATE TABLE dt (i SMALLINT); …© Continuent 2010
  25. 25. /  Statement replication: MySQL syntax is “permissive” /  Embedded binary / alternate charsets /  Different charsets for different clients /  Row replication: database/table/column charsets may differ /  Answer: Stick with one character set throughout; use row replication to move binary data MySQL PostgreSQL INSERT INTO embedded_blob ARGH!!! (SQL statement fails) (key, data) VALUES (1, ‘?0^Es 0^0’’) create table xlate(id int, d1 ARGH!!! (no way to translate to varchar(25) character set common charset) latin1, d2 varchar(25) character set utf8);© Continuent 2010
  26. 26. MySQL Versions /  Problem: Data stored on hard-to-replicate MySQL versions or configurations •  Row replication not enabled (5.1) •  No row replication support (5.0, 4.1) •  Tungsten cannot read binlog (4.1) /  Answer: MySQL blackhole replication •  (Blackhole = no store, just a binlog) •  Caveat: Check MySQL docs carefully Replicator© Continuent 2010
  27. 27. What’s Left? /  We have covered the basics but there is more… /  Data transformation (more Javascript filtering) /  Making it go *really* fast (think parallel replication)© Continuent 2010
  28. 28. © Continuent 2010
  29. 29. © Continuent 2010
  30. 30. HQ and Americas EMEA and APAC 560 S. Winchester Blvd., Suite 500 Lars Sonckin kaari 16 San Jose, CA 95128 02600 Espoo, Finland Tel (866) 998-3642 Tel +358 50 517 9059 Fax (408) 668-1009 Fax +358 9 863 0060 e-mail: sales@continuent.com Continuent Web Site: http://www.continuent.com© Continuent 2010

×