Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Liberating Your Data From MySQL: Cross-Database Replication to the Rescue!

2,682 views

Published on

Countless petabytes of data are sitting in MySQL database where they are perfectly useless for PostgreSQL users. Fortunately there is a solution: Tungsten Replicator can move data from MySQL to PostgreSQL, and in real time, too. In this talk we'll describe how to design cross-database replication, then set it up using Tungsten Replicator. We will cover some of the pitfalls and corner cases like SQL dialect differences, data types, character sets, and MySQL bugs that make implementation both exciting and fun. We'll conclude with a demo of database updates moving in real time between databases.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Liberating Your Data From MySQL: Cross-Database Replication to the Rescue!

  1. 1. Robert Hodges and Linas Virbalas Continuent, Inc.© Continuent 2010
  2. 2. /  Introductions /  What is Tungsten? /  A Word About MySQL Replication /  Tungsten’s MySQL to PostgreSQL Replication /  Demo! /  Questions and Answers© Continuent 2010
  3. 3. © Continuent 2010
  4. 4. /  Our Value: Make open source as robust as commercial databases •  Enterprise capability •  Cloud flexibility •  Open source cost /  Our Solution: Tungsten Database Clustering /  Our Company: •  Founded in Finland, HQ in USA •  US/European presence •  Deep experience in replication, distributed systems, and RDBMS© Continuent 2010
  5. 5. What is Tungsten and How Does It Work?© Continuent 2010
  6. 6. What Is Tungsten? /  Tungsten creates data services using off- the-shelf open source databases to: 1.  Ensure reliable, fast access to data 2.  Simplify common management tasks 3.  Enable disaster recovery …Without changing applications© Continuent 2010
  7. 7. Apache/Mod_PHP Apache/Mod_PHP Connector Connector Replicator Replicator Replicator Manager Manager Manager© Continuent 2010
  8. 8. Technology: Replication Pipelines© Continuent 2010
  9. 9. So Why Are We Here?© Continuent 2010
  10. 10. Why Would You Want to Do That? /  Feed MySQL web transactions to PostgreSQL applications /  Feed PG-based data warehouse and reporting /  Migrate MySQL data to PG with minimum downtime /  Scott McNealy says we should share (a PG-West joke)© Continuent 2010
  11. 11. © Continuent 2010
  12. 12. Slave readable Slave writeable MySQL Statement Based + + MySQL Row Based + + MySQL Mixed + + PostgreSQL WAL Shipping - - PostgreSQL Streaming + - Replication (Hot Standby)/  Tungsten for MySQL •  You can both read and write to the DB operating in slave mode •  This, potentially, allows you to break consistency •  In some scenarios that is a plus/  Tungsten for PostgreSQL •  In Streaming Replication you can read from slaves •  Still, you cannot write to them •  In effect, you cannot have different data/structure on a slave© Continuent 2010
  13. 13. /  Statement Based Replication /  Row Based Replication© Continuent 2010
  14. 14. Logical Physical MySQL Statement Based x MySQL Row Based x MySQL Mixed x PostgreSQL WAL Shipping x PostgreSQL Streaming Replication x Filters (data transformation) possible + - Different data/structure on slave + - possible© Continuent 2010
  15. 15. © Continuent 2010
  16. 16. © Continuent 2010
  17. 17. Tungsten Tungsten Replicator Replicator© Continuent 2010
  18. 18. Master Slave Replicator Replicator Transaction Transaction History Log History Log Filters Filters MySQL PostgreSQL Extractor Applier© Continuent 2010
  19. 19. /  Provisioning /  Data Type Differences /  Database vs. Schema /  Default (Implicitly Defined) Schema Selection /  SQL Dialect Differences •  Statement Replication vs. Row Replication /  Character Sets and Binary Data /  Old Versions of MySQL© Continuent 2010
  20. 20. Provisioning /  Harder way: Dump data explicitly /  Easier way: Replicate a mysqldump backup Replicator© Continuent 2010
  21. 21. /  Note the type differences between MySQL and PG MySQL PostgreSQL ! TINYINT SMALLINT SMALLINT SMALLINT INTEGER INTEGER BIGINT BIGINT ! CHAR(1) CHAR(5) = {‘true’, ‘false’} CHAR(x) CHAR(x) VARCHAR(x) VARCHAR(x) DATE DATE TIMESTAMP TIMESTAMP ! TEXT (diff. sizes) TEXT ! BLOB BYTEA …© Continuent 2010
  22. 22. Database vs. Schema /  In MySQL these are the same: ! !CREATE DATABASE foo! ! !CREATE SCHEMA foo! /  In PostgreSQL these are very different: CREATE DATABASE foo! ! !CREATE SCHEMA foo! /  Tungsten uses filters to rectify MySQL databases to PostgreSQL schemas© Continuent 2010
  23. 23. /  MySQL: Trivial to use `USE` /  MySQL: Going without `USE` generates different events MySQL Implicit MySQL Explicit CREATE SCHEMA s; CREATE SCHEMA s; USE s; ! CREATE TABLE t (i int); CREATE TABLE s.t (i int); ! INSERT INTO t (1); INSERT INTO s.t (1); /  PG: Extract the default schema from the event /  PG: Set it before applying MySQL PostgreSQL USE s; > SET search_path TO s, "$user”;© Continuent 2010
  24. 24. /  Differences between DDL and DML statement SQL dialects /  Row Replication resolves issues rising from differences in DML, but still leaves DDL to handle /  Tungsten Replicator Filters come to the rescue! •  Simple to develop Java or JavaScript extensions •  Event structure IN -> Filter -> Event structure OUT MySQL PostgreSQL CREATE TABLE complex (id CREATE TABLE complex (id INTEGER AUTO_INCREMENT SERIAL PRIMARY KEY, i INT); PRIMARY KEY, i INT); CREATE TABLE dt (i TINYINT); CREATE TABLE dt (i SMALLINT); …© Continuent 2010
  25. 25. /  Statement replication: MySQL syntax is “permissive” /  Embedded binary / alternate charsets /  Different charsets for different clients /  Row replication: database/table/column charsets may differ /  Answer: Stick with one character set throughout; use row replication to move binary data MySQL PostgreSQL INSERT INTO embedded_blob ARGH!!! (SQL statement fails) (key, data) VALUES (1, ‘?0^Es 0^0’’) create table xlate(id int, d1 ARGH!!! (no way to translate to varchar(25) character set common charset) latin1, d2 varchar(25) character set utf8);© Continuent 2010
  26. 26. MySQL Versions /  Problem: Data stored on hard-to-replicate MySQL versions or configurations •  Row replication not enabled (5.1) •  No row replication support (5.0, 4.1) •  Tungsten cannot read binlog (4.1) /  Answer: MySQL blackhole replication •  (Blackhole = no store, just a binlog) •  Caveat: Check MySQL docs carefully Replicator© Continuent 2010
  27. 27. What’s Left? /  We have covered the basics but there is more… /  Data transformation (more Javascript filtering) /  Making it go *really* fast (think parallel replication)© Continuent 2010
  28. 28. © Continuent 2010
  29. 29. © Continuent 2010
  30. 30. HQ and Americas EMEA and APAC 560 S. Winchester Blvd., Suite 500 Lars Sonckin kaari 16 San Jose, CA 95128 02600 Espoo, Finland Tel (866) 998-3642 Tel +358 50 517 9059 Fax (408) 668-1009 Fax +358 9 863 0060 e-mail: sales@continuent.com Continuent Web Site: http://www.continuent.com© Continuent 2010

×