Flexible Replication

Flexible Replication

• 1.
  Linas Virbalas Continuent, Inc. © Continuent 2010
• 2.
  /  Definition &Motivation /  Three Stories /  How? /  Questions and Answers © Continuent 2010
• 3.
  © Continuent 2010
• 4.
  Heterogeneous Replication  Replication between different types of DBMS © Continuent 2010
• 5.
  1.  Real-time integrationof data between different DBMS types 2.  Seamless migration out of one DBMS type to another 3.  Data warehousing (real-time) from different DBMS types © Continuent 2010
• 6.
  /  Name: Linas Virbalas /  Country: Lithuania /  Implementing for Tungsten: •  MySQL -> PostgreSQL •  MySQL -> Greenplum •  MySQL -> Oracle •  PostgreSQL WAL •  PostgreSQL Streaming Replication •  PostgreSQL Logical Replication via Slony logs /  Blog: http://flyingclusters.blogspot.com © Continuent 2010
• 7.
  © Continuent 2010
• 8.
  1.  Story ofMySQL -> … •  How to replicate from MySQL to PostgreSQL/Greenplum? Oracle? 2.  Story of PostgreSQL -> PostgreSQL •  Advanced Logical Replication 3.  Story of PostgreSQL -> … •  For now: how to replicate from PostgreSQL to MySQL © Continuent 2010
• 9.
  With Tungsten Replicator Why? Because It Can! :) © Continuent 2010
• 10.
  /  Interfaces to implement new: •  Extractors •  Filters •  Appliers /  Multiple replication services per one process /  JAVA /  Open Source GPL v2 © Continuent 2010
• 11.
  Technology: Replication Pipelines ©Continuent 2010
• 12.
  © Continuent 2010
• 13.
  /  Statement BasedReplication /  Row Based Replication © Continuent 2010
• 14.
  © Continuent 2010
• 15.
  Tungsten Tungsten Replicator Replicator © Continuent 2010
• 16.
  Master Slave Replicator Replicator Transaction Transaction History Log History Log Filters Filters MySQL PostgreSQL Extractor Applier © Continuent 2010
• 17.
  /  Provisioning /  Data Type Differences /  Database vs. Schema /  Default (Implicitly Defined) Schema Selection /  SQL Dialect Differences •  Statement Replication vs. Row Replication /  Character Sets and Binary Data /  Old Versions of MySQL © Continuent 2010
• 18.
  Provisioning /  Harderway: Dump data explicitly /  Easier way: Replicate a mysqldump backup Replicator © Continuent 2010
• 19.
  /  Note thetype differences between MySQL and PG MySQL PostgreSQL ! TINYINT SMALLINT SMALLINT SMALLINT INTEGER INTEGER BIGINT BIGINT ! CHAR(1) CHAR(5) = {‘true’, ‘false’} CHAR(x) CHAR(x) VARCHAR(x) VARCHAR(x) DATE DATE TIMESTAMP TIMESTAMP ! TEXT (diff. sizes) TEXT ! BLOB BYTEA … © Continuent 2010
• 20.
  Database vs. Schema /  In MySQL these are the same: ! !CREATE DATABASE foo! ! !CREATE SCHEMA foo! /  In PostgreSQL these are very different: CREATE DATABASE foo! ! !CREATE SCHEMA foo! /  Tungsten uses filters to rectify MySQL databases to PostgreSQL schemas © Continuent 2010
• 21.
  /  MySQL: Trivialto use `USE` /  MySQL: Going without `USE` generates different events MySQL Implicit MySQL Explicit CREATE SCHEMA s; CREATE SCHEMA s; USE s; ! CREATE TABLE t (i int); CREATE TABLE s.t (i int); ! INSERT INTO t (1); INSERT INTO s.t (1); /  PG: Extract the default schema from the event /  PG: Set it before applying MySQL PostgreSQL USE s; > SET search_path TO s, "$user”; © Continuent 2010
• 22.
  /  Differences betweenDDL and DML statement SQL dialects /  Row Replication resolves issues rising from differences in DML, but still leaves DDL to handle /  Tungsten Replicator Filters come to the rescue! •  Simple to develop Java or JavaScript extensions •  Event structure IN -> Filter -> Event structure OUT MySQL PostgreSQL CREATE TABLE complex (id CREATE TABLE complex (id INTEGER AUTO_INCREMENT SERIAL PRIMARY KEY, i INT); PRIMARY KEY, i INT); CREATE TABLE dt (i TINYINT); CREATE TABLE dt (i SMALLINT); … © Continuent 2010
• 23.
  /  Statement replication:MySQL syntax is “permissive” /  Embedded binary / alternate charsets /  Different charsets for different clients /  Row replication: database/table/column charsets may differ /  Answer: Stick with one character set throughout; use row replication to move binary data MySQL PostgreSQL INSERT INTO embedded_blob ARGH!!! (SQL statement fails) (key, data) VALUES (1, ‘?0^Es 0^0’’) create table xlate(id int, d1 ARGH!!! (no way to translate to varchar(25) character set common charset) latin1, d2 varchar(25) character set utf8); © Continuent 2010
• 24.
  © Continuent 2010
• 25.
  © Continuent 2010
• 26.
  Logical Physical MySQL Statement Based x MySQL Row Based x MySQL Mixed x PostgreSQL WAL Shipping x PostgreSQL Streaming Replication x Filters (data transformation) possible + - Different data/structure on slave + - possible /  A transaction is not accessible to the replicator under physical replication /  Tungsten Replicator manages WAL/Streaming Replication © Continuent 2010
• 27.
  Logical Physical MySQL Statement Based x MySQL Row Based x MySQL Mixed x PostgreSQL WAL Shipping x PostgreSQL Streaming Replication x Tungsten Replicator w/ x PostgreSQLSlonyExtractor Filters (data transformation) possible + - Different data/structure on slave + - possible /  With PostgreSQLSlonyExtractor transaction goes through the Replicator pipeline © Continuent 2010
• 28.
  Master Slave Replicator Replicator Transaction Transaction History Log History Log Filters Filters PostgreSQL SlonyExtractor
• 29.
  © Continuent 2010
• 30.
  © Continuent 2010
• 31.
  Master Slave Replicator Replicator Transaction Transaction History Log History Log Filters Filters PostgreSQL MySQLApplier SlonyExtractor
• 32.
  © Continuent 2010
• 33.
  © Continuent 2010
• 34.
  © Continuent 2010
• 35.
  Open Source Commercial http://tungsten-replicator.org sales@continuent.com #tungsten @ irc.freenode.net My Blog: http://flyingclusters.blogspot.com Continuent Web Site: http://www.continuent.com © Continuent 2010