• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Synchronise your data between MySQL and MongoDB
 

Synchronise your data between MySQL and MongoDB

on

  • 1,653 views

How to start replication between MySQL and MongoDB using Tungsten Replicator, an open source high performance replication engine.

How to start replication between MySQL and MongoDB using Tungsten Replicator, an open source high performance replication engine.

Statistics

Views

Total Views
1,653
Views on SlideShare
1,502
Embed Views
151

Actions

Likes
7
Downloads
18
Comments
0

5 Embeds 151

https://twitter.com 130
http://www.weebly.com 13
http://www.datascienceassn.org 6
https://stbeehive.oracle.com 1
http://cloud.feedly.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution-NonCommercial-NoDerivs LicenseCC Attribution-NonCommercial-NoDerivs LicenseCC Attribution-NonCommercial-NoDerivs License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Synchronise your data between MySQL and MongoDB Synchronise your data between MySQL and MongoDB Presentation Transcript

    • Synchronize your data between MySQL and MongoDB using Tungsten Replicator Giuseppe Maxia, Director of QA Continuent, Inc ©Continuent 2013 Tuesday, October 15, 13 1
    • About me • Giuseppe Maxia, a.k.a. "The Data Charmer" • Director of Quality Assurance, Continuent, Inc • 25+ years development and DB experience • Long timer MySQL community member. • Oracle ACE Director • Blog: http://datacharmer.blogspot.com • Twitter: @datacharmer ©Continuent 2013 Tuesday, October 15, 13 2 2
    • Introducing Continuent • The leading provider of clustering and replication for open source DBMS • Our Product: Continuent Tungsten • Clustering - Commercial-grade HA, performance scaling and data management for MySQL • Replication - Flexible, high-performance data movement ©Continuent 2013 Tuesday, October 15, 13 3 3
    • A Review of Tungsten Replicator ©Continuent 2013 Tuesday, October 15, 13 4 4
    • Tungsten Replicator Overview Master Replicator Download transactions via network DBMS Logs (Transactions + Metadata) Slave Replicator Apply using JDBC ©Continuent 2013 Tuesday, October 15, 13 THL THL (Transactions + Metadata) 5 5
    • Master Replication Service Pipeline Stage Extract Filter Stage Apply Extract Filter Apply tcp/ip Binlog MySQL Master ©Continuent 2013 Tuesday, October 15, 13 Slave Replicators Transaction History Log In-Memory Queue 6 6
    • Slave Replication Service Pipeline Stage Apply Extract Filter Stage Apply Extract Filter Apply tcp/ip Extract Filter Stage Master Replicator Transaction History Log ©Continuent 2013 Tuesday, October 15, 13 In-Memory Queue Slave DBMS 7 7
    • master-slave MySQL Oracle fan-in slave Oracle MySQL all-masters Heterogeneous Oracle MySQL MySQL Oracle star ©Continuent 2013 Tuesday, October 15, 13 8
    • MongoDB in a nutshell ©Continuent 2013 Tuesday, October 15, 13 9 9
    • What is MongoDB • • • • • • • ©Continuent 2013 Tuesday, October 15, 13 A non-relational database A document-oriented database Schema-free Open source High performance Scalable Developer-friendly (sort of) 10 10
    • What is MongoDB good for? • • • Storing large amount of unrelated data • IT IS NOT a drop-in replacement for a relational database ©Continuent 2013 Tuesday, October 15, 13 Data that can't be constrained in a schema Complement to relational data 11 11
    • Relational vs. document person p_id name age 1 Joe 30 2 Fred 23 3 Jack 26 p_id d_id 4 Sue 25 1 2 5 Pete 32 2 2 3 1 4 3 5 1 department d_id name 1 sales 2 dev 3 support ©Continuent 2013 Tuesday, October 15, 13 pers_dept Relational 12 12
    • Relational vs. document person p_id name age 1 Joe 30 2 Fred 23 3 Jack 26 p_id d_id 4 Sue 25 1 2 5 Pete 32 2 2 3 1 4 3 5 1 department d_id name 1 sales 2 dev 3 support ©Continuent 2013 Tuesday, October 15, 13 pers_dept Relational 13 13
    • Relational vs. document person p_id name age 1 Joe 30 2 Fred 23 3 Jack 26 p_id d_id 4 Sue 25 1 2 5 Pete 32 2 2 3 1 4 3 5 1 department d_id name 1 sales 2 dev 3 support ©Continuent 2013 Tuesday, October 15, 13 pers_dept Relational 14 14
    • Relational vs. document person p_id name age 1 Joe 30 2 Fred 23 3 Jack 26 p_id d_id 4 Sue 25 1 2 5 Pete 32 2 2 3 1 4 3 5 1 department d_id name 1 sales 2 dev 3 support ©Continuent 2013 Tuesday, October 15, 13 pers_dept Relational 15 15
    • Relational vs. document person p_id name age 1 Joe 30 2 Fred 23 3 Jack 26 p_id d_id 4 Sue 25 1 2 5 Pete 32 2 2 3 1 4 3 5 1 department d_id name 1 sales 2 dev 3 support ©Continuent 2013 Tuesday, October 15, 13 pers_dept Relational 16 16
    • Relational vs. document person p_id name age 1 Joe 30 2 Fred 23 3 Jack 26 p_id d_id 4 Sue 25 1 2 5 Pete 32 2 2 3 1 4 3 5 1 department d_id name 1 sales 2 dev 3 support ©Continuent 2013 Tuesday, October 15, 13 pers_dept Relational 17 17
    • Relational vs. document person _id name age department 1 Joe 30 dev 2 Fred 23 dev 3 Jack 26 sales 4 Sue 25 support 5 Pete 32 sales Document ©Continuent 2013 Tuesday, October 15, 13 18 18
    • How MongoDB keeps data • ©Continuent 2013 Tuesday, October 15, 13 three levels: • • • dbs collections documents 19 19
    • MongoDB insertion demo > show collections > > db.person.insert( {_id: 1, name: "Joe", age: 30, department: "dev"}) > show collections person system.indexes ©Continuent 2013 Tuesday, October 15, 13 20 20
    • MongoDB insertion demo > db.person.insert( "dev"}) > db.person.insert( "sales"}) > db.person.insert( "support"}) > db.person.insert( "sales"}) > db.person.find() { "_id" : 1, "name" { "_id" : 2, "name" { "_id" : 3, "name" { "_id" : 4, "name" { "_id" : 5, "name" ©Continuent 2013 Tuesday, October 15, 13 {_id: 2, name: "Fred", age: 23, department: {_id: 3, name: "Jack", age: 26, department: {_id: 4, name: "Sue", age: 25, department: {_id: 5, name: "Pete", age: 30, department: : : : : : "Joe", "age" : 30, "department" : "dev" } "Fred", "age" : 23, "department" : "dev" } "Jack", "age" : 26, "department" : "sales" } "Sue", "age" : 25, "department" : "support" } "Pete", "age" : 30, "department" : "sales" } 21 21
    • MySQL to MongoDB basics ©Continuent 2013 Tuesday, October 15, 13 22 22
    • Replication from MySQL to MongoDB • • • • • • • ©Continuent 2013 Tuesday, October 15, 13 Requires ROW-based-replication Replication happens by table There is no consolidation into "documents" DDL commands are ignored Statement commands are ignored Column names become document attributes enum and set columns are converted to strings 23 23
    • First example of replication # MySQL create schema oneschema; use oneschema ; create table myfirst( num int not null primary key, dt datetime, ts timestamp, going enum('yes', 'no')); # MongoDB > show dbs local 0.078125GB test 0.203125GB tungsten_mysql2mongodb 0.203125GB # NOTICE: no "oneschema" ©Continuent 2013 Tuesday, October 15, 13 24 24
    • Inserting data # MySQL insert into myfirst values (1, '2003-04-26 09:15:00', null, 'yes'); Query OK, 1 row affected (0.01 sec) select * from myfirst; +-----+---------------------+---------------------+-------+ | num | dt | ts | going | +-----+---------------------+---------------------+-------+ | 1 | 2003-04-26 09:15:00 | 2013-10-14 19:39:38 | yes | +-----+---------------------+---------------------+-------+ 1 row in set (0.00 sec) ©Continuent 2013 Tuesday, October 15, 13 25 25
    • Checking results in MongoDB # MongoDB > show dbs local 0.078125GB oneschema 0.203125GB test 0.203125GB tungsten_mysql2mongodb 0.203125GB > use oneschema switched to db oneschema > show collections myfirst system.indexes > db.myfirst.find() { "_id" : ObjectId("525c2c5af5d9ca820fcee01d"), "num" : "1", "dt" : "2003-04-26 11:15:00.0", "ts" : "2013-10-14 19:39:38.0", "going" : "yes" } ©Continuent 2013 Tuesday, October 15, 13 26 26
    • Another interesting insertion #MySQL create table t1(_id int not null primary key, c char(10)); insert into t1 values (1, 'abc'); select * from t1; +-----+------+ | _id | c | +-----+------+ | 1 | abc | +-----+------+ 1 row in set (0.00 sec) # MongoDB > db.t1.find() { "_id" : "1", "c" : "abc" } ©Continuent 2013 Tuesday, October 15, 13 27 27
    • More insertions # MySQL insert into t1 values (2,'def'), (3,'ghi'), (4,'jkl'), (5, 'mno'); Query OK, 4 rows affected (0.01 sec) Records: 4 Duplicates: 0 Warnings: 0 select * from t1; +-----+------+ | _id | c | +-----+------+ | 1 | abc | | 2 | def | | 3 | ghi | | 4 | jkl | | 5 | mno | +-----+------+ 5 rows in set (0.00 sec) ©Continuent 2013 Tuesday, October 15, 13 28 28
    • More insertions # MongoDB > { { { { { db.t1.find() "_id" : "1", "_id" : "2", "_id" : "3", "_id" : "4", "_id" : "5", ©Continuent 2013 Tuesday, October 15, 13 "c" "c" "c" "c" "c" : : : : : "abc" "def" "ghi" "jkl" "mno" } } } } } 29 29
    • Update and delete as seen on master update t1 set c = 'ZZZ' where _id = 3; Query OK, 1 row affected (0.00 sec) Rows matched: 1 Changed: 1 Warnings: 0 delete from t1 where _id=2; Query OK, 1 row affected (0.00 sec) select * from t1; +-----+------+ | _id | c | +-----+------+ | 1 | abc | | 3 | ZZZ | | 4 | jkl | | 5 | mno | +-----+------+ 4 rows in set (0.00 sec) ©Continuent 2013 Tuesday, October 15, 13 30 30
    • e!ects of update and delete on the slave # MongoDB > { { { { db.t1.find() "_id" : "1", "_id" : "3", "_id" : "4", "_id" : "5", ©Continuent 2013 Tuesday, October 15, 13 "c" "c" "c" "c" : : : : "abc" "ZZZ" "jkl" "mno" } } } } 31 31
    • Overview of Tungsten installer ©Continuent 2013 Tuesday, October 15, 13 32 32
    • Overview of Installation Process 1. Set up hosts 2. Prepare MySQL replicas 3. Download software 4. Install using tpm Amazon Setup: https://docs.continuent.com/wiki/display/TEDOC/ Preparing+EC2+Servers ©Continuent 2013 Tuesday, October 15, 13 33 33
    • How tungsten-installer Works for Basic Master/Slave Deployment Staging copy of files db1 db2 check prereqs copy code configure ©Continuent 2013 Tuesday, October 15, 13 db3 34 34
    • Tungsten master/slave replication alpha THL THL host1 alpha host2 alpha THL host3 installer ©Continuent 2013 Tuesday, October 15, 13 35 35
    • Bi-directional replication alpha alpha bravo bravo host1 host2 installer Install all master and slave services on all hosts at once ©Continuent 2013 Tuesday, October 15, 13 36 36
    • 4 nodes all-masters alpha alpha bravo host1 bravo charlie charlie delta host2 delta alpha bravo ©Continuent 2013 Tuesday, October 15, 13 bravo charlie charlie delta host3 alpha delta host4 37 37
    • Tungsten security layer • Tra!c encryption: Tuesday, October 15, 13 all data in transit (transaction history logs, or THL) is encrypted using SSL • • • ©Continuent 2013 • all administrative tra!c is encrypted with SSL Transparent to the user Independent of the database server (works also for heterogeneous replication) 38 38
    • Tungsten replicator without security THL alpha THL alpha alpha host2 plain text THL host1 THL host4 host3 master slave ©Continuent 2013 Tuesday, October 15, 13 alpha replicator services 39 39
    • Tungsten Replicator with security SSL alpha THL alpha THL THL alpha host2 SSL host1 SSL SSL slave ©Continuent 2013 Tuesday, October 15, 13 alpha host4 host3 master THL replicator services 40 40
    • Installing Master/Slave Replication ... alpha THL THL host1 alpha host2 alpha THL host3 ©Continuent 2013 Tuesday, October 15, 13 41 41
    • master/slave using tpm ./tools/tpm install alpha --topology=master-slave --home-directory=/opt/continuent/replicator --replication-user=tungsten --replication-password=secret --master=host1 --slaves=host2,host3,host4 --start ©Continuent 2013 Tuesday, October 15, 13 42 42
    • Installing Master/Slave Replication with MongoDB alpha alpha host1 host2 alpha alpha host3 ©Continuent 2013 Tuesday, October 15, 13 host4 43 43
    • master/slave with MongoDB ./tools/tpm configure mysql2mongodb --enable-heterogenous-service=true --topology=master-slave --master=host1 --replication-user=tungsten --replication-password=secret --slaves=host2,host3,host4 --home-directory=$MYSQL_DEPLOY --start-and-report ./tools/tpm configure mysql2mongodb --hosts=host4 --datasource-type=mongodb --replication-port=$MONGODB_PORT ./tools/tpm install ©Continuent 2013 Tuesday, October 15, 13 44 44
    • Installing Fan-In Replication alpha bravo host1 host2 alpha charlie bravo host3 ©Continuent 2013 Tuesday, October 15, 13 host4 charlie 45 45
    • fan-in using tpm ./tools/tpm install many_towns --replication-user=tungsten --replication-password=secret --home-directory=/opt/continuent/replication --masters=host1,host2,host3 --slaves=host4 --master-services=alpha,bravo,charlie --topology=fan-in --start ©Continuent 2013 Tuesday, October 15, 13 46 46
    • Installing Fan-In Replication with MongoDB alpha bravo host1 host2 alpha bravo host3 ©Continuent 2013 Tuesday, October 15, 13 charlie charlie host4 47 47
    • fan-in with MongoDB ./tools/tpm configure mysql2mongodb --enable-heterogenous-service=true --topology=fan-in --masters=host1,host2,host3 --master-services=alpha,bravo,charlie --slaves=host4 --replication-user=tungsten --replication-password=secret --home-directory=$MYSQL_DEPLOY --datasource-type=mysql --start-and-report ./tools/tpm configure mysql2mongodb --hosts=host4 --datasource-type=mongodb --replication-port=$MONGODB_PORT ./tools/tpm install ©Continuent 2013 Tuesday, October 15, 13 48 48
    • Install Multi-Master replication alpha alpha bravo host1 bravo charlie charlie host2 alpha bravo host3 ©Continuent 2013 Tuesday, October 15, 13 charlie 49 49
    • multi-master using tpm ../tools/tpm install musketeers --reset --topology=all-masters --home-directory=/opt/continuent/replicator --replication-user=tungsten --replication-password=secret --masters=host1,host2,host3 --master-services=alpha,bravo,charlie --start ©Continuent 2013 Tuesday, October 15, 13 50 50
    • Install Multi-Master replication with Mongodb alpha alpha bravo host1 bravo charlie charlie alpha host3 ©Continuent 2013 Tuesday, October 15, 13 alpha bravo bravo charlie charlie host2 host4 51 51
    • multi-master with MongoDB ./tools/tpm configure mysql2mongodb --enable-heterogenous-service=true --topology=all-masters --masters=host1,host2,host3 --slaves=host1,host2,host3,host4 --master-services=alpha,bravo,charlie --replication-user=tungsten --replication-password=secret --home-directory=$MYSQL_DEPLOY --datasource-type=mysql --start-and-report ./tools/tpm configure mysql2mongodb --hosts=host4 --datasource-type=mongodb --replication-port=$MONGODB_PORT ./tools/tpm install ©Continuent 2013 Tuesday, October 15, 13 52 52
    • MongoDB or TokuMX • TokuMX is a drop-in replacement for MongoDB • • • Open source project, developed by TokuTek ©Continuent 2013 Tuesday, October 15, 13 https://github.com/Tokutek/mongo it includes • • • • better indexing row-level locking (MongoDB locks at db level) transactions better compression 53 53
    • DEMO: MongoDB and multi master installation ©Continuent 2013 Tuesday, October 15, 13 54 54
    • Joining the Community ©Continuent 2013 Tuesday, October 15, 13 55 55
    • Tungsten Replicator is Open Source • Project home: http://code.google.com/p/tungsten-replicator/ • Log bugs, "nd builds, post in replicator discussion group • Documentation: https://docs.continuent.com/wiki/display/TEDOC/ Tungsten+Documentation+Home https://docs.continuent.com/wiki/display/TEDOC/ Deploying+MongoDB+Replication ©Continuent 2013 Tuesday, October 15, 13 56 56
    • 560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel +1 (866) 998-3642 Fax +1 (408) 668-1009 e-mail: sales@continuent.com Our Blogs: http://scale-out-blog.blogspot.com http://datacharmer.blogspot.com http://www.continuent.com/news/blogs Continuent Web Page: http://www.continuent.com Tungsten Replicator 2.1: http://code.google.com/p/tungsten-replicator ©Continuent 2012. Tuesday, October 15, 13 57