Synchronise your data between MySQL and MongoDB

Synchronize your data
between MySQL
and MongoDB
using Tungsten Replicator
Giuseppe Maxia, Director of QA
Continuent, Inc

©Continuent 2013
Tuesday, October 15, 13

1

About me

•

Giuseppe Maxia, a.k.a. "The Data Charmer"

• Director of Quality Assurance, Continuent, Inc
• 25+ years development and DB experience
• Long timer MySQL community member.
• Oracle ACE Director
• Blog: http://datacharmer.blogspot.com
• Twitter: @datacharmer

©Continuent 2013

2
2

Introducing Continuent

•

The leading provider of clustering and
replication for open source DBMS

•

Our Product: Continuent Tungsten

• Clustering - Commercial-grade HA, performance
scaling and data management for MySQL

• Replication - Flexible, high-performance data
movement

©Continuent 2013

3
3

A Review of Tungsten Replicator

©Continuent 2013

4
4

Tungsten Replicator Overview
Master
Replicator

Download
transactions
via network
DBMS
Logs

(Transactions + Metadata)

Slave

Replicator

Apply using JDBC

©Continuent 2013

THL

THL
(Transactions + Metadata)

5
5

Master Replication Service
Pipeline
Stage
Extract Filter

Stage
Apply

Extract Filter

Apply

tcp/ip

Binlog
MySQL
Master

©Continuent 2013

Slave
Replicators

Transaction
History Log

In-Memory
Queue

6
6

Slave Replication Service
Pipeline
Stage
Apply

Extract Filter

Stage
Apply

Extract Filter

Apply

tcp/ip

Extract Filter

Stage

Master
Replicator
Transaction
History Log

©Continuent 2013

In-Memory
Queue

Slave
DBMS

7
7

master-slave
MySQL

Oracle
fan-in slave

Oracle

MySQL
all-masters

Heterogeneous

Oracle

MySQL

MySQL

Oracle
star

©Continuent 2013

8

MongoDB in a nutshell

©Continuent 2013

9
9

What is MongoDB

•
•
•
•
•
•
•
©Continuent 2013

A non-relational database
A document-oriented database
Schema-free
Open source
High performance
Scalable
Developer-friendly (sort of)

10
10

What is MongoDB good for?

•
•
•

Storing large amount of unrelated data

•

IT IS NOT a drop-in replacement for a
relational database

©Continuent 2013

Data that can't be constrained in a schema
Complement to relational data

11
11

Relational vs. document
person
p_id name age
1

Joe

30

2

Fred

23

3

Jack

26

p_id

d_id

4

Sue

25

1

2

5

Pete

32

2

2

3

1

4

3

5

1

department
d_id
name
1
sales
2
dev
3
support

©Continuent 2013

pers_dept

Relational

12
12

person
p_id name age
1

Joe

30

2

Fred

23

3

Jack

26

p_id

d_id

4

Sue

25

1

2

5

Pete

32

2

2

3

1

4

3

5

1

department
d_id
name
1
sales
2
dev
3
support

©Continuent 2013

pers_dept

Relational

13
13

person
p_id name age
1

Joe

30

2

Fred

23

3

Jack

26

p_id

d_id

4

Sue

25

1

2

5

Pete

32

2

2

3

1

4

3

5

1

department
d_id
name
1
sales
2
dev
3
support

©Continuent 2013

pers_dept

Relational

14
14

person
p_id name age
1

Joe

30

2

Fred

23

3

Jack

26

p_id

d_id

4

Sue

25

1

2

5

Pete

32

2

2

3

1

4

3

5

1

department
d_id
name
1
sales
2
dev
3
support

©Continuent 2013

pers_dept

Relational

15
15

person
p_id name age
1

Joe

30

2

Fred

23

3

Jack

26

p_id

d_id

4

Sue

25

1

2

5

Pete

32

2

2

3

1

4

3

5

1

department
d_id
name
1
sales
2
dev
3
support

©Continuent 2013

pers_dept

Relational

16
16

person
p_id name age
1

Joe

30

2

Fred

23

3

Jack

26

p_id

d_id

4

Sue

25

1

2

5

Pete

32

2

2

3

1

4

3

5

1

department
d_id
name
1
sales
2
dev
3
support

©Continuent 2013

pers_dept

Relational

17
17

person
_id

name

age

department

1

Joe

30

dev

2

Fred

23

dev

3

Jack

26

sales

4

Sue

25

support

5

Pete

32

sales

Document
©Continuent 2013

18
18

How MongoDB keeps data

•

©Continuent 2013

three levels:

•
•
•

dbs
collections
documents

19
19

MongoDB insertion demo
> show collections
>
> db.person.insert( {_id: 1, name: "Joe", age: 30, department:
"dev"})
> show collections
person
system.indexes

©Continuent 2013

20
20

MongoDB insertion demo
> db.person.insert(
"dev"})
> db.person.insert(
"sales"})
> db.person.insert(
"support"})
> db.person.insert(
"sales"})
> db.person.find()
{ "_id" : 1, "name"
{ "_id" : 2, "name"
{ "_id" : 3, "name"
{ "_id" : 4, "name"
{ "_id" : 5, "name"

©Continuent 2013

{_id: 2, name: "Fred", age: 23, department:
{_id: 3, name: "Jack", age: 26, department:
{_id: 4, name: "Sue", age: 25, department:
{_id: 5, name: "Pete", age: 30, department:
:
:
:
:
:

"Joe", "age" : 30, "department" : "dev" }
"Fred", "age" : 23, "department" : "dev" }
"Jack", "age" : 26, "department" : "sales" }
"Sue", "age" : 25, "department" : "support" }
"Pete", "age" : 30, "department" : "sales" }

21
21

MySQL to MongoDB basics

©Continuent 2013

22
22

Replication from MySQL to MongoDB

•
•
•
•
•
•
•
©Continuent 2013

Requires ROW-based-replication
Replication happens by table
There is no consolidation into "documents"
DDL commands are ignored
Statement commands are ignored
Column names become document attributes
enum and set columns are converted to
strings
23
23

First example of replication
# MySQL
create schema oneschema;
use oneschema ;
create table myfirst( num int not null primary key,
dt datetime,
ts timestamp,
going enum('yes', 'no'));
# MongoDB
> show dbs
local 0.078125GB
test 0.203125GB
tungsten_mysql2mongodb

0.203125GB

# NOTICE: no "oneschema"
©Continuent 2013

24
24

Inserting data
# MySQL
insert into myfirst values (1, '2003-04-26 09:15:00', null, 'yes');
Query OK, 1 row affected (0.01 sec)
select * from myfirst;
+-----+---------------------+---------------------+-------+
| num | dt
| ts
| going |
+-----+---------------------+---------------------+-------+
|
1 | 2003-04-26 09:15:00 | 2013-10-14 19:39:38 | yes
|
+-----+---------------------+---------------------+-------+
1 row in set (0.00 sec)

©Continuent 2013

25
25

Checking results in MongoDB
# MongoDB
> show dbs
local 0.078125GB
oneschema 0.203125GB
test 0.203125GB
tungsten_mysql2mongodb

0.203125GB

> use oneschema
switched to db oneschema
> show collections
myfirst
system.indexes
> db.myfirst.find()
{ "_id" : ObjectId("525c2c5af5d9ca820fcee01d"), "num" : "1", "dt" :
"2003-04-26 11:15:00.0", "ts" : "2013-10-14 19:39:38.0", "going" :
"yes" }
©Continuent 2013

26
26

Another interesting insertion
#MySQL
create table t1(_id int not null primary key, c char(10));
insert into t1 values (1, 'abc');
select * from t1;
+-----+------+
| _id | c
|
+-----+------+
|
1 | abc |
+-----+------+
1 row in set (0.00 sec)
# MongoDB
> db.t1.find()
{ "_id" : "1", "c" : "abc" }

©Continuent 2013

27
27

More insertions
# MySQL
insert into t1 values (2,'def'), (3,'ghi'), (4,'jkl'), (5, 'mno');
Query OK, 4 rows affected (0.01 sec)
Records: 4 Duplicates: 0 Warnings: 0
select * from t1;
+-----+------+
| _id | c
|
+-----+------+
|
1 | abc |
|
2 | def |
|
3 | ghi |
|
4 | jkl |
|
5 | mno |
+-----+------+
5 rows in set (0.00 sec)

©Continuent 2013

28
28

More insertions
# MongoDB
>
{
{
{
{
{

db.t1.find()
"_id" : "1",
"_id" : "2",
"_id" : "3",
"_id" : "4",
"_id" : "5",

©Continuent 2013

"c"
"c"
"c"
"c"
"c"

:
:
:
:
:

"abc"
"def"
"ghi"
"jkl"
"mno"

}
}
}
}
}

29
29

Update and delete as seen on master
update t1 set c = 'ZZZ' where _id = 3;
Rows matched: 1 Changed: 1 Warnings: 0
delete from t1 where _id=2;
select * from t1;
+-----+------+
| _id | c
|
+-----+------+
|
1 | abc |
|
3 | ZZZ |
|
4 | jkl |
|
5 | mno |
+-----+------+
4 rows in set (0.00 sec)
©Continuent 2013

30
30

e!ects of update and delete on the
slave
# MongoDB
>
{
{
{
{

db.t1.find()
"_id" : "1",
"_id" : "3",
"_id" : "4",
"_id" : "5",

©Continuent 2013

"c"
"c"
"c"
"c"

:
:
:
:

"abc"
"ZZZ"
"jkl"
"mno"

}
}
}
}

31
31

Overview of Tungsten installer

©Continuent 2013

32
32

Overview of Installation Process
1. Set up hosts
2. Prepare MySQL replicas
3. Download software
4. Install using tpm

Amazon Setup:
https://docs.continuent.com/wiki/display/TEDOC/
Preparing+EC2+Servers

©Continuent 2013

33
33

How tungsten-installer Works for
Basic Master/Slave Deployment
Staging copy
of ﬁles

db1

db2

check prereqs
copy code
conﬁgure
©Continuent 2013

db3

34
34

Tungsten master/slave replication
alpha

THL

THL

host1

alpha

host2

alpha

THL

host3

installer

©Continuent 2013

35
35

Bi-directional replication
alpha

alpha

bravo

bravo

host1

host2

installer
Install all master and slave services on all hosts at once

©Continuent 2013

36
36

4 nodes all-masters
alpha

alpha

bravo

host1

bravo

charlie

charlie

delta

host2

delta

alpha
bravo

©Continuent 2013

bravo

charlie

charlie

delta

host3

alpha

delta

host4

37
37

Tungsten security layer

•

Tra!c encryption:


all data in transit (transaction history logs, or THL) is
encrypted using SSL

•
•
•

©Continuent 2013

•

all administrative tra!c is encrypted with SSL
Transparent to the user
Independent of the database server (works also for
heterogeneous replication)

38
38

Tungsten replicator without security
THL

alpha

THL

alpha

alpha

host2

plain text

THL

host1

THL

host4

host3

master
slave

©Continuent 2013

alpha

replicator
services

39
39

Tungsten Replicator with security
SSL
alpha

THL

alpha

THL

THL

alpha

host2

SSL

host1

SSL

SSL

slave

©Continuent 2013

alpha

host4

host3

master

THL

replicator
services

40
40

master/slave using tpm

./tools/tpm install alpha
--topology=master-slave
--home-directory=/opt/continuent/replicator
--replication-user=tungsten
--replication-password=secret
--master=host1
--slaves=host2,host3,host4
--start

©Continuent 2013

42
42

Installing Master/Slave Replication
with MongoDB
alpha

alpha

host1

host2

alpha

alpha

host3

©Continuent 2013

host4

43
43

master/slave with MongoDB
./tools/tpm configure mysql2mongodb
--enable-heterogenous-service=true
--topology=master-slave
--master=host1
--slaves=host2,host3,host4
--home-directory=$MYSQL_DEPLOY
--start-and-report
--hosts=host4
--datasource-type=mongodb
--replication-port=$MONGODB_PORT
./tools/tpm install

©Continuent 2013

44
44

Installing Fan-In Replication
alpha
bravo

host1

host2

alpha

charlie

bravo

host3

©Continuent 2013

host4

charlie

45
45

fan-in using tpm

./tools/tpm install many_towns
--home-directory=/opt/continuent/replication
--masters=host1,host2,host3
--slaves=host4
--master-services=alpha,bravo,charlie
--topology=fan-in
--start

©Continuent 2013

46
46

Installing Fan-In Replication with
MongoDB
alpha
bravo

host1

host2

alpha
bravo

host3

©Continuent 2013

charlie

charlie

host4

47
47

fan-in with MongoDB
--topology=fan-in
--slaves=host4
--datasource-type=mysql
--start-and-report
--hosts=host4
./tools/tpm install

©Continuent 2013

48
48

Install Multi-Master replication
alpha

alpha

bravo

host1

bravo

charlie

charlie

host2

alpha
bravo

host3

©Continuent 2013

charlie

49
49

multi-master using tpm

../tools/tpm install musketeers
--reset
--topology=all-masters
--home-directory=/opt/continuent/replicator
--start

©Continuent 2013

50
50

Install Multi-Master replication with
Mongodb
alpha

alpha

bravo

host1

bravo

charlie

charlie

alpha

host3

©Continuent 2013

alpha

bravo

bravo

charlie

charlie

host2

host4

51
51

multi-master with MongoDB
--topology=all-masters
--slaves=host1,host2,host3,host4
--datasource-type=mysql
--start-and-report
--hosts=host4
./tools/tpm install

©Continuent 2013

52
52

MongoDB or TokuMX

•

TokuMX is a drop-in replacement for
MongoDB

•
•
•

Open source project, developed by TokuTek

©Continuent 2013

https://github.com/Tokutek/mongo
it includes

•
•
•
•

better indexing
row-level locking (MongoDB locks at db level)
transactions
better compression
53
53

Tungsten Replicator is Open Source

•

Project home:
http://code.google.com/p/tungsten-replicator/

•

Log bugs, "nd builds, post in replicator discussion
group

•

Documentation:
Tungsten+Documentation+Home
Deploying+MongoDB+Replication

©Continuent 2013

56
56

560 S. Winchester Blvd., Suite 500
San Jose, CA 95128
Tel +1 (866) 998-3642
Fax +1 (408) 668-1009
e-mail: sales@continuent.com

Our Blogs:
http://scale-out-blog.blogspot.com
http://datacharmer.blogspot.com
http://www.continuent.com/news/blogs

Continuent Web Page:
http://www.continuent.com
Tungsten Replicator 2.1:
http://code.google.com/p/tungsten-replicator

©Continuent 2012.

57

Synchronise your data between MySQL and MongoDB

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Synchronise your data between MySQL and MongoDB

Similar to Synchronise your data between MySQL and MongoDB (20)

More from Giuseppe Maxia

More from Giuseppe Maxia (20)

Recently uploaded

Recently uploaded (20)

Synchronise your data between MySQL and MongoDB