Giuseppe Maxia, profile picture
Uploaded byGiuseppe Maxia
1,921 views

Moving data for the masses

The document discusses how database replication is important for scaling web applications and high availability. It describes how database replication can be used for load balancing reads, sharding writes, and replacing masters during failures. While basic MySQL replication provides some capabilities, it has limitations in supporting multiple masters, conflict prevention, and replicating to different database types. The Tungsten Replicator aims to address these limitations by providing features like parallel replication across databases, seamless failover, and replicating from MySQL to other databases like PostgreSQL. It benchmarks show that Tungsten Replicator's parallel replication approach can significantly reduce the time to catch up slave databases compared to native MySQL replication.

Embed presentation

Downloaded 39 times
1 / 141
2 / 141
3 / 141
4 / 141
5 / 141
6 / 141
7 / 141
8 / 141
9 / 141
10 / 141
11 / 141
12 / 141
13 / 141
14 / 141
15 / 141
16 / 141
17 / 141
18 / 141
19 / 141
20 / 141
21 / 141
22 / 141
23 / 141
24 / 141
25 / 141
26 / 141
27 / 141
28 / 141
29 / 141
30 / 141
31 / 141
32 / 141
33 / 141
34 / 141
35 / 141
36 / 141
37 / 141
38 / 141
39 / 141
40 / 141
41 / 141
42 / 141
43 / 141
44 / 141
45 / 141
46 / 141
47 / 141
48 / 141
49 / 141
50 / 141
51 / 141
52 / 141
53 / 141
54 / 141
55 / 141
56 / 141
57 / 141
58 / 141
59 / 141
60 / 141
61 / 141
62 / 141
63 / 141
64 / 141
65 / 141
66 / 141
67 / 141
68 / 141
69 / 141
70 / 141
71 / 141
72 / 141
73 / 141
74 / 141
75 / 141
76 / 141
77 / 141
78 / 141
79 / 141
80 / 141
81 / 141
82 / 141
83 / 141
84 / 141
85 / 141
86 / 141
87 / 141
88 / 141
89 / 141
90 / 141
91 / 141
92 / 141
93 / 141
94 / 141
95 / 141
96 / 141
97 / 141
98 / 141
99 / 141
100 / 141
101 / 141
102 / 141
103 / 141
104 / 141
105 / 141
106 / 141
107 / 141
108 / 141
109 / 141
110 / 141
111 / 141
112 / 141
113 / 141
114 / 141
115 / 141
116 / 141
117 / 141
118 / 141
119 / 141
120 / 141
121 / 141
122 / 141
123 / 141
124 / 141
125 / 141
126 / 141
127 / 141
128 / 141
129 / 141
130 / 141
131 / 141
132 / 141
133 / 141
134 / 141
135 / 141
136 / 141
137 / 141
138 / 141
139 / 141
140 / 141
141 / 141
MOVING DATA FOR THE MASSES Giuseppe Maxia QA Director Continuent, Inc @datacharmer This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License. Friday, November 11, 11 1
Databases are the backbone of the web economy www $$$ Friday, November 11, 11 2
Actually, database replication is the backbone of the web economy Friday, November 11, 11 3
Actually, database replication is the backbone of the web economy WHY? Friday, November 11, 11 3
Why moving data? • Scaling reads (load balancing) • Scaling writes (sharding) • Parallelize computation (sharding+map/reduce) • High availability (replacing master on failure) • Creating information (feeding data warehouses) Friday, November 11, 11 4
Scaling reads M S S S Friday, November 11, 11 5
Scaling writes (sharding) M1 M3 M2 S S S S S S Friday, November 11, 11 6
High availability M S S S Friday, November 11, 11 7
High availability S ✘ M S S Friday, November 11, 11 7
High availability S S M Friday, November 11, 11 8
Creating new information S M M M Friday, November 11, 11 9
Creating new information S S M S M M M Friday, November 11, 11 9
More about data replication Friday, November 11, 11 10
database server r/w requests web server clients Friday, November 11, 11 11
database server r/w requests web servers load balancer clients Friday, November 11, 11 12
database load on a simple web application r e write a d Friday, November 11, 11 13
write read database load on a successful web application Friday, November 11, 11 14
database server ✘ r/w requests web servers load balancer clients Friday, November 11, 11 15
read/write master a web application read/only slaves scheme with replication load balancer R/W R/O web servers load balancer clients Friday, November 11, 11 16
r e write a read d read/write master read/only slaves database load with replication Friday, November 11, 11 17
r e write a read d read/write master read/only slaves scaling database load with replication Friday, November 11, 11 18
• If you want a successful company Friday, November 11, 11 19
• If you want a successful company • in the 21st century Friday, November 11, 11 19
• If you want a successful company • in the 21st century • you need to move data Friday, November 11, 11 19
• If you want a successful company • in the 21st century • you need to move data • fast Friday, November 11, 11 19
• If you want a successful company • in the 21st century • you need to move data • fast • and often Friday, November 11, 11 19
MOVING DATA FOR THE MASSES Friday, November 11, 11 20
How do you move data? Friday, November 11, 11 21
How do you move data? • Commodity hardware Friday, November 11, 11 21
How do you move data? • Commodity hardware • Basic • Commodity software (MySQL) • MySQL native replication Friday, November 11, 11 21
How do you move data? • Commodity hardware • Basic • Commodity software (MySQL) • MySQL native replication • Advanced • MySQL and other DBMS • Powerful tools (Tungsten Replicator) Friday, November 11, 11 21
Why Tungsten? Friday, November 11, 11 22
transaction transaction MySQL transaction transaction transaction DBMS transaction transaction transaction transaction transaction transaction transaction BINARY LOG ns act act tio n tra ansio ns sac ntio ac ion ion n ac nr tr tra aniton tsa a n csio nts ct n rn a o taa scti r trnsa n n MySQL replication tio ct tra tra REPLICATION transaction transaction is single threaded MySQL DBMS Friday, November 11, 11 23
master master master master MySQL MySQL MySQL MySQL DBMS DBMS DBMS DBMS MySQL MySQL MySQL MySQL DBMS DBMS DBMS DBMS slave slave slave slave single source multi source (fan-in) multiple sources? Friday, November 11, 11 24
from this to this MySQL MySQL DBMS DBMS master master MySQL MySQL DBMS DBMS master master multiple masters? Friday, November 11, 11 25
INSERT INSERT RECORD RECORD A A MySQL MySQL DBMS DBMS master master MySQL MySQL DBMS DBMS MODIFY master master MODIFY RECORD RECORD B B Avoiding conflicts? Friday, November 11, 11 26
master Seamless failover? MySQL DBMS master MySQL DBMS master MySQL DBMS MySQL MySQL DBMS DBMS slave slave MySQL DBMS MySQL DBMS slave slave MySQL MySQL DBMS DBMS slave slave Friday, November 11, 11 27
Replicating to something else? mysql master MySQL DBMS MySQL DBMS mysql postgresql oracle mongodb Friday, November 11, 11 28
All these examples tell us: Nice dream, but MySQL can’t do it Friday, November 11, 11 29
Enter Tungsten Replicator Friday, November 11, 11 30
http://code.google.com/p/tungsten-replicator Friday, November 11, 11 31
http://code.google.com/p/tungsten-replicator Open Source Friday, November 11, 11 31
http://code.google.com/p/tungsten-replicator Open Source 100% GPL v2 Friday, November 11, 11 31
What can it do? • Easy failover • Multiple masters • Multiple sources to a single slave • Conflict prevention • Parallel replication • Replicate to Oracle and PostgreSQL database Friday, November 11, 11 32
MySQL to foreign services MySQL setup to run as MySQL master Master DB Data is applied to PostgreSQL or Oracle or to NoSQL entities Bin Logs (Binlogs enabled) Slave DB © Continuent 2010 Friday, November 11, 11 33
Main components • Transaction History Logs (THL) • roughly corresponding to MySQL relay logs • have a lot of metadata • Service database • contains metadata for latest transactions • Metadata is committed together with data • Makes slaves crash proof Friday, November 11, 11 34
A failover scenario 1: MySQL native replication Friday, November 11, 11 35
Defining the problem Slave 3 Master Replication Slave 2 Slave 1 Binary log Transactions Relay log Binary log -------- -------- Transactions Transactions -------- -------- ======== -------- -------- ======== -------- ======== -------- ======== Friday, November 11, 11 36
The master crashes Slave 3 Master Replication Slave 2 Slave 1 Binary log Transactions Relay log Binary log -------- -------- Transactions Transactions -------- -------- ======== -------- -------- ======== -------- ======== -------- ======== Friday, November 11, 11 37
Where do the slaves stand? Slave 1 Slave 2 Slave 3 Binary log Binary log Binary log Transactions Transactions Transactions ======== ======== ======== ======== ======== ======== ======== ======== ======== ======== ======== ======== Friday, November 11, 11 38
Slave 3 becomes master New master Slave 1 Slave 2 Binary log Transactions ======== Binary log Binary log ======== ======== Transactions Transactions ======== ======== ======== ======== ======== ======== ======== ======== ======== Friday, November 11, 11 39
Need to synch the missing transactions New master Slave 1 Slave 2 Binary log Transactions ======== Binary log Binary log ======== ======== Transactions Transactions ======== ======== ======== ======== ======== ======== ======== ======== ======== Friday, November 11, 11 40
Transaction position in the new master binlog may not match positions in the old master binlog New master Slave 1 Slave 2 Binary log Transactions ? ? ======== Binary log Binary log ======== ======== Transactions Transactions ======== ======== ======== ======== ======== ======== ? ======== ? ======== ======== Friday, November 11, 11 41
A failover scenario 1: Tungsten Replicator Friday, November 11, 11 42
Slave 3 Master Replication Slave 2 Slave 1 THL Transactions THL -------- -------- Transactions -------- ======== -------- ======== ======== ======== Includes Global Transaction ID Friday, November 11, 11 43
The master crashes Slave 3 Master Replication Slave 2 Slave 1 THL Transactions THL -------- -------- Transactions -------- ======== -------- ======== ======== ======== Friday, November 11, 11 44
Where do the slaves stand? Slave 1 Slave 2 Slave 3 THL THL THL Transactions Transactions Transactions ======== ======== ======== ======== ======== ======== ======== ======== ======== ======== ======== ======== Friday, November 11, 11 45
Slave 3 becomes master New master Slave 1 Slave 2 THL Transactions ======== THL THL ======== ======== Transactions Transactions ======== ======== ======== ======== ======== ======== ======== ======== ======== Friday, November 11, 11 46
Need to synch the missing transactions New master Slave 1 Slave 2 THL Transactions ======== THL THL ======== ======== Transactions Transactions ======== ======== ======== ======== ======== ======== ======== ======== ======== Friday, November 11, 11 47
Transaction IDs in the new master THL are understood immediately by the slaves New master Slave 1 Slave 2 Binary log Transactions ======== Binary log Binary log ======== ======== Transactions Transactions ======== ======== ======== ======== ======== ======== Give me ======== Give me ======== seqno #51 ======== seqno #67 Friday, November 11, 11 48
Parallel replication Friday, November 11, 11 49
parallel replication From here ... Friday, November 11, 11 50
parallel replication To here. Friday, November 11, 11 51
Parallel replication facts ✓Sharded by database ✓Good choice for slave lag problems ❖Bad choice for single database projects Friday, November 11, 11 52
Benchmarking parallel replication Friday, November 11, 11 53
db0 preparation (1) db1 sysbench db2 sysbench db3 sysbench db4 sysbench db5 sysbench db6 sysbench db7 sysbench db8 sysbench db9 sysbench sysbench Friday, November 11, 11 54
before the test (2) STOPPED binary logs MySQL slave OFFLINE Tungsten slave direct: alpha (slave) replicator alpha Friday, November 11, 11 55
starting the test STOPPED binary logs MySQL slave Concurrent sysbench on 30 databases running for 1 hour OFFLINE TOTAL DATA: 130 GB Tungsten slave direct: alpha RAM per server: 20GB (slave) replicator alpha Slaves will have 1 hour lag Friday, November 11, 11 56
measuring results START binary logs MySQL slave ONLINE Recording catch-up time Tungsten slave direct: alpha (slave) replicator alpha Friday, November 11, 11 57
MySQL native replication slave catch up in 04:29:30 Friday, November 11, 11 58
Tungsten parallel replication slave catch up in 00:55:40 Friday, November 11, 11 59
Parallel replication made simpler FROM HERE .... Friday, November 11, 11 60
Parallel replication made simpler TO HERE Friday, November 11, 11 61
Parallel replication made simpler Friday, November 11, 11 62
parallel replication direct slave facts Friday, November 11, 11 63
parallel replication direct slave facts ✓No need to install Tungsten on the master Friday, November 11, 11 63
parallel replication direct slave facts ✓No need to install Tungsten on the master ✓Tungsten runs only on the slave Friday, November 11, 11 63
parallel replication direct slave facts ✓No need to install Tungsten on the master ✓Tungsten runs only on the slave ✓Replication can revert to native slave with two commands (trepctl offline; start slave) Friday, November 11, 11 63
parallel replication direct slave facts ✓No need to install Tungsten on the master ✓Tungsten runs only on the slave ✓Replication can revert to native slave with two commands (trepctl offline; start slave) ✓Native replication can continue on other slaves Friday, November 11, 11 63
parallel replication direct slave facts ✓No need to install Tungsten on the master ✓Tungsten runs only on the slave ✓Replication can revert to native slave with two commands (trepctl offline; start slave) ✓Native replication can continue on other slaves ❖Failover (either native or Tungsten) becomes a manual task Friday, November 11, 11 63
Multiple masters Friday, November 11, 11 64
multiple masters • Tungsten Replicator recipe: use more services Friday, November 11, 11 65
Bi-directional replication Friday, November 11, 11 66
Bi-directional replication with slaves Friday, November 11, 11 67
Multiple sources Friday, November 11, 11 68
multiple sources • Tungsten Replicator recipe is still valid: use more services Friday, November 11, 11 69
Multiple source replication Friday, November 11, 11 70
Multiple masters replication: 3 nodes Friday, November 11, 11 71
Multiple masters replication: 4 nodes Friday, November 11, 11 72
4 nodes multiple masters simplified Friday, November 11, 11 73
star topologies Friday, November 11, 11 74
Conflict prevention Friday, November 11, 11 75
Conflict prevention facts Friday, November 11, 11 76
Conflict prevention facts • Sharded by database Friday, November 11, 11 76
Conflict prevention facts • Sharded by database • Defined dynamically Friday, November 11, 11 76
Conflict prevention facts • Sharded by database • Defined dynamically • Applied either at the master or at the slave Friday, November 11, 11 76
Conflict prevention facts • Sharded by database • Defined dynamically • Applied either at the master or at the slave • methods: Friday, November 11, 11 76
Conflict prevention facts • Sharded by database • Defined dynamically • Applied either at the master or at the slave • methods: • make replication fail Friday, November 11, 11 76
Conflict prevention facts • Sharded by database • Defined dynamically • Applied either at the master or at the slave • methods: • make replication fail • drop silently Friday, November 11, 11 76
Conflict prevention facts • Sharded by database • Defined dynamically • Applied either at the master or at the slave • methods: • make replication fail • drop silently • drop with warning Friday, November 11, 11 76
Tungsten Conflict prevention points host1 host3 M S S host2 host4 master S M S Friday, November 11, 11 77
Tungsten Conflict prevention points host1 host3 M S S host2 host4 master S M S Friday, November 11, 11 77
Tungsten Conflict prevention points host1 host3 M S S host2 host4 master S M S Friday, November 11, 11 77
Prevention methods: Fail on master host1 host3 M S S host2 host4 master S M S Friday, November 11, 11 78
INSERT A x,y Prevention methods: Fail on master host1 host3 M S S host2 host4 master S M S INSERT A x,z Friday, November 11, 11 78
INSERT A x,y Prevention methods: Fail on master host1 host3 M S S host2 host4 master S M S INSERT A x,z Friday, November 11, 11 78
Prevention methods: Fail on slave host1 host3 M S S host2 host4 master S M S Friday, November 11, 11 79
INSERT A x,y Prevention methods: Fail on slave host1 host3 M S S host2 host4 master S M S INSERT A x,z Friday, November 11, 11 79
INSERT A x,y Prevention methods: Fail on slave host1 host3 M S S host2 host4 master S M S INSERT A x,z Friday, November 11, 11 79
Prevention methods: Fail on slave host1 (Multiple sources) M host2 host3 master M S Friday, November 11, 11 80
INSERT A x,y Prevention methods: Fail on slave host1 (Multiple sources) M host2 host3 master M S INSERT A x,z Friday, November 11, 11 80
INSERT A x,y Prevention methods: Fail on slave host1 (Multiple sources) M host2 host3 master M S INSERT A x,z Friday, November 11, 11 80
Prevention methods: DROP on master host1 host3 M S S host2 host4 master S M S Friday, November 11, 11 81
INSERT A x,y Prevention methods: DROP on master host1 host3 M S S host2 host4 master S M S Friday, November 11, 11 81
INSERT A x,y Prevention methods: DROP on master host1 host3 M S S host2 host4 master S M S INSERT A x,z Friday, November 11, 11 81
Prevention methods: DROP on master host1 host3 M S S host2 host4 master S M S INSERT A x,z Friday, November 11, 11 81
Prevention methods: DROP on slave host1 (Multiple sources) M host2 host3 master M S Friday, November 11, 11 82
INSERT A x,y Prevention methods: DROP on slave host1 (Multiple sources) M host2 host3 master M S INSERT A x,z Friday, November 11, 11 82
Prevention methods: DROP on slave host1 (Multiple sources) M host2 INSERT A host3 x,y master INSERT A M x,z S Friday, November 11, 11 83
Prevention methods: DROP on slave host1 (Multiple sources) M host2 host3 master INSERT A M x,z S Friday, November 11, 11 83
Tungsten in practice Installation Friday, November 11, 11 84
Installation • Check the requirements • Get the binaries • Expand the tarball • Run ./tools/tungsten-installer Friday, November 11, 11 85
REQUIREMENTS • Java JRE or JDK (Sun/Oracle or Open-jdk) • Ruby 1.8 (only during installation) • ssh access to the same user in all nodes • MySQL user with all privileges Friday, November 11, 11 86
Installation types • master / slave • slave - direct Friday, November 11, 11 87
master/slave host2 host1 master slave THL binlog THL host3 slave THL Friday, November 11, 11 88
slave direct host2 host1 slave master relay log THL binlog host3 slave relay log THL Friday, November 11, 11 89
sample installation • 4 servers • qa.r1.continuent.com r1 • qa.r2.continuent.com r2 • qa.r3.continuent.com r3 • qa.r4.continuent.com r4 • MySQL 5.1 • tungsten-replicator-2.0.4 Friday, November 11, 11 90
Installation (1) # starting at node 4, but any would do for N in 1 2 3 4 do ssh r$N mkdir tinstall done cd tinstall tar -xzf /path/to/tungsten-replicator-2.0.4.tar.gz cd tungsten-replicator-2.0.4 Friday, November 11, 11 91
Installation (2) export TUNGSTEN_BASE=$HOME/tinstall 1 ./tools/tungsten-installer 2 --master-slave 3 --master-host=r1 4 --datasource-user=tungsten 5 --datasource-password=secret 6 --service-name=dragon 7 --home-directory=$TUNGSTEN_BASE 8 --cluster-hosts=r1,r2,r3,r4 9 --start Friday, November 11, 11 92
Installation (2) export TUNGSTEN_BASE=$HOME/tinstall ./tools/tungsten-installer --master-slave # installation mode --master-host=r1 # who’s the master --datasource-user=tungsten # mysql username --datasource-password=secret # mysql password --service-name=dragon # name of the service --home-directory=$TUNGSTEN_BASE # where we install --cluster-hosts=r1,r2,r3,r4 # hosts in cluster --start # start replicator after installing Friday, November 11, 11 93
What does the installation do 1: Validate all servers host4 host1 host2 host3 ✔ ✔ ✔ ✔ ✗ ✗ ✗ ✗ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Report all errors Friday, November 11, 11 94
What does the installation do 1 (again): Validate all servers host4 host1 host2 host3 ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Friday, November 11, 11 95
What does the installation do 2: install Tungsten in all servers host4 $HOME/ host1 tinstall/ host2 config/ host3 releases/ relay/ logs/ tungsten/ Friday, November 11, 11 96
example ssh r2 chmod 444 $HOME/tinstall ./tools/tungsten-installer --master-slave --master-host=r1 --datasource-user=tungsten --datasource-password=secret --service-name=dragon --home-directory=$HOME/tinstall --thl-directory=$HOME/tinstall/logs --relay-directory=$HOME/tinstall/relay --cluster-hosts=r1,r2,r3,r4 --start ERROR >> qa.r2.continuent.com >> /home/tungsten/ tinstall is not writeable Friday, November 11, 11 97
example ssh r2 chmod 755 $HOME/tinstall ./tools/tungsten-installer --master-slave --master-host=r1 --datasource-user=tungsten --datasource-password=secret --service-name=dragon --home-directory=$HOME/tinstall --thl-directory=$HOME/tinstall/logs --relay-directory=$HOME/tinstall/relay --cluster-hosts=r1,r2,r3,r4 --start # no errors Friday, November 11, 11 98
Installation - direct ./tools/tungsten-installer --direct --master-host=r1 --slave-host=r4 --master-user=tungsten --slave-user=tungsten --master-password=secret --slave-password=secret --service-name=Castor --channels=1 --home-directory=$TUNGSTEN_BASE --start-and-report Friday, November 11, 11 99
http://bit.ly/tr20_cookbook Friday, November 11, 11 100
Tungsten Replicator vs. Enterprise Feature Replicator Enterprise Simple admin with trepctl X X MySQL Row/Statement 5.0/5.1/5.5 X X Multi-master replication X X Parallel apply (2.0) X X Distributed management with cctrl X Automated failover and recovery X Zero-downtime upgrade X SQL routing and load-balancing X Floating IP support X © Continuent 2011 Friday, November 11, 11 101
Conclusion and Q&A Friday, November 11, 11 102
Contact Information Worldwide 560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 @datacharmer Tel (866) 998-3642 Fax (408) 668-1009 @continuent e-mail: sales@continuent.com Continuent Web Site: http://www.continuent.com Tungsten Project http://tungsten-replicator.org Getting started with Tungsten Replicator: http://datacharmer.blogspot.com © Continuent 2011 Friday, November 11, 11 103
Friday, November 11, 11 104

More Related Content

ZIP
Emily is Moving to Kenya
byguest6b1e4c4
 
PPS
Mighty ones of old
byczaragon
 
PDF
MySQL - Open Database
byGiuseppe Maxia
 
PPTX
Peacemaker
byguest85417
 
PDF
Testing early mysql releases in a sandbox
byGiuseppe Maxia
 
PDF
MySQL creative programming
byGiuseppe Maxia
 
PDF
MySQL Sandbox 3
byGiuseppe Maxia
 
PDF
Gearman For Beginners
byGiuseppe Maxia
 
Emily is Moving to Kenya
byguest6b1e4c4
 
Mighty ones of old
byczaragon
 
MySQL - Open Database
byGiuseppe Maxia
 
Peacemaker
byguest85417
 
Testing early mysql releases in a sandbox
byGiuseppe Maxia
 
MySQL creative programming
byGiuseppe Maxia
 
MySQL Sandbox 3
byGiuseppe Maxia
 
Gearman For Beginners
byGiuseppe Maxia
 

Viewers also liked

PDF
Nursing colleges
byfaithjk20
 
PPTX
Uma história em correspondência
byQuintaDaFidalga
 
PDF
Banda desenhada Maria Castanha
byQuintaDaFidalga
 
DOCX
Plan de mejora lectores 1°
byJuan Carlos Valencia V.
 
PDF
What is income insurance
byijimng3
 
PPS
Mainha,
bySandro Loureiro
 
ODP
Pract1
bymariaymishell95
 
PDF
Introdução
byEveraldo Lippi
 
ODT
Exámenes diciembre 2011
byPATRICIO CAPECE PINTADO
 
DOCX
Tutorial bytworld volume iii
byEmerson Anjos Videos
 
Nursing colleges
byfaithjk20
 
Uma história em correspondência
byQuintaDaFidalga
 
Banda desenhada Maria Castanha
byQuintaDaFidalga
 
Plan de mejora lectores 1°
byJuan Carlos Valencia V.
 
What is income insurance
byijimng3
 
Mainha,
bySandro Loureiro
 
Pract1
bymariaymishell95
 
Introdução
byEveraldo Lippi
 
Exámenes diciembre 2011
byPATRICIO CAPECE PINTADO
 
Tutorial bytworld volume iii
byEmerson Anjos Videos
 

Similar to Moving data for the masses

PDF
Advanced mysql replication techniques
byGiuseppe Maxia
 
PDF
Replication 101
byGiuseppe Maxia
 
PDF
Optimizing your site for contextual ads: SEO, Design and Content
byRaven Tools
 
PDF
Modern HTML & CSS Coding: Speed, Semantics & Structure
byRaven Tools
 
PDF
Distribute the workload, PHP Barcelona 2011
byHelgi Þormar Þorbjörnsson
 
PDF
15 QR Code Ideas to Blow Your Customers' Minds
byJustin Ferrell
 
PDF
Data Grids and Data Caching
byGalder Zamarreño
 
PDF
Infinispan for Dummies
byGalder Zamarreño
 
PDF
Building Data Driven Products With Ruby - RubyConf 2012
byRyan Weald
 
PDF
Scaling your web app with MySQL replication
byGiuseppe Maxia
 
PDF
Intro to HTML5 Game Programming
byJames Williams
 
PDF
Performance Optimization for Ext GWT 3.0
bySencha
 
PDF
What's new in HTML5, CSS3 and JavaScript, James Pearce
bySencha
 
PDF
David Mytton, Boxed Ice
byMashery
 
PDF
Kill bottlenecks with gearman, sphinx, and memcached, Confoo 2011
byBachkoutou Toutou
 
PDF
Lightning talks percona live mysql_2012
byGiuseppe Maxia
 
PDF
soft-shake.ch - Data grids and Data Caching
bysoft-shake.ch
 
PDF
Opensource Authentication and Authorization
byConFoo
 
PDF
The facilities of Features Drupal module
byKálmán Hosszu
 
PDF
LinkedIn Data Infrastructure (QCon London 2012)
bySid Anand
 
Advanced mysql replication techniques
byGiuseppe Maxia
 
Replication 101
byGiuseppe Maxia
 
Optimizing your site for contextual ads: SEO, Design and Content
byRaven Tools
 
Modern HTML & CSS Coding: Speed, Semantics & Structure
byRaven Tools
 
Distribute the workload, PHP Barcelona 2011
byHelgi Þormar Þorbjörnsson
 
15 QR Code Ideas to Blow Your Customers' Minds
byJustin Ferrell
 
Data Grids and Data Caching
byGalder Zamarreño
 
Infinispan for Dummies
byGalder Zamarreño
 
Building Data Driven Products With Ruby - RubyConf 2012
byRyan Weald
 
Scaling your web app with MySQL replication
byGiuseppe Maxia
 
Intro to HTML5 Game Programming
byJames Williams
 
Performance Optimization for Ext GWT 3.0
bySencha
 
What's new in HTML5, CSS3 and JavaScript, James Pearce
bySencha
 
David Mytton, Boxed Ice
byMashery
 
Kill bottlenecks with gearman, sphinx, and memcached, Confoo 2011
byBachkoutou Toutou
 
Lightning talks percona live mysql_2012
byGiuseppe Maxia
 
soft-shake.ch - Data grids and Data Caching
bysoft-shake.ch
 
Opensource Authentication and Authorization
byConFoo
 
The facilities of Features Drupal module
byKálmán Hosszu
 
LinkedIn Data Infrastructure (QCon London 2012)
bySid Anand
 

More from Giuseppe Maxia

PDF
MySQL NDB 8.0 clusters in your laptop with dbdeployer
byGiuseppe Maxia
 
PDF
Test like a_boss
byGiuseppe Maxia
 
PDF
Dbdeployer, the universal installer
byGiuseppe Maxia
 
PDF
Test complex database systems in your laptop with dbdeployer
byGiuseppe Maxia
 
PDF
Dbdeployer
byGiuseppe Maxia
 
PDF
Dbdeployer
byGiuseppe Maxia
 
PDF
A quick tour of Mysql 8 roles
byGiuseppe Maxia
 
PDF
MySQL document_store
byGiuseppe Maxia
 
PDF
Replication skeptic
byGiuseppe Maxia
 
PDF
Synchronise your data between MySQL and MongoDB
byGiuseppe Maxia
 
PDF
Juggle your data with Tungsten Replicator
byGiuseppe Maxia
 
PDF
MySQL in your laptop
byGiuseppe Maxia
 
PDF
Script it
byGiuseppe Maxia
 
PDF
Tungsten Replicator tutorial
byGiuseppe Maxia
 
PDF
Preventing multi master conflicts with tungsten
byGiuseppe Maxia
 
PDF
MySQL high availability power and usability
byGiuseppe Maxia
 
PDF
Solving MySQL replication problems with Tungsten
byGiuseppe Maxia
 
PDF
State of the art of MySQL replication and clustering
byGiuseppe Maxia
 
PDF
Testing mysql creatively in a sandbox
byGiuseppe Maxia
 
PDF
Mysql 5.5 and 5.6 replication
byGiuseppe Maxia
 
MySQL NDB 8.0 clusters in your laptop with dbdeployer
byGiuseppe Maxia
 
Test like a_boss
byGiuseppe Maxia
 
Dbdeployer, the universal installer
byGiuseppe Maxia
 
Test complex database systems in your laptop with dbdeployer
byGiuseppe Maxia
 
Dbdeployer
byGiuseppe Maxia
 
Dbdeployer
byGiuseppe Maxia
 
A quick tour of Mysql 8 roles
byGiuseppe Maxia
 
MySQL document_store
byGiuseppe Maxia
 
Replication skeptic
byGiuseppe Maxia
 
Synchronise your data between MySQL and MongoDB
byGiuseppe Maxia
 
Juggle your data with Tungsten Replicator
byGiuseppe Maxia
 
MySQL in your laptop
byGiuseppe Maxia
 
Script it
byGiuseppe Maxia
 
Tungsten Replicator tutorial
byGiuseppe Maxia
 
Preventing multi master conflicts with tungsten
byGiuseppe Maxia
 
MySQL high availability power and usability
byGiuseppe Maxia
 
Solving MySQL replication problems with Tungsten
byGiuseppe Maxia
 
State of the art of MySQL replication and clustering
byGiuseppe Maxia
 
Testing mysql creatively in a sandbox
byGiuseppe Maxia
 
Mysql 5.5 and 5.6 replication
byGiuseppe Maxia
 

Recently uploaded

PDF
UiPath Automation Developer Associate Training Series 2026 - Session 3
byDianaGray10
 
PDF
final.pdf
byomarbishtawi04
 
PDF
GDG Cloud Southlake #49: Pradeep R Kumar: Implications of Agentic AI for Iden...
byJames Anderson
 
PPTX
Automation Without Apprentices: How AI Challenges the Open Source Way
byIcinga
 
PDF
Agent to agent service discovery using HashiCorp Consul and Vault
byBram Vogelaar
 
PDF
Escape from the Forbidden Zone: Smuggling green and inclusive tech past the g...
byBookNet Canada
 
PPTX
Introducing VisualSim 2610 The Next Leap in System Level Modeling
byDeepak Shankar
 
PDF
UiPath Modern Automation Playbook -Session 2
bysuhanisingh58689
 
PDF
GenerationAI_Paris_2025_Architecting_Intelligence.pdf
byapidays
 
PPTX
Microsoft Azure News - February 2026 - BAUG
byDaniel Toomey
 
PDF
From Hallucinations to High-Confidence AI with Data Enrichment.pdf
byPrecisely
 
PDF
TrustArc Webinar - From Trends to Action: Fitting AI Governance into Privacy Ops
byTrustArc
 
PDF
apidays New York 2025 | AI in Application Security
byapidays
 
PDF
Explaining specific examples of purpose-specific BOM
byakipii ogaoga
 
PPTX
The Transformative Technology in Contemporary Businesses
bygreyaudrina
 
PDF
Automated Governance for FME Flow: Smarter Admin at Scale
bySafe Software
 
PDF
Chapter 5 Security Mechanisms iin computer security.pdf
byGetnet Tigabie Askale -(GM)
 
PDF
GTM-and-Sales-Plan for a cyber security product
byAshish Jangir
 
PDF
20260212 Security-JAWS activity results for 2025 and activity goals for 2026
byTyphon 666
 
PDF
From Artificial Intelligence to African Intelligence: Transforming Research &...
byMoses Kemibaro
 
UiPath Automation Developer Associate Training Series 2026 - Session 3
byDianaGray10
 
final.pdf
byomarbishtawi04
 
GDG Cloud Southlake #49: Pradeep R Kumar: Implications of Agentic AI for Iden...
byJames Anderson
 
Automation Without Apprentices: How AI Challenges the Open Source Way
byIcinga
 
Agent to agent service discovery using HashiCorp Consul and Vault
byBram Vogelaar
 
Escape from the Forbidden Zone: Smuggling green and inclusive tech past the g...
byBookNet Canada
 
Introducing VisualSim 2610 The Next Leap in System Level Modeling
byDeepak Shankar
 
UiPath Modern Automation Playbook -Session 2
bysuhanisingh58689
 
GenerationAI_Paris_2025_Architecting_Intelligence.pdf
byapidays
 
Microsoft Azure News - February 2026 - BAUG
byDaniel Toomey
 
From Hallucinations to High-Confidence AI with Data Enrichment.pdf
byPrecisely
 
TrustArc Webinar - From Trends to Action: Fitting AI Governance into Privacy Ops
byTrustArc
 
apidays New York 2025 | AI in Application Security
byapidays
 
Explaining specific examples of purpose-specific BOM
byakipii ogaoga
 
The Transformative Technology in Contemporary Businesses
bygreyaudrina
 
Automated Governance for FME Flow: Smarter Admin at Scale
bySafe Software
 
Chapter 5 Security Mechanisms iin computer security.pdf
byGetnet Tigabie Askale -(GM)
 
GTM-and-Sales-Plan for a cyber security product
byAshish Jangir
 
20260212 Security-JAWS activity results for 2025 and activity goals for 2026
byTyphon 666
 
From Artificial Intelligence to African Intelligence: Transforming Research &...
byMoses Kemibaro
 

Moving data for the masses

  • 1.
    MOVING DATA FORTHE MASSES Giuseppe Maxia QA Director Continuent, Inc @datacharmer This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License. Friday, November 11, 11 1
  • 2.
    Databases are thebackbone of the web economy www $$$ Friday, November 11, 11 2
  • 3.
    Actually, database replicationis the backbone of the web economy Friday, November 11, 11 3
  • 4.
    Actually, database replicationis the backbone of the web economy WHY? Friday, November 11, 11 3
  • 5.
    Why moving data? • Scaling reads (load balancing) • Scaling writes (sharding) • Parallelize computation (sharding+map/reduce) • High availability (replacing master on failure) • Creating information (feeding data warehouses) Friday, November 11, 11 4
  • 6.
    Scaling reads M S S S Friday, November 11, 11 5
  • 7.
    Scaling writes (sharding) M1 M3 M2 S S S S S S Friday, November 11, 11 6
  • 8.
    High availability M S S S Friday, November 11, 11 7
  • 9.
    High availability S ✘ M S S Friday, November 11, 11 7
  • 10.
    High availability S S M Friday, November 11, 11 8
  • 11.
    Creating new information S M M M Friday, November 11, 11 9
  • 12.
    Creating new information S S M S M M M Friday, November 11, 11 9
  • 13.
    More about data replication Friday, November 11, 11 10
  • 14.
    database server r/w requests web server clients Friday, November 11, 11 11
  • 15.
    database server r/w requests web servers load balancer clients Friday, November 11, 11 12
  • 16.
    database load on a simple web application r e write a d Friday, November 11, 11 13
  • 17.
    write read database load on a successful web application Friday, November 11, 11 14
  • 18.
    database server ✘ r/w requests web servers load balancer clients Friday, November 11, 11 15
  • 19.
    read/write master a web application read/only slaves scheme with replication load balancer R/W R/O web servers load balancer clients Friday, November 11, 11 16
  • 20.
    r e write a read d read/write master read/only slaves database load with replication Friday, November 11, 11 17
  • 21.
    r e write a read d read/write master read/only slaves scaling database load with replication Friday, November 11, 11 18
  • 22.
    • If you want a successful company Friday, November 11, 11 19
  • 23.
    • If you want a successful company • in the 21st century Friday, November 11, 11 19
  • 24.
    • If you want a successful company • in the 21st century • you need to move data Friday, November 11, 11 19
  • 25.
    • If you want a successful company • in the 21st century • you need to move data • fast Friday, November 11, 11 19
  • 26.
    • If you want a successful company • in the 21st century • you need to move data • fast • and often Friday, November 11, 11 19
  • 27.
    MOVING DATA FOR THE MASSES Friday, November 11, 11 20
  • 28.
    How do youmove data? Friday, November 11, 11 21
  • 29.
    How do youmove data? • Commodity hardware Friday, November 11, 11 21
  • 30.
    How do youmove data? • Commodity hardware • Basic • Commodity software (MySQL) • MySQL native replication Friday, November 11, 11 21
  • 31.
    How do youmove data? • Commodity hardware • Basic • Commodity software (MySQL) • MySQL native replication • Advanced • MySQL and other DBMS • Powerful tools (Tungsten Replicator) Friday, November 11, 11 21
  • 32.
  • 33.
    transaction transaction MySQL transaction transaction transaction DBMS transaction transaction transaction transaction transaction transaction transaction BINARY LOG ns act act tio n tra ansio ns sac ntio ac ion ion n ac nr tr tra aniton tsa a n csio nts ct n rn a o taa scti r trnsa n n MySQL replication tio ct tra tra REPLICATION transaction transaction is single threaded MySQL DBMS Friday, November 11, 11 23
  • 34.
    master master master master MySQL MySQL MySQL MySQL DBMS DBMS DBMS DBMS MySQL MySQL MySQL MySQL DBMS DBMS DBMS DBMS slave slave slave slave single source multi source (fan-in) multiple sources? Friday, November 11, 11 24
  • 35.
    from this to this MySQL MySQL DBMS DBMS master master MySQL MySQL DBMS DBMS master master multiple masters? Friday, November 11, 11 25
  • 36.
    INSERT INSERT RECORD RECORD A A MySQL MySQL DBMS DBMS master master MySQL MySQL DBMS DBMS MODIFY master master MODIFY RECORD RECORD B B Avoiding conflicts? Friday, November 11, 11 26
  • 37.
    master Seamless failover? MySQL DBMS master MySQL DBMS master MySQL DBMS MySQL MySQL DBMS DBMS slave slave MySQL DBMS MySQL DBMS slave slave MySQL MySQL DBMS DBMS slave slave Friday, November 11, 11 27
  • 38.
    Replicating to somethingelse? mysql master MySQL DBMS MySQL DBMS mysql postgresql oracle mongodb Friday, November 11, 11 28
  • 39.
    All these examplestell us: Nice dream, but MySQL can’t do it Friday, November 11, 11 29
  • 40.
  • 41.
  • 42.
    http://code.google.com/p/tungsten-replicator Open Source Friday, November 11, 11 31
  • 43.
    http://code.google.com/p/tungsten-replicator Open Source 100% GPL v2 Friday, November 11, 11 31
  • 44.
    What can itdo? • Easy failover • Multiple masters • Multiple sources to a single slave • Conflict prevention • Parallel replication • Replicate to Oracle and PostgreSQL database Friday, November 11, 11 32
  • 45.
    MySQL to foreignservices MySQL setup to run as MySQL master Master DB Data is applied to PostgreSQL or Oracle or to NoSQL entities Bin Logs (Binlogs enabled) Slave DB © Continuent 2010 Friday, November 11, 11 33
  • 46.
    Main components • Transaction History Logs (THL) • roughly corresponding to MySQL relay logs • have a lot of metadata • Service database • contains metadata for latest transactions • Metadata is committed together with data • Makes slaves crash proof Friday, November 11, 11 34
  • 47.
    A failover scenario 1: MySQL native replication Friday, November 11, 11 35
  • 48.
    Defining the problem Slave 3 Master Replication Slave 2 Slave 1 Binary log Transactions Relay log Binary log -------- -------- Transactions Transactions -------- -------- ======== -------- -------- ======== -------- ======== -------- ======== Friday, November 11, 11 36
  • 49.
    The master crashes Slave 3 Master Replication Slave 2 Slave 1 Binary log Transactions Relay log Binary log -------- -------- Transactions Transactions -------- -------- ======== -------- -------- ======== -------- ======== -------- ======== Friday, November 11, 11 37
  • 50.
    Where do theslaves stand? Slave 1 Slave 2 Slave 3 Binary log Binary log Binary log Transactions Transactions Transactions ======== ======== ======== ======== ======== ======== ======== ======== ======== ======== ======== ======== Friday, November 11, 11 38
  • 51.
    Slave 3 becomes master New master Slave 1 Slave 2 Binary log Transactions ======== Binary log Binary log ======== ======== Transactions Transactions ======== ======== ======== ======== ======== ======== ======== ======== ======== Friday, November 11, 11 39
  • 52.
    Need to synchthe missing transactions New master Slave 1 Slave 2 Binary log Transactions ======== Binary log Binary log ======== ======== Transactions Transactions ======== ======== ======== ======== ======== ======== ======== ======== ======== Friday, November 11, 11 40
  • 53.
    Transaction position inthe new master binlog may not match positions in the old master binlog New master Slave 1 Slave 2 Binary log Transactions ? ? ======== Binary log Binary log ======== ======== Transactions Transactions ======== ======== ======== ======== ======== ======== ? ======== ? ======== ======== Friday, November 11, 11 41
  • 54.
    A failover scenario 1: Tungsten Replicator Friday, November 11, 11 42
  • 55.
    Slave 3 Master Replication Slave 2 Slave 1 THL Transactions THL -------- -------- Transactions -------- ======== -------- ======== ======== ======== Includes Global Transaction ID Friday, November 11, 11 43
  • 56.
    The master crashes Slave 3 Master Replication Slave 2 Slave 1 THL Transactions THL -------- -------- Transactions -------- ======== -------- ======== ======== ======== Friday, November 11, 11 44
  • 57.
    Where do theslaves stand? Slave 1 Slave 2 Slave 3 THL THL THL Transactions Transactions Transactions ======== ======== ======== ======== ======== ======== ======== ======== ======== ======== ======== ======== Friday, November 11, 11 45
  • 58.
    Slave 3 becomes master New master Slave 1 Slave 2 THL Transactions ======== THL THL ======== ======== Transactions Transactions ======== ======== ======== ======== ======== ======== ======== ======== ======== Friday, November 11, 11 46
  • 59.
    Need to synchthe missing transactions New master Slave 1 Slave 2 THL Transactions ======== THL THL ======== ======== Transactions Transactions ======== ======== ======== ======== ======== ======== ======== ======== ======== Friday, November 11, 11 47
  • 60.
    Transaction IDs inthe new master THL are understood immediately by the slaves New master Slave 1 Slave 2 Binary log Transactions ======== Binary log Binary log ======== ======== Transactions Transactions ======== ======== ======== ======== ======== ======== Give me ======== Give me ======== seqno #51 ======== seqno #67 Friday, November 11, 11 48
  • 61.
  • 62.
    parallel replication From here ... Friday, November 11, 11 50
  • 63.
    parallel replication To here. Friday, November 11, 11 51
  • 64.
    Parallel replication facts ✓Sharded by database ✓Good choice for slave lag problems ❖Bad choice for single database projects Friday, November 11, 11 52
  • 65.
    Benchmarking parallel replication Friday, November 11, 11 53
  • 66.
    db0 preparation (1) db1 sysbench db2 sysbench db3 sysbench db4 sysbench db5 sysbench db6 sysbench db7 sysbench db8 sysbench db9 sysbench sysbench Friday, November 11, 11 54
  • 67.
    before the test(2) STOPPED binary logs MySQL slave OFFLINE Tungsten slave direct: alpha (slave) replicator alpha Friday, November 11, 11 55
  • 68.
    starting the test STOPPED binary logs MySQL slave Concurrent sysbench on 30 databases running for 1 hour OFFLINE TOTAL DATA: 130 GB Tungsten slave direct: alpha RAM per server: 20GB (slave) replicator alpha Slaves will have 1 hour lag Friday, November 11, 11 56
  • 69.
    measuring results START binary logs MySQL slave ONLINE Recording catch-up time Tungsten slave direct: alpha (slave) replicator alpha Friday, November 11, 11 57
  • 70.
    MySQL native replication slave catch up in 04:29:30 Friday, November 11, 11 58
  • 71.
    Tungsten parallel replication slave catch up in 00:55:40 Friday, November 11, 11 59
  • 72.
    Parallel replication madesimpler FROM HERE .... Friday, November 11, 11 60
  • 73.
    Parallel replication madesimpler TO HERE Friday, November 11, 11 61
  • 74.
    Parallel replication madesimpler Friday, November 11, 11 62
  • 75.
    parallel replication direct slave facts Friday, November 11, 11 63
  • 76.
    parallel replication direct slave facts ✓No need to install Tungsten on the master Friday, November 11, 11 63
  • 77.
    parallel replication direct slave facts ✓No need to install Tungsten on the master ✓Tungsten runs only on the slave Friday, November 11, 11 63
  • 78.
    parallel replication direct slave facts ✓No need to install Tungsten on the master ✓Tungsten runs only on the slave ✓Replication can revert to native slave with two commands (trepctl offline; start slave) Friday, November 11, 11 63
  • 79.
    parallel replication direct slave facts ✓No need to install Tungsten on the master ✓Tungsten runs only on the slave ✓Replication can revert to native slave with two commands (trepctl offline; start slave) ✓Native replication can continue on other slaves Friday, November 11, 11 63
  • 80.
    parallel replication direct slave facts ✓No need to install Tungsten on the master ✓Tungsten runs only on the slave ✓Replication can revert to native slave with two commands (trepctl offline; start slave) ✓Native replication can continue on other slaves ❖Failover (either native or Tungsten) becomes a manual task Friday, November 11, 11 63
  • 81.
  • 82.
    multiple masters • Tungsten Replicator recipe: use more services Friday, November 11, 11 65
  • 83.
  • 84.
    Bi-directional replication withslaves Friday, November 11, 11 67
  • 85.
  • 86.
    multiple sources • Tungsten Replicator recipe is still valid: use more services Friday, November 11, 11 69
  • 87.
  • 88.
    Multiple masters replication:3 nodes Friday, November 11, 11 71
  • 89.
    Multiple masters replication:4 nodes Friday, November 11, 11 72
  • 90.
    4 nodes multiplemasters simplified Friday, November 11, 11 73
  • 91.
  • 92.
  • 93.
    Conflict prevention facts Friday, November 11, 11 76
  • 94.
    Conflict prevention facts • Sharded by database Friday, November 11, 11 76
  • 95.
    Conflict prevention facts • Sharded by database • Defined dynamically Friday, November 11, 11 76
  • 96.
    Conflict prevention facts • Sharded by database • Defined dynamically • Applied either at the master or at the slave Friday, November 11, 11 76
  • 97.
    Conflict prevention facts • Sharded by database • Defined dynamically • Applied either at the master or at the slave • methods: Friday, November 11, 11 76
  • 98.
    Conflict prevention facts • Sharded by database • Defined dynamically • Applied either at the master or at the slave • methods: • make replication fail Friday, November 11, 11 76
  • 99.
    Conflict prevention facts • Sharded by database • Defined dynamically • Applied either at the master or at the slave • methods: • make replication fail • drop silently Friday, November 11, 11 76
  • 100.
    Conflict prevention facts • Sharded by database • Defined dynamically • Applied either at the master or at the slave • methods: • make replication fail • drop silently • drop with warning Friday, November 11, 11 76
  • 101.
    Tungsten Conflict preventionpoints host1 host3 M S S host2 host4 master S M S Friday, November 11, 11 77
  • 102.
    Tungsten Conflict preventionpoints host1 host3 M S S host2 host4 master S M S Friday, November 11, 11 77
  • 103.
    Tungsten Conflict preventionpoints host1 host3 M S S host2 host4 master S M S Friday, November 11, 11 77
  • 104.
    Prevention methods: Failon master host1 host3 M S S host2 host4 master S M S Friday, November 11, 11 78
  • 105.
    INSERT A x,y Prevention methods: Fail on master host1 host3 M S S host2 host4 master S M S INSERT A x,z Friday, November 11, 11 78
  • 106.
    INSERT A x,y Prevention methods: Fail on master host1 host3 M S S host2 host4 master S M S INSERT A x,z Friday, November 11, 11 78
  • 107.
    Prevention methods: Failon slave host1 host3 M S S host2 host4 master S M S Friday, November 11, 11 79
  • 108.
    INSERT A x,y Prevention methods: Fail on slave host1 host3 M S S host2 host4 master S M S INSERT A x,z Friday, November 11, 11 79
  • 109.
    INSERT A x,y Prevention methods: Fail on slave host1 host3 M S S host2 host4 master S M S INSERT A x,z Friday, November 11, 11 79
  • 110.
    Prevention methods: Failon slave host1 (Multiple sources) M host2 host3 master M S Friday, November 11, 11 80
  • 111.
    INSERT A x,y Prevention methods: Fail on slave host1 (Multiple sources) M host2 host3 master M S INSERT A x,z Friday, November 11, 11 80
  • 112.
    INSERT A x,y Prevention methods: Fail on slave host1 (Multiple sources) M host2 host3 master M S INSERT A x,z Friday, November 11, 11 80
  • 113.
    Prevention methods: DROPon master host1 host3 M S S host2 host4 master S M S Friday, November 11, 11 81
  • 114.
    INSERT A x,y Prevention methods: DROP on master host1 host3 M S S host2 host4 master S M S Friday, November 11, 11 81
  • 115.
    INSERT A x,y Prevention methods: DROP on master host1 host3 M S S host2 host4 master S M S INSERT A x,z Friday, November 11, 11 81
  • 116.
    Prevention methods: DROPon master host1 host3 M S S host2 host4 master S M S INSERT A x,z Friday, November 11, 11 81
  • 117.
    Prevention methods: DROPon slave host1 (Multiple sources) M host2 host3 master M S Friday, November 11, 11 82
  • 118.
    INSERT A x,y Prevention methods: DROP on slave host1 (Multiple sources) M host2 host3 master M S INSERT A x,z Friday, November 11, 11 82
  • 119.
    Prevention methods: DROPon slave host1 (Multiple sources) M host2 INSERT A host3 x,y master INSERT A M x,z S Friday, November 11, 11 83
  • 120.
    Prevention methods: DROPon slave host1 (Multiple sources) M host2 host3 master INSERT A M x,z S Friday, November 11, 11 83
  • 121.
    Tungsten in practice Installation Friday, November 11, 11 84
  • 122.
    Installation • Check the requirements • Get the binaries • Expand the tarball • Run ./tools/tungsten-installer Friday, November 11, 11 85
  • 123.
    REQUIREMENTS • Java JRE or JDK (Sun/Oracle or Open-jdk) • Ruby 1.8 (only during installation) • ssh access to the same user in all nodes • MySQL user with all privileges Friday, November 11, 11 86
  • 124.
    Installation types • master / slave • slave - direct Friday, November 11, 11 87
  • 125.
    master/slave host2 host1 master slave THL binlog THL host3 slave THL Friday, November 11, 11 88
  • 126.
    slave direct host2 host1 slave master relay log THL binlog host3 slave relay log THL Friday, November 11, 11 89
  • 127.
    sample installation • 4 servers • qa.r1.continuent.com r1 • qa.r2.continuent.com r2 • qa.r3.continuent.com r3 • qa.r4.continuent.com r4 • MySQL 5.1 • tungsten-replicator-2.0.4 Friday, November 11, 11 90
  • 128.
    Installation (1) # starting at node 4, but any would do for N in 1 2 3 4 do ssh r$N mkdir tinstall done cd tinstall tar -xzf /path/to/tungsten-replicator-2.0.4.tar.gz cd tungsten-replicator-2.0.4 Friday, November 11, 11 91
  • 129.
    Installation (2) export TUNGSTEN_BASE=$HOME/tinstall 1 ./tools/tungsten-installer 2 --master-slave 3 --master-host=r1 4 --datasource-user=tungsten 5 --datasource-password=secret 6 --service-name=dragon 7 --home-directory=$TUNGSTEN_BASE 8 --cluster-hosts=r1,r2,r3,r4 9 --start Friday, November 11, 11 92
  • 130.
    Installation (2) export TUNGSTEN_BASE=$HOME/tinstall ./tools/tungsten-installer --master-slave # installation mode --master-host=r1 # who’s the master --datasource-user=tungsten # mysql username --datasource-password=secret # mysql password --service-name=dragon # name of the service --home-directory=$TUNGSTEN_BASE # where we install --cluster-hosts=r1,r2,r3,r4 # hosts in cluster --start # start replicator after installing Friday, November 11, 11 93
  • 131.
    What does the installation do 1: Validate all servers host4 host1 host2 host3 ✔ ✔ ✔ ✔ ✗ ✗ ✗ ✗ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Report all errors Friday, November 11, 11 94
  • 132.
    What does the installation do 1 (again): Validate all servers host4 host1 host2 host3 ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Friday, November 11, 11 95
  • 133.
    What does the installation do 2: install Tungsten in all servers host4 $HOME/ host1 tinstall/ host2 config/ host3 releases/ relay/ logs/ tungsten/ Friday, November 11, 11 96
  • 134.
    example ssh r2 chmod 444 $HOME/tinstall ./tools/tungsten-installer --master-slave --master-host=r1 --datasource-user=tungsten --datasource-password=secret --service-name=dragon --home-directory=$HOME/tinstall --thl-directory=$HOME/tinstall/logs --relay-directory=$HOME/tinstall/relay --cluster-hosts=r1,r2,r3,r4 --start ERROR >> qa.r2.continuent.com >> /home/tungsten/ tinstall is not writeable Friday, November 11, 11 97
  • 135.
    example ssh r2 chmod 755 $HOME/tinstall ./tools/tungsten-installer --master-slave --master-host=r1 --datasource-user=tungsten --datasource-password=secret --service-name=dragon --home-directory=$HOME/tinstall --thl-directory=$HOME/tinstall/logs --relay-directory=$HOME/tinstall/relay --cluster-hosts=r1,r2,r3,r4 --start # no errors Friday, November 11, 11 98
  • 136.
    Installation - direct ./tools/tungsten-installer --direct --master-host=r1 --slave-host=r4 --master-user=tungsten --slave-user=tungsten --master-password=secret --slave-password=secret --service-name=Castor --channels=1 --home-directory=$TUNGSTEN_BASE --start-and-report Friday, November 11, 11 99
  • 137.
  • 138.
    Tungsten Replicator vs.Enterprise Feature Replicator Enterprise Simple admin with trepctl X X MySQL Row/Statement 5.0/5.1/5.5 X X Multi-master replication X X Parallel apply (2.0) X X Distributed management with cctrl X Automated failover and recovery X Zero-downtime upgrade X SQL routing and load-balancing X Floating IP support X © Continuent 2011 Friday, November 11, 11 101
  • 139.
    Conclusion and Q&A Friday,November 11, 11 102
  • 140.
    Contact Information Worldwide 560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 @datacharmer Tel (866) 998-3642 Fax (408) 668-1009 @continuent e-mail: sales@continuent.com Continuent Web Site: http://www.continuent.com Tungsten Project http://tungsten-replicator.org Getting started with Tungsten Replicator: http://datacharmer.blogspot.com © Continuent 2011 Friday, November 11, 11 103
  • 141.