Your SlideShare is downloading. ×
0
MySQL Labs :
Multi Source Replication

Shivji Kumar Jha,
Software Developer,
MySQL Replication Team

1

| Copyright © 2013...
Safe Harbour Statement
The following is intended to outline our general product direction. It is intended for
information ...
MySQL Labs features are not fit for production.
They are provided solely for testing purposes, to try the latest bug fixes...


Agenda

Background: Why Use Replication?



Multi-source Replication
– Introduction
– Use cases
– Internals
– Monitori...
Background: What is Replication Used For?
 Read scale-out

S

M

S

More
reads?
More
slaves!

M

S
S

write clients
write...
What about Write scale-out?

M?
M

More
writes?
More
Masters?

M?
M?

write clients
Enter MySQL Fabric ...
6

| Copyright ...
Background: What is Replication Used For?
 Redundancy: If master crashes, promote slave to master

B
A

B
Uh Oh!

C

B
Wh...
Background: What is Replication Used For?
 On-line Backup and Reporting

S
M




write clients



8

| Copyright © 201...
Background: What is Replication Used For?
A
C

A
B
B
C

Image from
www.ginkgomaps.com

9

| Copyright © 2013, Oracle and/o...


Agenda

Background: Why Use Replication?



Multi-source Replication
– Introduction
– Use cases
– Internals
– Monitori...
Multi-Source Replication: Introduction
 A Slave can have more than one master.
M1
M2
M3
M4
11

| Copyright © 2013, Oracle...
Multi-Source Replication: Introduction
 A Slave can have more than one master.
db1

12

M1

| Copyright © 2013, Oracle an...
Multi-Source Replication: Introduction
 A Slave can have more than one master.
db1
db2

13

M1
M2

| Copyright © 2013, Or...
Multi-Source Replication: Introduction
 A Slave can have more than one master.
db1
db2

M2

db3

M3

db4

14

M1

M4

| C...
Multi-Source Replication: Introduction
 A Slave can have more than one master.
db1

M1

db2

M2
S

db3
db4

15

M3
M4

| ...


Agenda

Background: Why Use Replication?



Multi-source Replication
– Introduction
– Use cases
– Internals
– Monitori...
Multi-Source Replication: Use Cases
 The main use cases of Multi-source replication are related to data aggregation.

17
...
Multi-Source Replication: Use Cases
Database 1

M1
Database 2

Database 1, 2, 3

M2
Database 3

M3

18

| Copyright © 2013...
Multi-Source Replication: Use Cases
Shard 1

M
Shard 2

Full table

M
Shard 3

M

19

| Copyright © 2013, Oracle and/or it...


Agenda

Background: Why Use Replication?



Multi-source Replication
– Introduction
– Use cases
– Internals
– Monitori...
Multi-Source Replication: More details..
 Slave can have more than one

master.

M1

– Receive transactions from

several...
Multi-Source Replication: What is a Channel?
 A channel is an an abstraction for a sender-receiver-applier path.

Client
...
Multi-Source Replication: How Many Channels?
 Number of channels on slave = Number of sources.

M1

Network

relay log1

...
Multi-Source Replication: Slave Appliers
 Each channel has its own single-threaded slave applier.

M1

Network

relay log...
Multi-Source Replication: Slave Appliers
 Each channel has its own multi-threaded slave applier.

M1

Network

relay log1...
Adding / Configuring A Channel
 Each channel can be configured individually.
 To add or alter a channel configuration us...
Replication Commands on a Channel
 The FOR CHANNEL=”<channel_name>” clause is consistent

across different replication co...
Working With All Channels
 The FOR ALL CHANNELS clause.

– START SLAVE [thread_type] FOR ALL CHANNELS
– STOP SLAVE [threa...
Compatibility With Other Replication Features
Multi-Threaded Slaves
Global Transaction Identifiers

Yes

Semi-Synchronous ...


Agenda

Background: Why Use Replication?



Multi-source Replication
– Introduction
– Use cases
– Internals
– Monitori...
Replication Monitoring: P_S Replication Tables
Slave Status

Connection
Configuration

Connection
Status

Execution
Config...
Replication Monitoring: P_S Replication Tables
 We have six performance schema tables for replication (MySQL-5.7.2):
– re...
Replication Monitoring: Connection Status
mysql> select * from performance_schema.replication_connection_statusG
*********...
Replication Monitoring: Connection Status
mysql> select * from performance_schema.replication_connection_statusG
*********...
Replication Monitoring: Connection Status
mysql> select * from performance_schema.replication_connection_statusG
*********...
Replication Monitoring: Connection Status
mysql> select * from performance_schema.replication_connection_statusG
*********...
Replication Monitoring: Connection Status
mysql> select * from performance_schema.replication_connection_statusG
*********...


Agenda

Background: Why Use Replication?



Multi-source Replication
– Introduction
– Use cases
– Internals
– Monitori...
Read More About Multi Source Replication






39

Read more about Multi-source replication from the design notes:
http...
Read More About Multi Source Replication




40

Official MySQL documentation for replication P_S tables:
http://dev.mys...
Upcoming SlideShare
Loading in...5
×

Open source India - MySQL Labs: Multi-Source Replication

654

Published on

This session was presented in Open source India 2013. The presentation covers MySQL multi-source replication which is released under MySQL Labs.

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
654
On Slideshare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
16
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Open source India - MySQL Labs: Multi-Source Replication"

  1. 1. MySQL Labs : Multi Source Replication Shivji Kumar Jha, Software Developer, MySQL Replication Team 1 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
  2. 2. Safe Harbour Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. 2 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
  3. 3. MySQL Labs features are not fit for production. They are provided solely for testing purposes, to try the latest bug fixes and generally to keep up with the development. Please, do not use these binaries in production. Instead, install them on a spare server. If you are looking for production ready binaries, please visit MySQL Downloads. 3 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
  4. 4.  Agenda Background: Why Use Replication?  Multi-source Replication – Introduction – Use cases – Internals – Monitoring  4 Reading More about Multi-source Replication | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
  5. 5. Background: What is Replication Used For?  Read scale-out S M S More reads? More slaves! M S S write clients write clients 5 read clients | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. | read clients
  6. 6. What about Write scale-out? M? M More writes? More Masters? M? M? write clients Enter MySQL Fabric ... 6 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. | write clients
  7. 7. Background: What is Replication Used For?  Redundancy: If master crashes, promote slave to master B A B Uh Oh! C B Whew! Crash A A C C B is the new master 7 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
  8. 8. Background: What is Replication Used For?  On-line Backup and Reporting S M   write clients  8 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |  business intelligent client apps reporting client apps big queries client apps
  9. 9. Background: What is Replication Used For? A C A B B C Image from www.ginkgomaps.com 9 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
  10. 10.  Agenda Background: Why Use Replication?  Multi-source Replication – Introduction – Use cases – Internals – Monitoring  10 Reading More about Multi-source Replication | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
  11. 11. Multi-Source Replication: Introduction  A Slave can have more than one master. M1 M2 M3 M4 11 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. | S
  12. 12. Multi-Source Replication: Introduction  A Slave can have more than one master. db1 12 M1 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
  13. 13. Multi-Source Replication: Introduction  A Slave can have more than one master. db1 db2 13 M1 M2 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
  14. 14. Multi-Source Replication: Introduction  A Slave can have more than one master. db1 db2 M2 db3 M3 db4 14 M1 M4 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
  15. 15. Multi-Source Replication: Introduction  A Slave can have more than one master. db1 M1 db2 M2 S db3 db4 15 M3 M4 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. | db1 db2 db3 db4
  16. 16.  Agenda Background: Why Use Replication?  Multi-source Replication – Introduction – Use cases – Internals – Monitoring  16 Reading More about Multi-source Replication | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
  17. 17. Multi-Source Replication: Use Cases  The main use cases of Multi-source replication are related to data aggregation. 17 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
  18. 18. Multi-Source Replication: Use Cases Database 1 M1 Database 2 Database 1, 2, 3 M2 Database 3 M3 18 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. | S Business Intelligence Data analytics Backup etc
  19. 19. Multi-Source Replication: Use Cases Shard 1 M Shard 2 Full table M Shard 3 M 19 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. | S
  20. 20.  Agenda Background: Why Use Replication?  Multi-source Replication – Introduction – Use cases – Internals – Monitoring  20 Reading More about Multi-source Replication | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
  21. 21. Multi-Source Replication: More details..  Slave can have more than one master. M1 – Receive transactions from several MySQL servers simultaneously. M2 S – Apply transactions from different masters simultaneously. M3 – No conflict detection or resolution. M4 21 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
  22. 22. Multi-Source Replication: What is a Channel?  A channel is an an abstraction for a sender-receiver-applier path. Client Insert... Receiver thread Sender thread Applier thread binary log binary log Insert... A relay log Insert... Insert... Network Channel 22 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. | B
  23. 23. Multi-Source Replication: How Many Channels?  Number of channels on slave = Number of sources. M1 Network relay log1 M2 binary log Network relay log2 M3 23 binary log binary log Network relay log3 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. | S binary log
  24. 24. Multi-Source Replication: Slave Appliers  Each channel has its own single-threaded slave applier. M1 Network relay log1 M2 binary log Network relay log2 M3 24 binary log binary log Network relay log3 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. | S binary log
  25. 25. Multi-Source Replication: Slave Appliers  Each channel has its own multi-threaded slave applier. M1 Network relay log1 M2 binary log Network relay log2 M3 25 binary log binary log Network relay log3 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. | S binary log
  26. 26. Adding / Configuring A Channel  Each channel can be configured individually.  To add or alter a channel configuration use: CHANGE MASTER TO master_def … FOR CHANNEL=”<channel_name>” 26 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
  27. 27. Replication Commands on a Channel  The FOR CHANNEL=”<channel_name>” clause is consistent across different replication commands. – start slave [...] FOR CHANNEL=”<channel_name>" – stop slave [thread_types] FOR CHANNEL=”<channel_name>” – reset slave [all] FOR CHANNEL=”<channel_name>” – flush relay logs FOR CHANNEL=”<channel_name>” – show relay log events FOR CHANNEL=”<channel_name>” ... 27 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
  28. 28. Working With All Channels  The FOR ALL CHANNELS clause. – START SLAVE [thread_type] FOR ALL CHANNELS – STOP SLAVE [thread_type] 28 FOR ALL CHANNELS | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
  29. 29. Compatibility With Other Replication Features Multi-Threaded Slaves Global Transaction Identifiers Yes Semi-Synchronous Replication Yes Crash Safe Slaves Yes Filters 29 Yes Yes | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
  30. 30.  Agenda Background: Why Use Replication?  Multi-source Replication – Introduction – Use cases – Internals – Monitoring  30 Reading More about Multi-source Replication | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
  31. 31. Replication Monitoring: P_S Replication Tables Slave Status Connection Configuration Connection Status Execution Configuration Execution Status Applier / Coordinator Status 31 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. | Workers Status
  32. 32. Replication Monitoring: P_S Replication Tables  We have six performance schema tables for replication (MySQL-5.7.2): – replication_connection_configuration – replication_connection_status One row per CHANNEL – replication_execute_configuration – replication_execute_status – replication_execute_status_by_coordinator – replication_execute_status_by_worker One row per CHANNEL per WORKER  Consistent semantics across tables. Lets explore one of them. 32 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
  33. 33. Replication Monitoring: Connection Status mysql> select * from performance_schema.replication_connection_statusG *************************** 1. row *************************** CHANNEL_NAME : CHANNEL1 SOURCE_UUID : 7cff7406-23ca-11e3-ac3e-5c260a83b12b THREAD_ID : 13 SERVICE_STATE : ON RECEIVED_TRANSACTION_SET : 7cff7406-23ca-11e3-ac3e-5c260a83b12b:1-4 LAST_ERROR_NUMBER :0 LAST_ERROR_MESSAGE : LAST_ERROR_TIMESTAMP : 0000-00-00 00:00:00 *************************** 2. row *************************** CHANNEL_NAME : CHANNEL2 ... One row per CHANNEL 33 33 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
  34. 34. Replication Monitoring: Connection Status mysql> select * from performance_schema.replication_connection_statusG *************************** 1. row *************************** CHANNEL_NAME : CHANNEL1 SOURCE_UUID : 7cff7406-23ca-11e3-ac3e-5c260a83b12b THREAD_ID : 13 SERVICE_STATE : ON RECEIVED_TRANSACTION_SET : 7cff7406-23ca-11e3-ac3e-5c260a83b12b:1-4 LAST_ERROR_NUMBER :0 LAST_ERROR_MESSAGE : Receiver thread & LAST_ERROR_TIMESTAMP : 0000-00-00 00:00:00 *************************** 2. row *************************** its service state CHANNEL_NAME : CHANNEL2 ... 34 34 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
  35. 35. Replication Monitoring: Connection Status mysql> select * from performance_schema.replication_connection_statusG *************************** 1. row *************************** CHANNEL_NAME : CHANNEL1 SOURCE_UUID : 7cff7406-23ca-11e3-ac3e-5c260a83b12b THREAD_ID : 13 SERVICE_STATE : ON RECEIVED_TRANSACTION_SET : 7cff7406-23ca-11e3-ac3e-5c260a83b12b:1-4 LAST_ERROR_NUMBER :0 LAST_ERROR_MESSAGE : LAST_ERROR_TIMESTAMP : 0000-00-00 00:00:00 *************************** 2. row *************************** CHANNEL_NAME : CHANNEL2 Set of transactions received ... through this channel 35 35 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
  36. 36. Replication Monitoring: Connection Status mysql> select * from performance_schema.replication_connection_statusG *************************** 1. row *************************** CHANNEL_NAME : CHANNEL1 SOURCE_UUID : 7cff7406-23ca-11e3-ac3e-5c260a83b12b THREAD_ID : 13 SERVICE_STATE : ON RECEIVED_TRANSACTION_SET : 7cff7406-23ca-11e3-ac3e-5c260a83b12b:1-4 LAST_ERROR_NUMBER : 1045 LAST_ERROR_MESSAGE : error connecting to master 'replssl@127.0.0.1:13000' ... LAST_ERROR_TIMESTAMP : 2013-11-04 13:37:23 *************************** 2. row *************************** CHANNEL_NAME : CHANNEL2 ... Oops! There was an error in the receiver thread on this channel 36 36 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
  37. 37. Replication Monitoring: Connection Status mysql> select * from performance_schema.replication_connection_statusG *************************** 1. row *************************** CHANNEL_NAME : CHANNEL1 Receiver SOURCE_UUID : 7cff7406-23ca-11e3-ac3e-5c260a83b12b thread on this channel stopped serving... THREAD_ID : NULL SERVICE_STATE : OFF RECEIVED_TRANSACTION_SET : 7cff7406-23ca-11e3-ac3e-5c260a83b12b:1-4 LAST_ERROR_NUMBER : 1045 LAST_ERROR_MESSAGE : error connecting to master 'replssl@127.0.0.1:13000' ... LAST_ERROR_TIMESTAMP : 2013-11-04 13:37:23 *************************** 2. row *************************** CHANNEL_NAME : CHANNEL2 ... 37 37 | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
  38. 38.  Agenda Background: Why Use Replication?  Multi-source Replication – Introduction – Use cases – Internals – Monitoring  38 Reading More about Multi-source Replication | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
  39. 39. Read More About Multi Source Replication    39 Read more about Multi-source replication from the design notes: http://dev.mysql.com/worklog/task/?id=1697 Read a feature preview of Multi-source replication on Rith's blog: http://on-mysql-replication.blogspot.in/2013/09/feature-preview-mysqlmulti-source-replication.html Read more about Multi-source replication on Rith's blog: http://on-mysql-replication.blogspot.in/2013/09/mysql-labs-multi-sour ce-replication.html | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
  40. 40. Read More About Multi Source Replication   40 Official MySQL documentation for replication P_S tables: http://dev.mysql.com/doc/refman/5.7/en/performance-schema-replicati on-tables.html Read more about replication P_S tables on Shiv's blog: http://shivjijha.blogspot.com/2013/09/Monitoring-Replication-with-t he-NEW-performance-schema-tables.html | Copyright © 2013, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, November 13 th, 2013. |
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×