• Like
  • Save

Repair & Recovery for your MySQL, MariaDB & MongoDB / TokuMX Clusters - Webinar Slides

  • 2,316 views
Uploaded on

Slides for the webinar held on January 21st 2014 …

Slides for the webinar held on January 21st 2014

Repair & Recovery for your MySQL, MariaDB & MongoDB / TokuMX Clusters

Galera Cluster, NDB Cluster, VIP with HAProxy and Keepalived, MongoDB Sharded Cluster, etc. all have their own availability models. We are aware of these availability models and will demonstrate in this webinar how to take corrective action in case of failures via our cluster management tool, ClusterControl.

In this webinar, Severalnines CTO Johan Andersson will show you how to leverage ClusterControl to detect failures in your database cluster and automatically repair them to maximize the availability of your database services. And Codership CEO Seppo Jaakola will be joining Johan to provide a deep-dive into Galera recovery internals.

Agenda:

Redundancy models for Galera, NDB and MongoDB/TokuMX
Failover & Recovery (Automatic vs Manual)
Zooming into Galera recovery procedures
Split brains in multi-datacenter setups

More in: Technology , Travel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
2,316
On Slideshare
0
From Embeds
0
Number of Embeds
5

Actions

Shares
Downloads
0
Comments
0
Likes
6

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Repair and Recovery for your MySQL, MariaDB and MongoDB/ TokuMX clusters January 2013 Johan Andersson Severalnines AB johan@severalnines.com Mobile +46 73 073 60 99
  • 2. Copyright 2014 Severalnines AB Agenda !  Redundancy models for Galera, NDB and MongoDB/ TokuMX !  Failover & Recovery (Automatic vs Manual) !  Zooming into Galera recovery procedures !  Split brains in multi-datacenter setups 2
  • 3. Copyright 2014 Severalnines AB Redundancy models for Galera, NDB and MongoDB/TokuMX !  Database Clustering can be divided in two areas !  Asynchronously replicated data !  MySQL Replication !  TokuMX/MongoDB !  Synchronously replicated data !  Galera for MySQL !  NDB Cluster (aka MySQL Cluster) 3
  • 4. Copyright 2014 Severalnines AB Asynchronous 1 2 3
  • 5. Copyright 2014 Severalnines AB Asynchronous Pros Cons !  Read scaling is easy !  Master handles all writes !  Read write splitting !  Lower write latency !  Automated failover is hard: !  Elect a new master !  Slave lag & Data loss !  Has everything been sent to the slave when the master crashes? !  Has the slave applied all data? 5
  • 6. Copyright 2014 Severalnines AB Synchronous 1 4 2 3
  • 7. Copyright 2014 Severalnines AB Synchronous Pros Cons !  All servers are masters !  Higher write latency !  Application can read and write on any server !  Sensitive to link latency !  No data loss !  Network partitioning !  Simple failover for apps 7
  • 8. Copyright 2014 Severalnines AB Redundancy Models TokuMX/MongoDB ReplicaSet (1/2) !  Up to 12 members !  Max 7 voting numbers !  Primary, Secondary, Arbiter !  Slave Delay !  Chaining Allowed !  Heartbeats every 2 seconds !  5 missed heartbeats: node dead Primary Secondary Arbiter 8
  • 9. Copyright 2014 Severalnines AB Redundancy Models TokuMX/MongoDB ReplicaSet (2/2) !  Automatic failover !  Priority (between 0 and 100.0) !  Primary has highest priority !  Influences election of new primary Data Center 1 Primary Secondary Data Center 2 Secondary 9
  • 10. Copyright 2014 Severalnines AB Redundancy Models TokuMX/MongoDB Sharded Cluster !  Built-in support for sharding !  Min 2 Query Routers !  3 Config Servers Primary Secondary Arbiter Primary Secondary Arbiter 10
  • 11. Copyright 2014 Severalnines AB Redundancy Models MySQL Replication !  No hard limits on no. of slaves !  Replication chaining !  Delayed slave !  Manual failover Slave Master Slave 11
  • 12. Copyright 2014 Severalnines AB Redundancy Models MySQL Cluster !  In-built sharding !  Separate SQL and Data layers !  Up to 4 replicas/node group !  Max 48 data nodes !  Auto-failover Node Group 1 Node Group 2 12
  • 13. Copyright 2014 Severalnines AB Redundancy Models Galera Cluster !  Synchronous replication plugin !  All masters have same dataset !  Slowest node determines speed of cluster !  No hard limits on no. of nodes !  Auto-failover !  Galera arbitrator (garbd) Master Master Master 13
  • 14. Copyright 2014 Severalnines AB Synchronous Replication Linear Two-phase commit !  Used by NDB data nodes !  Pessimistic locking !  Works well for cases with update contention 1 4 6 3 2 5 1. Prepare transaction 4. commit 14
  • 15. Copyright 2014 Severalnines AB Synchronous Replication Certificate-based !  Used by Galera/WSREP !  Optimistic locking !  Not optimal for cases with conflicting updates 1 2 3 4 1. begin transaction 2. commit 15
  • 16. Copyright 204Severalnines AB Zooming into Galera recovery procedures By Seppo Jaakola @ Codership 16
  • 17. Copyright 2014 Severalnines AB Split-brain !  Network partitioning (aka) split brain. N nodes divided into M equal parts Galera Servers - SYNCED DC1 WAN Galera Servers - SYNCED DC2 17
  • 18. Copyright 2014 Severalnines AB Split-brain !  WAN breaks - 2 equal splits (Network partitioning) !  Non-Primary – cannot handle transactions Galera – Non-Primary DC1 WAN Galera – Non-Primary DC2 18
  • 19. Copyright 2014 Severalnines AB Split-brain prevention !  Solution: The side with the majority of nodes will win: !  Add garbd or a galera node to DC1 or DC2 !  Add weight to existing node in DC1: wsrep_provider_options=“pc.weight=N;” (default = 1 ) Galera – Synced DC1 WAN Galera – Synced DC2 19
  • 20. Copyright 2014 Severalnines AB Split-brain prevention !  Add weight to existing node in DC1: wsrep_provider_options=“pc.weight=2;” Galera – Synced DC1 pc.weight=2 WAN 1 3 Galera – Synced DC2 1 1 2 20
  • 21. Copyright 2014 Severalnines AB Split-brain prevention !  Final Two Data center setup with an additional node. Galera – Synced DC1 WAN Galera – Synced DC2 21
  • 22. Copyright 2014 Severalnines AB Split-brain prevention !  Three data centers, split brain unlikely. !  Galera 3.0 supports segments (gmcast.segment) !  Only one node in each segment will relay to only one other node in another segment !  Segments also applies to Two Data centers, and between racks in one DC. WAN Galera – Segment 3 DC2 Galera – Segment 1 DC1 Galera – Segment 2 DC3
  • 23. Copyright 2014 Severalnines AB Q&A 23
  • 24. Copyright 2014 Severalnines AB Thank you. For further questions please contact John-Jerome Schmidt: jj@severalnines.com 24