Severalnines Self-Training: MySQL® Cluster - Part IV

MySQL Cluster Training presented by severalnines.com Address: Contact: SeveralninesAB Jean-Jérôme Schmidt c/o SICS, Box 1263 Email: services@severalnines.com Isafjordsgatan22 SE-164-29 KistaCopyright 2011 Severalnines AB Control your database infrastructure 1

Introduction • At Severalnines, we believe in sharing information and knowledge; we all come from an open source background • We know a lot of things about MySQL Cluster and think that MySQL Cluster is a great technology • These free MySQL Cluster Training slides are a contribution of ours to the knowledge and information sharing that‟s common practice in the open source community • If you have any questions on these slides or would like to book an actual training class, please contact us at: services@severalnines.comCopyright 2011 Severalnines AB Control your database infrastructure 2

Training Slides - Concept • Over the coming weeks we will be chronologically releasing slides for the different sections of our MySQL Cluster Training program on our website. • The full agenda of the training with all of its modules is outlined in the next slides so that you can see what topics will be covered over the coming weeks. • Particularly specialised topics such as Cluster/J or NDB API are not fully covered in the slides. We recommend our instructor-led training classes for such topics. • Please contact us for more details: services@severalnines.comCopyright 2011 Severalnines AB Control your database infrastructure 3

Full Training Agenda (1/4) • MySQL Cluster Introduction – MySQL eco system – Scale up, scale out, and sharding – MySQL Cluster Architecture – Use cases – Features – Node types and Roles • Detailed Concepts – Data Distribution – Verifying data distribution – Access Methods – Partitioning – Node failures and failure detection – Network Partitioning – Transactions and Locking – Consistency Model – Redo logging and Checkpointing • Internals – NDB Design InternalsCopyright 2011 Severalnines AB Control your database infrastructure 4

Agenda (2/4) • Installing MySQL Cluster – Setting up MySQL Cluster – Starting/stopping nodes – Recovery and restarts – Upgrading configuration – Upgrading Cluster • Performance Tuning (instructor-led only; contact us at services@severalnines.com) – Differences compared to Innodb/MyISAM – Designing efficient and fast applications – Identifying bottlenecks – Tweaking configuration (OS and MySQL Cluster) – Query Tuning – Schema Design – Index TuningCopyright 2011 Severalnines AB Control your database infrastructure 5

Agenda (3/4) • Management and Administration – Backup and Restore – Geographical Replication – Online and offline operations – Ndbinfo tables – Reporting – Single user mode – Scaling Cluster • Disk Data – Use cases – Limitations – Best practice configuration • Designing a Cluster – Capacity Planning and Dimensioning – Hardware recommendations – Best practice Configuration – Storage calculationsCopyright 2011 Severalnines AB Control your database infrastructure 6

Agenda (4/4) • Resolving Issues – Common problems – Error logs and Tracefiles – Recovery and Escalation procedures • Connectivity Overview – NDBAPI – Cluster/J – LDAP • Severalnines Tools – Monitoring and Management – Benchmarking – Sandboxes – Configuration and capacity planning • ConclusionCopyright 2011 Severalnines AB Control your database infrastructure 7

Agenda: Lab Exercises (only applicable to instructor-led training classes) • Lab Exercises – Installing and Loading data into MySQL Cluster – Starting/stopping nodes, recovery – Query tuning – Backup and Restore – Configuration Upgrade • Would you like to try something particular? – This is possible too, speak with your instructorCopyright 2011 Severalnines AB Control your database infrastructure 8

Prerequisites • Readers / Participants have understanding of SQL and basic database concepts. • Laptops/PCs for hands-on exercises • Linux: 1GB RAM • Windows: 2GB RAM • Approx. 20GB disk space and Virtualbox installed. • Virtualbox can be downloaded for free at http://www.virtualbox.org/wiki/Downloads • MySQL Cluster version 7.1 or laterCopyright 2011 Severalnines AB Control your database infrastructure 9

Recovery • There are four types of Recovery in MySQL Cluster – Node Recovery • Fastest recovery option – Initial Node Recovery – System Restart – Initial System Restart plus restoring backup • We will go over how to perform recovery in the administration part.Copyright 2011 Severalnines AB Control your database infrastructure 13

Node Recovery • Data Node 2 Crashes during LCP 1 LCP 0 Complete LCP 1 Complete LCP 1 Starts timeDATA NODE 1 1A C 2B C 3F C 4D C 5G C 8H C 1X C REDOLOG subid data subid data 1 A LCP 1 A 2 B Updated 2 B 3 C 1A 1X! 4 DDATA NODE 2 1A C 2B C 3F C 4D C REDOLOG subid data DATA NODE 2 1 A CRASHES JUST AFTER 2 B GCP 8 Copyright 2011 Severalnines AB Control your database infrastructure 15

Node Recovery • Data Node 2 will: – Read in its last completed LCP (LCP 0) – Read it its REDO LOG until GCP 8 • But Data Node 1 continued to live and has more changes. – These changes needs to be copied from Data Node 1 to Data Node 2. 5G C 8H C 1X C GCP 10 GCP 11 GCP 12Copyright 2011 Severalnines AB Control your database infrastructure 16

Node Recovery • Data Node 2 sends a request to Data Node 1 – Send all data from GCP 8 and onwards – Remember the record header contains when the record was last modified. • Data Node 1 starts a scan and scans partition by partition, table by table. Start scan GCP subid data GCP subid data 12 1 AX 2 1 A 4 2 B 4 2 B 6 3 C 6 3 C 8 4 D 8 4 D 10 5 G 11 8 HCopyright 2011 Severalnines AB Control your database infrastructure 17

Node Recovery • As soon as the Copying starts, Data node 2 will take part of the 2PC protocol – But TC is inactive until recovery is complete. • This means that data already copied will be changed to • UPDATE subsriber SET data=„AAA‟ WHERE subid=3 • INSERT INTO subscriber(subid, data) VALUES(10, „BB‟) Start scan GCP subid data GCP subid data COPY 12 1 AX 12 1 A X 4 2 B 4 2 B 2PC 32 3 AAA 32 3 AAA 8 4 D 8 4 D 10 5 G 10 5 G 11 8 H 11 8 H 24 10 BBCopyright 2011 Severalnines AB Control your database infrastructure 18

Node Recovery • As soon as the Copying starts, Data node 2 will take part of the 2PC protocol – But TC is inactive until recovery is complete. • This means that data already copied will be changed too. • UPDATE subsriber SET data=„AAA‟ WHERE subid=3 • INSERT INTO subscriber(subid, data) VALUES(10, „BB‟) Start scan GCP subid data GCP subid data COPY 12 1 AX 12 1 A X 4 2 B 4 2 B 2PC 32 3 AAA 32 3 AAA 8 4 D 8 4 D COPY 10 5 G 10 5 G COPY 11 8 H 11 8 H COPY 24 10 BB 24 10 BB End scanCopyright 2011 Severalnines AB Control your database infrastructure 19

Node Recovery • Scan completed on Node 1 of all partitions for all tables. • TC on Node 2 becomes Active DATA DATA NODE 1 NODE 2 GCP subid data GCP subid data 12 1 AX 12 1 A X 4 2 B 4 2 B 6 3 AAA 6 3 AAA 8 4 D 8 4 D 10 5 G 10 5 G 11 8 H 11 8 H 24 10 BB 24 10 BBCopyright 2011 Severalnines AB Control your database infrastructure 20

Node Failures • What happens if Data Node 2 fails? • Node failure detected. Fail-over of primary replica F1 to node 2 – Fail-over time dependent on underlying OS and configuration (e.g. OSE Delta 10 ms, Linux 100 ms) STORAGE LAYER subid data 1 A Partition 0 2 B DATA DATA DATA DATA NODE 1 NODE 2 NODE 3 NODE 4 3 C Partition 1 4 D P0 P1 P2 P3 5 E Partition 2 6 F S1 S0 S3 S2 7 G Partition 3 Node group 0 Node group 1 8 H Px == PRIMARY Partition x Sx == SECONDARY ParttionxCopyright 2011 Severalnines AB Control your database infrastructure 22

Node Failures – Data Node 2 Failed • Data Node 1 activates S1 and is now the PRIMARY for Data Node 2‟s partitions. • TC on Data Node 1 will take over any outstanding transactions of Node 2. STORAGE LAYER subid data 1 A Partition 0 2 B DATA DATA DATA DATA NODE 1 NODE 2 NODE 3 NODE 4 3 C Partition 1 4 D P0 P1 P2 P3 5 E Partition 2 6 F S1  P1 S0 S3 S2 7 G Partition 3 Node group 0 Node group 1 8 H Px == PRIMARY Partition x Sx == SECONDARY ParttionxCopyright 2011 Severalnines AB Control your database infrastructure 23

Node Failures – Data Node 3 Failed • Data Node 4 activates S2 and is now the PRIMARY for Data Node 3‟s partitions. STORAGE LAYER subid data 1 A Partition 0 2 B DATA DATA DATA DATA NODE 1 NODE 2 NODE 3 NODE 4 3 C Partition 1 4 D P0 P1 P2 P3 5 E Partition 2 6 F S1  P1 S0 S3 S2P2 7 G Partition 3 Node group 0 Node group 1 8 H Px == PRIMARY Partition x Sx == SECONDARY ParttionxCopyright 2011 Severalnines AB Control your database infrastructure 24

Node Failures • After recovery the role of the partitions are toggled back. STORAGE LAYER subid data 1 A Partition 0 2 B DATA DATA DATA DATA NODE 1 NODE 2 NODE 3 NODE 4 3 C Partition 1 4 D P0 P1 P2 P3 5 E Partition 2 6 F P1S1 S0 S3 P2S2 7 G Partition 3 Node group 0 Node group 1 8 H Px == PRIMARY Partition x Sx == SECONDARY ParttionxCopyright 2011 Severalnines AB Control your database infrastructure 25

If a Nodegroup goes down • Cluster goes down • Partition 2 and Partition 3 are lost, and all Partitions must be present to have a functioning Cluster. STORAGE LAYER subid data 1 A Partition 0 2 B DATA DATA DATA DATA NODE 1 NODE 2 NODE 3 NODE 4 3 C Partition 1 4 D P0 P1 P2 P3 5 E Partition 2 6 F P1S1 S0 S3 P2S2 7 G Partition 3 Node group 0 Node group 1 8 H Px == PRIMARY Partition x Sx == SECONDARY ParttionxCopyright 2011 Severalnines AB Control your database infrastructure 26

Node Failures and Error Detection • There are two ways the data nodes can detect that another node has failed. – Socket to remote host was closed (TCP Disconnect) – Using HeartbeatsCopyright 2011 Severalnines AB Control your database infrastructure 27

Node Failures and Error Detection • There are two ways the data nodes can detect that another node has failed. – TCP Disconnect – Heartbeats • Internally on each node a Watchdog ensures that a data node does not get stuck: – Monitors that a node does not end up in an eternal loop – Protects against software bugs • Missed Heartbeats are typically because of – Data node is swapping – Computer is using CPU for something else – Unreliable network – Bug in your NDBAPI applicationCopyright 2011 Severalnines AB Control your database infrastructure 28

Heartbeat Circle – Normal Operation • HearbeatIntervalDbDb regulates how often hearbeat checks are made. Default: 5000ms, Recommended in Telco: 1500ms • After 4xHearbeatIntervalDbDb node is declared dead STORAGE LAYER DATA DATA DATA OK DATA OK NODE 3 OK NODE 4 NODE 1 NODE 2 OKCopyright 2011 Severalnines AB Control your database infrastructure 29

Missed Heartbeats • Missed Heartbeats are typically because of – Data node is swapping – Computer is using CPU for something else – Unreliable network – Bug in your NDBAPI applicationCopyright 2011 Severalnines AB Control your database infrastructure 30

Heartbeat Circle – Node 3 misses HB • Node 3 has missed 4 heart beats from Node 2 • Node 2 declares Node 3 as dead. STORAGE LAYER DATA DATA DATA DATA NODE 1 OK NODE 2 NODE 3 NODE 4 OKCopyright 2011 Severalnines AB Control your database infrastructure 31

Node 3 Declared Dead • Node 3 is excluded and a new heartbeat circle is formed. STORAGE LAYER DATA NODE 3 DATA DATA DATA NODE 1 OK NODE 2 OK NODE 4 OKCopyright 2011 Severalnines AB Control your database infrastructure 32

Node 3 Rejoins Cluster • When Node 3 is restarted on of the first things to happens is that it rejoins the heartbeat circle. STORAGE LAYER DATA DATA DATA OK DATA OK NODE 3 OK NODE 4 NODE 1 NODE 2 OKCopyright 2011 Severalnines AB Control your database infrastructure 33

Network Partitioning • The Data Nodes are separated from each other and form two “Clusters”. – Also known as Split Brain. DATA DATA NODE 1 NODE 2 P0 P1 S1 S0Copyright 2011 Severalnines AB Control your database infrastructure 34

Network Partitioning • An Abitrator must be asked for advice. Management Server “Am I the chosen one?” “Am I the chosen one?” DATA DATA NODE 1 NODE 2 P0 P1 S1 S0Copyright 2011 Severalnines AB Control your database infrastructure 35

Arbitrator • By default the Management Servers are arbitrators. – ArbitrationRank=[0 | 1 | 2 ] regulates this • 0 – disabled • 1 and 2 are priority levels – ArbitrationTimeout=1000ms • Data nodes wait 1000ms for reply from arbitrator. If exceeded, data node will shutdown. – Arbitration = [ Default | Disabled | WaitExternal] • WaitExternal – your own application can do Arbitration. • The data nodes agrees on which arbitrator to use at Startup – This decision may never change unless the Arbitrator dies – The data nodes will in that case re-elect another arbitrator.Copyright 2011 Severalnines AB Control your database infrastructure 36

Network Partitioning • The elected Arbitrator is asked for advice. Management Management Server 1 Server 2 “Am I the chosen one?” “Am I the chosen one?” DATA DATA NODE 1 NODE 2 P0 P1 S1 S0Copyright 2011 Severalnines AB Control your database infrastructure 37

Network Partitioning • Only one set of nodes will get the positive answer • The nodes getting “NO” will shutdown. Management Management Server 1 Server 2 Yes NO!!! DATA DATA NODE 1 NODE 2 P0 P1 S1 S0Copyright 2011 Severalnines AB Control your database infrastructure 38

Network Partitioning • Nodes that cannot reach the Arbitrator will shutdown Management Management Server 1 Server 2 “Am I the chosen one?” “Am I the chosen one?” DATA DATA NODE 1 NODE 2 P0 P1 S1 S0Copyright 2011 Severalnines AB Control your database infrastructure 39

Arbitration • Arbitration is only used if there are equal splits. • At least three computers must be present to handle network partitioning • The data nodes will handle majority groups by themselves. – 2 nodes on one side and 4 nodes on the other. – The largest side has the majority.Copyright 2011 Severalnines AB Control your database infrastructure 40

Two computer setup • Requirements: – Use redundant switches – Use bonding • Eth0 to switch 0 • Eth1 to switch 1 – Set Arbitration=Disabled • This is a pretty solid setup – chances that two switches crashes at the same time is small • In that case you likely have a bigger problem in your data center.Copyright 2011 Severalnines AB Control your database infrastructure 41

We hope these training slides are useful to you! Please visit our website to view the next section of this training. For any questions, comments, feedback or to book a training class, please contact us at: services@severalnines.com Thank you!Copyright 2011 Severalnines AB Control your database infrastructure 43

Disclaimer © Copyright 2011 Severalnines AB. All rights reserved. Severalnines& the Severalnineslogo(s) are trademarks of Severalnines AB. MySQL is a registered trademark of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.Copyright 2011 Severalnines AB Control your database infrastructure 44

http://www.severalnines.com	207
http://severalnines.com	144

Severalnines Self-Training: MySQL® Cluster - Part IV

by Severalnines AB on Dec 21, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

2 Embeds 351

Statistics