Severalnines Self-Training: MySQL® Cluster - Part I

MySQL Cluster Trainingpresented by severalnines.com
Address:
c/o Severalnines AB
SICS, Box 1263
Isafjordsgatan22
SE-164-29 Kista
Contact:
Jean-Jérôme Schmidt
Email: services@severalnines.com

Introduction
At Severalnines, we believe in sharing information and knowledge; we all come from an open source background
We know a lot of things about MySQL Cluster and think that MySQL Cluster is a great technology
These free MySQL Cluster Training slides are a contribution of ours to the knowledge and information sharing that’s common practice in the open source community
If you have any questions on these slides or would like to book an actual training class, please contact us at: services@severalnines.com

Training Slides - Concept
Over the coming weeks we will be chronologically releasing slides for the different sections of our MySQL Cluster Training program on our website.
The full agenda of the training with all of its modules is outlined in the next slides so that you can see what topics will be covered over the coming weeks.
Particularly specialised topics such as Cluster/J or NDB API are not fully covered in the slides. We recommend our instructor-led training classes for such topics.
Please contact us for more details: services@severalnines.com

Full Training Agenda (1/4)
MySQL Cluster Introduction
MySQL eco system
Scale up, scale out, and sharding
MySQL Cluster Architecture
Use cases
Features
Node types and Roles
Detailed Concepts
Data Distribution
Verifying data distribution
Access Methods
Partitioning
Node failures and failure detection
Network Partitioning
Transactions and Locking
Consistency Model
Redo logging and Checkpointing

Agenda (2/4)
Installing MySQL Cluster
Setting up MySQL Cluster
Starting/stopping nodes
Recovery and restarts
Upgrading configuration
Upgrading Cluster
Performance Tuning (instructor-led only; contact us at services@severalnines.com)
Differences compared to Innodb/MyISAM
Designing efficient and fast applications
Identifying bottlenecks
Tweaking configuration (OS and MySQL Cluster)
Query Tuning
Schema Design
Index Tuning

Agenda (3/4)
Management and Administration
Backup and Restore
Geographical Replication
Online and offline operations
Ndbinfo tables
Reporting
Single user mode
Scaling Cluster
Disk Data
Use cases
Limitations
Best practice configuration
Designing a Cluster
Capacity Planning and Dimensioning
Hardware recommendations
Best practice Configuration
Storage calculations

Agenda (4/4)
Resolving Issues
Common problems
Error logs and Tracefiles
Recovery and Escalation procedures
Connectivity Overview
NDBAPI
Cluster/J
LDAP
Severalnines Tools
Monitoring and Management
Benchmarking
Sandboxes
Configuration and capacity planning
Conclusion

Agenda: Lab Exercises(only applicable to instructor-led training classes)
Lab Exercises
Installing and Loading data into MySQL Cluster
Starting/stopping nodes, recovery
Query tuning
Backup and Restore
Configuration Upgrade
Would you like to try something particular?
This is possible too, speak with your instructor

Prerequisites
Readers / Participants have understanding of SQL and basic database concepts.
Laptops/PCs for hands-on exercises
Linux: 1GB RAM
Windows: 2GB RAM
Approx. 20GB disk space and Virtualbox installed.
Virtualbox can be downloaded for free at http://www.virtualbox.org/wiki/Downloads
MySQL Cluster version 7.1 or later

Week 1 – Training SlidesSection 1: MySQL Cluster Introduction

Topics
Origins of MySQL Cluster
Scale up, scale out, and sharding
Use cases
Features
Node types and Roles

Origins of MySQL Cluster
Ericsson started development of NDB Cluster in 1996
Focus on HA/performance requirements of Telecom
Offered as a product in 2001 by Alzato, fully owned by Ericsson
Acquisition by MySQL in September 2003
Core team is transferred
Intellectual Property transferred (9 patents)
New product called MySQL Cluster
Original NDB Cluster + SQL engine + MySQL connectors
Major investment in usability
Co-developed with some telecom vendors

Telco-grade Data Management
High Availability
99.999% (< 5 mins/yr) or 99.9999% (< 30 secs/yr)
Scalability
Throughput in the range of 1000s or 10,000s per second
Predictable Real-time Performance
Typically 5 to 10 milliseconds
In-memory performance
Cost effective
ScaleOut vs. ScaleUp
COTS hardware

Scale up, scale out, sharding
Scale up means having one beefy machine handling load.
At some point it won’t scale
Depending on implementation, mutexes and semaphores these systems scales up to about 16 processors/cores.
With the MySQL Server you will be limited by the number of incoming connections at some stage.
Hard to increase the capacity
Reconfigure with more cores
Examples
Sybase. Customer has a Sun E10000, next step is to get a Sun E15000, and then license cost for additional processors
Gets expensive.
Scale up is old school.

Scale-out means that instead of using one beefy machine you use many.
This can be accomplished with
MySQL Replication
Master -> slave replication
Pros: easy to add slaves
Cons: Write scaling is hard, one single master, read/write split
MySQL Cluster
Will be addressed in this training.
Read
Read/writes
Asynch. replication
Read

Shardingmeans that instead of using one beefy machine you use many and partitions the data on the machines.
Problems:
Hard do find a good key to partition by
Adding nodes are hard – how to redistribute the data?
MySQL Cluster addresses a lot of these problems
Sharding Aware Layer in Application Code
Users1-1000
Users1001-2000
Users2001-3000

Application R/I/U/D
Application R/I/U/D
Application R/I/U/D
Java, C++, LDAP
Application R/I/U/D
Native API
DATANODE
DATANODE
Management Server
Management Server
DATANODE
DATANODE

Parallel Database Server
Shared-nothing architecture
Data partitioned across nodes
In a RAID10 fashion
Redundancy
Synchronous replication
Automatic failover and repair
Performance
Ability to mix and match disk and memory tables
Different access methods (SQL, C++, Java,LDAP)
MySQL Cluster basics

No single point of failure
Any node can fail without affecting the system
Configurable number of data copies
Up to 4 copies
Synchronous replication
Automatic failure detection and failover
Detection time determined by heartbeat
Failover is instantaneous
Automatic resynchronization
Delta or Full synchronization
Availability Features (1/2)

Handling of network partitioning
Uses an Arbitrator
Online Software Upgrade
Supports upgrades and downgrade of system without outages
Disaster Recovery
Geographical Replication using Asynchronous protocol
Online Backup
Availability Features (2/2)

Use Cases
Good fit
OLTP apps with short running queries
Application with real-time characteristics and requirements
A lot of concurrent requests
Write intensive applications
Typically the following are a poor fit:
Heavy reporting type (OLAP)
Data Warehouse
Complex JOINs (much improved in 7.2) scale badly.
A couple of tables and with about 1000 records meeting the JOIN criteria is just fine)
Complex and long running transactions in a realtime system is just wrong.
However, replicate from MySQL Cluster to regular MySQL (innodb) which runs the reporting.

Application R/I/U/D
Application R/I/U/D
Application R/I/U/D
Java, C++, LDAP
Application R/I/U/D
Native API
DATANODE
DATANODE
Management Server
Management Server
DATANODE
DATANODE

Node Types and Roles
MySQL Cluster is comprised of the following node types:
Management Nodes
Maintains the cluster configuration
Involved in handling network partitioning
Writes a cluster log containing events occurring in Cluster.
Typically two are deployed for redundancy
Data Nodes
Transaction, query handling and data storage
Atleast two for redundancy
Application Nodes
Mysql server, Cluster/J
Requests are ultimately converted to NDBAPI calls

Starting a MySQL Cluster
Pretend that we have four hosts:
app01
app02
storage01
storage02
We will start a HA cluster consisting of four nodes:
2 data nodes, one each on storage01 and storage02
2 mysql servers to access the data nodes, one on each of app01 and app02
2 management servers to deal with configuration and network partitioning, one of each on app01 and app02.

Management Nodes/Servers
Owns the Cluster configuration and loads a file called config.ini.
Config.ini describes the Cluster
Number of management nodes, data nodes access (mysql/ndbapi) nodes
Ports, Resources, Data Directories
The management node is started with
ndb_mgmd –f /etc/mysql/config.ini --configdir=/etc/mysql
The management server generates a binary versioned configuration file that is stored in configdir.
All nodes must first connect to the management nodes to fetch the configuration
Listens by default on port 1186
Writes a cluster log in Datadir (e.gDatadir=/data/)

MySQLClusterTwo management servers are started
The management servers loads in the config.ini file.
Compare with each other they have the same version. If not, they won’t start.
The management servers start writing a cluster log
Ndb_X_cluster.log
Located in the datadir.
Hostname=app01datadir=/data/Listens on *:1186
Management Server1
Management Server2
Compareconfig
app01> ndb_mgmd –f /etc/mysql/config.ini –configdir=/etc/mysql --reload
app02> ndb_mgmd –f /etc/mysql/config.ini –configdir=/etc/mysql --reload

Data Nodes
Connects to the management nodes and fetch the configuration
Needs a connectstring pointing to the management nodes
The data nodes are the processes
Storing data
Managing transaction, indexing, data distribution etc etc.
When the data node gets the configuration it will try and setup connections to other nodes.
Started with
ndbd –c “hostname1:1186,hostname2:1186”
ndbmtd –c “hostname1:1186,hostname2:1186”
Ndbmtd is the multithreaded daemon and can use up to 8 cores. Ndbd is for single/dual core machines.

MySQL ClusterData Node 1 Joins
Hostname=storage01datadir=/data/
Management Server1
Management Server2
DATANODE1
Get config
Starts listening for other data node
storage01> ndbd –c “app01:1186,app02:1186”

MySQL ClusterData Node 2 Joins
Management Server1
Management Server2
DATANODE1
Get config
DATANODE1
storage02> ndbd –c “app01:1186,app02:1186”

API Nodes
NDBAPI programs
Direct API (no SQL)
C++ programs, uses libndbclient
Cluster/J
Java, has a JNI interface that uses libndbclient
MySQL nodes/server
Convert SQL into NDBAPI code
Uses libndbclient
Stores data/logs in a data directory
Configuration for the server in a my.cnf file
In any case
A connectstring is a always needed pointing to the management nodes.

MySQL ClusterA MySQL Server Joins
Get config
Hostname=app03datadir=/data/
Management Server1
Management Server2
DATANODE1
DATANODE1
app03> mysqld &

MySQL ClusterA MySQL Server Joins
Hostname=app03datadir=/data/
Management Server1
Management Server2
DATANODE1
DATANODE1
app03> mysqld &

MySQL ClusterFullystarted
Hostname=app01datadir=/data/mysqlclusterListens on *:1186
Hostname=app02datadir=/data/mysqlclusterListens on *:1186
Hostname=storage01datadir=/data/mysqlcluster
Hostname=app01datadir=/data/mysql
Management Server1
Management Server2
DATANODE1
Hostname=storage01datadir=/data/mysqlcluster
Hostname=app02datadir=/data/mysql
DATANODE1

Fully Connected
All data nodes are connected to every other node
Epoll is used
Max 255 Nodes in the Cluster
Max 48 Data Nodes
Biggest Deployed Cluster
48 data nodes

Coming next in week 2:Section 2: Detailed Concepts

Topics covered in Week 2
Data Distribution
Verifying data distribution
Access Methods
Partitioning
Node failures and failure detection
Network Partitioning
Transactions and Locking
Consistency Model
Redo logging and Checkpointing

We hope these training slides are useful to you!Please visit our website to view the next section of this training.For any questions, comments, feedback or to book a training class, please contact us at: services@severalnines.comThank you!

Disclaimer
© Copyright 2011 Severalnines AB. All rights reserved.
Severalnines and the Severalnineslogo(s) are
trademarks of Severalnines AB.
MySQL is a registered trademark of Oracle and/or its
affiliates.
Other names may be trademarks of their
respective owners.

http://www.severalnines.com	2984
http://severalnines.com	1565
http://localhost	7
https://twitter.com	3
https://si0.twimg.com	2
http://twitter.com	2
http://translate.googleusercontent.com	1
http://www.linkedin.com	1

Severalnines Self-Training: MySQL® Cluster - Part I

by Severalnines AB

Accessibility

Categories

Upload Details

Usage Rights

8 Embeds 4,565

Statistics