The document provides an overview of MySQL Cluster, highlighting its design goals, features, and architecture aimed at achieving high availability, scalability, and performance for transactional and analytical workloads. It details the system's capabilities for synchronous replication, auto-sharding, and online operations, which can achieve 99.999% uptime. Additionally, it discusses the historical context and development of MySQL Cluster, emphasizing its suitability for high-volume applications across various industries.

1. 1 Copyright © 2011, Oracle and/or its affiliates. All rights
reserved.
MySQL Cluster overview
Frazer Clement - MySQL Cluster Engineering @ Oracle
September 2014
Node Group 2
F2
F4
Node3Node4
F2
F4
Node Group 1
F1
F3
Node3Node4
F1
F3
MySQL Cluster Data Nodes
MySQL Servers

2. 2 Copyright © 2014, Oracle and/or its affiliates. All rights
reserved.
MySQL Cluster overview
Two halves
●
Intro
●
What is MySQL Cluster?
●
MySQL Cluster development Node Group 2
F2
F4
Node3Node4
F2
F4
Node Group 1
F1
F3
Node3Node4
F1
F3
MySQL Cluster Data Nodes
MySQL Servers

3. 3 Copyright © 2014, Oracle and/or its affiliates. All rights
reserved.
Who are you?
Frazer Clement - Technical Lead MySQL Cluster @ Oracle
● Working on MySQL Cluster software at MySQL / Sun / Oracle for 7 years...
Client Apis, async replication, conflict detection, cluster membership
protocols, performance, observability and usability issues, a bit of everything.
● Prior to that, worked at Nortel Networks on 2G/3G HLR software for 8 years
(mobile phone profile, location, authentication database)
● Spare time HA / parallel / scalable / database / distributed systems geek
● Originally from Edinburgh, left for a while, then came back - working from
home/cafe/pop up Fringe venues in Edinburgh since 2009

4. 4 Copyright © 2014, Oracle and/or its affiliates. All rights
reserved.
Who are you? blog@ messagepassing.blogspot.com

5. 5 Copyright © 2014, Oracle and/or its affiliates. All rights
reserved.
What is MySQL Cluster?

6. 6 Copyright © 2014, Oracle and/or its affiliates. All rights
reserved.
MySQL Cluster design goals
• Auto-Sharding, Multi-Master
• ACID Compliant, OLTP + Real-Time Analytics
HIGH SCALE,
READS + WRITES
• Shared nothing, no Single Point of Failure
• Self Healing + On-Line Operations
99.999%
AVAILABILITY
• Open Source + Commercial Editions
• Commodity hardware + Management, Monitoring Tools
LOW TCO
• Key/Value + Complex, Relational Queries
• SQL + Memcached + Java + JPA + HTTP/REST + C++
SQL + NoSQL
• High Load, Real Time Performance
• Predictable Low-Latency, Bounded Access Time
REAL-TIME

7. 7 Copyright © 2014, Oracle and/or its affiliates. All rights
reserved.
MySQL Cluster architecture
MySQL Cluster Data Nodes
Data Layer
Clients
Application Layer
Management
Management

8. 8 Copyright © 2014, Oracle and/or its affiliates. All rights
reserved.
MySQL Cluster highlights
Distilled : HA, High performance, Relational, Transactional, Distributed, Parallel, SQL, NoSQL,
Scalable, Shared-Nothing, Commodity ...
SQL : Joins, Foreign Keys, Transactions, Row locks, Triggers, Views, Stored procedures, Blobs,
keyless tables, newSQL, MySQL compatible... connectors for most languages, ORMs etc...
NoSQL : Full C++ Api for best control and performance (MySQLD SE built on top), Other Apis : Java, JPA, Node.js,
Memcache....
HA : 99.999% uptime systems (five nines), No single point of failure (SPOF), Heartbeating, cluster
membership, automatic failover + recovery, automatic client failover, transactional DDL, CP, async replication,
advanced exception logging...
Performance and parallelism: High throughput, low bounded latency (71M read tx/s, 20M
write tx/s in 2012). Batching, optimised protocols, Intra and Inter query parallelism, pushed parallel filters, pushed
parallel joins, non-blocking event driven multithreaded...
Scalability: Scale-out nodegroups or stateless API clients online, Scale-up data nodes and clients online with
multithreading, scale up hardware online.

9. 9 Copyright © 2014, Oracle and/or its affiliates. All rights
reserved.
MySQL Cluster highlights
Distilled : HA, High performance, Relational, Transactional, Distributed, Parallel, SQL, NoSQL,
Shared-Nothing, Commodity ...
Replication : Synchronous two phase commit internally, Transactional HA
async replication between clusters, conflict detection+resolution...
Storage : Data transparently distributed and balanced by hash, Indexed columns in
memory, others on disk or memory, Secondary unique and ordered indexes, Redundant Redo
logs and periodic checkpoints...
Manageability : Online add + drop (index, column), Online consistent backup, Online
upgrade, Online OS or hardware upgrade, consolidated cluster logs, C management Api for tooling...
Shared nothing, Commodity : No need for shared storage, In-memory data uses disk
frugally, TCP over Ethernet / Infiniband etc, No special layer 2 requirements. Open source.

10. 10 Copyright © 2014, Oracle and/or its affiliates. All rights
reserved.
MySQL Cluster architecture
Data
Nodes
Node Group 1
F1
F4
F4
F1
Node Group 2
F2
F5
F5
F2
Application Nodes
Cluster
Mgmt
Cluster
Mgmt
RESTJPA
Node Group 3
F3
F6
F6
F3
F1
F2
F3
F4
F5
F6
Table 1

11. 11 Copyright © 2014, Oracle and/or its affiliates. All rights
reserved.
MySQL Cluster scaling and HA
Data
Nodes
Node Group 1
F1
F4
F4
Node Group 2
F2
F5
F5
Application Nodes
Cluster
Mgmt
Cluster
Mgmt
RESTJPA
Node Group 3
F3
F6
F6
Performance + HA
Online scale out of front end / Api nodes
F1 F2 F3

12. 12 Copyright © 2014, Oracle and/or its affiliates. All rights
reserved.
MySQL Cluster scaling and HA
Data
Nodes
Node Group 1
F1
F4
F4
Node Group 2
F2
F5
F5
Application Nodes
Cluster
Mgmt
Cluster
Mgmt
RESTJPA
Node Group 3
F3
F6
F6
Performance + HA
Online scale out of front end / Api nodes
F1 F2 F3
Performance + Capacity
Online scale out of back end by adding node groups
(Write scaling)

13. 13 Copyright © 2014, Oracle and/or its affiliates. All rights
reserved.
MySQL Cluster HA
Data
Nodes
Node Group 1
F1
F4
F4
Node Group 2
F2
F5
F5
Application Nodes
Cluster
Mgmt
Cluster
Mgmt
RESTJPA
Node Group 3
F3
F6
F6
Redundancy for availability – no SPOF
● Two (or more) management servers.
● Used for configuration, node startup/shutdown, triggering
backups, logging + 'split-brain' arbitration
● Not critical – not involved in transaction processing / querying
F1 F2 F3

14. 14 Copyright © 2014, Oracle and/or its affiliates. All rights
reserved.
MySQL Cluster HA
Data
Nodes
Node Group 1
F1
F4
F4
F1
Node Group 2
F2
F5
F5
F2
Application Nodes
Cluster
Mgmt
Cluster
Mgmt
RESTJPA
Node Group 3
F3
F6
F6
F3
Redundancy for availability – no SPOF
Two nodes per nodegroup, each storing the same data
Can survive data node failures so long as one node per
nodegroup is available.
Load balanced, Synchronous 2PC, heartbeating, automatic
failover, recovery etc
F1 F2 F3

15. 15 Copyright © 2014, Oracle and/or its affiliates. All rights
reserved.
MySQL Cluster HA
Data
Nodes
Node Group 1
F1
F4
F4
F1
Node Group 2
F2
F5
F5
F2
Application Nodes
Cluster
Mgmt
Cluster
Mgmt
RESTJPA
Node Group 3
F3
F6
F6
F3F1 F2 F3
Redundancy for availability
API nodes are stateless and consistent, can use n + m sparing
NdbApi automatically balances, fails over and back on data
node failures.
Network needs no SPOF too – no single failure takes out > 1
cluster member.

16. 16 Copyright © 2014, Oracle and/or its affiliates. All rights
reserved.
Who uses MySQL Cluster?
Extreme Scalability, Availability and Affordability
● Web
● High volume OLTP
● Ecommerce
● On-Line Gaming
● Digital Marketing
● User Profile Management
● Session Management & Caching
● Telecoms
● Subscriber Data Management
● Service Delivery Platforms
● VAS: VoIP, IPTV & VoD
● Mobile Content Delivery
● Mobile Payments
● ISP infrastructure

17. 17 Copyright © 2014, Oracle and/or its affiliates. All rights
reserved.
What is MySQL Cluster Not?
Many users are excited by the technology and so attempt to find some way
to use MySQL Cluster where something simpler will do...
●
Not a loosely coupled 'sharded' cluster of MySQL Servers (InnoDB)
See MySQL Fabric !
●
Not a drop-in replacement for InnoDB or MyISAM
Some functional and performance differences
●
Not 'infinitely scalable' (Suitable for 'lots of small data' – see MySQL@Facebook)
●
Not simple
●
Not in-memory only (Indexes must fit in aggregate memory)
But when nothing else will do, MySQL Cluster is exceptional...

18. 18 Copyright © 2014, Oracle and/or its affiliates. All rights
reserved.
MySQL Cluster development

19. 19 Copyright © 2014, Oracle and/or its affiliates. All rights
reserved.
Origins
MySQL Cluster is a combination of the familiar MySQL RDBMS Server
and the Ndb Cluster storage engine.
●
Ndb Cluster engine originally designed in Ericsson AB (Sweden) for telecoms
●
Team became startup called 'Alzato' in Ericsson tech incubator circa 1999
●
Team and IP sold to MySQL AB in 2003.
●
MySQL 4.1 released with Ndb storage engine in 2004
●
Since then : MySQL 5.0, MySQL 5.1, MySQL Cluster 6.2, 6.3, 7.0, 7.1, 7.2, 7.3,
7.4..., Sun, Oracle...
●
Mature, proven technology
https://www.flamingspork.com/blog/2013/05/13/the-mysql-cluster-storage-engine/

20. 20 Copyright © 2014, Oracle and/or its affiliates. All rights
reserved.
Distributed software and team
organizations which design systems ... are
constrained to produce designs which are copies of the
communication structures of these organizations
—M. Conway (1968)
Luckily we are building a distributed shared nothing database :)
●
Original Ericsson / Alzato Ndb team based in Stockholm
●
Now also have members from Sun/Clustra in Trondheim, Norway and in
Bangalore, Finland, US west coast, Edinburgh. Some in an office, others
at home.

21. 21 Copyright © 2014, Oracle and/or its affiliates. All rights
reserved.
Software engineering
●
Written in C/C++, 0.5M LOC unique / 1.7M LOC
●
Running on Linux, Solaris (x86+SPARC), Windows, Mac OSX++
●
Distributed source code management (Bitkeeper, bzr, …)
●
Agile development process
●
Continuous integration and test systems in Norway
●
Per-push automated testing plus daily distributed test suites
●
Oracle standard bug and customer support systems

22. 22 Copyright © 2014, Oracle and/or its affiliates. All rights
reserved.
MySQL Cluster software
● Different architectures in the MySQL Server code base and Ndb data node code base.
● MySQL Cluster data nodes use an internal async message passing architecture derived from
Ericsson AXE telecom switches. Non blocking, state machines, callbacks, fail fast, constrained
multithreading, local and distributed protocols etc.
● Incredibly efficient and with great runtime properties, but challenging to work with, understand,
debug and improve.
● MySQL Server has a more 'IT' style design – copious multithreading, locks, synchronously
blocking threads etc.
● NdbApi interfaces the two.
● Data nodes have around 20 components, 100 protocols.
● Team works right across the distributed stack – HLL to SQL to system call level and below

23. 23 Copyright © 2014, Oracle and/or its affiliates. All rights
reserved.
MySQL Server layers
Ndb storage engine implementation
Handler / Storage Engine Api
SQL parsing, optimisation, execution
Client threads
NdbApi
Network communication with data nodes
MySQL clients MySQL clients MySQL clients
Generic
MySQL
Server code
Generic NdbApi
code
Glue
SQL
Handler Api calls
NdbApi calls
MySQL
client protocol
'Protocol 6'
MySQL
server
Scale up by
reducing
contention
on
bottleneck
resources

24. 24 Copyright © 2014, Oracle and/or its affiliates. All rights
reserved.
MySQL Cluster data node components
Replication thread
Main thread
LDM threads
TC threads
Send threads
Request
processing
threads
TC and LDM threads
do most work, must
be well fed by Send +
Receive threads
Receive threads IO threads
Connect threads Watchdog
Data
node
Internal
shared
nothing* -
Scale up by
duplicating
components
and routing
signals

25. 25 Copyright © 2014, Oracle and/or its affiliates. All rights
reserved.
Distributed dataflow + HA
TC
TC
TC
TC
API
API
API
API
LDM / TUP
LDM / TUP
LDM / TUP
LDM / TUP
SUMA
SUMA
SUMA
SUMA
API
API
API
API
Subscriber 1
Subscriber 2
NdbApi NdbApi
events
A single row write transaction from an API node routes to one TC instance, from where it is
forwarded to the LDM instance managing the primary fragment replica for the fragment. Then it is
synchronously replicated to the nodegroup peer. Both nodes forward the event to their SUMA
instance, but only one SUMA forwards the event to the subscribing API nodes. The other buffers it.
Data nodes, NoOfReplicas=2

26. 26 Copyright © 2014, Oracle and/or its affiliates. All rights
reserved.
MySQL Cluster development
My day involves working
●
with the Cluster Engineering team to discuss and review designs,
protocols, code, tests, debug problems etc.
●
directly with our largest customers on field issues, performance, new
features, training.
●
with the MySQL Oracle support team to support all our customers.
Using
●
Emacs, gcc, phone, IM, and occasionally face-to-face meetings.
Working in a distributed team is hard, can be isolating, but there are benefits...

27. 27 Copyright © 2014, Oracle and/or its affiliates. All rights
reserved.
http://www.mysql.com/products/cluster/
http://dev.mysql.com

28. 28 Copyright © 2014, Oracle and/or its affiliates. All rights
reserved.
Attribution
http://paradiseintheworld.com/edinburgh-scotland/
http://en.wikipedia.org/wiki/Conway%27s_law