The document discusses MySQL NDB 8.0 and high availability solutions for MySQL. It summarizes MySQL NDB Cluster, MySQL InnoDB Cluster, and MySQL Replication as high availability solutions. It also discusses features and performance of MySQL NDB Cluster 8.0, including linear scalability, predictable low-latency performance, and improved backup throughput.

1. Ted Wennmark & Stuart Davey
MySQL Solution Engineering EMEA
MySQL Business Unit
February 24, 2021
MySQL News
MySQL NDB 8.0

2. Safe harbor statement
The following is intended to outline our general product direction. It is intended for information
purposes only, and may not be incorporated into any contract. It is not a commitment to deliver
any material, code, or functionality, and should not be relied upon in making purchasing
decisions.
The development, release, timing, and pricing of any features or functionality described for
Oracle’s products may change and remains at the sole discretion of Oracle Corporation.
2

3. • Long time MySQL user
– Started developing apps using MySQL over 20 years back
– Worked as MySQL DBA, Trainer and consultant the past
• MySQL Prinicipal Solution Architect at Oracle
• Work with normal MySQL but have focuse on distributed
databases with NDB Cluster.
• My workshops at Github: https://github.com/wwwted/
• Let’s stay in touch:
– https://www.linkedin.com/in/tedwennmark/
• Join us on slack: https://lefred.be/mysql-community-on-slack/
Ted Wennmark
3

4. MySQL High Availability Solutions
• MySQL NDB Cluster
– NDB storage engine.
– Memory database.
– Automatic sharding of data.
– SQL Access via MySQL with
cross shard join support.
– Native access via several
API’s.
– Read/write consistency.
– Read/write scalability (2pc).
– ACID and transactions.
• MySQL InnoDB Cluster
– Easy HA built into MySQL
5.7+ for InnoDB.
– MySQL Group Replication,
Shell and Router.
– Write consistency.
– Read Scalability.
– Native CRUD API in
MySQL 8.
– Synchrounous (Paxos).
• MySQL Replication
– Core part of MySQL, used
by almost everyone.
– Can be used by any
storage engine.
– Asynchronous and semi-
sync option.
– Scale out reads.
– ReplicaSet from 8.0.19 with
integration to Shell and
Router.
4

5. MySQL High Availability Solutions
• MySQL NDB Cluster
– NDB storage engine.
– Memory database.
– Automatic sharding of data.
– SQL Access via MySQL with
cross shard join support.
– Native access via several
API’s.
– Read/write consistency.
– Read/write scalability (2pc).
– ACID and transactions.
• MySQL InnoDB Cluster
– Easy HA built into MySQL
5.7+ for InnoDB.
– MySQL Group Replication,
Shell and Router.
– Write consistency.
– Read Scalability.
– Native CRUD API in
MySQL 8.
– Synchrounous (Paxos).
• MySQL Replication
– Core part of MySQL, used
by almost everyone.
– Can be used by any
storage engine.
– Asynchronous and semi-
sync option.
– Scale out reads.
– ReplicaSet from 8.0.19 with
integration to Shell and
Router.
5

6. Massively linear scale
Always-On 99.9999% Availability
Distributed In-Memory Datasets
Always Consistent
Parallel Real-Time Performance.
Auto-partitioning, data distribution
and replication built-in.
Read- and Write Scale-Out
to many TB on commodity hardware.
Designed for mission critical
systems. Masterless, shared-nothing
with no single point of failure.
Transactional consistency across
distributed and partitioned dataset.
Out of the box straightforward
application programming.
Ease of use
Open Source
Written in C++. Can be used standalone
or with MySQL as a SQL front-end.
6

7. Requirements on NDB Cluster
• Unavailable less than 30 seconds per year (Class 6)
• Predictable latency (transaction with 20 operations within 10 milliseconds, mixed
read/write)
• Transparent Distribution and Replication
• Write and Read Scalability
• Support SQL, LDAP, File System interface, …
• Mixed OLTP and OLAP for real-time data analysis
• Follow HW development on CPUs, Network, Disks and Memory
7

8. Class 6 Availability
• Handle many concurrent and sequential failures
• Automatic restart at failure
• Synchronize with live nodes
• Online schema changes
• Global Replication
• Online Add Node
8

9. Predictable Latency - Real-time
• Defaults to In-Memory storage
• Asynchronous File Operations
• Complex operations divided into multiple executions
• Memory lockable to avoid swapping
• Real-time mode supported
• CPU spinning modes supported
9

10. When TO consider MySQL Cluster
• You need High Availability 6-9’s (and strong consistency).
• You need Sharding, either due to size or write
performance.
• You need Linear Scalabillity when adding more nodes.
• You need predictable Real-ime response times.
• SQL and cross shard join support.
• You want a ACID distributed in-memory database.
10

11. Linear scale with MySQL Cluster
11

12. MySQL Cluster NoSQL Performance
200 Million NoSQL Reads/Second
• Memory optimized tables
− Durable
− Mix with disk-based tables
• Parallel table scans for non-indexed
searches
• MySQL Cluster FlexAsych
− 200M NoSQL Reads/Second
12

13. MySQL Cluster SQL Performance
2.5M SQL Statements/Second
• Memory optimized tables
− Durable
− Mix with disk-based tables
• Massively concurrent OLTP
• Distributed Joins for analytics
• Parallel table scans for non-indexed
searches
• MySQL Cluster DBT2 BM
− 2.5M SQL Statements/Second
13

14. YCSB Benchmark
YCSB : Yahoo Cloud Serving Benchmark
• YCSB – Yahoo Cloud Service Benchmark
− De-facto cloud benchmark
− Benchmark can not be changed
• NDB is #1 player in this realm
− NDB Cluster is the Fastest Distributed, In-memory, Transactional Database in
the world!
14

15. YCSB Benchmark – Scaling NBD
YCSB : Yahoo Cloud Serving Benchmark
YCSB 0.15.0 with JDBC / SQL
• 1kB records
• Uniform distribution
2, 4 and 8 data nodes
• Replication factor 2
• ACID (read committed)
8 DenseIO across 2 AD
• adding 400us network latency
Best throughput and latency on market
1M
2M
3M
4M
2 4 8
(2 ADs)
1.4M
2.8M
3.7M
Transactions
per
second
Nodes
15

16. YCSB Benchmark – NDB Real-Time
YCSB : Yahoo Cloud Serving Benchmark
4 data nodes with 300M and
600M rows using JDBC
99% SQL reads < 1ms
• 95% < 0.9ms
99% SQL writes < 2ms
• 95% < 1.7ms
1M
Transaction
per
second
2 ms
Same Throughput & Latency
300M rows 600M rows
1.25M
TPS
1.25M
TPS
Reads
Reads
Writes
Writes
1 ms
16

17. Product Nodes TPS/OPS
32 227k
2 275k
3 715k
6 1.6M
8 1.6M
4 2.8M
YCSB Benchmark – NDB Results
YCSB : Yahoo Cloud Serving Benchmark
• Developed at Yahoo for Cloud Scale
workloads
• Widely used to compare scale-out
databases, NoSQL databases, and (non-
durable) in-memory data grids
• A series of NoSQL workload types are
defined:
• Workload A: 50% reads, 50% Updates
• The YCSB Client cannot be changed
• DB Vendors implement the DB Client
interface in Java
17

18. Architecture
18

19. MySQL Cluster Architecture
• Multiple data nodes form a cluster
• Shared nothing architecture
• Data is automatically distributed to data
nodes
19

20. MySQL Cluster Architecture - Replicas
Copy of 1
Copy of 1
• Multiple copies of data are maintained for
availability
• A group of data nodes shares the same
data
• 1 - 4 replicas/copies of data can be
configured
20

21. User-id (PK) Service Data
1773467253 chat xxx
6257346892 chat xxx
1773467253 photos xxx
7234782739 photos xxx
8235602099 reminders xxx
8437829249 location xxx
MySQL Cluster Data Nodes
Partition Key
Data distribution
• Auto-partitioning and distribution
− No name-node or central master
• Each dataset is split into fragments and
distributed across data nodes.
• Within a cluster data is always consistent.
21

22. Data distribution
Cluster uses thousands of virtual partitions
22

23. Data distribution
These virtual partitions are distributed to data nodes
23

24. Data distribution
These virtual partitions are distributed to data nodes
24

25. Data distribution
These virtual partitions are distributed to data nodes
25

26. Data distribution
These virtual partitions are distributed to data nodes
26

27. On-line Scaling and Elasticity - Repartitioning
Virtual partitions re-distributed on-line when adding more data nodes
Designed to be a slow background process not impacting real-time performance.
27

28. On-line Scaling and Elasticity - Repartitioning
Minimal amount of data moved
No re-hashing necessary
Similar to consistent hashing
28

29. Fully replicated
• Datasets can be marked to be copied to all nodes for best possible read performance
• All copies can be configured to be read from
• Local copy can be preferred when reading
29

30. Writing data “ACID”
Data
Memory
Flush writes to disk in
background checkpoints
Commit Log (REDO)
writes
…
time
…
• Data in MySQL Cluster is written to
memory and disk in a way that allows
real-time access and recovery
• Memory is locked so it won’t swap
• Writes go to data memory and commit
log
• Background process checkpoints data
memory for recovery
• Reads always happen from memory - not
from disk
30

31. Reading Data
• Cluster always knows where its data is - without a
name node
• Key-value with hash on primary key
• Complemented by ordered in-memory-optimized T-
Tree indexes for fast searches
31

32. Cross partition joins
• Cluster queries distributed data as
if it was a single consolidated
database
• Joins are pushed down to data
nodes
• Parallel cross-shard execution in
the data nodes
• Result consolidation in MySQL
Server
32

33. NDB 8.0 News
33

34. MySQL Cluster Development: 7.5 -> 8.0 (GA)
- MySQL Server 5.7 (5.6)
- 5x faster restarts
- JSON Support
- 50% faster reads
- 40% faster read/write
Cluster 7.4/7.5
Cluster 7.6
- MySQL Server 5.7
- Designed for Terabyte clusters
- Designed for modern hardware
- Native csv import
- Parallel backup
- Faster restart and recovery
Cluster 8.0 (GA)
- MySQL 8.0
- More Data nodes (144)
- 1-4 replicas
- Dynamic memory mgmt
- Larger rows (32k)
- Faster SQL
- Faster Disk Data
34

35. Cluster 8.0 Multi-threaded backup
Each data node doing own backup - one data
manager handling all writing
Now each data manager handling own
writing
Data Node 1 Data Node 2 Data Node 1 Data Node 2
35

36. Cluster 8.0 Multi-threaded backup
Now using all Local Data-Managers (LDM):
− Better system balance, avoids local data-manager overload
− Local Data-Manager with local backup processing - more efficient
− Faster backups (but backup performance limited more by configurable checkpoint speed,
so this is not necessarily true)
N independent file sets, can restore in parallel
36

37. Cluster 8.0 Transactional Data Dictionary
• Goal: Atomic and Transactional DDL for MySQL Server.
• Centralized data dictionary schema that uniformly stores dictionary data.
• Serialized dictionary format (SDI).
• MySQL Server local dictionary information is synchronized via NDB in a synchronous
fashion.
37

38. Cluster 8.0 Synchronized privileges I
• Privilege information moved from MyISAM to transactional InnoDB (but not part of Data
Dictionary).
• Changes in code due to data dictionary mandated removal of previous distributed privileges
implementation.
• Privileges are now synchronized via NDB and schema distribution.
38

39. Cluster 8.0 Synchronized privileges II
mysql>
GRANT
NDB_STORED_USER
ON
db.t1
TO
`bo-l-2`@`localhost`
NDBAPI
Local privilege
tables (InnoDB)
ACL
ndb_sql_metadata table
ACL Replication
Events
39

40. Cluster 8.0 Dynamic resource allocation
config.ini today
config.ini
MaxNoOfConcurrentTransactions=70000
MaxNoOfConcurrentOperations=359500
# Don't touch the following parameter
unless you really know what you're
doing.
MaxNoOfConcurrentScans=200
MaxNoOfLocalScans=9000
40

41. Cluster 8.0 Dynamic resource allocation
• Transactional memory dynamically
allocated from pool
• No more MaxNoOfTransactions,
MaxNoOfOperations, MaxNoOf…
• Still possible to use old static allocation
for highest level of performance
• More resources types to follow
config.ini
MaxNoOfConcurrentTransactions=90000
MaxNoOfConcurrentOperations=259200
# Don't touch the following parameter
unless you really know what you're
doing.
MaxNoOfConcurrentScans=300
MaxNoOfLocalScans=10000
41

42. Cluster 8.0 Dynamic resource allocation
• Transaction resources have a
reserved minimum and can
allocate up to a maximum
amount.
• Transaction resources will first
allocate from fixed size
Transaction Memory pool.
• If Transaction Memory pool is
exhausted then allocation will
happen from Shared Global
Memory - up to maximum per
resource.
Shared Global
Memory
Reserved
per resource
Global
maximum
Transaction
Memory
Concurrent
Operations
Concurrent
Transactions
Concurrent
Scans
Exclusive for
all transaction
resources
42

43. Cluster 8.0 Dynamic resource allocation
High-level gains:
− Less configuration complexity
− Fewer operational issues due to 'resource X exhausted’
− Potential savings on memory due to over-configured resources.
− Avoids hand-crafted config.ini for every new setup variant
43

44. Node 2
Node 1
Cluster 8.0 Reading from backup fragment is default
• Entirely local read transactions for co-
located API-nodes! Improved latency.
• Reading from backup allows to read any
node containing a copy of data
• Previously reads were directed towards
the node containing the primary fragment
only
44

45. Cluster 8 TPC-H Cluster 8.0 versus 7.5
Across all SQL NDB 8.0 is equally fast or faster than 7.5
NDB
8.0
x
times
faster
0,00
17,50
35,00
52,50
70,00
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 *) Q9 Q10 *) Q11 Q12 Q13 Q14 Q15 Q16 Q17 Q18 *) Q19 Q20 Q21 Q22
Improvements made in 7.6 vs 7.5 Improvements made in 8.0 vs 7.6
45

46. Cluster 8.0 More New features
• Larger row sizes (14k -> 30k)
• Support for 4 replicas (max 2 today)
• Larger cluster:
− Up to 144 data nodes
46

47. Cluster 8.0 Road-map (continous delivery model)
• Security
− Encrypted backup (started, 8.0.21)
− Encrypted data at rest (ongoing)
− Key management
− TLS links
• Other
− SQL improvements (ongoing)
− IPv6
− Blob batching
47

48. Deployment
StatefulSet
NDB Cluster
Ndb operator
backup /
restore
controller
mysql
controll
NDB
controller
Data Node
Data Node
Data Node
Data Node
Mgm Node
StatefulSet
Mgm Node
CRD
Cluster 8.0 Road-map: Kubernetes operator for NDB

49. Use Cases
49

50. MySQL Cluster Use Cases
• Key-Value store
+ High Availability, Scale-Out, Durability
• Transactional object store
+ Multi row transactions and consistency
• Relational database with MySQL Server front-ends
+ SQL joins, foreign keys, triggers, stored procedures, generated columns, JSON
50

51. SQL, JDBC, ADO, ...
LB or Connector/J
arbitrator
MySQL Cluster: SQL - Read Optimized HA setup
• Optimized 2 server HA setup, 3rd node
with management node is needed for
arbitration only.
• With ReadBackup all data is read/joined
locally.
• MySQL API nodes use shared memory
transporter (UseShm).
• MySQL nodes know who is local data
node using configuration parameter
ndb_data_node_neighbor.
51

52. application
Data Nodes
NDB Native C++ API
MySQL Cluster: Key-Value store
• High volume OLTP system.
• Linear scalability.
• Real-time response times.
• +50TB systems
• Use cases:
✓ IoT
✓ Financial data
✓ Telco core network data
52

53. Application(s)
NDB Native C++ API
App
SQL
MySQL Cluster: Hybrid “New SQL”
• High volume OLTP system.
• Linear scalability.
• Real-time response times.
• +50TB systems
• Access via SQL:
✓ Analytics
✓ BI
✓ Fraud
Data Nodes
53

54. Application(s)
Load Balancers
• High volume OLTP system.
• +50TB systems
• All access via SQL!
• Use cases:
✓ IoT
✓ On-line gaming
✓ Trading or other financial data
✓ Scalable SQL database
MySQL Cluster: New SQL
Data Nodes
54

55. Node Group 2
Node Group 1
Application(s)
Data Nodes
MySQL Cluster: DR – One Stretched Cluster I
Load Balancers
Application(s)
Load Balancers
DC 1 DC 2
55

56. MySQL Cluster: DR – One Stretched Cluster II
• This architecture will impact your response times if the latency is high
between the two data center.
− With the use of multithreaded applications and batching NDB can still deliver
good throughput.
− Timeouts for heartbeat can be increased if needed.
• Make sure you configure cluster so node groups are spanning both DC
as seen in picture above.
• This architecture is best used if you have a predictable low latency
network.
• Supports running application active on both sites (DC’s)
56

57. Application(s)
Data Nodes
MySQL Cluster: DR – Asynchronous Replication I
Load Balancers
Application(s)
Load Balancers
DC 1 DC 2
Data Nodes
57

58. MySQL Cluster: DR – Asynchronous Replication II
• This architecture is our standard (Active/Passive) DR solution.
• Two independent NDB Custer on each site, no impact on response
times due to latency between the two sites (DS’s) since replication is
asynchronous.
• Asynchronous replication is not native to NDB so dedicated MySQL
nodes are need to take care of replication cross sites.
• Manual work is needed to manage the replication channel.
• Supports running application active/passive setup between sites (DC’s)
− Active/Active can be achieved using replication in both directions
and conflict resolution.
58

59. Who’s Using MySQL Cluster?
59

60. MySQL Replication vs InnoDB Cluster vs NDB Cluster
MySQL Replication MySQL InnoDB Cluster MySQL NDB Cluster
Storage Engine All InnoDB NDBCLUSTER
Distributed Architecture Shared Shared nothing Shared nothing
Clustering Mode Master + slaves Multi-master (possible) Multi-master (default)
Replication mode Asynchronous Paxos (Synchronous) 2PC (Synchronous)
Consistency Model Weak Consistency Medium Consistency Strong Consistency
Sharding No No Yes
Arbitration No Yes Yes
Load Balancing No Reads via MySQL Router Yes
NoSQL APIs MySQL CRUD API MySQL CRUD API Native NDB API
Operational Complexity Easy Medium High
Administration Standard (MySQL) Standard (MySQL) Custom (MySQL + NDB)
60

61. Thank you
Ted Wennmark
MySQL Solution Engineering EMEA
MySQL Business Unit
61