Breakthrough performance with MySQL Cluster (2012)

Insert Picture Here
Breakthrough performance
with MySQL Cluster
Frazer Clement
Senior Software Engineer, Oracle
frazer.clement@oracle.com
messagepassing.blogspot.com

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.3
Session Agenda
•
Introduction to MySQL Cluster performance
•
Understanding MySQL Cluster
•
Making Benchmarks
•
Monitoring and Measuring MySQL Cluster
•
Boosting performance

Disclaimer
THE FOLLOWING IS INTENDED TO OUTLINE OUR GENERAL
PRODUCT DIRECTION. IT IS INTENDED FOR INFORMATION
PURPOSES ONLY, AND MAY NOT BE INCORPORATED INTO ANY
CONTRACT. IT IS NOT A COMMITMENT TO DELIVER ANY
MATERIAL, CODE, OR FUNCTIONALITY, AND SHOULD NOT BE
RELIED UPON IN MAKING PURCHASING DECISION. THE
DEVELOPMENT, RELEASE, AND TIMING OF ANY FEATURES OR
FUNCTIONALITY DESCRIBED FOR ORACLE'S PRODUCTS
REMAINS AT THE SOLE DISCRETION OF ORACLE

Insert Picture Here
Introduction to MySQL
Cluster performance

MySQL Cluster – Users & Applications
Extreme Scalability, Availability and Affordability
•
Web
•
High volume OLTP
•
eCommerce
•
On-Line Gaming
•
Digital Marketing
•
User Profile Management
•
Session Management & Caching
•
Telecoms
•
Service Delivery Platforms
•
VAS: VoIP, IPTV & VoD
•
Mobile Content Delivery
•
Mobile Payments
•

MySQL Cluster Architecture
Data
Nodes
Node Group 1
F1
F3
F3
F1
Node Group 2
F2
F4
F4
F2
Application Nodes
Cluster
Mgmt
Cluster
Mgmt
RESTJPA

4.3

4.3
/ 1 Billion Updates per Minute

MySQL Cluster Performance Gains
4.3 Billion + reads per minute, 30 node Intel Xeon cluster
NoSQL C++ API, flexAsynch benchmark
Huge increase in peak performance 7.1 -> 7.2
7.1
7.2
0 10 20 30 40 50 60 70 80
Million queries per second

1.2 Billion updates per minute, 30 node Intel E5-2600 based cluster
All updates transactional and synchronously replicated using 2PC
NoSQL, C++ API flexAsynch benchmark
Improvements due to enhanced per-node multicore scalability, inter-node
communication improvements, faster CPUs.
MySQL Cluster Performance Gains
Huge increase in peak performance 7.1 -> 7.2
MySQL Cluster continues to scale on new hardware
mikaelronstrom.blogspot.com

Gains on More Complex Transactions
Billions per minute benchmarks are on very simple transactions.
7.2 aims to better support more complex transactions using Adaptive
Query Localization (AQL) aka SPJ.
AQL has the data nodes execute some table joins, reducing round trips,
data transfer and increasing parallelism.
Aimed at small to medium sized joins.
Combined with new Index Statistics feature in 7.2, gives great joins

Data
Node1
Data
Node2
MySQL
Server
Web-based Content Management System

MySQL Cluster 7.1 MySQL Cluster 7.2
0
10
20
30
40
50
60
70
80
90
100
Query Execution Time Seconds
87.23 seconds
1.26 seconds
Index statistics + AQL give huge join performance gains 7.1->7.2

Performance and Optimisation
Rules of optimisation (http://c2.com/cgi/wiki?RulesOfOptimization)
i. Don't
ii. Don't yet (experts only)
iii. Profile before optimising
If you somehow reach iii) :
●
Know your goals, so you know when to stop.
●
Understand the system being optimised
●
Ignore DBMS vendors benchmarks! Build your own for your
application / queries / use cases.
●
Benchmarks should be easy to run, giving repeatable results
●
Use DBMS, OS and third party tools to profile and understand

Insert Picture Here
Understanding MySQL
Cluster

Understand Latency in MySQL Cluster
MySQL client MySQL server Data node 1 Data node 2
Angle
Dependent on
Network
technology
Batching,
batchsizes
Pushed joins
Distribution
awareness.
Batching
Partition
pruning
Standard
MySQL client
roundtrip
optimisation
(Stored
procedures,
joins)
Time

Round trips more expensive the higher up the stack they are.
Round trips are forced by data dependencies between operations – also
sometimes by the inability of lower stack layers to understand them.
Some dependencies can only be handled by the client/application.
Some can be handled by the MySQL Server (e.g. joins).
Some can be handled by the MySQL Cluster data nodes (e.g. as pushed
equi-joins in 7.2).
Handling data dependencies lower down the stack improves
performance, but not always possible.

The MySQL Cluster data nodes have a huge request processing
capacity - the difficulty is in keeping them busy.
With synchronous clients, and requests which result in many roundtrips,
giving a high latency to work ratio, very many client threads are required
to drive a high load on the data nodes.
Reducing roundtrips makes it easier to load the cluster, avoiding
problems with heavy client thread concurrency.
Reducing roundtrips improves individual query response times
4.3 Bn reads/min benchmark uses asynchronous clients

1 1 2
INSERT INTO t1 VALUES (12, …);
…

1 1 2
INSERT INTO t1 VALUES
(12, …), (13, …), (14, …),
(15, …), (16, …), (17, …);
…

1 3 1
Unpushed
SELECT … FROM t1 JOIN t2
JOIN t3 ON t2.key =
t1.attr AND t3.key =
t2.attr AND t1.key =
<const>;

Understand Data Distribution in MySQL Cluster
●
Rows are spread around data nodes based on an MD5 hash of
(some/all) of their primary key columns. This is pseudo-random.
●
Random gives good balance, but defeats locality of access benefits.
●
Find (or add) some localising column in your primary key, partition by
this, and related data will be stored.
●
Benefits :
- Reduced data-node round trips for reads (Latency + efficiency)
- Fewer 2PC participants for writes (Latency + efficiency)
- Scans bounded by the localising column are 'pruned'
(Efficiency + scalability)
Ordered index scans must be pruned to scale with the number of data nodes

NdbApi client Data node 1 Data node 2 Data node 3 Data node 3
Non pruned scan, all nodes involved, no scaling with # nodes
SELECT … FROM t1 WHERE
<expression using ordered
index without EQ on all
t1 partition key columns>

Pruned scan, 2 nodes involved, scales with # nodes
part_key1=A AND
part_key2=B AND
ordered_index_key3
=><=Whatever;

Hinted pruned scan, 1 node involved, scales with # nodes
Ordered index scans must be pruned to scale with the number of data nodes
part_key1=A AND
part_key2=B AND
ordered_index_key3
=><=Whatever;

●
MySQL Cluster provides an abstraction of a single database spread
across a cluster of machines. You don't need to care about how data is
distributed to execute any query.
●
But to go fast you must think about data distribution.
●
One easy way is to consider how data would be distributed in an
equivalent sharded database system – to minimise or remove cross-
shard queries, joins, transactions etc. (e.g. shard by user id, by function,
by geography, by hierarchy)
●
Unlike a sharded database system, MySQL Cluster supports queries
which require cross-shard queries or transactions, but if your 'high
runner' queries are effectively 'sharded', they will benefit.

Understand Parallelism in MySQL Cluster
Batch of primary key reads, executing in parallel
Batching gives intra-transaction parallelism
Parallelism is good for large queries/transactions

Performance Goals with MySQL Cluster
Good
Primary key operations
Unique key operations
Batching
Pruned index scans
Distribution awareness
Pushed conditions
Pushed joins
Parallelism (for large
queries/operations)
Bad
Round trips
Non pruned scans
'High running' Non distribution
aware queries
Ugly
BLOB/TEXT columns
Generally require ~1 extra Api
round trip per row

Insert Picture Here
Making Benchmarks

Making a Benchmark
●
Ideally use your application in a known, repeatable, relevant
'traffic' scenario.
●
Use existing or add instrumentation to measure how it performs
against your goals.
●
If goals are not being met, look at how to break out parts of the
functionality so that they can be benchmarked and analysed
individually – ideally still via your application.
●
Benchmark, analyse, improve
●
Benchmark with application to check end-to-end gains.
●
Making microbenchmarks? Reuse and extend existing tools –
mysqlslap, flexAsynch, memslap, etc.

Making a Simple SQL Join Benchmark
create.sql:
CREATE TABLE sub_name (sub_id INT NOT NULL PRIMARY KEY, name
VARCHAR(30)) engine=ndb;
CREATE TABLE sub_age (sub_id INT NOT NULL PRIMARY KEY, age INT)
engine=ndb;
INSERT INTO sub_name VALUES (1,'Bill'),(2,'Fred'),(3,'Bill'),
(4,'Jane'),(5,'Andrew'),(6,'Anne'),(7,'Juliette'),(8,'Awen'),
(9,'Leo'),(10,'Bill');
INSERT INTO sub_age VALUES (1,40),(2,23),(3,33),(4,19),(5,21),
(6,50),(7,31),(8,65),(9,18),(10,101);
query.sql:
SELECT sub_age.age FROM sub_name, sub_age WHERE
sub_name.name='Bill' AND sub_name.sub_id=sub_age.sub_id;

Making a Simple SQL Join Benchmark
shell> mysqlslap --concurrency=5 --iterations=100 --query=query.sql
--create=create.sql
Benchmark
Average number of seconds to run all queries: 0.132
seconds
Minimum number of seconds to run all queries: 0.037
seconds
Maximum number of seconds to run all queries: 0.268
seconds
Number of clients running queries: 5
Average number of queries per client: 1

Insert Picture Here
Monitoring and Measuring
MySQL Cluster

Understand the MySQL Cluster Server stack
Ndb storage engine implementation
Handler / Storage Engine Api
SQL parsing, optimisation, execution
Client threads
NdbApi
Network communication with data nodes
MySQL clients MySQL clients MySQL clients
Generic
MySQL
Server code
Generic NdbApi
code
Glue
SQL
Handler Api calls
NdbApi calls
MySQL client
protocol
'Protocol 6'

MySQL Server Explain Plan
mysql> EXPLAIN <query>;
mysql> EXPLAIN PARTITIONS <query>;
mysql> EXPLAIN EXTENDED <query>;
mysql> SHOW WARNINGS;
Interpretation of output here is mostly the same as for generic MySQL

MySQL Server Stack Profiles
GLOBAL/LOCAL STATUS variables Session
SHOW LOCAL STATUS LIKE 'Com_%'; SQL commands
SHOW LOCAL STATUS LIKE 'Handler%'; Handler Api calls
SHOW STATUS LIKE 'Ndb_api%session'; Ndb Api calls
Server
SHOW GLOBAL STATUS LIKE 'Com_%'; SQL commands
SHOW GLOBAL STATUS LIKE 'Handler%'; Handler Api calls
SHOW STATUS LIKE 'Ndb_api%count'; Ndb Api calls
SHOW STATUS LIKE 'Ndb'; Various stats
SHOW ENGINE NDB STATUS; Connection level
Understand SQL -> Handler -> NdbApi mapping
Also available from
INFORMATION_SCHEMA for
use in SELECTs + Views

MySQL Server NdbApi Call Profile
mysql> show status like 'ndb_api%session';
+----------------------------------------------+------------+
| Variable_name | Value |
+----------------------------------------------+------------+
| Ndb_api_wait_exec_complete_count_session | 12 |
| Ndb_api_wait_scan_result_count_session | 9 |
| Ndb_api_wait_meta_request_count_session | 24 |
| Ndb_api_wait_nanos_count_session | 1086595954 |
| Ndb_api_bytes_sent_count_session | 8504 |
| Ndb_api_bytes_received_count_session | 12032 |
| Ndb_api_trans_start_count_session | 6 |
| Ndb_api_trans_commit_count_session | 4 |
| Ndb_api_trans_abort_count_session | 1 |
| Ndb_api_trans_close_count_session | 6 |
| Ndb_api_pk_op_count_session | 11 |
| Ndb_api_uk_op_count_session | 0 |
| Ndb_api_table_scan_count_session | 2 |
...
+----------------------------------------------+------------+
mysql>
Network roundtrips
Api Data transfer
NdbApi api call
profile
Details : http://messagepassing.blogspot.com/
2011/04/journey-upriver-to-dark-heart-of.html
+ MySQL Cluster Reference Manual
Also available
from NdbApi!

Use OS-level Tools to Understand Performance
top
IOSTAT
VMSTAT
mpstat
strace
perf
dtrace
oprofile
netstat
Ifconfig
PMP

MySQL Server Profiling in Production
Enable the slow query log
set global slow_query_log=1;
set global long_query_time=3; //3 seconds
set global log_queries_not_using_indexes=1;
Slow queries will be written in the slow query log:
mysql> show global variables like 'slow_query_log_file';
+---------------------+------------------------------+
| Variable_name | Value |
+---------------------+------------------------------+
| slow_query_log_file | /data1/mysql/mysqld-slow.log |
+---------------------+------------------------------+
Queries will be written in plain text

Query Analyzer?

Insert Picture Here
Boosting Performance

Using AQL to Perform Efficient Joins (7.2 only)
Activated when ndb_join_pushdown is on (default)
Rules for a Join to be pushed down:
- Joined columns use identical types
- No reference to BLOB or TEXT columns
- No explicit locking
- Child tables in the Join must be accessed using one of the ref, eq_ref, or
const
- Tables not explicitly partitioned by [LINEAR] HASH, LIST, or RANGE
- Query plan doesn’t select ‘Using join buffer'
- If root of Join is an eq_ref or const, child tables must be joined by eq_ref
Run ANALYZE TABLE <tab-name> on each table once
Use EXPLAIN to see what components are being pushed down:
Extra: Child of 'd' in pushed join@1
EXPLAIN EXTENDED <query>;SHOW WARNINGS;.
Joins are far more time
efficient than before,
but can still be
resource heavy.

Batching Within a Transaction
Batching amortises the round-trip latency across multiple operations and/or
results.
Primary and unique key operations can be sent in batches, which will be
executed concurrently + in parallel if on different nodes/fragments.
Ordered index and table scans send their results in batches, allowing buffer
space to be traded for round trips.
Example – Insert 1M records
No batching:
INSERT INTO t1(data) VALUES (<data>);
765 seconds to insert 1M records
Batching (batches of 16):
INSERT INTO t1(<columns>) VALUES (<data>), (<data>)...
50 seconds to insert 1M records
Time ratio here is 15.3:1 –
almost the same as the
batching factor. Most of
the response time is
request/response latency.

Batching Within a Transaction
SELECT * FROM t1 WHERE userid=1 AND serviceid IN
(1,2,3,4,5,7,8,9,10);
SET transaction_allow_batching=1; //must be set on the connection
BEGIN;
INSERT INTO t1 ....;
DELETE FROM t5 ....;
UPDATE t1 SET value='new value' WHERE id=1;
COMMIT;
Verify the effect of these statements using the SHOW STATUS LIKE
'ndb_api_wait%session' counters.
Errors associated with a
statement may be
reported against a later
statement in the
transaction.

Distribution Awareness
Select a partitioning key for your schema which maximises the locality of your
high-running queries (Subscriber id, Session id, Product id, Order id, Tree root
nodes)
Doesn't matter if low-running queries have no locality.
Consider adding extra key columns to your tables to enable this, using a
secondary unique index to enforce uniqueness on the original PK if necessary.
Change DDL to use partitioning key : CREATE TABLE t1 … ENGINE=NDB
PARTITION BY KEY <My key(s)>;
In 'high running' transactions, perform first query using EQ on partitioning key
column(s) to give placement hint to MySQL Server.
Consider splitting your transactions if the partitioning key value is not known at
the start of a high running transaction. Do you need transactional semantics?
Specified per-table,
used across tables
Monitor distribution awareness and pruning
using SHOW STATUS and NdbInfo

Simplify Transactions
Simpler transactions (fewer tables, fewer rows, primary key access) are much
simpler for the system to run fast.
Schema denormalisation can remove the need for some joins.
Some applications have data which need not be disk-persistent (WAL redo
logged and checkpointed). By removing their disk-persistence, they can sustain
high write rates without pressuring disk IO. After a system recovery, the table will
be empty.
mysql> set ndb_table_no_logging=1;
mysql> create table session_table(..) engine=ndb;
mysql> set ndb_table_no_logging=0;

Multithreaded Data Nodes
MySQL Cluster data nodes use many threads for disk and network IO, and
database request processing. Request processing threads use a shared-nothing,
non-blocking message passing model with very little inter-thread locking.
Multithreading in Ndb is to enable use of multiple HW cores, therefore, roughly
one request processing thread per HW core is used as a guideline when
configuring.
The threads are asymmetric – work is partitioned among them functionally. This
means that the balance of thread types and counts depends on the workload to
be handled. If in doubt, use the default configuration for your HW core
count.

Multithreaded Data Nodes
Replication thread
Main thread
LDM threads
TC threads
Send threads
Request
processing
threads
TC and LDM threads
do most work, must
be well fed by Send +
Receive threads
Receive threads IO threads
Connect threads Watchdog

Hardware for Performance
High bandwidth, low latency network
- Infiniband is nice
Multiple disks if disk IO is a problem
- Split checkpoints from Redo logs from DD Tablespaces
Consider SSDs
- Best for DD Tablespaces as logs and Checkpoints are mostly serialised writes
Refer to MySQL Cluster Evaluation Guide for more advice
MySQL Cluster runs great on low end hardware, especially where applications,
schemas and queries are tuned for it.

Choice of Api
Lower level Apis can give more control, as well as removing latency and
complexity.
However, they may require development to integrate with your systems.
For the best performance, use C++ NdbApi, or the ClusterJ Java Api. NdbApi
gives greater control, at the cost of greater complexity.
The memcached or HTTP/REST Apis also expose good performance through a
simplified Api.
Native memcached HTTP/REST
JDBC / ODBC
PHP / PERL
Python / Ruby

Breakthrough Performance?
1. Understand the system and its bottlenecks
MySQL manual, blogs, whitepapers, forum, mailing list,
Experimentation
2. Set goals
3. while (!goal.met())
{Benchmark, Monitor, Measure, Optimise}

Connection Pooling and the Query Cache
Connection pooling here refers to having more than one logical connection from a
MySQL Server to the Ndb data nodes. (Each logical connection has many
internal socket connections).
This can be used to avoid bottlenecks due to per-logical connection locking.
Consumes [API] slots.
The MySQL Query cache caches query results. However it is asynchronously
invalidated by Ndb as the cached tables are changed in the cluster.
With a high change rate, the rate of query cache invalidations can become
excessive.
Should only really be considered for very static tables, where applications can
tolerate some staleness.

Breakthrough performance with MySQL Cluster (2012)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Breakthrough performance with MySQL Cluster (2012)

Similar to Breakthrough performance with MySQL Cluster (2012) (20)

Recently uploaded

Recently uploaded (20)

Breakthrough performance with MySQL Cluster (2012)