The MySQL Cluster 7.x series introduced a number of features to allow for fine-grained control over the real-time behaviour of the NDB storage engine. New threads have been introduced, and users are able to control placement of these threads, as well as locking the memory such that no swapping occurs. In an ideal run-time environment, CPUs handling data node threads will not execute other threads apart from OS kernel threads or interrupt handling. Correct tuning of certain parameters can be specially important for certain types of workloads. This presentation covers the different tuning aspects of MySQL Cluster. - Application design guidelines - Schema Optimization - Index Selection and Tuning - Query Tuning - OS Tuning - Data Node internals - Optimizations for real-time behaviour This presentation looks closely at how to get the most out of your MySQL Cluster 7.x runtime environment.

1. Performance Tuning of MySQL Cluster
April 2014
Johan Andersson
Severalnines AB
johan@severalnines.com

2. Agenda
! 7.4 – What’s coming?
! 7.3 Feature Update
! OS Tuning
! Stability Tuning
! Application design
! Identifying bottlenecks
! Tuning tricks
2
Copyright 2011 Severalnines AB

3. 7.4 - what is coming
! Optimized Scans
! Overload/Overcommitting
Resource protection
! Improved Checkpointing
! 10% more performance
3
Copyright 2011 Severalnines AB

4. 7.3 Feature Update
! Node.js Connector
! JavaScript (V8 engine) to access data directly in the
Data nodes
! No SQL – bypasses the MySQL Server ! lower latency,
high throughput for simple queries (like PK operations,
simple scans from one table)
4
Copyright 2011 Severalnines AB

5. 7.3 Feature Update
! FOREIGN KEYs finally supported!
! Implemented at the Data Node level
! But..
ERROR 1506 (HY000): Foreign key clause is not yet
supported in conjunction with partitioning
! Hopefully fixed for the GA release
! What about the performance penalty?
5
Copyright 2011 Severalnines AB

6. 7.3 Foreign Key Perf
create table users_posts (
uid integer ,
fid integer ,
pid integer auto_increment,
message varchar(1024),
primary key(uid,fid, pid),
constraint fk_forum foreign key(fid) references forum(fid) on delete cascade,
constraint fk_user foreign key(uid) references users(uid) on delete cascade
) engine=ndb;
! Compare INSERT performance with and w/o FKs
! With FK, must check that forum(fid) and users(uid) exists.
! Populate with 1M records
6
Copyright 2011 Severalnines AB

7. 7.3 Foreign Key Perf
! Bencher drivers the load:
! https://github.com/severalnines/bencher
! 4 threads, 2 data nodes, 4 cores,
! App ! mysqld ! data nodes
! FOREIGN KEYs enabled
Summary:
--------------------------
Average Throughput = 1274.58 tps (stdev=59.71)
7
Copyright 2011 Severalnines AB

8. 7.3 Foreign Key Perf
! Bencher drivers the load:
! https://github.com/severalnines/bencher
! 4 threads, 2 data nodes, 4 cores,
! App ! mysqld ! data nodes
! Not using FOREIGN KEYs
Summary:
--------------------------
Average Throughput = 1428.57 tps (stdev=57.10)
! Foreign keys gave ~11% drop in performance.
8
Copyright 2011 Severalnines AB

9. 7.2 ! 7.3 Caveats
! A little gotcha:
! --engine-condition-pushdown ! no longer supported in
MySQL 5.6
! Mysqld will fail to start
! Take it out from my.cnf before upgrading!
9
Copyright 2011 Severalnines AB

10. Facts
! A single query will never run as fast as on Innodb
(served from RAM)
! Network latency is a issue
! More data nodes does not speed up query execution
time.
10
Copyright 2011 Severalnines AB

11. OS Tuning
! Disable NUMA in /etc/grub.conf
! echo ‘0’ > /proc/sys/vm/swappiness
echo ‘vm.swappiness=0’ >> /etc/sysctl.conf
! Bind data node threads to CPUs/cores
! cat /proc/interrupts | grep eth
cpu0 cpu1 cpu2 cpu3
44: 31 49432584 0 0 xen-dyn-event eth0"
45: 1633715292 0 0 0 xen-dyn-event eth1"
11
Copyright 2011 Severalnines AB
Avoid!
OK!
In config.ini [ndbd default]:
ThreadConfig=ldm={count=1,cpubind=1,2},main={cpubind=3} ..

12. Perf Config Tuning (1)
! How many Local Data Managers ?
! How many TC threads?
Copyright 2011 Severalnines AB
12

13. Perf Config Tuning (2)
Copyright 2011 Severalnines AB
13
3
4
5
9
10
11
IRQBALANCE_BANNED_CPUS
IRQ
CPU0
CPU1

14. Perf Config Tuning (3)
! Run representative load
! Does any NDB Thread run hot?
! Increase with threads of that type
! Enable HyperThreading
! Can give 40% more
! Enable:
! RealTimeScheduler=1
! Don’t forget:
! NoOfFragmentLogParts=<no of LDMs>
Copyright 2011 Severalnines AB
14

15. Stability Tuning
! Tuning the REDO log is key
! FragmentLogFileSize=256M
! NoOfFragmentLogFiles=<4-6> X DataMemory in MB / 4 x
FragmentLogFileSize
! RedoBuffer=64M for a write busy system
! Disk based data:
! SharedGlobalMemory=4096M
! In the LOGFILE GROUP: undo_buffer_size=128M
! Or higher (max is 600M)
15
Copyright 2011 Severalnines AB

16. Stability Tuning
! Make sure you don’t have more “execution threads” than cores
! You want to have
! Major page faults low
! Involuntary context switches low
mysql> SELECT node_id, thr_no,thr_nm , os_ru_majflt,
os_ru_nivcsw FROM threadstat;
+---------+--------+--------+--------------+--------------+
| node_id | thr_no | thr_nm | os_ru_majflt | os_ru_nivcsw |
+---------+--------+--------+--------------+--------------+
| 3 | 0 | main | 1 | 541719 |
| 4 | 0 | main | 0 | 561769 |
+---------+--------+--------+--------------+--------------+
2 rows in set (0.01 sec)
16
Copyright 2011 Severalnines AB

17. Application Design
! Define the most typical Use Cases
! List all my friends, session management etc etc.
! Optimize everything for the typical use case
! Engineer schema to cater for the Use Cases
! Keep it simple
! Complex access patterns does not scale
! Simple access patterns do ( Primay key and Partitioned Index Scans )
! Note! There is no parameter in config.ini that affects
performance – only availability.
! Everything is about the Schema and the Queries.
! Tune the mysql servers (sort buffers etc) as you would for innodb.
17
Copyright 2011 Severalnines AB

18. Simple Access
! PRIMARY KEY lookups are HASH lookup O(1)
! INDEX searches a T-tree and takes O(log n) time.
! In 7.2 and later JOINs are ok, but in 7.1 you should try
to avoid them.
18
Copyright 2011 Severalnines AB

19. Identifying Bottlenecks
! High CPU usage on data nodes
! Probably a lot of large index scans and full table scans are used.
! Check Slow query log or a query monitor
! High CPU usage on MySQL servers
! Probably a lot of GROUP BY/DISTINCT or aggregate functions.
! Hardly no CPU is used on either MySQL or data nodes
! Probably low load
! Time is spent on network (a lot of “ping pong” to satisfy a request).
! System is running slow in general
! Disks (io util), queries, swap (must never happen), network

20. Need To Add Data Nodes?
! (adding MySQL servers is easy)
! top –Hd1
! Is any of data nodes threads at 100%?
! Yes: add more data nodes (online)
! No: do nothing

21. Detecting Query Problems
! Here is a standard method for how to attack the problem.
! Performance tuning is a never-ending loop:
BEGIN
– Capture information – e.g, slow query log
• Change long_query_time if needed
– EXPLAIN the queries
• What indexes are used?
• Are tables JOINed in the correct order (small to big)
– Re-run the optimized typical use cases using bencher/
mysqlslap
GOTO BEGIN;
END;
! Never tune unless you can measure and test!
! Don't optimize unless you have a problem!

22. Enable Logging
! Slow query log
! set global slow_query_log=1;
! set global long_query_time=0.01;
! set global log_queries_not_using_indexes=1;
! General log (if you don’t get enough info in the Slow
Query Log)
! Activate for a very short period of time (30-60seconds) –
intrusive
! Can fill up disk very fast – make sure you turn it off.
! set global general_log=1;
! Use Severalnines ClusterControl
! Includes a Cluster-wide Query Monitor.
! Query frequency, EXPLAINs, lock time etc.
! Performance Monitor and Manager.

23. Setup
23
Copyright 2011 Severalnines AB
UID FID data
1 1 A
2 3 B
1 2 C
2 4 D
table t1
Partition 0
Partition 1
NETWORK!

24. Sharding
! By default, all index scans hit all data nodes
! good if result set is big – you want as many CPUs as possible to
help you.
! For smaller result sets (~a couple of hundred records) Partition
Pruning is key for scalability.
! User-defined partitioning can help to improve equality index
scans on part of a primary key.
! CREATE TABLE t1 (uid,
fid,
somedata,
PRIMARY KEY(uid, fid))
PARTITION BY KEY(uid);
! All data belonging to a particular uid will be on the same
partition.
! Great locality!
! select * from user where uid=1;
! Only one data node will be scanned (no matter how many
nodes you have)

25. Sharding
UID FID data
1 1 A
2 3 B
1 2 C
2 4 D
table t1
UID FID data
1 1 A
1 2 C
2 3 B
2 4 D
table t1
Partition 0
Partition 1
PARTITION BY KEY(UID)Default Partitioning

26. Sharding
mysql> show global status like 'ndb_pruned_scan_count’;
+-----------------------+-------+
| Variable_name | Value |
+-----------------------+-------+
| Ndb_pruned_scan_count | 0 |
+-----------------------+-------+
CREATE TABLE t1( … ) PARTITION BY KEY (userid);
An run query, and verify it works:
select * from user where uid=1;
mysql> show global status like 'ndb_pruned_scan_count’;
+-----------------------+-------+
| Variable_name | Value |
+-----------------------+-------+
| Ndb_pruned_scan_count | 1 |
+-----------------------+-------+

27. Sharding
mysql> show global status like 'ndb%pruned%';
| Ndb_api_table_scan_count | 264 |
| Ndb_api_range_scan_count | 18 |
| Ndb_api_pruned_scan_count | 3 |

28. Sharding - EXAMPLE
create table users_posts2 (
uid integer ,
fid integer ,
pid integer auto_increment,
message varchar(1024),
primary key(uid,fid, pid)
) engine=ndb partition by key(uid);
NO PARTITIONING
create table users_posts2 (
uid integer ,
fid integer ,
pid integer auto_increment,
message varchar(1024),
primary key(uid,fid, pid)
) engine=ndb;
PARTITION BY KEY

29. Sharding – EXPLAIN PARTITIONS
mysql> explain partitions select * from users_posts u where u.uid=1G
id: 1
select_type: SIMPLE
table: u
partitions: p0,p1
type: ref
possible_keys: PRIMARY
key: PRIMARY
key_len: 4
ref: const
rows: 2699
Extra: NULL
With PARTITION BY KEY (UID)
mysql> explain partitions select * from users_posts2 u where
u.uid=1G
id: 1
select_type: SIMPLE
table: u
partitions: p0
type: ref
possible_keys: PRIMARY
key: PRIMARY
key_len: 4
ref: const
rows: 2699
Extra: NULL

30. Data Types
BLOBs/TEXTs vs VARBINARY/VARCHAR
! BLOB/TEXT columns are stored in an external hidden table.
! First 256B are stored inline in main table
! Reading a BLOB/TEXT requires two reads
! One for reading the Main table + reading from hidden
table
! Change to VARBINARY/VARCHAR if:
! Your BLOB/TEXTs can fit within an 14000 Bytes record
! (record size is currently 14000 Bytes)
! Reading/writing VARCHAR/VARBINARY is less expensive
Note 1: BLOB/TEXT are also more expensive in Innodb as BLOB/TEXT data is
not inlined with the table. Thus, two disk seeks are needed to read a
BLOB.
Note 2: Store images, movies etc outside the database on the filesystem.

31. Query Tuning
! MySQL Cluster 7.2 and later has pushed down joins ! joins
are performed in the data nodes.
! OPTIMIZER in MySQL Cluster 7.1 and earlier is weak
! Statistics gathering is non-existing
! Optimizer thinks there are only 10 rows to examine in each
table!
! FORCE INDEX / STRAIGHt_JOIN to get queries run the way you
want

32. Query Tuning
! if you have two similar indexes:
! index(a)
! index(a,ts)
on the following table
CREATE TABLE `t1` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`a` bigint(20) DEFAULT NULL,
`ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `idx_t1_a` (`a`),
KEY `idx_t1_a_ts` (`a`,`ts`)) ENGINE=ndbcluster DEFAULT CHARSET=latin1

33. Query Tuning
mysql> select count(id) from t1 where a=5;
+-----------+
| count(id) |
+-----------+
| 3072000 |
+-----------+
1 row in set (0.02 sec)
mysql> select count(id) from t1 where a=5
and ts>'2013-04-18 14:34:08’;
+-----------+
| count(id) |
+-----------+
| 512 |
+-----------+
1 row in set (0.00 sec)

34. Query Tuning Pre 7.2
mysql> explain select * from t1 where a=2 and ts='2011-10-05 15:32:11';
+----+-------------+-------+------+----------------------+----------+---------+-------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref |
rows | Extra |
+----+-------------+-------+------+----------------------+----------+---------+-------+------+-------------+
| 1 | SIMPLE | t1 | ref | idx_t1_a,idx_t1_a_ts | idx_t1_a | 9 | const | 10 |
Using where |
+----+-------------+-------+------+----------------------+----------+---------+-------+------+-------------+
! Use FORCE INDEX(..) ...
mysql> explain select * from t1 FORCE INDEX (idx_t1_a_ts) where a=2 and ts='2011-10-05
15:32:11;
+| 1 | SIMPLE | t1 | ref | idx_t1_a_ts | idx_t1_a_ts | 13 | const,const | 10 |
Using where |
1 row in set (0.00 sec)
! ..to ensure the correct index is picked!
! The difference can be 1 record read instead of any
number of records!

35. Index Statistics
explain select * from t1 where a=5 and ts>'2013-04-18 14:34:08' G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: t1
type: range
possible_keys: idx_t1_a,idx_t1_a_ts
key: idx_t1_a
key_len: 9
ref: const
Rows: 17
Extra: Using where with pushed condition

36. Index Statistics
mysql> analyze table t1;
+---------+---------+----------+----------+
| Table | Op | Msg_type | Msg_text |
+---------+---------+----------+----------+
| test.t1 | analyze | status | OK |
+---------+---------+----------+----------+
1 row in set (3.40 sec)

37. Index Statistics
Mysql> explain select * from t1 where a=5
and ts>'2013-04-18 14:34:08' G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: t1
type: range
possible_keys: idx_t1_a,idx_t1_a_ts
key: idx_t1_a_ts
key_len: 13
ref: NULL
rows: 253
Extra: Using where with pushed condition; Using MRR
1 row in set (0.00 sec)

38. Ndb_cluster_connection_pool
! Improved in MC 7.3, set ndb_cluster_connection_pool
between 1 and 4.
! MySQL Cluster 7.2 and earlier:
! Problem:
! A Sendbuffer on the connection between mysqld and the
data nodes is protected by a Mutex.
! Connection threads in MySQL must acquire Mutex and the put
data in SendBuffer.
! Many threads gives more contention on the mutex
! Must scale out with many MySQL Servers.
! Workaround:
! Ndb_cluster_connection_pool (in my.cnf) creates more
connections from one mysqld to the data nodes
! Threads load balance on the connections gives less
contention on mutex which in turn gives increased scalabilty
! Less MySQL Servers needed to drive load!
! www.severalnines.com/cluster-configurator allows you to
specify the connection pool.

39. Ndb_cluster_connection_pool
! Improved in MC 7.3, set ndb_cluster_connection_pool
between 1 and 4.
! MySQL Cluster 7.2 and earlier:
! Gives at least 70% better performance and a MySQL Server that
can scale beyond four database connections.
! Set Ndb_cluster_connection_pool=2x<CPU cores>
! It is a good starting point
! One free [mysqld] slot is required in config.ini for each
Ndb_cluster_connection.
! 4 mysql servers,each with Ndb_cluster_connection_pool=8
requires 32 [mysqld] in config.ini
! Note that also memcached and node.js, cluster/j etc also has
the concept of the ndb_cluster_connection_pool.

40. Q&A
40
Copyright 2011 Severalnines AB

41. Resources
! MySQL Cluster Configurator
! www.severalnines.com/config
! MySQL Cluster Management + Monitoring
! www.severalnines.com/cmon
! MySQL Cluster Training Slides
! www.severalnines.com/mysql-cluster-training
! My Blog
! johanandersson.blogspot.com

42. Keep in touch…
! Facebook
! www.facebook.com/severalnines
! Twitter
! @severalnines
! Linked in:
! www.linkedin.com/company/severalnines

43. Thank you for your time!
johan@severalnines.com
43
Copyright 2011 Severalnines AB