Your SlideShare is downloading. ×
Performance Tuning of MySQL Cluster
April 2014
Johan Andersson
Severalnines AB
johan@severalnines.com
Agenda
!  7.4 – What’s coming?
!  7.3 Feature Update
!  OS Tuning
!  Stability Tuning
!  Application design
!  Identifying...
7.4 - what is coming
!  Optimized Scans
!  Overload/Overcommitting
Resource protection
!  Improved Checkpointing
!  10% mo...
7.3 Feature Update
!  Node.js Connector
!  JavaScript (V8 engine) to access data directly in the
Data nodes
!  No SQL – by...
7.3 Feature Update
!  FOREIGN KEYs finally supported!
!  Implemented at the Data Node level
!  But..
ERROR 1506 (HY000): F...
7.3 Foreign Key Perf
create table users_posts (
uid integer ,
fid integer ,
pid integer auto_increment,
message varchar(10...
7.3 Foreign Key Perf
!  Bencher drivers the load:
!  https://github.com/severalnines/bencher
!  4 threads, 2 data nodes, 4...
7.3 Foreign Key Perf
!  Bencher drivers the load:
!  https://github.com/severalnines/bencher
!  4 threads, 2 data nodes, 4...
7.2 ! 7.3 Caveats
!  A little gotcha:
!  --engine-condition-pushdown ! no longer supported in
MySQL 5.6
!  Mysqld will fai...
Facts
!  A single query will never run as fast as on Innodb
(served from RAM)
!  Network latency is a issue
!  More data n...
OS Tuning
!  Disable NUMA in /etc/grub.conf
!  echo ‘0’ > /proc/sys/vm/swappiness
echo ‘vm.swappiness=0’ >> /etc/sysctl.co...
Perf Config Tuning (1)
!  How many Local Data Managers ?
!  How many TC threads?
Copyright 2011 Severalnines AB
12
Perf Config Tuning (2)
Copyright 2011 Severalnines AB
13
3
4
5
9
10
11
IRQBALANCE_BANNED_CPUS
IRQ
CPU0
CPU1
Perf Config Tuning (3)
!  Run representative load
!  Does any NDB Thread run hot?
!  Increase with threads of that type
! ...
Stability Tuning
!  Tuning the REDO log is key
!  FragmentLogFileSize=256M
!  NoOfFragmentLogFiles=<4-6> X DataMemory in M...
Stability Tuning
!  Make sure you don’t have more “execution threads” than cores
!  You want to have
!  Major page faults ...
Application Design
!  Define the most typical Use Cases
!  List all my friends, session management etc etc.
!  Optimize ev...
Simple Access
!  PRIMARY KEY lookups are HASH lookup O(1)
!  INDEX searches a T-tree and takes O(log n) time.
!  In 7.2 an...
Identifying Bottlenecks
!  High CPU usage on data nodes
!  Probably a lot of large index scans and full table scans are us...
Need To Add Data Nodes?
!  (adding MySQL servers is easy)
!  top –Hd1
!  Is any of data nodes threads at 100%?
!  Yes: add...
Detecting Query Problems
!  Here is a standard method for how to attack the problem.
!  Performance tuning is a never-endi...
Enable Logging
!  Slow query log
!  set global slow_query_log=1;
!  set global long_query_time=0.01;
!  set global log_que...
Setup
23
Copyright 2011 Severalnines AB
UID FID data
1 1 A
2 3 B
1 2 C
2 4 D
table t1
Partition 0
Partition 1
NETWORK!
Sharding
!  By default, all index scans hit all data nodes
!  good if result set is big – you want as many CPUs as possibl...
Sharding
UID FID data
1 1 A
2 3 B
1 2 C
2 4 D
table t1
UID FID data
1 1 A
1 2 C
2 3 B
2 4 D
table t1
Partition 0
Partition...
Sharding
mysql> show global status like 'ndb_pruned_scan_count’;
+-----------------------+-------+
| Variable_name | Value...
Sharding
mysql> show global status like 'ndb%pruned%';
| Ndb_api_table_scan_count | 264 |
| Ndb_api_range_scan_count | 18 ...
Sharding - EXAMPLE
create table users_posts2 (
uid integer ,
fid integer ,
pid integer auto_increment,
message varchar(102...
Sharding – EXPLAIN PARTITIONS
mysql> explain partitions select * from users_posts u where u.uid=1G
id: 1
select_type: SIMP...
Data Types
BLOBs/TEXTs vs VARBINARY/VARCHAR
!  BLOB/TEXT columns are stored in an external hidden table.
!  First 256B are...
Query Tuning
!  MySQL Cluster 7.2 and later has pushed down joins ! joins
are performed in the data nodes.
!  OPTIMIZER in...
Query Tuning
!  if you have two similar indexes:
!  index(a)
!  index(a,ts)
on the following table
CREATE TABLE `t1` (
`id...
Query Tuning
mysql> select count(id) from t1 where a=5;
+-----------+
| count(id) |
+-----------+
| 3072000 |
+-----------...
Query Tuning Pre 7.2
mysql> explain select * from t1 where a=2 and ts='2011-10-05 15:32:11';
+----+-------------+-------+-...
Index Statistics
explain select * from t1 where a=5 and ts>'2013-04-18 14:34:08' G
*************************** 1. row ****...
Index Statistics
mysql> analyze table t1;
+---------+---------+----------+----------+
| Table | Op | Msg_type | Msg_text |...
Index Statistics
Mysql> explain select * from t1 where a=5
and ts>'2013-04-18 14:34:08' G
*************************** 1. r...
Ndb_cluster_connection_pool
!  Improved in MC 7.3, set ndb_cluster_connection_pool
between 1 and 4.
!  MySQL Cluster 7.2 a...
Ndb_cluster_connection_pool
!  Improved in MC 7.3, set ndb_cluster_connection_pool
between 1 and 4.
!  MySQL Cluster 7.2 a...
Q&A
40
Copyright 2011 Severalnines AB
Resources
!  MySQL Cluster Configurator
!  www.severalnines.com/config
!  MySQL Cluster Management + Monitoring
!  www.sev...
Keep in touch…
!  Facebook
!  www.facebook.com/severalnines
!  Twitter
!  @severalnines
!  Linked in:
!  www.linkedin.com/...
Thank you for your time!
johan@severalnines.com
43
Copyright 2011 Severalnines AB
Upcoming SlideShare
Loading in...5
×

MySQL Cluster 7.3 Performance Tuning - Severalnines Slides

1,295

Published on

The MySQL Cluster 7.x series introduced a number of features to allow for fine-grained control over the real-time behaviour of the NDB storage engine. New threads have been introduced, and users are able to control placement of these threads, as well as locking the memory such that no swapping occurs. In an ideal run-time environment, CPUs handling data node threads will not execute other threads apart from OS kernel threads or interrupt handling. Correct tuning of certain parameters can be specially important for certain types of workloads.

This presentation covers the different tuning aspects of MySQL Cluster.

- Application design guidelines
- Schema Optimization
- Index Selection and Tuning
- Query Tuning
- OS Tuning
- Data Node internals
- Optimizations for real-time behaviour

This presentation looks closely at how to get the most out of your MySQL Cluster 7.x runtime environment.

Published in: Technology, Business
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,295
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Transcript of "MySQL Cluster 7.3 Performance Tuning - Severalnines Slides"

  1. 1. Performance Tuning of MySQL Cluster April 2014 Johan Andersson Severalnines AB johan@severalnines.com
  2. 2. Agenda !  7.4 – What’s coming? !  7.3 Feature Update !  OS Tuning !  Stability Tuning !  Application design !  Identifying bottlenecks !  Tuning tricks 2 Copyright 2011 Severalnines AB
  3. 3. 7.4 - what is coming !  Optimized Scans !  Overload/Overcommitting Resource protection !  Improved Checkpointing !  10% more performance 3 Copyright 2011 Severalnines AB
  4. 4. 7.3 Feature Update !  Node.js Connector !  JavaScript (V8 engine) to access data directly in the Data nodes !  No SQL – bypasses the MySQL Server ! lower latency, high throughput for simple queries (like PK operations, simple scans from one table) 4 Copyright 2011 Severalnines AB
  5. 5. 7.3 Feature Update !  FOREIGN KEYs finally supported! !  Implemented at the Data Node level !  But.. ERROR 1506 (HY000): Foreign key clause is not yet supported in conjunction with partitioning !  Hopefully fixed for the GA release !  What about the performance penalty? 5 Copyright 2011 Severalnines AB
  6. 6. 7.3 Foreign Key Perf create table users_posts ( uid integer , fid integer , pid integer auto_increment, message varchar(1024), primary key(uid,fid, pid), constraint fk_forum foreign key(fid) references forum(fid) on delete cascade, constraint fk_user foreign key(uid) references users(uid) on delete cascade ) engine=ndb; !  Compare INSERT performance with and w/o FKs !  With FK, must check that forum(fid) and users(uid) exists. !  Populate with 1M records 6 Copyright 2011 Severalnines AB
  7. 7. 7.3 Foreign Key Perf !  Bencher drivers the load: !  https://github.com/severalnines/bencher !  4 threads, 2 data nodes, 4 cores, !  App ! mysqld ! data nodes !  FOREIGN KEYs enabled Summary: -------------------------- Average Throughput = 1274.58 tps (stdev=59.71) 7 Copyright 2011 Severalnines AB
  8. 8. 7.3 Foreign Key Perf !  Bencher drivers the load: !  https://github.com/severalnines/bencher !  4 threads, 2 data nodes, 4 cores, !  App ! mysqld ! data nodes !  Not using FOREIGN KEYs Summary: -------------------------- Average Throughput = 1428.57 tps (stdev=57.10) !  Foreign keys gave ~11% drop in performance. 8 Copyright 2011 Severalnines AB
  9. 9. 7.2 ! 7.3 Caveats !  A little gotcha: !  --engine-condition-pushdown ! no longer supported in MySQL 5.6 !  Mysqld will fail to start !  Take it out from my.cnf before upgrading! 9 Copyright 2011 Severalnines AB
  10. 10. Facts !  A single query will never run as fast as on Innodb (served from RAM) !  Network latency is a issue !  More data nodes does not speed up query execution time. 10 Copyright 2011 Severalnines AB
  11. 11. OS Tuning !  Disable NUMA in /etc/grub.conf !  echo ‘0’ > /proc/sys/vm/swappiness echo ‘vm.swappiness=0’ >> /etc/sysctl.conf !  Bind data node threads to CPUs/cores !  cat /proc/interrupts | grep eth cpu0 cpu1 cpu2 cpu3
 44: 31 49432584 0 0 xen-dyn-event eth0" 45: 1633715292 0 0 0 xen-dyn-event eth1" 11 Copyright 2011 Severalnines AB Avoid! OK! In config.ini [ndbd default]: ThreadConfig=ldm={count=1,cpubind=1,2},main={cpubind=3} ..
  12. 12. Perf Config Tuning (1) !  How many Local Data Managers ? !  How many TC threads? Copyright 2011 Severalnines AB 12
  13. 13. Perf Config Tuning (2) Copyright 2011 Severalnines AB 13 3 4 5 9 10 11 IRQBALANCE_BANNED_CPUS IRQ CPU0 CPU1
  14. 14. Perf Config Tuning (3) !  Run representative load !  Does any NDB Thread run hot? !  Increase with threads of that type !  Enable HyperThreading !  Can give 40% more !  Enable: !  RealTimeScheduler=1 !  Don’t forget: !  NoOfFragmentLogParts=<no of LDMs> Copyright 2011 Severalnines AB 14
  15. 15. Stability Tuning !  Tuning the REDO log is key !  FragmentLogFileSize=256M !  NoOfFragmentLogFiles=<4-6> X DataMemory in MB / 4 x FragmentLogFileSize !  RedoBuffer=64M for a write busy system !  Disk based data: !  SharedGlobalMemory=4096M !  In the LOGFILE GROUP: undo_buffer_size=128M !  Or higher (max is 600M) 15 Copyright 2011 Severalnines AB
  16. 16. Stability Tuning !  Make sure you don’t have more “execution threads” than cores !  You want to have !  Major page faults low !  Involuntary context switches low mysql> SELECT node_id, thr_no,thr_nm , os_ru_majflt, os_ru_nivcsw FROM threadstat; +---------+--------+--------+--------------+--------------+ | node_id | thr_no | thr_nm | os_ru_majflt | os_ru_nivcsw | +---------+--------+--------+--------------+--------------+ | 3 | 0 | main | 1 | 541719 | | 4 | 0 | main | 0 | 561769 | +---------+--------+--------+--------------+--------------+ 2 rows in set (0.01 sec) 16 Copyright 2011 Severalnines AB
  17. 17. Application Design !  Define the most typical Use Cases !  List all my friends, session management etc etc. !  Optimize everything for the typical use case !  Engineer schema to cater for the Use Cases !  Keep it simple !  Complex access patterns does not scale !  Simple access patterns do ( Primay key and Partitioned Index Scans ) !  Note! There is no parameter in config.ini that affects performance – only availability. !  Everything is about the Schema and the Queries. !  Tune the mysql servers (sort buffers etc) as you would for innodb. 17 Copyright 2011 Severalnines AB
  18. 18. Simple Access !  PRIMARY KEY lookups are HASH lookup O(1) !  INDEX searches a T-tree and takes O(log n) time. !  In 7.2 and later JOINs are ok, but in 7.1 you should try to avoid them. 18 Copyright 2011 Severalnines AB
  19. 19. Identifying Bottlenecks !  High CPU usage on data nodes !  Probably a lot of large index scans and full table scans are used. !  Check Slow query log or a query monitor !  High CPU usage on MySQL servers !  Probably a lot of GROUP BY/DISTINCT or aggregate functions. !  Hardly no CPU is used on either MySQL or data nodes !  Probably low load !  Time is spent on network (a lot of “ping pong” to satisfy a request). !  System is running slow in general !  Disks (io util), queries, swap (must never happen), network
  20. 20. Need To Add Data Nodes? !  (adding MySQL servers is easy) !  top –Hd1 !  Is any of data nodes threads at 100%? !  Yes: add more data nodes (online) !  No: do nothing
  21. 21. Detecting Query Problems !  Here is a standard method for how to attack the problem. !  Performance tuning is a never-ending loop: BEGIN –  Capture information – e.g, slow query log •  Change long_query_time if needed –  EXPLAIN the queries •  What indexes are used? •  Are tables JOINed in the correct order (small to big) –  Re-run the optimized typical use cases using bencher/ mysqlslap GOTO BEGIN; END; !  Never tune unless you can measure and test! !  Don't optimize unless you have a problem!
  22. 22. Enable Logging !  Slow query log !  set global slow_query_log=1; !  set global long_query_time=0.01; !  set global log_queries_not_using_indexes=1; !  General log (if you don’t get enough info in the Slow Query Log) !  Activate for a very short period of time (30-60seconds) – intrusive !  Can fill up disk very fast – make sure you turn it off. !  set global general_log=1; !  Use Severalnines ClusterControl !  Includes a Cluster-wide Query Monitor. !  Query frequency, EXPLAINs, lock time etc. !  Performance Monitor and Manager.
  23. 23. Setup 23 Copyright 2011 Severalnines AB UID FID data 1 1 A 2 3 B 1 2 C 2 4 D table t1 Partition 0 Partition 1 NETWORK!
  24. 24. Sharding !  By default, all index scans hit all data nodes !  good if result set is big – you want as many CPUs as possible to help you. !  For smaller result sets (~a couple of hundred records) Partition Pruning is key for scalability. !  User-defined partitioning can help to improve equality index scans on part of a primary key. !  CREATE TABLE t1 (uid, fid, somedata, PRIMARY KEY(uid, fid)) PARTITION BY KEY(uid); !  All data belonging to a particular uid will be on the same partition. !  Great locality! !  select * from user where uid=1; !  Only one data node will be scanned (no matter how many nodes you have)
  25. 25. Sharding UID FID data 1 1 A 2 3 B 1 2 C 2 4 D table t1 UID FID data 1 1 A 1 2 C 2 3 B 2 4 D table t1 Partition 0 Partition 1 PARTITION BY KEY(UID)Default Partitioning
  26. 26. Sharding mysql> show global status like 'ndb_pruned_scan_count’; +-----------------------+-------+ | Variable_name | Value | +-----------------------+-------+ | Ndb_pruned_scan_count | 0 | +-----------------------+-------+ CREATE TABLE t1( … ) PARTITION BY KEY (userid); An run query, and verify it works: select * from user where uid=1; mysql> show global status like 'ndb_pruned_scan_count’; +-----------------------+-------+ | Variable_name | Value | +-----------------------+-------+ | Ndb_pruned_scan_count | 1 | +-----------------------+-------+
  27. 27. Sharding mysql> show global status like 'ndb%pruned%'; | Ndb_api_table_scan_count | 264 | | Ndb_api_range_scan_count | 18 | | Ndb_api_pruned_scan_count | 3 |
  28. 28. Sharding - EXAMPLE create table users_posts2 ( uid integer , fid integer , pid integer auto_increment, message varchar(1024), primary key(uid,fid, pid) ) engine=ndb partition by key(uid); NO PARTITIONING create table users_posts2 ( uid integer , fid integer , pid integer auto_increment, message varchar(1024), primary key(uid,fid, pid) ) engine=ndb; PARTITION BY KEY
  29. 29. Sharding – EXPLAIN PARTITIONS mysql> explain partitions select * from users_posts u where u.uid=1G id: 1 select_type: SIMPLE table: u partitions: p0,p1 type: ref possible_keys: PRIMARY key: PRIMARY key_len: 4 ref: const rows: 2699 Extra: NULL With PARTITION BY KEY (UID) mysql> explain partitions select * from users_posts2 u where u.uid=1G id: 1 select_type: SIMPLE table: u partitions: p0 type: ref possible_keys: PRIMARY key: PRIMARY key_len: 4 ref: const rows: 2699 Extra: NULL
  30. 30. Data Types BLOBs/TEXTs vs VARBINARY/VARCHAR !  BLOB/TEXT columns are stored in an external hidden table. !  First 256B are stored inline in main table !  Reading a BLOB/TEXT requires two reads !  One for reading the Main table + reading from hidden table !  Change to VARBINARY/VARCHAR if: !  Your BLOB/TEXTs can fit within an 14000 Bytes record !  (record size is currently 14000 Bytes) !  Reading/writing VARCHAR/VARBINARY is less expensive Note 1: BLOB/TEXT are also more expensive in Innodb as BLOB/TEXT data is not inlined with the table. Thus, two disk seeks are needed to read a BLOB. Note 2: Store images, movies etc outside the database on the filesystem.
  31. 31. Query Tuning !  MySQL Cluster 7.2 and later has pushed down joins ! joins are performed in the data nodes. !  OPTIMIZER in MySQL Cluster 7.1 and earlier is weak !  Statistics gathering is non-existing !  Optimizer thinks there are only 10 rows to examine in each table! !  FORCE INDEX / STRAIGHt_JOIN to get queries run the way you want
  32. 32. Query Tuning !  if you have two similar indexes: !  index(a) !  index(a,ts) on the following table CREATE TABLE `t1` ( `id` int(11) NOT NULL AUTO_INCREMENT, `a` bigint(20) DEFAULT NULL, `ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY (`id`), KEY `idx_t1_a` (`a`), KEY `idx_t1_a_ts` (`a`,`ts`)) ENGINE=ndbcluster DEFAULT CHARSET=latin1
  33. 33. Query Tuning mysql> select count(id) from t1 where a=5; +-----------+ | count(id) | +-----------+ | 3072000 | +-----------+ 1 row in set (0.02 sec) mysql> select count(id) from t1 where a=5 and ts>'2013-04-18 14:34:08’; +-----------+ | count(id) | +-----------+ | 512 | +-----------+ 1 row in set (0.00 sec)
  34. 34. Query Tuning Pre 7.2 mysql> explain select * from t1 where a=2 and ts='2011-10-05 15:32:11'; +----+-------------+-------+------+----------------------+----------+---------+-------+------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+------+----------------------+----------+---------+-------+------+-------------+ | 1 | SIMPLE | t1 | ref | idx_t1_a,idx_t1_a_ts | idx_t1_a | 9 | const | 10 | Using where | +----+-------------+-------+------+----------------------+----------+---------+-------+------+-------------+ !  Use FORCE INDEX(..) ... mysql> explain select * from t1 FORCE INDEX (idx_t1_a_ts) where a=2 and ts='2011-10-05 15:32:11; +| 1 | SIMPLE | t1 | ref | idx_t1_a_ts | idx_t1_a_ts | 13 | const,const | 10 | Using where | 1 row in set (0.00 sec) !  ..to ensure the correct index is picked! !  The difference can be 1 record read instead of any number of records!
  35. 35. Index Statistics explain select * from t1 where a=5 and ts>'2013-04-18 14:34:08' G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: t1 type: range possible_keys: idx_t1_a,idx_t1_a_ts key: idx_t1_a key_len: 9 ref: const Rows: 17 Extra: Using where with pushed condition
  36. 36. Index Statistics mysql> analyze table t1; +---------+---------+----------+----------+ | Table | Op | Msg_type | Msg_text | +---------+---------+----------+----------+ | test.t1 | analyze | status | OK | +---------+---------+----------+----------+ 1 row in set (3.40 sec)
  37. 37. Index Statistics Mysql> explain select * from t1 where a=5 and ts>'2013-04-18 14:34:08' G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: t1 type: range possible_keys: idx_t1_a,idx_t1_a_ts key: idx_t1_a_ts key_len: 13 ref: NULL rows: 253 Extra: Using where with pushed condition; Using MRR 1 row in set (0.00 sec)
  38. 38. Ndb_cluster_connection_pool !  Improved in MC 7.3, set ndb_cluster_connection_pool between 1 and 4. !  MySQL Cluster 7.2 and earlier: !  Problem: !  A Sendbuffer on the connection between mysqld and the data nodes is protected by a Mutex. !  Connection threads in MySQL must acquire Mutex and the put data in SendBuffer. !  Many threads gives more contention on the mutex !  Must scale out with many MySQL Servers. !  Workaround: !  Ndb_cluster_connection_pool (in my.cnf) creates more connections from one mysqld to the data nodes !  Threads load balance on the connections gives less contention on mutex which in turn gives increased scalabilty !  Less MySQL Servers needed to drive load! !  www.severalnines.com/cluster-configurator allows you to specify the connection pool.
  39. 39. Ndb_cluster_connection_pool !  Improved in MC 7.3, set ndb_cluster_connection_pool between 1 and 4. !  MySQL Cluster 7.2 and earlier: !  Gives at least 70% better performance and a MySQL Server that can scale beyond four database connections. !  Set Ndb_cluster_connection_pool=2x<CPU cores> !  It is a good starting point !  One free [mysqld] slot is required in config.ini for each Ndb_cluster_connection. !  4 mysql servers,each with Ndb_cluster_connection_pool=8 requires 32 [mysqld] in config.ini !  Note that also memcached and node.js, cluster/j etc also has the concept of the ndb_cluster_connection_pool.
  40. 40. Q&A 40 Copyright 2011 Severalnines AB
  41. 41. Resources !  MySQL Cluster Configurator !  www.severalnines.com/config !  MySQL Cluster Management + Monitoring !  www.severalnines.com/cmon !  MySQL Cluster Training Slides !  www.severalnines.com/mysql-cluster-training !  My Blog !  johanandersson.blogspot.com
  42. 42. Keep in touch… !  Facebook !  www.facebook.com/severalnines !  Twitter !  @severalnines !  Linked in: !  www.linkedin.com/company/severalnines
  43. 43. Thank you for your time! johan@severalnines.com 43 Copyright 2011 Severalnines AB

×