Your SlideShare is downloading. ×
  • Like
  • Save

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

MySQL Cluster Performance Tuning - 2013 MySQL User Conference


Slides from a presentation given at Percona Live MySQL Conference 2013 in Santa Clara, US. …

Slides from a presentation given at Percona Live MySQL Conference 2013 in Santa Clara, US.
Topics include:
- How to look for performance bottlenecks
- Foreign Key performance in MySQL Cluster 7.3
- Sharding and table partitioning
- efficient use of datatypes (e.g. BLOBS vs varbinary)

Published in Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • allow download pdf pls...
    Are you sure you want to
    Your message goes here
No Downloads


Total Views
On SlideShare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. Performance Tuning of MySQL ClusterApril 2013Johan AnderssonSeveralnines
  • 2. Agenda  7.3 Feature Update  OS Tuning  Stability Tuning  Application design  Identifying bottlenecks  Tuning tricks2Copyright 2011 Severalnines AB
  • 3. 7.3 Feature Update  Node.js Connector  JavaScript (V8 engine) to access data directly in theData nodes  No SQL – bypasses the MySQL Server  lower latency,high throughput for simple queries (like PK operations,simple scans from one table)3Copyright 2011 Severalnines AB
  • 4. 7.3 Feature Update  FOREIGN KEYs finally supported!  Implemented at the Data Node level  But..ERROR 1506 (HY000): Foreign key clause is not yetsupported in conjunction with partitioning  Hopefully fixed for the GA release  What about the performance penalty?4Copyright 2011 Severalnines AB
  • 5. 7.3 Foreign Key Perfcreate table users_posts (uid integer ,fid integer ,pid integer auto_increment,message varchar(1024),primary key(uid,fid, pid),constraint fk_forum foreign key(fid) references forum(fid) on delete cascade,constraint fk_user foreign key(uid) references users(uid) on delete cascade) engine=ndb;  Compare INSERT performance with and w/o FKs  With FK, must check that forum(fid) and users(uid) exists.  Populate with 1M records5Copyright 2011 Severalnines AB
  • 6. 7.3 Foreign Key Perf  Bencher drivers the load:  4 threads, 2 data nodes, 4 cores,  App  mysqld  data nodes  FOREIGN KEYs enabledSummary:--------------------------Average Throughput = 1274.58 tps (stdev=59.71)6Copyright 2011 Severalnines AB
  • 7. 7.3 Foreign Key Perf  Bencher drivers the load:  4 threads, 2 data nodes, 4 cores,  App  mysqld  data nodes  Not using FOREIGN KEYsSummary:--------------------------Average Throughput = 1428.57 tps (stdev=57.10)  Foreign keys gave ~11% drop in performance.7Copyright 2011 Severalnines AB
  • 8. 7.2  7.3 Caveats  Rolling upgrade from 7.2.10 to 7.3.1 works!  A little gotcha:  --engine-condition-pushdown  no longer supported inMySQL 5.6  Mysqld will fail to start  Take it out from my.cnf before upgrading!8Copyright 2011 Severalnines AB
  • 9. Facts  A single query will never run as fast as on Innodb(served from RAM)  Network latency is a issue  More data nodes does not speed up query executiontime.9Copyright 2011 Severalnines AB
  • 10. OS Tuning  Disable NUMA in /etc/grub.conf  echo ‘0’ > /proc/sys/vm/swappinessecho ‘vm.swappiness=0’ >> /etc/sysctl.conf  Bind data node threads to CPUs/cores  cat /proc/interrupts | grep ethcpu0 cpu1 cpu2 cpu3
44: 31 49432584 0 0 xen-dyn-event eth0"45: 1633715292 0 0 0 xen-dyn-event eth1"10Copyright 2011 Severalnines ABAvoid!OK!In config.ini [ndbd default]:ThreadConfig=ldm={count=1,cpubind=1,2},main={cpubind=3} ..
  • 11. Stability Tuning  Tuning the REDO log is key  FragmentLogFileSize=256M  NoOfFragmentLogFiles=<4-6> X DataMemory in MB / 4 xFragmentLogFileSize  RedoBuffer=64M for a write busy system  Disk based data:  SharedGlobalMemory=4096M  In the LOGFILE GROUP: undo_buffer_size=128M  Or higher (max is 600M)11Copyright 2011 Severalnines AB
  • 12. Stability Tuning  Make sure you don’t have more “execution threads” than cores  You want to have  Major page faults low  Involuntary context switches lowmysql> SELECT node_id, thr_no,thr_nm , os_ru_majflt,os_ru_nivcsw FROM threadstat;+---------+--------+--------+--------------+--------------+| node_id | thr_no | thr_nm | os_ru_majflt | os_ru_nivcsw |+---------+--------+--------+--------------+--------------+| 3 | 0 | main | 1 | 541719 || 4 | 0 | main | 0 | 561769 |+---------+--------+--------+--------------+--------------+2 rows in set (0.01 sec)12Copyright 2011 Severalnines AB
  • 13. Application Design  Define the most typical Use Cases  List all my friends, session management etc etc.  Optimize everything for the typical use case  Engineer schema to cater for the Use Cases  Keep it simple  Complex access patterns does not scale  Simple access patterns do ( Primay key and Partitioned Index Scans )  Note! There is no parameter in config.ini that affectsperformance – only availability.  Everything is about the Schema and the Queries.  Tune the mysql servers (sort buffers etc) as you would for innodb.13Copyright 2011 Severalnines AB
  • 14. Simple Access  PRIMARY KEY lookups are HASH lookup O(1)  INDEX searches a T-tree and takes O(log n) time.  In 7.2 and later JOINs are ok, but in 7.1 you should tryto avoid them.14Copyright 2011 Severalnines AB
  • 15. Identifying Bottlenecks  A lot of CPU is used on the data nodes  Probably a lot of large index scans and full table scans are used.  Check Slow query log or a query monitor  A lot of CPU is used on the mysql servers  Probably a lot of GROUP BY/DISTINCT or aggregate functions.  Hardly no CPU is used on either mysql or data nodes  Probably low load  Time is spent on network (a lot of “ping pong” to satisfy a request).  System is running slow in general  Disks (io util), queries, swap (must never happen), network
  • 16. Need To Add Data Nodes?  (adding mysql servers is easy)  top –Hd1  Is any of data nodes threads at 100%?  Yes: add more data nodes (online)  No: do nothing
  • 17. Detecting Query Problems  Here is a standard method for how to attack the problem.  Performance tuning is a never-ending loop:BEGIN–  Capture information – e.g, slow query log•  Change long_query_time if needed–  EXPLAIN the queries•  What indexes are used?•  Are tables JOINed in the correct order (small to big)–  Re-run the optimized typical use cases using bencher/mysqlslapGOTO BEGIN;END;  Never tune unless you can measure and test!  Dont optimize unless you have a problem!
  • 18. Enable Logging  Slow query log  set global slow_query_log=1;  set global long_query_time=0.01;  set global log_queries_not_using_indexes=1;  General log (if you don’t get enough info in the SlowQuery Log)  Activate for a very short period of time (30-60seconds) –intrusive  Can fill up disk very fast – make sure you turn it off.  set global general_log=1;  Use Severalnines ClusterControl  Includes a Cluster-wide Query Monitor.  Query frequency, EXPLAINs, lock time etc.  Performance Monitor and Manager.
  • 19. Setup19Copyright 2011 Severalnines ABsubid data1 A3 B2 C4 DsubscriberPartition 0Partition 1NETWORK!
  • 20. Sharding  By default, all index scans hit all data nodes  good if result set is big – you want as many CPUs as possible tohelp you.  For smaller result sets (~a couple of hundred records) PartitionPruning is key for scalability.  User-defined partitioning can help to improve equality indexscans on part of a primary key.  CREATE TABLE t1 (uid,fid,somedata,PRIMARY KEY(uid, fid))PARTITION BY KEY(userid);  All data belonging to a particular uid will be on the samepartition.  Great locality!  select * from user where uid=1;  Only one data node will be scanned (no matter how manynodes you have)
  • 21. Shardingmysql> show global status like ndb_pruned_scan_count’;+-----------------------+-------+| Variable_name | Value |+-----------------------+-------+| Ndb_pruned_scan_count | 0 |+-----------------------+-------+CREATE TABLE t1( … ) PARTITION BY KEY (userid);An run query, and verify it works:select * from user where userid=1;mysql> show global status like ndb_pruned_scan_count’;+-----------------------+-------+| Variable_name | Value |+-----------------------+-------+| Ndb_pruned_scan_count | 1 |+-----------------------+-------+
  • 22. Shardingmysql> show global status like ndb%pruned%;| Ndb_api_table_scan_count | 264 || Ndb_api_range_scan_count | 18 || Ndb_api_pruned_scan_count | 3 |
  • 23. Sharding - EXAMPLE  create table users_posts2 (uid integer ,fid integer ,pid integer auto_increment,message varchar(1024),primary key(uid,fid, pid)) engine=ndbpartition by key(uid);NO PARTITIONINGcreate table users_posts2 (uid integer ,fid integer ,pid integer auto_increment,message varchar(1024),primary key(uid,fid, pid)) engine=ndb;PARTITION BY KEY
  • 24. Sharding – EXPLAIN PARTITIONSmysql> explain partitions select * from users_posts u where u.uid=1Gid: 1select_type: SIMPLEtable: upartitions: p0,p1type: refpossible_keys: PRIMARYkey: PRIMARYkey_len: 4ref: constrows: 2699Extra: NULLWith PARTITION BY KEY (UID)mysql> explain partitions select * from users_posts2 u whereu.uid=1Gid: 1select_type: SIMPLEtable: upartitions: p0type: refpossible_keys: PRIMARYkey: PRIMARYkey_len: 4ref: constrows: 2699Extra: NULL
  • 25. Data TypesBLOBs/TEXTs vs VARBINARY/VARCHAR  BLOB/TEXT columns are stored in an external hidden table.  First 256B are stored inline in main table  Reading a BLOB/TEXT requires two reads  One for reading the Main table + reading from hiddentable  Change to VARBINARY/VARCHAR if:  Your BLOB/TEXTs can fit within an 14000 Bytes record  (record size is currently 14000 Bytes)  Reading/writing VARCHAR/VARBINARY is less expensiveNote 1: BLOB/TEXT are also more expensive in Innodb as BLOB/TEXT data isnot inlined with the table. Thus, two disk seeks are needed to read aBLOB.Note 2: Store images, movies etc outside the database on the filesystem.
  • 26. Query Tuning  MySQL Cluster 7.2 and later has pushed down joins  joinsare performed in the data nodes.  OPTIMIZER in MySQL Cluster 7.1 and earlier is weak  Statistics gathering is non-existing  Optimizer thinks there are only 10 rows to examine in eachtable!  FORCE INDEX / STRAIGHt_JOIN to get queries run the way youwant
  • 27. Query Tuning  if you have two similar indexes:  index(a)  index(a,ts)on the following tableCREATE TABLE `t1` (`id` int(11) NOT NULL AUTO_INCREMENT,`a` bigint(20) DEFAULT NULL,`ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,PRIMARY KEY (`id`),KEY `idx_t1_a` (`a`),KEY `idx_t1_a_ts` (`a`,`ts`)) ENGINE=ndbcluster DEFAULT CHARSET=latin1
  • 28. Query Tuningmysql> select count(id) from t1 where a=5;+-----------+| count(id) |+-----------+| 3072000 |+-----------+1 row in set (0.02 sec)mysql> select count(id) from t1 where a=5and ts>2013-04-18 14:34:08’;+-----------+| count(id) |+-----------+| 512 |+-----------+1 row in set (0.00 sec)
  • 29. Query Tuning Pre 7.2mysql> explain select * from t1 where a=2 and ts=2011-10-05 15:32:11;+----+-------------+-------+------+----------------------+----------+---------+-------+------+-------------+| id | select_type | table | type | possible_keys | key | key_len | ref |rows | Extra |+----+-------------+-------+------+----------------------+----------+---------+-------+------+-------------+| 1 | SIMPLE | t1 | ref | idx_t1_a,idx_t1_a_ts | idx_t1_a | 9 | const | 10 |Using where |+----+-------------+-------+------+----------------------+----------+---------+-------+------+-------------+  Use FORCE INDEX(..) ...mysql> explain select * from t1 FORCE INDEX (idx_t1_a_ts) where a=2 and ts=2011-10-0515:32:11;+| 1 | SIMPLE | t1 | ref | idx_t1_a_ts | idx_t1_a_ts | 13 | const,const | 10 |Using where |1 row in set (0.00 sec) ensure the correct index is picked!  The difference can be 1 record read instead of anynumber of records!
  • 30. Index Statisticsexplain select * from t1 where a=5 and ts>2013-04-18 14:34:08 G*************************** 1. row ***************************id: 1select_type: SIMPLEtable: t1type: rangepossible_keys: idx_t1_a,idx_t1_a_tskey: idx_t1_akey_len: 9ref: constRows: 17Extra: Using where with pushed condition
  • 31. Index Statisticsmysql> analyze table t1;+---------+---------+----------+----------+| Table | Op | Msg_type | Msg_text |+---------+---------+----------+----------+| test.t1 | analyze | status | OK |+---------+---------+----------+----------+1 row in set (3.40 sec)
  • 32. Index StatisticsMysql> explain select * from t1 where a=5and ts>2013-04-18 14:34:08 G*************************** 1. row ***************************id: 1select_type: SIMPLEtable: t1type: rangepossible_keys: idx_t1_a,idx_t1_a_tskey: idx_t1_a_tskey_len: 13ref: NULLrows: 253Extra: Using where with pushed condition; Using MRR1 row in set (0.00 sec)
  • 33. Ndb_cluster_connection_pool  Problem:  A Sendbuffer on the connection between mysqld and thedata nodes is protected by a Mutex.  Connection threads in MySQL must acquire Mutex and theput data in SendBuffer.  Many threads gives more contention on the mutex  Must scale out with many MySQL Servers.  Workaround:  Ndb_cluster_connection_pool (in my.cnf) creates moreconnections from one mysqld to the data nodes  Threads load balance on the connections gives lesscontention on mutex which in turn gives increased scalabilty  Less MySQL Servers needed to drive load! allows you tospecify the connection pool.
  • 34. Ndb_cluster_connection_pool  Gives atleast 70% better performance and a MySQL Serverthat can scale beyond four database connections.  Set Ndb_cluster_connection_pool=2x<CPU cores>  It is a good starting point  One free [mysqld] slot is required in config.ini for eachNdb_cluster_connection.  4 mysql servers,each with Ndb_cluster_connection_pool=8requires 32 [mysqld] in config.ini  Note that also memcached and node.js, cluster/j etc also hasthe concept of the ndb_cluster_connection_pool.
  • 35. Q&A35Copyright 2011 Severalnines AB
  • 36. Resources  MySQL Cluster Configurator  MySQL Cluster Management + Monitoring  MySQL Cluster Training Slides  My Blog
  • 37. Keep in touch…  Facebook  Twitter  @severalnines  Linked in:
  • 38. Thank you for your time!johan@severalnines.com38Copyright 2011 Severalnines AB