MySQL Cluster Performance Tuning - 2013 MySQL User Conference


Published on

Slides from a presentation given at Percona Live MySQL Conference 2013 in Santa Clara, US.
Topics include:
- How to look for performance bottlenecks
- Foreign Key performance in MySQL Cluster 7.3
- Sharding and table partitioning
- efficient use of datatypes (e.g. BLOBS vs varbinary)

Published in: Technology
1 Comment
1 Like
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

MySQL Cluster Performance Tuning - 2013 MySQL User Conference

  1. 1. Performance Tuning of MySQL ClusterApril 2013Johan AnderssonSeveralnines
  2. 2. Agenda  7.3 Feature Update  OS Tuning  Stability Tuning  Application design  Identifying bottlenecks  Tuning tricks2Copyright 2011 Severalnines AB
  3. 3. 7.3 Feature Update  Node.js Connector  JavaScript (V8 engine) to access data directly in theData nodes  No SQL – bypasses the MySQL Server  lower latency,high throughput for simple queries (like PK operations,simple scans from one table)3Copyright 2011 Severalnines AB
  4. 4. 7.3 Feature Update  FOREIGN KEYs finally supported!  Implemented at the Data Node level  But..ERROR 1506 (HY000): Foreign key clause is not yetsupported in conjunction with partitioning  Hopefully fixed for the GA release  What about the performance penalty?4Copyright 2011 Severalnines AB
  5. 5. 7.3 Foreign Key Perfcreate table users_posts (uid integer ,fid integer ,pid integer auto_increment,message varchar(1024),primary key(uid,fid, pid),constraint fk_forum foreign key(fid) references forum(fid) on delete cascade,constraint fk_user foreign key(uid) references users(uid) on delete cascade) engine=ndb;  Compare INSERT performance with and w/o FKs  With FK, must check that forum(fid) and users(uid) exists.  Populate with 1M records5Copyright 2011 Severalnines AB
  6. 6. 7.3 Foreign Key Perf  Bencher drivers the load:  4 threads, 2 data nodes, 4 cores,  App  mysqld  data nodes  FOREIGN KEYs enabledSummary:--------------------------Average Throughput = 1274.58 tps (stdev=59.71)6Copyright 2011 Severalnines AB
  7. 7. 7.3 Foreign Key Perf  Bencher drivers the load:  4 threads, 2 data nodes, 4 cores,  App  mysqld  data nodes  Not using FOREIGN KEYsSummary:--------------------------Average Throughput = 1428.57 tps (stdev=57.10)  Foreign keys gave ~11% drop in performance.7Copyright 2011 Severalnines AB
  8. 8. 7.2  7.3 Caveats  Rolling upgrade from 7.2.10 to 7.3.1 works!  A little gotcha:  --engine-condition-pushdown  no longer supported inMySQL 5.6  Mysqld will fail to start  Take it out from my.cnf before upgrading!8Copyright 2011 Severalnines AB
  9. 9. Facts  A single query will never run as fast as on Innodb(served from RAM)  Network latency is a issue  More data nodes does not speed up query executiontime.9Copyright 2011 Severalnines AB
  10. 10. OS Tuning  Disable NUMA in /etc/grub.conf  echo ‘0’ > /proc/sys/vm/swappinessecho ‘vm.swappiness=0’ >> /etc/sysctl.conf  Bind data node threads to CPUs/cores  cat /proc/interrupts | grep ethcpu0 cpu1 cpu2 cpu3
44: 31 49432584 0 0 xen-dyn-event eth0"45: 1633715292 0 0 0 xen-dyn-event eth1"10Copyright 2011 Severalnines ABAvoid!OK!In config.ini [ndbd default]:ThreadConfig=ldm={count=1,cpubind=1,2},main={cpubind=3} ..
  11. 11. Stability Tuning  Tuning the REDO log is key  FragmentLogFileSize=256M  NoOfFragmentLogFiles=<4-6> X DataMemory in MB / 4 xFragmentLogFileSize  RedoBuffer=64M for a write busy system  Disk based data:  SharedGlobalMemory=4096M  In the LOGFILE GROUP: undo_buffer_size=128M  Or higher (max is 600M)11Copyright 2011 Severalnines AB
  12. 12. Stability Tuning  Make sure you don’t have more “execution threads” than cores  You want to have  Major page faults low  Involuntary context switches lowmysql> SELECT node_id, thr_no,thr_nm , os_ru_majflt,os_ru_nivcsw FROM threadstat;+---------+--------+--------+--------------+--------------+| node_id | thr_no | thr_nm | os_ru_majflt | os_ru_nivcsw |+---------+--------+--------+--------------+--------------+| 3 | 0 | main | 1 | 541719 || 4 | 0 | main | 0 | 561769 |+---------+--------+--------+--------------+--------------+2 rows in set (0.01 sec)12Copyright 2011 Severalnines AB
  13. 13. Application Design  Define the most typical Use Cases  List all my friends, session management etc etc.  Optimize everything for the typical use case  Engineer schema to cater for the Use Cases  Keep it simple  Complex access patterns does not scale  Simple access patterns do ( Primay key and Partitioned Index Scans )  Note! There is no parameter in config.ini that affectsperformance – only availability.  Everything is about the Schema and the Queries.  Tune the mysql servers (sort buffers etc) as you would for innodb.13Copyright 2011 Severalnines AB
  14. 14. Simple Access  PRIMARY KEY lookups are HASH lookup O(1)  INDEX searches a T-tree and takes O(log n) time.  In 7.2 and later JOINs are ok, but in 7.1 you should tryto avoid them.14Copyright 2011 Severalnines AB
  15. 15. Identifying Bottlenecks  A lot of CPU is used on the data nodes  Probably a lot of large index scans and full table scans are used.  Check Slow query log or a query monitor  A lot of CPU is used on the mysql servers  Probably a lot of GROUP BY/DISTINCT or aggregate functions.  Hardly no CPU is used on either mysql or data nodes  Probably low load  Time is spent on network (a lot of “ping pong” to satisfy a request).  System is running slow in general  Disks (io util), queries, swap (must never happen), network
  16. 16. Need To Add Data Nodes?  (adding mysql servers is easy)  top –Hd1  Is any of data nodes threads at 100%?  Yes: add more data nodes (online)  No: do nothing
  17. 17. Detecting Query Problems  Here is a standard method for how to attack the problem.  Performance tuning is a never-ending loop:BEGIN–  Capture information – e.g, slow query log•  Change long_query_time if needed–  EXPLAIN the queries•  What indexes are used?•  Are tables JOINed in the correct order (small to big)–  Re-run the optimized typical use cases using bencher/mysqlslapGOTO BEGIN;END;  Never tune unless you can measure and test!  Dont optimize unless you have a problem!
  18. 18. Enable Logging  Slow query log  set global slow_query_log=1;  set global long_query_time=0.01;  set global log_queries_not_using_indexes=1;  General log (if you don’t get enough info in the SlowQuery Log)  Activate for a very short period of time (30-60seconds) –intrusive  Can fill up disk very fast – make sure you turn it off.  set global general_log=1;  Use Severalnines ClusterControl  Includes a Cluster-wide Query Monitor.  Query frequency, EXPLAINs, lock time etc.  Performance Monitor and Manager.
  19. 19. Setup19Copyright 2011 Severalnines ABsubid data1 A3 B2 C4 DsubscriberPartition 0Partition 1NETWORK!
  20. 20. Sharding  By default, all index scans hit all data nodes  good if result set is big – you want as many CPUs as possible tohelp you.  For smaller result sets (~a couple of hundred records) PartitionPruning is key for scalability.  User-defined partitioning can help to improve equality indexscans on part of a primary key.  CREATE TABLE t1 (uid,fid,somedata,PRIMARY KEY(uid, fid))PARTITION BY KEY(userid);  All data belonging to a particular uid will be on the samepartition.  Great locality!  select * from user where uid=1;  Only one data node will be scanned (no matter how manynodes you have)
  21. 21. Shardingmysql> show global status like ndb_pruned_scan_count’;+-----------------------+-------+| Variable_name | Value |+-----------------------+-------+| Ndb_pruned_scan_count | 0 |+-----------------------+-------+CREATE TABLE t1( … ) PARTITION BY KEY (userid);An run query, and verify it works:select * from user where userid=1;mysql> show global status like ndb_pruned_scan_count’;+-----------------------+-------+| Variable_name | Value |+-----------------------+-------+| Ndb_pruned_scan_count | 1 |+-----------------------+-------+
  22. 22. Shardingmysql> show global status like ndb%pruned%;| Ndb_api_table_scan_count | 264 || Ndb_api_range_scan_count | 18 || Ndb_api_pruned_scan_count | 3 |
  23. 23. Sharding - EXAMPLE  create table users_posts2 (uid integer ,fid integer ,pid integer auto_increment,message varchar(1024),primary key(uid,fid, pid)) engine=ndbpartition by key(uid);NO PARTITIONINGcreate table users_posts2 (uid integer ,fid integer ,pid integer auto_increment,message varchar(1024),primary key(uid,fid, pid)) engine=ndb;PARTITION BY KEY
  24. 24. Sharding – EXPLAIN PARTITIONSmysql> explain partitions select * from users_posts u where u.uid=1Gid: 1select_type: SIMPLEtable: upartitions: p0,p1type: refpossible_keys: PRIMARYkey: PRIMARYkey_len: 4ref: constrows: 2699Extra: NULLWith PARTITION BY KEY (UID)mysql> explain partitions select * from users_posts2 u whereu.uid=1Gid: 1select_type: SIMPLEtable: upartitions: p0type: refpossible_keys: PRIMARYkey: PRIMARYkey_len: 4ref: constrows: 2699Extra: NULL
  25. 25. Data TypesBLOBs/TEXTs vs VARBINARY/VARCHAR  BLOB/TEXT columns are stored in an external hidden table.  First 256B are stored inline in main table  Reading a BLOB/TEXT requires two reads  One for reading the Main table + reading from hiddentable  Change to VARBINARY/VARCHAR if:  Your BLOB/TEXTs can fit within an 14000 Bytes record  (record size is currently 14000 Bytes)  Reading/writing VARCHAR/VARBINARY is less expensiveNote 1: BLOB/TEXT are also more expensive in Innodb as BLOB/TEXT data isnot inlined with the table. Thus, two disk seeks are needed to read aBLOB.Note 2: Store images, movies etc outside the database on the filesystem.
  26. 26. Query Tuning  MySQL Cluster 7.2 and later has pushed down joins  joinsare performed in the data nodes.  OPTIMIZER in MySQL Cluster 7.1 and earlier is weak  Statistics gathering is non-existing  Optimizer thinks there are only 10 rows to examine in eachtable!  FORCE INDEX / STRAIGHt_JOIN to get queries run the way youwant
  27. 27. Query Tuning  if you have two similar indexes:  index(a)  index(a,ts)on the following tableCREATE TABLE `t1` (`id` int(11) NOT NULL AUTO_INCREMENT,`a` bigint(20) DEFAULT NULL,`ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,PRIMARY KEY (`id`),KEY `idx_t1_a` (`a`),KEY `idx_t1_a_ts` (`a`,`ts`)) ENGINE=ndbcluster DEFAULT CHARSET=latin1
  28. 28. Query Tuningmysql> select count(id) from t1 where a=5;+-----------+| count(id) |+-----------+| 3072000 |+-----------+1 row in set (0.02 sec)mysql> select count(id) from t1 where a=5and ts>2013-04-18 14:34:08’;+-----------+| count(id) |+-----------+| 512 |+-----------+1 row in set (0.00 sec)
  29. 29. Query Tuning Pre 7.2mysql> explain select * from t1 where a=2 and ts=2011-10-05 15:32:11;+----+-------------+-------+------+----------------------+----------+---------+-------+------+-------------+| id | select_type | table | type | possible_keys | key | key_len | ref |rows | Extra |+----+-------------+-------+------+----------------------+----------+---------+-------+------+-------------+| 1 | SIMPLE | t1 | ref | idx_t1_a,idx_t1_a_ts | idx_t1_a | 9 | const | 10 |Using where |+----+-------------+-------+------+----------------------+----------+---------+-------+------+-------------+  Use FORCE INDEX(..) ...mysql> explain select * from t1 FORCE INDEX (idx_t1_a_ts) where a=2 and ts=2011-10-0515:32:11;+| 1 | SIMPLE | t1 | ref | idx_t1_a_ts | idx_t1_a_ts | 13 | const,const | 10 |Using where |1 row in set (0.00 sec) ensure the correct index is picked!  The difference can be 1 record read instead of anynumber of records!
  30. 30. Index Statisticsexplain select * from t1 where a=5 and ts>2013-04-18 14:34:08 G*************************** 1. row ***************************id: 1select_type: SIMPLEtable: t1type: rangepossible_keys: idx_t1_a,idx_t1_a_tskey: idx_t1_akey_len: 9ref: constRows: 17Extra: Using where with pushed condition
  31. 31. Index Statisticsmysql> analyze table t1;+---------+---------+----------+----------+| Table | Op | Msg_type | Msg_text |+---------+---------+----------+----------+| test.t1 | analyze | status | OK |+---------+---------+----------+----------+1 row in set (3.40 sec)
  32. 32. Index StatisticsMysql> explain select * from t1 where a=5and ts>2013-04-18 14:34:08 G*************************** 1. row ***************************id: 1select_type: SIMPLEtable: t1type: rangepossible_keys: idx_t1_a,idx_t1_a_tskey: idx_t1_a_tskey_len: 13ref: NULLrows: 253Extra: Using where with pushed condition; Using MRR1 row in set (0.00 sec)
  33. 33. Ndb_cluster_connection_pool  Problem:  A Sendbuffer on the connection between mysqld and thedata nodes is protected by a Mutex.  Connection threads in MySQL must acquire Mutex and theput data in SendBuffer.  Many threads gives more contention on the mutex  Must scale out with many MySQL Servers.  Workaround:  Ndb_cluster_connection_pool (in my.cnf) creates moreconnections from one mysqld to the data nodes  Threads load balance on the connections gives lesscontention on mutex which in turn gives increased scalabilty  Less MySQL Servers needed to drive load! allows you tospecify the connection pool.
  34. 34. Ndb_cluster_connection_pool  Gives atleast 70% better performance and a MySQL Serverthat can scale beyond four database connections.  Set Ndb_cluster_connection_pool=2x<CPU cores>  It is a good starting point  One free [mysqld] slot is required in config.ini for eachNdb_cluster_connection.  4 mysql servers,each with Ndb_cluster_connection_pool=8requires 32 [mysqld] in config.ini  Note that also memcached and node.js, cluster/j etc also hasthe concept of the ndb_cluster_connection_pool.
  35. 35. Q&A35Copyright 2011 Severalnines AB
  36. 36. Resources  MySQL Cluster Configurator  MySQL Cluster Management + Monitoring  MySQL Cluster Training Slides  My Blog
  37. 37. Keep in touch…  Facebook  Twitter  @severalnines  Linked in:
  38. 38. Thank you for your time!johan@severalnines.com38Copyright 2011 Severalnines AB