• Save
MySQL Cluster 7.3 Performance Tuning - Severalnines Slides

Like this? Share it with your network

Share

MySQL Cluster 7.3 Performance Tuning - Severalnines Slides

  • 524 views
Uploaded on

The MySQL Cluster 7.x series introduced a number of features to allow for fine-grained control over the real-time behaviour of the NDB storage engine. New threads have been introduced, and users......

The MySQL Cluster 7.x series introduced a number of features to allow for fine-grained control over the real-time behaviour of the NDB storage engine. New threads have been introduced, and users are able to control placement of these threads, as well as locking the memory such that no swapping occurs. In an ideal run-time environment, CPUs handling data node threads will not execute other threads apart from OS kernel threads or interrupt handling. Correct tuning of certain parameters can be specially important for certain types of workloads.

This presentation covers the different tuning aspects of MySQL Cluster.

- Application design guidelines
- Schema Optimization
- Index Selection and Tuning
- Query Tuning
- OS Tuning
- Data Node internals
- Optimizations for real-time behaviour

This presentation looks closely at how to get the most out of your MySQL Cluster 7.x runtime environment.

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
524
On Slideshare
524
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
0
Comments
0
Likes
2

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Performance Tuning of MySQL Cluster April 2014 Johan Andersson Severalnines AB johan@severalnines.com
  • 2. Agenda !  7.4 – What’s coming? !  7.3 Feature Update !  OS Tuning !  Stability Tuning !  Application design !  Identifying bottlenecks !  Tuning tricks 2 Copyright 2011 Severalnines AB
  • 3. 7.4 - what is coming !  Optimized Scans !  Overload/Overcommitting Resource protection !  Improved Checkpointing !  10% more performance 3 Copyright 2011 Severalnines AB
  • 4. 7.3 Feature Update !  Node.js Connector !  JavaScript (V8 engine) to access data directly in the Data nodes !  No SQL – bypasses the MySQL Server ! lower latency, high throughput for simple queries (like PK operations, simple scans from one table) 4 Copyright 2011 Severalnines AB
  • 5. 7.3 Feature Update !  FOREIGN KEYs finally supported! !  Implemented at the Data Node level !  But.. ERROR 1506 (HY000): Foreign key clause is not yet supported in conjunction with partitioning !  Hopefully fixed for the GA release !  What about the performance penalty? 5 Copyright 2011 Severalnines AB
  • 6. 7.3 Foreign Key Perf create table users_posts ( uid integer , fid integer , pid integer auto_increment, message varchar(1024), primary key(uid,fid, pid), constraint fk_forum foreign key(fid) references forum(fid) on delete cascade, constraint fk_user foreign key(uid) references users(uid) on delete cascade ) engine=ndb; !  Compare INSERT performance with and w/o FKs !  With FK, must check that forum(fid) and users(uid) exists. !  Populate with 1M records 6 Copyright 2011 Severalnines AB
  • 7. 7.3 Foreign Key Perf !  Bencher drivers the load: !  https://github.com/severalnines/bencher !  4 threads, 2 data nodes, 4 cores, !  App ! mysqld ! data nodes !  FOREIGN KEYs enabled Summary: -------------------------- Average Throughput = 1274.58 tps (stdev=59.71) 7 Copyright 2011 Severalnines AB
  • 8. 7.3 Foreign Key Perf !  Bencher drivers the load: !  https://github.com/severalnines/bencher !  4 threads, 2 data nodes, 4 cores, !  App ! mysqld ! data nodes !  Not using FOREIGN KEYs Summary: -------------------------- Average Throughput = 1428.57 tps (stdev=57.10) !  Foreign keys gave ~11% drop in performance. 8 Copyright 2011 Severalnines AB
  • 9. 7.2 ! 7.3 Caveats !  A little gotcha: !  --engine-condition-pushdown ! no longer supported in MySQL 5.6 !  Mysqld will fail to start !  Take it out from my.cnf before upgrading! 9 Copyright 2011 Severalnines AB
  • 10. Facts !  A single query will never run as fast as on Innodb (served from RAM) !  Network latency is a issue !  More data nodes does not speed up query execution time. 10 Copyright 2011 Severalnines AB
  • 11. OS Tuning !  Disable NUMA in /etc/grub.conf !  echo ‘0’ > /proc/sys/vm/swappiness echo ‘vm.swappiness=0’ >> /etc/sysctl.conf !  Bind data node threads to CPUs/cores !  cat /proc/interrupts | grep eth cpu0 cpu1 cpu2 cpu3
 44: 31 49432584 0 0 xen-dyn-event eth0" 45: 1633715292 0 0 0 xen-dyn-event eth1" 11 Copyright 2011 Severalnines AB Avoid! OK! In config.ini [ndbd default]: ThreadConfig=ldm={count=1,cpubind=1,2},main={cpubind=3} ..
  • 12. Perf Config Tuning (1) !  How many Local Data Managers ? !  How many TC threads? Copyright 2011 Severalnines AB 12
  • 13. Perf Config Tuning (2) Copyright 2011 Severalnines AB 13 3 4 5 9 10 11 IRQBALANCE_BANNED_CPUS IRQ CPU0 CPU1
  • 14. Perf Config Tuning (3) !  Run representative load !  Does any NDB Thread run hot? !  Increase with threads of that type !  Enable HyperThreading !  Can give 40% more !  Enable: !  RealTimeScheduler=1 !  Don’t forget: !  NoOfFragmentLogParts=<no of LDMs> Copyright 2011 Severalnines AB 14
  • 15. Stability Tuning !  Tuning the REDO log is key !  FragmentLogFileSize=256M !  NoOfFragmentLogFiles=<4-6> X DataMemory in MB / 4 x FragmentLogFileSize !  RedoBuffer=64M for a write busy system !  Disk based data: !  SharedGlobalMemory=4096M !  In the LOGFILE GROUP: undo_buffer_size=128M !  Or higher (max is 600M) 15 Copyright 2011 Severalnines AB
  • 16. Stability Tuning !  Make sure you don’t have more “execution threads” than cores !  You want to have !  Major page faults low !  Involuntary context switches low mysql> SELECT node_id, thr_no,thr_nm , os_ru_majflt, os_ru_nivcsw FROM threadstat; +---------+--------+--------+--------------+--------------+ | node_id | thr_no | thr_nm | os_ru_majflt | os_ru_nivcsw | +---------+--------+--------+--------------+--------------+ | 3 | 0 | main | 1 | 541719 | | 4 | 0 | main | 0 | 561769 | +---------+--------+--------+--------------+--------------+ 2 rows in set (0.01 sec) 16 Copyright 2011 Severalnines AB
  • 17. Application Design !  Define the most typical Use Cases !  List all my friends, session management etc etc. !  Optimize everything for the typical use case !  Engineer schema to cater for the Use Cases !  Keep it simple !  Complex access patterns does not scale !  Simple access patterns do ( Primay key and Partitioned Index Scans ) !  Note! There is no parameter in config.ini that affects performance – only availability. !  Everything is about the Schema and the Queries. !  Tune the mysql servers (sort buffers etc) as you would for innodb. 17 Copyright 2011 Severalnines AB
  • 18. Simple Access !  PRIMARY KEY lookups are HASH lookup O(1) !  INDEX searches a T-tree and takes O(log n) time. !  In 7.2 and later JOINs are ok, but in 7.1 you should try to avoid them. 18 Copyright 2011 Severalnines AB
  • 19. Identifying Bottlenecks !  High CPU usage on data nodes !  Probably a lot of large index scans and full table scans are used. !  Check Slow query log or a query monitor !  High CPU usage on MySQL servers !  Probably a lot of GROUP BY/DISTINCT or aggregate functions. !  Hardly no CPU is used on either MySQL or data nodes !  Probably low load !  Time is spent on network (a lot of “ping pong” to satisfy a request). !  System is running slow in general !  Disks (io util), queries, swap (must never happen), network
  • 20. Need To Add Data Nodes? !  (adding MySQL servers is easy) !  top –Hd1 !  Is any of data nodes threads at 100%? !  Yes: add more data nodes (online) !  No: do nothing
  • 21. Detecting Query Problems !  Here is a standard method for how to attack the problem. !  Performance tuning is a never-ending loop: BEGIN –  Capture information – e.g, slow query log •  Change long_query_time if needed –  EXPLAIN the queries •  What indexes are used? •  Are tables JOINed in the correct order (small to big) –  Re-run the optimized typical use cases using bencher/ mysqlslap GOTO BEGIN; END; !  Never tune unless you can measure and test! !  Don't optimize unless you have a problem!
  • 22. Enable Logging !  Slow query log !  set global slow_query_log=1; !  set global long_query_time=0.01; !  set global log_queries_not_using_indexes=1; !  General log (if you don’t get enough info in the Slow Query Log) !  Activate for a very short period of time (30-60seconds) – intrusive !  Can fill up disk very fast – make sure you turn it off. !  set global general_log=1; !  Use Severalnines ClusterControl !  Includes a Cluster-wide Query Monitor. !  Query frequency, EXPLAINs, lock time etc. !  Performance Monitor and Manager.
  • 23. Setup 23 Copyright 2011 Severalnines AB UID FID data 1 1 A 2 3 B 1 2 C 2 4 D table t1 Partition 0 Partition 1 NETWORK!
  • 24. Sharding !  By default, all index scans hit all data nodes !  good if result set is big – you want as many CPUs as possible to help you. !  For smaller result sets (~a couple of hundred records) Partition Pruning is key for scalability. !  User-defined partitioning can help to improve equality index scans on part of a primary key. !  CREATE TABLE t1 (uid, fid, somedata, PRIMARY KEY(uid, fid)) PARTITION BY KEY(uid); !  All data belonging to a particular uid will be on the same partition. !  Great locality! !  select * from user where uid=1; !  Only one data node will be scanned (no matter how many nodes you have)
  • 25. Sharding UID FID data 1 1 A 2 3 B 1 2 C 2 4 D table t1 UID FID data 1 1 A 1 2 C 2 3 B 2 4 D table t1 Partition 0 Partition 1 PARTITION BY KEY(UID)Default Partitioning
  • 26. Sharding mysql> show global status like 'ndb_pruned_scan_count’; +-----------------------+-------+ | Variable_name | Value | +-----------------------+-------+ | Ndb_pruned_scan_count | 0 | +-----------------------+-------+ CREATE TABLE t1( … ) PARTITION BY KEY (userid); An run query, and verify it works: select * from user where uid=1; mysql> show global status like 'ndb_pruned_scan_count’; +-----------------------+-------+ | Variable_name | Value | +-----------------------+-------+ | Ndb_pruned_scan_count | 1 | +-----------------------+-------+
  • 27. Sharding mysql> show global status like 'ndb%pruned%'; | Ndb_api_table_scan_count | 264 | | Ndb_api_range_scan_count | 18 | | Ndb_api_pruned_scan_count | 3 |
  • 28. Sharding - EXAMPLE create table users_posts2 ( uid integer , fid integer , pid integer auto_increment, message varchar(1024), primary key(uid,fid, pid) ) engine=ndb partition by key(uid); NO PARTITIONING create table users_posts2 ( uid integer , fid integer , pid integer auto_increment, message varchar(1024), primary key(uid,fid, pid) ) engine=ndb; PARTITION BY KEY
  • 29. Sharding – EXPLAIN PARTITIONS mysql> explain partitions select * from users_posts u where u.uid=1G id: 1 select_type: SIMPLE table: u partitions: p0,p1 type: ref possible_keys: PRIMARY key: PRIMARY key_len: 4 ref: const rows: 2699 Extra: NULL With PARTITION BY KEY (UID) mysql> explain partitions select * from users_posts2 u where u.uid=1G id: 1 select_type: SIMPLE table: u partitions: p0 type: ref possible_keys: PRIMARY key: PRIMARY key_len: 4 ref: const rows: 2699 Extra: NULL
  • 30. Data Types BLOBs/TEXTs vs VARBINARY/VARCHAR !  BLOB/TEXT columns are stored in an external hidden table. !  First 256B are stored inline in main table !  Reading a BLOB/TEXT requires two reads !  One for reading the Main table + reading from hidden table !  Change to VARBINARY/VARCHAR if: !  Your BLOB/TEXTs can fit within an 14000 Bytes record !  (record size is currently 14000 Bytes) !  Reading/writing VARCHAR/VARBINARY is less expensive Note 1: BLOB/TEXT are also more expensive in Innodb as BLOB/TEXT data is not inlined with the table. Thus, two disk seeks are needed to read a BLOB. Note 2: Store images, movies etc outside the database on the filesystem.
  • 31. Query Tuning !  MySQL Cluster 7.2 and later has pushed down joins ! joins are performed in the data nodes. !  OPTIMIZER in MySQL Cluster 7.1 and earlier is weak !  Statistics gathering is non-existing !  Optimizer thinks there are only 10 rows to examine in each table! !  FORCE INDEX / STRAIGHt_JOIN to get queries run the way you want
  • 32. Query Tuning !  if you have two similar indexes: !  index(a) !  index(a,ts) on the following table CREATE TABLE `t1` ( `id` int(11) NOT NULL AUTO_INCREMENT, `a` bigint(20) DEFAULT NULL, `ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY (`id`), KEY `idx_t1_a` (`a`), KEY `idx_t1_a_ts` (`a`,`ts`)) ENGINE=ndbcluster DEFAULT CHARSET=latin1
  • 33. Query Tuning mysql> select count(id) from t1 where a=5; +-----------+ | count(id) | +-----------+ | 3072000 | +-----------+ 1 row in set (0.02 sec) mysql> select count(id) from t1 where a=5 and ts>'2013-04-18 14:34:08’; +-----------+ | count(id) | +-----------+ | 512 | +-----------+ 1 row in set (0.00 sec)
  • 34. Query Tuning Pre 7.2 mysql> explain select * from t1 where a=2 and ts='2011-10-05 15:32:11'; +----+-------------+-------+------+----------------------+----------+---------+-------+------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+------+----------------------+----------+---------+-------+------+-------------+ | 1 | SIMPLE | t1 | ref | idx_t1_a,idx_t1_a_ts | idx_t1_a | 9 | const | 10 | Using where | +----+-------------+-------+------+----------------------+----------+---------+-------+------+-------------+ !  Use FORCE INDEX(..) ... mysql> explain select * from t1 FORCE INDEX (idx_t1_a_ts) where a=2 and ts='2011-10-05 15:32:11; +| 1 | SIMPLE | t1 | ref | idx_t1_a_ts | idx_t1_a_ts | 13 | const,const | 10 | Using where | 1 row in set (0.00 sec) !  ..to ensure the correct index is picked! !  The difference can be 1 record read instead of any number of records!
  • 35. Index Statistics explain select * from t1 where a=5 and ts>'2013-04-18 14:34:08' G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: t1 type: range possible_keys: idx_t1_a,idx_t1_a_ts key: idx_t1_a key_len: 9 ref: const Rows: 17 Extra: Using where with pushed condition
  • 36. Index Statistics mysql> analyze table t1; +---------+---------+----------+----------+ | Table | Op | Msg_type | Msg_text | +---------+---------+----------+----------+ | test.t1 | analyze | status | OK | +---------+---------+----------+----------+ 1 row in set (3.40 sec)
  • 37. Index Statistics Mysql> explain select * from t1 where a=5 and ts>'2013-04-18 14:34:08' G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: t1 type: range possible_keys: idx_t1_a,idx_t1_a_ts key: idx_t1_a_ts key_len: 13 ref: NULL rows: 253 Extra: Using where with pushed condition; Using MRR 1 row in set (0.00 sec)
  • 38. Ndb_cluster_connection_pool !  Improved in MC 7.3, set ndb_cluster_connection_pool between 1 and 4. !  MySQL Cluster 7.2 and earlier: !  Problem: !  A Sendbuffer on the connection between mysqld and the data nodes is protected by a Mutex. !  Connection threads in MySQL must acquire Mutex and the put data in SendBuffer. !  Many threads gives more contention on the mutex !  Must scale out with many MySQL Servers. !  Workaround: !  Ndb_cluster_connection_pool (in my.cnf) creates more connections from one mysqld to the data nodes !  Threads load balance on the connections gives less contention on mutex which in turn gives increased scalabilty !  Less MySQL Servers needed to drive load! !  www.severalnines.com/cluster-configurator allows you to specify the connection pool.
  • 39. Ndb_cluster_connection_pool !  Improved in MC 7.3, set ndb_cluster_connection_pool between 1 and 4. !  MySQL Cluster 7.2 and earlier: !  Gives at least 70% better performance and a MySQL Server that can scale beyond four database connections. !  Set Ndb_cluster_connection_pool=2x<CPU cores> !  It is a good starting point !  One free [mysqld] slot is required in config.ini for each Ndb_cluster_connection. !  4 mysql servers,each with Ndb_cluster_connection_pool=8 requires 32 [mysqld] in config.ini !  Note that also memcached and node.js, cluster/j etc also has the concept of the ndb_cluster_connection_pool.
  • 40. Q&A 40 Copyright 2011 Severalnines AB
  • 41. Resources !  MySQL Cluster Configurator !  www.severalnines.com/config !  MySQL Cluster Management + Monitoring !  www.severalnines.com/cmon !  MySQL Cluster Training Slides !  www.severalnines.com/mysql-cluster-training !  My Blog !  johanandersson.blogspot.com
  • 42. Keep in touch… !  Facebook !  www.facebook.com/severalnines !  Twitter !  @severalnines !  Linked in: !  www.linkedin.com/company/severalnines
  • 43. Thank you for your time! johan@severalnines.com 43 Copyright 2011 Severalnines AB