Presentation used at COSCUP 2014 in Taiwan.

1. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.1
MySQL Performance Tuning
MySQL Global Business Unit
Sales Consulting Senior Manager, JAPAC
梶山 隆輔 / Ryusuke Kajiyama

2. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.
MAKE MySQL
a Better MySQL

3. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.3
MySQL 5.5
MySQL Cluster 7.3
MySQL Enterprise Monitor 2.3
MySQL Enterprise Backup
Security
Scalability
HA
Audit
MySQL 5.6
MySQL Workbench 6.0
MySQL Utilities
MySQL Applier for
Hadoop
MySQL Workbench 5.2
MySQL Enterprise Oracle
Certifications
Over 3,5 Years of MySQL Innovation
MySQL Cluster Manager
Windows installer & Tools
MySQL Cluster 7.2
MySQL Cluster 7.1
MySQL Migration
Wizard

4. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.4
MySQL: Everywhere and Growing
Platform of Choice for Next Generation Web, Cloud & Embedded Applications
World’s Most
Popular Open
Source
Database
Database
Platform for
Next Gen
Web Apps
Leading
Database for
Web
Applications
Leading
Database in
the Cloud
Integrated with
Hadoop in
majority of Big
Data platforms
#1 Linux Career IT
skill: MySQL *
* Source: Linux Career IT Skills Watch update July 2013
Downloads & Sales
Growth
Best of both Worlds
SQL+NoSQL
LAMP + Python
+Node.js + Go +...
Awards & Social
Media Momentum

5. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.5
MySQL 5.6: Best Release Ever!

6. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.6
MySQL 5.6: Scalability
 Users can fully utilize latest generations of hardware and OS
 Scales as data volumes and users grow

7. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.7
MySQL 5.5
MySQL 5.6
MySQL 5.6 SysBench Benchmarks
Up to 234% Performance Gain

8. Replication Binary Log Group Commit
• Days of losing 50%+
master performance
are gone!
• 10% overhead when
configuring
sync_binlog=0
• Additional 5% when
configuring
sync_binlog=1
https://blogs.oracle.com/MySQL/entry/mysql_5_6_replication_performance

9. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.9
Same app can leverage:
 Key-value access to InnoDB via
familiar Memcached API
 SQL for rich queries, JOINs,
FKs, etc.
 Fully transactional
MySQL 5.6: InnoDB
NoSQL Key Value Access to InnoDB
 Up to 9x performance boost for
updates
 Great for fast data ingestion in Big
Data pipeline

10. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.10
MySQL 5.6: Optimizer

11. • Simple word but many meanings
• Main objective:
• Users (direct or indirect) should be satisfied
• Most typical performance metrics
• Throughput
• Latency / Response time
• Scalability
• Combined metrics
Starvation
transactions/time
Defining Performance

12. • Multi User applications
• Request waits in queue before being processed
• User response time = queueing delay + service time
• Non high tech example – support call center.
• “Hockey Stick” - queuing delay grows rapidly when
system getting close to saturation
• Need to improve queueing delay or service time
to improve performance
• Improving service time reduces queuing delay
Connections
ResponseTime
Queuing Theory

13. • Main Question – where does service time comes from ?
• network, cpu, disk, locks...
• Direct Measurements
• Sum up all query times from web pages
• Indirect measurements
• CPU usage
• Disk IO latency
• Network traffic
• Load Average
• Number of running queries
• etc.
Service Time: Key to the hotspot

14. Benchmark Tests
• Great tool to:
• Quantify application performance
• Measure performance effect of the changes
• Validate Scalability
• Plan deployment
• But
• Can be very misleading if done wrong
• Need to be able to read results correctly
• Typical Errors
• Testing with 1GB size with 100G in production
• Using uniform distribution
• “Harry Potter” ordered as frequent as Zulu dictionary
• Testing in single user scenario

15. Benchmark Tools for MySQL
• Creating own benchmark tool
– Generating from SQL statements in General Log
– Fetching SQL statements with MySQL Proxy or TCP Dump
• DBT2
– http://osdldbt.sourceforge.net/
– http://nippondanji.blogspot.com/2009/03/dbt-2.html
• mysqlslap MySQL 5.1 +
– http://dev.mysql.com/doc/refman/5.6/en/mysqlslap.html
• SysBench
– http://sysbench.sourceforge.net/

16. • Performance costs money, whatever road you take
• Investigate different possibilities
• Better hardware could be cheaper than a major rewrite
• How much performance/scalability/reliability do you need ?
• 99.999% could be a lot more expensive than 99.9%
• Is peak traffic requirements 100x average or just 3x ?
• Take a look at whole picture
• Is this the largest risk/bottleneck ?
• Identify which optimizations are critical for business
• Optimization of “everything” is often waste of resources
• What is the cost of suboptimal performance ?
Business side of optimization

17. Basic1: Checking Server Configurations
• The MySQL server is controlled by “System Variables”
• Set via:
• Option File: my.cnf / my.ini
• Temporary change: SET [GLOBAL] <vriable>=<value>
• Can be per connection (LOCAL) or server wide(GLOBAL)

18. Basic2: Checking Status of Server
 Monitor system performance using “Status Variables”
• Knowing internal commands of one query
Mysql> FLUSH STATUS; <run query>; SHOW STATUS;
• Checking status periodically
shell> mysqladmin -u -p ... ex -i 15 -r | grep -v ‘ 0 ‘
http://dev.mysql.com/doc/refman/5.6/en/server-status-variables.html

19. Client2 ClientN
Connection Thread Pool
Parser Query 101101
Client1
Optimizer
Storage Engines
 InnoDB
 MyISAM
 MERGE
 MEMORY
 ARCHIVE
Query Cache
MySQL Server
MySQL Server Architecture

20. • max_connections (151)
• number of connections server will
allow. May run out of memory if too
high, because of per connections
memory usage
• thread_cache_size (8)
• Keep up to this amount of threads
“cached” after disconnect
• Typical setting
max_connections/3
Client2 ClientN
Connection Thread Pool
Client1
mysql> show status;
• Max_used_connections
• check if it matches
max_connections, too low value or
sign of overload
• Threads_created
• thread_cache misses
• should be low.
Server Connections & Threads

21. • sort_buffer_size (2M)
• Memory to allocate for sort. Will use
disk based sort for larger data sets.
Often fine at 512K or 1M
• other buffers, read,
read_rnd, etc… smaller
defaults often OK
• You can change dynamically if
large sort operation is needed
in batch operation etc
Client2 ClientN
Connection Thread Pool
Client1
mysql> show status;
• Sort_merge_passes -
• number of passes made
during file merge sort.
• check if file sort needs to be
done at all
• use index if possible
Connection Thread Work Buffers

22. • query_cache_size (0)
• Amount of memory to use for query
cache
• Typically 32M is fine, some
databases need 128M
• query_cache_type (ON)
• Worst case performance overhead
is about 15%-20%
• Favors servers with higher
SELECT/WRITE ratios
• Best Practice
• Set to DEMAND
• Add SQL_CACHE to appropriate
queries
Connection Thread Pool
Parser Query 101101
Query Cache
mysql> show status;
• Qcache_hits, Qcache_inserts
• hits/inserts cache hit ratio, if small, maybe
disable query cache
• Qcache_lowmem_prunes
• times older queries were removed due to
low memory, need to increase
query_cache_size if high
Server Query Cache

23. • innodb_buffer_pool_size
• 80% of memory on Innodb only system
• caches data & indexes unlike MyISAM
• innodb_log_file_size
• A key parameter for write performance
• Recovery time is no more an issue.
• Bigger is better for write QPS stability
• innodb_flush_log_at_trx_commit
• 1 (slow) will flush (fsync) log at each commit. Truly
ACID
• 2 (fast) will only flush log buffer to OS cache on
commit, sync to disk once/sec.
• 0 (fastest) will flush (fsync) log every second or so
• innodb_file_per_table
• Always good choice to distribute i/o
• Default ON from 5.6
Storage Engines
 InnoDB
 MyISAM
 MERGE
 MEMORY
ARCHIVE
mysql> SHOW ENGINE
INNODB STATUS;
Great way to see what is
going on inside InnoDB,
hard to parse
• File IO
• Buffer Pool
• Log activity
• Row activity
InnoDB Performance Tips

24. • innodb_flush_method = O_DIRECT
• Not to consume OS cache
• innodb_buffer_pool_instances (5.5+)
• To avoid mutex contention
• 2 or more in can
• innodb_io_capacity (5.5+)
• Enlarge if you have fast disks
• Default 200 is good for 2 disks striped
• innodb_read_io_threads (5.5+)
• innodb_write_io_threads (5.5+)
• Enlarge if you have fast disks
• Default 4 is usually good enough
Storage Engines
 InnoDB
 MyISAM
 MERGE
 MEMORY
ARCHIVE
mysql> SHOW ENGINE
INNODB STATUS;
Great way to see what is
going on inside InnoDB,
hard to parse
• File IO
• Buffer Pool
• Log activity
• Row activity
InnoDB Performance Tips (Cont.)

25. DBT-2 (W200) Transactions per Minute %user %iowait
Buffer pool 1G 1125.44 2% 30%
Buffer pool 2G 1863.19 3% 28%
Buffer pool 5G 4385.18 5.5% 33%
Buffer pool 30G
(All data in cache) 36784.76 36% 8%
Semi-in-memory Database with InnoDB
• DBT-2 benchmark (write intensive)
• 20-25GB hot data (200 warehouses, running 1 hour)
• Nehalem 2.93GHz x 8 cores, MySQL 5.5.2, 4 RAID1+0 HDDs
• RAM size affects everything. Not only for SELECT,
but also for INSERT/UPDATE/DELETE
• INSERT: Random reads/writes happen when inserting into indexes in random order
• UPDATE/DELETE: Random reads/writes happen when modifying records

26. • Normalization
• good for OLTP, writes
• data redundancies eliminated
• join performance penalty
• smaller total data set
• E/R diagram clean translation
• Denormalization
• good for OLAP, reporting
• data redundancies across tables
for better indexing
• maybe eliminate joins
• Data types
– use tinyint, smallint, mediumint,
save space!
– join columns same data type
– varchar(64) instead of char(64)
– declare NOT NULL if true
– varchar(64) instead of varchar(255)
– INT not DECIMAL(9)
• Indexing
– multi-column
– ordered BTREE
– index prefixes
– covering
Schema Design

27. #Time: 08073101 16:25:24
#User@Host: root[root] @ localhost [127.0.0.1]
#Query_time: 8 Lock_time: 0 Rows_sent: 20 Rows_examined: 243661
SELECT part_num FROM `inventory`.`parts` WHERE
(`ven` = "foo") ORDER BY `delivery_datetime` DESC LIMIT 100;
Monitoring Queries - Slow Query Log
Cons
• Requires MySQL restart (5.0 and lower)
• Growth must be managed using FLUSH LOGS
• Entries must be parsed/sorted for relevance
• mysqldumpslow helps, but still tedious, takes time
Pros
• Logs queries that took longer than X (user defined)
• Logs queries that do not use indexes (5.0 and higher)
• Includes data needed to trace offending queries

28. Monitoring Queries – SHOW PROCESSLIST;
mysql> SHOW FULL PROCESSLISTG
******** 1. row *****************
Id: 1
User: MyUser
Host: localhost
db: inventory
Command: Query
Time: 1030455
State: Sending Data
Info: SELECT part_num from ‘inv’;
…..
2 rows in set (0.00 sec)
Pros
• Shows current processes
• Shows status of executing
queries
• Includes data needed to trace
offending queries
Con
Scripting needed to:
• automate,
• integrate with Slow Query Log,
• Aggregate/parse results for
analysis,
• notify DBA of problem

29. Fixing Problem Queries – EXPLAIN;
EXPLAIN SELECT part_num
FROM `inventory`.`parts`
WHERE (`ven` = "foo")
ORDER BY `delivery_datetime`
DESC LIMIT 100;G
******** 1. row *************
ID: 1
select_type: SIMPLE
table: parts
type: ref
possible_keys: ven, part#
key: ven
key_len: 3
ref: null
rows: 872
Extra: Using WHERE
1 row in set (0.00 sec)
Analyze
• How indexes are being used (or
not…)
• required filesorts
• What tables, columns are being
queried
Fix/Tune - involves iterations of:
• Add/alter indexes
• Alter tables, columns, datatypes
• Alter query structure
• Test, 10 GOTO 10 until done

30. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 1230
MySQL Enterprise Monitor
 Real-time MySQL performance and availability monitoring
 Visually find & fix problem queries
 Disk monitoring for capacity planning
 Cloud friendly architecture (no agents)
 Start monitoring MySQL in 10 minutes
 Remote agent option provides OS monitoring

31. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 1231
MySQL Query Analyzer
 Real-time query performance
 Visual correlation graphs
 Find & fix expensive queries
 Detailed query statistics
 Query Response Time index (QRTi)
“With the MySQL Query Analyzer, we were able to
identify and analyze problematic SQL code, and triple
our database performance. More importantly, we were
able to accomplish this in three days, rather than
taking weeks.”
Keith Souhrada
Software Development Engineer
Big Fish Games

32. MySQL Enterprise Scalability
MySQL Thread Pool
• MySQL default thread-handling – excellent
performance, can limit scalability as connections grow
• MySQL Thread Pool improves sustained
performance/scale as user connections grow
• Thread Pool API
New!

33. MySQL Enterprise Scalability
Configuration
MySQL 5.6.11
Oracle Linux 6.3, Unbreakable Kernel 2.6.32
4 sockets, 24 cores, 48 Threads
Intel(R) Xeon® E7540 2GHz CPUs
512GB DDR RAM

34. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.34
Performance Schema in MySQL 5.5 & MySQL 5.6

35. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.35
Performance Schema in MySQL 5.5
Laid out the framework for instrumentation
Focused early on low level instruments
– Prove it could handle load, build from ground up
Not turned on by default, had some overhead issues

36. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.36
Performance Schema in MySQL 5.5
17 Tables
222 Instruments
Instrument Event Class
File IO wait/io/file/%
Mutexes wait/synch/mutex/%
Read/Write
Locks
wait/synch/rwlock/%
Conditions wait/synch/cond/%

37. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.37
Performance Schema in MySQL 5.5

38. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.38
Performance Schema in MySQL 5.6
Fixed lots of performance issues
Focused on bringing the instrumentation to the DBA / Developer
Statements / Statement Digests
Execution Stages
Object tracking (table / index IO, table locks)
Network IO
Turned on by default, with a subset of instrumentation disabled

39. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.39
Performance Schema in MySQL 5.6
52 Tables (+35)
545 Instruments (+323)
Instrument
Type
Event Class
Statements statement/%
Stages stage/%
Table IO wait/io/table/%
Table Locks wait/lock/table/%
Network IO wait/io/socket/%
Idle Timing idle

40. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.40
Performance Schema in MySQL 5.6

41. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.41
Improvements made to date in MySQL 5.7

42. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 12
Insert Picture Here
42
Memory Usage
Metadata Locking
Replication Configuration & Status
Prepared Statements
Transactions
Stored Programs

43. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.43
Improvements made to date in MySQL 5.7
75 Tables (+23)
784 Instruments (+239)
Instrument Type Event Class
Transactions transaction
Memory memory/%
Metadata Locks wait/lock/metadata/%

44. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.44
Improvements made to date in MySQL 5.7

45. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.45
DMRs: Development Milestone Releases
 Fully Functional Release Candidate Quality
 Two to Four per Year
 Early Community Testing, Use, & Feedback
 Accelerates Rate of Enhancements
 Improves Quality

46. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.46
MySQL 5.7
 InnoDB for better transactional throughput, availability
 Replication for better scalability and availability
 Utilities for dev/ops automation
 Performance Schema for better performance metrics
 Optimizer for better EXPLAINing query performance
Available Now! Get it here:
MySQL 5.7.2 builds on MySQL 5.6 by improving:
dev.mysql.com/downloads/mysql/
New!

47. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.47
MySQL 5.7 Sysbench Benchmarks
Sysbench Point Select
Intel(R) Xeon(R) CPU X7560 x86_64
4 sockets x 8 cores-HT (64 CPU threads)
2.27GHz, 256G RAM
Oracle Linux 6.2
95% Faster than MySQL 5.6
172% Faster than MySQL 5.5
500,000 QPS

48. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.48
RO In-Memory @MySQL 5.7
 Sysbench OLTP_RO 8-tables, 32cores-HT :

49. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.49
RO In-Memory @MySQL 5.7

50. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.50

51. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.51
第一次台灣 MySQL 社群聚會
 我們將在 7 月 22 日下午 6:00 到 8:00 借用台北市忠孝西路
一段 35 樓甲骨文的會議室，舉辦第一次台灣 MySQL 社群
聚會，歡迎您抽空參加。在聚會中我們將會中提供簡單的晚
餐，同時分享 MySQL 的最新動態，包括：
– 展示如何用新發表的 MySQL Fabric 支持上 PB 級資料的應用系統，
做資料庫分表分庫，高可用和讀寫分離
– MySQL 5.7.4 DMR 的新功能和産品發展路線
 也非常歡迎您分享您的心得和意見如果您打算參加 22 日的
聚會，或打算分享您的心得
1st
TWMUG meeting