MySQL Performance Tuning at COSCUP 2014

7,878 views

Published on

Presentation used at COSCUP 2014 in Taiwan.

0 Comments
19 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
7,878
On SlideShare
0
From Embeds
0
Number of Embeds
4,036
Actions
Shares
0
Downloads
168
Comments
0
Likes
19
Embeds 0
No embeds

No notes for slide
  • Since Oracle acquired Sun we have delivered A TON of MySQL Innovation
    Go through the different deliverables
    We’ve improved the MySQL experience for users, customers, for folks using MySQL on Windows, for Oracle customers using MySQL...etc
    In case anyone still has any doubt about Oracle’s commitment to MySQL, those doubts clearly can be put to rest. We’re delivering more than ever before in MySQL’s history, and will continue to do so.
  • MySQL is in an incredibly favorable position, and the platform of choice for next generation Web, mobile and cloud applications.
    It is....(the 4 key points)
    We’re also seeing great momentum for MySQL: talk about 3 points on the side
    We want to ensure that MySQL remains the platform of choice for next gen web apps, and we are evolving the database by developing new features, enhancements and integrating in it new capabilities useful for modern apps. Give example of the 2 other points in orange (NoSQL and new languages)
  • MySQL 5.6 delivers across the board improvements designed to enable innovative DBAs and Developers to create and deploy the next generation of web, embedded and Cloud/SaaS/DaaS applications on the latest generation of development frameworks and hardware platforms.
    At a glance, MySQL 5.6 is simply a better MySQL with improvements that enhance every functional area of the database kernel, including:
     
    Improved Performance Scalability
    5.6 scales up 48 CPU threads which enables users to leverage the full processing power of modern architectures
    5.6 performs up to 230% faster over 5.5 on transactional work loads.
    Improved InnoDB
    Allow 5.6 to provide better transactional throughput and availability over previous versions.
    Improved Optimizer
    Provides better query execution times and more diagnostics for monitoring, tuning and debugging
    Impoved Replication
    Provides higher performance, availability and data integrity
    Improved Performance Schema
    Provides better instrumentation and user/application level statistics and monitoring
    New! NoSQL Access to InnoDB
    Provides fast, Key Value access with full ACID compliance, better developer agility
    Improved InnoDB storage engine for better transactional throughput
    Improved Optimizer for better query execution times and diagnostics
    Better Application Availability with Online DDL/Schema changes
    Better Developer Agility with NoSQL Access with Memcached API to InnoDB
    Improved Replication for high performance, self-healing distributed deployments
    Improved Performance Schema for better instrumentation
    Improved Security for worry-free application deployments
    And other Important Enhancements
  • In MySQL 5.6 performance has been improved to provide better linear performance and scale on systems supporting multi-processors and high CPU thread concurrency.
    Internal benchmarks for SysBench Read/Write and Read Only workloads show a marked improvement in scale over the most current version of MySQL 5.5.
    For Linux, SysBench Read/Write benchmarks show that 5.6 scales “up and to the right” on > 48 CPU threads/cores.
    Similar improvements have been delivered on Windows.
  • For Linux, SysBench Read Only benchmarks show that 5.6 performance is improved up to 234% over 5.5 at the same user concurrency.
    Similar improvements have been delivered on Windows.
  • Many of the latest generation of web, cloud, social and mobile applications require fast operations against simple Key/Value pairs. At the same time, they must retain the ability to run complex queries against the same data, as well as ensure the data is protected with ACID guarantees. With the new NoSQL API for InnoDB, developers have all the benefits of a transactional RDBMS, coupled with the performance capabilities of Key/Value store.
     
    MySQL 5.6 provides simple, key-value interaction with InnoDB data via the familiar Memcached API. Implemented via a new Memcached daemon plug-in to mysqld, the new Memcached protocol is mapped directly to the native InnoDB API and enables developers to use existing Memcached clients to bypass the expense of query parsing and go directly to InnoDB data for lookups and transactional compliant updates. The API makes it possible to re-use standard Memcached libraries and clients, while extending Memcached functionality by integrating a persistent, crash-safe, transactional database back-end.
  • Real-time MySQL Performance and Availability Monitoring
    Visual Dashboad for Better Visibility
    Visually Find and Fix Problem Queries
    Pinpoint SQL code that causing a slowdown
    Visual Capacity Planning
    Forecasting using Projections and Trend Analysis
    Agentless, Cloud Friendly Architecture
    Remotely Monitor MySQL without agents
    Start Monitoring MySQL in 10 minutes
    Easy to use with zero configuration
  • MySQL Performance Tuning at COSCUP 2014

    1. 1. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.1 MySQL Performance Tuning MySQL Global Business Unit Sales Consulting Senior Manager, JAPAC 梶山 隆輔 / Ryusuke Kajiyama
    2. 2. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. MAKE MySQL a Better MySQL
    3. 3. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.3 MySQL 5.5 MySQL Cluster 7.3 MySQL Enterprise Monitor 2.3 MySQL Enterprise Backup Security Scalability HA Audit MySQL 5.6 MySQL Workbench 6.0 MySQL Utilities MySQL Applier for Hadoop MySQL Workbench 5.2 MySQL Enterprise Oracle Certifications Over 3,5 Years of MySQL Innovation MySQL Cluster Manager Windows installer & Tools MySQL Cluster 7.2 MySQL Cluster 7.1 MySQL Migration Wizard
    4. 4. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.4 MySQL: Everywhere and Growing Platform of Choice for Next Generation Web, Cloud & Embedded Applications World’s Most Popular Open Source Database Database Platform for Next Gen Web Apps Leading Database for Web Applications Leading Database in the Cloud Integrated with Hadoop in majority of Big Data platforms #1 Linux Career IT skill: MySQL * * Source: Linux Career IT Skills Watch update July 2013 Downloads & Sales Growth Best of both Worlds SQL+NoSQL LAMP + Python +Node.js + Go +... Awards & Social Media Momentum
    5. 5. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.5 MySQL 5.6: Best Release Ever!
    6. 6. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.6 MySQL 5.6: Scalability  Users can fully utilize latest generations of hardware and OS  Scales as data volumes and users grow
    7. 7. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.7 MySQL 5.5 MySQL 5.6 MySQL 5.6 SysBench Benchmarks Up to 234% Performance Gain
    8. 8. Replication Binary Log Group Commit • Days of losing 50%+ master performance are gone! • 10% overhead when configuring sync_binlog=0 • Additional 5% when configuring sync_binlog=1 https://blogs.oracle.com/MySQL/entry/mysql_5_6_replication_performance
    9. 9. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.9 Same app can leverage:  Key-value access to InnoDB via familiar Memcached API  SQL for rich queries, JOINs, FKs, etc.  Fully transactional MySQL 5.6: InnoDB NoSQL Key Value Access to InnoDB  Up to 9x performance boost for updates  Great for fast data ingestion in Big Data pipeline
    10. 10. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.10 MySQL 5.6: Optimizer
    11. 11. • Simple word but many meanings • Main objective: • Users (direct or indirect) should be satisfied • Most typical performance metrics • Throughput • Latency / Response time • Scalability • Combined metrics Starvation transactions/time Defining Performance
    12. 12. • Multi User applications • Request waits in queue before being processed • User response time = queueing delay + service time • Non high tech example – support call center. • “Hockey Stick” - queuing delay grows rapidly when system getting close to saturation • Need to improve queueing delay or service time to improve performance • Improving service time reduces queuing delay Connections ResponseTime Queuing Theory
    13. 13. • Main Question – where does service time comes from ? • network, cpu, disk, locks... • Direct Measurements • Sum up all query times from web pages • Indirect measurements • CPU usage • Disk IO latency • Network traffic • Load Average • Number of running queries • etc. Service Time: Key to the hotspot
    14. 14. Benchmark Tests • Great tool to: • Quantify application performance • Measure performance effect of the changes • Validate Scalability • Plan deployment • But • Can be very misleading if done wrong • Need to be able to read results correctly • Typical Errors • Testing with 1GB size with 100G in production • Using uniform distribution • “Harry Potter” ordered as frequent as Zulu dictionary • Testing in single user scenario
    15. 15. Benchmark Tools for MySQL • Creating own benchmark tool – Generating from SQL statements in General Log – Fetching SQL statements with MySQL Proxy or TCP Dump • DBT2 – http://osdldbt.sourceforge.net/ – http://nippondanji.blogspot.com/2009/03/dbt-2.html • mysqlslap MySQL 5.1 + – http://dev.mysql.com/doc/refman/5.6/en/mysqlslap.html • SysBench – http://sysbench.sourceforge.net/
    16. 16. • Performance costs money, whatever road you take • Investigate different possibilities • Better hardware could be cheaper than a major rewrite • How much performance/scalability/reliability do you need ? • 99.999% could be a lot more expensive than 99.9% • Is peak traffic requirements 100x average or just 3x ? • Take a look at whole picture • Is this the largest risk/bottleneck ? • Identify which optimizations are critical for business • Optimization of “everything” is often waste of resources • What is the cost of suboptimal performance ? Business side of optimization
    17. 17. Basic1: Checking Server Configurations • The MySQL server is controlled by “System Variables” • Set via: • Option File:   my.cnf / my.ini • Temporary change: SET [GLOBAL] <vriable>=<value> • Can be per connection (LOCAL) or server wide(GLOBAL)
    18. 18. Basic2: Checking Status of Server  Monitor system performance using “Status Variables” • Knowing internal commands of one query Mysql> FLUSH STATUS; <run query>; SHOW STATUS; • Checking status periodically shell> mysqladmin -u -p ... ex -i 15 -r | grep -v ‘ 0 ‘ http://dev.mysql.com/doc/refman/5.6/en/server-status-variables.html
    19. 19. Client2 ClientN Connection Thread Pool Parser Query 101101 Client1 Optimizer Storage Engines  InnoDB  MyISAM  MERGE  MEMORY  ARCHIVE Query Cache MySQL Server MySQL Server Architecture
    20. 20. • max_connections (151) • number of connections server will allow. May run out of memory if too high, because of per connections memory usage • thread_cache_size (8) • Keep up to this amount of threads “cached” after disconnect • Typical setting max_connections/3 Client2 ClientN Connection Thread Pool Client1 mysql> show status; • Max_used_connections • check if it matches max_connections, too low value or sign of overload • Threads_created • thread_cache misses • should be low. Server Connections & Threads
    21. 21. • sort_buffer_size (2M) • Memory to allocate for sort. Will use disk based sort for larger data sets. Often fine at 512K or 1M • other buffers, read, read_rnd, etc… smaller defaults often OK • You can change dynamically if large sort operation is needed in batch operation etc Client2 ClientN Connection Thread Pool Client1 mysql> show status; • Sort_merge_passes - • number of passes made during file merge sort. • check if file sort needs to be done at all • use index if possible Connection Thread Work Buffers
    22. 22. • query_cache_size (0) • Amount of memory to use for query cache • Typically 32M is fine, some databases need 128M • query_cache_type (ON) • Worst case performance overhead is about 15%-20% • Favors servers with higher SELECT/WRITE ratios • Best Practice • Set to DEMAND • Add SQL_CACHE to appropriate queries Connection Thread Pool Parser Query 101101 Query Cache mysql> show status; • Qcache_hits, Qcache_inserts • hits/inserts cache hit ratio, if small, maybe disable query cache • Qcache_lowmem_prunes • times older queries were removed due to low memory, need to increase query_cache_size if high Server Query Cache
    23. 23. • innodb_buffer_pool_size • 80% of memory on Innodb only system • caches data & indexes unlike MyISAM • innodb_log_file_size • A key parameter for write performance • Recovery time is no more an issue. • Bigger is better for write QPS stability • innodb_flush_log_at_trx_commit • 1 (slow) will flush (fsync) log at each commit. Truly ACID • 2 (fast) will only flush log buffer to OS cache on commit, sync to disk once/sec. • 0 (fastest) will flush (fsync) log every second or so • innodb_file_per_table • Always good choice to distribute i/o • Default ON from 5.6 Storage Engines  InnoDB  MyISAM  MERGE  MEMORY ARCHIVE mysql> SHOW ENGINE INNODB STATUS; Great way to see what is going on inside InnoDB, hard to parse • File IO • Buffer Pool • Log activity • Row activity InnoDB Performance Tips
    24. 24. • innodb_flush_method = O_DIRECT • Not to consume OS cache • innodb_buffer_pool_instances (5.5+) • To avoid mutex contention • 2 or more in can • innodb_io_capacity (5.5+) • Enlarge if you have fast disks • Default 200 is good for 2 disks striped • innodb_read_io_threads (5.5+) • innodb_write_io_threads (5.5+) • Enlarge if you have fast disks • Default 4 is usually good enough Storage Engines  InnoDB  MyISAM  MERGE  MEMORY ARCHIVE mysql> SHOW ENGINE INNODB STATUS; Great way to see what is going on inside InnoDB, hard to parse • File IO • Buffer Pool • Log activity • Row activity InnoDB Performance Tips (Cont.)
    25. 25. DBT-2 (W200) Transactions per Minute %user %iowait Buffer pool 1G 1125.44 2% 30% Buffer pool 2G 1863.19 3% 28% Buffer pool 5G 4385.18 5.5% 33% Buffer pool 30G (All data in cache) 36784.76 36% 8% Semi-in-memory Database with InnoDB • DBT-2 benchmark (write intensive) • 20-25GB hot data (200 warehouses, running 1 hour) • Nehalem 2.93GHz x 8 cores, MySQL 5.5.2, 4 RAID1+0 HDDs • RAM size affects everything. Not only for SELECT, but also for INSERT/UPDATE/DELETE • INSERT: Random reads/writes happen when inserting into indexes in random order • UPDATE/DELETE: Random reads/writes happen when modifying records
    26. 26. • Normalization • good for OLTP, writes • data redundancies eliminated • join performance penalty • smaller total data set • E/R diagram clean translation • Denormalization • good for OLAP, reporting • data redundancies across tables for better indexing • maybe eliminate joins • Data types – use tinyint, smallint, mediumint, save space! – join columns same data type – varchar(64) instead of char(64) – declare NOT NULL if true – varchar(64) instead of varchar(255) – INT not DECIMAL(9) • Indexing – multi-column – ordered BTREE – index prefixes – covering Schema Design
    27. 27. #Time: 08073101 16:25:24 #User@Host: root[root] @ localhost [127.0.0.1] #Query_time: 8 Lock_time: 0 Rows_sent: 20 Rows_examined: 243661 SELECT part_num FROM `inventory`.`parts` WHERE (`ven` = "foo") ORDER BY `delivery_datetime` DESC LIMIT 100; Monitoring Queries - Slow Query Log Cons • Requires MySQL restart (5.0 and lower) • Growth must be managed using FLUSH LOGS • Entries must be parsed/sorted for relevance • mysqldumpslow helps, but still tedious, takes time Pros • Logs queries that took longer than X (user defined) • Logs queries that do not use indexes (5.0 and higher) • Includes data needed to trace offending queries
    28. 28. Monitoring Queries – SHOW PROCESSLIST; mysql> SHOW FULL PROCESSLISTG ******** 1. row ***************** Id: 1 User: MyUser Host: localhost db: inventory Command: Query Time: 1030455 State: Sending Data Info: SELECT part_num from ‘inv’; ….. 2 rows in set (0.00 sec) Pros • Shows current processes • Shows status of executing queries • Includes data needed to trace offending queries Con Scripting needed to: • automate, • integrate with Slow Query Log, • Aggregate/parse results for analysis, • notify DBA of problem
    29. 29. Fixing Problem Queries – EXPLAIN; EXPLAIN SELECT part_num FROM `inventory`.`parts` WHERE (`ven` = "foo") ORDER BY `delivery_datetime` DESC LIMIT 100;G ******** 1. row ************* ID: 1 select_type: SIMPLE table: parts type: ref possible_keys: ven, part# key: ven key_len: 3 ref: null rows: 872 Extra: Using WHERE 1 row in set (0.00 sec) Analyze • How indexes are being used (or not…) • required filesorts • What tables, columns are being queried Fix/Tune - involves iterations of: • Add/alter indexes • Alter tables, columns, datatypes • Alter query structure • Test, 10 GOTO 10 until done
    30. 30. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 1230 MySQL Enterprise Monitor  Real-time MySQL performance and availability monitoring  Visually find & fix problem queries  Disk monitoring for capacity planning  Cloud friendly architecture (no agents)  Start monitoring MySQL in 10 minutes  Remote agent option provides OS monitoring
    31. 31. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 1231 MySQL Query Analyzer  Real-time query performance  Visual correlation graphs  Find & fix expensive queries  Detailed query statistics  Query Response Time index (QRTi) “With the MySQL Query Analyzer, we were able to identify and analyze problematic SQL code, and triple our database performance. More importantly, we were able to accomplish this in three days, rather than taking weeks.” Keith Souhrada Software Development Engineer Big Fish Games
    32. 32. MySQL Enterprise Scalability MySQL Thread Pool • MySQL default thread-handling – excellent performance, can limit scalability as connections grow • MySQL Thread Pool improves sustained performance/scale as user connections grow • Thread Pool API New!
    33. 33. MySQL Enterprise Scalability Configuration MySQL 5.6.11 Oracle Linux 6.3, Unbreakable Kernel 2.6.32 4 sockets, 24 cores, 48 Threads Intel(R) Xeon® E7540 2GHz CPUs 512GB DDR RAM
    34. 34. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.34 Performance Schema in MySQL 5.5 & MySQL 5.6
    35. 35. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.35 Performance Schema in MySQL 5.5 Laid out the framework for instrumentation Focused early on low level instruments – Prove it could handle load, build from ground up Not turned on by default, had some overhead issues
    36. 36. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.36 Performance Schema in MySQL 5.5 17 Tables 222 Instruments Instrument Event Class File IO wait/io/file/% Mutexes wait/synch/mutex/% Read/Write Locks wait/synch/rwlock/% Conditions wait/synch/cond/%
    37. 37. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.37 Performance Schema in MySQL 5.5
    38. 38. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.38 Performance Schema in MySQL 5.6 Fixed lots of performance issues Focused on bringing the instrumentation to the DBA / Developer Statements / Statement Digests Execution Stages Object tracking (table / index IO, table locks) Network IO Turned on by default, with a subset of instrumentation disabled
    39. 39. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.39 Performance Schema in MySQL 5.6 52 Tables (+35) 545 Instruments (+323) Instrument Type Event Class Statements statement/% Stages stage/% Table IO wait/io/table/% Table Locks wait/lock/table/% Network IO wait/io/socket/% Idle Timing idle
    40. 40. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.40 Performance Schema in MySQL 5.6
    41. 41. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.41 Improvements made to date in MySQL 5.7
    42. 42. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 12 Insert Picture Here 42 Memory Usage Metadata Locking Replication Configuration & Status Prepared Statements Transactions Stored Programs
    43. 43. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.43 Improvements made to date in MySQL 5.7 75 Tables (+23) 784 Instruments (+239) Instrument Type Event Class Transactions transaction Memory memory/% Metadata Locks wait/lock/metadata/%
    44. 44. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.44 Improvements made to date in MySQL 5.7
    45. 45. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.45 DMRs: Development Milestone Releases  Fully Functional Release Candidate Quality  Two to Four per Year  Early Community Testing, Use, & Feedback  Accelerates Rate of Enhancements  Improves Quality
    46. 46. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.46 MySQL 5.7  InnoDB for better transactional throughput, availability  Replication for better scalability and availability  Utilities for dev/ops automation  Performance Schema for better performance metrics  Optimizer for better EXPLAINing query performance Available Now! Get it here: MySQL 5.7.2 builds on MySQL 5.6 by improving: dev.mysql.com/downloads/mysql/ New!
    47. 47. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.47 MySQL 5.7 Sysbench Benchmarks Sysbench Point Select Intel(R) Xeon(R) CPU X7560 x86_64 4 sockets x 8 cores-HT (64 CPU threads) 2.27GHz, 256G RAM Oracle Linux 6.2 95% Faster than MySQL 5.6 172% Faster than MySQL 5.5 500,000 QPS
    48. 48. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.48 RO In-Memory @MySQL 5.7  Sysbench OLTP_RO 8-tables, 32cores-HT :
    49. 49. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.49 RO In-Memory @MySQL 5.7
    50. 50. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.50
    51. 51. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.51 第一次台灣 MySQL 社群聚會  我們將在 7 月 22 日下午 6:00 到 8:00 借用台北市忠孝西路 一段 35 樓甲骨文的會議室,舉辦第一次台灣 MySQL 社群 聚會,歡迎您抽空參加。在聚會中我們將會中提供簡單的晚 餐,同時分享 MySQL 的最新動態,包括: – 展示如何用新發表的 MySQL Fabric 支持上 PB 級資料的應用系統, 做資料庫分表分庫,高可用和讀寫分離 – MySQL 5.7.4 DMR 的新功能和産品發展路線  也非常歡迎您分享您的心得和意見如果您打算參加 22 日的 聚會,或打算分享您的心得 1st TWMUG meeting

    ×