Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

MariaDB Performance Tuning and Optimization

3,627 views

Published on

MariaDB Performance Tuning and Optimization

Published in: Data & Analytics

MariaDB Performance Tuning and Optimization

  1. 1. MariaDB Performance Tuning and Optimization Jonathan Day, MariaDB Solution Architect Shree Nair, Monyog Product Manager
  2. 2. Agenda • MariaDB Performance Tuning - Jon Day – Common Principles and Best Practices – Server Hardware and OS – MariaDB Configuration Settings – Database Design – Monitoring and Query Tuning • Monyog Presentation - Shree Nair • Q&A
  3. 3. Why Tune? ● Efficient Use of Hardware Resources ● Best Performance for Users ● Avoid Outages Due to Server Slowness ● Extra Capacity ○ Be Prepared for Application Development Requirements ○ Allow for Unexpected Traffic Spikes or Other Changes in Demand
  4. 4. Performance Common Principles and Best Practices
  5. 5. Tuning Routine - When to Tune • Tune from Start of the Application Lifecycle – Start Early to Ensure Schema is Well Constructed – Test Queries on Real Data — Watch for Bottlenecks – Over Tuning without Production Data or Traffic is Counter Productive • Conduct Periodic Reviews of Production Systems – Watch for Schema, Query and Significant Changes – Check Carefully New Application Features – Monitor System Resources — Disk, Memory, Network, CPU – Be Proactive in Solving Problems
  6. 6. Tuning Routine - When Not to Tune • Identify Objectives in Advance – Adhere to Objectives • Decide on Best Use of Resources – Tune Appropriately – Don't Excessively Tune • Be Aware of Data Integrity – Where is Speed Most Important? – Where is Integrity Most Important? – Adhere to these Boundaries
  7. 7. The my.cnf configuration file • Change one setting at a time... – This is the only way to determine if a change is beneficial. • Most settings can be changed at runtime with SET GLOBAL. – It is very handy and it allows you to quickly revert the change if needed. – To make the change permanent, you need to update the configuration file. • If a change in the configuration is not visible even after a MariaDB restart… – Did you use the correct configuration file? – Did you put the setting in the right section? •Normally the [mysqld] section for these settings
  8. 8. The my.cnf configuration file • The server refuses to start after a change: – Did you use the correct units? •For instance, innodb_buffer_pool_size should be set in bytes while max_connection is dimensionless. • Do not allow duplicate settings in the configuration file. – If you want to keep track of the changes, use version control. • Don’t do naive math, like “my new server has 2x RAM, I’ll just make all the values 2x the previous ones”.
  9. 9. Performance Server Hardware and OS Tuning
  10. 10. Server Hardware • One Service per Server is Ideal to Prevent Contention – Have the database server be only a database server etc. • More CPU Cores is generally Good • More Disk is usually Better – Large Datasets, Fast Disks are Ideal • More RAM is usually Best — Traffic Dependent – More of Dataset in Memory, Fewer Slow Disk Operations
  11. 11. OS Settings Linux Settings •Swappiness ○ Value for propensity of the OS to swap to disk ○ Defaults are usually 60 ○ Commonly set low to 10 or so (not 0) •Noatime ○ Mount disks with this option ○ Turns off writing of access time to disk with every file access ○ Without this option every read becomes an additional write
  12. 12. Which Filesystem to use? • The short answer is the best filesystem for a MariaDB server is ext4, XFS, or Btrfs. All are solid enterprise journaling filesystems that scale well from small to very large files and very large storage volumes. Filesystems XFS Ext4 Btrfs Maximum filesystem size 8EB 1EB 16EB Maximum file size 8EB 16TB 16EB
  13. 13. Performance MariaDB Configuration Settings
  14. 14. Configuration Settings innodb_buffer_pool_size •The first setting to update •The buffer pool is where data and indexes are cached • Utilize memory for read operations rather than disk •80% RAM rule of thumb •Typical values are ✓ 5-6GB (8GB RAM) ✓ 20-25GB (32GB RAM) ✓ 100-120GB (128GB RAM)
  15. 15. Configuration Settings query_cache_size ● Query cache is a well known bottleneck ● Consider setting query_cache_size = 0 ● Use other ways to speed up read queries: ○ Good indexing ○ Adding replicas to spread the read load
  16. 16. Configuration Settings innodb_log_file_size ● Size of the redo logs - 25 to 50% of innodb_buffer_pool usually recommended ● Redo logs are used to make sure writes are fast and durable and also during crash recovery ● Larger log files can lead to slower recovery in the event of a server crash ● But! Larger log files also reduce the number of checkpoints needed and reduce disk I/O
  17. 17. Configuration Settings innodb_file_per_table ● Each .ibd file represents a tablespace of its own. ● Database operations such as “TRUNCATE” can be completed faster and you may also reclaim unused space when dropping or truncating a database table. ● Allows some of the database tables to be kept in separate storage device. This can greatly improve the I/O load on your disks.
  18. 18. Configuration Settings Disable MySQL Reverse DNS Lookups ● MariaDB performs a DNS lookup of the user’s IP address and Hostname with connection ● The IP address is checked by resolving it to a host name. The hostname is then resolved to an IP to verify ● This allows DNS issues to cause delays ● You can disable and use IP addresses only ○ skip-name-resolve under [mysqld] in my.cnf
  19. 19. Configuration Settings max_connections •‘Too many connections’ error? •Using a connection pool at the application level or a thread pool at the MariaDB level can help
  20. 20. Configuration Settings Check for MySQL idle Connections ● Idle connections consume resources and should be interrupted or refreshed when possible. ● Idle connections are in “sleep” state and usually stay that way for long period of time. ● To look for idled connections: ● # mysqladmin processlist -u root -p | grep “Sleep” ● You can check the code for the cause if many idled ● You can also change the wait_timeout value
  21. 21. Configuration Settings thread_cache_size ● The thread_cache_size directive sets the amount of threads that your server should cache. ● To find the thread cache hit rate, you can use the following technique: ○ show status like 'Threads_created'; ○ show status like 'Connections'; ● calculate the thread cache hit rate percentage: ○ 100 - ((Threads_created / Connections) * 100) ● Dynamically set to a new value: ○ set global thread_cache_size = 16;
  22. 22. Configuration Settings memory parameters ● MariaDB uses temporary tables when processing complex queries involving joins and sorting ● The default size of a temporary table is very small ○ The size is configured in your my.cnf: tmp-table-size = 1G max-heap-table-size = 1G ● Both should have the same size and will help prevent disk writes ● A rule of thumb is giving 64Mb for every GB of RAM on the server
  23. 23. Configuration Settings Buffer Sizes ● join buffer size ○ used to process joins – but only full joins on which no keys are possible ● sort buffer size ○ Sort buffer size is used to sort data. ○ The system status variable sort_merge_passes will indicates need to increase ○ This variable should be as low as possible. ● These buffers are allocated per connection and play a significant role in the performance of the system.
  24. 24. Configuration Settings max_allowed_packet ● MariaDB splits data into packets. Usually a single packet is considered a row that is sent to a client. ● The max_allowed_packet directive defines the maximum size of packet that can be sent. ● Setting this value too low can cause a query to stall and you will receive an error in your error log. ● It is recommended to set the value to the size of your largest packet. ○ Some suggest 11 times the largest BLOB
  25. 25. Performance Database Design
  26. 26. Database Design Choosing Data Types ● Use Appropriate Data Type (INT for Numbers, VARCHAR) ● Use Smallest Useful Type ● Variable Length Fields are often Padded ● Use NOT NULL, where Practical ○ A NULL field uses slightly More Disk and Memory (Depends on Storage Engine) ● Use PROCEDURE ANALYSE( )
  27. 27. Database Design Keys and Indexes ● Make PRIMARY KEY as Small as Practical ○ InnoDB Stores Copy of PRIMARY KEY with each Index Entry ● Secondary Indexes Grow with PRIMARY KEY ● Use Surrogate PRIMARY KEY , Not Natural Ones ● Consider Partial (Prefix) Index for String Indexes
  28. 28. Advantages • Included in Backup and Replication • Guaranteed to be Consistent with Other Data Disadvantages • Database gets Too Large • Sometimes Best to Store Files in the File System and the Filename and Path in Table Database Design: File Storage in BLOB
  29. 29. Database Design Tuning Tables Overall ● Minimize Table Size on Disk and in Memory ● Archive Table Data if Possible and Appropriate ● Remove Duplicate or Unused Indexes ● Use Appropriate Data Types — Smaller is Better ● Consider Sharding Large Tables across Multiple Servers
  30. 30. Normalizing • Makes a Data Set as Smaller by Eliminating Redundant Data • Ensures Data Set Consistency and Integrity • Often Improves Concurrency by Reducing Locking • Increases Number of Tables and Associated Maintenance Denormalizing • Complete Normalization can Slow Queries • More Complex JOIN Queries Harder to Maintain • Denormalization Adds Redundant Data to Schema • Write Queries become More Complex or Numerous as Multiple Locations must be Maintained • It's Easier to Normalize First, Then Denormalize when Appropriate Database Design: Normalization of Tables
  31. 31. Storage Engines ● XtraDB is the best choice in the majority of cases. It is a performance-enhanced fork of InnoDB and is the MariaDB default engine until MariaDB 10.1. ● InnoDB is a good general transactional storage engine. It is the default MySQL storage engine, and default MariaDB 10.2 storage engine, but in earlier releases XtraDB is a performance enhanced fork of InnoDB, and is usually preferred. ● Aria, MariaDB's more modern improvement on MyISAM, has a small footprint and allows for easy copying between systems. ● MyISAM has a small footprint and allows for easy copying between systems. MyISAM is MySQL's oldest storage engine. There is usually little reason to use it except for legacy purposes. Aria is MariaDB's more modern improvement. ● Spider uses partitioning to provide data sharding through multiple servers. ● ColumnStore utilizes a massively parallel distributed data architecture and is designed for big data scaling to process petabytes of ● MyRocks enables greater compression than InnoDB, as well as less write amplification giving better endurance of flash storage and improving overall throughput. (Currently Alpha in MariaDB 10.2)
  32. 32. Performance Monitoring and Query Tuning
  33. 33. Monitoring and Query Tuning Enable Slow query Logs Logging slow queries can help you determine issues with your database and help you debug them. This can be easily enabled by adding the following values in your my.cnf configuration file: slow-query-log = 1 slow-query-log-file = /var/lib/mysql/mysql-slow.log long_query_time = 1 The first directive enables the logging of slow queries, while the second one tells MariaDB where to store the actual log file. Use long_query_time to define the amount of time that is considered long for a MariaDB query to be completed.
  34. 34. EXPLAIN SELECT * FROM employees WHERE MONTH(birth_date) = 8 G id: 1 select_type: SIMPLE table: employees type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 299587 Extra: Using where Monitoring and Query Tuning Query Analysis ● Use the Slow Log to find Problem Queries ● Use mysqldumpslow Utility for Manageable Reports ● Use EXPLAIN to see how MariaDB Executes a Troublesome Query and if Indexes are Used ● Use EXPLAIN EXTENDED and SHOW WARNINGS to see how MariaDB Rearranges a Query before Execution EXPLAIN SELECT * FROM employees WHERE MONTH(birth_date) = 8 G id: 1 select_type: SIMPLE table: employees type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 299587 Extra: Using where
  35. 35. EXPLAIN SELECT * FROM employees WHERE MONTH(birth_date) = 8 G id: 1 select_type: SIMPLE table: employees type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 299587 Extra: Using where Monitoring and Query Tuning Query Tuning Overview ● Try Not to Query Tune on Production Server ● Use Test Server with same Hardware and OS ● Use Copy of Production Data Set ● Query Frequency is as Important as Query Speed ○ Moderately Slow Queries are often a Bigger Problem than a Rarely Run Very Slow Query
  36. 36. EXPLAIN SELECT * FROM employees WHERE MONTH(birth_date) = 8 G id: 1 select_type: SIMPLE table: employees type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 299587 Extra: Using where Monitoring and Query Tuning Indexing ● Indexes Improve Read Performance ● Without Index, MariaDB Must Read Every Row — Full Table Scan ● With Index, MariaDB can jump to Requested Rows ● Reduced I/O and Improving Performance ● Index Increase cost of Writes ● Find Balance ● Index for Speed, but Avoid Indexing Excessively or Arbitrarily ● Remove Dead or Redundant Indexes
  37. 37. EXPLAIN SELECT * FROM employees WHERE MONTH(birth_date) = 8 G id: 1 select_type: SIMPLE table: employees type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 299587 Extra: Using where Monitoring and Query Tuning Index Size ● Keep Indexes as Small as Practical ○ Faster since More Likely to Fit in Memory ○ Rebuilds Faster after Writes ○ PRIMARY KEY should be Minimum Useful Size ● Use Partial Prefix Indexes for String Columns ○ May Slow Searches Slightly, but Reduce Index Size ● Use Index Cardinality (Uniqueness Measure) Only If Necessary — Re-evaluate as Data Grows ○ Low Cardinality Indicates many Duplicates ○ High Cardinality is More Useful
  38. 38. EXPLAIN SELECT * FROM employees WHERE MONTH(birth_date) = 8 G id: 1 select_type: SIMPLE table: employees type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 299587 Extra: Using where Monitoring and Query Tuning Tools & Statistics ● Identify Accurately and Carefully Trouble Spots ○ Guessing is Rarely Useful ● Gather Performance Stats with MariaDB and OS Tools ○ SHOW Statements ○ PERFORMANCE_SCHEMA ○ CPU, Disk, Network, Memory, & Swap Stats ● Retain Snapshots of Multiple Stats ○ Data from a Single Point Shows very Little ● Automate the Collection of Stats into Logs ○ Can be Useful for Emergency Tuning
  39. 39. EXPLAIN SELECT * FROM employees WHERE MONTH(birth_date) = 8 G id: 1 select_type: SIMPLE table: employees type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 299587 Extra: Using where Monitoring and Query Tuning SHOW PROCESSLIST ● Snapshot of mysqld Activity ● mysqld is Multi-Threaded, One Thread per ○ Client Connection (i.e., query, transaction) — a "process" is a "thread" ● Accumulate SHOW PROCESSLIST Snapshots to build History of Thread Activities
  40. 40. EXPLAIN SELECT * FROM employees WHERE MONTH(birth_date) = 8 G id: 1 select_type: SIMPLE table: employees type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 299587 Extra: Using where Monitoring and Query Tuning SHOW STATUS Global or Session ● Returns List of Internal Counters ● GLOBAL for System-Wide Status — Since Start-Up ● SESSION for Local to Client Connection ● FLUSH STATUS Resets Local Counters ● Monitor Changes to Counters to Identify Hot Spots ● Collect Periodically Status Snapshots to Profile Traffic
  41. 41. EXPLAIN SELECT * FROM employees WHERE MONTH(birth_date) = 8 G id: 1 select_type: SIMPLE table: employees type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 299587 Extra: Using where Monitoring and Query Tuning PERFORMANCE_SCHEMA ● Similar to INFORMATION_SCHEMA , but Performance Tuning ● Monitors MariaDB Server Events ● Function Calls, Operating System Waits, Internal Mutexes, I/O Calls ● Detailed Query Execution Stages (Parsing, Statistics, Sorting) ● Some Features Storage Engine Specific ● Monitoring Lightweight and Requires No Dedicated Thread ● Designed to be Used Iteratively with Successive Refinement
  42. 42. Monitoring and Query Tuning Monitoring Tools Monyog - Agentless and Cost-effective MySQL monitoring tool Box Anemometer - a MariaDB Slow Query Monitor. This tool is used to analyze slow query logs collected from MariaDB instances to identify problematic queries pt-query-digest - Analyzes MariaDB queries from logs, processlist, and tcpdump
  43. 43. Agentless and Cost-effective MariaDB monitoring tool.
  44. 44. Monitoring databases should be easy Avoid Visibility Gap Across Servers Get a bird’s-eye view of your data tier, keep up with the production demands and avoid the visibility gap. Achieve Faster Issue Resolution Time Uncover problematic queries and performance problems in complex data layer. Industry-Leading Replication Overview View the replication hierarchy of servers along with the details of each replicated server.
  45. 45. SQLITE REPOSITORY MYSQL DATA & OS METRICS COLLECTOR DATA AGGREGATOR MONYOG WEB CLIENT TIMELY ALERTS VIA MAIL (SMTP) & TRAPS (SNMP) MONYOG MYSQL/ MARIADB CLOUDBASED MYSQL/ MARIADB DATA, LOGS OS METRICS SFTP / MARIADB C-CONNECTOR MARIADB C-CONNECTOR / REST API / CLOUDWATCH API DATA, LOGS OS METRICS Monyog Architecture MYSQL DATA & OS METRICS COLLECTOR
  46. 46. Benefits of Monyog •On-premise solution. Secure. Scalable. •600+ monitors and advisors. •Minuscule Performance Overhead. •Saves tons of time on a daily basis. •Avoids critical visibility gap across MySQL servers. •Faster Issue Resolution Time.
  47. 47. Why Monyog is a need-to-have? •Proactively prevents problems before customers notice. •Ensures great performance of mission-critical applications. •No deep database expertise needed to use Monyog. •Makes MariaDB performance tuning easy.
  48. 48. Hardware requirement • Small & medium size deployments • Double core CPU with 4GB of memory • 5GB of storage space per server registered with Monyog • Large size deployments (>500 servers) • 16 core CPU with 32GB of memory • 1.5 TB of storage
  49. 49. How can you try out Monyog? •Free trial available on www.monyog.com •Monyog documentation - http://monyogkb.webyog.com/ •Included in a MariaDB Enterprise Subscription. •Contact us at sales@webyog.com
  50. 50. Thank you Jonathan.Day@mariadb.com.
  51. 51. Q&A

×