MySQL Scaling Presentation

3,454
-1

Published on

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,454
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
235
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

MySQL Scaling Presentation

  1. 1. Strategic MySQL Planning for Complexity & Growth Or MySQL Scaling for Dummies Thomas Falgout Sr. Developer Yahoo!
  2. 2. The Basics <ul><li>Ways to squeeze extra performance: </li><ul><li>Query Optimization
  3. 3. Schema Definition
  4. 4. Hardware
  5. 5. Software
  6. 6. MySQL Config
  7. 7. Cache
  8. 8. Monitoring </li></ul></ul>
  9. 9. Query Optimizations <ul><li>KNOW EVERY QUERY!
  10. 10. Developers usually don't write proper SQL
  11. 11. SQL Statements will directly affect your performance
  12. 12. Example: </li><ul><li>Repeating SQL Statements for no benefit
  13. 13. 1000 very quick unnecessary queries vs 1 slow query </li></ul></ul>
  14. 14. Query Optimizations <ul><li>Don't Develop in an island </li><ul><li>Talk it over.
  15. 15. Your coworkers might surprise you.
  16. 16. In a good way. </li></ul></ul>
  17. 17. Query Optimizations - EXPLAIN <ul><li>Only works for SELECT
  18. 18. Quickly show you how much data your queries are processing
  19. 19. Could spend an entire session on just this.
  20. 20. But won't.
  21. 21. Read for yourself.
  22. 22. http://dev.mysql.com/doc/refman/5.1/en/explain.html
  23. 23. http://dev.mysql.com/doc/refman/5.1/en/using-explain.html
  24. 24. http://www.percona.com/files//presentations/EXPLAIN_demystified.pdf
  25. 25. LUUUUUCCCYYYYYYY!!!!! </li></ul>
  26. 26. Query Optimizations <ul><li>SELECTS </li><ul><li>SELECT * is bad </li><ul><li>Hurts performance (cache, maintainability, bandwidth) </li></ul><li>Joining two 100,000 row tables gets big
  27. 27. “ SELECT COUNT(*) FROM users” </li><ul><li>Performs full table scan on Innodb
  28. 28. Faster on MyISAM (stored in table info) </li></ul></ul></ul>
  29. 29. Query Optimizations <ul><li>INSERTS </li><ul><li>Can you batch INSERT?
  30. 30. INSERT INTO a VALUES (1,23),(2,34),(4,33);
  31. 31. To speed up INSERT operations that require multiple statements, LOCK your tables
  32. 32. Can you use LOAD DATA? (Import via CSV) </li></ul></ul>
  33. 33. Query Optimizations <ul><li>UPDATE </li><ul><li>Updates are usually pretty good.
  34. 34. Not much to say. </li></ul></ul>
  35. 35. Query Optimizations <ul><li>DELETE </li><ul><li>DELETES can be costly. </li><ul><li>Sometimes causes DB to rebuild index </li></ul><li>IDEAS </li><ul><li>Add a “to_delete” column and then delete batch
  36. 36. If having to delete A LOT of rows </li><ul><li>SELECT INTO rows to keep into temp table
  37. 37. DROP table
  38. 38. RENAME tmp table to table </li></ul></ul></ul></ul>
  39. 39. Query Optimizations <ul><li>Find queries that do COUNT(*) or create tables with results </li><ul><li>Create tables with results
  40. 40. Aggregate data hourly/daily/per update/etc. </li></ul></ul>
  41. 41. Query Optimization – Handling Logs <ul><li>Don't talk directly to DB
  42. 42. Send updates through daemon to load data
  43. 43. Don't update for each request </li><ul><li>Aggregate per 1000 req/2 min/app start
  44. 44. UPDATE counts SET count = count + 42 WHERE id=1 </li></ul></ul>
  45. 45. Schema Definition - MyISAM <ul><li>Original Engine for MySQL
  46. 46. Does not handle transactions
  47. 47. Faster to Write, Slightly faster to Read
  48. 48. Uses separate Data (.MYD), Index (.MYI) and Frame (.frm) </li></ul>
  49. 49. Schema Definition - INNODB <ul><li>2 nd Engine for MySQL – From Oracle/Innobase
  50. 50. Handles transactions </li><ul><li>START TRANS; INSERT xxx; COMMIT </li></ul><li>Uses one file for both data and index for entire DB (Can change in my.cnf to separate to table)
  51. 51. Slower to write and read since it stores data per transaction. </li><ul><li>Can be tuned to be almost as fast as MyISAM </li></ul></ul>
  52. 52. Schema Definition - BLACKHOLE <ul><li>Fastest Engine available
  53. 53. Writes Binary logs to disk
  54. 54. Doesn't store data
  55. 55. Useful for offloading replication from the master
  56. 56. Blackhole Sun. Won't you come. Wash away my data. </li></ul>
  57. 57. Schema Definitions <ul><li>If you can, stick to one storage engine
  58. 58. Falcon </li><ul><li>New Transactional Engine
  59. 59. http://forge.mysql.com/wiki/Falcon </li></ul></ul>
  60. 60. Schema Definition - Size <ul><li>Make your data Small
  61. 61. http://dev.mysql.com/doc/refman/5.1/en/data-size.html
  62. 62. http://dev.mysql.com/doc/refman/5.0/en/storage-requirements.html </li></ul>
  63. 63. Schema Definition – Size <ul><li>TINYINT – 1 Byte
  64. 64. SMALLINT – 2 Bytes
  65. 65. MEDIUMINT – 3 Bytes
  66. 66. INT – 4 Bytes
  67. 67. BIGINT – 8 Bytes
  68. 68. http://dev.mysql.com/doc/refman/5.0/en/numeric-types.html </li></ul>
  69. 69. Schema Definition - Size <ul><li>CHAR(N) = N + 1 byte. 2 bytes if N > 255 bytes
  70. 70. VARCHAR(N) = X + 1 byte. X = Number of actual characters in string.
  71. 71. N & X = 1 Byte. Unless using utf8 or a multi-byte format
  72. 72. Same rules apply for BINARY & VARBINARY.
  73. 73. Extra data is TRUNCATED!!!!!!
  74. 74. That's bad.
  75. 75. http://dev.mysql.com/doc/refman/5.0/en/string-types.html </li></ul>
  76. 76. Schema Definition - VARCHAR <ul><li>VARCHAR(255) is usually the sign of a careless design
  77. 77. VARCHAR(255) is 4.x limitation
  78. 78. When creating a temp table, it allocates the greatest size of data (255) </li></ul>
  79. 79. Schema Definition - INT <ul><li>BIGINT is not needed for AUTO_INCREMENT
  80. 80. INT UNSIGNED = 4.3 Billion values. </li><ul><li>If you hit this, you should be partitioning </li></ul><li>Unless you have a reason, all of your INT columns should be UNSIGNED </li><ul><li>Adds data integrity for negative values </li></ul></ul>
  81. 81. Schema Definition - Date <ul><li>TIMESTAMP = 4 bytes
  82. 82. DATETIME = 8 bytes </li><ul><li>DATE = 3 bytes
  83. 83. TIME = 3 bytes
  84. 84. ??? </li></ul><li>INT as UNIX Timestamp = 4 bytes </li></ul>
  85. 85. Schema Definition – Storing IP's <ul><li>IP = 4 Bytes
  86. 86. 192.168.101.201 = 11000000.10101000.01100101.11001001
  87. 87. INT UNSIGNED = 4 Bytes!
  88. 88. INET_NTOA => NumberTOAddress
  89. 89. INET_ATON => AddressTONumber
  90. 90. Don't use VARCHAR(15) </li><ul><li>Avg of 12 bytes vs 4 bytes </li></ul></ul>
  91. 91. Schema Definition – IP Addr <ul>mysql> SELECT INET_ATON('192.168.101.201'); +------------------------------+ | INET_ATON('192.168.101.201') | +------------------------------+ | 3232261577 | +------------------------------+ mysql> SELECT ... WHERE ip_col BETWEEN INET_ATON('12.13.14.15') AND INET_ATON('13.14.15.16') </ul>
  92. 92. Schema Definition - ENUM <ul><li>Use Codes for enum </li><ul><li>m / f instead of male / female </li></ul><li>Must alter table to add an enum </li></ul>
  93. 93. Schema Definition - BLOB <ul><li>BE WARY! </li><ul><li>SELECT <field> will return THE WHOLE THING!
  94. 94. Stores first as 768 Bytes, and then increments into 16k data pages per row per TEXT/BLOB field </li></ul></ul>
  95. 95. Hardware <ul><li>DB typically require hot on RAM before CPU
  96. 96. Separate your App and DB </li><ul><li>Keep them close.
  97. 97. Network latency affects experience </li></ul></ul>
  98. 98. Hardware <ul><li>Split up READS vs WRITES </li><ul><li>Create a Slave to replicate your DB to
  99. 99. Send all READS to Slave
  100. 100. Send all WRITES to Master </li></ul></ul>
  101. 101. Hardware – MySQL Cluster <ul><li>Realtime Transactional DB Engine
  102. 102. Provides HA w/ no Single Point of Failure
  103. 103. Requires 3 servers (2 Data, 1 Mgmt Nodes)
  104. 104. Inserts on one DB are replicated to others
  105. 105. Selects can be performed on any DB
  106. 106. http://dev.mysql.com/tech-resources/articles/mysql-cluster-for-two-servers.html
  107. 107. http://www.lod.com/whitepapers/mysql-cluster-howto.html </li></ul>
  108. 108. Software – MySQL Proxy <ul><li>Program between your client and your DB
  109. 109. Can be used for </li><ul><li>Load Balancing
  110. 110. Failover
  111. 111. Query Analysis
  112. 112. Query Filtering / Modification </li></ul><li>http://forge.mysql.com/wiki/MySQL_Proxy </li></ul>
  113. 113. Software – Maatkit <ul><li>Series of Tools for monitoring & administration </li><ul><li>Table Checksum (Compare Master/Slave)
  114. 114. Table Sync (Resolve Differences)
  115. 115. Visual Explain (Explain! Visually!)
  116. 116. Heartbeat (Shows Slave Replication Delay)
  117. 117. And Much, Much More! </li></ul><li>http://www.maatkit.org/ </li></ul>
  118. 118. Scaling Reads <ul><li>Get More Servers
  119. 119. Point Slaves to Master
  120. 120. Put Slaves Behind Load Balancer </li></ul>
  121. 121. Scaling Writes <ul><li>No Easy button
  122. 122. Batch Insert / Update
  123. 123. Don't Delete
  124. 124. Partition your Data
  125. 125. If you're at this point, this presentation probably won't help you. </li><ul><li>Read Cal Henderson's Scaling MySQL </li></ul></ul>
  126. 126. MySQL Config <ul><li>Do you know what those config variables do? </li><ul><li>SHOW VARIABLES </li><ul><li>key_buffer_size
  127. 127. table_cache
  128. 128. innodb_buffer_pool_size
  129. 129. query_cache_limit </li></ul><li>http://dev.mysql.com/doc/refman/5.0/en/server-system-variables.html </li></ul></ul>
  130. 130. MySQL Config <ul><li>key_buffer_size </li><ul><li>MyISAM Only
  131. 131. Size of cache for MyISAM Indexes
  132. 132. Can check performance with SHOW STATUS </li><ul><li>key_read_requests
  133. 133. key_reads
  134. 134. key_write_requests
  135. 135. key_writes
  136. 136. key_reads/key_read_requests < 0.01 == GOOD </li></ul></ul></ul>
  137. 137. MySQL Config <ul><li>table_cache / table_open_cache (5.1.3) </li><ul><li>Number of open tables for all threads
  138. 138. This dictates the number of open file descriptors </li></ul></ul>
  139. 139. MySQL Config <ul><li>innodb_buffer_pool_size </li><ul><li>INNODB only
  140. 140. Size in bytes of the memory buffer InnoDB uses to cache data and indexes.
  141. 141. Default 8MB </li></ul></ul>
  142. 142. MySQL Config <ul><li>Query Cache </li><ul><li>The query cache stores the text of a SELECT statement with the corresponding result that was sent to the client
  143. 143. Great for tables that don't change often
  144. 144. Doesn't return stale data
  145. 145. Cache is invalidated if returned entry is modified
  146. 146. Another reason why SELECT(*) is bad </li></ul></ul>
  147. 147. MySQL Config <ul><li>Memory </li><ul><li>INNODB – Let MySQL use all memory
  148. 148. MyISAM – Leave more memory for OS page caches </li></ul></ul>
  149. 149. MySQL Config <ul><li>MySQL Slow Log </li><ul><li>Used to log any queries that take longer than x seconds
  150. 150. Use long_query_time to specify x
  151. 151. ~1 sec is good starting point
  152. 152. http://dev.mysql.com/doc/refman/5.1/en/slow-query-log.html </li></ul></ul>
  153. 153. Caching - External <ul><li>Memcache </li><ul><li>Used by all the big players (Y!, Facebook, etc.)
  154. 154. Originally created by LiveJournal to handle scale
  155. 155. EASY Button
  156. 156. $memcache_obj = memcache_connect(&quot;localhost&quot;, 11211);
  157. 157. $memcache_obj->add('var_key', 'test variable');
  158. 158. $var = $memcache_obj->get('var_key'); </li></ul></ul>
  159. 159. Caching - Internal <ul><li>Query Cache </li><ul><li>http://dev.mysql.com/doc/refman/5.1/en/query-cache.html </li></ul><li>key_buffer_size </li><ul><li>http://dev.mysql.com/doc/refman/5.0/en/server-system-variables.html </li></ul><li>innodb_buffer_pool_size </li><ul><li>http://dev.mysql.com/doc/refman/5.0/en/innodb-parameters.htm </li></ul></ul>
  160. 160. Benchmarking - mysqlslap <ul><li>Mysqlslap
  161. 161. mysqlslap -u root -concurrency=10000 -iterations=5 -number-of-queries=10000 -query=”SELECT name FROM city WHERE city_id=12” </li></ul>
  162. 162. Benchmarking – Apache Bench <ul><li>Built for testing webservices
  163. 163. Like mysqlslap
  164. 164. GOOD NEWS </li><ul><li>Benchmark whole web app </li></ul><li>BAD NEWS </li><ul><li>Benchmark whole web app </li></ul><li>http://httpd.apache.org/docs/2.2/programs/ab.html </li></ul>
  165. 165. Monitoring <ul><li>Monitoring is as important as benchmarking
  166. 166. Basics: </li><ul><li>SHOW FULL PROCESSLIST
  167. 167. SHOW GLOBAL STATUS
  168. 168. SHOW GLOBAL VARIABLES </li></ul></ul>
  169. 169. Monitoring Tools <ul><li>Cacti – RRDtool based networking graphing tool
  170. 170. MUNIN – Similar to Cacti, but easier to maintain
  171. 171. Nagios – More of a notification tool
  172. 172. MySQL Enterprise Dashboard – Notification & Monitoring </li></ul>
  173. 173. Other <ul><li>Write Proper SQL – Naming Convention </li><ul><li>customer_id, order_id NOT id
  174. 174. Be descriptive
  175. 175. Avoid reserve words (date, time, timestamp) </li></ul></ul>
  176. 176. Other <ul><li>NEVER test on a production system
  177. 177. Goal of test environment: </li><ul><li>NOT to test your software
  178. 178. Break your software. </li></ul></ul>
  179. 179. Other <ul><li>SET sql_mode = 'STRICT_TRANS_TABLES' </li><ul><li>Forces you to use good data </li></ul></ul>
  180. 180. The Most Important Lesson This part is absolutely important that you get. It's time to wake up.
  181. 181. The Most Important Lesson When you need to, Ignore everything I've suggested.
  182. 182. Thanks Thanks for listening.
  183. 183. Links Flickr - Scaling MySQL http://www.scribd.com/doc/21593/Scaling-PHPMySQL Very Basic PHP & MySQL: http://www.cs.gsu.edu/~stefan/teaching/csc-6370/notes/phpmysql.pdf JPIPES! http://jpipes.com/presentations/joinfu/joinfu_part_one.pdf http://jpipes.com/presentations/joinfu/joinfu_part_two.pdf http://jpipes.com/presentations/dqm/legend_drunken_query_master.pdf EXPLAIN: http://www.percona.com/files//presentations/EXPLAIN_demystified.pdf Query Optimization: http://www.percona.com/files//presentations/Zurich2007-QueryOptimization.pdf Server Settings Tuning: http://www.percona.com/files//presentations/UC2007-MySQL-Server-Settings-Tuning.pdf http://www.flupps.org/mysql/MySQL_Microscope.ppt http://dev.mysql.com/doc/refman/5.0/en/working-with-null.html http://www.devshed.com/c/a/MySQL/MySQL-Server-Tuning-Tips-and-Tricks/ http://www.planetmysql.org/
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×