Your SlideShare is downloading. ×
  • Like
MySQL Scaling Presentation
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

MySQL Scaling Presentation

  • 3,294 views
Published

 

Published in Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
3,294
On SlideShare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
226
Comments
0
Likes
5

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Strategic MySQL Planning for Complexity & Growth Or MySQL Scaling for Dummies Thomas Falgout Sr. Developer Yahoo!
  • 2. The Basics
    • Ways to squeeze extra performance:
  • 9. Query Optimizations
    • KNOW EVERY QUERY!
    • 10. Developers usually don't write proper SQL
    • 11. SQL Statements will directly affect your performance
    • 12. Example:
      • Repeating SQL Statements for no benefit
      • 13. 1000 very quick unnecessary queries vs 1 slow query
  • 14. Query Optimizations
    • Don't Develop in an island
      • Talk it over.
      • 15. Your coworkers might surprise you.
      • 16. In a good way.
  • 17. Query Optimizations - EXPLAIN
    • Only works for SELECT
    • 18. Quickly show you how much data your queries are processing
    • 19. Could spend an entire session on just this.
    • 20. But won't.
    • 21. Read for yourself.
    • 22. http://dev.mysql.com/doc/refman/5.1/en/explain.html
    • 23. http://dev.mysql.com/doc/refman/5.1/en/using-explain.html
    • 24. http://www.percona.com/files//presentations/EXPLAIN_demystified.pdf
    • 25. LUUUUUCCCYYYYYYY!!!!!
  • 26. Query Optimizations
    • SELECTS
      • SELECT * is bad
        • Hurts performance (cache, maintainability, bandwidth)
      • Joining two 100,000 row tables gets big
      • 27. “ SELECT COUNT(*) FROM users”
        • Performs full table scan on Innodb
        • 28. Faster on MyISAM (stored in table info)
  • 29. Query Optimizations
    • INSERTS
      • Can you batch INSERT?
      • 30. INSERT INTO a VALUES (1,23),(2,34),(4,33);
      • 31. To speed up INSERT operations that require multiple statements, LOCK your tables
      • 32. Can you use LOAD DATA? (Import via CSV)
  • 33. Query Optimizations
    • UPDATE
      • Updates are usually pretty good.
      • 34. Not much to say.
  • 35. Query Optimizations
    • DELETE
      • DELETES can be costly.
        • Sometimes causes DB to rebuild index
      • IDEAS
        • Add a “to_delete” column and then delete batch
        • 36. If having to delete A LOT of rows
          • SELECT INTO rows to keep into temp table
          • 37. DROP table
          • 38. RENAME tmp table to table
  • 39. Query Optimizations
    • Find queries that do COUNT(*) or create tables with results
      • Create tables with results
      • 40. Aggregate data hourly/daily/per update/etc.
  • 41. Query Optimization – Handling Logs
    • Don't talk directly to DB
    • 42. Send updates through daemon to load data
    • 43. Don't update for each request
      • Aggregate per 1000 req/2 min/app start
      • 44. UPDATE counts SET count = count + 42 WHERE id=1
  • 45. Schema Definition - MyISAM
    • Original Engine for MySQL
    • 46. Does not handle transactions
    • 47. Faster to Write, Slightly faster to Read
    • 48. Uses separate Data (.MYD), Index (.MYI) and Frame (.frm)
  • 49. Schema Definition - INNODB
    • 2 nd Engine for MySQL – From Oracle/Innobase
    • 50. Handles transactions
      • START TRANS; INSERT xxx; COMMIT
    • Uses one file for both data and index for entire DB (Can change in my.cnf to separate to table)
    • 51. Slower to write and read since it stores data per transaction.
      • Can be tuned to be almost as fast as MyISAM
  • 52. Schema Definition - BLACKHOLE
    • Fastest Engine available
    • 53. Writes Binary logs to disk
    • 54. Doesn't store data
    • 55. Useful for offloading replication from the master
    • 56. Blackhole Sun. Won't you come. Wash away my data.
  • 57. Schema Definitions
    • If you can, stick to one storage engine
    • 58. Falcon
      • New Transactional Engine
      • 59. http://forge.mysql.com/wiki/Falcon
  • 60. Schema Definition - Size
    • Make your data Small
    • 61. http://dev.mysql.com/doc/refman/5.1/en/data-size.html
    • 62. http://dev.mysql.com/doc/refman/5.0/en/storage-requirements.html
  • 63. Schema Definition – Size
    • TINYINT – 1 Byte
    • 64. SMALLINT – 2 Bytes
    • 65. MEDIUMINT – 3 Bytes
    • 66. INT – 4 Bytes
    • 67. BIGINT – 8 Bytes
    • 68. http://dev.mysql.com/doc/refman/5.0/en/numeric-types.html
  • 69. Schema Definition - Size
    • CHAR(N) = N + 1 byte. 2 bytes if N > 255 bytes
    • 70. VARCHAR(N) = X + 1 byte. X = Number of actual characters in string.
    • 71. N & X = 1 Byte. Unless using utf8 or a multi-byte format
    • 72. Same rules apply for BINARY & VARBINARY.
    • 73. Extra data is TRUNCATED!!!!!!
    • 74. That's bad.
    • 75. http://dev.mysql.com/doc/refman/5.0/en/string-types.html
  • 76. Schema Definition - VARCHAR
    • VARCHAR(255) is usually the sign of a careless design
    • 77. VARCHAR(255) is 4.x limitation
    • 78. When creating a temp table, it allocates the greatest size of data (255)
  • 79. Schema Definition - INT
    • BIGINT is not needed for AUTO_INCREMENT
    • 80. INT UNSIGNED = 4.3 Billion values.
      • If you hit this, you should be partitioning
    • Unless you have a reason, all of your INT columns should be UNSIGNED
      • Adds data integrity for negative values
  • 81. Schema Definition - Date
    • TIMESTAMP = 4 bytes
    • 82. DATETIME = 8 bytes
    • INT as UNIX Timestamp = 4 bytes
  • 85. Schema Definition – Storing IP's
    • IP = 4 Bytes
    • 86. 192.168.101.201 = 11000000.10101000.01100101.11001001
    • 87. INT UNSIGNED = 4 Bytes!
    • 88. INET_NTOA => NumberTOAddress
    • 89. INET_ATON => AddressTONumber
    • 90. Don't use VARCHAR(15)
      • Avg of 12 bytes vs 4 bytes
  • 91. Schema Definition – IP Addr
      mysql> SELECT INET_ATON('192.168.101.201'); +------------------------------+ | INET_ATON('192.168.101.201') | +------------------------------+ | 3232261577 | +------------------------------+ mysql> SELECT ... WHERE ip_col BETWEEN INET_ATON('12.13.14.15') AND INET_ATON('13.14.15.16')
  • 92. Schema Definition - ENUM
    • Use Codes for enum
      • m / f instead of male / female
    • Must alter table to add an enum
  • 93. Schema Definition - BLOB
    • BE WARY!
      • SELECT <field> will return THE WHOLE THING!
      • 94. Stores first as 768 Bytes, and then increments into 16k data pages per row per TEXT/BLOB field
  • 95. Hardware
    • DB typically require hot on RAM before CPU
    • 96. Separate your App and DB
      • Keep them close.
      • 97. Network latency affects experience
  • 98. Hardware
    • Split up READS vs WRITES
      • Create a Slave to replicate your DB to
      • 99. Send all READS to Slave
      • 100. Send all WRITES to Master
  • 101. Hardware – MySQL Cluster
    • Realtime Transactional DB Engine
    • 102. Provides HA w/ no Single Point of Failure
    • 103. Requires 3 servers (2 Data, 1 Mgmt Nodes)
    • 104. Inserts on one DB are replicated to others
    • 105. Selects can be performed on any DB
    • 106. http://dev.mysql.com/tech-resources/articles/mysql-cluster-for-two-servers.html
    • 107. http://www.lod.com/whitepapers/mysql-cluster-howto.html
  • 108. Software – MySQL Proxy
    • Program between your client and your DB
    • 109. Can be used for
    • http://forge.mysql.com/wiki/MySQL_Proxy
  • 113. Software – Maatkit
    • Series of Tools for monitoring & administration
      • Table Checksum (Compare Master/Slave)
      • 114. Table Sync (Resolve Differences)
      • 115. Visual Explain (Explain! Visually!)
      • 116. Heartbeat (Shows Slave Replication Delay)
      • 117. And Much, Much More!
    • http://www.maatkit.org/
  • 118. Scaling Reads
    • Get More Servers
    • 119. Point Slaves to Master
    • 120. Put Slaves Behind Load Balancer
  • 121. Scaling Writes
    • No Easy button
    • 122. Batch Insert / Update
    • 123. Don't Delete
    • 124. Partition your Data
    • 125. If you're at this point, this presentation probably won't help you.
      • Read Cal Henderson's Scaling MySQL
  • 126. MySQL Config
    • Do you know what those config variables do?
      • SHOW VARIABLES
      • http://dev.mysql.com/doc/refman/5.0/en/server-system-variables.html
  • 130. MySQL Config
    • key_buffer_size
      • MyISAM Only
      • 131. Size of cache for MyISAM Indexes
      • 132. Can check performance with SHOW STATUS
        • key_read_requests
        • 133. key_reads
        • 134. key_write_requests
        • 135. key_writes
        • 136. key_reads/key_read_requests < 0.01 == GOOD
  • 137. MySQL Config
    • table_cache / table_open_cache (5.1.3)
      • Number of open tables for all threads
      • 138. This dictates the number of open file descriptors
  • 139. MySQL Config
    • innodb_buffer_pool_size
      • INNODB only
      • 140. Size in bytes of the memory buffer InnoDB uses to cache data and indexes.
      • 141. Default 8MB
  • 142. MySQL Config
    • Query Cache
      • The query cache stores the text of a SELECT statement with the corresponding result that was sent to the client
      • 143. Great for tables that don't change often
      • 144. Doesn't return stale data
      • 145. Cache is invalidated if returned entry is modified
      • 146. Another reason why SELECT(*) is bad
  • 147. MySQL Config
    • Memory
      • INNODB – Let MySQL use all memory
      • 148. MyISAM – Leave more memory for OS page caches
  • 149. MySQL Config
    • MySQL Slow Log
      • Used to log any queries that take longer than x seconds
      • 150. Use long_query_time to specify x
      • 151. ~1 sec is good starting point
      • 152. http://dev.mysql.com/doc/refman/5.1/en/slow-query-log.html
  • 153. Caching - External
    • Memcache
      • Used by all the big players (Y!, Facebook, etc.)
      • 154. Originally created by LiveJournal to handle scale
      • 155. EASY Button
      • 156. $memcache_obj = memcache_connect(&quot;localhost&quot;, 11211);
      • 157. $memcache_obj->add('var_key', 'test variable');
      • 158. $var = $memcache_obj->get('var_key');
  • 159. Caching - Internal
    • Query Cache
      • http://dev.mysql.com/doc/refman/5.1/en/query-cache.html
    • key_buffer_size
      • http://dev.mysql.com/doc/refman/5.0/en/server-system-variables.html
    • innodb_buffer_pool_size
      • http://dev.mysql.com/doc/refman/5.0/en/innodb-parameters.htm
  • 160. Benchmarking - mysqlslap
    • Mysqlslap
    • 161. mysqlslap -u root -concurrency=10000 -iterations=5 -number-of-queries=10000 -query=”SELECT name FROM city WHERE city_id=12”
  • 162. Benchmarking – Apache Bench
    • Built for testing webservices
    • 163. Like mysqlslap
    • 164. GOOD NEWS
      • Benchmark whole web app
    • BAD NEWS
      • Benchmark whole web app
    • http://httpd.apache.org/docs/2.2/programs/ab.html
  • 165. Monitoring
    • Monitoring is as important as benchmarking
    • 166. Basics:
      • SHOW FULL PROCESSLIST
      • 167. SHOW GLOBAL STATUS
      • 168. SHOW GLOBAL VARIABLES
  • 169. Monitoring Tools
    • Cacti – RRDtool based networking graphing tool
    • 170. MUNIN – Similar to Cacti, but easier to maintain
    • 171. Nagios – More of a notification tool
    • 172. MySQL Enterprise Dashboard – Notification & Monitoring
  • 173. Other
    • Write Proper SQL – Naming Convention
      • customer_id, order_id NOT id
      • 174. Be descriptive
      • 175. Avoid reserve words (date, time, timestamp)
  • 176. Other
    • NEVER test on a production system
    • 177. Goal of test environment:
      • NOT to test your software
      • 178. Break your software.
  • 179. Other
    • SET sql_mode = 'STRICT_TRANS_TABLES'
      • Forces you to use good data
  • 180. The Most Important Lesson This part is absolutely important that you get. It's time to wake up.
  • 181. The Most Important Lesson When you need to, Ignore everything I've suggested.
  • 182. Thanks Thanks for listening.
  • 183. Links Flickr - Scaling MySQL http://www.scribd.com/doc/21593/Scaling-PHPMySQL Very Basic PHP & MySQL: http://www.cs.gsu.edu/~stefan/teaching/csc-6370/notes/phpmysql.pdf JPIPES! http://jpipes.com/presentations/joinfu/joinfu_part_one.pdf http://jpipes.com/presentations/joinfu/joinfu_part_two.pdf http://jpipes.com/presentations/dqm/legend_drunken_query_master.pdf EXPLAIN: http://www.percona.com/files//presentations/EXPLAIN_demystified.pdf Query Optimization: http://www.percona.com/files//presentations/Zurich2007-QueryOptimization.pdf Server Settings Tuning: http://www.percona.com/files//presentations/UC2007-MySQL-Server-Settings-Tuning.pdf http://www.flupps.org/mysql/MySQL_Microscope.ppt http://dev.mysql.com/doc/refman/5.0/en/working-with-null.html http://www.devshed.com/c/a/MySQL/MySQL-Server-Tuning-Tips-and-Tricks/ http://www.planetmysql.org/