Howmysqlworks

761 views

Published on

How MySQL works

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
761
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
12
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Howmysqlworks

  1. 1. MySQL stuff
  2. 2. About me • Before REA I worked for a hosting company for 4 years • Learnt a lot about MySQL during this period • I started as an OPS engineer at REA and did some database stuff here • I havent done much MySQL for two years ! • In this period we’ve had • MySQL 5.6 come out • RDS MySQL improvements ! !
  3. 3. Today • High level overview of MySQL • Look inside MySQL server • Talk through how InnoDB performs queries • Schema changes and online schema changes ! !
  4. 4. Background • Second biggest relational database in the world • Anyone know what the biggest is? ! • Used by many of the big web shops • Facebook for all feed data • Etsy for user & data • Twitter for tweet persistence • … ! • Sun bought MySQL AB in 2008, people freaked out • Oracle bought Sun in 2010, people freaked out • Big improvements over the past few years (5.5 / 5.6) !
  5. 5. How it works mysqld mysql client tcp:3306 filesocket ServerClient libmysql mysql lib ORM host$ mysql host$ mysqld_safe! host$ mysqld
  6. 6. Big features • Pluggable storage engines is massive flexibility • You can use a different storage engine per table ! ! • Common storage engines • InnoDB - ACID compliant store • MyISAM - one of the original storage engines - avoid • Lots and lots of others !
  7. 7. Inside mysqld mysqld storage engine(s) MyISAM / InnoDB / Memory / etc tcp:3306 filesocket Getting the query and responding bits and bytes stuff API
  8. 8. Inside mysqld mysqld storage engine(s) MyISAM / InnoDB / Memory / etc tcp:3306 filesocket API
  9. 9. Inside mysqld mysqld storage engine(s) MyISAM / InnoDB / Memory / etc tcp:3306 filesocket Parse query and convert to SE API calls. API
  10. 10. Inside mysqld mysqld Storage engines(s) tcp:3306 filesocket API query cache binary log join / sort buffers Binary Log for replication Buffers: Kinda how much batching to storage API to save cost
  11. 11. The bad parts - binary log mysqld tcp:3306 filesocket query cache binary log join / sort buffers Because you can mix and match storage engines, replicating state has to be done at the lowest common denominator Innodb Table 1 MyISAM Table 2
  12. 12. The bad parts - binary log mysqld Innodb Table 1 tcp:3306 filesocket query cache binary log join / sort buffers Highly concurrent InnoDB needs its queries, serialized down to replicate to slave MyISAM Table 2
  13. 13. Compare to other DBs DB server tcp:x filesocket? performance knobs transaction log
  14. 14. The bad parts - query cache mysqld Innodb Table 1 tcp:3306 filesocket query cache binary log join / sort buffers Can only work on an entire table level & uses LRU MyISAM Table 2
  15. 15. The bad parts - query cache mysqld Innodb Table 1 tcp:3306 filesocket query cache binary log join / sort buffers All ‘update/insert/delete’ blocks all transactions to ensure correct state. It actually slows ! things down MyISAM Table 2
  16. 16. Inside mysqld mysqld storage engine(s) MyISAM / InnoDB / Memory / etc tcp:3306 filesocket API Where 10%* of ! the work is done Where 90% ! of the work ! is done *Finger in the air calculation
  17. 17. Summary • Turn off query cache! It is hurting you! • Binary log is a necessary evil for replication • Dont play with the performance knobs (sort buffer / join buffer / etc) • Turning the 10% - Focus on the 90% • Use InnoDB for all the tables (unless they’re system tables) ! ! • So…lets look at the 90% ! !
  18. 18. InnoDB mysqld InnoDB tcp:3306 filesocket API
  19. 19. Unknown gems mysqld InnoDB tcp:3306 filesocket API handlersocket/ memcache Innodb
  20. 20. Unknown gems mysqld InnoDB tcp:3306 filesocket API handlersocket/ memcache Innodb Memcache API that talks directly to InnoDB ! Removes the SQL layer and is just K/V ! Google: handlersocket memcache InnoDB
  21. 21. Key parts of InnoDB Innodb API
  22. 22. Key parts of InnoDB Table space (whats on disk) redo log Innodb bufferpool memory! ib_log1! ib_log2! sometable.ibd! ib_data!
  23. 23. Key parts of InnoDB Table space redo log bufferpool • Each block is a page • Just think of it as a row with some extra stuff like version number • InnoDB is ACID meaning each connection has to have its own view of the world! • Extra metadata at the top about where everything is
  24. 24. Key parts of InnoDB Table space redo log bufferpool • Black is the base metadata
  25. 25. What is bufferpool Table space redo log bufferpool • In memory version of hot tablespace pages
  26. 26. What is redo log Table space redo log bufferpool • An append only log file of changes
  27. 27. What is table space Table space redo log bufferpool • The raw files - but it may not be always up-to-date
  28. 28. Lets run some queries Table space redo log bufferpool • First - lets fill up our database with data
  29. 29. Lets run some queries Table space redo log bufferpool • mysql> select * from red;
  30. 30. Lets run some queries Table space redo log bufferpool • mysql> select * from red; mysqld parses the query and sends ! API calls to InnoDB
  31. 31. Lets run some queries Table space redo log bufferpool • mysql> select * from red; Is red in bufferpool?
  32. 32. Lets run some queries Table space redo log bufferpool • mysql> select * from red; Load red into buffer pool perform random ! read disk activity! At this point, the time to respond! is dependent on disk speed!
  33. 33. Lets run some queries Table space redo log bufferpool • mysql> select * from red; Respond to API calls to mysqld
  34. 34. Lets run some queries Table space redo log bufferpool • mysql> select * from red; Parse results ! second pass sort if necessary ! return to client
  35. 35. Lets run some queries Table space redo log bufferpool • mysql> select * from red; #again
  36. 36. Lets run some queries Table space redo log bufferpool • mysql> select * from red; mysqld parses the query and sends ! API calls to InnoDB
  37. 37. Lets run some queries Table space redo log bufferpool • mysql> select * from red; Is red in bufferpool? - yes
  38. 38. Lets run some queries Table space redo log bufferpool • mysql> select * from red; Respond to API calls to mysqld
  39. 39. Lets run some queries Table space redo log bufferpool • mysql> select * from red; Parse results ! second pass sort if necessary ! return to client
  40. 40. Lets run some queries Table space redo log bufferpool • mysql> select * from brown; What if bufferpool is full?
  41. 41. Lets run some queries Table space redo log bufferpool • mysql> select * from blue; Assume metadata knows page requirements! LRU on the bufferpool to find space and evict
  42. 42. Lets run some queries Table space redo log bufferpool • mysql> select * from blue; ! Load in blue as before and return
  43. 43. Lets run some queries Table space redo log bufferpool • mysql> update blue set A=B where ID=1; ! Is page in bufferpool? yes
  44. 44. Lets run some queries Table space redo log bufferpool • mysql> update blue set A=B where ID=1; ! Update the page in bufferpool and increment ! page version number
  45. 45. Lets run some queries Table space redo log bufferpool • mysql> update blue set A=B where ID=1; ! Write the new page to the redo log! what tablespace is up to!
  46. 46. Lets run some queries Table space redo log bufferpool • mysql> update blue set A=B where ID=1; ! Update metadata telling it that this is the ! new state of the page! perform sequential ! write disk activity to redo log! This is pretty fast! what tablespace is up to!
  47. 47. Lets run some queries Table space redo log bufferpool • mysql> update blue set A=B where ID=1; ! Send the OK back to mysqld! what tablespace is up to!
  48. 48. Lets run some queries Table space redo log bufferpool • mysql> update blue set A=B where ID=1; ! At sometime in the near future, do a semi sequential ! parse changes and update the tablespace what tablespace is up to! semi sequential write! Not that bad ! !
  49. 49. Lets run some queries Table space redo log bufferpool • mysql> insert into yellow values (1,2,3,4); ! Is space in bufferpool?!
  50. 50. Lets run some queries Table space redo log bufferpool • mysql> insert into yellow values (1,2,3,4); ! Is adjacent pages in bufferpool?! Random read ! (but not too big)
  51. 51. Lets run some queries Table space redo log bufferpool • mysql> insert into yellow values (1,2,3,4); ! Send OK back to client!
  52. 52. Key take aways Table space redo log bufferpool • Cold databases are slow! • bufferpool is empty and everything requires random disk reads • The bigger the buffer pool you have, the quicker you will be! (NUMBER 1 factor on performance!)! • Redo log is the second slowest part • we can tune this innodb_flush_logs_at_trx_commit=1 innodb_buffer_pool_size= ~80% memory!
  53. 53. Schema changes • Everyone hates them • They are not transactional • They are blocking (well most) • How do they work? !
  54. 54. Remember this? mysqld Storage engines(s) tcp:3306 filesocket API query cache binary log join / sort buffers
  55. 55. Lets add a few more Storage engines(s) tcp:3306 filesocket API table meta data binary log Table triggers functions / procedures
  56. 56. mysqld tcp:3306 filesocket mytbl Schema changes mytbl meta
  57. 57. mytbl mysql> CREATE TABLE mytbl 
 (a INT, b CHAR (20), INDEX (a)) ENGINE=InnoDB; Schema changes
  58. 58. mytbl Schema changes mysql> ALTER TABLE mytbl 
 ADD COLUMN age INT;
  59. 59. mytbl Schema changes mysql> ALTER TABLE mytbl 
 ADD COLUMN age INT; ! Wait for any open transactions to close and then ! lock access to the table! block
  60. 60. mytbl Schema changes mysql> ALTER TABLE mytbl 
 ADD COLUMN age INT; ! Create a new ‘hidden’ table with the new schema! block .mytbl
  61. 61. mytbl Schema changes mysql> ALTER TABLE mytbl 
 ADD COLUMN age INT; ! Do an internal ‘mysqldump’ / ‘mysqlrestore’! block .mytbl
  62. 62. mytbl Schema changes mysql> ALTER TABLE mytbl 
 ADD COLUMN age INT; ! Do a switch-a-roo! block .mytbl
  63. 63. Schema changes mysql> ALTER TABLE mytbl 
 ADD COLUMN age INT; ! Unblock and send OK back to client! mytbl
  64. 64. Online schema changes • Some changes can be done without ‘locking’ • Column additions • Index additions ! • Any removal cannot be done online • Unless you do it yourself !
  65. 65. mysqld mytbl Online schema changes 5.1/5.5 mytbl meta
  66. 66. mysqld mytbl Online schema changes 5.1/5.5 mytbl meta client mysql> ALTER… pt-online-schema-change soundclouds lhm etc
  67. 67. mysqld mytbl Online schema changes 5.1/5.5 mytbl meta client mysql> ALTER… .mytbl meta .mytbl ! Creates another table with the updated schema in the client!
  68. 68. mysqld mytbl Online schema changes 5.1/5.5 mytbl meta client mysql> ALTER… .mytbl meta .mytbl ! Creates a set of triggers on the main table! triggers
  69. 69. mysqld mytbl Online schema changes 5.1/5.5 mytbl meta client mysql> ALTER… .mytbl meta .mytbl ! selects the entire table (in batches) and inserts into the new table! ! ! This can take a long time! triggers
  70. 70. mysqld mytbl Online schema changes 5.1/5.5 mytbl meta client mysql> ALTER… .mytbl meta .mytbl triggers client mysql> select ! Whilst the new table is being built, selects still go to the old table!
  71. 71. mytbl Online schema changes 5.1/5.5 mytbl meta client mysql> ALTER… .mytbl meta .mytbl triggers client mysql> UPDATE ! Updates cause the trigger to fire which updates the new table being built. (Inserts the row if its still hasn't been copied)! ! Deletes happen the same way!
  72. 72. mytbl Online schema changes 5.1/5.5 mytbl meta client mysql> ALTER… .mytbl meta .mytbl triggers client mysql> INSERT ! Inserts also happen the same way
  73. 73. mytbl Online schema changes 5.1/5.5 mytbl meta client mysql> ALTER… .mytbl meta .mytbl triggers ! Eventually the table population finishes and a ‘RENAME table’ action is performed (which is quick)
  74. 74. mytbl Online schema changes 5.1/5.5 mytbl meta client mysql> ALTER… .mytbl meta .mytbl triggers ! Eventually the table population finishes. This is the only ‘blocking’ time block block
  75. 75. mytbl Online schema changes 5.1/5.5 mytbl meta client mysql> ALTER… ! The schema change tool finishes
  76. 76. Online schema changes in 5.6 • All that ‘switch-a-roo’ logic is now within mysqld itself !
  77. 77. Summary about schema changes • You ‘double your IO’ when you’re online schema changes! • Binary log is outside of this - so watch out for your slaves table locking if you don't use the right flags ! !

×