Downloaded 12 times
More Related Content
![Microsoft Power Point Brett Campbellv2 [Compatibility Mode]](https://cdn.slidesharecdn.com/ss_thumbnails/microsoftpowerpointbrettcampbellv2compatibilitymode-13390722806022-phpapp02-120607073218-phpapp02-thumbnail.jpg?width=640&height=640&fit=bounds)
Howmysqlworks
- 1.
- 2. About me • BeforeREA I worked for a hosting company for 4 years • Learnt a lot about MySQL during this period • I started as an OPS engineer at REA and did some database stuff here • I havent done much MySQL for two years ! • In this period we’ve had • MySQL 5.6 come out • RDS MySQL improvements ! !
- 3. Today • High leveloverview of MySQL • Look inside MySQL server • Talk through how InnoDB performs queries • Schema changes and online schema changes ! !
- 4. Background • Second biggestrelational database in the world • Anyone know what the biggest is? ! • Used by many of the big web shops • Facebook for all feed data • Etsy for user & data • Twitter for tweet persistence • … ! • Sun bought MySQL AB in 2008, people freaked out • Oracle bought Sun in 2010, people freaked out • Big improvements over the past few years (5.5 / 5.6) !
- 5. How it works mysqld mysqlclient tcp:3306 filesocket ServerClient libmysql mysql lib ORM host$ mysql host$ mysqld_safe! host$ mysqld
- 6. Big features • Pluggablestorage engines is massive flexibility • You can use a different storage engine per table ! ! • Common storage engines • InnoDB - ACID compliant store • MyISAM - one of the original storage engines - avoid • Lots and lots of others !
- 7. Inside mysqld mysqld storage engine(s) MyISAM/ InnoDB / Memory / etc tcp:3306 filesocket Getting the query and responding bits and bytes stuff API
- 8. Inside mysqld mysqld storage engine(s) MyISAM/ InnoDB / Memory / etc tcp:3306 filesocket API
- 9. Inside mysqld mysqld storage engine(s) MyISAM/ InnoDB / Memory / etc tcp:3306 filesocket Parse query and convert to SE API calls. API
- 10. Inside mysqld mysqld Storage engines(s) tcp:3306 filesocket API querycache binary log join / sort buffers Binary Log for replication Buffers: Kinda how much batching to storage API to save cost
- 11. The bad parts- binary log mysqld tcp:3306 filesocket query cache binary log join / sort buffers Because you can mix and match storage engines, replicating state has to be done at the lowest common denominator Innodb Table 1 MyISAM Table 2
- 12. The bad parts- binary log mysqld Innodb Table 1 tcp:3306 filesocket query cache binary log join / sort buffers Highly concurrent InnoDB needs its queries, serialized down to replicate to slave MyISAM Table 2
- 13. Compare to otherDBs DB server tcp:x filesocket? performance knobs transaction log
- 14. The bad parts- query cache mysqld Innodb Table 1 tcp:3306 filesocket query cache binary log join / sort buffers Can only work on an entire table level & uses LRU MyISAM Table 2
- 15. The bad parts- query cache mysqld Innodb Table 1 tcp:3306 filesocket query cache binary log join / sort buffers All ‘update/insert/delete’ blocks all transactions to ensure correct state. It actually slows ! things down MyISAM Table 2
- 16. Inside mysqld mysqld storage engine(s) MyISAM/ InnoDB / Memory / etc tcp:3306 filesocket API Where 10%* of ! the work is done Where 90% ! of the work ! is done *Finger in the air calculation
- 17. Summary • Turn offquery cache! It is hurting you! • Binary log is a necessary evil for replication • Dont play with the performance knobs (sort buffer / join buffer / etc) • Turning the 10% - Focus on the 90% • Use InnoDB for all the tables (unless they’re system tables) ! ! • So…lets look at the 90% ! !
- 18.
- 19.
- 20. Unknown gems mysqld InnoDB tcp:3306 filesocket API handlersocket/ memcache Innodb Memcache API thattalks directly to InnoDB ! Removes the SQL layer and is just K/V ! Google: handlersocket memcache InnoDB
- 21. Key parts ofInnoDB Innodb API
- 22. Key parts ofInnoDB Table space (whats on disk) redo log Innodb bufferpool memory! ib_log1! ib_log2! sometable.ibd! ib_data!
- 23. Key parts ofInnoDB Table space redo log bufferpool • Each block is a page • Just think of it as a row with some extra stuff like version number • InnoDB is ACID meaning each connection has to have its own view of the world! • Extra metadata at the top about where everything is
- 24. Key parts ofInnoDB Table space redo log bufferpool • Black is the base metadata
- 25. What is bufferpool Tablespace redo log bufferpool • In memory version of hot tablespace pages
- 26. What is redolog Table space redo log bufferpool • An append only log file of changes
- 27. What is tablespace Table space redo log bufferpool • The raw files - but it may not be always up-to-date
- 28. Lets run somequeries Table space redo log bufferpool • First - lets fill up our database with data
- 29. Lets run somequeries Table space redo log bufferpool • mysql> select * from red;
- 30. Lets run somequeries Table space redo log bufferpool • mysql> select * from red; mysqld parses the query and sends ! API calls to InnoDB
- 31. Lets run somequeries Table space redo log bufferpool • mysql> select * from red; Is red in bufferpool?
- 32. Lets run somequeries Table space redo log bufferpool • mysql> select * from red; Load red into buffer pool perform random ! read disk activity! At this point, the time to respond! is dependent on disk speed!
- 33. Lets run somequeries Table space redo log bufferpool • mysql> select * from red; Respond to API calls to mysqld
- 34. Lets run somequeries Table space redo log bufferpool • mysql> select * from red; Parse results ! second pass sort if necessary ! return to client
- 35. Lets run somequeries Table space redo log bufferpool • mysql> select * from red; #again
- 36. Lets run somequeries Table space redo log bufferpool • mysql> select * from red; mysqld parses the query and sends ! API calls to InnoDB
- 37. Lets run somequeries Table space redo log bufferpool • mysql> select * from red; Is red in bufferpool? - yes
- 38. Lets run somequeries Table space redo log bufferpool • mysql> select * from red; Respond to API calls to mysqld
- 39. Lets run somequeries Table space redo log bufferpool • mysql> select * from red; Parse results ! second pass sort if necessary ! return to client
- 40. Lets run somequeries Table space redo log bufferpool • mysql> select * from brown; What if bufferpool is full?
- 41. Lets run somequeries Table space redo log bufferpool • mysql> select * from blue; Assume metadata knows page requirements! LRU on the bufferpool to find space and evict
- 42. Lets run somequeries Table space redo log bufferpool • mysql> select * from blue; ! Load in blue as before and return
- 43. Lets run somequeries Table space redo log bufferpool • mysql> update blue set A=B where ID=1; ! Is page in bufferpool? yes
- 44. Lets run somequeries Table space redo log bufferpool • mysql> update blue set A=B where ID=1; ! Update the page in bufferpool and increment ! page version number
- 45. Lets run somequeries Table space redo log bufferpool • mysql> update blue set A=B where ID=1; ! Write the new page to the redo log! what tablespace is up to!
- 46. Lets run somequeries Table space redo log bufferpool • mysql> update blue set A=B where ID=1; ! Update metadata telling it that this is the ! new state of the page! perform sequential ! write disk activity to redo log! This is pretty fast! what tablespace is up to!
- 47. Lets run somequeries Table space redo log bufferpool • mysql> update blue set A=B where ID=1; ! Send the OK back to mysqld! what tablespace is up to!
- 48. Lets run somequeries Table space redo log bufferpool • mysql> update blue set A=B where ID=1; ! At sometime in the near future, do a semi sequential ! parse changes and update the tablespace what tablespace is up to! semi sequential write! Not that bad ! !
- 49. Lets run somequeries Table space redo log bufferpool • mysql> insert into yellow values (1,2,3,4); ! Is space in bufferpool?!
- 50. Lets run somequeries Table space redo log bufferpool • mysql> insert into yellow values (1,2,3,4); ! Is adjacent pages in bufferpool?! Random read ! (but not too big)
- 51. Lets run somequeries Table space redo log bufferpool • mysql> insert into yellow values (1,2,3,4); ! Send OK back to client!
- 52. Key take aways Tablespace redo log bufferpool • Cold databases are slow! • bufferpool is empty and everything requires random disk reads • The bigger the buffer pool you have, the quicker you will be! (NUMBER 1 factor on performance!)! • Redo log is the second slowest part • we can tune this innodb_flush_logs_at_trx_commit=1 innodb_buffer_pool_size= ~80% memory!
- 53. Schema changes • Everyonehates them • They are not transactional • They are blocking (well most) • How do they work? !
- 54. Remember this? mysqld Storage engines(s) tcp:3306 filesocket API querycache binary log join / sort buffers
- 55. Lets add afew more Storage engines(s) tcp:3306 filesocket API table meta data binary log Table triggers functions / procedures
- 56.
- 57. mytbl mysql> CREATE TABLEmytbl (a INT, b CHAR (20), INDEX (a)) ENGINE=InnoDB; Schema changes
- 58. mytbl Schema changes mysql> ALTERTABLE mytbl ADD COLUMN age INT;
- 59. mytbl Schema changes mysql> ALTERTABLE mytbl ADD COLUMN age INT; ! Wait for any open transactions to close and then ! lock access to the table! block
- 60. mytbl Schema changes mysql> ALTERTABLE mytbl ADD COLUMN age INT; ! Create a new ‘hidden’ table with the new schema! block .mytbl
- 61. mytbl Schema changes mysql> ALTERTABLE mytbl ADD COLUMN age INT; ! Do an internal ‘mysqldump’ / ‘mysqlrestore’! block .mytbl
- 62. mytbl Schema changes mysql> ALTERTABLE mytbl ADD COLUMN age INT; ! Do a switch-a-roo! block .mytbl
- 63. Schema changes mysql> ALTERTABLE mytbl ADD COLUMN age INT; ! Unblock and send OK back to client! mytbl
- 64. Online schema changes •Some changes can be done without ‘locking’ • Column additions • Index additions ! • Any removal cannot be done online • Unless you do it yourself !
- 65. mysqld mytbl Online schema changes5.1/5.5 mytbl meta
- 66. mysqld mytbl Online schema changes5.1/5.5 mytbl meta client mysql> ALTER… pt-online-schema-change soundclouds lhm etc
- 67. mysqld mytbl Online schema changes5.1/5.5 mytbl meta client mysql> ALTER… .mytbl meta .mytbl ! Creates another table with the updated schema in the client!
- 68. mysqld mytbl Online schema changes5.1/5.5 mytbl meta client mysql> ALTER… .mytbl meta .mytbl ! Creates a set of triggers on the main table! triggers
- 69. mysqld mytbl Online schema changes5.1/5.5 mytbl meta client mysql> ALTER… .mytbl meta .mytbl ! selects the entire table (in batches) and inserts into the new table! ! ! This can take a long time! triggers
- 70. mysqld mytbl Online schema changes5.1/5.5 mytbl meta client mysql> ALTER… .mytbl meta .mytbl triggers client mysql> select ! Whilst the new table is being built, selects still go to the old table!
- 71. mytbl Online schema changes5.1/5.5 mytbl meta client mysql> ALTER… .mytbl meta .mytbl triggers client mysql> UPDATE ! Updates cause the trigger to fire which updates the new table being built. (Inserts the row if its still hasn't been copied)! ! Deletes happen the same way!
- 72. mytbl Online schema changes5.1/5.5 mytbl meta client mysql> ALTER… .mytbl meta .mytbl triggers client mysql> INSERT ! Inserts also happen the same way
- 73. mytbl Online schema changes5.1/5.5 mytbl meta client mysql> ALTER… .mytbl meta .mytbl triggers ! Eventually the table population finishes and a ‘RENAME table’ action is performed (which is quick)
- 74. mytbl Online schema changes5.1/5.5 mytbl meta client mysql> ALTER… .mytbl meta .mytbl triggers ! Eventually the table population finishes. This is the only ‘blocking’ time block block
- 75. mytbl Online schema changes5.1/5.5 mytbl meta client mysql> ALTER… ! The schema change tool finishes
- 76. Online schema changesin 5.6 • All that ‘switch-a-roo’ logic is now within mysqld itself !
- 77. Summary about schemachanges • You ‘double your IO’ when you’re online schema changes! • Binary log is outside of this - so watch out for your slaves table locking if you don't use the right flags ! !