Moshe Kaplan  Database2011, Jan 13 [email_address] http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspo...
Presentation Objectives <ul><li>Who is using MySQL? </li></ul><ul><li>MySQL Limitations </li></ul><ul><li>How to get over ...
Who is Using MySQL?
MySQL Limitations <ul><li>50-100M Records/Table </li></ul><ul><li>50 SELECT Statements/Second </li></ul>
700 Inserts/Sec In Memory Engine 3000 Inserts/Sec InnoDB Engine 700 Inserts/Sec Amazon AWS Standard Large Instance
Why Do I Care? <ul><li>From 0 to 100  (US mass adaptation) </li></ul><ul><ul><li>Phone: 100 yrs </li></ul></ul><ul><ul><li...
100K New Users/Week
The Network Effect
What Should I Do? <ul><li>Oracle </li></ul><ul><li>SQL Server </li></ul><ul><li>$$$ </li></ul>
Sharding
Vertical Sharding
Horizontal Sharding <ul><li>Static Hashing </li></ul><ul><ul><li>Complex growth </li></ul></ul><ul><ul><li>Simple </li></u...
<ul><li>Key locations are defined in a directory </li></ul><ul><ul><li>Simple growth </li></ul></ul><ul><ul><li>Directory ...
Horizontal Sharding <ul><li>Static Hashing with Directory Mapping </li></ul><ul><ul><li>Simple Growth </li></ul></ul><ul><...
Horizontal Sharding <ul><li>Each key signed by DB#  generated on creation </li></ul><ul><ul><li>Simple growth </li></ul></...
Limitations: Reporting
Best Practices <ul><li>$connection =  new_db_connection(&quot;customer://1234&quot;); </li></ul><ul><li>$statement =  $con...
Lessons <ul><li>Vertical Sharding:  </li></ul><ul><ul><li>User Actions, Users, Comments, Items </li></ul></ul><ul><li>Hori...
‘ s Path to Sharding <ul><li>Single Server </li></ul><ul><li>Single Master with Multiple Read Slaves </li></ul><ul><li>Par...
‘ s Path <ul><li>Master-Master replication </li></ul><ul><li>Each Shard is 50% loaded </li></ul><ul><li>40K queries/second...
Ad Network Reference Architecture
Sharding Management http://dev.mysql.com/tech-resources/articles/mysql_5.0_psea1.html Storage Engine MySQL Proxy
Grizzard http://readwriteweb.com.br/search/scala-framework/
ScaleBase
The Bottom Line: Grow ∞
Startup your Engines  Thank you Moshe Kaplan: mokplan@gmail.com http://top-performance.blogspot.com
Upcoming SlideShare
Loading in...5
×

Database2011 MySQL Sharding

5,114

Published on

How MySQL become an Internet standard? based on a presentation I gave at the Database2011 conference: Who uses MySQL? (everyone inc. Facebook, Twitter and Google), What are MySQL limitations? (multi threading, INSERTS, SELECTS), How the big boys solved these issues? (Sharding, Partitioning), What are the pros and cons for each Sharding strategy? (Vertical, Horizonal: directory based, static hashing, signature based), How to implement Sharding from an applicative view? How to take care of reporting? What are the emerging solutions in this field? (Gizzard, ScaleBase, Xeround and Akibian)

Published in: Technology
1 Comment
20 Likes
Statistics
Notes
  • The best alternative to sharding is CLUSTRIX ! It is seemlesss to implement and let's all developers wory about developing! not sharding. I can tell you more about it if you like, mail me at kenwaj@yahoo.com
    KW
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
5,114
On Slideshare
0
From Embeds
0
Number of Embeds
22
Actions
Shares
0
Downloads
3
Comments
1
Likes
20
Embeds 0
No embeds

No notes for slide
  • The world is changing more and more fast You have to minimize NRE You must support unexpected demand You must provide top service (people now leaves cell operator after single incident, rather then 5, 10 years ago Firms a vulnerable – Citi worth 20Bill $ instead of 200 Bill a year ago Break and Mortar bookstores 15 years ago and Amazon…. Will it happen again to banks, insurance, real estate agencies… IS YOUR MARKET THE NEXT FOR PENETRATION – Finance? Real Estate? How to win a rival that is not existing yet? http://www.johnmwillis.com/ibm/cloud-computing-and-the-enterprise/ -
  • Sharded database – OLTP Little/No reporting OLAP – must be implemented for reporting Loads data from sharded DBs Custom mechanism Any commercial
  • Start with nothing: storage, FW, LB, Server and grow… Can buy servers for more than hour
  • Database2011 MySQL Sharding

    1. 1. Moshe Kaplan Database2011, Jan 13 [email_address] http://blogs.microsoft.co.il/blogs/vprnd http://top-performance.blogspot.com
    2. 2. Presentation Objectives <ul><li>Who is using MySQL? </li></ul><ul><li>MySQL Limitations </li></ul><ul><li>How to get over this? </li></ul><ul><li>Sharding Case Studies </li></ul><ul><li>Products </li></ul>
    3. 3. Who is Using MySQL?
    4. 4. MySQL Limitations <ul><li>50-100M Records/Table </li></ul><ul><li>50 SELECT Statements/Second </li></ul>
    5. 5. 700 Inserts/Sec In Memory Engine 3000 Inserts/Sec InnoDB Engine 700 Inserts/Sec Amazon AWS Standard Large Instance
    6. 6. Why Do I Care? <ul><li>From 0 to 100 (US mass adaptation) </li></ul><ul><ul><li>Phone: 100 yrs </li></ul></ul><ul><ul><li>Radio: 40 yrs </li></ul></ul><ul><ul><li>TV: 30 yrs </li></ul></ul><ul><ul><li>Mobile: 20 yrs </li></ul></ul><ul><ul><li>Internet: 10 yrs </li></ul></ul><ul><ul><li>Facebook: 2 yrs </li></ul></ul>
    7. 7. 100K New Users/Week
    8. 8. The Network Effect
    9. 9. What Should I Do? <ul><li>Oracle </li></ul><ul><li>SQL Server </li></ul><ul><li>$$$ </li></ul>
    10. 10. Sharding
    11. 11. Vertical Sharding
    12. 12. Horizontal Sharding <ul><li>Static Hashing </li></ul><ul><ul><li>Complex growth </li></ul></ul><ul><ul><li>Simple </li></ul></ul>Mod 10 = 0 Mod 10 = 1 Mod 10 = 2 Mod 10 = 3 Mod 10 = 4 Mod 10 = 5 Mod 10 = 6 Mod 10 = 7 Mod 10 = 8 Mod 10 = 9
    13. 13. <ul><li>Key locations are defined in a directory </li></ul><ul><ul><li>Simple growth </li></ul></ul><ul><ul><li>Directory is SPOF </li></ul></ul>Horizontal Sharding
    14. 14. Horizontal Sharding <ul><li>Static Hashing with Directory Mapping </li></ul><ul><ul><li>Simple Growth </li></ul></ul><ul><ul><li>Small Directory still SPOF </li></ul></ul>Mod 1000 = 4
    15. 15. Horizontal Sharding <ul><li>Each key signed by DB# generated on creation </li></ul><ul><ul><li>Simple growth </li></ul></ul><ul><ul><li>New key generation is SPOF </li></ul></ul>
    16. 16. Limitations: Reporting
    17. 17. Best Practices <ul><li>$connection = new_db_connection(&quot;customer://1234&quot;); </li></ul><ul><li>$statement = $connection->prepare( $sql_statement, $params ); </li></ul><ul><li>$result = $statement->execute(); </li></ul>
    18. 18. Lessons <ul><li>Vertical Sharding: </li></ul><ul><ul><li>User Actions, Users, Comments, Items </li></ul></ul><ul><li>Horizontal Sharding </li></ul><ul><li>Denormalization </li></ul><ul><li>MySQL Replication </li></ul>
    19. 19. ‘ s Path to Sharding <ul><li>Single Server </li></ul><ul><li>Single Master with Multiple Read Slaves </li></ul><ul><li>Partitioned </li></ul><ul><li>Sharding </li></ul>
    20. 20. ‘ s Path <ul><li>Master-Master replication </li></ul><ul><li>Each Shard is 50% loaded </li></ul><ul><li>40K queries/second </li></ul>
    21. 21. Ad Network Reference Architecture
    22. 22. Sharding Management http://dev.mysql.com/tech-resources/articles/mysql_5.0_psea1.html Storage Engine MySQL Proxy
    23. 23. Grizzard http://readwriteweb.com.br/search/scala-framework/
    24. 24. ScaleBase
    25. 25. The Bottom Line: Grow ∞
    26. 26. Startup your Engines Thank you Moshe Kaplan: mokplan@gmail.com http://top-performance.blogspot.com

    ×