Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
MySQL Sharding  MySQL User Group 4/3/2009 [email_address] http://top-performance.blogspot.com
RockeTier Methodology <ul><li>Detect </li></ul><ul><li>Rate </li></ul><ul><li>Immediate Effective Relief </li></ul><ul><li...
Presentation Objectives <ul><li>Who is using MySQL? </li></ul><ul><li>MySQL Limitations </li></ul><ul><li>How to get over ...
Who is Using MySQL?
MySQL Limitations <ul><li>Table sizes:  50-100M records per table </li></ul><ul><li>Reads:  50 queries/second </li></ul>
Why Do I Care? <ul><li>From 0 to 100  (US mass adaptation) </li></ul><ul><ul><li>Phone: 100 yrs </li></ul></ul><ul><ul><li...
100K New Users/Week
The Network Effect
What Should I Do? <ul><li>Oracle </li></ul><ul><li>SQL Server </li></ul><ul><li>$$$ </li></ul>
Sharding
Vertical Sharding
Horizontal Sharding <ul><li>Static Hashing </li></ul><ul><ul><li>Complex growth </li></ul></ul><ul><ul><li>Simple </li></u...
<ul><li>Key locations are defined in a directory </li></ul><ul><ul><li>Simple growth </li></ul></ul><ul><ul><li>Directory ...
Horizontal Sharding <ul><li>Static hashing with directory mapping </li></ul><ul><ul><li>Simple growth </li></ul></ul><ul><...
Horizontal Sharding <ul><li>Each key signed by DB#  generated on creation </li></ul><ul><ul><li>Simple growth </li></ul></...
Sharding Management <ul><li>No mature tools in the market </li></ul><ul><li>Hibernate Shards – not recommended </li></ul><...
Reporting
Best Practices <ul><li>$connection  =   new_db_connection (&quot; customer :// 1234 &quot;) ; </li></ul><ul><li>$statement...
Lessons <ul><li>Vertical Sharding:  </li></ul><ul><ul><li>User Actions, Users, Comments, Items </li></ul></ul><ul><li>Hori...
Lessons <ul><li>Vertical Sharding:  </li></ul><ul><ul><li>User Actions, Users, Comments, Items </li></ul></ul><ul><li>Hori...
Lessons <ul><li>100M views per day </li></ul><ul><li>The path to Sharding:  </li></ul><ul><ul><li>Single server </li></ul>...
Lessons <ul><li>Master-Master replication </li></ul><ul><li>Each Shard is 50% loaded </li></ul><ul><li>40K queries/second ...
Ad Network Reference Architecture
The Bottom Line: Grow ∞
Startup your Engines  Thank you [email_address] http://top-performance.blogspot.com
Upcoming SlideShare
Loading in …5
×

"How Sharding turned MySQL into the Internet de-facto database standard?", Moshe Kaplan, RockeTier

11,840 views

Published on

A common belief in the enterprise software world is that MySQL cannot scale to large databases sizes. The Internet industry proved it can be done. These days many of the Internet giants, processing billions of events every day, are based on MySQL. Most of these giants were able to turn MySQL into a mighty database machines by implementing Sharding.
What is Sharding? What kinds of Sharding can you implement? What are the best practices? All these issues will be address in this lecture by Moshe Kaplan from RockeTier. the performance experts

Published in: Technology
  • First of all, thank you for this very insightful presentation!

    I have a question about slide 5: here it is stated that one of MySQL limitations is to support 50 queries/second: this number surprised me since it seems quite low.
    Of course it very much depends on the query and on the amount of data that is fetched each time.

    In any case, both in my experience and from the evidence that I've found googling on the subject, it seems that this limit can be set to much a higher value.

    Can you please comment on how you came out with it?
    Thanks!
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

"How Sharding turned MySQL into the Internet de-facto database standard?", Moshe Kaplan, RockeTier

  1. 1. MySQL Sharding MySQL User Group 4/3/2009 [email_address] http://top-performance.blogspot.com
  2. 2. RockeTier Methodology <ul><li>Detect </li></ul><ul><li>Rate </li></ul><ul><li>Immediate Effective Relief </li></ul><ul><li>Roadmap </li></ul><ul><li>Scale up and Scale out </li></ul>
  3. 3. Presentation Objectives <ul><li>Who is using MySQL? </li></ul><ul><li>MySQL Limitations </li></ul><ul><li>How to get over this? </li></ul><ul><ul><li>Move to another DB and scale up… </li></ul></ul><ul><ul><li>Vertical Sharding </li></ul></ul><ul><ul><li>Horizontal Sharding </li></ul></ul><ul><li>Sharding test case </li></ul>
  4. 4. Who is Using MySQL?
  5. 5. MySQL Limitations <ul><li>Table sizes: 50-100M records per table </li></ul><ul><li>Reads: 50 queries/second </li></ul>
  6. 6. Why Do I Care? <ul><li>From 0 to 100 (US mass adaptation) </li></ul><ul><ul><li>Phone: 100 yrs </li></ul></ul><ul><ul><li>Radio: 40 yrs </li></ul></ul><ul><ul><li>TV: 30 yrs </li></ul></ul><ul><ul><li>Mobile: 20 yrs </li></ul></ul><ul><ul><li>Internet: 10 yrs </li></ul></ul><ul><ul><li>Facebook: 2 yrs </li></ul></ul>
  7. 7. 100K New Users/Week
  8. 8. The Network Effect
  9. 9. What Should I Do? <ul><li>Oracle </li></ul><ul><li>SQL Server </li></ul><ul><li>$$$ </li></ul>
  10. 10. Sharding
  11. 11. Vertical Sharding
  12. 12. Horizontal Sharding <ul><li>Static Hashing </li></ul><ul><ul><li>Complex growth </li></ul></ul><ul><ul><li>Simple </li></ul></ul>Mod 10 = 0 Mod 10 = 1 Mod 10 = 2 Mod 10 = 3 Mod 10 = 4 Mod 10 = 5 Mod 10 = 6 Mod 10 = 7 Mod 10 = 8 Mod 10 = 9
  13. 13. <ul><li>Key locations are defined in a directory </li></ul><ul><ul><li>Simple growth </li></ul></ul><ul><ul><li>Directory is SPOF </li></ul></ul>Horizontal Sharding
  14. 14. Horizontal Sharding <ul><li>Static hashing with directory mapping </li></ul><ul><ul><li>Simple growth </li></ul></ul><ul><ul><li>Small Directory still SPOF </li></ul></ul>Mod 1000 = 4
  15. 15. Horizontal Sharding <ul><li>Each key signed by DB# generated on creation </li></ul><ul><ul><li>Simple growth </li></ul></ul><ul><ul><li>New key generation is SPOF </li></ul></ul>
  16. 16. Sharding Management <ul><li>No mature tools in the market </li></ul><ul><li>Hibernate Shards – not recommended </li></ul><ul><ul><li>Hibernate… </li></ul></ul><ul><ul><li>Beta </li></ul></ul><ul><li>Required Mechanisms </li></ul><ul><ul><li>Distribution of changes in DB schema </li></ul></ul>
  17. 17. Reporting
  18. 18. Best Practices <ul><li>$connection = new_db_connection (&quot; customer :// 1234 &quot;) ; </li></ul><ul><li>$statement = $connection->prepare( $sql_statement, $params ); </li></ul><ul><li>$result = $statement->execute(); </li></ul>
  19. 19. Lessons <ul><li>Vertical Sharding: </li></ul><ul><ul><li>User Actions, Users, Comments, Items </li></ul></ul><ul><li>Horizontal Sharding </li></ul><ul><li>Denormalization </li></ul><ul><li>MySQL Replication </li></ul>
  20. 20. Lessons <ul><li>Vertical Sharding: </li></ul><ul><ul><li>User Actions, Users, Comments, Items </li></ul></ul><ul><li>Horizontal Sharding </li></ul><ul><li>Denormalization </li></ul><ul><li>MySQL Replication </li></ul>
  21. 21. Lessons <ul><li>100M views per day </li></ul><ul><li>The path to Sharding: </li></ul><ul><ul><li>Single server </li></ul></ul><ul><ul><li>Single master with multiple read slaves </li></ul></ul><ul><ul><li>Partitioned </li></ul></ul><ul><ul><li>Sharding </li></ul></ul>
  22. 22. Lessons <ul><li>Master-Master replication </li></ul><ul><li>Each Shard is 50% loaded </li></ul><ul><li>40K queries/second </li></ul>
  23. 23. Ad Network Reference Architecture
  24. 24. The Bottom Line: Grow ∞
  25. 25. Startup your Engines Thank you [email_address] http://top-performance.blogspot.com

×