The Problem
                    The Tests
                Breakthroughs




Scaling MySQL writes through partitioning

   ...
The Problem
                               The Tests
                           Breakthroughs


$ finger philip



      Ph...
The Problem     Our data
                           The Tests    DB infrastructure
                       Breakthroughs   ...
The Problem     Our data
                           The Tests    DB infrastructure
                       Breakthroughs   ...
The Problem     Our data
                           The Tests    DB infrastructure
                       Breakthroughs   ...
The Problem     Our data
           The Tests    DB infrastructure
       Breakthroughs    Performance




Why not use a d...
The Problem     Our data
               The Tests    DB infrastructure
           Breakthroughs    Performance




I like ...
The Problem     Our data
                            The Tests    DB infrastructure
                        Breakthroughs ...
The Problem     Our data
                         The Tests    DB infrastructure
                     Breakthroughs    Per...
The Problem     Our data
                          The Tests    DB infrastructure
                      Breakthroughs    P...
The Problem     Our data
                           The Tests    DB infrastructure
                       Breakthroughs   ...
The Problem     Our data
                           The Tests    DB infrastructure
                       Breakthroughs   ...
The Problem     Our data
                           The Tests    DB infrastructure
                       Breakthroughs   ...
The Problem     Our data
                           The Tests    DB infrastructure
                       Breakthroughs   ...
The Problem     Our data
                           The Tests    DB infrastructure
                       Breakthroughs   ...
The Problem     Our data
                  The Tests    DB infrastructure
              Breakthroughs    Performance


Tes...
The Problem     Basic tests
                        The Tests    Going crazy
                    Breakthroughs    Insights...
The Problem     Basic tests
                        The Tests    Going crazy
                    Breakthroughs    Insights...
The Problem     Basic tests
                            The Tests    Going crazy
                        Breakthroughs    ...
The Problem     Basic tests
                            The Tests    Going crazy
                        Breakthroughs    ...
The Problem     Basic tests
                      The Tests    Going crazy
                  Breakthroughs    Insights


T...
The Problem     Basic tests
                      The Tests    Going crazy
                  Breakthroughs    Insights


T...
The Problem     Basic tests
                         The Tests    Going crazy
                     Breakthroughs    Insigh...
The Problem     Basic tests
                          The Tests    Going crazy
                      Breakthroughs    Insi...
The Problem     Basic tests
                             The Tests    Going crazy
                         Breakthroughs  ...
The Problem     Basic tests
                           The Tests    Going crazy
                       Breakthroughs    In...
The Problem     Basic tests
                            The Tests    Going crazy
                        Breakthroughs    ...
The Problem     Bulk inserts
                          The Tests    Partitioning
                      Breakthroughs    Lo...
The Problem     Bulk inserts
                           The Tests    Partitioning
                       Breakthroughs    ...
The Problem     Bulk inserts
                          The Tests    Partitioning
                      Breakthroughs    Lo...
The Problem     Bulk inserts
                            The Tests    Partitioning
                        Breakthroughs  ...
The Problem     Bulk inserts
                       The Tests    Partitioning
                   Breakthroughs    Long run...
The Problem     Bulk inserts
                         The Tests    Partitioning
                     Breakthroughs    Long...
The Problem    Bulk inserts
                              The Tests   Partitioning
                          Breakthroughs...
The Problem
                            The Tests
                        Breakthroughs


Summary




    Bulk inserts pus...
The Problem
                          The Tests
                      Breakthroughs


Thanks, Merci




     ConFoo organi...
The Problem
                              The Tests
                          Breakthroughs


contact me



     Philip Te...
The Problem
                            The Tests
                        Breakthroughs


Photo credits


     Disused war...
Upcoming SlideShare
Loading in...5
×

Scaling MySQL writes through Partitioning

17,681

Published on

MySQL is fairly fast for most applications, but every now and then you might come across an application that needs fast writes at a very large scale. The problem with this is that if your inserts are not ordered according to your primary key, then once you cross a limit known as the innodb_buffer_pool_size (for InnoDB tables), write performance starts to degrade because you're now hitting disk a lot. This limit is dependent on the amount of RAM you have, and that in turn is limited by how much money you can throw at the problem. At some point it makes sense to close your pocket and start thinking of a way to hack around the problem.

Faced with a similar problem, we figured out a way to hack it with MySQL 5's partitioning to scale writes to a consistently high rate.

In this talk, I'll cover all the steps we went through to get to this solution in the hopes that either the solution itself or the thought processes behind it will help others solve their own scaling problems.

Published in: Technology
6 Comments
55 Likes
Statistics
Notes
No Downloads
Views
Total Views
17,681
On Slideshare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
643
Comments
6
Likes
55
Embeds 0
No embeds

No notes for slide

Scaling MySQL writes through Partitioning

  1. 1. The Problem The Tests Breakthroughs Scaling MySQL writes through partitioning Philip Tellis / philip@bluesmoon.info ConFoo / 2010-03-10 ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  2. 2. The Problem The Tests Breakthroughs $ finger philip Philip Tellis philip@bluesmoon.info bluesmoon.info @bluesmoon yahoo geek ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  3. 3. The Problem Our data The Tests DB infrastructure Breakthroughs Performance Web requests Millions of beacons from a web page No response required Can be batch processed Very small amounts of data loss is acceptable ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  4. 4. The Problem Our data The Tests DB infrastructure Breakthroughs Performance Large volume 2000 requests/second on most days up to 8000 requests/second on some days 200MM requests/day Some data is fake or abusive ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  5. 5. The Problem Our data The Tests DB infrastructure Breakthroughs Performance Access patterns Lots of writes throughout the day One huge read at the end of the day Summarise data and throw out the details Many reads of summary data over several months ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  6. 6. The Problem Our data The Tests DB infrastructure Breakthroughs Performance Why not use a data warehouse? ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  7. 7. The Problem Our data The Tests DB infrastructure Breakthroughs Performance I like to get the most out of my hardware ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  8. 8. The Problem Our data The Tests DB infrastructure Breakthroughs Performance Hardware setup MySQL 5.1 Multi-master replication in two colos, 1 remote slave per master Only one master writable at any point of time 4GB RAM (later 16GB), Big disk with RAID 10 ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  9. 9. The Problem Our data The Tests DB infrastructure Breakthroughs Performance DB config innodb_buffer_pool_size=2078M innodb_flush_log_at_trx_commit=1 innodb_log_buffer_size=8M innodb_max_dirty_pages_pct=90 innodb_doublewrite=1, innodb_support_xa=1 sync_binlog=0 key_buffer_size=32M, myisam_sort_buffer_size=512k transaction_isolation=REPEATABLE-READ ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  10. 10. The Problem Our data The Tests DB infrastructure Breakthroughs Performance Data setup Each row 120bytes + InnoDB overhead innodb_file_per_table so we can see how the table grows No Autoincrement fields PRIMARY KEY derived from data + one other index ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  11. 11. The Problem Our data The Tests DB infrastructure Breakthroughs Performance Test requirements Insert records until the system breaks down Find out why it broke down Find out how to make it not break down Find out how fast we can insert records (must be >2000 i/s) ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  12. 12. The Problem Our data The Tests DB infrastructure Breakthroughs Performance Test requirements Insert records until the system breaks down Find out why it broke down Find out how to make it not break down Find out how fast we can insert records (must be >2000 i/s) ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  13. 13. The Problem Our data The Tests DB infrastructure Breakthroughs Performance Test requirements Insert records until the system breaks down Find out why it broke down Find out how to make it not break down Find out how fast we can insert records (must be >2000 i/s) ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  14. 14. The Problem Our data The Tests DB infrastructure Breakthroughs Performance Test requirements Insert records until the system breaks down Find out why it broke down Find out how to make it not break down Find out how fast we can insert records (must be >2000 i/s) ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  15. 15. The Problem Our data The Tests DB infrastructure Breakthroughs Performance How I tested Insertion script measured insertion speed v/s number of records Number of records roughly translates to table size On DB box we measure disk performance and table size ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  16. 16. The Problem Our data The Tests DB infrastructure Breakthroughs Performance Test 1 ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  17. 17. The Problem Basic tests The Tests Going crazy Breakthroughs Insights Test 2 - Drop the secondary index ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  18. 18. The Problem Basic tests The Tests Going crazy Breakthroughs Insights Test 3 - innodb_buffer_pool_size=1000 ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  19. 19. The Problem Basic tests The Tests Going crazy Breakthroughs Insights Realisation Max table size directly proportional to innodb_buffer_pool_size Extra index reduces insertion rate Extra index reduces max table size Possible solution: increase RAM and innodb_buffer_pool_size But this only postpones the problem ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  20. 20. The Problem Basic tests The Tests Going crazy Breakthroughs Insights Realisation Max table size directly proportional to innodb_buffer_pool_size Extra index reduces insertion rate Extra index reduces max table size Possible solution: increase RAM and innodb_buffer_pool_size But this only postpones the problem ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  21. 21. The Problem Basic tests The Tests Going crazy Breakthroughs Insights Test 4 - innodb_flush_log_at_trx_commit=2 ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  22. 22. The Problem Basic tests The Tests Going crazy Breakthroughs Insights Test 5 - innodb_max_dirty_pages_pct=60 ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  23. 23. The Problem Basic tests The Tests Going crazy Breakthroughs Insights Test 6 - Let’s try MyISAM ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  24. 24. The Problem Basic tests The Tests Going crazy Breakthroughs Insights Test 7 - Inserts in a transaction ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  25. 25. The Problem Basic tests The Tests Going crazy Breakthroughs Insights Other stuff we tried innodb_doublewrite=0 - no effect Server side prepared statements - no effect transaction_isolation=READ-COMMITTED - no effect innodb_support_xa=0 - 12% increase in insertion rate Combination of the best options - negligible effect ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  26. 26. The Problem Basic tests The Tests Going crazy Breakthroughs Insights What we knew at this point Sticking with InnoDB We need a large buffer pool We need to drop extra indices flush_log_at_trx_commit=2 is good enough Transactions are good ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  27. 27. The Problem Basic tests The Tests Going crazy Breakthroughs Insights Our big problem Insert rate was barely reaching the rate of incoming data! Still breaks down before getting a day’s worth of data ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  28. 28. The Problem Bulk inserts The Tests Partitioning Breakthroughs Long running test Test 8 - Single bulk insert ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  29. 29. The Problem Bulk inserts The Tests Partitioning Breakthroughs Long running test Bulk insert specifications 40,000 records in one insert statement Use INSERT IGNORE 4-6 seconds per statement PRIMARY KEY drops duplicates We still have a breakdown when we cross the buffer pool ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  30. 30. The Problem Bulk inserts The Tests Partitioning Breakthroughs Long running test Test 9 - bulk inserts + partitioning ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  31. 31. The Problem Bulk inserts The Tests Partitioning Breakthroughs Long running test What happened? Split the table into partitions Each partition < 0.5 x innodb_buffer_pool_size current and next partition fit in memory at any time Partition key is based on incoming data and not on SELECTs ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  32. 32. The Problem Bulk inserts The Tests Partitioning Breakthroughs Long running test Schema CREATE TABLE ( ... ) PARTITION BY RANGE( ( time DIV 3600 ) MOD 24 ) ( Partition p0 values less than (2), Partition p1 values less than (4), ... Partition p10 values less than (22), Partition p11 values less than (24) ); ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  33. 33. The Problem Bulk inserts The Tests Partitioning Breakthroughs Long running test Test 10 - Ran for 7 days ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  34. 34. The Problem Bulk inserts The Tests Partitioning Breakthroughs Long running test Still running Terabytes of data around 8500 inserts per second Potentially 700+ MM inserts per day ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  35. 35. The Problem The Tests Breakthroughs Summary Bulk inserts push up your insert rate Partitioning lets you insert more records Partition based on incoming data key for fast inserts http://tech.bluesmoon.info/2009/09/scaling-writes-in-mysql.html ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  36. 36. The Problem The Tests Breakthroughs Thanks, Merci ConFoo organisers Exceptional Performance team @ Yahoo! Monitoring team @ Yahoo! MySQL Geeks at Yahoo! ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  37. 37. The Problem The Tests Breakthroughs contact me Philip Tellis philip@bluesmoon.info bluesmoon.info @bluesmoon yahoo geek ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  38. 38. The Problem The Tests Breakthroughs Photo credits Disused warehouse on Huddersfield Broad Canal / by TDR1 http://www.flickr.com/photos/tdr1/3578203727/ Hardware store dog / by sstrudeau http://www.flickr.com/photos/sstrudeau/330379020/ North Dakota, Broken Down Van / by mattdente http://www.flickr.com/photos/mattdente/46944898/ One red tree / by EssjayNZ http://www.flickr.com/photos/essjay/155223631/ The Leaning Tree / by stage88 http://www.flickr.com/photos/stage88/3179612722/ ConFoo / 2010-03-10 Scaling MySQL writes through partitioning
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×