Scaling MySQL writes through Partitioning - IPC Spring Edition

The Problem The Tests Breakthroughs Scaling MySQL writes through partitioning Philip Tellis / philip@bluesmoon.info IPC Spring – 2010-06-01 – Berlin IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem The Tests Breakthroughs $ ﬁnger philip Philip Tellis geek yahoo @bluesmoon http://bluesmoon.info/ philip@bluesmoon.info IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem Our data The Tests DB infrastructure Breakthroughs Performance Web requests Millions of beacons from a web page No response required Can be batch processed Very small amounts of data loss is acceptable IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem Our data The Tests DB infrastructure Breakthroughs Performance Large volume 2000 requests/second on most days up to 8000 requests/second on some days 200MM requests/day Some data is fake or abusive IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem Our data The Tests DB infrastructure Breakthroughs Performance Access patterns Lots of writes throughout the day One huge read at the end of the day Summarise data and throw out the details Many reads of summary data over several months IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem Our data The Tests DB infrastructure Breakthroughs Performance Why not use a data warehouse? IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem Our data The Tests DB infrastructure Breakthroughs Performance I like to get the most out of my hardware IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem Our data The Tests DB infrastructure Breakthroughs Performance Hardware setup MySQL 5.1 Multi-master replication in two colos, 1 remote slave per master Only one master writable at any point of time 4GB RAM (later 16GB), Big disk with RAID 10 RAID 10 7200rpm 6x1TB IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem Our data The Tests DB infrastructure Breakthroughs Performance DB conﬁg innodb_buffer _pool_size = 2078M innodb_ﬂush_log_at_trx_commit = 1 innodb_log_buffer _size = 8M innodb_max_dirty _pages_pct = 90 innodb_doublewrite = 1, innodb_support_xa = 1 sync_binlog = 0 key _buffer _size = 32M, myisam_sort_buffer _size = 512k transaction_isolation = REPEATABLE-READ IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem Our data The Tests DB infrastructure Breakthroughs Performance Data setup Each row 120bytes + InnoDB overhead innodb_ﬁle_per_table so we can see how the table grows No Autoincrement ﬁelds PRIMARY KEY derived from data + one other index IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem Our data The Tests DB infrastructure Breakthroughs Performance Schema page identiﬁer - INT timestamp - INT one-way hash of IP - CHAR(32) page performance information request country useragent IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem Our data The Tests DB infrastructure Breakthroughs Performance Test requirements Insert records until the system breaks down Find out why it broke down Find out how to make it not break down Find out how fast we can insert records (must be >2000 i/s) IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem Our data The Tests DB infrastructure Breakthroughs Performance How I tested Insertion script measured insertion speed v/s number of records Number of records roughly translates to table size On DB box we measure disk performance and table size IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem Our data The Tests DB infrastructure Breakthroughs Performance Test 1 IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem Basic tests The Tests Going crazy Breakthroughs Insights Test 2 - Drop the secondary index IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem Basic tests The Tests Going crazy Breakthroughs Insights Test 3 - innodb_buffer_pool_size=1000 IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem Basic tests The Tests Going crazy Breakthroughs Insights Realisation Max table size directly proportional to innodb_buffer_pool_size Extra index reduces insertion rate Extra index reduces max table size Possible solution: increase RAM and innodb_buffer_pool_size But this only postpones the problem IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem Basic tests The Tests Going crazy Breakthroughs Insights Test 4 - innodb_flush_log_at_trx_commit=2 IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem Basic tests The Tests Going crazy Breakthroughs Insights Test 5 - innodb_max_dirty_pages_pct=60 IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem Basic tests The Tests Going crazy Breakthroughs Insights Test 6 - Let’s try MyISAM IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem Basic tests The Tests Going crazy Breakthroughs Insights Test 7 - Inserts in a transaction IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem Basic tests The Tests Going crazy Breakthroughs Insights Other stuff we tried innodb_doublewrite=0 – no effect Server side prepared statements – no effect transaction_isolation=READ-COMMITTED – no effect innodb_support_xa=0 – 12% increase in insertion rate Combination of the best options – negligible effect IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem Basic tests The Tests Going crazy Breakthroughs Insights What we knew at this point Sticking with InnoDB We need a large buffer pool We need to drop extra indices ﬂush_log_at_trx_commit = 2 is good enough Transactions are good IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem Basic tests The Tests Going crazy Breakthroughs Insights Our big problem Insert rate was barely reaching the rate of incoming data! Still breaks down before getting a day’s worth of data IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem Bulk inserts The Tests Partitioning Breakthroughs Long running test Test 8 - Single bulk insert IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem Bulk inserts The Tests Partitioning Breakthroughs Long running test Bulk insert speciﬁcations 40,000 records in one insert statement Use INSERT IGNORE 4-6 seconds per statement PRIMARY KEY drops duplicates We still have a breakdown when we cross the buffer pool IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem Bulk inserts The Tests Partitioning Breakthroughs Long running test Handling insert failures Typically if one record is bad, the entire insert fails INSERT IGNORE solves this problem Bonus is easy recovery from hardware/network failures IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem Bulk inserts The Tests Partitioning Breakthroughs Long running test Test 9 - bulk inserts + partitioning IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem Bulk inserts The Tests Partitioning Breakthroughs Long running test What happened? Split the table into partitions Each partition < 0.5 × innodb_buffer _pool_size current and next partition ﬁt in memory at any time Partition key is based on incoming data and not on SELECTs IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem Bulk inserts The Tests Partitioning Breakthroughs Long running test Schema CREATE TABLE ( ... ) PARTITION BY RANGE( ( time DIV 3600 ) MOD 24 ) ( Partition p0 values less than (2), Partition p1 values less than (4), ... Partition p10 values less than (22), Partition p11 values less than (24) ); IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem Bulk inserts The Tests Partitioning Breakthroughs Long running test Test 10 - Ran for 7 days IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem Bulk inserts The Tests Partitioning Breakthroughs Long running test Still running Terabytes of data around 8500 inserts per second Potentially 700+ MM inserts per day IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem Bulk inserts The Tests Partitioning Breakthroughs Long running test A Note on read performance Data volume before and after is very different Current read rate is approx 8000 records per second IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem The Tests Breakthroughs What about a key-value store? Some summarisation queries are complex but not impossible with a k/v store Summary data is still relational in nature Avoid sharing resources (RAM/CPU) between two separate data stores We have not yet discarded this idea IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem The Tests Breakthroughs Lots of critics The title should be "how to get poor performance by using a completely inappropriate tool" Dude, that’s a really impressive bit of engineering. However, you failed the interview. it’s just another case of RDBMS blinding people for all other possible solutions IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem The Tests Breakthroughs Summary Bulk inserts push up your insert rate Partitioning lets you insert more records Partition based on incoming data key for fast inserts http://tech.bluesmoon.info/2009/09/scaling-writes-in-mysql.html IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem The Tests Breakthroughs Photo credits Disused warehouse on Huddersfield Broad Canal / by TDR1 http://www.flickr.com/photos/tdr1/3578203727/ Hardware store dog / by sstrudeau http://www.flickr.com/photos/sstrudeau/330379020/ North Dakota, Broken Down Van / by mattdente http://www.flickr.com/photos/mattdente/46944898/ One red tree / by EssjayNZ http://www.flickr.com/photos/essjay/155223631/ The Leaning Tree / by stage88 http://www.flickr.com/photos/stage88/3179612722/ IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

The Problem The Tests Breakthroughs contact me Philip Tellis yahoo geek @bluesmoon http://bluesmoon.info/ slideshare.net/bluesmoon philip@bluesmoon.info IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning

http://www.slideshare.net	50
http://developer.yahoo.net	42
http://talks.bluesmoon.info	12
http://static.slidesharecdn.com	1

Scaling MySQL writes through Partitioning - IPC Spring Edition

by Philip Tellis

Accessibility

Categories

Upload Details

Usage Rights

4 Embeds 105

Statistics

Scaling MySQL writes through Partitioning - IPC Spring Edition Presentation Transcript