Solving the Challenges of Big Databases
with MySQL
Bradley C. Kuszmaul
MySQL Connect 2012 Solving the Challenges of Big Da...
Challenging Activities
Some of the top challenges I hear:
• Loading a big database takes a long time.
• Adding rows to an ...
Storage Engines Can Help MySQL
Storage Engines
Client
MySQL front end
MyISAM InnoDB TokuDB ...
OS file storage
Client Clie...
Big Data
• Big data can range from a terabyte to multiple
petabytes.
• The difficulties start showing up when your data
doe...
Is a Gigabyte Big?
• No, since it fits in main memory even on a cheap
server.
• No, since scanning the entire data set off ...
Is a Terabyte Big?
• A terabyte fits on a small $500 server with a single
disk drive. But even the small end of a terabyte ...
Is 10 Terabytes Big?
• You might spend $5,000–$20,000 on a server that
can hold ten terabytes on a RAID with with a
batter...
Is a Petabyte Big?
• Yes.
• Some MySQL users have multiple petabytes of
data.
• They typically shard the data up across hu...
Challenges
As you move from millions of rows and gigabytes of
data to billions of rows and terabytes of data, these
proble...
Challenge I: Loading Data
• The first problem is loading the data.
• Two reasons for slow performance:
1. Fetching one row ...
Where does the TokuDB Loader Find
Parallelism?
1. Merge sort (especially the in-memory stages)
contains a huge amount of p...
Challenge II: Maintaining Indexes
Setup: A big database with many indexes adds,
removes, or updates thousands of rows per ...
B-Trees are Everywhere
B-Trees show up in
database indexes (such
as MyISAM and
InnoDB), file systems
(such as XFS), and man...
B-Trees are Fast at Sequential Inserts
In Memory
Insertions are
into this leaf node
BB
B
···
···
• One disk I/O per leaf (...
B-Trees are Slow for High-Entropy Inserts
In Memory
BB
B
···
···
• Most nodes are not in main memory.
• Most insertions re...
Recipe for a fractal tree
Start with a B-tree:
11 29 61 79
43
47,53,59,61
13,19,23,29
83,89,97,10131,37,41,43
1,3,5,7,11
6...
Added 17
11 29 61 79
43
47,53,59,61
13,17,19,23,29
83,89,97,10131,37,41,43
1,3,5,7,11
67,71,73,79
Maybe needed several dis...
InnoDB Adds a Buffer
11 29 61 79
43
Not in RAM
Buffer
47,53,59,61
13,19,23,29
83,89,97,10131,37,41,43
1,3,5,7,11
67,71,73,...
A Buffer Helps (a Little)
Sometimes an InnoDB-style buffer works great: The
buffer collects several rows for a single tree...
Fractal Trees Indexes Add Many Buffers
11 29 61 79
43 Buffers
47,53,59,61
13,17,19,23,29
83,89,97,10131,37,41,43
1,3,5,7,1...
When a Buffer Overflows
11 29 61 79
43 Buffers
insert(17)
insert(21)
insert(39)
47,53,59,61
13,17,19,23,29
83,89,97,10131,3...
Deletes are Also Messages
11 29 61 79
43 Buffers
insert(17)
insert(21)
insert(39)
delete(29)
47,53,59,61
13,17,19,23,29
83...
Comparing Data Structures
Disk I/Os are the important metric. How many does it
take to do an insertion or a query?
Data In...
Maintaining Indexes on SSD
Inserting one billion rows, maintaining three
secondary indexes. TokuDB’s terminal insertion ra...
Reducing Wear on SSD
• 161 times fewer write operations.
• 17 times fewer bytes written.
MySQL Connect 2012 Solving the Ch...
SSDs Wear Out
• Modern SSDs wear out after about 1000 overwrites.
• sysbench achieves 100MB/s on random writes on a
256GB ...
Covering Indexes
Arranging for your queries to be index-only can make
huge speedups. For example see this guest blog
posti...
Challenge III: Adding a Column
Many DBAS have had the following experience:
• Load a bunch of data into a table and set up...
A Painful Way to Add a Column
The MyISAM / InnoDB way:
• Lock the table.
• Scan through the table, changing every row to t...
A Fast Way to Add a Column
11 29 61 79
43 Buffers
addrow(x)
47,53,59,61
13,17,19,23,29
83,89,97,10131,37,41,43
1,3,5,7,11
...
Q: How Fast is Hot Column Addition?
Zardosht wrote about a (typical) example:
alter table ontime add column totalTime int ...
Challenge IV: Adding an Index
Adding an index to a big database can be painful.
The problem: What if someone adds a row wh...
A Fast Way to Add an Index
Scan through the table creating the index.
UnindexedIndexed
2 17 29 3741 7 31 13 5 11 3
Partial...
A Fast Way to Add an Index
Scan through the table creating the index.
UnindexedIndexed
17
2 17 29 3741 7 31 13 5 11 3
Part...
A Fast Way to Add an Index
Scan through the table creating the index.
UnindexedIndexed
1713
2 17 29 3741 7 31 13 5 11 3
Pa...
Handling Updates I
If another thread updates something that’s been
scanned, then update the index
UnindexedIndexed
2 17 29...
Handling Updates II
If another thread updates something that has not been
scanned, then do not update the index, since the...
Q: How Fast is Hot Index Addition?
Zardosht wrote about a (typical) example:
• InnoDB took 31m 34s.
• TokuDB took 9m 30s.
...
Challenge V: Replication (Slave Lag)
Many systems employ MySQL’s master-slave
replication
• to distribute read workloads
•...
TokuDB Keeps Up
After 60s at 3000TPS, InnoDB was 140s behind, but
TokuDB is still keeping up.
MySQL Connect 2012 Solving t...
Challenge VI: Compression
Compression can reduce the cost of maintaining your
database.
• For SSD, space costs money, so c...
Aggressive Compression
Log-style data with stored procedure names, database instance names, begin and ending execution tim...
Compression Speeds Up Loading
MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 43
Compression Speeds Up The DB
MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 44
Why not Always Use Aggressive
Compression?
• If you have less than six cores in your server, the
extra CPU work from the c...
Big Data Challenges
If you have big data, you will find these challenges
(and others):
• Loading data.
• Inserting, updatin...
More Information
You can download this talk and others at
http://tokutek.com/technology
MySQL Connect 2012 Solving the Cha...
Upcoming SlideShare
Loading in …5
×

Big challenges

521 views
439 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
521
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Big challenges

  1. 1. Solving the Challenges of Big Databases with MySQL Bradley C. Kuszmaul MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 1
  2. 2. Challenging Activities Some of the top challenges I hear: • Loading a big database takes a long time. • Adding rows to an existing database is slow. • Adding or removing a column takes my table offline for a long time. • Adding an index takes my table offline for a long time. • Backup is difficult. (Not in this talk.) These activities are painless for small data, but can be painful for big data. MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 2
  3. 3. Storage Engines Can Help MySQL Storage Engines Client MySQL front end MyISAM InnoDB TokuDB ... OS file storage Client ClientClient Rest of this talk is about what a storage engine can do, focusing on TokuDB because that’s what I know. Tokutek sells TokuDB, a closed-source storage engine for MySQL. MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 3
  4. 4. Big Data • Big data can range from a terabyte to multiple petabytes. • The difficulties start showing up when your data does not fit in main memory. MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 4
  5. 5. Is a Gigabyte Big? • No, since it fits in main memory even on a cheap server. • No, since scanning the entire data set off of disk is on the order of seconds. MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 5
  6. 6. Is a Terabyte Big? • A terabyte fits on a small $500 server with a single disk drive. But even the small end of a terabyte can contain billions of rows of data. The little server probably cannot handle much load. • This machine has too little memory to hold the entire dataset. • It can take hours to scan the data. So yes, a terabyte can be big. MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 6
  7. 7. Is 10 Terabytes Big? • You might spend $5,000–$20,000 on a server that can hold ten terabytes on a RAID with with a battery-backed-up controller. • Now it’s really tough to fit the data into memory... MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 7
  8. 8. Is a Petabyte Big? • Yes. • Some MySQL users have multiple petabytes of data. • They typically shard the data up across hundreds or thousands of servers. • Each server suffers from big data problems, and the system as a whole suffers really big data problems. • Let’s focus on the problems a single server faces. These problems show up at a terabyte or so. MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 8
  9. 9. Challenges As you move from millions of rows and gigabytes of data to billions of rows and terabytes of data, these problems get harder: Challenge I: Loading data Challenge II: Maintaining indexes under insertions, deletions, and updates. Challenge III: Adding or removing columns. Challenge IV: Adding an index online. Challenge V: Replication (slave lag). Challenge VI: Compression. MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 9
  10. 10. Challenge I: Loading Data • The first problem is loading the data. • Two reasons for slow performance: 1. Fetching one row at a time from a file. 2. Storing one row at a time onto the database. • TokuDB speeds up #2, but does not address #1. • Rich found an 8-fold speedup loading TPCC. • Perhaps more speedup on more cores (especially if someone addresses #1). MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 10
  11. 11. Where does the TokuDB Loader Find Parallelism? 1. Merge sort (especially the in-memory stages) contains a huge amount of parallelism. 2. Compression offers a huge amount of parallelism. TokuDB compresses 4MB blocks. MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 11
  12. 12. Challenge II: Maintaining Indexes Setup: A big database with many indexes adds, removes, or updates thousands of rows per second. I’ll talk about: • Why B-trees bottleneck insertion rates. InnoDB and MyISAM use B-trees. • How fractal trees improve speed. TokuDB uses fractal trees. MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 12
  13. 13. B-Trees are Everywhere B-Trees show up in database indexes (such as MyISAM and InnoDB), file systems (such as XFS), and many other storage systems. MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 13
  14. 14. B-Trees are Fast at Sequential Inserts In Memory Insertions are into this leaf node BB B ··· ··· • One disk I/O per leaf (which contains many rows). • Sequential disk I/O. • Performance is limited by disk bandwidth. MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 14
  15. 15. B-Trees are Slow for High-Entropy Inserts In Memory BB B ··· ··· • Most nodes are not in main memory. • Most insertions require a random disk I/O. • Performance is limited by disk head movement. • Only 100’s of inserts/s/disk (≤ 0.2% of disk bandwidth). MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 15
  16. 16. Recipe for a fractal tree Start with a B-tree: 11 29 61 79 43 47,53,59,61 13,19,23,29 83,89,97,10131,37,41,43 1,3,5,7,11 67,71,73,79 Oops, I forgot 17. MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 16
  17. 17. Added 17 11 29 61 79 43 47,53,59,61 13,17,19,23,29 83,89,97,10131,37,41,43 1,3,5,7,11 67,71,73,79 Maybe needed several disk I/Os to bring blocks in to store 17. MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 17
  18. 18. InnoDB Adds a Buffer 11 29 61 79 43 Not in RAM Buffer 47,53,59,61 13,19,23,29 83,89,97,10131,37,41,43 1,3,5,7,11 67,71,73,79 17 • If a block is on disk (not in RAM) then keep the row in a buffer. • Later, move rows from the buffer to the tree nodes. MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 18
  19. 19. A Buffer Helps (a Little) Sometimes an InnoDB-style buffer works great: The buffer collects several rows for a single tree node, and for the cost of one I/O, several rows are moved to disk. Sometimes the buffer fills up, and the system slows down. Even in these situations, an insertion buffer helps (but not by much). 43 Buffer , 85, 103 17,4,39,49,72, MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 19
  20. 20. Fractal Trees Indexes Add Many Buffers 11 29 61 79 43 Buffers 47,53,59,61 13,17,19,23,29 83,89,97,10131,37,41,43 1,3,5,7,11 67,71,73,79 insert(17) • All the internal nodes of the tree include buffers. • To insert a row, put a message in the root’s buffer. MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 20
  21. 21. When a Buffer Overflows 11 29 61 79 43 Buffers insert(17) insert(21) insert(39) 47,53,59,61 13,17,19,23,29 83,89,97,10131,37,41,43 1,3,5,7,11 67,71,73,79 • Move messages down the tree when buffers fill. • When a message arrives at a leaves, apply it. MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 21
  22. 22. Deletes are Also Messages 11 29 61 79 43 Buffers insert(17) insert(21) insert(39) delete(29) 47,53,59,61 13,17,19,23,29 83,89,97,10131,37,41,43 1,3,5,7,11 67,71,73,79 MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 22
  23. 23. Comparing Data Structures Disk I/Os are the important metric. How many does it take to do an insertion or a query? Data Insertion cost Query cost structure (in practice) B-tree Many I/Os One I/O LSM tree Fraction of an I/O Many I/Os Fractal Tree Fraction of an I/O One I/O • Fractal tree indexes reduce the cost of index maintenance by two to three orders of magnitude. • So you can ingest data faster and maintain more indexes. MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 23
  24. 24. Maintaining Indexes on SSD Inserting one billion rows, maintaining three secondary indexes. TokuDB’s terminal insertion rate was 14,532 inserts/s. InnoDB’s was 1,607 inserts/s. Platform: TokuDB v6.5, Centos 5.6; 2x Xeon L5520; 72GB RAM; LSI MegaRaid 9285; 2x 256GB Samsung 830 in RAID0. MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 24
  25. 25. Reducing Wear on SSD • 161 times fewer write operations. • 17 times fewer bytes written. MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 25
  26. 26. SSDs Wear Out • Modern SSDs wear out after about 1000 overwrites. • sysbench achieves 100MB/s on random writes on a 256GB Samsung 830, overwriting in 7 hours, and wearing out in 300 days. • In this scenario, the FTL amplifies writes by another 4x, so it will last only 75 days. • InnoDB is a little slower than sysbench, and would wear it out in about 246 days. • Under the same workload, TokuDB would last 11 years. MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 26
  27. 27. Covering Indexes Arranging for your queries to be index-only can make huge speedups. For example see this guest blog posting from Baron Schwartz of Percona: http://www.tokutek.com/2011/09/ are-you-forcing-mysql-to-do-twice -as-many-joins-as-necessary/ MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 27
  28. 28. Challenge III: Adding a Column Many DBAS have had the following experience: • Load a bunch of data into a table and set up the associated indexes. • Realize that adding a column would be helpful. This becomes painful for big data. MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 28
  29. 29. A Painful Way to Add a Column The MyISAM / InnoDB way: • Lock the table. • Scan through the table, changing every row to the new format (adding the column). • Unlock the table. This process can take many hours during which time the system permits no modifications to the table. The bigger the data the longer the downtime. MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 29
  30. 30. A Fast Way to Add a Column 11 29 61 79 43 Buffers addrow(x) 47,53,59,61 13,17,19,23,29 83,89,97,10131,37,41,43 1,3,5,7,11 67,71,73,79 Send a “broadcast” message into the fractal tree index. • As the message descends tree the message is replicated. • When it arrives at a leaf, update the leaf’s rows. TokuDB can add or delete a column with essentially no additional I/O. MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 30
  31. 31. Q: How Fast is Hot Column Addition? Zardosht wrote about a (typical) example: alter table ontime add column totalTime int default 0; Hot column addition reduced the schema change time from 17 hours to 3 seconds: http://www.tokutek.com/2011/03/ hot-column-addition-and-deletion-part-i-performance A: It’s fast MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 31
  32. 32. Challenge IV: Adding an Index Adding an index to a big database can be painful. The problem: What if someone adds a row while we are building the index? The row might not make it into the index, so the index is corrupted. Many storage engines lock the table when adding an index. MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 32
  33. 33. A Fast Way to Add an Index Scan through the table creating the index. UnindexedIndexed 2 17 29 3741 7 31 13 5 11 3 Partially constructed index Table 2 7 31 41 MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 33
  34. 34. A Fast Way to Add an Index Scan through the table creating the index. UnindexedIndexed 17 2 17 29 3741 7 31 13 5 11 3 Partially constructed index Table 2 7 31 41 MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 34
  35. 35. A Fast Way to Add an Index Scan through the table creating the index. UnindexedIndexed 1713 2 17 29 3741 7 31 13 5 11 3 Partially constructed index Table 2 7 31 41 MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 35
  36. 36. Handling Updates I If another thread updates something that’s been scanned, then update the index UnindexedIndexed 2 17 29 3741 7 31 5 11 3 Partially constructed index Table 2 7 31 41 23 23 13 MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 36
  37. 37. Handling Updates II If another thread updates something that has not been scanned, then do not update the index, since the indexer will pick up the value. UnindexedIndexed 2 17 29 3741 7 31 5 11 3 Partially constructed index Table 2 7 31 41 23 23 19 13 MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 37
  38. 38. Q: How Fast is Hot Index Addition? Zardosht wrote about a (typical) example: • InnoDB took 31m 34s. • TokuDB took 9m 30s. • But the table was locked for less than 2s. http://www.tokutek.com/2011/04/hot-indexing-part-i-new-feature A: It’s fast MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 38
  39. 39. Challenge V: Replication (Slave Lag) Many systems employ MySQL’s master-slave replication • to distribute read workloads • to build backups, and • to implement geographically distributed disaster recovery. But MySQL slaves often fall behind, since the replication is single-threaded (even the latest multithreaded slaves are single threaded in each database). MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 39
  40. 40. TokuDB Keeps Up After 60s at 3000TPS, InnoDB was 140s behind, but TokuDB is still keeping up. MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 40
  41. 41. Challenge VI: Compression Compression can reduce the cost of maintaining your database. • For SSD, space costs money, so compression saves capital costs. • For rotating disks, compression improves performance. MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 41
  42. 42. Aggressive Compression Log-style data with stored procedure names, database instance names, begin and ending execution timestamps, duration row counts, and parameter values. MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 42
  43. 43. Compression Speeds Up Loading MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 43
  44. 44. Compression Speeds Up The DB MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 44
  45. 45. Why not Always Use Aggressive Compression? • If you have less than six cores in your server, the extra CPU work from the compression workload compression work might slow you down: Use standard compression. • For modern servers, there are plenty of CPU cycles to do the compression: Use aggressive compression. MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 45
  46. 46. Big Data Challenges If you have big data, you will find these challenges (and others): • Loading data. • Inserting, updating, and deleting rows while maintaining indexes. • Schema changes. • Replication (slave lag) • Compression. TokuDB 6.5 was announced this week and is available now. MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 46
  47. 47. More Information You can download this talk and others at http://tokutek.com/technology MySQL Connect 2012 Solving the Challenges of Big Databases with MySQL 47

×