What Every Developer Should Know About Database Scalability
Upcoming SlideShare
Loading in...5
×
 

What Every Developer Should Know About Database Scalability

on

  • 36,282 views

Replication. Partitioning. Relational databases. Bigtable. Dynamo. There is no one-size-fits-all approach to scaling your database, and the CAP theorem proved that there never will be. This talk will ...

Replication. Partitioning. Relational databases. Bigtable. Dynamo. There is no one-size-fits-all approach to scaling your database, and the CAP theorem proved that there never will be. This talk will explain the advantages and limits of the approaches to scaling traditional relational databases, as well as the tradeoffs made by the designers of newer distributed systems like Cassandra. These slides are from Jonathan Ellis's OSCON 09 talk: http://en.oreilly.com/oscon2009/public/schedule/detail/7955

Statistics

Views

Total Views
36,282
Views on SlideShare
33,860
Embed Views
2,422

Actions

Likes
75
Downloads
1,063
Comments
6

51 Embeds 2,422

http://spyced.blogspot.com 1365
http://www.scoop.it 380
http://www.slideshare.net 273
http://training-india.blogspot.com 112
http://saltfactory.textcube.com 40
http://www.pearltrees.com 39
http://sqlsolutions.blogspot.com 38
http://planetcassandra.org 26
http://spyced.blogspot.in 24
url_unknown 13
http://spyced.blogspot.co.uk 12
http://training-india.blogspot.in 7
http://theoldreader.com 7
http://spyced.blogspot.com.br 6
http://spyced.blogspot.jp 6
http://spyced.blogspot.fr 5
http://localhost 5
http://spyced.blogspot.ca 5
http://translate.googleusercontent.com 5
http://profeo.sf.ac 4
https://twitter.com 4
http://spyced.blogspot.com.ar 3
http://spyced.blogspot.de 3
http://saltfactory-textcube.blogspot.com 3
http://webcache.googleusercontent.com 3
http://hyubiz.com 2
http://pinterest.com 2
http://spyced.blogspot.sg 2
http://www.techgig.com 2
http://spyced.blogspot.com.au 2
http://timesjobs.techgig.com 2
http://spyced.blogspot.se 2
http://spyced.blogspot.hk 2
http://www.blogger.com 1
http://spyced.blogspot.co.il 1
http://www.iss.nl 1
http://www.pinterest.com 1
http://adrianamap.wikispaces.com 1
http://spyced.blogspot.kr 1
http://sqlsolutions.blogspot.com.ar 1
http://www.4624.info 1
http://infosiftr.com 1
http://a0.twimg.com 1
http://spyced.blogspot.it 1
https://twimg0-a.akamaihd.net 1
http://spyced.blogspot.ro 1
http://profeo.pt 1
http://www.pertinence.net 1
http://spyced.blogspot.com.es 1
http://74.125.155.132 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • Related to your 'partitioning' class of approaches to scalability is horizontal scale with traditional relational databases. ParElastic (full disclosure, I'm a founder & CTO) makes a product that provides elastic horizontal scalability for MySQL. Check out www.parelastic.com.
    Are you sure you want to
    Your message goes here
    Processing…
  • scalability
    Are you sure you want to
    Your message goes here
    Processing…
  • Slide 12 says 'replication' but looks more like sharding to me. Replication would be storing the same smiley face on more than one disk/database. Instead, you have several databases and each smiley is going to one of those. Slide 22 is the same but the cylinders are replaced with puzzle pieces, and this time the title is 'partitioning'. Slide 30 finally seems to have it right. Slide 48 is identical to slide 2. This probably makes more sense if you were there to hear this presentation.
    Are you sure you want to
    Your message goes here
    Processing…
  • slide 17: MySQL Cluster is synchronous and fault tolerant. It has other problems, but failing on master crash is not one of them (every node is a peer).
    Are you sure you want to
    Your message goes here
    Processing…
  • Below products can be used for database scalability

    Object caching products: memcached

    Database Caching Products: CSQL, Timesten
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

What Every Developer Should Know About Database Scalability What Every Developer Should Know About Database Scalability Presentation Transcript

  • Database scalability Jonathan Ellis
  • Classic RDBMS persistence Index Data
  • Disk is the new tape* • ~8ms to seek – ~4ms on expensive 15k rpm disks
  • performance What scaling means money
  • Performance • Latency • Throughput
  • Two kinds of operations • Reads • Writes
  • Caching • Memcached • Ehcache • etc cache DB
  • Cache invalidation • Implicit • Explicit
  • Cache set invalidation get_cached_cart(cart=13, offset=10, limit=10) get('cart:13:10:10') ?
  • Set invalidation 2 prefix = get('cart_prefix:13') get(prefix + ':10:10') del('cart_prefix:13') http://www.aminus.org/blogs/index.php/2007/1
  • Replication
  • Types of replication • Master → slave – Master → slave → other slaves • Master ↔ master – multi-master
  • Types of replication 2 • Synchronous • Asynchronous
  • Synchronous • Synchronous = slow(er) • Complexity (e.g. 2pc) • PGCluster • Oracle
  • Asynchronous master/slave • Easiest • Failover • MySQL replication • Slony, Londiste, WAL shipping • Tungsten
  • Asynchronous multi-master • Conflict resolution – O(N3) or O(N2) as you add nodes – http://research.microsoft.com/~gray/replicas.ps • Bucardo • MySQL Cluster
  • Achtung! • Asynchronous replication can lose data if the master fails
  • “Architecture” • Primarily about how you cope with failure scenarios
  • Replication does not scale writes
  • Scaling writes • Partitioning aka sharding – Key / horizontal – Vertical – Directed
  • Partitioning
  • Key based partitioning • PK of “root” table controls destination – e.g. user id • Retains referential integrity
  • Example: blogger.com Users Blogs Comments
  • Example: blogger.com Users Blogs Comments' Comments
  • Vertical partitioning • Tables on separate nodes • Often a table that is too big to keep with the other tables, gets too big for a single node
  • Growing is hard
  • Directed partitioning • Central db that knows what server owns a key • Makes adding machines easier • Single point of failure
  • Partitioning
  • Partitioning with replication
  • What these have in common • Ad hoc • Error-prone • Manpower-intensive
  • To summarize • Scaling reads sucks • Scaling writes sucks more
  • * Distributed databases • Data is automatically partitioned • Transparent to application • Add capacity without downtime • Failure tolerant *Like Bigtable, not Lotus Notes
  • Two famous papers • Bigtable: A distributed storage system for structured data, 2006 • Dynamo: amazon's highly available key- value store, 2007
  • The world doesn't need another half-assed key/value store (See also Olin Shivers' 100% and 80% solutions)
  • Two approaches • Bigtable: “How can we build a distributed database on top of GFS?” • Dynamo: “How can we build a distributed hash table appropriate for the data center?”
  • Bigtable architecture
  • Lookup in Bigtable
  • Dynamo
  • Eventually consistent • Amazon: http://www.allthingsdistributed.com/2008/12 • eBay: http://queue.acm.org/detail.cfm?id=139412
  • Consistency in a BASE world • If W + R > N, you are 100% consistent • W=1, R=N • W=N, R=1 • W=Q, R=Q where Q = N / 2 + 1
  • Cassandra
  • Memtable / SSTable Disk Commit log
  • ColumnFamilies keyA column1 column2 column3 keyC column1 column7 column11 Column Byte[] Name Byte[] Value I64 timestamp
  • LSM write properties • No reads • No seeks • Fast • Atomic within ColumnFamily
  • vs MySQL with 50GB of data • MySQL – ~300ms write – ~350ms read • Cassandra – ~0.12ms write – ~15ms read • Achtung!
  • Classic RDBMS persistence Index Data
  • Questions