Scale Your Database And Be Happy


Published on

Today we're facing a paramount change in the data management field: more and more business applications are going to be contaminated with "social" aspects, requiring your data layer to be always available and perform well under increasing load conditions.
And while your relational database will be there to keep your transactional data in safe, you will need a whole new breed of data store to accommodate your availability and scalability needs: a so called "no-SQL" store.
In this talk you will learn about the forces driving this data layer revolution, and the most important patterns and products which will help you scale, stay available and smile happily at your "social" needs.

Published in: Technology
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Scale Your Database And Be Happy

  1. Scale Your Database And Be Happy Sergio Bossa @sbtourist Spring Framework Italian Meeting 2009
  2. About Me ➔ Software architect and engineer ➔ Gioco Digitale (online gambling and casinos) ➔ Open Source enthusiast ➔ Terracotta Messaging ( ➔ Actorom ( ➔ Terrastore (coming soon…) ➔ (Micro-)Blogger ➔ ➔ Sergio Bossa -
  3. Premise #1 Database ≠ Relational Database Sergio Bossa -
  4. Premise #2 Relational Databases Are Not Dead Sergio Bossa -
  5. Premise #3 You'll never hear the word NoSQL Here Sergio Bossa -
  6. Scaling Your Database … what? ● Scaling used as a loose term here. ● Scale to handle heterogeneous data. ● Scale to handle more data. ● Scale to handle more load. ● Scale to handle topology changes due to: ● Unplanned growth. ● Unpredictable failures. Sergio Bossa -
  7. Scaling Your Database … why? ● Scaling the way you handle your data is going to be more and more important. ● Business is moving toward data-centric applications. ● Let's call them “social”. ● Interest is toward efficient ways of: ● Storing … ● Serving … ● Analyzing … ● Data! Sergio Bossa -
  8. Scaling Your Relational Database Sergio Bossa -
  9. Replication ● Master - Slave replication. ● One (and only one) master database. ● One or more slaves. ● All writes goes to the master. ● Replicated to slaves. ● Reads are balanced among master and slaves. ● Major issues: ● Single point of failure. ● Single point of bottleneck. ● Static topology. Sergio Bossa -
  10. Replication ● Master - Master replication. ● One or more masters. ● Writes and reads can go to any master node. ● Writes are replicated among masters. ● Major issues: ● Limited performance and scalability (due to quorum). ● Complexity. ● Static topology. Sergio Bossa -
  11. Partitioning ● Vertical partitioning. ● Put tables belonging to different functional areas on different database nodes. ● Scale your data and load by function. ● Move joins to the application level. ● Major issues: ● No more truly relational. ● Limited scalability (what if a functional area grows too much?). Sergio Bossa -
  12. Partitioning ● Horizontal partitioning. ● Split tables by key and put partitions (shards) on different nodes. ● Scale your data and load by key. ● Move joins to the application level. ● Needs some kind of routing. ● Major issues: ● No more truly relational. ● Limited scalability (what if you need to rebalance?). Sergio Bossa -
  13. Caching ● Put a cache in front of your database. ● Distribute. ● Write-through for scaling reads. ● Write-behind for scaling reads and writes. ● Saves you a lot of pain, but ... ● “Only” scales read/write load. Sergio Bossa -
  14. Still left out ... ● We didn't scale our data model. ● Still bound to the relational data model. ● We didn't scale our topology. ● Still static. ● Hard to add nodes for handling growth. ● Hard to tolerate nodes leaving due to failures. Sergio Bossa -
  15. Non Relational Databases, coming... Sergio Bossa -
  16. Friends or Foes? We come in peace. To help our old friend: the relational database. Sergio Bossa -
  17. Requirements ● Flexible data model. ● Extreme reliability. ● Scale as you need. ● Scale at unplanned change in the data model. ● Scale at unplanned growth in data size. ● Scale at unplanned growth in load. Sergio Bossa -
  18. Data Model ● Column oriented (hybrid). ● Group by columns. ● Hybrid: group by keys and column families. ● Dynamically add columns. ● Different key-identified values may have different number of columns. ● Efficiently access the same group of columns (column family). Sergio Bossa -
  19. Data Model ● Document oriented. ● Group by named collections. ● Identify by key. ● Store a schema-less document. JSON. ● ● XML. ● Whatever ... ● Dynamically update your data model by simply changing your documents. ● Efficiently access whole documents. Sergio Bossa -
  20. Data Model ● Key/Value oriented. ● Group by named collections. ● Identify by key. ● Store an opaque value (whatever). ● Maybe the ancestor of modern non relationals. Sergio Bossa -
  21. Data Partitioning ● Consistent Hashing. ● Nodes mapped on a ring space of integers. Each node mapped on multiple locations. ● ● Each node owns a range of integers. ● Keys assigned to integers in the ring space. ● Stored on the owner node. ● Joining/Leaving nodes only affect the partition they're mapped to. ● Hence, keys re-balancing is limited to that specific range (efficient). Sergio Bossa -
  22. Data Partitioning Sergio Bossa -
  23. Data Consistency ● Strict (ACID) Consistency. ● All nodes ... ●At every point in time ... ● Hold a consistent view of the stored data. ● Reads and writes can executed on every node. ●Results will be always consistent and up-to- date. ● Due to the CAP Theorem you will sacrifice one of: ● Availability. ● Partition tolerance. Sergio Bossa -
  24. Data Consistency ● Eventual (BASE) Consistency. ● N: number of nodes you want to replicate to. ● W: number of required writes to succeed. ● R: number of required reads to succeed. ● W<N ●Nodes not receiving the write may eventually get that value later. ● R<N ● Nodes not holding the read value are ignored. Sergio Bossa -
  25. Data Consistency ● Eventual (BASE) Consistency. ● High read/write availability. ●Work even when some nodes fail to read and write values. ● Partition tolerance. ●Work even when some nodes cannot be reached anymore. ● Due to the CAP Theorem you are sacrificing consistency. Sergio Bossa -
  26. Data Versioning ● Vector Clocks. ● List of (node, counter) values associated to each object version. ● Every time a given object is read by a node, all its vector clocks are transferred. ● Every time a given object is written back by a node, counter for that node is incremented. ● A vector clock can express causal ordering. ● A vector clock can express branching. ● Read-time reconciliation (read repair). Sergio Bossa -
  27. Data Versioning ● Other... ● Multi-Version Concurrency Control. ●Each read/write operation works on a consistent snapshot. ● Optimistic concurrency. ●Write operations succeed only if their version is the current one. ● Last Wins (optionally with timestamps). ● Last write operation wins. ● Optionally, with the highest timestamp. Sergio Bossa -
  28. Data Recovery ● Hinted Handoff. ● Writes to unavailable nodes get directed to “secondary” nodes. ● Secondary nodes get an hint about the original destination node. ● When the node is available again, the secondary node send back the value. Sergio Bossa -
  29. Data Recovery ● Merkle Trees. ● For nodes missing large number of values (i.e. after disaster recovery). ● Nodes exchange a tree composed of: Leaves containing each the hash of a value ● hosted by the node. ● Parents containing each the hash of the children. ● Updated values are recovered by comparing hashes and reading back from healthy nodes. Sergio Bossa -
  30. Membership ● Master-based. ● Registry-like. ● Membership information maintained and broadcasted by one or more master nodes. ● Consistent. ● No SPOF with active/passive master. ● Prone to partitioning failures. Sergio Bossa -
  31. Membership ● Gossip-based. ● Peer-to-Peer. ● Membership information is randomly spread among nodes. ● Each node picks one or more nodes, broadcasting them its own topology view. ● All nodes will eventually reach a consistent view of the cluster topology. Sergio Bossa -
  32. Data Analysis ● The importance of data locality. ● A distributed system is built by: Moving data toward its behavior. ● ● ... or ... ● Moving behavior toward its data. ● An efficient distributed system is built by: ● Moving behavior toward its data. Sergio Bossa -
  33. Data Analysis ● Map-Reduce. ● Map data analysis and computation tasks toward the data itself. ● Reduce results. ● No need to move data around. Sergio Bossa -
  34. Use Cases (1) ● Runtime data. ● “Runtime” VS “Transactional”. ● Not all data need complex relations. ● Not all data need to be persisted forever. ●That is, everything regarding the current “runtime” state. ● User session and everything related. ● Put the “runtime” state into your N-RDBMS. ● When the “runtime” state turns into “transactional”, put it into your RDBMS. Sergio Bossa -
  35. Use Cases (2) ● Hot spots. ● For read-intensive data: Use your N-RDBMS as a primary database ● for reads. ● Use your RDBMS as a primary database for writes and load data into the N-RDBMS from a background thread. ● For read/write-intensive data: ● Use your N-RDBMS as a primary database for writes and reads. ● Put your data in your RDBMS from a background thread (if needed). Sergio Bossa -
  36. Use Cases (3) ● Intense data computations. ● When the relational model doesn't efficiently represent your data ... ● And join operations are just too expensive ... ● N-RDBMS come to rescue! ● Providing more efficient data representation/storage. ● Providing grid-style computations (i.e. Map- Reduce). Sergio Bossa -
  37. Products (1) ● MongoDB ● ● Document-based. ●(Binary) Json. ● Support for indexes and object queries. ● Full support for master-slave replication. ● Alpha support for sharding. ● ACID (unless failure scenarios during replication). Sergio Bossa -
  38. Products (2) ● Cassandra ● ● Column-based (hybrid). Keys. ● ● Column Families. ● Columns. ● Super-Columns. ● Support for ordered range queries. ● Fully distributed. ●Peer-to-Peer. ● Eventually consistent. Sergio Bossa -
  39. Products (3) ● Voldemort ● ● Key/Value. ●Pluggable data serialization. ● No support for queries. ● Fully distributed. ●Peer-to-Peer. ● Eventually consistent. Sergio Bossa -
  40. Products (4) ● Riak ● ● Document-based. Json. ● ● Links. ● Support for Map-Reduce. ● Fully distributed. ●Peer-to-Peer. ● Eventually consistent. ● With runtime dynamic tuning. Sergio Bossa -
  41. Final words ● Know how to scale your relational database. ● Don't dismiss it just to follow the hype. ● Know how non-relational databases scale. ● There are many choices around. ● Know your use cases. ● Make sensible decisions. ● Enjoy! ● And be happy! Sergio Bossa -
  42. Thank you! Q&A Sergio Bossa -