Design Patterns for Distributed
  Non-Relational Databases
                  aka
 Just Enough Distributed Systems To Be
  ...
Introduction
Common Underlying Assumptions
Design Patterns
   Consistent Hashing
   Consistency Models
   Data Models
   S...
Why We’re All Here


     Scaling up doesn’t work
     Scaling out with traditional RDBMSs isn’t so
     hot either
      ...
Closed-source NRDBMSs
“The Inspiration”




           Google BigTable
                    Applications: webtable, Reader,...
Data Interfaces
“This is the NOSQL meetup, right?”




          Every row has a key (PK)
          Key/value get/put
    ...
Underlying Assumptions
Assumptions - Data Size


      The data does not fit on one node.
      The data may not fit on one rack.
      SANs are to...
Assumptions - Reliability
      The system must be highly available to serve
      web (and other) applications.
      Sin...
Assumptions - Performance
...and price thereof


            All systems we’re talking about today are
            meant f...
Design Patterns
Partitioning Schemes
“Where does a key live?”




          Given a key, we need to determine which
          node(s) it b...
Consistent Hashing
Maintaining hashing in a dynamic cluster
Consistent Hashing
Key Placement
Consistency Models

     A consistency model determines rules for
     visibility and apparent order of updates.
     Exam...
Strict Consistency

     All read operations must return the data from
     the latest completed write operation, regardle...
Eventual Consistency

     As t → ∞, readers will see writes.
     In a steady state, the system is guaranteed to
     eve...
Timestamps and Vector Clocks
Determining a history of a row



           Eventual consistency relies on deciding what
   ...
Vector Clocks

     Definition:
         A vector clock is a tuple {t1 , t2 , ..., tn } of clock
         values from each ...
Data Models
What’s in a row?




          Primary Key → Value
          Value could be:
                   Blob
         ...
Multi-Version Storage
Using Timestamps for a 3rd dimension



          Each table cell has a timestamp
          Timestam...
Storage Layouts
How do we lay out rows and columns on disk?




          Determines performance of different access
      ...
Row-based Storage




     Pros:
         Good locality of access (on disk and in cache) of
         different columns
    ...
Columnar Storage




     Pros:
         Data for a given column is stored sequentially
         Scanning a single column ...
Columnar Storage with Locality Groups




     Columns are organized into families (“locality
     groups”)
     Benefits o...
Log Structured Merge Trees
aka “The BigTable model”

           Random IO for writes is bad (and impossible in
           ...
LSM Data Layout
LSM Write Path
LSM Read Path
LSM Read Path + Bloom Filters
LSM Memtable Flush
LSM Compaction
Cluster Management

     Clients need to know where to find data
     (consistent hashing tokens, etc)
     Internal nodes ...
Omniscient Master

     When nodes join/leave or change state, they
     talk to a master
     That master holds the autho...
Gossip
     Gossip is one method to propagate a view of
     cluster status.
     Every t seconds, on each node:
         ...
Gossip - Initial State
Gossip - Round 1
Gossip - Round 2
Gossip - Round 3
Gossip - Round 4
Questions to Ask Presenters
Scalability and Reliability

      What are the scaling bottlenecks? How does it
      react when overloaded?
      Are th...
Performance

     What’s the goal? Batch throughput or request
     latency?
     How many seeks for reads? For writes? Ho...
Consistency

     What consistency model does the system
     provide?
     What situations would cause a lapse of
     co...
Cluster Management and Topology

     Does the system have a single master? Does it
     use gossip to spread cluster mana...
Data Model and Storage

     What data model and storage system does the
     system provide?
     Is it pluggable?
     W...
Data Access Methods


     What methods exist for accessing data? Can I
     access it from language X?
     Is there a wa...
Real Life Considerations
(I was talking about fake life in the first 45 slides)

            Who uses this system? How big ...
Questions?
http://cloudera-todd.s3.amazonaws.com/nosql.pdf
Upcoming SlideShare
Loading in...5
×

Design Patterns for Distributed Non-Relational Databases

59,862

Published on

Intro slides from http://nosql.net event (Jun 11, 09)

Published in: Technology
8 Comments
184 Likes
Statistics
Notes
  • 6 years after it was written, this is still an amazingly good summary of non-relational database architectures. It clarifies and summarizes a huge amount of still very relevent concepts (LSM trees, consistent hashing, etc)
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Really Great and helpful
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • GameAZ.us is a directory of free flash games - A web build on Cassandra Database
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Great presso. Good coverage. Thanks for sharing.
    Would be awesome if slideshare could also embed audio into it :)
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • The pattern format is well-suited to recap every way to address each aspect of the subject matter. This presentation is very useful for that. I would suggest to group between the patterns that are more relevant for the typical user of such distributed systems, and the patterns more useful for the implementors of such systems.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
59,862
On Slideshare
0
From Embeds
0
Number of Embeds
26
Actions
Shares
0
Downloads
2,431
Comments
8
Likes
184
Embeds 0
No embeds

No notes for slide

Transcript of "Design Patterns for Distributed Non-Relational Databases"

  1. 1. Design Patterns for Distributed Non-Relational Databases aka Just Enough Distributed Systems To Be Dangerous (in 40 minutes) Todd Lipcon (@tlipcon) Cloudera June 11, 2009
  2. 2. Introduction Common Underlying Assumptions Design Patterns Consistent Hashing Consistency Models Data Models Storage Layouts Log-Structured Merge Trees Cluster Management Omniscient Master Gossip Questions to Ask Presenters
  3. 3. Why We’re All Here Scaling up doesn’t work Scaling out with traditional RDBMSs isn’t so hot either Sharding scales, but you lose all the features that make RDBMSs useful! Sharding is operationally obnoxious. If we don’t need relational features, we want a distributed NRDBMS.
  4. 4. Closed-source NRDBMSs “The Inspiration” Google BigTable Applications: webtable, Reader, Maps, Blogger, etc. Amazon Dynamo Shopping Cart, ? Yahoo! PNUTS Applications: ?
  5. 5. Data Interfaces “This is the NOSQL meetup, right?” Every row has a key (PK) Key/value get/put multiget/multiput Range scan? With predicate pushdown? MapReduce? SQL?
  6. 6. Underlying Assumptions
  7. 7. Assumptions - Data Size The data does not fit on one node. The data may not fit on one rack. SANs are too expensive. Conclusion: The system must partition its data across many nodes.
  8. 8. Assumptions - Reliability The system must be highly available to serve web (and other) applications. Since the system runs on many nodes, nodes will crash during normal operation. Data must be safe even though disks and nodes will fail. Conclusion: The system must replicate each row to multiple nodes and remain available despite certain node and disk failure.
  9. 9. Assumptions - Performance ...and price thereof All systems we’re talking about today are meant for real-time use. 95th or 99th percentile is more important than average latency Commodity hardware and slow disks. Conclusion: The system needs to perform well on commodity hardware, and maintain low latency even during recovery operations.
  10. 10. Design Patterns
  11. 11. Partitioning Schemes “Where does a key live?” Given a key, we need to determine which node(s) it belongs on. If that node is down, we need to find another copy elsewhere. Difficulties: Unbounded number of keys. Dynamic cluster membership. Node failures.
  12. 12. Consistent Hashing Maintaining hashing in a dynamic cluster
  13. 13. Consistent Hashing Key Placement
  14. 14. Consistency Models A consistency model determines rules for visibility and apparent order of updates. Example: Row X is replicated on nodes M and N Client A writes row X to node N Some period of time t elapses. Client B reads row X from node M Does client B see the write from client A? Consistency is a continuum with tradeoffs
  15. 15. Strict Consistency All read operations must return the data from the latest completed write operation, regardless of which replica the operations went to Implies either: All operations for a given row go to the same node (replication for availability) or nodes employ some kind of distributed transaction protocol (eg 2 Phase Commit or Paxos) CAP Theorem: Strict Consistency can’t be achieved at the same time as availability and partition-tolerance.
  16. 16. Eventual Consistency As t → ∞, readers will see writes. In a steady state, the system is guaranteed to eventually return the last written value For example: DNS, or MySQL Slave Replication (log shipping) Special cases of eventual consistency: Read-your-own-writes consistency (“sent mail” box) Causal consistency (if you write Y after reading X, anyone who reads Y sees X) gmail has RYOW but not causal!
  17. 17. Timestamps and Vector Clocks Determining a history of a row Eventual consistency relies on deciding what value a row will eventually converge to In the case of two writers writing at “the same” time, this is difficult Timestamps are one solution, but rely on synchronized clocks and don’t capture causality Vector clocks are an alternative method of capturing order in a distributed system
  18. 18. Vector Clocks Definition: A vector clock is a tuple {t1 , t2 , ..., tn } of clock values from each node v1 < v2 if: For all i, v1i ≤ v2i For at least one i, v1i < v2i v1 < v2 implies global time ordering of events When data is written from node i, it sets ti to its clock value. This allows eventual consistency to resolve consistency between writes on multiple replicas.
  19. 19. Data Models What’s in a row? Primary Key → Value Value could be: Blob Structured (set of columns) Semi-structured (set of column families with arbitrary columns, eg linkto:<url> in webtable) Each has advantages and disadvantages Secondary Indexes? Tables/namespaces?
  20. 20. Multi-Version Storage Using Timestamps for a 3rd dimension Each table cell has a timestamp Timestamps don’t necessarily need to correspond to real life Multiple versions (and tombstones) can exist concurrently for a given row Reads may return “most recent”, “most recent before T”, etc. (free snapshots) System may provide optimistic concurrency control with compare-and-swap on timestamps
  21. 21. Storage Layouts How do we lay out rows and columns on disk? Determines performance of different access patterns Storage layout maps directly to disk access patterns Fast writes? Fast reads? Fast scans? Whole-row access or subsets of columns?
  22. 22. Row-based Storage Pros: Good locality of access (on disk and in cache) of different columns Read/write of a single row is a single IO operation. Cons: But if you want to scan only one column, you still read all.
  23. 23. Columnar Storage Pros: Data for a given column is stored sequentially Scanning a single column (eg aggregate queries) is fast Cons: Reading a single row may seek once per column.
  24. 24. Columnar Storage with Locality Groups Columns are organized into families (“locality groups”) Benefits of row-based layout within a group. Benefits of column-based - don’t have to read groups you don’t care about.
  25. 25. Log Structured Merge Trees aka “The BigTable model” Random IO for writes is bad (and impossible in some DFSs) LSM Trees convert random writes to sequential writes Writes go to a commit log and in-memory storage (Memtable) The Memtable is occasionally flushed to disk (SSTable) The disk stores are periodically compacted P. E. O’Neil, E. Cheng, D. Gawlick, and E. J. O’Neil. The log-structured merge-tree (LSM-tree). Acta Informatica. 1996.
  26. 26. LSM Data Layout
  27. 27. LSM Write Path
  28. 28. LSM Read Path
  29. 29. LSM Read Path + Bloom Filters
  30. 30. LSM Memtable Flush
  31. 31. LSM Compaction
  32. 32. Cluster Management Clients need to know where to find data (consistent hashing tokens, etc) Internal nodes may need to find each other as well Since nodes may fail and recover, a configuration file doesn’t really suffice We need a way of keeping some kind of consistent view of the cluster state
  33. 33. Omniscient Master When nodes join/leave or change state, they talk to a master That master holds the authoritative view of the world Pros: simplicity, single consistent view of the cluster Cons: potential SPOF unless master is made highly available. Not partition-tolerant.
  34. 34. Gossip Gossip is one method to propagate a view of cluster status. Every t seconds, on each node: The node selects some other node to chat with. The node reconciles its view of the cluster with its gossip buddy. Each node maintains a “timestamp” for itself and for the most recent information it has from every other node. Information about cluster state spreads in O(lgn) rounds (eventual consistency) Scalable and no SPOF, but state is only eventually consistent
  35. 35. Gossip - Initial State
  36. 36. Gossip - Round 1
  37. 37. Gossip - Round 2
  38. 38. Gossip - Round 3
  39. 39. Gossip - Round 4
  40. 40. Questions to Ask Presenters
  41. 41. Scalability and Reliability What are the scaling bottlenecks? How does it react when overloaded? Are there any single points of failure? When nodes fail, does the system maintain availability of all data? Does the system automatically re-replicate when replicas are lost? When new nodes are added, does the system automatically rebalance data?
  42. 42. Performance What’s the goal? Batch throughput or request latency? How many seeks for reads? For writes? How many net RTTs? What 99th percentile latencies have been measured in practice? How do failures impact serving latencies? What throughput has been measured in practice for bulk loads?
  43. 43. Consistency What consistency model does the system provide? What situations would cause a lapse of consistency, if any? Can consistency semantics be tweaked by configuration settings? Is there a way to do compare-and-swap on row contents for optimistic locking? Multirow?
  44. 44. Cluster Management and Topology Does the system have a single master? Does it use gossip to spread cluster management data? Can it withstand network partitions and still provide some level of service? Can it be deployed across multiple datacenters for disaster recovery? Can nodes be commissioned/decomissioned automatically without downtime? Operational hooks for monitoring and metrics?
  45. 45. Data Model and Storage What data model and storage system does the system provide? Is it pluggable? What IO patterns does the system cause under different workloads? Is the system best at random or sequential access? For read-mostly or write-mostly? Are there practical limits on key, value, or row sizes? Is compression available?
  46. 46. Data Access Methods What methods exist for accessing data? Can I access it from language X? Is there a way to perform filtering or selection at the server side? Are there bulk load tools to get data in/out efficiently? Is there a provision for data backup/restore?
  47. 47. Real Life Considerations (I was talking about fake life in the first 45 slides) Who uses this system? How big are the clusters it’s deployed on, and what kind of load do they handle? Who develops this system? Is this a community project or run by a single organization? Are outside contributions regularly accepted? Who supports this system? Is there an active community who will help me deploy it and debug issues? Docs? What is the open source license? What is the development roadmap?
  48. 48. Questions? http://cloudera-todd.s3.amazonaws.com/nosql.pdf
  1. ¿Le ha llamado la atención una diapositiva en particular?

    Recortar diapositivas es una manera útil de recopilar información importante para consultarla más tarde.

×