Dynamo and BigTable in light of the CAP theorem

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Dynamo and BigTable
In light of the CAP Theorem

22953 Research Seminar: Databases and Data Mining
November 2013
Open Univ...

Overview
• Introduction to DDBS and NoSQL
• CAP Theorem
• Dynamo (AP)

• BigTable (CP)
• Dynamo vs. BigTable

Distributed Database Systems
• Data is stored across several sites that share no
physical component.
• Systems that run on...

Distributed Data Storage
Partitioning :
Data is partitioned into
several fragments and
stored in different sites.
Horizont...

Partitioning
A

B

key
key

value

x

10

w

12

value

x

5

y

7

z

10

A

B

12

5

7
10
12

value

w

x

w

key

valu...

Replication
A

B

key

value

x

x

5

y

7

z

10

10

w

5

7

z

value

5

y

key

x
key

value

12

C

D

key

value

...

Updating distributed data
• Quorum voting (Gifford SOSP’79) :






N – number of replicas.
At least R copies should ...

NoSQL
• No SQL :
– Not RDBMS.
– Not using SQL language.
– Not only SQL ?

• Flexible schema
• Horizontal scalability
• Rel...

Overview
• Introduction to DDBS and NoSQL √
• CAP Theorem
• Dynamo (AP)

• BigTable (CP)
• Dynamo vs. BigTable

CAP Theorem
• Eric A. Brewer. Towards robust distributed systems (Invited Talk)
, July 2000
• S. Gilbert and N. Lynch, Bre...

CAP Theorem
• Consistency – equivalent to having a single up-to-date
copy of the data.
• Availability - every request rece...

CAP Theorem
Consistency

x=5

x?

5

Availability

Partition tolerance

CAP Theorem - Proof
x=0

x=0

x=0

x=5

x?

1. x=0  Not consistent
2. No response  Not available

CAP – 2 of 3

Consistency

Availability

Partition
Tolerance

• Trivial:
– The trivial system that ignores all requests me...

CAP – 2 of 3

Consistency

Availability

Partition
Tolerance

• Trivial:
– The service can trivially return the initial va...

CAP – 2 of 3

Consistency

Availability

Partition
Tolerance

• If there are no partitions, it is clearly possible to prov...

Overview
• Introduction to DDBS and NoSQL √
• CAP Theorem √
• Dynamo (AP)

• BigTable (CP)
• Dynamo vs. BigTable

Dynamo - Introduction
•
•
•
•

Highly available key-value storage system.
Provides an “always-on” experience.
Prefers avai...

Dynamo - API
• Distributed hash table :
– put(key, context, object) – Associates given object with
specified key and conte...

Dynamo - Partitioning
• Naive approach :
– Hash the key.
– Apply modulo n (n=number of nodes).

key = “John Smith”

hash(k...

Dynamo - Partitioning
• Consistent hashing (STOC’97) :
– Each node is assigned to a random position on the ring.
– Key is ...

Dynamo - Partitioning
• Virtual nodes :
– Each physical node is assigned to multiple points in the ring.

Dynamo - Replication
•

Each data item is replicated at N nodes.

•

Preference list - the list of nodes that store a part...

Dynamo - Data Versioning
• Eventual consistency(replicas are updated
asynchronously) :
– A put() call may return to its ca...

Dynamo - Data Versioning
• Semantic Reconciliation :
Item 1
Item 6

get()
{ item1, item4}, {item1, item6}
put({ item1, ite...

Dynamo - Data Versioning
•

Vector clocks(Lamport CACM’78)
– List of <node, counter> pairs.
– One vector clock is associat...

Dynamo – get() and put()
• Two strategies that a client can use :
– Load balancer that will select a node based on load
in...

Dynamo – get() and put()
• Quorum-like system :
 R - number of nodes that must read a key.
 W - number of nodes that mus...

Dynamo – Handling Failures
• Sloppy Quorum and Hinted Handoff :
– All read and write operations are performed on the first...

Dynamo – Handling Failures
• Anti-entropy :
– Protocol to keep the replicas synchronized.
– To detect inconsistencies – Me...

Dynamo – Membership detection
• Gossip-based protocol :
– Periodically, each node contacts another node in the network,
ra...

Dynamo - Summary

Overview
• Introduction to DDBS and NoSQL √
• CAP Theorem √
• Dynamo (AP) √

• BigTable (CP)
• Dynamo vs. BigTable

BigTable - Introduction
• Distributed storage system for managing
structured data that is designed to scale to a
very larg...

BigTable – Data Model
• Bigtable is a sparse, distributed, persistent
multidimensional sorted map.
• The map is indexed by...

BigTable – Data Model
user_id

phone

15
row_key

name
John

178145

(29, name, t2)  “Robert”
user_id

column_key

name

...

BigTable – Data Model
• Columns are grouped into Column Families:
– family : optional qualifier
Column
Family

Optional
Qu...

BigTable – Data Model
• Rows :
– The row keys in a table are arbitrary strings.
– Every read or write of data under a sing...

BigTable – Data Model
• Column Families :
– Column keys are grouped into sets called column families.
– A column family mu...

BigTable – Data Model
• Timestamps :
– Each cell in a Bigtable can contain multiple versions of the
same data; these versi...

BigTable – Data Model
• Tablets :
– Large tables broken into tablets at row boundaries.
– Tablet holds contiguous range of...

BigTable – API
• Metadata operations :
– Creating and deleting tables, column families, modify access
control rights.

• C...

BigTable – Building Blocks
• GFS – large-scale distributed file system.
GHEMAWAT, GOBIOFF, LEUNG, The Google file system. ...

BigTable – Building Blocks
• GFS :
– Files broken into chunks (typically 64 MB)
– Master manages metadata.
– Data transfer...

BigTable – Building Blocks
• Chubby :
– Provides a namespace of directories and small files :
• Each directory or file can...

BigTable – Building Blocks
• SSTable :
– Stored in GFS.
– Persistent, ordered, immutable map from keys to values.
– Provid...

BigTable – System Structure
• Three major components:
– Client library
– Master (exactly one) :
•
•
•
•
•

Assigning table...

BigTable – System Structure

BigTable – Tablet Location
• Three-level hierarchy analogous to that of a B+ tree to store
tablet location information.
• ...

BigTable – Tablet Assignment
•

Tablet Server:
– When a tablet server starts, it creates, and acquires an exclusive lock o...

BigTable - Tablet Assignment
Master

Metadata
Table

Tablet Server 1

Tablet Server 2

Tablet 1 Tablet 8

Tablet 7 Tablet ...

BigTable – Tablet Serving
•

Writes :
– Updates committed to a commit log.
– Recently committed updates are stored in memo...

BigTable - Compactions
• Minor compaction:
– Converts the memtable into SSTable.
– Reduces memory usage.
– Reduces log rea...

BigTable – Bloom Filters
Bloom Filter :
1. Empty array a of m bits, all set to 0.
2. Hash function h, such that h hashes e...

BigTable – Bloom Filters

• Drastically reduces the number of disk seeks
required for read operations !

Overview
• Introduction to DDBS and NoSQL √
• CAP Theorem √
• Dynamo (AP) √

• BigTable (CP) √
• Dynamo vs. BigTable

Dynamo vs. BigTable
Dynamo

BigTable

data model

key-value

multidimensional map

operations

by key

by key range

parti...

Overview
• Introduction to DDBS and NoSQL √
• CAP Theorem √
• Dynamo (AP) √

• BigTable (CP) √
• Dynamo vs. BigTable √

References
•

R.Ramakrishnan and J.Gehrke, Database Management Systems, 3rd edition, pp. 736-751

•

S. Gilbert and N. Lyn...

Dynamo and BigTable in light of the CAP theorem

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

Thinking Outside the Table by Graphwise (Previo... 66955 views

View on Slideshare

1 of 60 Ad

View on Slideshare

1 of 60 Ad