Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Consistent Hashing
The problem

•

Distribute items into buckets

•

Fetch an item from its bucket
Naive Solution

hash(key) % buckets
Naive solution
0

foo

3
4

baz

1

1
2

0

bar

2
parker

3
warby
Other considerations
•

Given a resource key and a list of servers, how do you find a
primary, second, tertiary (and on dow...
Benefits
•

Weighted distributions based on the number of
replicas per server.

•

New servers can be brought online gradua...
Real world usage
•

memcached

•

redis

•

graphite

•

Dynamo (Amazon)

•

BigTable (Google)
•

Consistent Hashing and Random Trees: Distributed
Caching Protocols for Relieving Hot Spots on the
World Wide Web

http:...
Consistent hashing
Consistent hashing
Consistent hashing
Upcoming SlideShare
Loading in …5
×

Consistent hashing

1,514 views

Published on

Tech roulette presentation given at Warby Parker tech team meeting.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Consistent hashing

  1. 1. Consistent Hashing
  2. 2. The problem • Distribute items into buckets • Fetch an item from its bucket
  3. 3. Naive Solution hash(key) % buckets
  4. 4. Naive solution 0 foo 3 4 baz 1 1 2 0 bar 2 parker 3 warby
  5. 5. Other considerations • Given a resource key and a list of servers, how do you find a primary, second, tertiary (and on down the line) server for the resource? • If you have different size servers, how do you assign each of them an amount of work that corresponds to their capacity? • How do you smoothly add capacity to the system without downtime? Specifically, this means solving two problems: • How do you avoid dumping 1/N of the total load on a new server as soon as you turn it on? • How do you avoid rehashing more existing keys than necessary?
  6. 6. Benefits • Weighted distributions based on the number of replicas per server. • New servers can be brought online gradually. • Simple data replication.
  7. 7. Real world usage • memcached • redis • graphite • Dynamo (Amazon) • BigTable (Google)
  8. 8. • Consistent Hashing and Random Trees: Distributed Caching Protocols for Relieving Hot Spots on the World Wide Web
 http://thor.cs.ucsb.edu/~ravenben/papers/coreos/kll+97.pdf • http://www.tomkleinpeter.com/2008/03/17/ programmers-toolbox-part-3-consistent-hashing/ • https://weblogs.java.net/blog/tomwhite/archive/ 2007/11/consistent_hash.html

×