Riak Use Cases:         Dissecting the Solutions to         Hard Problems         Andy Gross <@argv0>         Principal Ar...
Riak              Dynamo-inspired key value database                   with full text search, map/reduce, secondary indice...
Choosing a NoSQL         Database              At small scale, everything works.              NoSQL DBs trade off traditio...
Tradeoffs              If you’re evaluating Mongo vs. Riak, or Couch vs.              Cassandra, you don’t understand your...
Distributed Systems:         Desirable Properties              Highly Available   Fault Tolerant              Low Latency ...
Medical Records Store         Danish Health Authorities              Implemented by Trifork A/S              Won the Digit...
User/Metadata Store         Comcast              User profile storage for xfinityTV mobile              application         ...
Notification Service         YammerWednesday, November 16, 11
Session Store         Mochi Media              First Basho Customer (late 2009)              Every hit to a Mochi web prop...
Document Store         Github Pages / Git.io              Riak as a web server for Github Pages (in              staging) ...
Distributed Systems:         Desirable Properties              High Availability        Fault Tolerance              Low L...
High Availability              Failure to accept a read/write results in:                   lost revenue                  ...
Low Latency           Sometimes late answer is useless or wrong           Users perceive slow sites as unavailable        ...
Fault Tolerance              Everything fails                   Especially in the cloud              When a host/disk/netw...
Predictability                       “It’s a piece of plumbing; it has never been                       a root cause of an...
CostWednesday, November 16, 11
Operational Costs              Sound familiar?                   “we chose a bad shard key...”                   “the fail...
High Availability:         Erlang              Ericcson AXD-301: 99.9999999% uptime (31ms/year)              Shared-nothin...
High Availability:         Riak Core              Dynamo abstracted: distributed systems toolkit              Exhaustively...
Low Latency:         Bitcask             Low Latency: All reads = hash lookup + 1 seek                                  Af...
Wednesday, November 16, 11
Low Latency:         Erlang VM              Erlang VM was designed for soft-realtime apps                   Preemptively s...
Wednesday, November 16, 11
Questions?Wednesday, November 16, 11
Upcoming SlideShare
Loading in...5
×

Riak Use Cases : Dissecting The Solutions To Hard Problems

4,335

Published on

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
4,335
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
38
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Riak Use Cases : Dissecting The Solutions To Hard Problems

  1. 1. Riak Use Cases: Dissecting the Solutions to Hard Problems Andy Gross <@argv0> Principal Architect, Basho Technologies QCon SF 2011Wednesday, November 16, 11
  2. 2. Riak Dynamo-inspired key value database with full text search, map/reduce, secondary indices, link traversal, commit hooks, HTTP and binary interfaces Written in Erlang (and C/C++) Open Source, Apache 2 licensed Enterprise features (multi-datacenter replication) and support available from BashoWednesday, November 16, 11
  3. 3. Choosing a NoSQL Database At small scale, everything works. NoSQL DBs trade off traditional features to better support new and emerging use cases Knowledge of the underlying system is essential NoSQL marketing is... “confusing”Wednesday, November 16, 11
  4. 4. Tradeoffs If you’re evaluating Mongo vs. Riak, or Couch vs. Cassandra, you don’t understand your problem By choosing Riak, you’ve already made tradeoffs: Sacrificing consistency for availability in failure scenarios A rich data/query model for a simple, scalable oneWednesday, November 16, 11
  5. 5. Distributed Systems: Desirable Properties Highly Available Fault Tolerant Low Latency Ops-Friendly Scalable PredictableWednesday, November 16, 11
  6. 6. Medical Records Store Danish Health Authorities Implemented by Trifork A/S Won the Digitization Prize as one of the “best government IT projects in Denmark” Stores medical prescription history for all Danish Citizens, replicated in 2 data centers. Accessed from pharmacies, hospital, mobile devices Replicated in multiple data centersWednesday, November 16, 11
  7. 7. User/Metadata Store Comcast User profile storage for xfinityTV mobile application Storage of metadata on content providers, and content licensing info Strict latency requirementsWednesday, November 16, 11
  8. 8. Notification Service YammerWednesday, November 16, 11
  9. 9. Session Store Mochi Media First Basho Customer (late 2009) Every hit to a Mochi web property = 1 read, maybe one write to Riak Unavailability, high latency = lost ad revenueWednesday, November 16, 11
  10. 10. Document Store Github Pages / Git.io Riak as a web server for Github Pages (in staging) Webmachine is an awesome HTTP server! Git.io URL shortenerWednesday, November 16, 11
  11. 11. Distributed Systems: Desirable Properties High Availability Fault Tolerance Low Latency Ops-Friendliness Horizontal Scalability PredictabilityWednesday, November 16, 11
  12. 12. High Availability Failure to accept a read/write results in: lost revenue lost users Availability and latency are intertwinedWednesday, November 16, 11
  13. 13. Low Latency Sometimes late answer is useless or wrong Users perceive slow sites as unavailable SLA violations SOA approaches magnify SLA failuresWednesday, November 16, 11
  14. 14. Fault Tolerance Everything fails Especially in the cloud When a host/disk/network fails, what is the impact on Availability Latency Operations staffWednesday, November 16, 11
  15. 15. Predictability “It’s a piece of plumbing; it has never been a root cause of any of our problems.” Coda Hale, YammerWednesday, November 16, 11
  16. 16. CostWednesday, November 16, 11
  17. 17. Operational Costs Sound familiar? “we chose a bad shard key...” “the failover script did not run as expected...” “the root cause was traced to a configuration error...” Staying up all night fighting your database does not make you a hero.Wednesday, November 16, 11
  18. 18. High Availability: Erlang Ericcson AXD-301: 99.9999999% uptime (31ms/year) Shared-nothing, immutable, message-passing, functional, concurrent Distributed systems primitives in core language OTP (Open Telecom Platform)Wednesday, November 16, 11
  19. 19. High Availability: Riak Core Dynamo abstracted: distributed systems toolkit Exhaustively tested In production use at AOL, Yahoo, others Insulates local storage and client API code from the hard problemsWednesday, November 16, 11
  20. 20. Low Latency: Bitcask Low Latency: All reads = hash lookup + 1 seek After the append completes, an in-memory structure called a ”keydir” is updated. A keydir is simply a hash table that maps every key in a Bitcask to a fixed-size structure giving the file, offset, and size of the most recently written entry for that key. When a write occurs, the keydir is atomically updated with the location of the newest data. The old data is still present on disk, but any new reads will use the latest version available in the keydir. As we’ll see later, the Tradeoff: Index must fit in memory merge process will eventually remove the old value. Reading a value is simple, and doesn’t ever require more than a single disk seek. We look up the key in our keydir, and from there we read the data using the file id, position, and size that are returned from that lookup. In many cases, the operating system’s filesystem read-ahead cache makes this a much faster operation than would be otherwise expected.Wednesday, November 16, 11
  21. 21. Wednesday, November 16, 11
  22. 22. Low Latency: Erlang VM Erlang VM was designed for soft-realtime apps Preemptively scheduled lightweight threads GC is per-thread, not stop-the-world Sophisticated scheduler + message passing = effective use of multicore machines.Wednesday, November 16, 11
  23. 23. Wednesday, November 16, 11
  24. 24. Questions?Wednesday, November 16, 11
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×