SlideShare a Scribd company logo
1 of 24
Download to read offline
Riak Use Cases:
         Dissecting the Solutions to
         Hard Problems
         Andy Gross <@argv0>
         Principal Architect, Basho Technologies
         QCon SF 2011




Wednesday, November 16, 11
Riak
              Dynamo-inspired key value database
                   with full text search, map/reduce, secondary indices,
                   link traversal, commit hooks, HTTP and binary
                   interfaces
              Written in Erlang (and C/C++)
              Open Source, Apache 2 licensed
              Enterprise features (multi-datacenter replication) and
              support available from Basho


Wednesday, November 16, 11
Choosing a NoSQL
         Database

              At small scale, everything works.
              NoSQL DBs trade off traditional features to better
              support new and emerging use cases
              Knowledge of the underlying system is essential
              NoSQL marketing is... “confusing”




Wednesday, November 16, 11
Tradeoffs

              If you’re evaluating Mongo vs. Riak, or Couch vs.
              Cassandra, you don’t understand your problem
              By choosing Riak, you’ve already made tradeoffs:
                   Sacrificing consistency for availability in failure
                   scenarios
                   A rich data/query model for a simple, scalable one



Wednesday, November 16, 11
Distributed Systems:
         Desirable Properties
              Highly Available   Fault Tolerant
              Low Latency        Ops-Friendly
              Scalable           Predictable




Wednesday, November 16, 11
Medical Records Store
         Danish Health Authorities
              Implemented by Trifork A/S
              Won the Digitization Prize as one of the “best
              government IT projects in Denmark”
              Stores medical prescription history for all
              Danish Citizens, replicated in 2 data centers.
              Accessed from pharmacies, hospital, mobile
              devices
              Replicated in multiple data centers


Wednesday, November 16, 11
User/Metadata Store
         Comcast

              User profile storage for xfinityTV mobile
              application
              Storage of metadata on content providers, and
              content licensing info
              Strict latency requirements




Wednesday, November 16, 11
Notification Service
         Yammer




Wednesday, November 16, 11
Session Store
         Mochi Media

              First Basho Customer (late 2009)
              Every hit to a Mochi web property = 1 read,
              maybe one write to Riak
              Unavailability, high latency = lost ad revenue




Wednesday, November 16, 11
Document Store
         Github Pages / Git.io

              Riak as a web server for Github Pages (in
              staging)
              Webmachine is an awesome HTTP server!
              Git.io URL shortener




Wednesday, November 16, 11
Distributed Systems:
         Desirable Properties
              High Availability        Fault Tolerance
              Low Latency              Ops-Friendliness
              Horizontal Scalability   Predictability




Wednesday, November 16, 11
High Availability

              Failure to accept a read/write results in:
                   lost revenue
                   lost users
              Availability and latency are intertwined




Wednesday, November 16, 11
Low Latency

           Sometimes late answer is useless or wrong
           Users perceive slow sites as unavailable
           SLA violations
           SOA approaches magnify SLA failures




Wednesday, November 16, 11
Fault Tolerance
              Everything fails
                   Especially in the cloud
              When a host/disk/network fails, what is the impact on
                   Availability
                   Latency
                   Operations staff



Wednesday, November 16, 11
Predictability

                       “It’s a piece of plumbing; it has never been
                       a root cause of any of our problems.”

                       Coda Hale, Yammer




Wednesday, November 16, 11
Cost




Wednesday, November 16, 11
Operational Costs
              Sound familiar?
                   “we chose a bad shard key...”
                   “the failover script did not run as expected...”
                   “the root cause was traced to a configuration
                   error...”
              Staying up all night fighting your database does
              not make you a hero.



Wednesday, November 16, 11
High Availability:
         Erlang

              Ericcson AXD-301: 99.9999999% uptime (31ms/year)
              Shared-nothing, immutable, message-passing,
              functional, concurrent
              Distributed systems primitives in core language
              OTP (Open Telecom Platform)




Wednesday, November 16, 11
High Availability:
         Riak Core

              Dynamo abstracted: distributed systems toolkit
              Exhaustively tested
              In production use at AOL, Yahoo, others
              Insulates local storage and client API code from the
              hard problems




Wednesday, November 16, 11
Low Latency:
         Bitcask
             Low Latency: All reads = hash lookup + 1 seek
                                  After the append completes, an in-memory structure called a ”keydir” is updated. A keydir is simply a hash
                               table that maps every key in a Bitcask to a fixed-size structure giving the file, offset, and size of the most recently
                               written entry for that key.




                                   When a write occurs, the keydir is atomically updated with the location of the newest data. The old data is
                               still present on disk, but any new reads will use the latest version available in the keydir. As we’ll see later, the

                             Tradeoff: Index must fit in memory
                               merge process will eventually remove the old value.
                                   Reading a value is simple, and doesn’t ever require more than a single disk seek. We look up the key in our
                               keydir, and from there we read the data using the file id, position, and size that are returned from that lookup. In
                               many cases, the operating system’s filesystem read-ahead cache makes this a much faster operation than would
                               be otherwise expected.
Wednesday, November 16, 11
Wednesday, November 16, 11
Low Latency:
         Erlang VM

              Erlang VM was designed for soft-realtime apps
                   Preemptively scheduled lightweight threads
                   GC is per-thread, not stop-the-world
              Sophisticated scheduler + message passing = effective
              use of multicore machines.




Wednesday, November 16, 11
Wednesday, November 16, 11
Questions?




Wednesday, November 16, 11

More Related Content

What's hot

soft-shake.ch - Data grids and Data Caching
soft-shake.ch - Data grids and Data Cachingsoft-shake.ch - Data grids and Data Caching
soft-shake.ch - Data grids and Data Cachingsoft-shake.ch
 
Docker 101 for Oracle DBAs - Oracle OpenWorld 2017
Docker 101 for Oracle DBAs - Oracle OpenWorld 2017Docker 101 for Oracle DBAs - Oracle OpenWorld 2017
Docker 101 for Oracle DBAs - Oracle OpenWorld 2017Adeesh Fulay
 
Paul Dix [InfluxData] | InfluxDays Opening Keynote | InfluxDays Virtual Exper...
Paul Dix [InfluxData] | InfluxDays Opening Keynote | InfluxDays Virtual Exper...Paul Dix [InfluxData] | InfluxDays Opening Keynote | InfluxDays Virtual Exper...
Paul Dix [InfluxData] | InfluxDays Opening Keynote | InfluxDays Virtual Exper...InfluxData
 
Codemotion 2015 Infinispan Tech lab
Codemotion 2015 Infinispan Tech labCodemotion 2015 Infinispan Tech lab
Codemotion 2015 Infinispan Tech labUgo Landini
 
CockroachDB: Architecture of a Geo-Distributed SQL Database
CockroachDB: Architecture of a Geo-Distributed SQL DatabaseCockroachDB: Architecture of a Geo-Distributed SQL Database
CockroachDB: Architecture of a Geo-Distributed SQL DatabaseC4Media
 
{code} and Containers - Open Source Infrastructure within Dell Technologies
{code} and Containers - Open Source Infrastructure within Dell Technologies{code} and Containers - Open Source Infrastructure within Dell Technologies
{code} and Containers - Open Source Infrastructure within Dell TechnologiesThe {code} Team
 
NoSQL overview implementation free
NoSQL overview implementation freeNoSQL overview implementation free
NoSQL overview implementation freeBenoit Perroud
 
Designing for Massive Scalability at BackType #bigdatacamp
Designing for Massive Scalability at BackType #bigdatacampDesigning for Massive Scalability at BackType #bigdatacamp
Designing for Massive Scalability at BackType #bigdatacampMichael Montano
 
The Hows and Whys of a Distributed SQL Database - Strange Loop 2017
The Hows and Whys of a Distributed SQL Database - Strange Loop 2017The Hows and Whys of a Distributed SQL Database - Strange Loop 2017
The Hows and Whys of a Distributed SQL Database - Strange Loop 2017Alex Robinson
 
keyvi the key value index @ Cliqz
keyvi the key value index @ Cliqzkeyvi the key value index @ Cliqz
keyvi the key value index @ CliqzHendrik Muhs
 
D conf 2016 - Using D for Primary Storage
D conf 2016 - Using D for Primary StorageD conf 2016 - Using D for Primary Storage
D conf 2016 - Using D for Primary StorageLiran Zvibel
 
Containers in depth – Understanding how containers work to better work with c...
Containers in depth – Understanding how containers work to better work with c...Containers in depth – Understanding how containers work to better work with c...
Containers in depth – Understanding how containers work to better work with c...All Things Open
 
Microservices and Cloud Native Apps Meetup with Diamanti and Nirmata
Microservices and Cloud Native Apps Meetup with Diamanti and NirmataMicroservices and Cloud Native Apps Meetup with Diamanti and Nirmata
Microservices and Cloud Native Apps Meetup with Diamanti and NirmataFred Love
 

What's hot (13)

soft-shake.ch - Data grids and Data Caching
soft-shake.ch - Data grids and Data Cachingsoft-shake.ch - Data grids and Data Caching
soft-shake.ch - Data grids and Data Caching
 
Docker 101 for Oracle DBAs - Oracle OpenWorld 2017
Docker 101 for Oracle DBAs - Oracle OpenWorld 2017Docker 101 for Oracle DBAs - Oracle OpenWorld 2017
Docker 101 for Oracle DBAs - Oracle OpenWorld 2017
 
Paul Dix [InfluxData] | InfluxDays Opening Keynote | InfluxDays Virtual Exper...
Paul Dix [InfluxData] | InfluxDays Opening Keynote | InfluxDays Virtual Exper...Paul Dix [InfluxData] | InfluxDays Opening Keynote | InfluxDays Virtual Exper...
Paul Dix [InfluxData] | InfluxDays Opening Keynote | InfluxDays Virtual Exper...
 
Codemotion 2015 Infinispan Tech lab
Codemotion 2015 Infinispan Tech labCodemotion 2015 Infinispan Tech lab
Codemotion 2015 Infinispan Tech lab
 
CockroachDB: Architecture of a Geo-Distributed SQL Database
CockroachDB: Architecture of a Geo-Distributed SQL DatabaseCockroachDB: Architecture of a Geo-Distributed SQL Database
CockroachDB: Architecture of a Geo-Distributed SQL Database
 
{code} and Containers - Open Source Infrastructure within Dell Technologies
{code} and Containers - Open Source Infrastructure within Dell Technologies{code} and Containers - Open Source Infrastructure within Dell Technologies
{code} and Containers - Open Source Infrastructure within Dell Technologies
 
NoSQL overview implementation free
NoSQL overview implementation freeNoSQL overview implementation free
NoSQL overview implementation free
 
Designing for Massive Scalability at BackType #bigdatacamp
Designing for Massive Scalability at BackType #bigdatacampDesigning for Massive Scalability at BackType #bigdatacamp
Designing for Massive Scalability at BackType #bigdatacamp
 
The Hows and Whys of a Distributed SQL Database - Strange Loop 2017
The Hows and Whys of a Distributed SQL Database - Strange Loop 2017The Hows and Whys of a Distributed SQL Database - Strange Loop 2017
The Hows and Whys of a Distributed SQL Database - Strange Loop 2017
 
keyvi the key value index @ Cliqz
keyvi the key value index @ Cliqzkeyvi the key value index @ Cliqz
keyvi the key value index @ Cliqz
 
D conf 2016 - Using D for Primary Storage
D conf 2016 - Using D for Primary StorageD conf 2016 - Using D for Primary Storage
D conf 2016 - Using D for Primary Storage
 
Containers in depth – Understanding how containers work to better work with c...
Containers in depth – Understanding how containers work to better work with c...Containers in depth – Understanding how containers work to better work with c...
Containers in depth – Understanding how containers work to better work with c...
 
Microservices and Cloud Native Apps Meetup with Diamanti and Nirmata
Microservices and Cloud Native Apps Meetup with Diamanti and NirmataMicroservices and Cloud Native Apps Meetup with Diamanti and Nirmata
Microservices and Cloud Native Apps Meetup with Diamanti and Nirmata
 

Viewers also liked

Riak add presentation
Riak add presentationRiak add presentation
Riak add presentationIlya Bogunov
 
Building Distributed Systems With Riak and Riak Core
Building Distributed Systems With Riak and Riak CoreBuilding Distributed Systems With Riak and Riak Core
Building Distributed Systems With Riak and Riak CoreAndy Gross
 
Scaling with Riak at Showyou
Scaling with Riak at ShowyouScaling with Riak at Showyou
Scaling with Riak at ShowyouJohn Muellerleile
 
Redis : Play buzz uses Redis
Redis : Play buzz uses RedisRedis : Play buzz uses Redis
Redis : Play buzz uses RedisRedis Labs
 
Modelling Data as Graphs (Neo4j)
Modelling Data as Graphs (Neo4j)Modelling Data as Graphs (Neo4j)
Modelling Data as Graphs (Neo4j)Michal Bachman
 
An intro to Neo4j and some use cases (JFokus 2011)
An intro to Neo4j and some use cases (JFokus 2011)An intro to Neo4j and some use cases (JFokus 2011)
An intro to Neo4j and some use cases (JFokus 2011)Emil Eifrem
 
Recommendations with Neo4j (FOSDEM 2015)
Recommendations with Neo4j (FOSDEM 2015)Recommendations with Neo4j (FOSDEM 2015)
Recommendations with Neo4j (FOSDEM 2015)Michal Bachman
 
Advanced Neo4j Use Cases with the GraphAware Framework
Advanced Neo4j Use Cases with the GraphAware FrameworkAdvanced Neo4j Use Cases with the GraphAware Framework
Advanced Neo4j Use Cases with the GraphAware FrameworkMichal Bachman
 
Modelling Data in Neo4j (plus a few tips)
Modelling Data in Neo4j (plus a few tips)Modelling Data in Neo4j (plus a few tips)
Modelling Data in Neo4j (plus a few tips)Michal Bachman
 
Best Buy Web 2.0
Best Buy Web 2.0Best Buy Web 2.0
Best Buy Web 2.0Lee Aase
 
Introduction to Redis Data Structures: Sorted Sets
Introduction to Redis Data Structures: Sorted SetsIntroduction to Redis Data Structures: Sorted Sets
Introduction to Redis Data Structures: Sorted SetsScaleGrid.io
 
Redis and its many use cases
Redis and its many use casesRedis and its many use cases
Redis and its many use casesChristian Joudrey
 
Introduction to some top Redis use cases
Introduction to some top Redis use casesIntroduction to some top Redis use cases
Introduction to some top Redis use casesJosiah Carlson
 
The BestBuy.com Cloud Architecture
The BestBuy.com Cloud ArchitectureThe BestBuy.com Cloud Architecture
The BestBuy.com Cloud Architecturejoelcrabb
 
Using Redis at Facebook
Using Redis at FacebookUsing Redis at Facebook
Using Redis at FacebookRedis Labs
 
Bootstrapping Recommendations with Neo4j
Bootstrapping Recommendations with Neo4jBootstrapping Recommendations with Neo4j
Bootstrapping Recommendations with Neo4jMax De Marzi
 
Neo4j GraphTalks - Einführung in Graphdatenbanken
Neo4j GraphTalks - Einführung in GraphdatenbankenNeo4j GraphTalks - Einführung in Graphdatenbanken
Neo4j GraphTalks - Einführung in GraphdatenbankenNeo4j
 
Redis Use Patterns (DevconTLV June 2014)
Redis Use Patterns (DevconTLV June 2014)Redis Use Patterns (DevconTLV June 2014)
Redis Use Patterns (DevconTLV June 2014)Itamar Haber
 
Neo4j the Anti Crime Database
Neo4j the Anti Crime DatabaseNeo4j the Anti Crime Database
Neo4j the Anti Crime DatabaseNeo4j
 

Viewers also liked (20)

Riak add presentation
Riak add presentationRiak add presentation
Riak add presentation
 
Building Distributed Systems With Riak and Riak Core
Building Distributed Systems With Riak and Riak CoreBuilding Distributed Systems With Riak and Riak Core
Building Distributed Systems With Riak and Riak Core
 
Scaling with Riak at Showyou
Scaling with Riak at ShowyouScaling with Riak at Showyou
Scaling with Riak at Showyou
 
Redis : Play buzz uses Redis
Redis : Play buzz uses RedisRedis : Play buzz uses Redis
Redis : Play buzz uses Redis
 
Modelling Data as Graphs (Neo4j)
Modelling Data as Graphs (Neo4j)Modelling Data as Graphs (Neo4j)
Modelling Data as Graphs (Neo4j)
 
An intro to Neo4j and some use cases (JFokus 2011)
An intro to Neo4j and some use cases (JFokus 2011)An intro to Neo4j and some use cases (JFokus 2011)
An intro to Neo4j and some use cases (JFokus 2011)
 
Recommendations with Neo4j (FOSDEM 2015)
Recommendations with Neo4j (FOSDEM 2015)Recommendations with Neo4j (FOSDEM 2015)
Recommendations with Neo4j (FOSDEM 2015)
 
Redis use cases
Redis use casesRedis use cases
Redis use cases
 
Advanced Neo4j Use Cases with the GraphAware Framework
Advanced Neo4j Use Cases with the GraphAware FrameworkAdvanced Neo4j Use Cases with the GraphAware Framework
Advanced Neo4j Use Cases with the GraphAware Framework
 
Modelling Data in Neo4j (plus a few tips)
Modelling Data in Neo4j (plus a few tips)Modelling Data in Neo4j (plus a few tips)
Modelling Data in Neo4j (plus a few tips)
 
Best Buy Web 2.0
Best Buy Web 2.0Best Buy Web 2.0
Best Buy Web 2.0
 
Introduction to Redis Data Structures: Sorted Sets
Introduction to Redis Data Structures: Sorted SetsIntroduction to Redis Data Structures: Sorted Sets
Introduction to Redis Data Structures: Sorted Sets
 
Redis and its many use cases
Redis and its many use casesRedis and its many use cases
Redis and its many use cases
 
Introduction to some top Redis use cases
Introduction to some top Redis use casesIntroduction to some top Redis use cases
Introduction to some top Redis use cases
 
The BestBuy.com Cloud Architecture
The BestBuy.com Cloud ArchitectureThe BestBuy.com Cloud Architecture
The BestBuy.com Cloud Architecture
 
Using Redis at Facebook
Using Redis at FacebookUsing Redis at Facebook
Using Redis at Facebook
 
Bootstrapping Recommendations with Neo4j
Bootstrapping Recommendations with Neo4jBootstrapping Recommendations with Neo4j
Bootstrapping Recommendations with Neo4j
 
Neo4j GraphTalks - Einführung in Graphdatenbanken
Neo4j GraphTalks - Einführung in GraphdatenbankenNeo4j GraphTalks - Einführung in Graphdatenbanken
Neo4j GraphTalks - Einführung in Graphdatenbanken
 
Redis Use Patterns (DevconTLV June 2014)
Redis Use Patterns (DevconTLV June 2014)Redis Use Patterns (DevconTLV June 2014)
Redis Use Patterns (DevconTLV June 2014)
 
Neo4j the Anti Crime Database
Neo4j the Anti Crime DatabaseNeo4j the Anti Crime Database
Neo4j the Anti Crime Database
 

Similar to Riak Use Cases : Dissecting The Solutions To Hard Problems

soft-shake.ch - Data grids and Data Grids
soft-shake.ch - Data grids and Data Gridssoft-shake.ch - Data grids and Data Grids
soft-shake.ch - Data grids and Data Gridssoft-shake.ch
 
LSC@LDAPCon 2011
LSC@LDAPCon 2011LSC@LDAPCon 2011
LSC@LDAPCon 2011sbahloul
 
CloudFoundry and MongoDb, a marriage made in heaven
CloudFoundry and MongoDb, a marriage made in heavenCloudFoundry and MongoDb, a marriage made in heaven
CloudFoundry and MongoDb, a marriage made in heavenPatrick Chanezon
 
Spotify: Playing for millions, tuning for more
Spotify: Playing for millions, tuning for moreSpotify: Playing for millions, tuning for more
Spotify: Playing for millions, tuning for moreNick Barkas
 
Apache Hadoop 3.0 Community Update
Apache Hadoop 3.0 Community UpdateApache Hadoop 3.0 Community Update
Apache Hadoop 3.0 Community UpdateDataWorks Summit
 
Introduction to NoSQL with Couchbase
Introduction to NoSQL with CouchbaseIntroduction to NoSQL with Couchbase
Introduction to NoSQL with CouchbaseTugdual Grall
 
Hopsworks - Self-Service Spark/Flink/Kafka/Hadoop
Hopsworks - Self-Service Spark/Flink/Kafka/HadoopHopsworks - Self-Service Spark/Flink/Kafka/Hadoop
Hopsworks - Self-Service Spark/Flink/Kafka/HadoopJim Dowling
 
An Open Source Case Study
An Open Source Case StudyAn Open Source Case Study
An Open Source Case Studywebhostingguy
 
Netflix Open Source Meetup Season 4 Episode 2
Netflix Open Source Meetup Season 4 Episode 2Netflix Open Source Meetup Season 4 Episode 2
Netflix Open Source Meetup Season 4 Episode 2aspyker
 
Data Management and Streaming Strategies in Drakensang Online
Data Management and Streaming Strategies in Drakensang OnlineData Management and Streaming Strategies in Drakensang Online
Data Management and Streaming Strategies in Drakensang OnlineAndre Weissflog
 
New Oracle Infrastructure2
New Oracle Infrastructure2New Oracle Infrastructure2
New Oracle Infrastructure2markleeuw
 
Interop 2011 - Scaling Platform As A Service
Interop 2011 - Scaling Platform As A ServiceInterop 2011 - Scaling Platform As A Service
Interop 2011 - Scaling Platform As A ServicePatrick Chanezon
 
Deliver Big Data, Database and AI/ML as-a-Service anywhere
Deliver Big Data, Database and AI/ML as-a-Service anywhereDeliver Big Data, Database and AI/ML as-a-Service anywhere
Deliver Big Data, Database and AI/ML as-a-Service anywhereRavikumar Alluboyina
 
Predictable Big Data Performance in Real-time
Predictable Big Data Performance in Real-timePredictable Big Data Performance in Real-time
Predictable Big Data Performance in Real-timeAerospike, Inc.
 
Cncf storage-final-filip
Cncf storage-final-filipCncf storage-final-filip
Cncf storage-final-filipJuraj Hantak
 
Netflix Open Source Meetup Season 3 Episode 2
Netflix Open Source Meetup Season 3 Episode 2Netflix Open Source Meetup Season 3 Episode 2
Netflix Open Source Meetup Season 3 Episode 2aspyker
 

Similar to Riak Use Cases : Dissecting The Solutions To Hard Problems (20)

soft-shake.ch - Data grids and Data Grids
soft-shake.ch - Data grids and Data Gridssoft-shake.ch - Data grids and Data Grids
soft-shake.ch - Data grids and Data Grids
 
LSC@LDAPCon 2011
LSC@LDAPCon 2011LSC@LDAPCon 2011
LSC@LDAPCon 2011
 
CloudFoundry and MongoDb, a marriage made in heaven
CloudFoundry and MongoDb, a marriage made in heavenCloudFoundry and MongoDb, a marriage made in heaven
CloudFoundry and MongoDb, a marriage made in heaven
 
Spotify: Playing for millions, tuning for more
Spotify: Playing for millions, tuning for moreSpotify: Playing for millions, tuning for more
Spotify: Playing for millions, tuning for more
 
Apache Hadoop 3.0 Community Update
Apache Hadoop 3.0 Community UpdateApache Hadoop 3.0 Community Update
Apache Hadoop 3.0 Community Update
 
Introduction to NoSQL with Couchbase
Introduction to NoSQL with CouchbaseIntroduction to NoSQL with Couchbase
Introduction to NoSQL with Couchbase
 
The World is Flat
The World is FlatThe World is Flat
The World is Flat
 
Hopsworks - Self-Service Spark/Flink/Kafka/Hadoop
Hopsworks - Self-Service Spark/Flink/Kafka/HadoopHopsworks - Self-Service Spark/Flink/Kafka/Hadoop
Hopsworks - Self-Service Spark/Flink/Kafka/Hadoop
 
An Open Source Case Study
An Open Source Case StudyAn Open Source Case Study
An Open Source Case Study
 
Techmeeting-17feb2016
Techmeeting-17feb2016Techmeeting-17feb2016
Techmeeting-17feb2016
 
Netflix Open Source Meetup Season 4 Episode 2
Netflix Open Source Meetup Season 4 Episode 2Netflix Open Source Meetup Season 4 Episode 2
Netflix Open Source Meetup Season 4 Episode 2
 
Data Management and Streaming Strategies in Drakensang Online
Data Management and Streaming Strategies in Drakensang OnlineData Management and Streaming Strategies in Drakensang Online
Data Management and Streaming Strategies in Drakensang Online
 
New Oracle Infrastructure2
New Oracle Infrastructure2New Oracle Infrastructure2
New Oracle Infrastructure2
 
How swift is your Swift - SD.pptx
How swift is your Swift - SD.pptxHow swift is your Swift - SD.pptx
How swift is your Swift - SD.pptx
 
Interop 2011 - Scaling Platform As A Service
Interop 2011 - Scaling Platform As A ServiceInterop 2011 - Scaling Platform As A Service
Interop 2011 - Scaling Platform As A Service
 
Deliver Big Data, Database and AI/ML as-a-Service anywhere
Deliver Big Data, Database and AI/ML as-a-Service anywhereDeliver Big Data, Database and AI/ML as-a-Service anywhere
Deliver Big Data, Database and AI/ML as-a-Service anywhere
 
Linux1
Linux1Linux1
Linux1
 
Predictable Big Data Performance in Real-time
Predictable Big Data Performance in Real-timePredictable Big Data Performance in Real-time
Predictable Big Data Performance in Real-time
 
Cncf storage-final-filip
Cncf storage-final-filipCncf storage-final-filip
Cncf storage-final-filip
 
Netflix Open Source Meetup Season 3 Episode 2
Netflix Open Source Meetup Season 3 Episode 2Netflix Open Source Meetup Season 3 Episode 2
Netflix Open Source Meetup Season 3 Episode 2
 

Recently uploaded

Benefits Of Flutter Compared To Other Frameworks
Benefits Of Flutter Compared To Other FrameworksBenefits Of Flutter Compared To Other Frameworks
Benefits Of Flutter Compared To Other FrameworksSoftradix Technologies
 
Key Features Of Token Development (1).pptx
Key  Features Of Token  Development (1).pptxKey  Features Of Token  Development (1).pptx
Key Features Of Token Development (1).pptxLBM Solutions
 
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 3652toLead Limited
 
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machineInstall Stable Diffusion in windows machine
Install Stable Diffusion in windows machinePadma Pradeep
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationSafe Software
 
Human Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsHuman Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsMark Billinghurst
 
Injustice - Developers Among Us (SciFiDevCon 2024)
Injustice - Developers Among Us (SciFiDevCon 2024)Injustice - Developers Among Us (SciFiDevCon 2024)
Injustice - Developers Among Us (SciFiDevCon 2024)Allon Mureinik
 
Unblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen FramesUnblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen FramesSinan KOZAK
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdfhans926745
 
Breaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path MountBreaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path MountPuma Security, LLC
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationMichael W. Hawkins
 
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...shyamraj55
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking MenDelhi Call girls
 
AI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsAI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsMemoori
 
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge
 
Pigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions
 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreternaman860154
 
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024BookNet Canada
 
Salesforce Community Group Quito, Salesforce 101
Salesforce Community Group Quito, Salesforce 101Salesforce Community Group Quito, Salesforce 101
Salesforce Community Group Quito, Salesforce 101Paola De la Torre
 

Recently uploaded (20)

Benefits Of Flutter Compared To Other Frameworks
Benefits Of Flutter Compared To Other FrameworksBenefits Of Flutter Compared To Other Frameworks
Benefits Of Flutter Compared To Other Frameworks
 
Key Features Of Token Development (1).pptx
Key  Features Of Token  Development (1).pptxKey  Features Of Token  Development (1).pptx
Key Features Of Token Development (1).pptx
 
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
 
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machineInstall Stable Diffusion in windows machine
Install Stable Diffusion in windows machine
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
 
Human Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsHuman Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR Systems
 
Injustice - Developers Among Us (SciFiDevCon 2024)
Injustice - Developers Among Us (SciFiDevCon 2024)Injustice - Developers Among Us (SciFiDevCon 2024)
Injustice - Developers Among Us (SciFiDevCon 2024)
 
Unblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen FramesUnblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen Frames
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
 
Breaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path MountBreaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path Mount
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
 
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men
 
AI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsAI as an Interface for Commercial Buildings
AI as an Interface for Commercial Buildings
 
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
 
Pigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food Manufacturing
 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreter
 
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
 
Salesforce Community Group Quito, Salesforce 101
Salesforce Community Group Quito, Salesforce 101Salesforce Community Group Quito, Salesforce 101
Salesforce Community Group Quito, Salesforce 101
 

Riak Use Cases : Dissecting The Solutions To Hard Problems

  • 1. Riak Use Cases: Dissecting the Solutions to Hard Problems Andy Gross <@argv0> Principal Architect, Basho Technologies QCon SF 2011 Wednesday, November 16, 11
  • 2. Riak Dynamo-inspired key value database with full text search, map/reduce, secondary indices, link traversal, commit hooks, HTTP and binary interfaces Written in Erlang (and C/C++) Open Source, Apache 2 licensed Enterprise features (multi-datacenter replication) and support available from Basho Wednesday, November 16, 11
  • 3. Choosing a NoSQL Database At small scale, everything works. NoSQL DBs trade off traditional features to better support new and emerging use cases Knowledge of the underlying system is essential NoSQL marketing is... “confusing” Wednesday, November 16, 11
  • 4. Tradeoffs If you’re evaluating Mongo vs. Riak, or Couch vs. Cassandra, you don’t understand your problem By choosing Riak, you’ve already made tradeoffs: Sacrificing consistency for availability in failure scenarios A rich data/query model for a simple, scalable one Wednesday, November 16, 11
  • 5. Distributed Systems: Desirable Properties Highly Available Fault Tolerant Low Latency Ops-Friendly Scalable Predictable Wednesday, November 16, 11
  • 6. Medical Records Store Danish Health Authorities Implemented by Trifork A/S Won the Digitization Prize as one of the “best government IT projects in Denmark” Stores medical prescription history for all Danish Citizens, replicated in 2 data centers. Accessed from pharmacies, hospital, mobile devices Replicated in multiple data centers Wednesday, November 16, 11
  • 7. User/Metadata Store Comcast User profile storage for xfinityTV mobile application Storage of metadata on content providers, and content licensing info Strict latency requirements Wednesday, November 16, 11
  • 8. Notification Service Yammer Wednesday, November 16, 11
  • 9. Session Store Mochi Media First Basho Customer (late 2009) Every hit to a Mochi web property = 1 read, maybe one write to Riak Unavailability, high latency = lost ad revenue Wednesday, November 16, 11
  • 10. Document Store Github Pages / Git.io Riak as a web server for Github Pages (in staging) Webmachine is an awesome HTTP server! Git.io URL shortener Wednesday, November 16, 11
  • 11. Distributed Systems: Desirable Properties High Availability Fault Tolerance Low Latency Ops-Friendliness Horizontal Scalability Predictability Wednesday, November 16, 11
  • 12. High Availability Failure to accept a read/write results in: lost revenue lost users Availability and latency are intertwined Wednesday, November 16, 11
  • 13. Low Latency Sometimes late answer is useless or wrong Users perceive slow sites as unavailable SLA violations SOA approaches magnify SLA failures Wednesday, November 16, 11
  • 14. Fault Tolerance Everything fails Especially in the cloud When a host/disk/network fails, what is the impact on Availability Latency Operations staff Wednesday, November 16, 11
  • 15. Predictability “It’s a piece of plumbing; it has never been a root cause of any of our problems.” Coda Hale, Yammer Wednesday, November 16, 11
  • 17. Operational Costs Sound familiar? “we chose a bad shard key...” “the failover script did not run as expected...” “the root cause was traced to a configuration error...” Staying up all night fighting your database does not make you a hero. Wednesday, November 16, 11
  • 18. High Availability: Erlang Ericcson AXD-301: 99.9999999% uptime (31ms/year) Shared-nothing, immutable, message-passing, functional, concurrent Distributed systems primitives in core language OTP (Open Telecom Platform) Wednesday, November 16, 11
  • 19. High Availability: Riak Core Dynamo abstracted: distributed systems toolkit Exhaustively tested In production use at AOL, Yahoo, others Insulates local storage and client API code from the hard problems Wednesday, November 16, 11
  • 20. Low Latency: Bitcask Low Latency: All reads = hash lookup + 1 seek After the append completes, an in-memory structure called a ”keydir” is updated. A keydir is simply a hash table that maps every key in a Bitcask to a fixed-size structure giving the file, offset, and size of the most recently written entry for that key. When a write occurs, the keydir is atomically updated with the location of the newest data. The old data is still present on disk, but any new reads will use the latest version available in the keydir. As we’ll see later, the Tradeoff: Index must fit in memory merge process will eventually remove the old value. Reading a value is simple, and doesn’t ever require more than a single disk seek. We look up the key in our keydir, and from there we read the data using the file id, position, and size that are returned from that lookup. In many cases, the operating system’s filesystem read-ahead cache makes this a much faster operation than would be otherwise expected. Wednesday, November 16, 11
  • 22. Low Latency: Erlang VM Erlang VM was designed for soft-realtime apps Preemptively scheduled lightweight threads GC is per-thread, not stop-the-world Sophisticated scheduler + message passing = effective use of multicore machines. Wednesday, November 16, 11