SlideShare a Scribd company logo
The NoSQL Ecosystem



        Adam Marcus
         MIT CSAIL
marcua@csail.mit.edu / @marcua
About Me
     ●
         Social Computing + Database Systems
     ●
         Easily Distracted: Wrote The NoSQL Ecosystem in
         The Architecture of Open Source Applications   1




1
    http://www.aosabook.org/en/nosql.html
History, Compressed




Late 1990s                         Today
History, Compressed




 Buy RAM

Late 1990s                         Today
History, Compressed




 Buy RAM Wrap RDBMSs

Late 1990s                         Today
History, Compressed




 Buy RAM Wrap RDBMSs   NoSQL

Late 1990s                         Today
History, Compressed


                          Not Only SQL




 Buy RAM Wrap RDBMSs   NoSQL

Late 1990s                          Today
History, Compressed




 Buy RAM Wrap RDBMSs   NoSQL   Commercialize

Late 1990s                            Today
History, Compressed




 Buy RAM Wrap RDBMSs   NoSQL   Commercialize

Late 1990s                            Today
The List, So You Don't Yell at Me

                HBase                  CouchDB
                               Neo4j
     Cassandra          Riak            InfoGrid
BerkeleyDB      Voldemort      HyperTable
       Redis
                         MongoDB        Sones
AllegroGraph HyperGraphDB
                                 DEX    FlockDB
             VertexDB
                                    Oracle NoSQL
Tokyo Cabinet       MemcacheDB
The List, So You Don't Yell at Me
      Marcus' Law of Databases:
           HBase             CouchDB
                             Neo4j
     Cassandra        Riak           InfoGrid
       The number of persistence
                      HyperTable
BerkeleyDB doubles every 1.5 years
            Voldemort
    options
       Redis
                       MongoDB       Sones
AllegroGraph HyperGraphDB
                               DEX   FlockDB
           VertexDB
                                 Oracle NoSQL
Tokyo Cabinet    MemcacheDB
The List, So You Don't Yell at Me
      Marcus' Law of Databases:
           HBase             CouchDB
                           Neo4j
     Cassandra      Riak           InfoGrid
       The number of persistence
                      HyperTable
BerkeleyDB doubles every 1.5 years
            Voldemort
    options
        Redis
                     MongoDB       Sones
                  Corollary:
AllegroGraph HyperGraphDB
          I'm sorry I missedDEX FlockDB
                             your
             persistence optionOracle NoSQL
            VertexDB
Tokyo Cabinet    MemcacheDB
interesting properties
  real-world usage
     takeaways
      questions
interesting properties
  real-world usage
     takeaways
      questions
Conventional Wisdom on NoSQL
●
    Key-based data model
●
    Sloppy schema
●
    Single-key transactions
●
    In-app joins
●
    Eventually consistent
Exceptions are More Interesting
●
    Data Model
●
    Query Model
●
    Transactions
●
    Consistency
Data Model
●
    Usually key-based...
    ●
        Column-families
    ●
        Documents
    ●
        Data structures
Data Model
●
    Usually key-based...
    ●
        Column-families
    ●
        Documents
    ●
        Data structures
●
    ...but not always
    ●
        Graph stores
Query Model
●
    Redis: data structure-specific operations
●
    CouchDB, Riak: MapReduce
●
    Cassandra, MongoDB: SQL-like languages, no
    joins or transactions
Query Model
●
    Redis: data structure-specific operations
●
    CouchDB, Riak: MapReduce
●
    Cassandra, MongoDB: SQL-like languages, no
    joins or transactions
●
    Third-party
    ●
        High-level: PigLatin, HiveQL
    ●
        Library: Cascading, Crunch
    ●
        Streaming: Flume, Kafka, S4, Scribe
Transactions
●
    Full ACID for single key
●
    Redis: multi-key single-node transactions
Consistency
●   Strong: Appears that all replicas see all writes
●   Eventual: Replicas may have different, divergent versions
Consistency
●   Strong: Appears that all replicas see all writes
●   Eventual: Replicas may have different, divergent versions

●   Dynamo: strong or eventual (quorum size)
Consistency
  ●   Strong: Appears that all replicas see all writes
  ●   Eventual: Replicas may have different, divergent versions

  ●   Dynamo: strong or eventual (quorum size)


  ●   PNUTs: Timeline consistency
  ●   ...many consistency models!




See: http://www.allthingsdistributed.com/2007/12/eventually_consistent.html
interesting properties
  real-world usage
     takeaways
      questions
NoSQL Use-cases
●
    Cassandra
●
    HBase
●
    MongoDB
Cassandra
●
    BigTable data model: key   column family
●
    Dynamo sharding model: consistent hashing
●
    Eventual or strong consistency
Cassandra at Netflix
  ●
      Transitioned from Oracle
  ●
      Store customer profiles, customer:movie watch
      log, and detailed usage logging
         Note: no multi-record locking (e.g., bank transfer)
      In-datacenter: 3 replicas, per-app consistency
         read/write quorum = 1, ~1ms latency
         read/write quorum = 2, ~3ms latency


“Replicating Datacenter Oracle with Global Apache Cassandra on AWS” by Adrian Cockcroft
http://www.slideshare.net/adrianco/migrating-netflix-from-oracle-to-global-cassandra
Cassandra at Netflix
  ●
      Transitioned from Oracle
  ●
      Store customer profiles, customer:movie watch
      log, and detailed usage logging
      ●
          Note: no multi-record locking (e.g., bank transfer)
      In-datacenter: 3 replicas, per-app consistency
          read/write quorum = 1, ~1ms latency
          read/write quorum = 2, ~3ms latency


“Replicating Datacenter Oracle with Global Apache Cassandra on AWS” by Adrian Cockcroft
http://www.slideshare.net/adrianco/migrating-netflix-from-oracle-to-global-cassandra
Cassandra at Netflix
  ●
      Transitioned from Oracle
  ●
      Store customer profiles, customer:movie watch
      log, and detailed usage logging
      ●
          Note: no multi-record locking (e.g., bank transfer)
  ●
      In-datacenter: 3 replicas, per-app consistency




“Replicating Datacenter Oracle with Global Apache Cassandra on AWS” by Adrian Cockcroft
http://www.slideshare.net/adrianco/migrating-netflix-from-oracle-to-global-cassandra
Cassandra at Netflix (cont'd)
●
    Benefit: async inter-datacenter replication
●
    Benefit: no downtime for schema changes
●
    Benefit: hooks for live backups
HBase
●
    Data model: key       column family
●
    Sharding model: range partitioning
●
    Strong consistency


    Applications
      Logging events/crawls, storing analytics
      Twitter: replicate data from MySQL, Hadoop analytics
      Facebook Messages
HBase
●
    Data model: key         column family
●
    Sharding model: range partitioning
●
    Strong consistency


●
    Applications
    ●
        Logging events/crawls, storing analytics
    ●
        Twitter: replicate data from MySQL, Hadoop analytics
    ●
        Facebook Messages
HBase for Facebook Messages
●
    Cassandra/Dynamo eventual consistency was
    difficult to program against


    Benefit: simple consistency model
    Benefit: flexible data model
    Benefit: simple sharding, load balancing,
    replication
HBase for Facebook Messages
●
    Cassandra/Dynamo eventual consistency was
    difficult to program against


●
    Benefit: simple consistency model
●
    Benefit: flexible data model
●
    Benefit: simple sharding, load balancing,
    replication
MongoDB
●
    Document-based data model
●
    Range-based partitioning
●
    Consistency depends on how you use it
MongoDB: Two use-cases
●
    Archiving at Craigslist
    ●
        2.2B historical posts, semi-structured
    ●
        Relatively large blobs: avg 2KB, max > 4 MB
MongoDB: Two use-cases
●
    Archiving at Craigslist
    ●
        2.2B historical posts, semi-structured
    ●
        Relatively large blobs: avg 2KB, max > 4 MB
●
    Checkins at Foursquare
    ●
        Geospatial indexing
    ●
        Small location-based updates, sharded on user
interesting properties
  real-world usage
     takeaways
      questions
Takeaways
●
    Developer accessibility
●
    Ecosystem of reuse
●
    Soft spot
Developer Accessibility
Developer Accessibility
    devops




@DEVOPS_BORAT
Developer Accessibility
    devops          self-taught dev




@DEVOPS_BORAT
A Different Five-Minute Rule



What does a first-time user of your system
experience in their first five minutes?
(idea credit: Justin Sheehy, Basho)
Redis
http://simonwillison.net/2009/Oct/22/redis/
PostgreSQL
http://www.postgresql.org/docs/9.1/interactive/sql-createtable.html
Cassandra
http://www.datastax.com/docs/0.7/getting_started/using_cli
Accessibility is Forever
Beyond five minutes:
●
    changing schemas
●
    scaling up
●
    modifying topology
Accessibility is Forever
Beyond five minutes:
●
    changing schemas
●
    scaling up
●
    modifying topology


       An accessible data store will ease
        first-time users in, and support
       experienced users as they expand
Ecosystem of Reuse
Spectrum of Reusability


Monolithic System         System Kernel
Spectrum of Reusability


Monolithic System            System Kernel


MongoDB
 Redis


 Use as
provided
Spectrum of Reusability


Monolithic System            System Kernel


MongoDB      Cassandra
 Redis       Voldemort


 Use as     Pick between
provided    components
Spectrum of Reusability


Monolithic System                System Kernel


MongoDB      Cassandra   ZooKeeper
 Redis       Voldemort    LevelDB


 Use as     Pick between Reusable
provided    components components
Spectrum of Reusability


Monolithic System                System Kernel


MongoDB      Cassandra   ZooKeeper
 Redis       Voldemort    LevelDB     riak_core


 Use as     Pick between Reusable     Build your
provided    components components    own system
And finally, a soft spot
polyglot persistence
            =
right tool for right task
Polyglot persistence: Where's the data?




“HBase at Mendeley,” Dan Harvey
http://www.slideshare.net/danharvey/hbase-at-mendeley
Polyglot persistence: Where's the data?




“HBase at Mendeley,” Dan Harvey
http://www.slideshare.net/danharvey/hbase-at-mendeley
Polyglot persistence: Where's the data?




Aid developers in reasoning about data consistency
          across multiple storage engines
“HBase at Mendeley,” Dan Harvey
http://www.slideshare.net/danharvey/hbase-at-mendeley
interesting properties
  real-world usage
     takeaways
      questions
●
    Systems: Polyglot persistence + data consistency?
●
    Systems: Polyglot persistence + data consistency?
●
    Operations: Availability/consistency/latency tradeoffs in
    datacenters?
●
    Systems: Polyglot persistence + data consistency?
●
    Operations: Availability/consistency/latency tradeoffs in
    datacenters?
●
    Accessibility: RDBMS five-minute usability?
●
    Systems: Polyglot persistence + data consistency?
●
    Operations: Availability/consistency/latency tradeoffs in
    datacenters?
●
    Accessibility: RDBMS five-minute usability?
●
    Accessibility: Scale-up wizard for storage systems?
●
    Systems: Polyglot persistence + data consistency?
●
    Operations: Availability/consistency/latency tradeoffs in
    datacenters?
●
    Accessibility: RDBMS five-minute usability?
●
    Accessibility: Scale-up wizard for storage systems?
●
    Comparisons: Design or implementation?
●
    Systems: Polyglot persistence + data consistency?
●
    Operations: Availability/consistency/latency tradeoffs in
    datacenters?
●
    Accessibility: RDBMS five-minute usability?
●
    Accessibility: Scale-up wizard for storage systems?
●
    Comparisons: Design or implementation?
●
    Future: Next-generation NoSQL stores?
Thank You!
●
    Systems: Polyglot persistence + data consistency?
●
    Operations: Availability/consistency/latency tradeoffs in
    datacenters?
●
    Accessibility: RDBMS five-minute usability?
●
    Accessibility: Scale-up wizard for storage systems?
●
    Comparisons: Design or implementation?
●
    Future: Next-generation NoSQL stores?

                    Adam Marcus
            marcua@csail.mit.edu / @marcua
Photo Credits
http://www.flickr.com/photos/dhannah/457765
7955/
http://www.flickr.com/photos/27316226@N02/3
000888100/sizes/m/in/photostream/
http://www.texample.net/tikz/examples/valenti
ne-heart/
http://voltdb.com/sites/default/files/VCE_button
.png

More Related Content

What's hot

Couchbase
CouchbaseCouchbase
Couchbase
Arpit Aggarwal
 
NoSQL databases - An introduction
NoSQL databases - An introductionNoSQL databases - An introduction
NoSQL databases - An introduction
Pooyan Mehrparvar
 
The Hive Think Tank: Rocking the Database World with RocksDB
The Hive Think Tank: Rocking the Database World with RocksDBThe Hive Think Tank: Rocking the Database World with RocksDB
The Hive Think Tank: Rocking the Database World with RocksDB
The Hive
 
An Intro to NoSQL Databases
An Intro to NoSQL DatabasesAn Intro to NoSQL Databases
An Intro to NoSQL Databases
Rajith Pemabandu
 
Nosql databases for the .net developer
Nosql databases for the .net developerNosql databases for the .net developer
Nosql databases for the .net developerJesus Rodriguez
 
No sql
No sqlNo sql
Introduction to Cassandra (June 2010)
Introduction to Cassandra (June 2010)Introduction to Cassandra (June 2010)
Introduction to Cassandra (June 2010)
gdusbabek
 
NoSQL
NoSQLNoSQL
NoSQL
Radu Potop
 
Comparative study of modern databases
Comparative study of modern databasesComparative study of modern databases
Comparative study of modern databases
Anirban Konar
 
Four NoSQL Databases You Should Know
Four NoSQL Databases You Should KnowFour NoSQL Databases You Should Know
Four NoSQL Databases You Should Know
Mahmoud Khaled
 
A Seminar on NoSQL Databases.
A Seminar on NoSQL Databases.A Seminar on NoSQL Databases.
A Seminar on NoSQL Databases.
Navdeep Charan
 
No SQL - A Simple Intro
No SQL - A Simple IntroNo SQL - A Simple Intro
No SQL - A Simple Intro
Karthi Keyan
 
NoSQL and MongoDB
NoSQL and MongoDBNoSQL and MongoDB
NoSQL and MongoDB
Rajesh Menon
 
First steps to Azure Cosmos DB: Getting Started with MongoDB and NoSQL
First steps to Azure Cosmos DB: Getting Started with MongoDB and NoSQLFirst steps to Azure Cosmos DB: Getting Started with MongoDB and NoSQL
First steps to Azure Cosmos DB: Getting Started with MongoDB and NoSQL
Hansamali Gamage
 
Big data stores
Big data  storesBig data  stores
Big data stores
Kumaran Ramanujam
 
Java BigData Full Stack Development (version 2.0)
Java BigData Full Stack Development (version 2.0)Java BigData Full Stack Development (version 2.0)
Java BigData Full Stack Development (version 2.0)
Alexey Zinoviev
 
MongoDB
MongoDBMongoDB

What's hot (20)

Couchbase
CouchbaseCouchbase
Couchbase
 
NoSQL databases - An introduction
NoSQL databases - An introductionNoSQL databases - An introduction
NoSQL databases - An introduction
 
The Hive Think Tank: Rocking the Database World with RocksDB
The Hive Think Tank: Rocking the Database World with RocksDBThe Hive Think Tank: Rocking the Database World with RocksDB
The Hive Think Tank: Rocking the Database World with RocksDB
 
An Intro to NoSQL Databases
An Intro to NoSQL DatabasesAn Intro to NoSQL Databases
An Intro to NoSQL Databases
 
Nosql databases for the .net developer
Nosql databases for the .net developerNosql databases for the .net developer
Nosql databases for the .net developer
 
NoSql
NoSqlNoSql
NoSql
 
No sql
No sqlNo sql
No sql
 
NoSQL Seminer
NoSQL SeminerNoSQL Seminer
NoSQL Seminer
 
Introduction to Cassandra (June 2010)
Introduction to Cassandra (June 2010)Introduction to Cassandra (June 2010)
Introduction to Cassandra (June 2010)
 
NoSQL
NoSQLNoSQL
NoSQL
 
Comparative study of modern databases
Comparative study of modern databasesComparative study of modern databases
Comparative study of modern databases
 
NoSQL
NoSQLNoSQL
NoSQL
 
Four NoSQL Databases You Should Know
Four NoSQL Databases You Should KnowFour NoSQL Databases You Should Know
Four NoSQL Databases You Should Know
 
A Seminar on NoSQL Databases.
A Seminar on NoSQL Databases.A Seminar on NoSQL Databases.
A Seminar on NoSQL Databases.
 
No SQL - A Simple Intro
No SQL - A Simple IntroNo SQL - A Simple Intro
No SQL - A Simple Intro
 
NoSQL and MongoDB
NoSQL and MongoDBNoSQL and MongoDB
NoSQL and MongoDB
 
First steps to Azure Cosmos DB: Getting Started with MongoDB and NoSQL
First steps to Azure Cosmos DB: Getting Started with MongoDB and NoSQLFirst steps to Azure Cosmos DB: Getting Started with MongoDB and NoSQL
First steps to Azure Cosmos DB: Getting Started with MongoDB and NoSQL
 
Big data stores
Big data  storesBig data  stores
Big data stores
 
Java BigData Full Stack Development (version 2.0)
Java BigData Full Stack Development (version 2.0)Java BigData Full Stack Development (version 2.0)
Java BigData Full Stack Development (version 2.0)
 
MongoDB
MongoDBMongoDB
MongoDB
 

Similar to HPTS 2011: The NoSQL Ecosystem

Introduction to NoSQL | Big Data Hadoop Spark Tutorial | CloudxLab
Introduction to NoSQL | Big Data Hadoop Spark Tutorial | CloudxLabIntroduction to NoSQL | Big Data Hadoop Spark Tutorial | CloudxLab
Introduction to NoSQL | Big Data Hadoop Spark Tutorial | CloudxLab
CloudxLab
 
Drop acid
Drop acidDrop acid
Drop acid
Mike Feltman
 
JPoint'15 Mom, I so wish Hibernate for my NoSQL database...
JPoint'15 Mom, I so wish Hibernate for my NoSQL database...JPoint'15 Mom, I so wish Hibernate for my NoSQL database...
JPoint'15 Mom, I so wish Hibernate for my NoSQL database...
Alexey Zinoviev
 
NoSQL Data Stores in Research and Practice - ICDE 2016 Tutorial - Extended Ve...
NoSQL Data Stores in Research and Practice - ICDE 2016 Tutorial - Extended Ve...NoSQL Data Stores in Research and Practice - ICDE 2016 Tutorial - Extended Ve...
NoSQL Data Stores in Research and Practice - ICDE 2016 Tutorial - Extended Ve...
Felix Gessert
 
No SQL Technologies
No SQL TechnologiesNo SQL Technologies
No SQL Technologies
Cris Holdorph
 
No Sql Introduction
No Sql IntroductionNo Sql Introduction
No Sql IntroductionDingding Ye
 
NoSQL Intro with cassandra
NoSQL Intro with cassandraNoSQL Intro with cassandra
NoSQL Intro with cassandra
Brian Enochson
 
Introduction to NoSql
Introduction to NoSqlIntroduction to NoSql
Introduction to NoSql
Omid Vahdaty
 
Using Spring with NoSQL databases (SpringOne China 2012)
Using Spring with NoSQL databases (SpringOne China 2012)Using Spring with NoSQL databases (SpringOne China 2012)
Using Spring with NoSQL databases (SpringOne China 2012)
Chris Richardson
 
Cassandra for mission critical data
Cassandra for mission critical dataCassandra for mission critical data
Cassandra for mission critical data
Oleksandr Semenov
 
On Rails with Apache Cassandra
On Rails with Apache CassandraOn Rails with Apache Cassandra
On Rails with Apache Cassandra
Stu Hood
 
SQL? NoSQL? NewSQL?!? What’s a Java developer to do? - JDC2012 Cairo, Egypt
SQL? NoSQL? NewSQL?!? What’s a Java developer to do? - JDC2012 Cairo, EgyptSQL? NoSQL? NewSQL?!? What’s a Java developer to do? - JDC2012 Cairo, Egypt
SQL? NoSQL? NewSQL?!? What’s a Java developer to do? - JDC2012 Cairo, Egypt
Chris Richardson
 
Mongo db transcript
Mongo db transcriptMongo db transcript
Mongo db transcript
foliba
 
Spring one2gx2010 spring-nonrelational_data
Spring one2gx2010 spring-nonrelational_dataSpring one2gx2010 spring-nonrelational_data
Spring one2gx2010 spring-nonrelational_data
Roger Xia
 
No sq lv1_0
No sq lv1_0No sq lv1_0
No sq lv1_0
Tuan Luong
 
Introduction to NoSQL
Introduction to NoSQLIntroduction to NoSQL
Introduction to NoSQL
balwinders
 

Similar to HPTS 2011: The NoSQL Ecosystem (20)

Introduction to NoSQL | Big Data Hadoop Spark Tutorial | CloudxLab
Introduction to NoSQL | Big Data Hadoop Spark Tutorial | CloudxLabIntroduction to NoSQL | Big Data Hadoop Spark Tutorial | CloudxLab
Introduction to NoSQL | Big Data Hadoop Spark Tutorial | CloudxLab
 
Drop acid
Drop acidDrop acid
Drop acid
 
JPoint'15 Mom, I so wish Hibernate for my NoSQL database...
JPoint'15 Mom, I so wish Hibernate for my NoSQL database...JPoint'15 Mom, I so wish Hibernate for my NoSQL database...
JPoint'15 Mom, I so wish Hibernate for my NoSQL database...
 
NoSQL
NoSQLNoSQL
NoSQL
 
NoSQL Data Stores in Research and Practice - ICDE 2016 Tutorial - Extended Ve...
NoSQL Data Stores in Research and Practice - ICDE 2016 Tutorial - Extended Ve...NoSQL Data Stores in Research and Practice - ICDE 2016 Tutorial - Extended Ve...
NoSQL Data Stores in Research and Practice - ICDE 2016 Tutorial - Extended Ve...
 
No SQL Technologies
No SQL TechnologiesNo SQL Technologies
No SQL Technologies
 
No Sql Introduction
No Sql IntroductionNo Sql Introduction
No Sql Introduction
 
NoSQL Intro with cassandra
NoSQL Intro with cassandraNoSQL Intro with cassandra
NoSQL Intro with cassandra
 
Nosql seminar
Nosql seminarNosql seminar
Nosql seminar
 
Introduction to NoSql
Introduction to NoSqlIntroduction to NoSql
Introduction to NoSql
 
NoSQL
NoSQLNoSQL
NoSQL
 
Using Spring with NoSQL databases (SpringOne China 2012)
Using Spring with NoSQL databases (SpringOne China 2012)Using Spring with NoSQL databases (SpringOne China 2012)
Using Spring with NoSQL databases (SpringOne China 2012)
 
Cassandra for mission critical data
Cassandra for mission critical dataCassandra for mission critical data
Cassandra for mission critical data
 
On Rails with Apache Cassandra
On Rails with Apache CassandraOn Rails with Apache Cassandra
On Rails with Apache Cassandra
 
SQL? NoSQL? NewSQL?!? What’s a Java developer to do? - JDC2012 Cairo, Egypt
SQL? NoSQL? NewSQL?!? What’s a Java developer to do? - JDC2012 Cairo, EgyptSQL? NoSQL? NewSQL?!? What’s a Java developer to do? - JDC2012 Cairo, Egypt
SQL? NoSQL? NewSQL?!? What’s a Java developer to do? - JDC2012 Cairo, Egypt
 
Mongo db transcript
Mongo db transcriptMongo db transcript
Mongo db transcript
 
Spring one2gx2010 spring-nonrelational_data
Spring one2gx2010 spring-nonrelational_dataSpring one2gx2010 spring-nonrelational_data
Spring one2gx2010 spring-nonrelational_data
 
No sq lv1_0
No sq lv1_0No sq lv1_0
No sq lv1_0
 
NoSql Databases
NoSql DatabasesNoSql Databases
NoSql Databases
 
Introduction to NoSQL
Introduction to NoSQLIntroduction to NoSQL
Introduction to NoSQL
 

Recently uploaded

FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdfFIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance
 
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
Product School
 
How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...
Product School
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Jeffrey Haguewood
 
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
Sri Ambati
 
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Thierry Lestable
 
Search and Society: Reimagining Information Access for Radical Futures
Search and Society: Reimagining Information Access for Radical FuturesSearch and Society: Reimagining Information Access for Radical Futures
Search and Society: Reimagining Information Access for Radical Futures
Bhaskar Mitra
 
Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*
Frank van Harmelen
 
When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...
Elena Simperl
 
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
DanBrown980551
 
The Future of Platform Engineering
The Future of Platform EngineeringThe Future of Platform Engineering
The Future of Platform Engineering
Jemma Hussein Allen
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
DianaGray10
 
"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi
Fwdays
 
Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........
Alison B. Lowndes
 
To Graph or Not to Graph Knowledge Graph Architectures and LLMs
To Graph or Not to Graph Knowledge Graph Architectures and LLMsTo Graph or Not to Graph Knowledge Graph Architectures and LLMs
To Graph or Not to Graph Knowledge Graph Architectures and LLMs
Paul Groth
 
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdfSmart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
91mobiles
 
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
James Anderson
 
UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4
DianaGray10
 
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Product School
 
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualitySoftware Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Inflectra
 

Recently uploaded (20)

FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdfFIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
 
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
 
How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
 
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
 
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
 
Search and Society: Reimagining Information Access for Radical Futures
Search and Society: Reimagining Information Access for Radical FuturesSearch and Society: Reimagining Information Access for Radical Futures
Search and Society: Reimagining Information Access for Radical Futures
 
Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*
 
When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...
 
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
 
The Future of Platform Engineering
The Future of Platform EngineeringThe Future of Platform Engineering
The Future of Platform Engineering
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
 
"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi
 
Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........
 
To Graph or Not to Graph Knowledge Graph Architectures and LLMs
To Graph or Not to Graph Knowledge Graph Architectures and LLMsTo Graph or Not to Graph Knowledge Graph Architectures and LLMs
To Graph or Not to Graph Knowledge Graph Architectures and LLMs
 
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdfSmart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
 
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
 
UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4
 
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
 
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualitySoftware Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
 

HPTS 2011: The NoSQL Ecosystem

  • 1. The NoSQL Ecosystem Adam Marcus MIT CSAIL marcua@csail.mit.edu / @marcua
  • 2. About Me ● Social Computing + Database Systems ● Easily Distracted: Wrote The NoSQL Ecosystem in The Architecture of Open Source Applications 1 1 http://www.aosabook.org/en/nosql.html
  • 4. History, Compressed Buy RAM Late 1990s Today
  • 5. History, Compressed Buy RAM Wrap RDBMSs Late 1990s Today
  • 6. History, Compressed Buy RAM Wrap RDBMSs NoSQL Late 1990s Today
  • 7. History, Compressed Not Only SQL Buy RAM Wrap RDBMSs NoSQL Late 1990s Today
  • 8. History, Compressed Buy RAM Wrap RDBMSs NoSQL Commercialize Late 1990s Today
  • 9. History, Compressed Buy RAM Wrap RDBMSs NoSQL Commercialize Late 1990s Today
  • 10. The List, So You Don't Yell at Me HBase CouchDB Neo4j Cassandra Riak InfoGrid BerkeleyDB Voldemort HyperTable Redis MongoDB Sones AllegroGraph HyperGraphDB DEX FlockDB VertexDB Oracle NoSQL Tokyo Cabinet MemcacheDB
  • 11. The List, So You Don't Yell at Me Marcus' Law of Databases: HBase CouchDB Neo4j Cassandra Riak InfoGrid The number of persistence HyperTable BerkeleyDB doubles every 1.5 years Voldemort options Redis MongoDB Sones AllegroGraph HyperGraphDB DEX FlockDB VertexDB Oracle NoSQL Tokyo Cabinet MemcacheDB
  • 12. The List, So You Don't Yell at Me Marcus' Law of Databases: HBase CouchDB Neo4j Cassandra Riak InfoGrid The number of persistence HyperTable BerkeleyDB doubles every 1.5 years Voldemort options Redis MongoDB Sones Corollary: AllegroGraph HyperGraphDB I'm sorry I missedDEX FlockDB your persistence optionOracle NoSQL VertexDB Tokyo Cabinet MemcacheDB
  • 13. interesting properties real-world usage takeaways questions
  • 14. interesting properties real-world usage takeaways questions
  • 15. Conventional Wisdom on NoSQL ● Key-based data model ● Sloppy schema ● Single-key transactions ● In-app joins ● Eventually consistent
  • 16. Exceptions are More Interesting ● Data Model ● Query Model ● Transactions ● Consistency
  • 17. Data Model ● Usually key-based... ● Column-families ● Documents ● Data structures
  • 18. Data Model ● Usually key-based... ● Column-families ● Documents ● Data structures ● ...but not always ● Graph stores
  • 19. Query Model ● Redis: data structure-specific operations ● CouchDB, Riak: MapReduce ● Cassandra, MongoDB: SQL-like languages, no joins or transactions
  • 20. Query Model ● Redis: data structure-specific operations ● CouchDB, Riak: MapReduce ● Cassandra, MongoDB: SQL-like languages, no joins or transactions ● Third-party ● High-level: PigLatin, HiveQL ● Library: Cascading, Crunch ● Streaming: Flume, Kafka, S4, Scribe
  • 21. Transactions ● Full ACID for single key ● Redis: multi-key single-node transactions
  • 22. Consistency ● Strong: Appears that all replicas see all writes ● Eventual: Replicas may have different, divergent versions
  • 23. Consistency ● Strong: Appears that all replicas see all writes ● Eventual: Replicas may have different, divergent versions ● Dynamo: strong or eventual (quorum size)
  • 24. Consistency ● Strong: Appears that all replicas see all writes ● Eventual: Replicas may have different, divergent versions ● Dynamo: strong or eventual (quorum size) ● PNUTs: Timeline consistency ● ...many consistency models! See: http://www.allthingsdistributed.com/2007/12/eventually_consistent.html
  • 25. interesting properties real-world usage takeaways questions
  • 26. NoSQL Use-cases ● Cassandra ● HBase ● MongoDB
  • 27. Cassandra ● BigTable data model: key column family ● Dynamo sharding model: consistent hashing ● Eventual or strong consistency
  • 28. Cassandra at Netflix ● Transitioned from Oracle ● Store customer profiles, customer:movie watch log, and detailed usage logging Note: no multi-record locking (e.g., bank transfer) In-datacenter: 3 replicas, per-app consistency read/write quorum = 1, ~1ms latency read/write quorum = 2, ~3ms latency “Replicating Datacenter Oracle with Global Apache Cassandra on AWS” by Adrian Cockcroft http://www.slideshare.net/adrianco/migrating-netflix-from-oracle-to-global-cassandra
  • 29. Cassandra at Netflix ● Transitioned from Oracle ● Store customer profiles, customer:movie watch log, and detailed usage logging ● Note: no multi-record locking (e.g., bank transfer) In-datacenter: 3 replicas, per-app consistency read/write quorum = 1, ~1ms latency read/write quorum = 2, ~3ms latency “Replicating Datacenter Oracle with Global Apache Cassandra on AWS” by Adrian Cockcroft http://www.slideshare.net/adrianco/migrating-netflix-from-oracle-to-global-cassandra
  • 30. Cassandra at Netflix ● Transitioned from Oracle ● Store customer profiles, customer:movie watch log, and detailed usage logging ● Note: no multi-record locking (e.g., bank transfer) ● In-datacenter: 3 replicas, per-app consistency “Replicating Datacenter Oracle with Global Apache Cassandra on AWS” by Adrian Cockcroft http://www.slideshare.net/adrianco/migrating-netflix-from-oracle-to-global-cassandra
  • 31. Cassandra at Netflix (cont'd) ● Benefit: async inter-datacenter replication ● Benefit: no downtime for schema changes ● Benefit: hooks for live backups
  • 32. HBase ● Data model: key column family ● Sharding model: range partitioning ● Strong consistency Applications Logging events/crawls, storing analytics Twitter: replicate data from MySQL, Hadoop analytics Facebook Messages
  • 33. HBase ● Data model: key column family ● Sharding model: range partitioning ● Strong consistency ● Applications ● Logging events/crawls, storing analytics ● Twitter: replicate data from MySQL, Hadoop analytics ● Facebook Messages
  • 34. HBase for Facebook Messages ● Cassandra/Dynamo eventual consistency was difficult to program against Benefit: simple consistency model Benefit: flexible data model Benefit: simple sharding, load balancing, replication
  • 35. HBase for Facebook Messages ● Cassandra/Dynamo eventual consistency was difficult to program against ● Benefit: simple consistency model ● Benefit: flexible data model ● Benefit: simple sharding, load balancing, replication
  • 36. MongoDB ● Document-based data model ● Range-based partitioning ● Consistency depends on how you use it
  • 37. MongoDB: Two use-cases ● Archiving at Craigslist ● 2.2B historical posts, semi-structured ● Relatively large blobs: avg 2KB, max > 4 MB
  • 38. MongoDB: Two use-cases ● Archiving at Craigslist ● 2.2B historical posts, semi-structured ● Relatively large blobs: avg 2KB, max > 4 MB ● Checkins at Foursquare ● Geospatial indexing ● Small location-based updates, sharded on user
  • 39. interesting properties real-world usage takeaways questions
  • 40. Takeaways ● Developer accessibility ● Ecosystem of reuse ● Soft spot
  • 42. Developer Accessibility devops @DEVOPS_BORAT
  • 43. Developer Accessibility devops self-taught dev @DEVOPS_BORAT
  • 44. A Different Five-Minute Rule What does a first-time user of your system experience in their first five minutes? (idea credit: Justin Sheehy, Basho)
  • 48. Accessibility is Forever Beyond five minutes: ● changing schemas ● scaling up ● modifying topology
  • 49. Accessibility is Forever Beyond five minutes: ● changing schemas ● scaling up ● modifying topology An accessible data store will ease first-time users in, and support experienced users as they expand
  • 51. Spectrum of Reusability Monolithic System System Kernel
  • 52. Spectrum of Reusability Monolithic System System Kernel MongoDB Redis Use as provided
  • 53. Spectrum of Reusability Monolithic System System Kernel MongoDB Cassandra Redis Voldemort Use as Pick between provided components
  • 54. Spectrum of Reusability Monolithic System System Kernel MongoDB Cassandra ZooKeeper Redis Voldemort LevelDB Use as Pick between Reusable provided components components
  • 55. Spectrum of Reusability Monolithic System System Kernel MongoDB Cassandra ZooKeeper Redis Voldemort LevelDB riak_core Use as Pick between Reusable Build your provided components components own system
  • 56. And finally, a soft spot
  • 57. polyglot persistence = right tool for right task
  • 58. Polyglot persistence: Where's the data? “HBase at Mendeley,” Dan Harvey http://www.slideshare.net/danharvey/hbase-at-mendeley
  • 59. Polyglot persistence: Where's the data? “HBase at Mendeley,” Dan Harvey http://www.slideshare.net/danharvey/hbase-at-mendeley
  • 60. Polyglot persistence: Where's the data? Aid developers in reasoning about data consistency across multiple storage engines “HBase at Mendeley,” Dan Harvey http://www.slideshare.net/danharvey/hbase-at-mendeley
  • 61. interesting properties real-world usage takeaways questions
  • 62. Systems: Polyglot persistence + data consistency?
  • 63. Systems: Polyglot persistence + data consistency? ● Operations: Availability/consistency/latency tradeoffs in datacenters?
  • 64. Systems: Polyglot persistence + data consistency? ● Operations: Availability/consistency/latency tradeoffs in datacenters? ● Accessibility: RDBMS five-minute usability?
  • 65. Systems: Polyglot persistence + data consistency? ● Operations: Availability/consistency/latency tradeoffs in datacenters? ● Accessibility: RDBMS five-minute usability? ● Accessibility: Scale-up wizard for storage systems?
  • 66. Systems: Polyglot persistence + data consistency? ● Operations: Availability/consistency/latency tradeoffs in datacenters? ● Accessibility: RDBMS five-minute usability? ● Accessibility: Scale-up wizard for storage systems? ● Comparisons: Design or implementation?
  • 67. Systems: Polyglot persistence + data consistency? ● Operations: Availability/consistency/latency tradeoffs in datacenters? ● Accessibility: RDBMS five-minute usability? ● Accessibility: Scale-up wizard for storage systems? ● Comparisons: Design or implementation? ● Future: Next-generation NoSQL stores?
  • 68. Thank You! ● Systems: Polyglot persistence + data consistency? ● Operations: Availability/consistency/latency tradeoffs in datacenters? ● Accessibility: RDBMS five-minute usability? ● Accessibility: Scale-up wizard for storage systems? ● Comparisons: Design or implementation? ● Future: Next-generation NoSQL stores? Adam Marcus marcua@csail.mit.edu / @marcua