SlideShare a Scribd company logo
A Survey of Advanced Non-relational Database
    Systems: Approaches and Applications



             Speaker: LIN Qian
  http://www.comp.nus.edu.sg/~linqian
Outline
• Introduction
• Non-relational database system
  –   Requirement
  –   Concepts
  –   Approaches
  –   Optimization
  –   Examples
• Comparison between RDBMS and non-relational
  database system



                                                1
Problem
• The Web introduces a new scale for applications, in
  terms of:
   –   Concurrent users (millions of reqs/second)
   –   Data (peta-bytes generated daily)
   –   Processing (all this data needs processing)
   –   Exponential growth (surging unpredictable demands)
• Shortage of existing RDBMS
   – Oracle, MS SQL, Sybase, MySQL, PostgreSQL, …
   – Trouble when dealing with very large traffic
   – Even with their high-end clustering solutions


                                                            2
Problem
• Why?
  – Applications using normalized database schema require the
    use of join, which doesn't perform well under lots of data
    and/or nodes
  – Existing RDBMS clustering solutions require scale-up, which
    is limited and not really scalable when dealing with
    exponential growth (e.g., 1000+ nodes)
  – Machines have upper limits on capacity




                                                                  3
Problem
• Why not just use sharding?
    – Very complex and application-specific
         •   Increased complexity of SQL
         •   Single point of failure
         •   Failover servers more complex
         •   Backups more complex
         •   Operational complexity added
    – Very problematic when adding/removing nodes
    – Basically, you end up denormalizing everything and loosing
      all benefits of relational databases

Sharding: Split one or more tables by row across potentially multiple instances of the
schema and database servers.

                                                                                         4
Who faced this problem?
• Web applications dealing with high traffic and massive
  data
   – Web service providers
      • Google, Yahoo!, Amazon, Facebook, Twitter, LinkedIn, …
   – Scientific data analysis
      • Weather, Ocean, tide, geothermy, …
   – Complex information processing
      • Financial, stock, telecommunication, …




                                                                 5
Solution
• A new kind of DBMS, capable of handling web scale
    – Possibly sacrificing some level of feature
• CAP theorem*: You can only optimize 2 out of these 3
    – Consistency - the system is in a consistent state after an operation
         • All nodes see the same data at the same time
         • Strong consistency (ACID) vs. eventual consistency (BASE)
    – Availability - the system is “always on”, no downtime
         • Node failure tolerance: All clients can find some available replica.
         • software/hardware upgrade tolerance
    – Partition tolerance
         • The system continues to operate (read/write) despite arbitrary message
           loss or failure of part of the system


* Eric A. Brewer, Towards Robust Distributed Systems, Proceedings of the 19th annual
ACM symposium on Principles of Distributed Computing (PODC), 2000                      6
Non-relational database systems
• Various solutions & products
   –   BigTable, LevelDB (developed at Google)
   –   Hbase (developed at Yahoo!)
   –   Dynamo (developed at Amazon)
   –   Cassandra (developed at FaceBook)
   –   Voldemort (developed at LinkedIn)
   –   Riak, Redis, CouchDB, MongoDB, Berkeley DB, …
• Researches
   – NoDB, Walnut, LogBase, Albatross, Citrusleaf, HadoopDB
   – PIQL, RAMCloud


                                                              7
Benefits
• Massively scalable
• Extremely fast
• Highly available, decentralized and fault tolerant
   – no single-point-of-failure
• Transparent sharding (consistent hashing)
• Elasticity
• Parallel processing
• Dynamic schema
• Automatic conflict resolution

                                                       8
Cost
• Allows sacrificing consistency (ACID)
   – at certain circumstances, but can deal with it
• Non-standard new API model
• Non-standard new Schema model
• New knowledge required to tune/optimize
• Less mature




                                                      9
Data/API/Schema model
• Data model: Key-Value store
   – (row:string, column:string, time:int64) → string
   – An opaque serialized object
• API model
   –   Get(key)
   –   Put(key, value)
   –   Delete(key)
   –   Execute(operation, key_list)
• Schema model
   – None
   – Kind of sparse table
                                                        10
Data processing
• MapReduce*
     – An API exposed by non-relational databases to process data
     – A functional programming pattern for parallelizing work
     – Brings the workers to the data
          • excellent fit for non-relational databases
     – Minimizes the programming to 2 simple functions
          • map & reduce



*: Jeffrey Dean and Sanjay Ghemawat,
MapReduce: Simplified Data Processing
on Large Clusters, Proceedings of the
6th Symposium on Operating Systems
Design and Implementation (OSDI),
2004.
                                                                    11
Optimization: Distributed indexing
• Exploits the characteristics of Cayley graphs to provide the scalability for
  supporting multiple distributed indexes of different types.
• Define a methodology to map various types of data and P2P overlays to a
  generalized Cayley graph structure.
• Propose self-tuning strategies to optimize the performance of the indexes
  defined over the generic Cayley overlay.




                                                                                 12
Optimization: Data migration
• Albatross is a technique for live migration in a
  multitenant database which can migrate a live tenant
  database with no aborted transactions.
   – Phase 1: Begin Migration.
   – Phase 2: Iterative Copying.
   – Phase 3: Atomic Handover.




                                                         13
Example: Oracle Berkeley DB
• High-performance embeddable database providing
  SQL, Java Object and Key-Value storage
  – Relational Storage - Support SQL.
  – Synchronization - extend the reach of existing applications
    to mobile devices by supporting unparalleled performance
    and a robust data store on the mobile device.
  – Replication - Provide a single-master multi-replica highly
    available database configuration.




                                             Storage engine

                                                                  14
Example: Amazon DynamoDB
• Fully managed NoSQL database service providing fast
  and predictable performance with seamless scalability
   – Provisioned throughput
      • Allocate dedicated resources to table to performance requirements,
        and automatically partitions data over a sufficient number of
        servers to meet request capacity.
   – Consistency model
      • The eventual consistency option maximizes read throughput.
   – Data Model
      • Attributes, Items and Tables




                                                                             15
Example: HBase
• Non-relational, distributed database running on top of
  HDFS providing Bigtable-like capabilities for Hadoop
   –   Strongly consistent reads/writes
   –   Automatic sharding
   –   Hadoop/HDFS Integration
   –   Block Cache and Bloom Filters
   –   Operational Management




                                                           16
Example: CouchDB
• Scalable, fault-tolerant, and schema-free document-
  oriented database
   –   Document Storage
   –   Distributed Architecture with Replication
   –   Map/Reduce Views and Indexes
   –   ACID Semantics
   –   Eventual Consistency
   –   Built for Offline




                                                        17
Example: Riak
• A distributed database architected for availability,
  fault-tolerance, operational simplicity and scalability.
   –   Operate in highly distributed environments
   –   Scale simply and intelligently
   –   Master-less
   –   Highly fault-tolerant
   –   Incredibly stable




                                                             18
Example: MongoDB
• Document-oriented NoSQL database system
  –   Scale horizontally without compromising functionality
  –   Document-oriented storage
  –   Full index support
  –   Master-slave replication
  –   Rich, document-based queries




                                                              19
Comparison with RDBMS
• Transaction
   – Web apps can (usually) do without transactions / strong
     consistency / integrity
   – Bigtable does not support transactions across multiple rows
      • support single-row transactions
      • provide an interface for batching writes across row keys at the
        clients

• Scalability
   – Parallel DBMS vs. MapReduce-base system




                                                                          20
THANK YOU!




             21
Backup




         22
Example of the CAP theorem
• When you have a lot of data which needs to be highly
  available, you'll usually need to partition it across
  machines & also replicate it to be more fault-tolerant
• This means, that when writing a record, all replica's
  must be updated too
• Now you need to choose between:
   – Lock all relevant replicas during update => be less available
   – Don't lock the replicas => be less consistent




                                                                     23

More Related Content

What's hot

NoSQL_Night
NoSQL_NightNoSQL_Night
NoSQL_Night
Clarence J M Tauro
 
Big Data and Hadoop - History, Technical Deep Dive, and Industry Trends
Big Data and Hadoop - History, Technical Deep Dive, and Industry TrendsBig Data and Hadoop - History, Technical Deep Dive, and Industry Trends
Big Data and Hadoop - History, Technical Deep Dive, and Industry Trends
Esther Kundin
 
1. introduction to no sql
1. introduction to no sql1. introduction to no sql
1. introduction to no sql
Anuja Gunale
 
Big data stores
Big data  storesBig data  stores
Big data stores
Kumaran Ramanujam
 
Cloud Deployments with Apache Hadoop and Apache HBase
Cloud Deployments with Apache Hadoop and Apache HBaseCloud Deployments with Apache Hadoop and Apache HBase
Cloud Deployments with Apache Hadoop and Apache HBase
DATAVERSITY
 
Relational and non relational database 7
Relational and non relational database 7Relational and non relational database 7
Relational and non relational database 7
abdulrahmanhelan
 
DBaaS with EDB Postgres on AWS
DBaaS with EDB Postgres on AWSDBaaS with EDB Postgres on AWS
DBaaS with EDB Postgres on AWS
EDB
 
Hadoop and WANdisco: The Future of Big Data
Hadoop and WANdisco: The Future of Big DataHadoop and WANdisco: The Future of Big Data
Hadoop and WANdisco: The Future of Big Data
WANdisco Plc
 
Lviv EDGE 2 - NoSQL
Lviv EDGE 2 - NoSQLLviv EDGE 2 - NoSQL
Lviv EDGE 2 - NoSQL
zenyk
 
Introduction to GlusterFS Webinar - September 2011
Introduction to GlusterFS Webinar - September 2011Introduction to GlusterFS Webinar - September 2011
Introduction to GlusterFS Webinar - September 2011
GlusterFS
 
Introduction to NuoDB
Introduction to NuoDBIntroduction to NuoDB
Introduction to NuoDB
Sandun Perera
 
Hadoop World 2011: Building Realtime Big Data Services at Facebook with Hadoo...
Hadoop World 2011: Building Realtime Big Data Services at Facebook with Hadoo...Hadoop World 2011: Building Realtime Big Data Services at Facebook with Hadoo...
Hadoop World 2011: Building Realtime Big Data Services at Facebook with Hadoo...
Cloudera, Inc.
 
NoSQL in Real-time Architectures
NoSQL in Real-time ArchitecturesNoSQL in Real-time Architectures
NoSQL in Real-time Architectures
Ronen Botzer
 
Harmonizing Multi-tenant HBase Clusters for Managing Workload Diversity
Harmonizing Multi-tenant HBase Clusters for Managing Workload DiversityHarmonizing Multi-tenant HBase Clusters for Managing Workload Diversity
Harmonizing Multi-tenant HBase Clusters for Managing Workload Diversity
HBaseCon
 
Apache HBase™
Apache HBase™Apache HBase™
Apache HBase™
Prashant Gupta
 
Big data - Online Training
Big data - Online TrainingBig data - Online Training
Big data - Online Training
Learntek1
 
Storage Systems For Scalable systems
Storage Systems For Scalable systemsStorage Systems For Scalable systems
Storage Systems For Scalable systems
elliando dias
 
Hbase: an introduction
Hbase: an introductionHbase: an introduction
Hbase: an introduction
Jean-Baptiste Poullet
 
Big Data Strategy for the Relational World
Big Data Strategy for the Relational World Big Data Strategy for the Relational World
Big Data Strategy for the Relational World
Andrew Brust
 
NoSQL and The Big Data Hullabaloo
NoSQL and The Big Data HullabalooNoSQL and The Big Data Hullabaloo
NoSQL and The Big Data Hullabaloo
Andrew Brust
 

What's hot (20)

NoSQL_Night
NoSQL_NightNoSQL_Night
NoSQL_Night
 
Big Data and Hadoop - History, Technical Deep Dive, and Industry Trends
Big Data and Hadoop - History, Technical Deep Dive, and Industry TrendsBig Data and Hadoop - History, Technical Deep Dive, and Industry Trends
Big Data and Hadoop - History, Technical Deep Dive, and Industry Trends
 
1. introduction to no sql
1. introduction to no sql1. introduction to no sql
1. introduction to no sql
 
Big data stores
Big data  storesBig data  stores
Big data stores
 
Cloud Deployments with Apache Hadoop and Apache HBase
Cloud Deployments with Apache Hadoop and Apache HBaseCloud Deployments with Apache Hadoop and Apache HBase
Cloud Deployments with Apache Hadoop and Apache HBase
 
Relational and non relational database 7
Relational and non relational database 7Relational and non relational database 7
Relational and non relational database 7
 
DBaaS with EDB Postgres on AWS
DBaaS with EDB Postgres on AWSDBaaS with EDB Postgres on AWS
DBaaS with EDB Postgres on AWS
 
Hadoop and WANdisco: The Future of Big Data
Hadoop and WANdisco: The Future of Big DataHadoop and WANdisco: The Future of Big Data
Hadoop and WANdisco: The Future of Big Data
 
Lviv EDGE 2 - NoSQL
Lviv EDGE 2 - NoSQLLviv EDGE 2 - NoSQL
Lviv EDGE 2 - NoSQL
 
Introduction to GlusterFS Webinar - September 2011
Introduction to GlusterFS Webinar - September 2011Introduction to GlusterFS Webinar - September 2011
Introduction to GlusterFS Webinar - September 2011
 
Introduction to NuoDB
Introduction to NuoDBIntroduction to NuoDB
Introduction to NuoDB
 
Hadoop World 2011: Building Realtime Big Data Services at Facebook with Hadoo...
Hadoop World 2011: Building Realtime Big Data Services at Facebook with Hadoo...Hadoop World 2011: Building Realtime Big Data Services at Facebook with Hadoo...
Hadoop World 2011: Building Realtime Big Data Services at Facebook with Hadoo...
 
NoSQL in Real-time Architectures
NoSQL in Real-time ArchitecturesNoSQL in Real-time Architectures
NoSQL in Real-time Architectures
 
Harmonizing Multi-tenant HBase Clusters for Managing Workload Diversity
Harmonizing Multi-tenant HBase Clusters for Managing Workload DiversityHarmonizing Multi-tenant HBase Clusters for Managing Workload Diversity
Harmonizing Multi-tenant HBase Clusters for Managing Workload Diversity
 
Apache HBase™
Apache HBase™Apache HBase™
Apache HBase™
 
Big data - Online Training
Big data - Online TrainingBig data - Online Training
Big data - Online Training
 
Storage Systems For Scalable systems
Storage Systems For Scalable systemsStorage Systems For Scalable systems
Storage Systems For Scalable systems
 
Hbase: an introduction
Hbase: an introductionHbase: an introduction
Hbase: an introduction
 
Big Data Strategy for the Relational World
Big Data Strategy for the Relational World Big Data Strategy for the Relational World
Big Data Strategy for the Relational World
 
NoSQL and The Big Data Hullabaloo
NoSQL and The Big Data HullabalooNoSQL and The Big Data Hullabaloo
NoSQL and The Big Data Hullabaloo
 

Viewers also liked

C-MR: Continuously Executing MapReduce Workflows on Multi-Core Processors
C-MR: Continuously Executing MapReduce Workflows on Multi-Core ProcessorsC-MR: Continuously Executing MapReduce Workflows on Multi-Core Processors
C-MR: Continuously Executing MapReduce Workflows on Multi-Core Processors
Qian Lin
 
Adaptive Execution Support for Malleable Computation
Adaptive Execution Support for Malleable ComputationAdaptive Execution Support for Malleable Computation
Adaptive Execution Support for Malleable Computation
Qian Lin
 
Kineograph: Taking the Pulse of a Fast-Changing and Connected World
Kineograph: Taking the Pulse of a Fast-Changing and Connected WorldKineograph: Taking the Pulse of a Fast-Changing and Connected World
Kineograph: Taking the Pulse of a Fast-Changing and Connected World
Qian Lin
 
In-situ MapReduce for Log Processing
In-situ MapReduce for Log ProcessingIn-situ MapReduce for Log Processing
In-situ MapReduce for Log Processing
Qian Lin
 
C-Cube: Elastic Continuous Clustering in the Cloud
C-Cube: Elastic Continuous Clustering in the CloudC-Cube: Elastic Continuous Clustering in the Cloud
C-Cube: Elastic Continuous Clustering in the Cloud
Qian Lin
 
Реляционные или нереляционные (Josh Berkus)
Реляционные или нереляционные (Josh Berkus)Реляционные или нереляционные (Josh Berkus)
Реляционные или нереляционные (Josh Berkus)
Ontico
 
Presto: Distributed Machine Learning and Graph Processing with Sparse Matrices
Presto: Distributed Machine Learning and Graph Processing with Sparse MatricesPresto: Distributed Machine Learning and Graph Processing with Sparse Matrices
Presto: Distributed Machine Learning and Graph Processing with Sparse Matrices
Qian Lin
 
Optimizing Virtual Machines Using Hybrid Virtualization
Optimizing Virtual Machines Using Hybrid VirtualizationOptimizing Virtual Machines Using Hybrid Virtualization
Optimizing Virtual Machines Using Hybrid Virtualization
Qian Lin
 
Trinity: A Distributed Graph Engine on a Memory Cloud
Trinity: A Distributed Graph Engine on a Memory CloudTrinity: A Distributed Graph Engine on a Memory Cloud
Trinity: A Distributed Graph Engine on a Memory Cloud
Qian Lin
 
RDBMS vs NoSQL-MongoDB
RDBMS vs NoSQL-MongoDBRDBMS vs NoSQL-MongoDB
RDBMS vs NoSQL-MongoDB
Intellipaat
 
Relational vs. Non-Relational
Relational vs. Non-RelationalRelational vs. Non-Relational
Relational vs. Non-Relational
PostgreSQL Experts, Inc.
 
Relational databases vs Non-relational databases
Relational databases vs Non-relational databasesRelational databases vs Non-relational databases
Relational databases vs Non-relational databases
James Serra
 

Viewers also liked (12)

C-MR: Continuously Executing MapReduce Workflows on Multi-Core Processors
C-MR: Continuously Executing MapReduce Workflows on Multi-Core ProcessorsC-MR: Continuously Executing MapReduce Workflows on Multi-Core Processors
C-MR: Continuously Executing MapReduce Workflows on Multi-Core Processors
 
Adaptive Execution Support for Malleable Computation
Adaptive Execution Support for Malleable ComputationAdaptive Execution Support for Malleable Computation
Adaptive Execution Support for Malleable Computation
 
Kineograph: Taking the Pulse of a Fast-Changing and Connected World
Kineograph: Taking the Pulse of a Fast-Changing and Connected WorldKineograph: Taking the Pulse of a Fast-Changing and Connected World
Kineograph: Taking the Pulse of a Fast-Changing and Connected World
 
In-situ MapReduce for Log Processing
In-situ MapReduce for Log ProcessingIn-situ MapReduce for Log Processing
In-situ MapReduce for Log Processing
 
C-Cube: Elastic Continuous Clustering in the Cloud
C-Cube: Elastic Continuous Clustering in the CloudC-Cube: Elastic Continuous Clustering in the Cloud
C-Cube: Elastic Continuous Clustering in the Cloud
 
Реляционные или нереляционные (Josh Berkus)
Реляционные или нереляционные (Josh Berkus)Реляционные или нереляционные (Josh Berkus)
Реляционные или нереляционные (Josh Berkus)
 
Presto: Distributed Machine Learning and Graph Processing with Sparse Matrices
Presto: Distributed Machine Learning and Graph Processing with Sparse MatricesPresto: Distributed Machine Learning and Graph Processing with Sparse Matrices
Presto: Distributed Machine Learning and Graph Processing with Sparse Matrices
 
Optimizing Virtual Machines Using Hybrid Virtualization
Optimizing Virtual Machines Using Hybrid VirtualizationOptimizing Virtual Machines Using Hybrid Virtualization
Optimizing Virtual Machines Using Hybrid Virtualization
 
Trinity: A Distributed Graph Engine on a Memory Cloud
Trinity: A Distributed Graph Engine on a Memory CloudTrinity: A Distributed Graph Engine on a Memory Cloud
Trinity: A Distributed Graph Engine on a Memory Cloud
 
RDBMS vs NoSQL-MongoDB
RDBMS vs NoSQL-MongoDBRDBMS vs NoSQL-MongoDB
RDBMS vs NoSQL-MongoDB
 
Relational vs. Non-Relational
Relational vs. Non-RelationalRelational vs. Non-Relational
Relational vs. Non-Relational
 
Relational databases vs Non-relational databases
Relational databases vs Non-relational databasesRelational databases vs Non-relational databases
Relational databases vs Non-relational databases
 

Similar to A Survey of Advanced Non-relational Database Systems: Approaches and Applications

Introduction to NoSQL
Introduction to NoSQLIntroduction to NoSQL
Introduction to NoSQL
PolarSeven Pty Ltd
 
Master.pptx
Master.pptxMaster.pptx
Master.pptx
KarthikR780430
 
No SQL- The Future Of Data Storage
No SQL- The Future Of Data StorageNo SQL- The Future Of Data Storage
No SQL- The Future Of Data Storage
Bethmi Gunasekara
 
Java scalability considerations yogesh deshpande
Java scalability considerations   yogesh deshpandeJava scalability considerations   yogesh deshpande
Java scalability considerations yogesh deshpande
IndicThreads
 
Nosql data models
Nosql data modelsNosql data models
Nosql data models
Viet-Trung TRAN
 
Learning from google megastore (Part-1)
Learning from google megastore (Part-1)Learning from google megastore (Part-1)
Learning from google megastore (Part-1)
Schubert Zhang
 
Nosql databases
Nosql databasesNosql databases
Nosql databases
Fayez Shayeb
 
Simple, Modular and Extensible Big Data Platform Concept
Simple, Modular and Extensible Big Data Platform ConceptSimple, Modular and Extensible Big Data Platform Concept
Simple, Modular and Extensible Big Data Platform Concept
Satish Mohan
 
module4-cloudcomputing-180131071200.pdf
module4-cloudcomputing-180131071200.pdfmodule4-cloudcomputing-180131071200.pdf
module4-cloudcomputing-180131071200.pdf
SumanthReddy540432
 
VTU 6th Sem Elective CSE - Module 4 cloud computing
VTU 6th Sem Elective CSE - Module 4  cloud computingVTU 6th Sem Elective CSE - Module 4  cloud computing
VTU 6th Sem Elective CSE - Module 4 cloud computing
Sachin Gowda
 
ScaleBase Webinar: Scaling MySQL - Sharding Made Easy!
ScaleBase Webinar: Scaling MySQL - Sharding Made Easy!ScaleBase Webinar: Scaling MySQL - Sharding Made Easy!
ScaleBase Webinar: Scaling MySQL - Sharding Made Easy!
ScaleBase
 
Data engineering
Data engineeringData engineering
Data engineering
Parimala Killada
 
NoSQL Architecture Overview
NoSQL Architecture OverviewNoSQL Architecture Overview
NoSQL Architecture Overview
Christopher Foot
 
NoSQLDatabases
NoSQLDatabasesNoSQLDatabases
NoSQLDatabases
Adi Challa
 
Chapter1: NoSQL: It’s about making intelligent choices
Chapter1: NoSQL: It’s about making intelligent choicesChapter1: NoSQL: It’s about making intelligent choices
Chapter1: NoSQL: It’s about making intelligent choices
Maynooth University
 
Big Data Storage Concepts from the "Big Data concepts Technology and Architec...
Big Data Storage Concepts from the "Big Data concepts Technology and Architec...Big Data Storage Concepts from the "Big Data concepts Technology and Architec...
Big Data Storage Concepts from the "Big Data concepts Technology and Architec...
raghdooosh
 
Intro to Big Data and NoSQL
Intro to Big Data and NoSQLIntro to Big Data and NoSQL
Intro to Big Data and NoSQL
Don Demcsak
 
Big Data and Cloud Computing
Big Data and Cloud ComputingBig Data and Cloud Computing
Big Data and Cloud Computing
Farzad Nozarian
 
BigData, NoSQL & ElasticSearch
BigData, NoSQL & ElasticSearchBigData, NoSQL & ElasticSearch
BigData, NoSQL & ElasticSearch
Sanura Hettiarachchi
 
SpringPeople - Introduction to Cloud Computing
SpringPeople - Introduction to Cloud ComputingSpringPeople - Introduction to Cloud Computing
SpringPeople - Introduction to Cloud Computing
SpringPeople
 

Similar to A Survey of Advanced Non-relational Database Systems: Approaches and Applications (20)

Introduction to NoSQL
Introduction to NoSQLIntroduction to NoSQL
Introduction to NoSQL
 
Master.pptx
Master.pptxMaster.pptx
Master.pptx
 
No SQL- The Future Of Data Storage
No SQL- The Future Of Data StorageNo SQL- The Future Of Data Storage
No SQL- The Future Of Data Storage
 
Java scalability considerations yogesh deshpande
Java scalability considerations   yogesh deshpandeJava scalability considerations   yogesh deshpande
Java scalability considerations yogesh deshpande
 
Nosql data models
Nosql data modelsNosql data models
Nosql data models
 
Learning from google megastore (Part-1)
Learning from google megastore (Part-1)Learning from google megastore (Part-1)
Learning from google megastore (Part-1)
 
Nosql databases
Nosql databasesNosql databases
Nosql databases
 
Simple, Modular and Extensible Big Data Platform Concept
Simple, Modular and Extensible Big Data Platform ConceptSimple, Modular and Extensible Big Data Platform Concept
Simple, Modular and Extensible Big Data Platform Concept
 
module4-cloudcomputing-180131071200.pdf
module4-cloudcomputing-180131071200.pdfmodule4-cloudcomputing-180131071200.pdf
module4-cloudcomputing-180131071200.pdf
 
VTU 6th Sem Elective CSE - Module 4 cloud computing
VTU 6th Sem Elective CSE - Module 4  cloud computingVTU 6th Sem Elective CSE - Module 4  cloud computing
VTU 6th Sem Elective CSE - Module 4 cloud computing
 
ScaleBase Webinar: Scaling MySQL - Sharding Made Easy!
ScaleBase Webinar: Scaling MySQL - Sharding Made Easy!ScaleBase Webinar: Scaling MySQL - Sharding Made Easy!
ScaleBase Webinar: Scaling MySQL - Sharding Made Easy!
 
Data engineering
Data engineeringData engineering
Data engineering
 
NoSQL Architecture Overview
NoSQL Architecture OverviewNoSQL Architecture Overview
NoSQL Architecture Overview
 
NoSQLDatabases
NoSQLDatabasesNoSQLDatabases
NoSQLDatabases
 
Chapter1: NoSQL: It’s about making intelligent choices
Chapter1: NoSQL: It’s about making intelligent choicesChapter1: NoSQL: It’s about making intelligent choices
Chapter1: NoSQL: It’s about making intelligent choices
 
Big Data Storage Concepts from the "Big Data concepts Technology and Architec...
Big Data Storage Concepts from the "Big Data concepts Technology and Architec...Big Data Storage Concepts from the "Big Data concepts Technology and Architec...
Big Data Storage Concepts from the "Big Data concepts Technology and Architec...
 
Intro to Big Data and NoSQL
Intro to Big Data and NoSQLIntro to Big Data and NoSQL
Intro to Big Data and NoSQL
 
Big Data and Cloud Computing
Big Data and Cloud ComputingBig Data and Cloud Computing
Big Data and Cloud Computing
 
BigData, NoSQL & ElasticSearch
BigData, NoSQL & ElasticSearchBigData, NoSQL & ElasticSearch
BigData, NoSQL & ElasticSearch
 
SpringPeople - Introduction to Cloud Computing
SpringPeople - Introduction to Cloud ComputingSpringPeople - Introduction to Cloud Computing
SpringPeople - Introduction to Cloud Computing
 

A Survey of Advanced Non-relational Database Systems: Approaches and Applications

  • 1. A Survey of Advanced Non-relational Database Systems: Approaches and Applications Speaker: LIN Qian http://www.comp.nus.edu.sg/~linqian
  • 2. Outline • Introduction • Non-relational database system – Requirement – Concepts – Approaches – Optimization – Examples • Comparison between RDBMS and non-relational database system 1
  • 3. Problem • The Web introduces a new scale for applications, in terms of: – Concurrent users (millions of reqs/second) – Data (peta-bytes generated daily) – Processing (all this data needs processing) – Exponential growth (surging unpredictable demands) • Shortage of existing RDBMS – Oracle, MS SQL, Sybase, MySQL, PostgreSQL, … – Trouble when dealing with very large traffic – Even with their high-end clustering solutions 2
  • 4. Problem • Why? – Applications using normalized database schema require the use of join, which doesn't perform well under lots of data and/or nodes – Existing RDBMS clustering solutions require scale-up, which is limited and not really scalable when dealing with exponential growth (e.g., 1000+ nodes) – Machines have upper limits on capacity 3
  • 5. Problem • Why not just use sharding? – Very complex and application-specific • Increased complexity of SQL • Single point of failure • Failover servers more complex • Backups more complex • Operational complexity added – Very problematic when adding/removing nodes – Basically, you end up denormalizing everything and loosing all benefits of relational databases Sharding: Split one or more tables by row across potentially multiple instances of the schema and database servers. 4
  • 6. Who faced this problem? • Web applications dealing with high traffic and massive data – Web service providers • Google, Yahoo!, Amazon, Facebook, Twitter, LinkedIn, … – Scientific data analysis • Weather, Ocean, tide, geothermy, … – Complex information processing • Financial, stock, telecommunication, … 5
  • 7. Solution • A new kind of DBMS, capable of handling web scale – Possibly sacrificing some level of feature • CAP theorem*: You can only optimize 2 out of these 3 – Consistency - the system is in a consistent state after an operation • All nodes see the same data at the same time • Strong consistency (ACID) vs. eventual consistency (BASE) – Availability - the system is “always on”, no downtime • Node failure tolerance: All clients can find some available replica. • software/hardware upgrade tolerance – Partition tolerance • The system continues to operate (read/write) despite arbitrary message loss or failure of part of the system * Eric A. Brewer, Towards Robust Distributed Systems, Proceedings of the 19th annual ACM symposium on Principles of Distributed Computing (PODC), 2000 6
  • 8. Non-relational database systems • Various solutions & products – BigTable, LevelDB (developed at Google) – Hbase (developed at Yahoo!) – Dynamo (developed at Amazon) – Cassandra (developed at FaceBook) – Voldemort (developed at LinkedIn) – Riak, Redis, CouchDB, MongoDB, Berkeley DB, … • Researches – NoDB, Walnut, LogBase, Albatross, Citrusleaf, HadoopDB – PIQL, RAMCloud 7
  • 9. Benefits • Massively scalable • Extremely fast • Highly available, decentralized and fault tolerant – no single-point-of-failure • Transparent sharding (consistent hashing) • Elasticity • Parallel processing • Dynamic schema • Automatic conflict resolution 8
  • 10. Cost • Allows sacrificing consistency (ACID) – at certain circumstances, but can deal with it • Non-standard new API model • Non-standard new Schema model • New knowledge required to tune/optimize • Less mature 9
  • 11. Data/API/Schema model • Data model: Key-Value store – (row:string, column:string, time:int64) → string – An opaque serialized object • API model – Get(key) – Put(key, value) – Delete(key) – Execute(operation, key_list) • Schema model – None – Kind of sparse table 10
  • 12. Data processing • MapReduce* – An API exposed by non-relational databases to process data – A functional programming pattern for parallelizing work – Brings the workers to the data • excellent fit for non-relational databases – Minimizes the programming to 2 simple functions • map & reduce *: Jeffrey Dean and Sanjay Ghemawat, MapReduce: Simplified Data Processing on Large Clusters, Proceedings of the 6th Symposium on Operating Systems Design and Implementation (OSDI), 2004. 11
  • 13. Optimization: Distributed indexing • Exploits the characteristics of Cayley graphs to provide the scalability for supporting multiple distributed indexes of different types. • Define a methodology to map various types of data and P2P overlays to a generalized Cayley graph structure. • Propose self-tuning strategies to optimize the performance of the indexes defined over the generic Cayley overlay. 12
  • 14. Optimization: Data migration • Albatross is a technique for live migration in a multitenant database which can migrate a live tenant database with no aborted transactions. – Phase 1: Begin Migration. – Phase 2: Iterative Copying. – Phase 3: Atomic Handover. 13
  • 15. Example: Oracle Berkeley DB • High-performance embeddable database providing SQL, Java Object and Key-Value storage – Relational Storage - Support SQL. – Synchronization - extend the reach of existing applications to mobile devices by supporting unparalleled performance and a robust data store on the mobile device. – Replication - Provide a single-master multi-replica highly available database configuration. Storage engine 14
  • 16. Example: Amazon DynamoDB • Fully managed NoSQL database service providing fast and predictable performance with seamless scalability – Provisioned throughput • Allocate dedicated resources to table to performance requirements, and automatically partitions data over a sufficient number of servers to meet request capacity. – Consistency model • The eventual consistency option maximizes read throughput. – Data Model • Attributes, Items and Tables 15
  • 17. Example: HBase • Non-relational, distributed database running on top of HDFS providing Bigtable-like capabilities for Hadoop – Strongly consistent reads/writes – Automatic sharding – Hadoop/HDFS Integration – Block Cache and Bloom Filters – Operational Management 16
  • 18. Example: CouchDB • Scalable, fault-tolerant, and schema-free document- oriented database – Document Storage – Distributed Architecture with Replication – Map/Reduce Views and Indexes – ACID Semantics – Eventual Consistency – Built for Offline 17
  • 19. Example: Riak • A distributed database architected for availability, fault-tolerance, operational simplicity and scalability. – Operate in highly distributed environments – Scale simply and intelligently – Master-less – Highly fault-tolerant – Incredibly stable 18
  • 20. Example: MongoDB • Document-oriented NoSQL database system – Scale horizontally without compromising functionality – Document-oriented storage – Full index support – Master-slave replication – Rich, document-based queries 19
  • 21. Comparison with RDBMS • Transaction – Web apps can (usually) do without transactions / strong consistency / integrity – Bigtable does not support transactions across multiple rows • support single-row transactions • provide an interface for batching writes across row keys at the clients • Scalability – Parallel DBMS vs. MapReduce-base system 20
  • 23. Backup 22
  • 24. Example of the CAP theorem • When you have a lot of data which needs to be highly available, you'll usually need to partition it across machines & also replicate it to be more fault-tolerant • This means, that when writing a record, all replica's must be updated too • Now you need to choose between: – Lock all relevant replicas during update => be less available – Don't lock the replicas => be less consistent 23

Editor's Notes

  1. Machines have upper limits on capacity
  2. Increased complexity of SQL - Increased bugs because the developers have to write more complicated SQL to handle sharding logic.Single point of failure - Corruption of one shard due to network/hardware/systems problems causes failure of the entire table.Failover servers more complex - Failover servers must themselves have copies of the fleets of database shards.Backups more complex - Database backups of the individual shards must coordinated with the backups of the other shards.Operational complexity added - Adding/removing indexes, adding/deleting columns, modifying the schema become much more difficult.
  3. Requirements to distributed systems