NoSQL Databases Jon Meredith [email_address]

NOT a product.

NOT a single technology.

Mostly created in response to scaling and reliability problems.

Huge differences between 'NoSQL' systems – but have elements in common.

Object databases

Graph databases

e-commerce

Social networking

All shambling under the NoSQL banner.

My application needs transactions

Data needs to be nicely normalized

I have replication for scalabilty/reliability

Sets

Arrays

Upgrade/rollback scripts have to operate on the whole database – could be millions of rows.

Doing phased rollouts is hard … the application needs to do work

Google's protocol buffers

Version objects

Every change on the master happens on the slave.

Slaves are read-only. Does not scale INSERT, UPDATE, DELETE queries.

Application responsible for distributing queries to correct server.

Updates travel around the ring

Add back in to the ring

Replication takes time – there is time lag between the first and last server to see an update.

You may not read your writes – not getting aCid properties any more.

The application needs to know how to resolve it

...Available...

...Maintainable...

with an RDBMs requires large efforts by application developers and operational staff

App needs to know data location

App needs to handle failover

Network partitions common

Network routes are flapping

Data centers are being destroyed by tornadoes

Best seller lists

Fault tolerant: Keeps N copies of the data

Designed for inconsistency

Totally decentralized – nodes 'gossip' state

Self-healing

Availability

Amazon chose A-P over C

Read operations (get) require 'R' nodes to respond

Write operations (put) require 'W' nodes to respond

If R+W > N nodes will read their writes (if no failure)

NRW tunes the cluster – typically (3,2,2)

Dynamo minimizes with vector clocks

Vector Clocks

Partitioning

Shopping Cart - Conflict Network Failure

Shopping Cart - Merge

Project Voldemort

Google Earth

Table indexed by {key,timestamp} and a variable number of sparse columns

Columns are grouped into column families. Columns in a family are stored together.

Each table is broken into tablets.

Tablets are stored on a cluster file system (GFS).

BigTable – Column Families Copyright Google

Programmers write two functions map() and reduce().

Table is mapped, then reduced.

Job control system monitors and resubmits.

Map/Reduce Source: institutes.lanl.gov

CouchDB Map/Reduce

http://www.vineetgupta.com/ 2010/01/nosql-databases-part-1-landscape.html

So many projects! Dynamo, BigTables, Redis Riak, Voldemort, CouchDb, Peanuts Hadoop/Hbase, Cassandra, Hypertable MongoDb, Terrastore, Scalaris, BerkleyDB MemcacheDB, Dynomite, Neo4J, TokyoCabinet … and more

Sparse Column Family

Decentralized

RESTful HTTP interface

Fully distributed

Clients for multiple languages

Filesystem

Key/Value Store with structured values

Written in C

Memcache-like protocol

Engine Yard

VideoWiki

Operations like increment, decrement, intersection, push, pop

In-memory (can be backed by disk)

Auto-sharding in client libraries

No fault tolerance (coming after 2.0)

Example: retwis – Twitter clone in PHP

BigTable ColumnFamily data model

Dynamo data distribution

Written in Java

Thrift based interface

Twitter

Used by Ubuntu One

HTTP interface

Uses Javascript for indexing/mapreduce

Incremental replication

Multi-process

Replicated

Neo4J – Graph Database

Peanuts – Yahoo

Even range queries are hard for distributed hash systems.

No transactions – rules out some classes of applications.

Space is still evolving

They force you to think about distributed design issues like consistency.

Play with them!