Front Range PHP NoSQL Databases

NoSQL Databases Jon Meredith [email_address]

What isn't NoSQL?
NOT a standard.
NOT a product.
NOT a single technology.

Well, what is it?
It's a buzzword .
A banner for non-relational databases to organize under.
Mostly created in response to scaling and reliability problems.
Huge differences between 'NoSQL' systems – but have elements in common.

Where did it come from?
They've been around for a while
Local key/value stores
Object databases
Graph databases
XML databases
New problems are emerging
Internet search
e-commerce
Social networking

Where did it come from?
Some efforts came from scaling the web...
Several papers published
2006 – Google BigTable
2007 – Dynamo Paper
In 2008 - explosion of data storage projects
All shambling under the NoSQL banner.

Really, why not use RDBMs?
I need to perform arbitrary queries
My application needs transactions
Data needs to be nicely normalized
I have replication for scalabilty/reliability

Data Mapping Woes
Relational databases divide data into columns made up of tables.
Programmers use complex nested data structures
Hashes
Sets
Arrays
Things of things
Have to map between the two

Data Mapping Woes (2)
Data in systems evolve over time … which means changes to the schema.
Upgrade/rollback scripts have to operate on the whole database – could be millions of rows.
Doing phased rollouts is hard … the application needs to do work

Alternative!
Let the application do it
Use convenient language features
PHP serialize/unserialize
… or use standards for mixed platforms
JSON very popular and well supported
Google's protocol buffers
… even XML
Design for forward compatibility
Preserve unknown fields
Version objects

Scalability and Availability
Scalability
How many requests you can process
Availability
How does your service degrade as things break.
RDBMS solutions - replication and sharding

Scaling RDBMs - Replication
Master-Slave replication is easiest
Every change on the master happens on the slave.
Slaves are read-only. Does not scale INSERT, UPDATE, DELETE queries.
Application responsible for distributing queries to correct server.

Scaling RDBMs - Replication
Multi-master ring replication
Can update any master
Updates travel around the ring
What happens when it fails?
Reconfigure the ring
What happens on return
Synchronize the master
Add back in to the ring

Replication
Replication is usually asynchronous for performance – you don't want to wait for the slowest slave on each update.
Replication takes time – there is time lag between the first and last server to see an update.
You may not read your writes – not getting aCid properties any more.

Scaling RDBMS – Sharding
Do application level splitting of data
Split large table into N smaller tables
Use Id modulo N to find the right table
Tables could be spread across multiple database servers
But the application needs to know where to query

Availability
If you want availability you need multiple servers – maybe even multiple sites.
In the real world you get network partitions
Just because you can't see your other data center doesn't mean users can't.
What should you do if you can't see the other data center?

Availability
Degrade one site to read-only
Defeats availability
If you allow both sites to operate
There's a chance two users could modify the same data.
The application needs to know how to resolve it

The bottom line...
Building systems that are
...Scalable...
...Available...
...Maintainable...
with an RDBMs requires large efforts by application developers and operational staff

It's hard because...
Significant work for developers.
App needs to convert data to table/columns
App needs to know data location
App needs to handle failover
App needs to handle inconsistency
Work for operational staff
Fixing replication topologies and synchronizing servers is fiddly work.

Last decades bleeding edge is here
Organizations with big problems started experimenting with alternatives
Developed internal systems during the mid 2000s
Distributed by design
Different data models
Published details in 2006/2007

Amazon
Huge e-commerce vendor.
Amazon cares about customer experience
Availabilty
Latency at the 99 th percentile
Built as an SOA – pages built from hundreds of services.
Amazon runs multiple data centers.
Hardware failure is their normal state
Network partitions common

Amazon Requirements
Shopping cart service must always be available
Customers should be able to view and add to their carts (in their words)
If disks are failing
Network routes are flapping
Data centers are being destroyed by tornadoes

Amazon Observations
Many services just stored state.
Access by primary key
No queries
Examples
Shopping carts
Best seller lists
Customer profiles
Hard to scale out relational databases

Amazon Solution: Dynamo
Primary key access only
Fault tolerant: Keeps N copies of the data
Designed for inconsistency
Totally decentralized – nodes 'gossip' state
Self-healing

Eventual Consistency 1
Brewer's CAP Theorem
Consistency
Availability
Partition tolerance
Pick two out of three!
Amazon chose A-P over C

N copies of each value
Read operations (get) require 'R' nodes to respond
Write operations (put) require 'W' nodes to respond
If R+W > N nodes will read their writes (if no failure)
NRW tunes the cluster – typically (3,2,2)

Consequence of availability: Conflicts
Conflicts can come from
Network partitions
Applications themselves – no transactions or locking
Applications must handle conflicts
Dynamo minimizes with vector clocks

Vector Clocks

Partitioning

Example: Shopping Cart
User browses site – adds 3 widgets

Shopping Cart - Conflict Network Failure

Shopping Cart - Merge

Open Source Dynamo
Dynamo is internal to Amazon
Open source options
Riak from Basho
Project Voldemort

Google BigTables
Used internally at Google
Indexing the web
Google Earth
Finance
Distributed storage system for structured data

Data representation
Data stored in tables.
Table indexed by {key,timestamp} and a variable number of sparse columns
Columns are grouped into column families. Columns in a family are stored together.
Each table is broken into tablets.
Tablets are stored on a cluster file system (GFS).

BigTable – Column Families Copyright Google

Map/Reduce
Processing framework that sits on top of BigTable.
Programmers write two functions map() and reduce().
Table is mapped, then reduced.
Job control system monitors and resubmits.

Map/Reduce Source: institutes.lanl.gov

BigTable has inspired...
Hadoop/Hbase
Cassandra
Riak
CouchDB
Map/Reduce

Explosion of NoSQL Dbs
Too many projects
Two good resources
http://nosql.mypopescu.com/
http://www.vineetgupta.com/ 2010/01/nosql-databases-part-1-landscape.html

So many projects! Dynamo, BigTables, Redis Riak, Voldemort, CouchDb, Peanuts Hadoop/Hbase, Cassandra, Hypertable MongoDb, Terrastore, Scalaris, BerkleyDB MemcacheDB, Dynomite, Neo4J, TokyoCabinet … and more

NoSQL Characteristics
Broad types
Key/Value
Sparse Column Family
Document oriented
Persistence
In memory
On disk
Distribution
Replicated
Decentralized

Riak from Basho http://riak.basho.com
Dynamo clone written in Erlang
RESTful HTTP interface
Fully distributed
Clients for multiple languages
Multiple storage backends
In-memory
Filesystem
Embedded InnoDB
I work there now!

Redis 1.2
http://code.google.com/p/redis/
Key/Value Store with structured values
Written in C
Memcache-like protocol
In use at
Github
Engine Yard
VideoWiki

Redis 1.2 (cont)
Values can be strings, sets, ordered sets, lists
Operations like increment, decrement, intersection, push, pop
In-memory (can be backed by disk)
Auto-sharding in client libraries
No fault tolerance (coming after 2.0)
Example: retwis – Twitter clone in PHP

Cassandra
http://incubator.apache.org/cassandra/
BigTable ColumnFamily data model
Dynamo data distribution
Written in Java
Thrift based interface
In use at
Facebook
Twitter

CouchDB
Document oriented database
All JSON documents
Written in Erlang
Used by Ubuntu One
HTTP interface
Uses Javascript for indexing/mapreduce
Incremental replication

BerkleyDB
Sleepycat now owned by Oracle
Key/Value Store
Multi-threaded
Multi-process
Replicated
Tranactional
Alternative: Tokyo Cabinet

I'm out of time
MongoDB
Neo4J – Graph Database
Peanuts – Yahoo

This is all great but...
Relational databases provide a lot of functionality.
Giving up queries
Even range queries are hard for distributed hash systems.
No transactions – rules out some classes of applications.
Space is still evolving

Conclusion
NoSQL systems give applications the tools they need for scalability/availability
They force you to think about distributed design issues like consistency.
Play with them!

http://frontrangephp.org	370
http://jonsbraindump.blogspot.com	157
http://www.frontrangephp.org	111
http://www.slideshare.net	31
http://jonsbraindump.blogspot.co.uk	5
http://translate.googleusercontent.com	2
http://jonsbraindump.blogspot.ca	2
http://www.blogger.com	1
http://jonsbraindump.blogspot.com.es	1
http://jonsbraindump.blogspot.jp	1
http://jonsbraindump.blogspot.in	1
http://jonsbraindump.blogspot.it	1

Front Range PHP NoSQL Databases

by Jon Meredith on Mar 11, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

12 Embeds 683

Statistics

Front Range PHP NoSQL Databases — Presentation Transcript