Front Range PHP NoSQL Databases
Upcoming SlideShare
Loading in...5
×
 

Front Range PHP NoSQL Databases

on

  • 6,661 views

The presentation I did for the FrontRange PHP User Group on 3/10/2010.

The presentation I did for the FrontRange PHP User Group on 3/10/2010.

Statistics

Views

Total Views
6,661
Views on SlideShare
5,495
Embed Views
1,166

Actions

Likes
3
Downloads
92
Comments
0

28 Embeds 1,166

http://frontrangephp.org 548
http://jonsbraindump.blogspot.com 242
http://www.aalizwel.com 118
http://www.frontrangephp.org 111
http://aalizwel.com 32
http://www.slideshare.net 31
http://jonsbraindump.blogspot.co.uk 14
http://jonsbraindump.blogspot.ru 13
http://jonsbraindump.blogspot.in 8
http://jonsbraindump.blogspot.se 5
http://jonsbraindump.blogspot.de 5
http://jonsbraindump.blogspot.it 4
http://jonsbraindump.blogspot.jp 4
http://jonsbraindump.blogspot.com.br 4
http://jonsbraindump.blogspot.com.au 4
http://jonsbraindump.blogspot.fr 3
http://jonsbraindump.blogspot.nl 3
http://jonsbraindump.blogspot.com.es 3
http://jonsbraindump.blogspot.ca 3
http://translate.googleusercontent.com 3
http://web.archive.org 1
http://jonsbraindump.blogspot.kr 1
http://jonsbraindump.blogspot.ch 1
http://jonsbraindump.blogspot.cz 1
http://jonsbraindump.blogspot.gr 1
http://jonsbraindump.blogspot.ie 1
http://www.blogger.com 1
http://jonsbraindump.blogspot.no 1
More...

Accessibility

Categories

Upload Details

Uploaded via as OpenOffice

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Introduce Disclose work for Basho Working on Dynamo clone for the last couple of years

Front Range PHP NoSQL Databases Front Range PHP NoSQL Databases Presentation Transcript

  • NoSQL Databases Jon Meredith [email_address]
  • What isn't NoSQL?
    • NOT a standard.
    • NOT a product.
    • NOT a single technology.
  • Well, what is it?
      It's a buzzword .
    • A banner for non-relational databases to organize under.
    • Mostly created in response to scaling and reliability problems.
    • Huge differences between 'NoSQL' systems – but have elements in common.
  • Where did it come from?
    • They've been around for a while
      • Local key/value stores
      • Object databases
      • Graph databases
      • XML databases
    • New problems are emerging
      • Internet search
      • e-commerce
      • Social networking
  • Where did it come from?
    • Some efforts came from scaling the web...
    • Several papers published
      • 2006 – Google BigTable
      • 2007 – Dynamo Paper
    • In 2008 - explosion of data storage projects
    • All shambling under the NoSQL banner.
  • Really, why not use RDBMs?
    • I need to perform arbitrary queries
    • My application needs transactions
    • Data needs to be nicely normalized
    • I have replication for scalabilty/reliability
  • Data Mapping Woes
    • Relational databases divide data into columns made up of tables.
    • Programmers use complex nested data structures
      • Hashes
      • Sets
      • Arrays
      • Things of things
    • Have to map between the two
  • Data Mapping Woes (2)
    • Data in systems evolve over time … which means changes to the schema.
    • Upgrade/rollback scripts have to operate on the whole database – could be millions of rows.
    • Doing phased rollouts is hard … the application needs to do work
  • Alternative!
    • Let the application do it
    • Use convenient language features
      • PHP serialize/unserialize
    • … or use standards for mixed platforms
      • JSON very popular and well supported
      • Google's protocol buffers
      • … even XML
    • Design for forward compatibility
      • Preserve unknown fields
      • Version objects
  • Scalability and Availability
    • Scalability
      • How many requests you can process
    • Availability
      • How does your service degrade as things break.
    • RDBMS solutions - replication and sharding
  • Scaling RDBMs - Replication
    • Master-Slave replication is easiest
    • Every change on the master happens on the slave.
    • Slaves are read-only. Does not scale INSERT, UPDATE, DELETE queries.
    • Application responsible for distributing queries to correct server.
  • Scaling RDBMs - Replication
    • Multi-master ring replication
      • Can update any master
      • Updates travel around the ring
      • What happens when it fails?
        • Reconfigure the ring
      • What happens on return
        • Synchronize the master
        • Add back in to the ring
  • Replication
    • Replication is usually asynchronous for performance – you don't want to wait for the slowest slave on each update.
    • Replication takes time – there is time lag between the first and last server to see an update.
    • You may not read your writes – not getting aCid properties any more.
  • Scaling RDBMS – Sharding
    • Do application level splitting of data
      • Split large table into N smaller tables
      • Use Id modulo N to find the right table
    • Tables could be spread across multiple database servers
      • But the application needs to know where to query
  • Availability
    • If you want availability you need multiple servers – maybe even multiple sites.
    • In the real world you get network partitions
      • Just because you can't see your other data center doesn't mean users can't.
    • What should you do if you can't see the other data center?
  • Availability
    • Degrade one site to read-only
      • Defeats availability
    • If you allow both sites to operate
      • There's a chance two users could modify the same data.
      • The application needs to know how to resolve it
  • The bottom line...
    • Building systems that are
      • ...Scalable...
      • ...Available...
      • ...Maintainable...
      • with an RDBMs requires large efforts by application developers and operational staff
  • It's hard because...
    • Significant work for developers.
      • App needs to convert data to table/columns
      • App needs to know data location
      • App needs to handle failover
      • App needs to handle inconsistency
    • Work for operational staff
      • Fixing replication topologies and synchronizing servers is fiddly work.
  • Last decades bleeding edge is here
    • Organizations with big problems started experimenting with alternatives
    • Developed internal systems during the mid 2000s
      • Distributed by design
      • Different data models
    • Published details in 2006/2007
  • Amazon
    • Huge e-commerce vendor.
    • Amazon cares about customer experience
      • Availabilty
      • Latency at the 99 th percentile
    • Built as an SOA – pages built from hundreds of services.
    • Amazon runs multiple data centers.
      • Hardware failure is their normal state
      • Network partitions common
  • Amazon Requirements
    • Shopping cart service must always be available
    • Customers should be able to view and add to their carts (in their words)
      • If disks are failing
      • Network routes are flapping
      • Data centers are being destroyed by tornadoes
  • Amazon Observations
    • Many services just stored state.
      • Access by primary key
      • No queries
    • Examples
      • Shopping carts
      • Best seller lists
      • Customer profiles
    • Hard to scale out relational databases
  • Amazon Solution: Dynamo
    • Primary key access only
    • Fault tolerant: Keeps N copies of the data
    • Designed for inconsistency
    • Totally decentralized – nodes 'gossip' state
    • Self-healing
  • Eventual Consistency 1
    • Brewer's CAP Theorem
      • Consistency
      • Availability
      • Partition tolerance
    • Pick two out of three!
    • Amazon chose A-P over C
  • Eventual Consistency 2
    • N copies of each value
    • Read operations (get) require 'R' nodes to respond
    • Write operations (put) require 'W' nodes to respond
    • If R+W > N nodes will read their writes (if no failure)
    • NRW tunes the cluster – typically (3,2,2)
  • Eventual Consistency 3
    • Consequence of availability: Conflicts
    • Conflicts can come from
      • Network partitions
      • Applications themselves – no transactions or locking
    • Applications must handle conflicts
    • Dynamo minimizes with vector clocks
  • Vector Clocks
  • Partitioning
  • Example: Shopping Cart
    • User browses site – adds 3 widgets
  • Shopping Cart - Conflict Network Failure
  • Shopping Cart - Merge
  • Open Source Dynamo
    • Dynamo is internal to Amazon
    • Open source options
      • Riak from Basho
      • Project Voldemort
  • Google BigTables
    • Used internally at Google
      • Indexing the web
      • Google Earth
      • Finance
    • Distributed storage system for structured data
  • Data representation
    • Data stored in tables.
    • Table indexed by {key,timestamp} and a variable number of sparse columns
    • Columns are grouped into column families. Columns in a family are stored together.
    • Each table is broken into tablets.
    • Tablets are stored on a cluster file system (GFS).
  • BigTable – Column Families Copyright Google
  • Map/Reduce
    • Processing framework that sits on top of BigTable.
    • Programmers write two functions map() and reduce().
    • Table is mapped, then reduced.
    • Job control system monitors and resubmits.
  • Map/Reduce Source: institutes.lanl.gov
  • BigTable has inspired...
    • Hadoop/Hbase
    • Cassandra
    • Riak
    • CouchDB
    Map/Reduce
  • Explosion of NoSQL Dbs
    • Too many projects
    • Two good resources
      • http://nosql.mypopescu.com/
      • http://www.vineetgupta.com/ 2010/01/nosql-databases-part-1-landscape.html
  • So many projects! Dynamo, BigTables, Redis Riak, Voldemort, CouchDb, Peanuts Hadoop/Hbase, Cassandra, Hypertable MongoDb, Terrastore, Scalaris, BerkleyDB MemcacheDB, Dynomite, Neo4J, TokyoCabinet … and more
  • NoSQL Characteristics
    • Broad types
      • Key/Value
      • Sparse Column Family
      • Document oriented
    • Persistence
      • In memory
      • On disk
    • Distribution
      • Replicated
      • Decentralized
  • Riak from Basho http://riak.basho.com
    • Dynamo clone written in Erlang
    • RESTful HTTP interface
    • Fully distributed
    • Clients for multiple languages
    • Multiple storage backends
      • In-memory
      • Filesystem
      • Embedded InnoDB
    • I work there now!
  • Redis 1.2
    • http://code.google.com/p/redis/
    • Key/Value Store with structured values
    • Written in C
    • Memcache-like protocol
    • In use at
      • Github
      • Engine Yard
      • VideoWiki
  • Redis 1.2 (cont)
    • Values can be strings, sets, ordered sets, lists
    • Operations like increment, decrement, intersection, push, pop
    • In-memory (can be backed by disk)
    • Auto-sharding in client libraries
    • No fault tolerance (coming after 2.0)
    • Example: retwis – Twitter clone in PHP
  • Cassandra
    • http://incubator.apache.org/cassandra/
    • BigTable ColumnFamily data model
    • Dynamo data distribution
    • Written in Java
    • Thrift based interface
    • In use at
      • Facebook
      • Twitter
  • CouchDB
    • Document oriented database
      • All JSON documents
    • Written in Erlang
    • Used by Ubuntu One
    • HTTP interface
    • Uses Javascript for indexing/mapreduce
    • Incremental replication
  • BerkleyDB
    • Sleepycat now owned by Oracle
    • Key/Value Store
      • Multi-threaded
      • Multi-process
      • Replicated
      • Tranactional
    • Alternative: Tokyo Cabinet
  • I'm out of time
    • MongoDB
    • Neo4J – Graph Database
    • Peanuts – Yahoo
  • This is all great but...
    • Relational databases provide a lot of functionality.
      • Giving up queries
      • Even range queries are hard for distributed hash systems.
      • No transactions – rules out some classes of applications.
      • Space is still evolving
  • Conclusion
    • NoSQL systems give applications the tools they need for scalability/availability
    • They force you to think about distributed design issues like consistency.
    • Play with them!