• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
SDEC2011 NoSQL concepts and models

SDEC2011 NoSQL concepts and models






Total Views
Views on SlideShare
Embed Views



0 Embeds 0

No embeds



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

    SDEC2011 NoSQL concepts and models SDEC2011 NoSQL concepts and models Presentation Transcript

    • What Exactly is NoSQL?Document databases, Column-family stores, Key-value pairs, moreShashank Tiwariblog: shanky.org | twitter: @tshankyst@treasuryofideas.com
    • NoSQL?
    • NoSQL : Various Shapes and Sizes• Document Databases• Column-family Oriented Stores• Key/value Data stores• XML Databases• Object Databases• Graph Databases
    • Document Databases• mostly MongoDB, little CouchDB
    • What is a document db?• One that stores documents• Popular options: • MongoDB -- C++ • CouchDB -- Erlang • Also Amazon’s SimpleDB• ...what exactly is a document?
    • In the real world• (Source: http://guide.couchdb.org/draft/why.html)
    • In terms of JSON• {name: “John Doe”,• zip: 10001}
    • What about db schema?• Schema-less• Different documents could be stored in a single collection
    • Data types: MongoDB• Essential JSON types:• string• integer• boolean• double
    • Data types: MongoDB (...cont)• Additional JSON types• null, array and object• BSON types -- binary encoded serialization of JSON like documents • date, binary data, object id, regular expression and code • (Reference: bsonspec.org)
    • A BSON example: object id
    • Data types: CouchDB• Everything JSON• Large objects: attachments
    • CRUD operations for documents• Create• Read• Update• Delete
    • MongoDB: Create Document• use mydb• w = {name: “John Doe”, zip: 10001};• db.location.save(w);
    • Create db and collection• Lazily created• Implicitly created• use mydb• db.collection.save(w)
    • MongoDB: Read Document• db.location.find({zip: 10001});• { "_id" : ObjectId("4c97053abe67000000003857"), "name" : "John Doe", "zip" : 10001 }
    • MongoDB: Read Document (...cont)• db.location.find({name: "John Doe"});• { "_id" : ObjectId("4c97053abe67000000003857"), "name" : "John Doe", "zip" : 10001 }
    • MongoDB: Update Document• Atomic operations on single documents• db.location.update( { name:"John Doe" }, { $set: { name: "Jane Doe" } } );
    • Indexes(explain)• db.ratings.find().explain();
    • Indexes(explain output)•{• "cursor" : "BasicCursor",• "nscanned" : 1000209,• "nscannedObjects" : 1000209,• "n" : 1000209,• "millis" : 1549,• "indexBounds" : {
    • Indexes(ensure index)• db.ratings.ensureIndex({ movie_id:1 });• db.ratings.ensureIndex({ movie_id:-1 });
    • Indexes(explain when index used)•{• "cursor" : "BtreeCursor movie_id_1",• "nscanned" : 2077,• "nscannedObjects" : 2077,• "n" : 2077,• "millis" : 2,• "indexBounds" : {
    • Indexes(get indexes)• db.ratings.getIndexes();
    • Sorted Ordered Column-family Datastores• Sorted• Ordered• Distributed• Map
    • Essential schema
    • Multi-dimensional View
    • A Map/Hash View•{• "row_key_1" : { "name" : {• "first_name" : "Jolly", "last_name" : "Goodfellow"• } } },• "location" : { "zip": "94301" },
    • Architectural View (HBase)
    • The Persistence Mechanism
    • The underlying file format
    • Model Wrappers (The GAE Way)• Python • Model, Expando, PolyModel• Java • JDO, JPA
    • HBase Data Access• Thrift + Avro• Java API -- HTable, HBaseAdmin• Hive (SQL like)• MapReduce -- sink and/or source
    • Transactions• Atomic row level• GAE Entity Groups
    • Indexes• Row ordered• Secondary indexes• GAE style multiple indexes • thinking from output to query
    • Use cases• Many Google’s Products• Facebook Messaging• StumbleUpon • Open TSDB• Mahalo, Ning, Meetup, Twitter, Yahoo!• Lily -- open source CMS built on HBase & Solr
    • Brewer’s CAP Theorem• http://www.cs.berkeley.edu/~brewer/cs262b-2004/PODC-keynote.pdf• http://theory.lcs.mit.edu/tds/papers/Gilbert/Brewer6.ps
    • Distributed Systems & Consistency (case: success)
    • Distributed Systems & Consistency (case: failure)
    • Binding by Transactions
    • Consistency Spectrum
    • Inconsistency Window
    • RWN Math• R – Number of nodes that are read from.• W – Number of nodes that are written to.• N – Total number of nodes in the cluster.• In general: R < N and W < N for higher availability
    • R+W>N• Easy to determine consistent state• R + W = 2N • absolutely consistent, can provide ACID gaurantee• In all cases when R + W > N there is some overlap between read and write nodes.
    • R = 1, W = N• more reads than writes•W=N • 1 node failure = entire system unavailable
    • R = N, W =1•W=N • Chance of data inconsistency quite high•R=N • Read only possible when all nodes in the cluster are available
    • R = W = ceiling ((N + 1)/2)Effective quorum for eventual consistency
    • Eventual consistency variants• Causal consistency -- A writes and informs B then B always sees updated value• Read-your-writes-consistency -- A writes a new value and never see the old one• Session consistency -- read-your-writes-consistency within a client session• Monotonic read consistency -- once seen a new value, never return previous value• Monotonic write consistency -- serialize writes by the same process
    • Dynamo Techniques• Consistent Hashing (Incremental scalability)• Vector clocks (high availability for writes)• Sloppy quorum and hinted handoff (recover from temporary failure)• Gossip based membership protocol (periodic, pair wise, inter-process interactions, low reliability, random peer selection)• Anti-entropy using Merkle trees• (source: http://s3.amazonaws.com/AllThingsDistributed/sosp/amazon- dynamo-sosp2007.pdf)
    • Consistent Hashing
    • Vector clocks (a trivial example)• 4 hackers: Joe, Hillary, Eric and Ajay decide to meetup• Joe -- suggests Palo Alto (t0)• Hillary and Eric -- decide to meet in Mountain View (t1)• Eric and Ajay -- decide to meet in Los Altos (t2)• Joe mails: PA, Hillary responds: Mtn View, Ajay responds: Los Altos (t3) • both Hillary and Ajay say: Eric knows
    • Vector clocks (how it works)• Venue : Palo Alto• Vector Clock: Joe (ver 1)• Venue: Mountain View• Vector Clock: Joe (ver 1), Hillary (ver 1), Eric (ver 1)• Venue: Los Altos• Vector Clock: Joe (ver 1), Ajay (ver 1), Eric (ver 1)
    • Vector clock (resolution)• Venue : Palo Alto• Vector Clock: Joe (ver 1)• Venue: Mountain View• Vector Clock: Joe (ver 1), Hillary (ver 1), Ajay (ver 0), Eric (ver 2)• Venue: Los Altos• Vector Clock: Joe (ver 1), Hillary (ver 0), Ajay (ver 1), Eric (ver 1)
    • CouchDB MVCC Style• (Source: http://guide.couchdb.org/draft/consistency.html)
    • Key/value Stores• Memcached• Membase• Redis• Tokyo Cabinet• Kyoto Cabinet• Berkeley DB
    • Redis -- a key-value data structure server• open source key-value store• a data structure server • values in key-value pairs can be strings, hashes, lists, sets, sorted sets
    • Where to find it?• redis.io• download a copy from http://redis.io/download
    • Who is building it?• Core developers • Salvatore Sanfilippo, twitter: @antirez • Pieter Noordhuis, twitter: @pnoordhuis• Main sponsor • VMware
    • Written in• ANSI C • runs on POSIX compliant systems with no external dependencies
    • How can it be used?• as an in memory data store • with option to persist to disk• in standalone mode or as a master-slave replicated set • Redis cluster -- coming soon! (June 2011)• as cache
    • Redis Architecture
    • Download and install• curl -O http://redis.googlecode.com/files/redis-2.2.0-rc4.tar.gz • (just a 436kb download)• tar zxvf redis-2.2.0-rc4.tar.gz• cd redis-2.2.0-rc4• make & make install (installs in /usr/local/bin)• make test (to be sure you install it correctly)
    • Start the redis-server• /usr/local/bin/redis-server• ...Server started, Redis version 2.1.12• ...The server is now ready to accept connections on port 6379
    • Connect with redis-cli• /usr/local/bin/redis-cli• redis> set key1 val1• OK• redis> get key1• "val1"
    • String key-value pairs• like memcached • with persistence• key and value -- binary-safe strings
    • Binary-safe?• redis> set "a key _" "another value"• OK• redis> get "a key _"• "another value"
    • Questions?• blog: shanky.org | twitter: @tshanky• st@treasuryofideas.com