NoSQL @ Qbranch -2010-04-15

Not only SQL Mårten Gustafson Qbranch CODE tech-meet @ 2010-04-15

What? “NoSQL is a movement promoting a loosely deﬁned class of non-relational data stores that break with a long history of relational databases” - Wikipedia

What? “NoSQL is a movement promoting a loosely deﬁned class of non-relational data stores that break with a long history of relational databases” - Wikipedia Not a single technique Not a single type of data Not a single type of use case

Why? • Non-relational • Schema-less • “Easily” scalable • REST/JSON API = web friendly

What’s out there? Storage type License Implemented in Amazon Dynamo Key/Value n/a ? Cassandra Columnfamily ASL 2.0 Java CouchDB Document ASL 2.0 Erlang Dynomite Key/Value BSD/MIT-style Erlang HBase Columnfamily ASL 2.0 Java MongoDB Document AGPL v3.0 C++ Neo4J Graph AGPL v3.0 / Comm Java Riak Key/Value ASL 2.0 Erlang Redis Key/Value BSD/MIT-style C Scalaris Key/Value ASL 2.0 Erlang Tokyo Cabinet Key/Value LGPL C Voldemort Key/Value ASL 2.0 Java

Distribution • Master / Slave • Master / Slave(s) • Masterless (Master / Master)

Distribution Masterless Master/Slave Hot standby Amazon Dynamo X Cassandra X CouchDB X Dynomite X HBase ? MongoDB X X Neo4J* Riak X Redis X Scalaris X Tokyo Cabinet Voldemort X * Neo4J HA coming “soon”

Distribution Masterless Master/Slave ie Hot standbyw Amazon Dynamo X d v Cassandra X iﬁe l CouchDB X Dynomite X m p i HBase ? MongoDB y s X X Neo4J* e r v Riak X a Redis X is Scalaris X h i s Tokyo Cabinet Voldemort X T * Neo4J HA coming “soon”

Common factor “...of the web...” Of the who?!

Of the web “...Django may be built for the Web, but CouchDB is built of the Web. I’ve never seen software that so completely embraces the philosophies behind HTTP. CouchDB makes Django look old-school in the same way that Django makes ASP look outdated” - http://jacobian.org/writing/of-the-web/

Of the web “...CouchDB may succeeded, and it may fail; who knows. I’m sure of one thing, though — this is what the software of the future looks like” - http://jacobian.org/writing/of-the-web/

So freakin’ what?! All your webish skillz and tools apply...

So freakin’ what?! language-, platform- and OS-neutral load balancers proxies MIME / Content-Type All your webish skillz and tools apply... HTTP client libs (etag, if-modiﬁed-since, etc) caches

These guys can just suck it HTTP/REST is integration that works (YMMV)

Buckle Up Dorothy. Cause' Kansas, Is Going Bye-Bye

Key/Value Store I got keys but no locks

Riak Decentralized key-value store A ﬂexible map/reduce engine HTTP/JSON API A database ideally suited for Web applications

The Ring

The Ring 12 1 11 2 10 3 9 4 8 5 7 6 ring size = 12

The Ring One Ring size to rule them all, One Ring size to ﬁnd them, One Ring size to bring them all and in the cluster bind them...

Consistent Hashing Store/Save (PUT)

Consistent Hashing Read (GET) “I want “ is answered by: where is on the ring?

Cluster Instance A Instance B Instance C ring size = 12 instances = 3 ring size / nodes = ~slices per instances

Cluster - Read (GET) Instance A Instance B Instance C

Cluster - Read (GET) I can haz ? Instance A Instance B Instance C Hm, lives in a slice of the ring owned by instance C.

Cluster - Read (GET) I can haz ? Instance A Instance B Instance C Okidoki, now Hey C! I need where’s he...a yeah in my fourth slice

Cluster - Read (GET) I can haz ? Here ya go Instance A Instance B Instance C Cheers!

Riak “stuff”

Riak “stuff” Bucket Container/keyspace. Determines number of replicas for its contents

Riak “stuff” Consistent Hashing Key hashing technique used to distribute keys on the ring Bucket Container/keyspace. Determines number of replicas for its contents

Shares state, bucket and ring knowledge in the cluster Riak “stuff” Gossiping Consistent Hashing Key hashing technique used to distribute keys on the ring Bucket Container/keyspace. Determines number of replicas for its contents

Shares state, bucket and ring knowledge in the cluster Riak “stuff” Gossiping Consistent Hashing Key hashing technique used to distribute keys on the ring Hinted Handoff Covering for a Bucket failed “neighbor” node while gone Container/keyspace. Determines number of replicas for its contents

Shares state, bucket and ring knowledge in the cluster Riak “stuff” Allows retrieval of “weakly” linked objects Gossiping Links Consistent Hashing Key hashing technique used to distribute keys on the ring Hinted Handoff Covering for a Bucket failed “neighbor” node while gone Container/keyspace. Determines number of replicas for its contents

Shares state, bucket and ring knowledge in the cluster Riak “stuff” Allows retrieval of “weakly” linked objects Gossiping Links Consistent Hashing Key hashing technique used to distribute keys on the ring Hinted Handoff Merkle Tree Covering for a Bucket Data structure for failed “neighbor” node while gone Container/keyspace. efﬁcient summary about Determines number of objects. Gossiped. replicas for its contents

Shares state, bucket and ring knowledge in the cluster Riak “stuff” Allows retrieval of “weakly” linked objects Gossiping Links Node Consistent Hashing One server. Runs vnodes which claims partitions. Key hashing technique used to distribute keys on the ring Hinted Handoff Merkle Tree Covering for a Bucket Data structure for failed “neighbor” node while gone Container/keyspace. efﬁcient summary about Determines number of objects. Gossiped. replicas for its contents

Shares state, bucket and ring knowledge in the cluster Riak “stuff” Allows retrieval of “weakly” linked objects Partition Gossiping Links One slice (part) of the ring. Node Consistent Hashing One server. Runs vnodes which claims partitions. Key hashing technique used to distribute keys on the ring Hinted Handoff Merkle Tree Covering for a Bucket Data structure for failed “neighbor” node while gone Container/keyspace. efﬁcient summary about Determines number of objects. Gossiped. replicas for its contents

Shares state, bucket and ring knowledge in the cluster Riak “stuff” Allows retrieval of “weakly” linked objects Partition Gossiping Links One slice (part) of the ring. Node Auto correction of out-of-date objects Consistent Hashing One server. Runs vnodes which claims partitions. Read Repair Key hashing technique used to distribute keys on the ring Hinted Handoff Merkle Tree Covering for a Bucket Data structure for failed “neighbor” node while gone Container/keyspace. efﬁcient summary about Determines number of objects. Gossiped. replicas for its contents

Shares state, bucket and ring knowledge in the cluster Riak “stuff” Allows retrieval of “weakly” linked objects Partition Gossiping Links One slice (part) of the ring. Node Auto correction of out-of-date objects Consistent Hashing One server. Runs vnodes which claims partitions. Read Repair Key hashing technique used to distribute keys on the ring Hinted Handoff Merkle Tree Covering for a Bucket Data structure for failed “neighbor” node while gone Number of copies Container/keyspace. efﬁcient summary about of the same object Determines number of replicas for its contents objects. Gossiped. Replica in the cluster

Shares state, bucket and ring knowledge in the cluster Riak “stuff” Allows retrieval of “weakly” linked objects Partition Gossiping Links The complete “space”, One slice (part) of the ring. divided into partitions which are claimed by vnodes Ring Node Auto correction of out-of-date objects Consistent Hashing One server. Runs vnodes which claims partitions. Read Repair Key hashing technique used to distribute keys on the ring Hinted Handoff Merkle Tree Covering for a Bucket Data structure for failed “neighbor” node while gone Number of copies Container/keyspace. efﬁcient summary about of the same object Determines number of replicas for its contents objects. Gossiped. Replica in the cluster

Shares state, bucket and ring knowledge in the cluster Riak “stuff” Allows retrieval of “weakly” linked objects Partition Gossiping Links The complete “space”, One slice (part) of the ring. divided into partitions which Vector Clock are claimed by vnodes Conﬂic detection technique for objects. Ring Node Auto correction of out-of-date objects Consistent Hashing One server. Runs vnodes which claims partitions. Read Repair Key hashing technique used to distribute keys on the ring Hinted Handoff Merkle Tree Covering for a Bucket Data structure for failed “neighbor” node while gone Number of copies Container/keyspace. efﬁcient summary about of the same object Determines number of replicas for its contents objects. Gossiped. Replica in the cluster

Shares state, bucket and ring knowledge in the cluster Riak “stuff” Allows retrieval of “weakly” linked objects Partition Gossiping Links The complete “space”, One slice (part) of the ring. Vnode divided into partitions which Vector Clock are claimed by vnodes Conﬂic detection technique for objects. Ring Node Runs in a node and claims one Auto correction of out-of-date objects Consistent Hashing One server. Runs vnodes which claims partition on the ring partitions. Read Repair Key hashing technique used to distribute keys on the ring Hinted Handoff Merkle Tree Covering for a Bucket Data structure for failed “neighbor” node while gone Number of copies Container/keyspace. efﬁcient summary about of the same object Determines number of replicas for its contents objects. Gossiped. Replica in the cluster

Riak - Takeaways • No single point of failure • Choose your levels for: • availability • consistency • partition tolerance

But wait, there’s more... • Binary data + Content-Type = whatever • MP3’s, Images, Text, ... • Map/Reduce • Local data, parallel

This slide intentionally left blank

Document Store Relax

CouchDB Document oriented databased Kick ass replication HTTP/JSON API Map/reduce view (index) deﬁnitions

World view One document == JSON One document == One record Many documents == One database Many databases == One instance No schema

World view Documents can have attachments (binary + mime type) be rendered differently (HTML, XML)

A document Key, either you choose it or CouchDB does it for you { "_id": "b098445d587b1f347e48e1a79301de02", "_rev": "1-80bfd8302e0f08eec2396c8107cafc19", "platform": { "browser": "mozilla", "version": "1.9.1.8" }, "timestamp": 1270131033337 Revision } number

Views Filter Collate Aggregate

Views { "_id": "b098445d587b1f347e48e1a79301de02", "_rev": "1-80bfd8302e0f08eec2396c8107cafc19", "platform": { "browser": "mozilla", "version": "1.9.1.8" }, "timestamp": 1270131033337 } + function(doc) { emit(doc.platform.browser, doc.platform.version); } = { "total_rows": 1, "offset": 0, "rows": [ "id": "b098445d587b1f347e48e1a79301de02", "key": "mozilla", "value": "1.9.1.8" ]

Views Views are stored as an accessible web resource on disk and incrementally updated as well as replicated with the database

Replication Peer to peer Online/Ofﬂine Conﬂict detection and resolution Any number of nodes Local Remote

Replication

CouchDB “stuff”

CouchDB “stuff” Append only Hence, won’t corrupt its data ﬁles

CouchDB “stuff” MVCC Multi version concurrency control. Writers do not block readers. Readers do not block writers. Append only Hence, won’t corrupt its data ﬁles

CouchDB “stuff” BDCRR MVCC Bi-directional, conﬂict resolving, replication Multi version concurrency control. Writers do not block readers. Readers do not block writers. Append only Hence, won’t corrupt its data ﬁles

CouchDB “stuff” BDCRR MVCC Bi-directional, conflict resolving, replication Multi version concurrency control. Writers do not block readers. Readers do not block writers. Append only Compaction Hence, won’t corrupt its data files Append only will cause data files to grow. Compaction to the rescue, in the background - for your pleasure.

CouchDB “stuff” BDCRR MVCC Bi-directional, conflict resolving, replication Multi version concurrency control. Writers do not block readers. Readers do not block writers. Append only Compaction Hence, won’t corrupt its data files Append only will cause data files to grow. Compaction to the rescue, in ACID the background - for your pleasure. Awesome, Cool, Impressive, Dope

CouchDB - Takeaways • Kick ass replication • Views are fast • Can host and serve complete webapps

Outro • Test one or more NoSQL thingys • Get familiar with Brewers CAP theorem • Get familiar with the Dynamo paper

Over and out. Mårten Gustafson @martengustafson http://marten.gustafson.pp.se/ marten.gustafson@gmail.com

http://bcho.tistory.com	1569
http://nosql.mypopescu.com	791
http://jujo00obo2o234ungd3t8qjfcjrs3o6k-a-sites-opensocial.googleusercontent.com	376
http://www.slideshare.net	21
https://jujo00obo2o234ungd3t8qjfcjrs3o6k-a-sites-opensocial.googleusercontent.com	16
http://mgustafson.wordpress.com	15
http://thinkery.me	5
http://www.techgig.com	3
http://webcache.googleusercontent.com	2
http://blog.naver.com	2
http://static.slidesharecdn.com	2
http://devmeat.com	1

NoSQL @ Qbranch -2010-04-15

by Mårten Gustafson on Apr 16, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

12 Embeds 2,803

Statistics

NoSQL @ Qbranch -2010-04-15 — Presentation Transcript