Embracing Constraints with CouchDB (ZC10 2010-11-04)

EMBRACING CONSTRAINTS WITH COUCHDB

David Zülke

David Zuelke

http://en.wikipedia.org/wiki/File:EU-Germany.svg

http://en.wikipedia.org/wiki/File:München_Panorama.JPG

Founder

Lead Developer

@dzuelke

A DISCLAIMER FIRST Before You All Figure This Out Yourselves...

NEIN NEIN NEIN NEIN DAS IST BETRUG

This talk is not really about embracing constraints

I’ll tell you what it’s really about when we’re ﬁnished

I’ll also apologize to you for lying at that point, and not now

(it’s always easier to apologize than to ask for permission)

COUCHDB IN THREE SLIDES Full Of DIS IS SRS BSNS Bullet Points

COUCHDB STORES DOCUMENTS • CouchDB stores documents with arbitrary keys and values • Each document is identiﬁed by an ID and has a revision • Documents can have ﬁle attachments • Stored as JSON, so it’s easy to interface with

COUCHDB SPEAKS HTTP • CouchDB uses HTTP to communicate with clients & servers • That means scalability • That means a lot of kick ass stuff totally for free • Caching • Load Balancing • Content Negotiation

COUCHDB USES MVCC • Multiversion Concurrency Control • When updating, you must supply a revision number • Your change will be rejected if the revision is not the latest • All writes are serialized • No need for locks, but puts some responsibility on developers

SOME DETAILS An In-Depth Look At What Makes CouchDB Different

availability Do you know the partition CAP X theorem? tolerance consistency

“So, CouchDB does not have consistency of CAP?”

“Booh, that means my data will be inconsistent. Fail!”

psssshhh

YOUR MOM IS INCONSISTENT

CouchDB is eventually consistent

When replicating, conﬂicting revisions will be marked as such

These conﬂicts can then be resolved (users, daemons,...)

and everything will be ﬁne o/

which brings us to...

REPLICATION • You can do Master-Master replication • Conﬂicts are detected and marked automatically • Conﬂicts are supposed to be resolved by applications • Or by users, who usually know best what to do!

CouchDB is Ground Computing

Imagine a world where every computer runs CouchDB

Ubuntu One already does, to sync bookmarks etc!

MAP/REDUCE

BASIC PRINCIPLE: MAPPER • The Mapper reads records and emits <key, value> pairs • Example: Apache access.log • Each line is a record • Extract client IP address and number of bytes transferred • Emit IP address as key, number of bytes as value • For hourly rotating logs, the job can be split across 24 nodes* * In pratice, it’s a lot smarter than that

BASIC PRINCIPLE: REDUCER •A Reducer is given a key and all values for this speciﬁc key • Even if there are many Mappers on many computers; the results are aggregated before they are handed to Reducers • Example: Apache access.log • The Reducer is called once for each client IP (that’s our key), with a list of values (transferred bytes) • We simply sum up the bytes to get the total trafﬁc per IP!

EXAMPLE OF MAPPED INPUT IP Bytes 212.122.174.13 18271 212.122.174.13 191726 212.122.174.13 198 74.119.8.111 91272 74.119.8.111 8371 212.122.174.13 43

REDUCER WILL RECEIVE THIS IP Bytes 18271 191726 212.122.174.13 198 43 91272 74.119.8.111 8371

AFTER REDUCTION IP Bytes 212.122.174.13 210238 74.119.8.111 99643

COUCHDB INCREMENTAL MAPREDUCE

THE KEY DIFFERENCE • Maps and Reduces are incremental: • If one document changes, only that one document needs: • mapping • reduction • Then a few new reduce runs are performed to compute the ﬁnal result

MAPPER: DOCS BY TAGS function(doc) { if(doc.type == 'product') { (doc.tags || []).forEach(function(tag) { emit(tag, doc); }); } }

MAPREDUCE: COUNT TAGS function(doc) { if(doc.type == 'product') { (doc.tags || []).forEach(function(tag) { emit(tag, 1); }); } } function(key, values) { return sum(values); }

BUT WAIT! There are no tables :(

so... how do you join data from related documents?

JOIN CATEGORIES & ITEMS function(doc) { if(doc.type == 'product') { emit([doc._id, 0], doc); emit([doc._id, 1], { _id: doc.category_id }); } } ["123", 0] {_id: "123", _rev: "5-‐a72", type: "product", "name": "Laser Beam"} ["123", 1] {_id: "est", _rev: "2-‐9af", type: "category", "name": "Evil Stuff"} ["817", 0] {_id: "817", _rev: "2-‐aa8", type: "product", "name": "Rocketship"} ["817", 1] {_id: "cst", _rev: "3-‐d8a", type: "category", "name": "Cool Stuff"} ["441", 0] {_id: "441", _rev: "19-‐fdf", type: "product", "name": "Sharks"} ["441", 1] {_id: "est", _rev: "2-‐9af", type: "category", "name": "Evil Stuff"}

JOIN PRODUCTS & ITEMS function(doc) { if(doc.type == 'category') { emit([doc._id, 0], doc); } elseif(doc.type == 'product') { emit([doc.category_id, doc._id], doc); } } ["est", 0] {_id: "est", _rev: "2-‐9af", type: "category", "name": "Evil Stuff"} ["est", "123"] {_id: "123", _rev: "5-‐a72", type: "product", "name": "Laser Beam"} ["est", "441"] {_id: "441", _rev: "19-‐fdf", type: "product", "name": "Sharks"} ["cst", 0] {_id: "cst", _rev: "3-‐d8a", type: "category", "name": "Cool Stuff"} ["cst", "817"] {_id: "817", _rev: "2-‐aa8", type: "product", "name": "Rocketship"}

BUT... BUT... WAIT! How do you validate document structures if it’s all schema-less?

VALIDATE DOCUMENTS function (newDoc, savedDoc, userCtx) { if(savedDoc && savedDoc.created_at != newDoc.created_at) { throw({forbidden: 'created_at is immutable'}); } if(doc.type == 'product') { if(!doc.price) { throw({forbidden: 'product must have a price'}); } } }

VALIDATE DOCUMENTS function (newDoc, savedDoc, userCtx) { function require(beTrue, message) { if(!beTrue) throw({forbidden: message}); } require(savedDoc && savedDoc.created_at != newDoc.created_at, 'created_at is immutable' ); if(doc.type == 'product') { require(!doc.price, 'product must have a price' ); } }

LUCENE INTEGRATION Full Control Over What Is Indexed, And How

COUCHAPP Python Tool For Development And Deployment

DEMO TIME Let’s Relax On The Couch

!e End

FURTHER READING • http://books.couchdb.org/ • http://couchdb.apache.org/ • http://github.com/couchapp/couchapp • http://github.com/rnewson/couchdb-lucene/ • http://janl.github.com/couchdbx/ • http://j.mp/oqbQs (E4X in CouchDB for XML parsing)

DID YOU SEE THE HEAD FAKE? This Talk Was Not About Embracing Constraints It Was About Embracing Awesomeness

Questions?

THANK YOU! This was http://joind.in/2315 by @dzuelke

Embracing Constraints with CouchDB (ZC10 2010-11-04)

by David Zuelke on Nov 04, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

2 Embeds 46

Statistics

Embracing Constraints with CouchDB (ZC10 2010-11-04) — Presentation Transcript

http://www.berejeb.com	40
http://www.devetdesign.com	6