OSCON 2011 Learning CouchDB

Learning CouchDB

About Me

Co-Founder andTechnical Director

from Vermont Battery Park - Burlington, Vermont by Marty Desilets, on Flickr

Organizer BTV

(Minor) Contributor

Author http://oreilly.com/catalog/9781449303129/ http://oreilly.com/catalog/9781449303433/

About You?

Installing CouchDB

OS X$ brew install couchdbOR$ sudo port install couchdbORCouchbase Server:http://info.couchbase.com/couchbaseEEdownload.html

Ubuntu$ sudo aptitude install couchdbORCouchbase Server:http://info.couchbase.com/couchbaseEEdownload.html

Red Hat Enterprise Linux 5$ sudo yum install couchdbORCouchbase Server:http://info.couchbase.com/couchbaseEEdownload.html

WindowsBinary:http://wiki.apache.org/couchdb/Windows_binary_installerORCouchbase Server:http://info.couchbase.com/couchbaseEEdownload.html

Futon:http://localhost:5984/_utils/

cURL$ curl -iX GET http://localhost:5984/HTTP/1.1 200 OKServer: CouchDB/1.0.2 (Erlang OTP/R14B){ "couchdb": "Welcome", "version": "1.0.2"}

CouchDB Basics

Document-OrientedSchema-lessStores key/value pairsKey is a string representing a document’s IDValue is a document represented as a JSON Object:{ "_id": "fe8a4a1ca998daafaaa04d7d9000187f", "title": "Learning CouchDB"}

{"title":"JSON Documents"}

Bene ts of JSONHuman-readable and simple data interchange formatData structures from many programming languages can be easily convertedto and from JSONLightweight—doesn’t consume much bandwidthEasily integrated into JavaScript applications (e.g. CouchApps)

JSON ObjectsA JSON Object is a collection of name/value pairs.The name is a String.The value can be a: • Boolean (false or true) • JSON Array (e.g. ["a", "b", "c"]) • JSON Object • Number • String • JSON NULL (null)

Example JSON Object{ "_id":"978-0-596-15589-6", "title":"CouchDB: The De nitive Guide", "subtitle":"Time to Relax", "authors":[ "J. Chris Anderson", "Jan Lehnardt", "Noah Slater" ], "publisher":"OReilly Media", "released":"2010-01-19", "pages":272}

Indexed QueriesMapReduce “views” generate B+ tree indexesIndexes are incrementally updated by Map functionsViews are queried for result sets:{ // … "rows": [ { "id": "fe8a4a1ca998daafaaa04d7d9000187f", "key":"Learning CouchDB", "value": null } ]}

Multi-VersionConcurrency Control (MVCC)Each document stores its current revision number and hash:{ "_id": "fe8a4a1ca998daafaaa04d7d9000187f", "_rev": "1-bd6ed00168af92895217953f69504bb4", "title":"Learning CouchDB"}Optimistic concurrency ensures that a document has not been modi ed sinceit was last retrievedOld revisions are saved for replication purposes, but are eventually pruned

ReplicationPeer-based, bi-directional replication of databasesIn multi-master scenarios, replicate database A → Band database B → AReplication can happen continuously or after having being disconnected for longperiods of time (or never connected)Document update con icts are agged and can then be processed by yourapplication or by the end-user

Shared NothingHorizontally scalableEventually consistent (once replication completes)Always available (just add nodes)Partition-tolerant (replication can happen later)

HTTP APINo custom binary protocol (well, there is Hovercraft)Everyone speaks HTTP—trivially easy to interface with CouchDBHTTP is: • distributed • scalable • cacheable

Under The HoodAtomic, Consistent, Isolated, and Durable (ACID)“Crash-only” design—no shutdown processAppend-only B-tree lesRuns on the Erlang Open Telecom Platform (OTP)Erlang is a highly-concurrent functional programming languageErlang OTP is well suited for distributed systems

Right Tool for the Job

When You MightConsider CouchDBYou’ve found yourself denormalizing your SQL database for better performanceYour domain model is a “ t” for documents (e.g. a CMS)Your application is read-heavyYou need a high level of concurrency and can give up consistency in exchangeYou need horizontal scalabilityYou want your database to be able to run anywhere, even on mobile devices,and even when disconnected from the cluster

Trade-OﬀsNo ad hoc queries. You need to know what you’re going to want to queryahead of timeNo concept of “joins”. You can relate data, but watch out for consistency issuesTransactions are limited to document boundariesCouchDB trades storage space for performanceReads scale well, writes scale well with third-party software (open source)

Other Alternatives to SQLMongoDBhttp://www.mongodb.org/Redishttp://redis.io/Cassandrahttp://cassandra.apache.org/Riakhttp://www.basho.com/HBase (a database for Hadoop)http://hbase.apache.org/

Don’t be so quick to get rid of SQL! There aremany problems for which an SQL database is agood t. SQL is a very powerful and exiblequery language.

Libraries

JavaScriptcouch.js (synchronous)http://localhost:5984/_utils/script/couch.jsjquery.couch.js (asynchronous)http://localhost:5984/_utils/script/jquery.couch.jsjqCouchhttp://plugins.jquery.com/project/jqcouch

PerlCouchDB-Clienthttp://search.cpan.org/dist/CouchDB-Client/POE-Component-Client-CouchDBhttp://search.cpan.org/dist/POE-Component-Client-CouchDB/CouchDB-Viewhttp://search.cpan.org/dist/CouchDB-View/CouchDB-Deployhttp://search.cpan.org/dist/CouchDB-Deploy/

PHPSaghttp://www.saggingcouch.com/PHPCouchhttp://www.phpcouch.org/PHPillowhttp://arbitracker.org/phpillow.htmlPHP CouchDB Extension (PECL)http://www.topdog.za.net/php_couchdb_extensionDoctrine2 ODM

PythonCouchDB Python Libraryhttp://code.google.com/p/couchdb-python/CouchDBKithttp://couchdbkit.org/Paisley: CouchDB client for Twistedhttps://launchpad.net/paisley/

RubyCouchRest (“close to the metal”)https://github.com/couchrest/couchrestCouchRest Model (“close to shiny metal with rounded edges”)https://github.com/couchrest/couchrest_modelCouch Potatohttps://github.com/langalex/couch_potatoCouchFoo (ActiveRecord API)https://github.com/georgepalmer/couch_fooCouchObjecthttp://couchobject.rubyforge.org/

JavaEktorphttp://code.google.com/p/ektorp/CouchDB4Jhttps://github.com/mbreese/couchdb4j

.NETEasyCouchDBhttps://github.com/hhariri/EasyCouchDBRelaxhttps://github.com/arobson/Relax/Divan (C#)https://github.com/foretagsplatsen/DivanFunctionalDivan (F#)https://github.com/kolosy/FunctionalDivanOttomanhttps://github.com/sinesignal/ottomanSharpCouchhttp://code.google.com/p/couchbrowse/

ErlangCouchbeam—Erlang CouchDB Kithttps://github.com/benoitc/couchbeamerlang_couchdbhttps://github.com/ngerakines/erlang_couchdb/eCouchhttp://code.google.com/p/ecouch/

ScalaSCouchDBhttps://github.com/debasishg/scouchdb

LispChillaxhttps://github.com/sykopomp/chillaxClouchDB (a Common Lisp library)http://common-lisp.net/project/clouchdb/

RESTful HTTP APIcurl -iX PUT http://localhost:5984/mydb

REST and RelaxationMessages are self-described via HTTP headers and HTTP status codesURIs identify resourcesHTTP methods de ne operations on resourcesHypermedia controls are possible through show and list functions(if you don’t consider JSON to be hypermedia)

HTTP Headers

Request Header ExamplesUser-Agent: curl/7.21.6…Host: localhost:5984Accept: application/jsonContent-Type: application/jsonContent-Length: 15If-None-Match: "2-bbd27429fd1a0daa2b946cbacb22dc3e"If-Match: "2-bbd27429fd1a0daa2b946cbacb22dc3e"

Response Header ExamplesServer: CouchDB/1.0.2 (Erlang OTP/R14B)Date: Tue, 07 Jun 2011 19:30:33 GMTContent-Type: application/jsonContent-Length: 211Cache-Control: must-revalidateLocation: http://localhost:5984/mydb/mydocEtag: "1-967a00dﬀ5e02add41819138abb3284d"

HTTP Status Codes

Successful 2xx200 OK201 Created202 Accepted

Redirection 3xx301 Moved Permanently304 Not Modi ed

Client Error 4xx400 Bad Request401 Unauthorized403 Forbidden405 Method Not Allowed404 Not Found409 Con ict412 Precondition Failed

Server Error 5xx500 Internal Server

HTTP Methods

GET a Resource$ curl -iX GET http://localhost:5984/HTTP/1.1 200 OKServer: CouchDB/1.0.2 (Erlang OTP/R14B){ "couchdb": "Welcome", "version": "1.0.2"}

Get Just HEADers$ curl -iX HEAD http://localhost:5984/HTTP/1.1 200 OKServer: CouchDB/1.0.2 (Erlang OTP/R14B)

PUT a Database$ curl -iX PUT http://localhost:5984/mydbHTTP/1.1 201 CreatedLocation: http://localhost:5984/mydb{"ok":true}

POST a Document$ curl -iX POST http://localhost:5984/mydb-H "Content-Type: application/json"-d {}HTTP/1.1 201 CreatedLocation: http://localhost:5984/mydb/0a72c9c36bd169818dc97ed18b000aa4{ "ok":true, "id":"0a72c9c36bd169818dc97ed18b000aa4", "rev":"1-967a00dﬀ5e02add41819138abb3284d"}

PUT a Document$ curl -iX PUT http://localhost:5984/mydb/0a72c9c36bd169818dc97ed18b000aa4-H "Content-Type: application/json"-d { "_rev":"1-967a00dﬀ5e02add41819138abb3284d", "title":"Learning CouchDB"}HTTP/1.1 201 Created{ "ok":true, "id":"0a72c9c36bd169818dc97ed18b000aa4", "rev":"2-516027e3179a22a22e06874c374e8ef0"}

DELETE a Document$ curl -iX DELETE http://localhost:5984/mydb/0a72c9c36bd169818dc97ed18b000aa4?rev=2-516027e3179a22a22e06874c374e8ef0HTTP/1.1 200 OK{ "ok":true, "id":"0a72c9c36bd169818dc97ed18b000aa4", "rev":"3-e9a5aa1c486eee23c84fa028bc904991"}

More API Examples

Create a Document$ curl -iX POST http://localhost:5984/mydb-H "Content-Type: application/json"-d {"_id":"mydoc"}HTTP/1.1 201 CreatedLocation: http://localhost:5984/mydb/mydoc{ "ok":true, "id":"mydoc", "rev":"1-967a00dﬀ5e02add41819138abb3284d"}

Read a Document$ curl -iX GET http://localhost:5984/mydb/mydocHTTP/1.1 200 OKEtag: "1-967a00dﬀ5e02add41819138abb3284d"{ "_id":"mydoc", "_rev":"1-967a00dﬀ5e02add41819138abb3284d"}

Update a Document$ curl -iX PUT http://localhost:5984/mydb/mydoc-H "Content-Type: application/json"-d { "_id":"mydoc", "_rev":"1-967a00dﬀ5e02add41819138abb3284d", "title":"CouchDB: The De nitive Guide"}HTTP/1.1 201 Created{ "ok":true, "id":"mydoc", "rev":"2-bbd27429fd1a0daa2b946cbacb22dc3e"}

CouchDB allows for conditional requests,saving bandwidth and processing.

Conditional GET$ curl -iX GET http://localhost:5984/mydb/mydoc-H If-None-Match: "2-bbd27429fd1a0daa2b946cbacb22dc3e"HTTP/1.1 304 Not Modi edEtag: "2-bbd27429fd1a0daa2b946cbacb22dc3e"Content-Length: 0

Delete a Document$ curl -iX DELETE http://localhost:5984/mydb/mydoc-H If-Match: "2-bbd27429fd1a0daa2b946cbacb22dc3e"HTTP/1.1 200 OK{ "ok":true, "id":"mydoc", "rev":"3-29d2ef6e0d3558a3547a92dac51f3231"}

Read a Deleted Document$ curl -iX GET http://localhost:5984/mydb/mydocHTTP/1.1 404 Object Not Found{ "error":"not_found", "reason":"deleted"}

Read a Deleted Document$ curl -iX GET http://localhost:5984/mydb/mydoc?rev=3-29d2ef6e0d3558a3547a92dac51f3231HTTP/1.1 200 OKEtag: "3-29d2ef6e0d3558a3547a92dac51f3231"{ "_id":"mydoc", "_rev":"3-29d2ef6e0d3558a3547a92dac51f3231", "_deleted":true}

Fetch Revisions$ curl -iX GET http://localhost:5984/mydb/mydoc?rev=3-29d2ef6e0d3558a3547a92dac51f3231&revs=trueHTTP/1.1 200 OK{ // … "_revisions":{ "start":3, "ids":[ "29d2ef6e0d3558a3547a92dac51f3231", "bbd27429fd1a0daa2b946cbacb22dc3e", "967a00dﬀ5e02add41819138abb3284d" ] }}

Do not rely on older revisions! Revisionsare used for concurrency control and forreplication. Old revisions are removedduring database compaction.

Replication Party! Cloned by Asha ten Broeke, on Flickr

My Business Card(business_card.json){ "fn":"Bradley Holt", "url":"http://bradley-holt.com/", "role":"Technical Director", "org":"Found Line, Inc.", "email":"bradley.holt@foundline.com", "adr":{ "street-address":"7 Kilburn Street", "locality":"Burlington", "region":"Vermont", "postal-code":"05401", "country-name":"USA" }, "tel":"802-383-4737 ext. 11"}

When POSTing a document, CouchDB willassign a UUID as the document’s ID if oneis not speci ed.

POST My Business Card$ curl -iX POST http://localhost:5984/cards-H "Content-Type: application/json"-d @business_card.jsonHTTP/1.1 201 CreatedLocation: http://localhost:5984/cards/fe8a4a1ca998daafaaa04d7d90000574{ "ok":true, "id":"fe8a4a1ca998daafaaa04d7d90000574", "rev":"1-2526abc0e10aa39167ﬀ7b16a77b4dad"}

New Document in Futon

You Replicate Me$ curl -iX POST http://localhost:5984/_replicate-H "Content-Type: application/json"-d { "source": "https://oscon-tutorial.iriscouch.com/cards", "target": "cards", "create_target": true}HTTP/1.1 200 OK{ "ok":true, //…}

Futon’s Replicator

GET All Documents$ curl -iX GET http://localhost:5984/cards/_all_docsHTTP/1.1 200 OK{ // … "rows": [ { "id": "fe8a4a1ca998daafaaa04d7d90000574", // … } ]}

All Documents in Futon

GET One Document$ curl -iX GET http://localhost:5984/cards/fe8a4a1ca998daafaaa04d7d90000574HTTP/1.1 200 OKEtag: "2-3a584544d14db477911bd420a24b7055"{ "_id": "fe8a4a1ca998daafaaa04d7d90000574", "_rev": "2-3a584544d14db477911bd420a24b7055", "fn": "Bradley Holt", "url": "http://bradley-holt.com/", // …}

Get Document in Futon

Create Your Business Card(business_card.json){ "fn":"Your Name Here", "url":"http://yoururl.com/", "role":"Your Role", "org":"Your Organization", "email":"youremail@yoururl.com", "adr":{ "street-address":"Your Street", "locality":"Your City", "region":"Your State", "postal-code":"Your Postal Code", "country-name":"Your Country" }, "tel":"Your Phone Number"}

POST Your Business Card$ curl -iX POST http://localhost:5984/cards-H "Content-Type: application/json"-d @business_card.jsonHTTP/1.1 201 Created

ContinuousPush Replication$ curl -iX POST http://localhost:5984/_replicate-H "Content-Type: application/json"-d { "source": "cards", "target": "https://oscon-tutorial.iriscouch.com/cards", "continuous": true}HTTP/1.1 200 OK{ "ok":true, //…}

…from Futon

Continuous Pull Replication$ curl -iX POST http://localhost:5984/_replicate-H "Content-Type: application/json"-d { "source": "https://oscon-tutorial.iriscouch.com/cards", "target": "cards", "continuous": true}HTTP/1.1 200 OK{ "ok":true, //…}

…from Futon

Viewing Replication Status

GET All Documents Again$ curl -iX GET http://localhost:5984/tutorial/_all_docsHTTP/1.1 200 OK{ // … "rows": [ { "id": "fe8a4a1ca998daafaaa04d7d90000574", // … }, // … ]}

Cancel Push Replication$ curl -iX POST http://localhost:5984/_replicate-H "Content-Type: application/json"-d { "source": "cards", "target": "https://oscon-tutorial.iriscouch.com/cards/", "cancel": true}HTTP/1.1 200 OK{ "ok":true, //…}

Cancel Pull Replication$ curl -iX POST http://localhost:5984/_replicate-H "Content-Type: application/json"-d { "source": "https://oscon-tutorial.iriscouch.com/cards/", "target": "cards", "cancel": true}HTTP/1.1 200 OK{ "ok":true, //…}

MapReduce Viewsfunction(doc) { if (doc.title) { emit(doc.title); } }

MapReduce consists of Map and Reduce stepswhich can be distributed in a way that takesadvantage of the multiple processor coresfound in modern hardware.

Create a Database in Futon

Create a Database with cURL$ curl -iX PUT http://localhost:5984/booksHTTP/1.1 201 CreatedLocation: http://localhost:5984/books{"ok":true}

First, some sample documents

The First Document(978-0-596-15589-6.json){ "_id":"978-0-596-15589-6", "title":"CouchDB: The De nitive Guide", "subtitle":"Time to Relax", "authors":[ "J. Chris Anderson", "Jan Lehnardt", "Noah Slater" ], "publisher":"OReilly Media", "formats":[ "Print", "Ebook", "Safari Books Online" ], "released":"2010-01-19", "pages":272}

POST the First Document$ curl -iX POST http://localhost:5984/books-H "Content-Type: application/json"-d @978-0-596-15589-6.jsonHTTP/1.1 201 CreatedLocation: http://localhost:5984/books/978-0-596-15589-6{ "ok":true, "id":"978-0-596-15589-6", "rev":"1-8c1d735475a0401d984c4aﬀ2fe796a5"}

The Second Document(978-0-596-52926-0.json){ "_id":"978-0-596-52926-0", "title":"RESTful Web Services", "subtitle":"Web services for the real world", "authors":[ "Leonard Richardson", "Sam Ruby" ], "publisher":"OReilly Media", "formats":[ "Print", "Ebook", "Safari Books Online" ], "released":"2007-05-08", "pages":448}

POST the Second Document$ curl -iX POST http://localhost:5984/books-H "Content-Type: application/json"-d @978-0-596-52926-0.jsonHTTP/1.1 201 CreatedLocation: http://localhost:5984/books/978-0-596-52926-0{ "ok":true, "id":"978-0-596-52926-0", "rev":"1-12538b07773519133043157220a63d8b"}

The Third Document(978-1-565-92580-9.json){ "_id":"978-1-565-92580-9", "title":"DocBook: The De nitive Guide", "authors":[ "Norman Walsh", "Leonard Muellner" ], "publisher":"OReilly Media", "formats":[ "Print" ], "released":"1999-10-28", "pages":648}

POST the Third Document$ curl -iX POST http://localhost:5984/books-H "Content-Type: application/json"-d @978-1-565-92580-9.jsonHTTP/1.1 201 CreatedLocation: http://localhost:5984/books/978-1-565-92580-9{ "ok":true, "id":"978-1-565-92580-9", "rev":"1-b945cb4799a1ccdd1689eae0e44124f1"}

The Fourth Document(978-0-596-80579-1.json){ "_id":"978-0-596-80579-1", "title":"Building iPhone Apps with HTML, CSS, and JavaScript", "subtitle":"Making App Store Apps Without Objective-C or Cocoa", "authors":[ "Jonathan Stark" ], "publisher":"OReilly Media", "formats":[ "Print", "Ebook", "Safari Books Online" ], "released":"2010-01-08", "pages":192}

POST the Fourth Document$ curl -iX POST http://localhost:5984/books-H "Content-Type: application/json"-d @978-0-596-80579-1.jsonHTTP/1.1 201 CreatedLocation: http://localhost:5984/books/978-0-596-80579-1{ "ok":true, "id":"978-0-596-80579-1", "rev":"1-09ce09fef75068834da99957c7b14cf2"}

…or Just Replicate$ curl -iX POST http://localhost:5984/_replicate-H "Content-Type: application/json"-d { "source": "https://oscon-tutorial.iriscouch.com/books", "target": "books"}HTTP/1.1 200 OK{ "ok":true, //…}

GET All Documents$ curl -iX GET http://localhost:5984/books/_all_docsHTTP/1.1 200 OK{ // … "rows": [ { "id": "978-0-596-15589-6", // … }, { "id": "978-0-596-52926-0", // … }, { "id": "978-0-596-80579-1", // … }, { "id": "978-1-565-92580-9", // … } ]}

Map and Reduce are written as JavaScriptfunctions that are de ned within views.

emit(key, value)Accepts two arguments—both are optional and default to nullFirst argument is the key to be indexedSecond argument is the value to be associated with the keyArbitrary JSON values are allowed for both the key and valueThe ID of the document being mapped is implicitly included with eachcall to the emit function

One-To-One Mapping(On Emit Per Document)

Map Book Titlesfunction(doc) { // JSON object representing a doc to be mapped if (doc.title) { // make sure this doc has a title emit(doc.title); // emit the doc’s title as the key }}

Temporary views can be used duringdevelopment but should be savedpermanently to design documentsfor production.

Titles View(titles_view.json){ "map": "function(doc) { if (doc.title) { emit(doc.title); } }"}

POST Titles Temporary View$ curl -iX POST http://localhost:5984/books/_temp_view-H "Content-Type: application/json"-d @titles_view.jsonHTTP/1.1 200 OK{ "total_rows": 4, "oﬀset": 0, "rows": [ // … ]}

“Titles” View Rows key id value"Building iPhone Apps with "978-0-596-80579-1" nullHTML, CSS, and JavaScript""CouchDB: The Deﬁnitive "978-0-596-15589-6" null Guide""DocBook: The Deﬁnitive "978-1-565-92580-9" null Guide" "RESTful Web Services" "978-0-596-52926-0" null

“Titles” View in Futon

One-To-Many Mapping(Multiple Emits Per Document)

Map Book Formatsfunction(doc) { // JSON object representing a doc to be mapped if (doc.formats) { // make sure this doc has a formats eld for (var i in doc.formats) { emit(doc.formats[i]); // emit each format as the key } }}

Formats View(formats_view.json){ "map": "function(doc) { if (doc.formats) { for (var i in doc.formats){ emit(doc.formats[i]); } } }"}

POST FormatsTemporary View$ curl -iX POST http://localhost:5984/books/_temp_view-H "Content-Type: application/json"-d @formats_view.jsonHTTP/1.1 200 OK{ "total_rows": 10, "oﬀset": 0, "rows": [ // … ]}

“Formats” View Rows key id value "Ebook" "978-0-596-15589-6" null "Ebook" "978-0-596-52926-0" null "Ebook" "978-0-596-80579-1" null "Print" "978-0-596-15589-6" null "Print" "978-0-596-52926-0" null "Print" "978-0-596-80579-1" null "Print" "978-1-565-92580-9" null"Safari Books Online" "978-0-596-15589-6" null"Safari Books Online" "978-0-596-52926-0" null"Safari Books Online" "978-0-596-80579-1" null

“Formats” View in Futon

Map Book Authorsfunction(doc) { // JSON object representing a doc to be mapped if (doc.authors) { // make sure this doc has an authors eld for (var i in doc.authors) { emit(doc.authors[i]); // emit each author as the key } }}

Authors View(authors_view.json){ "map": "function(doc) { if (doc.authors) { for (var i in doc.authors){ emit(doc.authors[i]); } } }"}

POST AuthorsTemporary View$ curl -iX POST http://localhost:5984/books/_temp_view-H "Content-Type: application/json"-d @authors_view.jsonHTTP/1.1 200 OK{ "total_rows": 8, "oﬀset": 0, "rows": [ // … ]}

“Authors” View Rows key id value "J. Chris Anderson" "978-0-596-15589-6" null "Jan Lehnardt" "978-0-596-15589-6" null "Jonathan Stark" "978-0-596-80579-1" null "Leonard Muellner" "978-1-565-92580-9" null"Leonard Richardson" "978-0-596-52926-0" null "Noah Slater" "978-0-596-15589-6" null "Norman Walsh" "978-1-565-92580-9" null "Sam Ruby" "978-0-596-52926-0" null

“Authors” View in Futon

When querying a view, the key and id elds can be used to select a row or rangeof rows, and to group. Rows can optionallybe grouped by key elds.

Reduce

Built-in Reduce Functions Function Output_count Returns the number of mapped values in the set_sum Returns the sum of the set of mapped values Returns numerical statistics of the mapped values in the set_stats including the sum, count, min, and max

Updated Formats View(formats_view.json){ "map": "function(doc) { if (doc.formats) { for (var i in doc.formats){ emit(doc.formats[i]); } } }", "reduce": "_count"}

POST Updated FormatsTemporary View$ curl -iX POST http://localhost:5984/books/_temp_view-H "Content-Type: application/json"-d @formats_view.jsonHTTP/1.1 200 OK{ "rows": [ { "key": null, "value": 10 } ]}

Format Count, Not Grouped key value null 10

…in Futon

POST Updated FormatsTemporary View, Grouped$ curl -iX POST http://localhost:5984/books/_temp_view?group=true-H "Content-Type: application/json"-d @formats_view.jsonHTTP/1.1 200 OK{ "rows": [ // … ]}

Format Count, Grouped key value "Ebook" 3 "Print" 4 "Safari Books Online" 3

…in Futon

The _count function can count arbitrary values,including NULLs. However, the _sum and _statsfunctions require numbers as values.

Updated Book Formats Mapfunction(doc) { if (doc.formats) { for (var i in doc.formats) { emit(doc.formats[i], doc.pages); // now emit pages as value } }}

Updated Formats View(formats_view.json){ "map": "function(doc) { if (doc.formats) { for (var i in doc.formats){ emit(doc.formats[i], doc.pages); } } }", "reduce": "_sum"}

Sum of Pages by Format key value "Ebook" 912 "Print" 1560 "Safari Books Online" 912

…in Futon

Statssum, count, minimum, maximum,sum over all square roots

Updated Formats View(formats_view.json){ "map": "function(doc) { if (doc.formats) { for (var i in doc.formats){ emit(doc.formats[i], doc.pages); } } }", "reduce": "_stats"}

Stats of Pages by Format key value {"sum":912,"count":3,"min":192, "Ebook" "max":448,"sumsqr":311552} {"sum":1560,"count":4,"min":192, "Print" "max":648,"sumsqr":731456} {"sum":912,"count":3,"min":192,"Safari Books Online" "max":448,"sumsqr":311552}

…in Futon

Custom Reduce Functions

The built-in Reduce functions should serve yourneeds most of the time.

ParametersKeys: An array of mapped key and document IDs in the form of [key,id] where id isthe document ID.Values: An array of mapped values.Rereduce: Whether or not the Reduce function is being called recursively on itsown output.

Reduce Function Skeletonfunction(keys, values, rereduce) { }

Count Equivalentfunction(keys, values, rereduce) { if (rereduce) { return sum(values); } else { return values.length; }}

Sum Equivalentfunction(keys, values, rereduce) { return sum(values);}

MapReduce LimitationsFull-text indexing and ad hoc searching • couchdb-lucene https://github.com/rnewson/couchdb-lucene • ElasticSearch and CouchDB https://github.com/elasticsearch/elasticsearch/wiki/Couchdb-integrationGeospatial indexing and search (two dimensional) • GeoCouch https://github.com/couchbase/geocouch • Geohash (e.g. c20g0b2t2882) http://geohash.org/

Querying Views

You can query for all rows, a single contiguousrange of rows, or even rows matching aspeci ed key.

Map Book Releasesfunction(doc) { if (doc.released) { emit(doc.released.split("-"), doc.pages); }}

Save the “Releases”View in Futon

Default Design Document(default_design.json){ "_id": "_design/default", "language": "javascript", "views": { "releases": { "map": "function(doc) { if (doc.released) { emit(doc.released.split(-),doc.pages); } }", "reduce": "_stats" } }}

Save the “Default”Design Document with cURL$ curl -iX POST http://localhost:5984/books-H "Content-Type: application/json"-d @default_design.jsonHTTP/1.1 201 CreatedLocation: http://localhost:5984/books/_design/default{ "ok": true, "id": "_design/default", "rev": "1-1776c504d0927e80e515acb952327797"}

Exact Grouping

“Releases”

“Releases”$ curl -iX GET http://localhost:5984/books/_design/default/_view/releases?group=trueHTTP/1.1 200 OK{ "rows": [ {"key": ["1999","10","28"], /* … */ }, {"key": ["2007","05","08"], /* … */ }, {"key": ["2010","01","08"], /* … */ }, {"key": ["2010","01","19"], /* … */ } ]}

“Releases” key value {"sum":648,"count":1,"min":648, ["1999","10","28"] "max":648,"sumsqr":419904} {"sum":448,"count":1,"min":448, ["2007","05","08"] "max":448,"sumsqr":200704} {"sum":192,"count":1,"min":192, ["2010","01","08"] "max":192,"sumsqr":36864} {"sum":272,"count":1,"min":272, ["2010","01","19"] "max":272,"sumsqr":73984}

Group Levels

“Releases”, Level 1 Grouping

“Releases”, Level 1 Grouping$ curl -iX GET http://localhost:5984/books/_design/default/_view/releases?group=true&group_level=1HTTP/1.1 200 OK{ "rows": [ {"key": ["1999"], /* … */ }, {"key": ["2007"], /* … */ }, {"key": ["2010"], /* … */ } ]}

“Releases”, Level 1 Grouping key value {"sum":648,"count":1,"min":648, ["1999"] "max":648,"sumsqr":419904} {"sum":448,"count":1,"min":448, ["2007"] "max":448,"sumsqr":200704} {"sum":464,"count":2,"min":192, ["2010"] "max":272,"sumsqr":110848}

“Releases”, Level 2 Grouping

“Releases”, Level 2 Grouping$ curl -iX GET http://localhost:5984/books/_design/default/_view/releases?group=true&group_level=2HTTP/1.1 200 OK{ "rows": [ {"key": ["1999","10"], /* … */ }, {"key": ["2007","05"], /* … */ }, {"key": ["2010","01"], /* … */ } ]}

“Releases”, Level 2 Grouping key value {"sum":648,"count":1,"min":648, ["1999","10"] "max":648,"sumsqr":419904} {"sum":448,"count":1,"min":448, ["2007","05"] "max":448,"sumsqr":200704} {"sum":464,"count":2,"min":192, ["2010","01"] "max":272,"sumsqr":110848}

Range Queries

By Key$ curl -iGX GET http://localhost:5984/books/_design/default/_view/releases-d reduce=false--data-urlencode key=["1999", "10", "28"]HTTP/1.1 200 OK{"total_rows":4,"oﬀset":0,"rows":[{"id":"978-1-565-92580-9","key":["1999","10","28"],"value":648}]}

By Start and End Keys$ curl -iGX GET http://localhost:5984/books/_design/default/_view/formats-d reduce=false--data-urlencode startkey="Ebook"--data-urlencode endkey="Print"HTTP/1.1 200 OK{"total_rows":10,"oﬀset":0,"rows":[{"id":"978-0-596-15589-6","key":"Ebook","value":272},{"id":"978-0-596-52926-0","key":"Ebook","value":448},{"id":"978-0-596-80579-1","key":"Ebook","value":192},{"id":"978-0-596-15589-6","key":"Print","value":272},{"id":"978-0-596-52926-0","key":"Print","value":448},{"id":"978-0-596-80579-1","key":"Print","value":192},{"id":"978-1-565-92580-9","key":"Print","value":648}]}

By Start and End Keysand Document IDs$ curl -iGX GET http://localhost:5984/books/_design/default/_view/formats-d reduce=false--data-urlencode startkey="Ebook"--data-urlencode startkey_docid=978-0-596-52926-0--data-urlencode endkey="Print"--data-urlencode endkey_docid=978-0-596-52926-0HTTP/1.1 200 OK{"total_rows":10,"oﬀset":1,"rows":[{"id":"978-0-596-52926-0","key":"Ebook","value":448},{"id":"978-0-596-80579-1","key":"Ebook","value":192},{"id":"978-0-596-15589-6","key":"Print","value":272},{"id":"978-0-596-52926-0","key":"Print","value":448}]}

Limiting, Skipping & Reversing Results

Limit$ curl -iGX GET http://localhost:5984/books/_design/default/_view/formats-d reduce=false-d limit=5HTTP/1.1 200 OK{"total_rows":10,"oﬀset":0,"rows":[{"id":"978-0-596-15589-6","key":"Ebook","value":272},{"id":"978-0-596-52926-0","key":"Ebook","value":448},{"id":"978-0-596-80579-1","key":"Ebook","value":192},{"id":"978-0-596-15589-6","key":"Print","value":272},{"id":"978-0-596-52926-0","key":"Print","value":448}]}

Skip$ curl -iGX GET http://localhost:5984/books/_design/default/_view/formats-d reduce=false-d limit=5-d skip=5HTTP/1.1 200 OK{"total_rows":10,"oﬀset":5,"rows":[{"id":"978-0-596-80579-1","key":"Print","value":192},{"id":"978-1-565-92580-9","key":"Print","value":648},{"id":"978-0-596-15589-6","key":"Safari Books Online","value":272},{"id":"978-0-596-52926-0","key":"Safari Books Online","value":448},{"id":"978-0-596-80579-1","key":"Safari Books Online","value":192}]}

The skip and limit parameters can be usedto implement pagination. When skipping alarger number of rows, it is more eﬃcient toset the skip parameter’s value to 1 and usethe previous page’s last key and documentID as the startkey and startkey_docid.

Reversing Output$ curl -iGX GET http://localhost:5984/books/_design/default/_view/titles-d reduce=false-d descending=trueHTTP/1.1 200 OK{"total_rows":4,"oﬀset":0,"rows":[{"id":"978-0-596-52926-0","key":"RESTful Web Services","value":448},{"id":"978-1-565-92580-9","key":"DocBook: The De nitive Guide","value":648},{"id":"978-0-596-15589-6","key":"CouchDB: The De nitive Guide","value":272},{"id":"978-0-596-80579-1","key":"Building iPhone Apps with HTML, CSS, andJavaScript","value":192}]}

When reversing output, the values for startkeyand endkey must be swapped as well as thevalues for startkey_docid and endkey_docid, ifspeci ed. This is because output is reversedbefore rows are ltered.

Including Documents

include_docs Parameter$ curl -iGX GET http://localhost:5984/books/_design/default/_view/titles-d reduce=false-d include_docs=trueHTTP/1.1 200 OK{ // … "rows": [ { // … "doc": { "_id": "978-0-596-80579-1", "_rev": "1-09ce09fef75068834da99957c7b14cf2", "title": "Building iPhone Apps with HTML, CSS, and JavaScript", // …

Emit Document as Valuefunction(doc) { if (doc.title) { emit(doc.title, doc); }}

Get Documents Separately

Get the View$ curl -iGX GET http://localhost:5984/books/_design/default/_view/titles-d reduce=false-d limit=1HTTP/1.1 200 OK{"total_rows":4,"oﬀset":0,"rows":[{"id":"978-0-596-80579-1","key":"Building iPhone Apps with HTML, CSS, andJavaScript","value":192}]}

Get a DocumentFrom the View$ curl -iX GET http://localhost:5984/books/978-0-596-80579-1HTTP/1.1 200 OK{ "_id": "978-0-596-80579-1", "_rev": "1-09ce09fef75068834da99957c7b14cf2", "title": "Building iPhone Apps with HTML, CSS, and JavaScript", "subtitle": "Making App Store Apps Without Objective-C or Cocoa", "authors": ["Jonathan Stark"], "publisher": "OReilly Media", "formats": ["Print", "Ebook", "Safari Books Online"], "released":" 2010-01-08", "pages":192}

Get a Cached Document From theView$ curl -iX GET http://localhost:5984/books/978-0-596-80579-1-H If-None-Match: "1-09ce09fef75068834da99957c7b14cf2"HTTP/1.1 304 Not Modi edEtag: "1-09ce09fef75068834da99957c7b14cf2"Content-Length: 0

Scaling

Load BalancingSend POST, PUT, and DELETE requests to a write-only master nodeSetup continuous replication from the master node to multiple read-only nodesLoad balance GET, HEAD, and OPTIONS requests amongstread-only nodes • Apache HTTP Server (mod_proxy) • nginx • HAProxy • Varnish • Squid

Load Balancing

Clustering(Partitioning/Sharding)BigCouchhttps://github.com/cloudant/bigcouch • Clusters modeled after Amazon’s Dynamo approachLoungehttp://tilgovi.github.com/couchdb-lounge/ • Proxy, partitioning, and shardingPillowhttps://github.com/khellan/Pillow • “…a combined router and rereducer for CouchDB.”

Server Con guration

local.ini[couchdb];max_document_size = 4294967296 ; bytes[httpd];port = 5984;bind_address = 127.0.0.1; Uncomment next line to trigger basic-auth popup on unauthorized requests.;WWW-Authenticate = Basic realm="administrator";…

GET Con g$ curl -iX GET http://localhost:5984/_con gHTTP/1.1 200 OK{ "httpd_design_handlers": { /* … */ }, "uuids": { /* … */ }, "stats": { /* … */ }, "httpd_global_handlers": { /* … */ }, "attachments": { /* … */ }, "query_server_con g": { /* … */ }, // …}

GET Con g Section$ curl -iX GET http://localhost:5984/_con g/uuidsHTTP/1.1 200 OK{ "algorithm": "sequential"}

GET Con g Section Key$ curl -iX GET http://localhost:5984/_con g/uuids/algorithmHTTP/1.1 200 OK"sequential"

PUT Con g Section Key$ curl -iX PUT http://localhost:5984/_con g/uuids/algorithm-H "Content-Type: application/json"-d "utc_random"HTTP/1.1 200 OK"sequential"

Con guration settings can also be edited fromwithin Futon

Security

Admin PartyDefault settings after install are that everyone is adminBy default, CouchDB will only listen on the loopback addressYou’ll want to change the default settings if you let CouchDB listen on a public IPaddress!

SSL Con guration[daemons];…httpsd = {couch_httpd, start_link, [https]};…[ssl]cert_ le = /full/path/to/server_cert.pemkey_ le = /full/path/to/server_key.pem;…

Connect to Port 6984 for SSL$ curl -iX GET https://localhost:6984/HTTP/1.1 200 OKServer: CouchDB/1.1.0 (Erlang OTP/R14B){ "couchdb": "Welcome", "version": "1.1.0"}

Users Database

GET All Users$ curl -iX GET http://localhost:5984/users/_all_docsHTTP/1.1 200 OK{ // … "rows": [ // … ]}

GET a UUID forPassword Salting$ curl -iX GET http://localhost:5984/_uuidsHTTP/1.1 200 OK{ "uuids": [ "f0f1353b46013ec80533346f7f000c8b" ]}

Concatenate and Hash Passwordand Salt$ echo -n "supersecurepasswordf0f1353b46013ec80533346f7f000c8b" | opensslsha1(stdin)= 3cfa095ef255121dcc2fe223b907246a6d1591e5

Bob (bob.json){ "_id": "org.couchdb.user:bob", "type": "user", "name": "bob", "roles": [], "password_sha": "3cfa095ef255121dcc2fe223b907246a6d1591e5", "salt": "f0f1353b46013ec80533346f7f000c8b"}

POST a User$ curl -iX POST http://localhost:5984/_users-H "Content-Type: application/json"-d @bob.jsonHTTP/1.1 201 CreatedLocation: http://localhost:5984/_users/org.couchdb.user:bob{ "ok": true, "id": "org.couchdb.user:bob", "rev": "1-fbca8d4c136c2c322e88249b78eeafe6"}

Create User Accountin Futon

Authentication

Basic Authentication$ curl -iX GET http://bob:supersecurepassword@localhost:5984/HTTP/1.1 200 OKServer: CouchDB/1.0.2 (Erlang OTP/R14B){ "couchdb": "Welcome", "version": "1.0.2"}

Cookie Authentication$ curl -iX POST http://localhost:5984/_session-d "name=bob"-d "password=supersecurepassword"HTTP/1.1 200 OKSet-Cookie: AuthSession=Ym9iOjRERjExRjk3On4WFg1s…; Version=1; Path=/;HttpOnly{ "ok": true, "name": "bob", "roles": []}

Cookie Authentication$ curl -iX GET http://localhost:5984/-b "AuthSession=Ym9iOjRERjExRjk3On4WFg1s…"HTTP/1.1 200 OKServer: CouchDB/1.0.2 (Erlang OTP/R14B){ "couchdb": "Welcome", "version": "1.0.2"}

Logging in Through Futon

CouchDB also supports OAuth

Authorization

Server Level Authorization

Server AdminCan: • Create databases (PUT /db) • Delete databases (DELETE /db) • Create design documents (PUT /db/_design/foo) • Update design documents (PUT /db/_design/foo?rev=1-3BA) • Delete design documents (DELETE /db/_design/foo?rev=2-C8F) • Trigger compaction (POST /db/_compact) • Trigger view cleanup (POST /db/_view_cleanup) • View active tasks (GET /_active_tasks) • Restart the server (POST /_restart) • Read con g settings (GET /_con g) • Update con g settings (PUT /_con g)

Create a Server Admin$ curl -iX PUT http://localhost:5984/_con g/admins/bob-d "supersecurepassword"HTTP/1.1 200 OK""

Delete Server Admin$ curl -iX DELETE http://bob:supersecurepassword@localhost:5984/_con g/admins/bobHTTP/1.1 200 OK"-hashed-54520b20f1b3a5c24d36c051aa1268de7543144c,20edd87862631adab51442b9b10e0a65"

Create Server Adminsfrom Futon

Database Level Authorization

Database AdminCan: • Create design documents (PUT /db/_design/foo) • Update design documents (PUT /db/_design/foo?rev=1-3BA) • Delete design documents (DELETE /db/_design/foo?rev=2-C8F) • Add database admins and database readers • Remove database admins and database readers • Set database revision limit (PUT /db/_revs_limit) • Execute temporary views (POST /db/_temp_view) • Trigger compaction (POST /db/_compact) • Trigger view cleanup (POST /db/_view_cleanup)

Database ReaderCan: • Read all types of documents • Except for design documents: • Create documents (POST /mydb/mydoc) • Update documents (PUT /mydb/mydoc?rev=1-3BA) • Delete documents (DELETE /mydb/mydoc?rev=2-C8F)

Database Security Object(security.json){ "admins" : { "names": ["bob"], "roles": [] }, "readers" : { "names": ["joe"], "roles": [] }}

PUT Security Object$ curl -iX PUT http://localhost:5984/mydb/_security-H "Content-Type: application/json"-d @security.jsonHTTP/1.1 200 OK{ "ok":true}

Database Security in Futon

Validation FunctionsDe ned within design documentsFunction passed: • New document • Old document • User context: • Database name (db) • User name (name) • Roles (roles)

Read-Only for Non-Adminsfunction(newDoc, oldDoc, userCtx) { if (-1 === userCtx.roles.indexOf(_admin)) { throw({forbidden: Only admins may update documents.}); }}

Auth Design Document{ "_id": "_design/auth", "language": "javascript", "validate_doc_update": "function(newDoc, oldDoc, userCtx) { if (-1 ===userCtx.roles.indexOf(_admin)) { throw({forbidden: Only admins may updatedocuments.}); } }"}

What else?

Con ict ResolutionReplication will eventually lead to con icting revisionsBoth con icted revisions will be preserved, but one “wins”Con icted documents will get a special "_con icts" agCreate a view to nd con icted documents:if (doc._con icts) { for (var i in doc._con icts) { emit(doc._con icts[i]); } }Have end-user or application logic handle a mergeClear the "_con icts" ag with a document update

Changes Feed$ curl -iX GET http://localhost:5984/books/_changesHTTP/1.1 200 OK{"results":[{"seq":1,"id":"978-0-596-52926-0","changes":[{"rev":"1-12538b0…"}]},{"seq":2,"id":"978-1-565-92580-9","changes":[{"rev":"1-b945cb4…"}]},{"seq":3,"id":"978-0-596-80579-1","changes":[{"rev":"1-09ce09fe…"}]},{"seq":4,"id":"978-0-596-15589-6","changes":[{"rev":"1-8c1d7354…"}]}],"last_seq":4}

Partial Replicas:Filter FunctionsFilter functions are de ned within design documents under the " lters" key.Example of an "authors" lter function:function(doc, req) { return author == doc.collection }The lter function (and its containing design document) can then be referencedduring replication:" lter":"default/authors"Filter functions can be parameterized:function(doc, req) { return req.query.collection == doc.collection }Parameters values can be speci ed during replication:"query_params": { "collection":"publisher" }

Partial Replicas:Specifying IDsProvide an array of IDs when replicating:"doc_ids": [ "be231efa93502b3286aae0ed7b000aed", "be231efa93502b3286aae0ed7b001a6a", "be231efa93502b3286aae0ed7b002916", "be231efa93502b3286aae0ed7b003458", "be231efa93502b3286aae0ed7b00375e"]

Show Functionsfunction(doc, req) { return { "body": "<h1>" + doc.title + "</h1>", "headers": { "Content-Type": "text/html" } };}GET /db/_design/<design_name>/_show/<show_name>/<doc_id>

List Functionsfunction(head, req) { var row; start({ "headers": { "Content-Type": "text/html" } }); while (row = getRow()) { send("<h1>" + row.key + "</h1>"); }}GET /db/_design/<design_name>/_list/<list_name>/<view_name>

Rewrites

Rewriting to an Attachment{ "from": "", "to": "index.html", "method": "GET", "query": {}}From:GET /db/_design/app/_rewriteTo:GET /db/_design/app/index.html

Rewriting to a List{ "from": "books", "to": "_list/books/all", "query": { "limit": 10 }}From:GET /db/_design/app/_rewrite/booksTo:GET /db/_design/app/_list/books/all?limit=10

Rewriting to a Show{ "from": "books/:id", "to": "_show/books/:id", "query": {}}From:GET /db/_design/app/_rewrite/books/978-0-596-15589-6To:GET /db/_design/app/_show/books/978-0-596-15589-6

Virtual Hosts

“vhosts” Con g Section{ "example.com": "/db/_design/app/_rewrite", "www.example.com": "/db/_design/app/_rewrite"}All requests with a Host HTTP header of example.com or www.example.com willnow be handled by the rewrite rules de ned in the app design document.

Maintenance

Database Compaction$ curl -iX POST http://localhost:5984/db/_compact-H "Content-Type: application/json"HTTP/1.1 202 Accepted{ "ok": true}

View Compaction$ curl -iX POST http://localhost:5984/db/_compact/default-H "Content-Type: application/json"HTTP/1.1 202 Accepted{ "ok": true}

View Cleanup$ curl -iX POST http://localhost:5984/db/_view_cleanup-H "Content-Type: application/json"HTTP/1.1 202 Accepted{ "ok": true}

Compact & Cleanup from Futon

CouchApps

CouchAppApplications built using CouchDB, JavaScript and HTML5CouchDB is a database, web server and application serverNo more middle-tier between your data and presentation layers: • CouchDB becomes both the data and application tiers • User agent becomes the presentation tierData and application can be replicated togetherCouchApp Tool:http://couchapp.org/

Mobile Couchbase for Android & iOShttp://www.couchbase.com/products-and-services/mobile-couchbase

Hosting is available through Iris Couch orCloudant.

CouchDB ResourcesCouchDB: The De nitive Guide CouchDB Wikiby J. Chris Anderson, Jan Lehnardt, and http://wiki.apache.org/couchdb/Noah Slater (O’Reilly)978-0-596-15589-6 Beginning CouchDB by Joe Lennon (Apress)Writing and Querying MapReduce Views in 978-1-430-27237-3CouchDBby Bradley Holt (O’Reilly)978-1-449-30312-9Scaling CouchDBby Bradley Holt (O’Reilly)063-6-920-01840-7

Questions?

http://bradley-holt.com	1427
http://blog.nosqlfan.com	273
http://architects.dzone.com	201
http://feed.feedsky.com	49
http://lanyrd.com	16
http://xianguo.com	10
http://feeds.feedburner.com	9
http://zhuaxia.com	4
http://reader.youdao.com	3
http://www.zhuaxia.com	3
http://dzone.com	2
http://safe.tumblr.com	1
http://paper.li	1
http://www.netvibes.com	1
http://cliveboulton.com	1
http://webcache.googleusercontent.com	1

OSCON 2011 Learning CouchDB

by Bradley Holt

Accessibility

Categories

Upload Details

Usage Rights

16 Embeds 2,002

Statistics

OSCON 2011 Learning CouchDB Presentation Transcript