Couchdb: No SQL? No driver? No problem

* CouchDB No SQL? No Driver? No problem. Angel Pizarro angel@upenn.edu * www.bauwel-movement.co.uk/ sculpture.php

About Me Me: CBIL alumni! Work in mass spec proteomics Lots of data in lots of formats in bioinformatics Ruby for programming and Ruby on Rails for Web apps But that doesn’t matter for CouchDB! Interested in CouchDB for AWS deployment

Overview Talk about Key-Value stores Introduce some general theory and concepts CouchDB speciﬁcs Example problem More CouchDB speciﬁcs Questions?

Key-Value Databases Datastore of values indexed by keys (duh!) Hash or B-Tree index for keys Cassandra Hash is FAST, but only allows single-value lookups B-Tree is slower, but allows range queries Horizontally scalable - via key partitioning

The CAP theory : applies when business logic is separate from storage Consistency vs. Availability vs. Partition tolerance RDBMS = enforced consistency PAXOS = quorum consistency CouchDB (and others) = eventual consistency and horizontally scalable http://www.julianbrowne.com/article/viewer/brewers-cap-theorem

CouchDB

CouchDB Document Oriented Database JSON documents

CouchDB Document Oriented Database JSON documents HTTP protocol using REST operations No direct native language drivers * Javascript is the lingua franca * Hovercraft: http://github.com/jchris/hovercraft/

CouchDB Document Oriented Database JSON documents HTTP protocol using REST operations No direct native language drivers * Javascript is the lingua franca ACID & MVCC guarantees on a per- document basis * Hovercraft: http://github.com/jchris/hovercraft/

CouchDB Document Oriented Database JSON documents HTTP protocol using REST operations No direct native language drivers * Javascript is the lingua franca ACID & MVCC guarantees on a per- document basis Map-Reduce indexing and views * Hovercraft: http://github.com/jchris/hovercraft/

CouchDB Document Oriented Database JSON documents HTTP protocol using REST operations No direct native language drivers * Javascript is the lingua franca ACID & MVCC guarantees on a per- document basis Map-Reduce indexing and views Back-ups and replication are easy-peasy * Hovercraft: http://github.com/jchris/hovercraft/

Javascript Object Notation

Javascript Object Notation * * http://www.json.org

Example JSON { “name”: “J. Doe”, “friends”: 0, “traits”: [“nice”, “outgoing”] }

REST Representational State Transfer

REST Representational State Transfer Clients-Server separation with uniform interface (HTTP)

REST Representational State Transfer Clients-Server separation with uniform interface (HTTP) Load-balancing, caching, authorization & authentication, proxies

REST Representational State Transfer Clients-Server separation with uniform interface (HTTP) Load-balancing, caching, authorization & authentication, proxies Stateless - client is responsible for creating a self- sufﬁcient request

REST Representational State Transfer Clients-Server separation with uniform interface (HTTP) Load-balancing, caching, authorization & authentication, proxies Stateless - client is responsible for creating a self- sufﬁcient request Resources are cacheable - servers must mark non-cacheable resources as such

REST Representational State Transfer Clients-Server separation with uniform interface (HTTP) Load-balancing, caching, authorization & authentication, proxies Stateless - client is responsible for creating a self- sufﬁcient request Resources are cacheable - servers must mark non-cacheable resources as such Only 5 HTTP verbs

REST Representational State Transfer Clients-Server separation with uniform interface (HTTP) Load-balancing, caching, authorization & authentication, proxies Stateless - client is responsible for creating a self- sufﬁcient request Resources are cacheable - servers must mark non-cacheable resources as such Only 5 HTTP verbs GET, PUT, POST, DELETE, HEAD

CouchDB REST/CRUD GET read PUT create or update DELETE delete something POST bulk operations

CouchDB passes the ACID test Each document is completely self-sufﬁcient Each document has a version number An update operation writes a complete new copy of the the record and is assigned the new version number Append-only ﬁle structure allows the write to occur while still serving read requests

MVCC RDBMS CouchDB Multi-Version Concurrency Control RDBMS enforces consistency using read/write locks Instead of locks, CouchDB just serve up old data Multi-document (mutli-row) transactional semantics must be handled by the application

Database API Create a DB: $ curl -X PUT http://127.0.0.1:5984/friendbook {"ok":true}

Database API Create a DB: Protocol $ curl -X PUT http://127.0.0.1:5984/friendbook {"ok":true}

Database API Create a DB: CouchDB server $ curl -X PUT http://127.0.0.1:5984/friendbook {"ok":true}

Database API Create a DB: DB name $ curl -X PUT http://127.0.0.1:5984/friendbook {"ok":true}

Database API Create a DB: $ curl -X PUT http://127.0.0.1:5984/friendbook {"ok":true} Try it Again: {"error":"db_exists"}

Database API Create a DB: $ curl -X PUT http://127.0.0.1:5984/friendbook {"ok":true} Try it Again: {"error":"db_exists"} Not recoverable! Delete a DB: $ curl -X DELETE http://localhost:5984/friendbook {"ok":true}

Inserting a document All insert require that you give a unique ID. You can request one from CouchDB: $ curl -X GET http://localhost:5984/_uuids {"uuids":["d1dde0996a4db7c1ebc78fb89c01b9e6"]}

Inserting a document All insert require that you give a unique ID. You can request one from CouchDB: $ curl -X GET http://localhost:5984/_uuids {"uuids":["d1dde0996a4db7c1ebc78fb89c01b9e6"]} We’ll just give one: $ curl -X PUT http://localhost:5984/friendbook/j_doe -d @j_doe.json {"ok":true, "id":"j_doe", "rev":"1-062af1c4ac73287b7e07396c86243432"}

Inserting a document All insert require that you give a unique ID. You can request one from CouchDB: $ curl -X GET http://localhost:5984/_uuids {"uuids":["d1dde0996a4db7c1ebc78fb89c01b9e6"]} We’ll just give one: $ curl -X PUT http://localhost:5984/friendbook/j_doe -d @j_doe.json Read a JSON ﬁle {"ok":true, "id":"j_doe", "rev":"1-062af1c4ac73287b7e07396c86243432"}

Full JSON document Before { "name": "J. Doe", "friends": 0 } After { "_id": "j_doe", "_rev": "1-062af1c4ac73287b7e07396c86243432", "name": "J. Doe", "friends": 0 }

Updating a document $ curl -X PUT http://localhost:5984/friendbook/j_doe -d '{"name": "J. Doe", "friends": 1 }' {"error":"conﬂict","reason":"Document update conﬂict."}

Updating a document $ curl -X PUT http://localhost:5984/friendbook/j_doe -d '{"name": "J. Doe", "friends": 1 }' {"error":"conﬂict","reason":"Document update conﬂict."} Must give _rev (revision number) for updates! revised.json { "_rev":"1-062af1c4ac73287b7e07396c86243432", "name":"J. Doe", "friends": 1 } $ curl -X PUT http://localhost:5984/friendbook/j_doe -d @revised.json {"ok":true,"id":"j_doe","rev":"2-0629239b53a8d146a3a3c4c63e 2dbfd0"}

Deleting a document $ curl -X DELETE http://localhost:5984/friendbook/j_doe {"error":"conﬂict","reason":"Document update conﬂict."} Must give revision number for deletes! $ curl -X DELETE http://localhost:5984/friendbook/j_doe? rev=2-0629239b53a8d146a3a3c4c63e2dbfd0 {"ok":true,"id":"j_doe", "rev":"3-57673a4b7b662bb916cc374a92318c6b"} Returns a revision number for the delete $ curl -X GET http://localhost:5984/friendbook/j_doe {"error":"not_found","reason":"deleted"}

Bulk operation POST /database/_bulk_docs with a JSON document containing all of the new or updated documents. // documents to bulk upload { "docs": [ {"_id": "0", "integer": 0, "string": "0"}, {"_id": "1", "integer": 1, "string": "1"}, {"_id": "2", "integer": 2, "string": "2"} ] // reply from CouchDB } [ {"id":"0","rev":"1-62657917"}, {"id":"1","rev":"1-2089673485"}, {"id":"2","rev":"1-2063452834"} ]

GOTCHA’s! Version storage is not guaranteed! Do not use this as a VCS! POST to /db/_compact deletes all older vesions To “roll back a transaction” you must: Retrieve all related records, cache these Insert any updates to records. On failure, use the returned revision numbers to re-insert the older record as a new one

Our Example Problem

Our Example Problem Hello world? Blog? Twitter clone?

Our Example Problem Hello world? Blog? Twitter clone? Let’s store all human proteins instead

Our Example Problem Hello world? Blog? Twitter clone? Let’s store all human proteins instead LOCUS YP_003024029 227 aa linear PRI 09-JUL-2009 DEFINITION cytochrome c oxidase subunit II [Homo sapiens]. ACCESSION YP_003024029 VERSION YP_003024029.1 GI:251831110 DBLINK Project:30353 DBSOURCE REFSEQ: accession NC_012920.1 KEYWORDS . SOURCE mitochondrion Homo sapiens (human) ORGANISM Homo sapiens Eukaryota; Metazoa; Chordata; Craniata; Vertebrata; Euteleostomi; Mammalia; Eutheria; Euarchontoglires; Primates; Haplorrhini; Catarrhini; Hominidae; Homo.

Our Example Problem Hello world? Blog? Twitter clone? Let’s store all human proteins instead LOCUS YP_003024029 227 aa linear PRI 09-JUL-2009 DEFINITION cytochrome c oxidase subunit II [Homo sapiens]. ACCESSION YP_003024029 VERSION YP_003024029.1 GI:251831110 DBLINK Project:30353 FEATURES DBSOURCE REFSEQ: accession NC_012920.1 Location/Qualiﬁers KEYWORDS . source 1..227 SOURCE /organism="Homo sapiens" mitochondrion Homo sapiens (human) ORGANISM Homo sapiens /organelle="mitochondrion" /isolation_source="caucasian" Eukaryota; Metazoa; Chordata; Craniata; Vertebrata; Euteleostomi; /db_xref="taxon:9606" Mammalia; Eutheria; Euarchontoglires; Primates; Haplorrhini; Catarrhini; Hominidae; Homo./tissue_type="placenta" /country="United Kingdom: Great Britain" /note="this is the rCRS" Protein 1..227 /product="cytochrome c oxidase subunit II" /calculated_mol_wt=25434 http://www.ncbi.nlm.nih.gov/

Futon : A Couchapp

Futon : A Couchapp This one is going to be a bit tougher

Design Documents

Design Documents The key to using CouchDB as more than a key-value store

Design Documents The key to using CouchDB as more than a key-value store Just another JSON document, but contain javascript functions that CouchDB treats as application code Functions are executed within CouchDB

Design Documents The key to using CouchDB as more than a key-value store Just another JSON document, but contain javascript functions that CouchDB treats as application code Functions are executed within CouchDB Contain sections for map-reduce views, data validation, alternate formatting, ... Also library code & data structures speciﬁc to the design document

Soy Map!

Soy Map! Views use a Map-Reduce model for indexing and deﬁning “virtual” documents Fits well with assumptions of self-sufﬁcient documents and eventual consistency

Soy Map! Views use a Map-Reduce model for indexing and defining “virtual” documents Fits well with assumptions of self-sufficient documents and eventual consistency Map function is applied to all documents in the database Emits (parts of) documents that pass mustard Indexing is incremental after an initial definition You can choose to defer an index update for insert speed

Map function example

Complex Map

View Result

GET by the indexed key

GET by the indexed key GET /refseq_human/_design/gb/_view/dbXref?key="GeneID:10" {"total_rows":7,"offset":2,"rows":[ {"id":"NP_000006", "key":"GeneID:10", "value":"NP_000006"} ]}

Reduce functions Optional and used in concert with a speciﬁc map function Great for summarizing or collating numerical data points E.g. counts, number of over time X, average load, probability of conversion Not really applicable to our example, so we’ll not cover it today

Show me the ... HTML? JSON is great, but what about, ya know, something useful? You can make a separate app to reformat the JSON OR you can use the “shows” section of a _design document. Rich formating possible with functions, templates, and special include macros

FASTA format “shows” : { “fasta” : “function(doc, req) { return ‘>’ + doc._id + ‘n’ + doc.seq; }”, ...

FASTA format “shows” : { “fasta” : “function(doc, req) { return ‘>’ + doc._id + ‘n’ + doc.seq; }”, ... GET /refseq_human/_design/gb/_show/fasta/NP_000006 >NP_000006 MDIEAYFERIGYKNSRNKLDLETLTDILEHQIRAVPFENLNMHCGQAMELGLEAIFDHIVRR NRGGWCLQVNQLLYWALTTIGFQTTMLGGYFYIPPVNKYSTGMVHLLLQVTIDGRNYIV DAGSGSSSQMWQPLELISGKDQPQVPCIFCLTEERGIWYLDQIRREQYITNKEFLNSHLLPK KKHQKIYLFTLEPRTIEDFESMNTYLQTSPTSSFITTSFCSLQTPEGVYCLVGFILTYRKFNYKD NTDLVEFKTLTEEEVEEVLRNIFKISLGRNLVPKPGDGSLTI

Backups & Replication

Backups & Replication Backup: simply copy the database ﬁle

Backups & Replication Backup: simply copy the database ﬁle Replicate: send a POST request with a source and target database

Backups & Replication Backup: simply copy the database ﬁle Replicate: send a POST request with a source and target database Source and target DB’s can either be local (just the db name) or remote (full URL)

Backups & Replication Backup: simply copy the database ﬁle Replicate: send a POST request with a source and target database Source and target DB’s can either be local (just the db name) or remote (full URL) “continous”: true option will register the target to the source’s _changes notiﬁcation API

Backups & Replication Backup: simply copy the database ﬁle Replicate: send a POST request with a source and target database Source and target DB’s can either be local (just the db name) or remote (full URL) “continous”: true option will register the target to the source’s _changes notiﬁcation API $ curl -X POST http://localhost:5984/_replicate -d '{"source":"db", "target":"db-replica", "continuous":true}'

Data normalization? Schema? Foreign Keys? Column Constraints?

Data normalization? Schema? Foreign Keys? Column Constraints? forgetaboutit Italian for “forget about it” … “or die”

Data normalization? Schema? Foreign Keys? Column Constraints? forgetaboutit Italian for “forget about it” … “or die” Denormalize “until it hurts”

Data normalization? Schema? Foreign Keys? Column Constraints? forgetaboutit Italian for “forget about it” … “or die” Denormalize “until it hurts” But there are validations are available

Data normalization? Schema? Foreign Keys? Column Constraints? forgetaboutit Italian for “forget about it” … “or die” Denormalize “until it hurts” But there are validations are available Validates a record on update with a JS function

Required Fields function(newDoc, oldDoc, userCtx) { function require(field, message) { message = message || "Document must have a " + field; if (!newDoc[field]) throw({forbidden : message}); }; if (newDoc.type == "blogPost") { require("title"); require("created_at"); require("body"); Convention alert! require("author"); } ... }

Thank You! Learn http://couchdb.apache.org/ http://books.couchdb.org/relax http://wiki.apache.org/couchdb/ Awesome posts by community http://planet.couchdb.org Development Libraries http://github.com/jchris/couchrest http://github.com/couchapp/couchapp

http://www.slideshare.net	36
http://nosql.mypopescu.com	17

Couchdb: No SQL? No driver? No problem

by delagoya on Oct 21, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

2 Embeds 53

Statistics

Couchdb: No SQL? No driver? No problem — Presentation Transcript