Couch Foo: CouchDB on rails

CouchDB and Ruby George Palmer

“Spending more time on the couch: Using CouchDB to avoid migrations, create ofﬂine applications and scale with greater ease”

What is CouchDB?

Database Landscape

Database Landscape MySQL

Database Landscape MySQL Postgres

Database Landscape MySQL Postgres SQLite

Database Landscape Relational Database MySQL Postgres SQLite

Database Landscape Relational Database MySQL Postgres ToykoCabinet SQLite

Database Landscape Relational Database MySQL Postgres ToykoCabinet SQLite Redis

Database Landscape Relational Database MySQL Postgres ToykoCabinet SQLite Redis Memcachedb

Database Landscape Relational Database Key-Value Databases MySQL Postgres ToykoCabinet SQLite Redis Memcachedb

Database Landscape Amazon SimpleDB Relational Database Key-Value Databases MySQL Postgres ToykoCabinet SQLite Redis Memcachedb

Database Landscape Amazon SimpleDB MongoDB Relational Database Key-Value Databases MySQL Postgres ToykoCabinet SQLite Redis Memcachedb

Database Landscape Amazon SimpleDB MongoDB CouchDB Relational Database Key-Value Databases MySQL Postgres ToykoCabinet SQLite Redis Memcachedb

Database Landscape Amazon SimpleDB MongoDB CouchDB Document Orientated Databases Relational Database Key-Value Databases MySQL Postgres ToykoCabinet SQLite Redis Memcachedb

Document Orientated Databases • Data is stored in documents • Like real life there’s no limit on how the information is stored • ie schema free • Documents stored and accessed by an identiﬁer

Stored as JSON { “_id”: “A312C72B”, “_rev”: “AB746C”, “name”: “Rails Underground”, “start-date”: “July 24th”, “tags”: [“ruby”, “rails”, “conference”] }

REST interface • Create HTTP POST or PUT /db/docid • Read HTTP GET /db/docid • Destroy HTTP DELETE /db/docid • Update HTTP PUT /db/docid

HTTP REST Beneﬁts • Load Balancing • Caching • ...

Introducing Views { { “type”: “car” “type”: “van” “colour”... “colour”... } } { { “type”: “lorry” “type”: “van” “colour”... “colour”... } } CouchDB

Introducing Views { { We need to look at a “type”: “car” “colour”... “type”: “van” “colour”... subset of the } } database documents - { { or a ‘view’ of the “type”: “lorry” “colour”... “type”: “van” “colour”... database } } CouchDB

Views • These views are saved in the database itself under a special identiﬁer • they must start: _design/

Simple View function(doc) { if (doc.Type == "van") { emit(null, doc); } }

Executing the query http://localhost:5984/rails-underground/_view/vans/all {"total_rows":2, "offset":0, "rows":[ {"id":"1", "key":null, "value": {"_id":"1","_rev":"4062949995","type":"van","name":"Tran sit"}}, {"id":"3", "key":null, "value": {"_id":"3","_rev":"1728947327","type":"van","name":"Tran sporter"}} ] }

Executing the query http://localhost:5984/rails-underground/_view/vans/all? limit=1 {"total_rows":1, "offset":0, "rows":[ {"id":"1", "key":null, "value": {"_id":"1","_rev":"4062949995","type":"van","name":"Tran sit"}} ] }

Simple View function(doc) { if (doc.Type == "van") { emit(doc.Name, doc); } }

Executing the query http://localhost:5984/rails-underground/_view/vans/all? startkey=Trans {"total_rows":1, "offset":0, "rows":[ {"id":"3", "key":”Transporter”, "value": {"_id":"3","_rev":"1728947327","type":"van","name":"Tran sporter"}} ] }

Reduce Query • map: function(doc) {if (doc.Type == "van") {emit(null, doc);}} // previous example • reduce: function(keys, values) { return keys.length } • This will return a result of 2 • You can optionally rereduce the result set by having a third argument on the reduce function

Inline Associatons { “user” : “George”, “roles” : [“supervisor”, “teamleader”], ... }

Associations in 1 query function(doc) { if (doc.type == "post") { map([doc._id, 0], doc); } else if (doc.type == "comment") { map([doc.post, 1], doc); } }

Associations in 1 query http://localhost:5984/rails-underground/_view/ posts_comments/all “key”:[“1”,0], “value”:{“_id”:”1”, “type”:”post”, “text”:”My Blog Post”} “key”:[“2”,0], “value”:{“_id”:”2”, “type”:”post”, “text”:”My 2nd Blog Post”} “key”:[“3”,0], “value”:{“_id”:”3”, “type”:”post”, “text”:”My 3rd Blog Post”} “key”:[“1”,1], “value”:{“_id”:”3”, “type”:”comment”, “text”:”You rock dude”, “post”:”1”} “key”:[“2”,1], “value”:{“_id”:”3”, “type”:”comment”, “text”:”Man you suck”, “post”:”2”}

Associations in 1 query http://localhost:5984/rails-underground/_view/ posts_comments/all?startkey=[“1”]&endkey=[“1”,2] “key”:[“1”,0], “value”:{“_id”:”1”, “type”:”post”, “text”:”My Blog Post”} “key”:[“2”,0], “value”:{“_id”:”2”, “type”:”post”, “text”:”My 2nd Blog Post”} “key”:[“3”,0], “value”:{“_id”:”3”, “type”:”post”, “text”:”My 3rd Blog Post”} “key”:[“1”,1], “value”:{“_id”:”3”, “type”:”comment”, “text”:”You rock dude”, “post”:”1”} “key”:[“2”,1], “value”:{“_id”:”3”, “type”:”comment”, “text”:”Man you suck”, “post”:”2”}

Indexes • Relational databases typically pay cost at point of insertion • CouchDB pay cost at time of checking view • Expensive if you write lots and read infrequently • Can create an update script CouchDB calls when database updated

Replication

Replication Nodes that go down catch up

Replication Partial Replicas

Replication Partial Replicas Ofﬂine copies

Conflict Management • As well as an _id field each document has a _rev • Each time a document is updated so is the _rev • Used to help determine ‘winning’ document when conflicts occur • Old revisions are available until the database is compressed

When to use CouchDB (and also when not)

Scenario I: Schema-less databases • FriendFeed moved to using MySQL in schema-less fashion because: • Adding new features became so difﬁcult. In particular adding new indexes became too expensive for tables with 10-20 million rows

Rules of Database degradation • Fields become optional • Relationships become Many-to-Many [ Taken from http://push.cx/2009/rules-of-database- app-aging ]

Scenario II: Real world models “The real world doesn’t map to a relational database very well”

5ft Shelf - Envisaged Database Design Shelf M M Book 1 1 M M M M Recommendat Price Category ions

5ft Shelf - Actual Database Design Book 1 M Recommendat ions 1 M Shelf M M Book Edition 1 1 M M M M ISBN Price Category

5ft Shelf - CouchDB Database Design Shelf M M Book 1 M Recommendat ions Prices ISBNs Categories Inline Associations on Book

Scenario III: Using replication/sharding • Ofﬂine Capability: • Satellite Ofﬁce • Desktop Apps • ‘Edge’ Databases • Large amount of databases needed • Even 1 database per user

When not to use CouchDB • Estate Agent application • Each house has x bedrooms, y bathrooms... • Financial Applications • (Generally stuff that’s very ﬁxed)

Introducing couch_foo (An exercise in learning how ActiveRecord really works)

Why? • Used all the existing ruby libraries at the time and found frustrations coming from ActiveRecord API • Opinion split on whether providing a full API for CouchDB is a good thing or whether more of the logic should be in application

CouchFoo Models class Address < CouchFoo::Base property :number, Integer property :street, String property :postcode end # - Generic types are fine if to_json # from_json are defined on the # object

CouchFoo Models (2) • The _id field is automatically assigned by CouchFoo and is guaranteed to be a UUID • View generation is handled automatically so you can just call the finders • Nearly all AR finders are available • Address.all • User.find_by_login, ... • (Obviously #find_by_sql isn’t)

CouchFoo Models (3) • Associations • Validations • Callbacks • Inheritance • Calculations • ...

How finders work • Each model stores its name in a ruby_class attribute in the document. • This makes it easy to filter out in a view • And then regenerate the model on retrieval • Views are generated automatically with one for each attribute lookup on a given model • Users can define their own custom view in the model as well

Finders • User.find(:first, :conditions => {:login => “george”}) • User.find(:first, :conditions => {:login => “george”}, :update => false) • User.find(:all, :limit => 10, :offset => 3) • User.find(:all, :use_key => [:login], :startkey => “fred”, :endkey => “george”)

Ordering Confusion • Relational databases order records by id which normally keeps incrementing • Thus newer records have a higher id • In CouchDB inserting a record may have a key starting ‘a126’ whereas the next insertion may have a key starting ‘3a82’ • Thus ﬁnd(:all) operations aren’t always in the order you expect • This can confuse the user too - “Why is my newest photo in the middle of the list?”

Ordering Confusion (2) class Address < CouchFoo::Base property :number, Integer property :street, String property :postcode property :created_at, DateTime default_sort :created_at end

Gotchas • To keep views(indexes) at a minimum ordering is done by CouchDB if the attribute isn’t exposed in the key • This means using :order and :limit may not provide the results expected • Can get round by adding extra attributes to the query eg User.find(:all, :use_key => [:name, :admin], :conditions => {:admin => true}, :order => :name, :limit => 10)

Bulk Saving CouchFoo::Base.bulk_save_default = true User.all.size # => 27 User.create(:name => "george") User.all.size # => 27 User.database.commit User.all.size # => 28 # It’s the developers responsibility to commit # In Rails the after_ﬁlter can be used to avoid forgetting

Performance • For updates will perform worse when not in memory - eg update_all • With bulk_save can perform much better than relational databases • CouchDB 0.9 offers a few performance enhancements over 0.8 • Performance implications fully documented in CouchFoo#Base

ActiveRecord tips • model.column_names # => {“id”, “login”,...} • also columns and columns_hash • instance.attributes # => {“name” => “George”, ...} • model.with_scope User.with_scope(:find => { :conditions => "state = 3" }) do find(1) # => SELECT * from articles WHERE state = 3 AND id = 1 end

Resources • Rails Freelancer • @Georgio_1999 • http://rowtheboat.com • http://github.com/georgepalmer

Couch Foo: CouchDB on rails

by Georgio_1999 on Jul 27, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

1 Embed 16

Statistics

Couch Foo: CouchDB on rails — Presentation Transcript