Persistence Smoothie

Persistence Michael Bleigh
Intridea, Inc.

Smoothie
Blending SQL and NoSQL

photo by Nikki L. via Flickr

Thursday, March 11, 2010

present.ly


tweetstream hashie
acts-as-taggable-on
subdomain-fu seed-fu
mustache_json

github.com/intridea


@mbleigh


You’ve (probably)
heard a lot about
NoSQL


NoSQL is a new way
to think about
persistence


Atomicity
Consistency
Isolation
Durability


Denormalization
Eventual Consistency
Schema-Free
Horizontal Scale


NoSQL tries to scale
(more) simply


NoSQL is going
mainstream


New York Times
Business Insider
BBC ShopWiki
GitHub Meebo
Disqus SourceForge
Sony Digg


...but not THAT
mainstream.


A word of caution...


NoSQL can
divide by zero


sn’t
d oe s
QL wait
oS , it
N
s le ep
NoSQL can
divide by zero
NoSQL
to infin counte
ity, twi d
ce

NoSQL is a (growing)
collection of tools, not
a new way of life


Key-Value Stores
Document Databases
Column Stores
Graph Databases


Key-Value Stores


Redis

• Key-value store + datatypes
• Lists, (Scored) Sets, Hashes
• Cache-like functions
(expiration)
• (Mostly) In-Memory

Riak

• Combo key-value store and
document database
• HTTP REST interface
• “Link walking”
• Map-Reduce

Map/Reduce
• Massively parallel way to
process large datasets
• First you scour data and “map” a
new set of data
• Then you “reduce” the data
down to a salient result


map = function() {
this.tags.forEach(function(tag) {
emit(tag, {count: 1});
});
}

reduce = function(key, values) {
var total = 0;
for (var i = 0; i < values.length; i++) {
total += values[i].count;
return {count: total};
}


Tokyo Cabinet
Dynomite
MemcachedDB
Voldemort


Document Databases


MongoDB

• Document store that speaks
BSON (Binary JSON)
• Indexing, simple query syntax
• GridFS
• Deliberate MapReduce

CouchDB

• JSON Document Store
• HTTP REST Interface
• Incremental MapReduce
• Intelligent Replication

Column-Oriented
Datastores


Cassandra

• Built by Facebook,
used by Twitter
• Pure horizontal scalability
• Schemaless


Graph Databases


Neo4J


When should I use
this stuff?


Complex, slow joins
for “activity stream”


Complex, slow joins
for “activity stream”

Denormalize,
use Key-Value Store

Variable schema,
vertical interaction


Variable schema,
vertical interaction

Document Database
or Column Store

Modeling multi-step
relationships


Modeling multi-step
relationships

Graph Database


NoSQL solves real
scalability and data
design issues


Ben Scofield
bit.ly/state-of-nosql


Ready to go?


Just one problem...


Your data is already
in a SQL database


We CAN all just
get along.


Three Ways


The Hard(ish) Way


The Easy Way


A Better Way?


The Hard Way:
Do it by hand.


class Post
include MongoMapper::Document

key :title, String
key :body, String
key :tags, Array
key :user_id, Integer

def user
User.find_by_id(self.user_id)
end

def user=(some_user)
self.user_id = some_user.id
end
end

class User < ActiveRecord::Base
def posts(options = {})
Post.all({:conditions => {:user_id => self.id}}.merge(options))
end
end


Pros & Cons
• Simple, maps to your domain

• Works for small, simple ORM intersections

• MUCH simpler in Rails 3

• Complex relationships are a mess

• Makes your models fat

• As DRY as the ocean


The Easy Way:
DataMapper


DataMapper

• Generic, relational ORM
• Speaks pretty much everything
you’ve ever heard of
• Implements Identity Map
• Module-based inclusion

DataMapper.setup(:default, "mysql://localhost")
DataMapper.setup(:mongodb, "mongo://localhost/posts")

class Post
include DataMapper::Resource
def self.default_repository_name; :mongodb; end

property :title, String
property :body, String
property :tags, Array

belongs_to :user
end

class User
include DataMapper::Resource

property :email, String
property :name, String

has n, :posts
end


Pros & Cons
• The ultimate Polyglot ORM

• Simple relationships between persistence
engines are easy

• Jack of all trades, master of none

• Perpetuates (sometimes) false assumptions

• Legacy stuﬀ is in ActiveRecord anyway


Is there a better way?


Maybe.


Gloo: Cross-ORM
Relationship Mapper

github.com/intridea/gloo


0.0.0.prealpha.1


Can’t we just sit
down and talk to
each other?


class Post
include MongoMapper::Resource

key :title, String
key :body, String
key :tags, Array

gloo :active_record do
belongs_to :user
end
end

class User < ActiveRecord::Base
gloo :mongo_mapper do
many :posts
end
end


Goals/Status
• Be able to deﬁne relationships
on the terms of any ORM from
any class, ORM or not
• Right Now: Partially working
ActiveRecord relationships
• Doing it wrong? Maybe

Code Time:
Schema4Less


Social Storefront
• Dummy application of a store that
lets others “follow” your purchases (a
less creepy Blippy?)
• Four requirements:
• users

• purchasing

• listings

• social graph


Users

• I already have an authentication
system
• I’m happy with it
• It’s Devise and ActiveRecord
• Stick with SQL

Purchasing

• Users need to be able to purchase
items from my storefront
• I can’t lose their transactions
• I need full ACID
• I’ll use MySQL

Social Graph

• I want activity streams and one
and two way relationships
• I need speed
• I don’t need consistency
• I’ll use Redis

Product Listings
• I am selling both movies and
books
• They have very diﬀerent
properties
• Products are relatively non-
relational
• I’ll use MongoDB

Demo and
Walkthrough


Wrapping Up


These systems can
(and should) live and
work together


Most important step
is to actually think
about data design


When you have a
whole bag of tools,
things stop looking
like nails


Questions?


Persistence Smoothie

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (8)

Similar to Persistence Smoothie

Similar to Persistence Smoothie (20)

More from Michael Bleigh

More from Michael Bleigh (9)

Recently uploaded

Recently uploaded (20)

Persistence Smoothie