Persistence Smoothie

Persistence Michael Bleigh
Intridea, Inc.

Smoothie
Blending SQL and NoSQL

photo by Nikki L. via Flickr

Thursday, March 11, 2010

present.ly


tweetstream hashie
acts-as-taggable-on
subdomain-fu seed-fu
mustache_json

github.com/intridea


@mbleigh


You’ve (probably)
heard a lot about
NoSQL


NoSQL is a new way
to think about
persistence


Atomicity
Consistency
Isolation
Durability


Denormalization
Eventual Consistency
Schema-Free
Horizontal Scale


NoSQL tries to scale
(more) simply


NoSQL is going
mainstream


New York Times
Business Insider
BBC ShopWiki
GitHub Meebo
Disqus SourceForge
Sony Digg


...but not THAT
mainstream.


A word of caution...


NoSQL can
divide by zero


sn’t
d oe s
QL wait
oS , it
N
s le ep
NoSQL can
divide by zero
NoSQL
to infin counte
ity, twi d
ce

NoSQL is a (growing)
collection of tools, not
a new way of life


Key-Value Stores
Document Databases
Column Stores
Graph Databases


Key-Value Stores


Redis

• Key-value store + datatypes
• Lists, (Scored) Sets, Hashes
• Cache-like functions
(expiration)
• (Mostly) In-Memory

Riak

• Combo key-value store and
document database
• HTTP REST interface
• “Link walking”
• Map-Reduce

Map/Reduce
• Massively parallel way to
process large datasets
• First you scour data and “map” a
new set of data
• Then you “reduce” the data
down to a salient result


map = function() {
this.tags.forEach(function(tag) {
emit(tag, {count: 1});
});
}

reduce = function(key, values) {
var total = 0;
for (var i = 0; i < values.length; i++) {
total += values[i].count;
return {count: total};
}


Tokyo Cabinet
Dynomite
MemcachedDB
Voldemort


Document Databases


MongoDB

• Document store that speaks
BSON (Binary JSON)
• Indexing, simple query syntax
• GridFS
• Deliberate MapReduce

CouchDB

• JSON Document Store
• HTTP REST Interface
• Incremental MapReduce
• Intelligent Replication

Column-Oriented
Datastores


Cassandra

• Built by Facebook,
used by Twitter
• Pure horizontal scalability
• Schemaless


Graph Databases


Neo4J


When should I use
this stuff?


Complex, slow joins
for “activity stream”


Complex, slow joins
for “activity stream”

Denormalize,
use Key-Value Store

Variable schema,
vertical interaction


Variable schema,
vertical interaction

Document Database
or Column Store

Modeling multi-step
relationships


Modeling multi-step
relationships

Graph Database


NoSQL solves real
scalability and data
design issues


Ben Scofield
bit.ly/state-of-nosql


Ready to go?


Just one problem...


Your data is already
in a SQL database


We CAN all just
get along.


Three Ways


The Hard(ish) Way


The Easy Way


A Better Way?


The Hard Way:
Do it by hand.


class Post
include MongoMapper::Document

key :title, String
key :body, String
key :tags, Array
key :user_id, Integer

def user
User.find_by_id(self.user_id)
end

def user=(some_user)
self.user_id = some_user.id
end
end

class User < ActiveRecord::Base
def posts(options = {})
Post.all({:conditions => {:user_id => self.id}}.merge(options))
end
end


Pros & Cons
• Simple, maps to your domain

• Works for small, simple ORM intersections

• MUCH simpler in Rails 3

• Complex relationships are a mess

• Makes your models fat

• As DRY as the ocean


The Easy Way:
DataMapper


DataMapper

• Generic, relational ORM
• Speaks pretty much everything
you’ve ever heard of
• Implements Identity Map
• Module-based inclusion

DataMapper.setup(:default, "mysql://localhost")
DataMapper.setup(:mongodb, "mongo://localhost/posts")

class Post
include DataMapper::Resource
def self.default_repository_name; :mongodb; end

property :title, String
property :body, String
property :tags, Array

belongs_to :user
end

class User
include DataMapper::Resource

property :email, String
property :name, String

has n, :posts
end


Pros & Cons
• The ultimate Polyglot ORM

• Simple relationships between persistence
engines are easy

• Jack of all trades, master of none

• Perpetuates (sometimes) false assumptions

• Legacy stuﬀ is in ActiveRecord anyway


Is there a better way?


Maybe.


Gloo: Cross-ORM
Relationship Mapper

github.com/intridea/gloo


0.0.0.prealpha.1


Can’t we just sit
down and talk to
each other?


class Post
include MongoMapper::Resource

key :title, String
key :body, String
key :tags, Array

gloo :active_record do
belongs_to :user
end
end

class User < ActiveRecord::Base
gloo :mongo_mapper do
many :posts
end
end


Goals/Status
• Be able to deﬁne relationships
on the terms of any ORM from
any class, ORM or not
• Right Now: Partially working
ActiveRecord relationships
• Doing it wrong? Maybe

Code Time:
Schema4Less


Social Storefront
• Dummy application of a store that
lets others “follow” your purchases (a
less creepy Blippy?)
• Four requirements:
• users

• purchasing

• listings

• social graph


Users

• I already have an authentication
system
• I’m happy with it
• It’s Devise and ActiveRecord
• Stick with SQL

Purchasing

• Users need to be able to purchase
items from my storefront
• I can’t lose their transactions
• I need full ACID
• I’ll use MySQL

Social Graph

• I want activity streams and one
and two way relationships
• I need speed
• I don’t need consistency
• I’ll use Redis

Product Listings
• I am selling both movies and
books
• They have very diﬀerent
properties
• Products are relatively non-
relational
• I’ll use MongoDB

Demo and
Walkthrough


Wrapping Up


These systems can
(and should) live and
work together


Most important step
is to actually think
about data design


When you have a
whole bag of tools,
things stop looking
like nails


Questions?


Persistence Smoothie

More Related Content

Viewers also liked

Similar to Persistence Smoothie

More from Michael Bleigh

Recently uploaded

Persistence Smoothie