Practical Ruby Projects with MongoDB - Ruby Midwest

PRACTICAL PROJECTS WITH Saturday, July 17, 2010

WHO AM I? Alex Sharp Lead Developer at OptimisDev @ajsharp (on the twitters) alexjsharp.com (on the ‘tubes) github.com/ajsharp (on the ‘hubs) Saturday, July 17, 2010

MAJOR POINTS • Brief intro to MongoDB • Practical Projects • Making the case for Mongo (time permitting) Saturday, July 17, 2010

BRIEF INTRO Saturday, July 17, 2010

WHAT IS MONGODB? mongodb is a high-performance, schema-less, document-oriented database Saturday, July 17, 2010

STRENGTHS Saturday, July 17, 2010

SCHEMA-LESS Saturday, July 17, 2010

SCHEMA-LESS • Great for rapid, iterative, agile development Saturday, July 17, 2010

SCHEMA-LESS • Great for rapid, iterative, agile development • Makes things possible that in an rdbms are either a. impossibly hard b. virtually impossible c. way harder than they should be Saturday, July 17, 2010

DOCUMENT-ORIENTED Saturday, July 17, 2010

DOCUMENT-ORIENTED • Mongo stores documents, not rows • documents are stored as binary json Saturday, July 17, 2010

DOCUMENT-ORIENTED • Mongo stores documents, not rows • documents are stored as binary json • Allows for a rich query syntax Saturday, July 17, 2010

DOCUMENT-ORIENTED • Mongo stores documents, not rows • documents are stored as binary json • Allows for a rich query syntax • Embedded documents eliminate many modeling headaches Saturday, July 17, 2010

BIG DATA Saturday, July 17, 2010

BIG DATA • Built to scale horizontally into the cloudz Saturday, July 17, 2010

BIG DATA • Built to scale horizontally into the cloudz • Auto-sharding • set a shard-key, a cluster of nodes, go • still in alpha, but production-ready soon Saturday, July 17, 2010

FAST WRITES Saturday, July 17, 2010

FAST WRITES • Mongo writes are “ﬁre-and-forget” • you can get a response with getLastError() Saturday, July 17, 2010

FAST WRITES • Mongo writes are “ﬁre-and-forget” • you can get a response with getLastError() • In-place updates Saturday, July 17, 2010

FAST WRITES • Mongo writes are “ﬁre-and-forget” • you can get a response with getLastError() • In-place updates • Lot’s of db commands for fast updates Saturday, July 17, 2010

AGGREGATION Map/reduce for aggregation Saturday, July 17, 2010

READS FAST TOO Optimize reads with indexes, just like an rdbms Saturday, July 17, 2010

WEAKNESSES Saturday, July 17, 2010

JOINS nope. Saturday, July 17, 2010

MULTI-DOCUMENT TRANSACTIONS no-sir-ee. Saturday, July 17, 2010

RELATIONAL INTEGRITY not a chance. Saturday, July 17, 2010

PRACTICAL PROJECTS Saturday, July 17, 2010

1. Accounting Application 2. Capped Collection Logging 3. Blogging app 4. Reporting app Saturday, July 17, 2010

GENERAL LEDGER ACCOUNTING APPLICATION Saturday, July 17, 2010

THE OBJECT MODEL ledger * transactions * entries Saturday, July 17, 2010

THE OBJECT MODEL # Credits Debits Line item { 1 { :account => “Cash”, :amount => 100.00 } { :account => “Notes Pay.”, :amount => 100.00 } Ledger Entries Line item { 2 { :account => “A/R”, :amount => 25.00 } { :account => “Gross Revenue”, :amount => 25.00 } Saturday, July 17, 2010

THE OBJECT MODEL Saturday, July 17, 2010

THE OBJECT MODEL • Each ledger line item belongs to a ledger Saturday, July 17, 2010

THE OBJECT MODEL • Each ledger line item belongs to a ledger • Each ledger line item has two ledger entries • must have at least one credit and one debit • credits and debits must balance Saturday, July 17, 2010

THE OBJECT MODEL • Each ledger line item belongs to a ledger • Each ledger line item has two ledger entries • must have at least one credit and one debit • credits and debits must balance • These objects must be transactional Saturday, July 17, 2010

SQL-BASED OBJECT MODEL @debit = Entry.new(:account => 'Cash', :amount => 100.00, :type => 'credit') @credit = Entry.new(:account => 'Notes Pay.', :amount => 100.00, :type => 'debit') @line_item = LineItem.new :ledger_id => 1, :entries => [@debit, @credit] Saturday, July 17, 2010

SQL-BASED OBJECT MODEL @debit = Entry.new(:account => 'Cash', :amount => 100.00, :type => 'credit') @credit = Entry.new(:account => 'Notes Pay.', :amount => 100.00, :type => 'debit') @line_item = LineItem.new :ledger_id => 1, :entries => [@debit, @credit] In a SQL schema, we need a database transaction to ensure multi-row atomicity Saturday, July 17, 2010

SQL-BASED OBJECT MODEL @debit = Entry.new(:account => 'Cash', :amount => 100.00, :type => 'credit') @credit = Entry.new(:account => 'Notes Pay.', :amount => 100.00, :type => 'debit') @line_item = LineItem.new :ledger_id => 1, :entries => [@debit, @credit] Remember: we have to create 3 new records in the database here Saturday, July 17, 2010

MONGO-FIED MODELING @line_item = LineItem.new(:ledger_id => 1, :entries => [ {:account => 'Cash', :type => 'debit', :amount => 100}, {:account => 'Notes pay.', :type => 'debit', :amount => 100} ] ) Saturday, July 17, 2010

MONGO-FIED MODELING @line_item = LineItem.new(:ledger_id => 1, :entries => [ {:account => 'Cash', :type => 'debit', :amount => 100}, {:account => 'Notes pay.', :type => 'debit', :amount => 100} ] ) This is the perfect case for embedded documents. Saturday, July 17, 2010

MONGO-FIED MODELING @line_item = LineItem.new(:ledger_id => 1, :entries => [ {:account => 'Cash', :type => 'debit', :amount => 100}, {:account => 'Notes pay.', :type => 'debit', :amount => 100} ] ) We will never have more than a few entries (usually two) Saturday, July 17, 2010

MONGO-FIED MODELING @line_item = LineItem.new(:ledger_id => 1, :entries => [ {:account => 'Cash', :type => 'debit', :amount => 100}, {:account => 'Notes pay.', :type => 'debit', :amount => 100} ] ) Embedded docs are perfect for “contains” many relationships Saturday, July 17, 2010

MONGO-FIED MODELING @line_item = LineItem.new(:ledger_id => 1, :entries => [ {:account => 'Cash', :type => 'debit', :amount => 100}, {:account => 'Notes pay.', :type => 'debit', :amount => 100} ] ) DB-level transactions no-longer needed. Mongo has single document atomicity. Saturday, July 17, 2010

LESSONS Saturday, July 17, 2010

LESSONS • Simpliﬁed, de-normalized data model • Objects modeled differently in Mongo than SQL Saturday, July 17, 2010

LESSONS • Simpliﬁed, de-normalized data model • Objects modeled differently in Mongo than SQL • Simpler data model + no transactions = WIN Saturday, July 17, 2010

LOGGING WITH CAPPED COLLECTIONS Saturday, July 17, 2010

BASELESS STATISTIC 95% of the time logs are NEVER used. Saturday, July 17, 2010

MOST LOGS ARE ONLY USED WHEN SOMETHING GOES WRONG Saturday, July 17, 2010

CAPPED COLLECTIONS • Fixed-sized, limitedoperation, auto age-out collections (kinda like memcached, except persistent) • Fixed insertion order • Super fast (faster than normal writes) • Ideal for logging and caching Saturday, July 17, 2010

GREAT. WHAT’S THAT MEAN? We can log additional pieces of arbitrary data effortlessly due to Mongo’s schemaless nature. Saturday, July 17, 2010

THIS IS AWESOME Saturday, July 17, 2010

QUERY. ANALYZE. PROFIT. Saturday, July 17, 2010

ALSO A REALLY HANDY TROUBLESHOOTING TOOL Saturday, July 17, 2010

User-reported errors are difﬁcult to diagnose Saturday, July 17, 2010

Wouldn’t it be useful to see the complete click- path? Saturday, July 17, 2010

BUNYAN http://github.com/ajsharp/bunyan Thin ruby layer around a MongoDB capped collection Saturday, July 17, 2010

BUNYAN http://github.com/ajsharp/bunyan Still needs a middleware api. Want to contribute? Saturday, July 17, 2010

require 'bunyan' Bunyan::Logger.configure do |c| c.database 'my_bunyan_db' c.collection "#{Rails.env}_bunyan_log" c.size 1073741824 # 1.gigabyte end Saturday, July 17, 2010

PAPERMILL http://github.com/ajsharp/papermill SINATRA FRONT-END TO BUNYAN Saturday, July 17, 2010

BLOGGING APPLICATION Saturday, July 17, 2010

BLOGGING APPLICATION Much easier to model with Mongo than a relational database Saturday, July 17, 2010

BLOGGING APPLICATION A post has an author Saturday, July 17, 2010

BLOGGING APPLICATION A post has an author A post has many tags Saturday, July 17, 2010

BLOGGING APPLICATION A post has an author A post has many tags A post has many comments Saturday, July 17, 2010

BLOGGING APPLICATION A post has an author A post has many tags A post has many comments Instead of JOINing separate tables, we can use embedded documents. Saturday, July 17, 2010

MONGOMAPPER Saturday, July 17, 2010

MONGOMAPPER •MongoDB “ORM” developed by John Nunemaker Saturday, July 17, 2010

MONGOMAPPER •MongoDB “ORM” developed by John Nunemaker •Very similar syntax to DataMapper Saturday, July 17, 2010

MONGOMAPPER •MongoDB “ORM” developed by John Nunemaker •Very similar syntax to DataMapper •Declarative rather than inheritance-based Saturday, July 17, 2010

MONGOMAPPER •MongoDB “ORM” developed by John Nunemaker •Very similar syntax to DataMapper •Declarative rather than inheritance-based •Very easy to drop into rails Saturday, July 17, 2010

MONGOMAPPER require 'mongo' connection = Mongo::Connection.new.db("#{Rails.env}_app") class Post include MongoMapper::Document belongs_to :author, :class_name => "User" key :title, String, :reuqired => true key :body, String, :required => true key :author_id, Integer, :required => true key :published_at, Time key :published, Boolean, :default => false key :tags, Array, :default => [] timestamps! end Saturday, July 17, 2010

REPORTING APP Saturday, July 17, 2010

Saturday, July 17, 2010

REPORTING IS HARD • Synchronization • Data, application API, schema • Schema synchronization is the hard one Saturday, July 17, 2010

WHY MONGO? • Aggregation support with Mongo’s map/reduce support • Mongo is good (and getting better) at handling big data sets • Schemaless is perfect for a ﬂattened data set Saturday, July 17, 2010

Amazon EC2 Rackspace Transform MongoDB Server Reporting App WAN Main App MySQL Saturday, July 17, 2010

CLIENT REQUESTS Amazon EC2 MongoDB Server map/reduce Reporting App User requests Client request GET /reports/some_report Saturday, July 17, 2010

TRANFORMATION Data needs to extracted from the the main app, transormed (i.e. ﬂattened) into some other form, and loaded into the reporting app for aggregation Saturday, July 17, 2010

TRANSFORMATION Rackspace Amazon EC2 1. GET /object_dependencies Transform MongoDB Server { :visit => [ :insurances, :patients ] } 2. read from active record MySQL 2. write to mongo Reporting App Main App Saturday, July 17, 2010

MAKING THE CASE Saturday, July 17, 2010

THE PROBLEMS OF SQL Saturday, July 17, 2010

A BRIEF HISTORY OF SQL • Developed at IBM in early 1970’s. • Designed to manipulate and retrieve data stored in relational databases Saturday, July 17, 2010

A BRIEF HISTORY OF SQL • Developed at IBM in early 1970’s. • Designed to manipulate and retrieve data stored in relational databases this is a problem Saturday, July 17, 2010

WHY?? Saturday, July 17, 2010

WE DON’T WORK WITH DATA... Saturday, July 17, 2010

WE WORK WITH OBJECTS Saturday, July 17, 2010

WE DON’T CARE ABOUT STORING DATA Saturday, July 17, 2010

WE CARE ABOUT PERSISTING STATE Saturday, July 17, 2010

DATA != OBJECTS Saturday, July 17, 2010

THEREFORE... Saturday, July 17, 2010

RELATIONAL DBS ARE AN ANTIQUATED TOOL FOR OUR NEEDS Saturday, July 17, 2010

OK, SO WHAT? SQL schemas are designed for storing and querying data, not persisting objects. Saturday, July 17, 2010

To reconcile this mismatch, we have ORM’s Saturday, July 17, 2010

Object Relational Mappers Saturday, July 17, 2010

OK, SO WHAT? We need ORMs to bridge the gap (map) between our native Ruby object model and a SQL schema Saturday, July 17, 2010

We’re forced to create relationships for data when what we really want is properties for our objects. Saturday, July 17, 2010

THIS IS NOT EFFICIENT Saturday, July 17, 2010

@alex = Person.new( :name => "alex", :stalkings => [ Friend.new("Jim"), Friend.new("Bob")] ) Saturday, July 17, 2010

Native ruby object <Person:0x10017d030 @name="alex", @stalkings= [#<Friend:0x10017d0a8 @name="Jim">, #<Friend:0x10017d058 @name="Bob"> ]> Saturday, July 17, 2010

JSON/Mongo Representation @alex.to_json { name: "alex", stalkings: [{ name: "Jim" }, { name: "Bob" }] } Saturday, July 17, 2010

SQL Schema Representation people: - name stalkings: - name - stalker_id Saturday, July 17, 2010

SQL Schema Representation people: - name stalkings: What!? - name - stalker_id Saturday, July 17, 2010

SQL Schema Representation people: - name stalkings: what’s a stalker_id? - name - stalker_id Saturday, July 17, 2010

SQL Schema Representation people: - name stalkings: this is our sql mapping - name - stalker_id Saturday, July 17, 2010

SQL Schema Representation people: - name stalkings: this is the - name “pointless” join - stalker_id Saturday, July 17, 2010

Ruby -> JSON -> SQL <Person:0x10017d030 @name="alex", @stalkings= Ruby [#<Friend:0x10017d0a8 @name="Jim">, #<Friend:0x10017d058 @name="Bob"> ]> @alex.to_json JSON { name: "alex", stalkings: [{ name: "Jim" }, { name: "Bob" }] } people: - name SQL stalkings: - name - stalker_id Saturday, July 17, 2010

Ruby -> JSON -> SQL <Person:0x10017d030 @name="alex", @stalkings= Ruby [#<Friend:0x10017d0a8 @name="Jim">, #<Friend:0x10017d058 @name="Bob"> ]> @alex.to_json JSON { name: "alex", stalkings: [{ name: "Jim" }, { name: "Bob" }] } people: - name Feels like we’re having SQL stalkings: to work too hard here - name - stalker_id Saturday, July 17, 2010

You’re probably thinking... “Listen GUY, SQL isn’t that bad” Saturday, July 17, 2010

Maybe. Saturday, July 17, 2010

But we can do much, much better. Saturday, July 17, 2010

NEEDS FOR A PERSISTENCE LAYER: Saturday, July 17, 2010

NEEDS FOR A PERSISTENCE LAYER: 1. To persist the native state of our objects Saturday, July 17, 2010

NEEDS FOR A PERSISTENCE LAYER: 1. To persist the native state of our objects 2. Should NOT interfere w/ application development Saturday, July 17, 2010

NEEDS FOR A PERSISTENCE LAYER: 1. To persist the native state of our objects 2. Should NOT interfere w/ application development 3. Provides features necessary to build modern web apps Saturday, July 17, 2010

NEXT STEPS Saturday, July 17, 2010

USING MONGO W/ RUBY • Ruby mongo driver • MongoMapper (github.com/jnunemaker/mongomapper) • MongoID (github.com/durran/mongoid) • Many other ORM/ODM’s • mongo-sequel (github.com/jeremyevans/sequel-mongo) • moneta (github.com/wycats/moneta) Saturday, July 17, 2010

MONGOMAPPER • DataMapper-like API • good for moving from a sql schema to mongo • easy to drop into rails • works with rails 3 Saturday, July 17, 2010

MONGOID • DataMapper-like API • uses ActiveModel, so familiar validation syntax • more opinionated embedded docs • easy to drop into rails • rails 2 - mongoid 1.x • rails 3 - mongoid 2.x Saturday, July 17, 2010

QUESTIONS? Saturday, July 17, 2010

THANKS! Saturday, July 17, 2010

http://www.mongodb.org	2648
http://www.nosqldatabases.com	67
http://speakerrate.com	20
http://ru.wiki.mongodb.org	19
http://pt.wiki.mongodb.org	15
http://vinublogs.blogspot.com	15
http://es.wiki.mongodb.org	12
http://billchristian.posterous.com	6
http://www.slideshare.net	4
http://mongodb.onconfluence.com	4
http://vinublogs.blogspot.in	3
http://wiki.mongodb.org	2
http://10gen.com	1
https://wiki.10gen.com	1
http://vinublogs.blogspot.ca	1
http://feeds.feedburner.com	1
http://vinublogs.blogspot.kr	1
http://api.mongodb.org	1
http://67.207.142.125	1
http://htmledit.squarefree.com	1
https://p.yammer.com	1
http://translate.googleusercontent.com	1
http://www.opencracy.com	1

Practical Ruby Projects with MongoDB - Ruby Midwest

by Alex Sharp

Accessibility

Categories

Upload Details

Usage Rights

23 Embeds 2,826

Statistics

Practical Ruby Projects with MongoDB - Ruby Midwest Presentation Transcript