Zero to Mongo in 60 Hours

Zero to Mongo in 60
Hours
Ryan Angilly
MyPunchbowl.com

@angilly

Wednesday, April 28, 2010

I’m a pretty awesome dude


• Electrical Engineer by
education


education

• Ex-EMC’er


education

• Ex-EMC’er

• Founded
MessageSling.com


education

• Ex-EMC’er

• Founded
MessageSling.com

• Entered the deadpool
in January


education

• Ex-EMC’er

• Founded
MessageSling.com

in January

• Senior developer @
MyPunchbowl.com


education
• Built several web apps

• Ex-EMC’er

• Founded
MessageSling.com

in January

MyPunchbowl.com


education

• Ex-EMC’er
• All Ruby. Mostly Rails

• Founded
MessageSling.com

in January

MyPunchbowl.com


education

• Ex-EMC’er

• Founded
• All SQL
MessageSling.com

in January

MyPunchbowl.com


education

• Ex-EMC’er

• Founded
• All SQL
MessageSling.com
• No experience w/
NoSQL/Mongo as of
in January
September ’09

MyPunchbowl.com


education

• Ex-EMC’er

• Founded
• All SQL
MessageSling.com
• No experience w/
NoSQL/Mongo as of
in January
September ’09

• Want to provide a
beginner’s perspective
MyPunchbowl.com


MyPunchbowl is a pretty awesome company



• Leader in start to ﬁnish party planning




• Connecting party planners to party
vendors -- subject of this talk





• Currently employs 11 people






• Funded over 2 rounds by Intel & Contour







• Strategic partnership w/ OTC among
others







others

• Amount of trafﬁc, users, events, invites,
vendors, etc... make our data sets large
enough to be interesting :)







others

• Amount of trafﬁc, users, events, invites,
vendors, etc... make our data sets large
enough to be interesting :)

• We’re talking many millions of ‘things’


Our engineers play with everything


Search by category and
location


Search by business name Search by category and
and location location


Track searches


Track searches

Track vendor impressions


MongoDB +
MyPunchbowl
A tale in conversations


Tracking requirement a good excuse to
ﬁnally use MongoDB



Me: Hey Blake I’m gonna use MongoDB to
keep track of all this search stuff.



Blake: Cool.



Blake: Cool.
Matt (CEO): What’s that?



Blake: Cool.
Blake & Me: Something cool.



Blake: Cool.
Blake & Me: Something cool.
Matt (CEO): K


6 reasons to use MongoDB
• Easy to get running (~5 minutes on OSX)
• Open Source
• Support in multiple (computer) languages
• Prototype in Ruby, move to Java if needed
• Very active development
• Full featured
• Great ecosystem

MongoDB feels right


MongoDB feels right
Me: MongoDB gives me the warm fuzzies
that Rails did.


MongoDB feels right
that Rails did.
Darren: Just like that Nunemaker post.


MongoDB feels right
that Rails did.
Darren: Just like that Nunemaker post.
Me: ? *runs to google*


Support is INSANE

11:35pm. Wednesday. Founder.
60 second response time.


2.5 days? Really?


2.5 days? Really?

Yes


2.5 days? Really?

Yes*


2.5 days to MongoDB

0 Days 1 Day 2 Days 200+ Days
and on...


2.5 days to MongoDB
Decision
to use
MongoDB

and on...


2.5 days to MongoDB
Decision
to use
MongoDB

and on...

Play around with
available OSS


2.5 days to MongoDB
Decision
to use
MongoDB

Figure out
document
‘schema’

and on...

Play around with
available OSS


2.5 days to MongoDB
Decision
to use
MongoDB

Figure out
document
‘schema’

and on...

Play around with
available OSS

Write some stuff on top
of mongo-ruby-driver


2.5 days to MongoDB
Decision Build test rig
to use
MongoDB

Figure out
document
‘schema’

and on...

Play around with
available OSS



2.5 days to MongoDB
to use
MongoDB

Figure out
document
‘schema’

and on...

Play around with Setup Conﬁguration
available OSS Management



2.5 days to MongoDB
to use
MongoDB

Figure out
document Get production ready
‘schema’

and on...




2.5 days to MongoDB
to use
MongoDB

Figure out
document Get production ready
‘schema’

and on...


Deploy


A stroll through Mongo’s OSS ecosystem



• mongodb (C++)



• mongodb (C++)
• mongo shell (JS via SpiderMonkey)



• mongodb (C++)
• mongo-ruby-driver



• mongodb (C++)
• mongo-java-driver



• mongodb (C++)
• mongo_record (Ruby)



• mongodb (C++)
• mongo_mapper (Ruby)



• mongodb (C++)
• mongo_mapper (Ruby)
• Integration w/ Rails

Document ‘schema’


• What do documents look like?


• What do documents look like?
• Example Document
{
city: “Boston”,
state: “MA”,
date: 1272153600,
occurrences: 1,
category_id: 1
}


• What do documents look like? • How do you update the documents?
• Example Document
{
city: “Boston”,
state: “MA”,
date: 1272153600,
occurrences: 1,
category_id: 1
}


• Example Document • Mongo shell command
{ coll.update({‘city’: ‘Boston’,
city: “Boston”, ‘state’: ‘MA’,
state: “MA”, ‘date’: 1272153600,
date: 1272153600, ‘category_id’: 1},
occurrences: 1, {‘$inc’: {‘occurrences’: 1}},
category_id: 1 true)
}


state: “MA”, ‘date’: 1272153600,
date: 1272153600, ‘category_id’: 1},
}
Document to match


state: “MA”, ‘date’: 1272153600,
date: 1272153600, ‘category_id’: 1},
}
Document to match
Operation to perform


state: “MA”, ‘date’: 1272153600,
date: 1272153600, ‘category_id’: 1},
}
Document to match
Operation to perform
‘upsert’: update if it’s there
insert if it’s not


Cuhhraazy Indexes


Cuhhraazy Indexes
• Original Document
{
city: “Boston”,
state: “MA”,
date: 1272153600,
occurrences: 1,
category_id: 1
}


Cuhhraazy Indexes
{
city: “Boston”,
state: “MA”,
date: 1272153600,
occurrences: 1,
category_id: 1
}
• 40k cities, 200 days, 60 categories


Cuhhraazy Indexes
{
city: “Boston”,
state: “MA”,
date: 1272153600,
occurrences: 1,
category_id: 1
}
• 480M potential documents


Cuhhraazy Indexes
{
city: “Boston”,
state: “MA”,
date: 1272153600,
occurrences: 1,
category_id: 1
}
• Composite indexes
• state_1_city_1
• state_1_date_1_city_1
• category_id_1_date_1
• date_1


Cuhhraazy Indexes
{
city: “Boston”,
state: “MA”,
date: 1272153600,
occurrences: 1,
category_id: 1
}
• Composite indexes
• state_1_city_1
• state_1_date_1_city_1
• category_id_1_date_1
• date_1
• Flexibility & performance in
querying and aggregating


Cuhhraazy Indexes
• Original Document • More Complex Document w/ Embedded Document
{ {
city: “Boston”, city: “Boston”,
state: “MA”, state: “MA”,
date: 1272153600, date: 1272153600,
occurrences: 1, occurrences: 9581555,
category_id: 1 category_id: 1,
} mobile_sources: {
• 40k cities, 200 days, 60 categories browsers: {
windows_mobile: 1,
• 480M potential documents palm_os: 0,
• Composite indexes },
iphone_4g: 9481554

• state_1_city_1 zip_codes: [“01518”]
• state_1_date_1_city_1 }
• category_id_1_date_1 }
• date_1


Cuhhraazy Indexes
{ {
date: 1272153600, date: 1272153600,
} mobile_sources: {
windows_mobile: 1,
iphone_4g: 9481554

• date_1 • Additional, complex data down the road?


Cuhhraazy Indexes
{ {
date: 1272153600, date: 1272153600,
} mobile_sources: {
windows_mobile: 1,
iphone_4g: 9481554

• Flexibility & performance in • Add deep embedded indexes!


Cuhhraazy Indexes
{ {
date: 1272153600, date: 1272153600,
} mobile_sources: {
windows_mobile: 1,
iphone_4g: 9481554

querying and aggregating • ‘mobile_sources.zip_codes’


Cuhhraazy Indexes
{ {
date: 1272153600, date: 1272153600,
} mobile_sources: {
windows_mobile: 1,
iphone_4g: 9481554

querying and aggregating • ‘mobile_sources.zip_codes’
• ‘mobile_sources.browsers.iphone_4g’


Let’s write some Ruby


class SearchStats
cattr_accessor :db, :collection

self.collection = $mongo_db.collection ‘searches‘

def self.record_search(city, state, category_id, vendor_id)
collection.update({:vendor_id => vendor_id,
:city => city,
:state => state,
:category_id => category_id,
:date => Time.now.utc.beginning_of_day.to_i},
{'$inc' => {:occurrence => 1}},
{:upsert => true})
end
end


class SearchStats


:city => city,
:state => state,
{:upsert => true})
end
end

irb> SearchStats.record_search(‘Boston’, ‘MA’, 9, 42)


class SearchStats


:city => city,
:state => state,
{:upsert => true})
end
end

irb> SearchStats.record_search(‘Boston’, ‘MA’, 9, 42)

mongo> db.searches.findOne()
{ "city" : "Boston",
"date" : 1272326400,
"impression" : 1,
"category_id" : 9,
"state" : "MA",
"vendor_id" : 42 }


TESTING


TESTING
• Pretty much the same as anything else


TESTING
• test/unit/search_stats_test.rb


TESTING
• conﬁg/environments/test.rb


TESTING
• $mongo_db = Mongo::Connection.new.db ‘mongo-sf-db’


TESTING

• SearchStatsTest#setup & #teardown


TESTING

• No transactions


TESTING

• No transactions
• ActiveSupport spoils us

TESTING

• No transactions
• ActiveSupport spoils us
• Gotta clean up after yourself

Other developers gotta
use it too


use it too
• Rails’ database.yml takes care of MySQL


use it too
• Gotta switch up MongoDBs on your own


use it too
• Simple conﬁg system ﬁle inspired by
another Nunemaker post.


use it too
• No database migrations!


use it too
• No database migrations!
• Must write rake tasks to create indexes
unless your ORM takes care of it for you


Making production
ready


Making production
ready
• Replication was iffy in 1.0.0 (solid now)


Making production
ready
• Nightly dumps w/ mongoexport


Making production
ready
• Upload to S3


Making production
ready
• Upload to S3
• Monitor process w/ monit


Deploy


Deploy
• Original deployment uneventful


Deploy
• Just worked


Deploy
• Just worked
• Future deployments that required a
database upgrade required downtime
(bummer)


Deploy
• Just worked
• Future deployments that required a
database upgrade required downtime
(bummer)
• We wanted to play with the geospatial
search


Day 200


Day 200
• One production process


Day 200
• One database


Day 200
• One database
• Daily dumps using mongoexport + upload
to S3


Day 200
• One database
to S3
• Replication not running


Day 200
• One database
to S3
• Several collections


Day 200
• One database
to S3
• 100k to >10M documents


Day 200
• One database
to S3
• 100k to >10M documents
• Scores of deep, composite indexes

What tripped me up?


Simple Test
should "get impressions_by_date" do
record1 = {:date => 100, :impression => 10}
record2 = {:date => 90, :impression => 8}
Mongo::Vendor.collection.insert record1
Mongo::Vendor.collection.insert record2

assert_same_elements [record1, record2],
Mongo::Vendor.impressions_by_date
end


Crazy Failure
1) Failure:
test: Mongo::Vendor should get
impressions_by_date. (Mongo::VendorTest)
...
-<{{"date"=>100.0, "csum"=>10.0}=>1,
{"date"=>90.0, "csum"=>8.0}=>1}>
+<{{"date"=>90.0, "csum"=>8.0}=>1,
{"date"=>100.0, "csum"=>10.0}=>1}>


Mongo’s BSON OrderedHash doesn’t behave
like ActiveSupport::OrderedHash



irb(main):001:0> require 'active_support'
irb(main):002:0> include ActiveSupport
irb(main):003:0> oh = OrderedHash.new
irb(main):004:0> oh[:b] = 2
irb(main):005:0> oh[:c] = 3
irb(main):006:0> oh[:a] = 1
irb(main):007:0> a = {:a => 1, :b => 2, :c => 3}
irb(main):008:0> a == oh
=> true
irb(main):009:0> oh == a
=> true



irb(main):001:0> require 'active_support' irb(main):001:0> require 'mongo'
irb(main):002:0> include ActiveSupport irb(main):002:0> oh = OrderedHash.new
irb(main):003:0> oh = OrderedHash.new irb(main):003:0> oh[:b] = 2
irb(main):004:0> oh[:b] = 2 irb(main):004:0> oh[:c] = 3
irb(main):005:0> oh[:c] = 3 irb(main):005:0> oh[:a] = 1
irb(main):006:0> oh[:a] = 1 irb(main):006:0> a = {:a => 1, :b => 2, :c
irb(main):007:0> a = {:a => 1, :b => 2, :c => 3} => 3}
irb(main):008:0> a == oh irb(main):007:0> a == oh
=> true => true
irb(main):009:0> oh == a
=> true



irb(main):001:0> require 'active_support' irb(main):001:0> require 'mongo'
irb(main):002:0> include ActiveSupport irb(main):002:0> oh = OrderedHash.new
irb(main):003:0> oh = OrderedHash.new irb(main):003:0> oh[:b] = 2
irb(main):004:0> oh[:b] = 2 irb(main):004:0> oh[:c] = 3
irb(main):005:0> oh[:c] = 3 irb(main):005:0> oh[:a] = 1
irb(main):006:0> oh[:a] = 1 irb(main):006:0> a = {:a => 1, :b => 2, :c
irb(main):007:0> a = {:a => 1, :b => 2, :c => 3} => 3}
irb(main):008:0> a == oh irb(main):007:0> a == oh
=> true => true
irb(main):009:0> oh == a irb(main):008:0> oh == a
=> true => false


And it turns out...


And it turns out...

MongoDB will not add indexes to your
collections.


And it turns out...

MongoDB will not add indexes to your
collections.

Unless you tell it to.


add indexes

add indexes

add indexes

add indexes

add indexes

add indexes

add indexes

add indexes

add indexes

add indexes

add indexes

add indexes

add indexes


But it’ll help pick up the pieces

Before index
> db.vendors.ﬁndOne({'date': {'$gt': 1271895200}})
{ ... }
Sun Apr 25 01:19:51 query v_production.vendors ntoreturn:1 reslen:114 nscanned:~10M { date: { $gt: 1271895200.0 } } nreturned:1 3492ms

During index
> db.vendors.ensureIndex({'date': 1})
Sun Apr 25 01:20:17 building new index on { date: 1.0 } for v_production.vendors
Sun Apr 25 01:20:17 Buildindex v_production.vendors idxNo:2 { _id: ObjId(4bd398d1dbf523027368d566), ns: "v_production.vendors", key: { date: 1.0 }, name:
"date_1" }
! ! ...
Sun Apr 25 01:21:35 ! done building bottom layer, going to commit
Sun Apr 25 01:21:35 done for ~10M records 78.56secs
Sun Apr 25 01:21:35 insert v_production.system.indexes 78560ms

After Index
> db.vendors.ﬁnd({'date': 1272153600}).explain()
{
! ...
! "nscanned" : 11220,
! "millis" : 12,
! ...
}



Before index
{ ... }

Full collection scan ~3.5s

During index
"date_1" }
! ! ...
Sun Apr 25 01:21:35 insert v_production.system.indexes 78560ms

After Index
{
! ...
! "nscanned" : 11220,
! "millis" : 12,
! ...
}



Before index
{ ... }


During index
"date_1" }
! ! ...
Sun Apr 25 01:21:35 insert v_production.system.indexes 78560ms Index added ~79s
No locks.

After Index
{
! ...
! "nscanned" : 11220,
! "millis" : 12,
! ...
}



Before index
{ ... }


During index
"date_1" }
! ! ...
No locks.

After Index
{
! ...
! "nscanned" : 11220,
! "millis" : 12, 12ms!
! ...
}



Before index
{ ... }


During index
"date_1" }
! ! ...
No locks.

After Index
During index creation, system completely
{
! ...
CRUD responsive. No impact to
!
!
"nscanned" : 11220,
"millis" : 12, 12ms! MyPunchbowl.com
! ...
}


Up next: Top Secret Project


• Project involving


• SCREAMS for a document-based solution


• mongosphinx
•
MongoDB’s ad-hoc full text search ok, lacks inﬁx
matching and bulk index building
•
Geo-spatial search not yet ellipsoidal (or even spherical)


• mongosphinx
•
•
• paperclip+mongomapper


• mongosphinx
•
•
• MyPunchbowl user/event/invite data mining/visualization


• mongosphinx
•
•
• Charding + MapReduce = Nerdstorm


• mongosphinx
•
•
• Charding + MapReduce = Nerdstorm
• Building tools to speed up ETL out of MySQL and into
MongoDB

The end

Questions?


Contact Me

ryan@angilly.com
http://www.mypunchbowl.com
http://ryanangilly.com
@angilly


Links

Slide 3/4 logos, wget’d from all over the Internets.

Dunce Cap http://steynian.files.wordpress.com/2009/09/dunce-cap.jpg

http://railstips.org/blog/archives/2009/12/18/why-i-think-mongo-is-to-databases-what-rails-was-to-frameworks/

http://railstips.org/blog/archives/2009/11/10/config-so-simple-your-mama-could-use-it/

Vendors configuration http://gist.github.com/380213


Zero to Mongo in 60 Hours

More Related Content

Viewers also liked

Similar to Zero to Mongo in 60 Hours

More from MongoSF

Recently uploaded

Zero to Mongo in 60 Hours