Zero to Mongo in 60 Hours
Upcoming SlideShare
Loading in...5
×
 

Zero to Mongo in 60 Hours

on

  • 12,179 views

 

Statistics

Views

Total Views
12,179
Slideshare-icon Views on SlideShare
11,331
Embed Views
848

Actions

Likes
50
Downloads
476
Comments
1

7 Embeds 848

http://www.slideshare.net 588
http://www.nosqldatabases.com 159
http://www.10gen.com 89
http://www.mongodb.org 7
http://127.0.0.1:2194 2
http://translate.googleusercontent.com 2
http://localhost 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

11 of 1

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • = ENTREPRENEURS WANTED =

    We Are a Global Community of Entrepreneurs Looking for New Leaders to Increase the Synergy of our Qualified Team.

    We are Entrepreneurs Working Together from the Comfort of Home. Come Join Us, Let’s Do this Together!!

    Review: http://bit.ly/db01RT
    Website: http://myentrepreneuronlinebusiness.com
    <br /><object type="application/x-shockwave-flash" data="http://www.youtube.com/v/DKzEHqygElA&amp;hl=es_MX&amp;fs=1?rel=0" width="350" height="288"><param name="movie" value="http://www.youtube.com/v/DKzEHqygElA&amp;hl=es_MX&amp;fs=1?rel=0"></param><embed src="http://www.youtube.com/v/DKzEHqygElA&amp;hl=es_MX&amp;fs=1?rel=0" width="350" height="288" type="application/x-shockwave-flash"></embed></object>
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Zero to Mongo in 60 Hours Zero to Mongo in 60 Hours Presentation Transcript

    • Zero to Mongo in 60 Hours Ryan Angilly MyPunchbowl.com @angilly Wednesday, April 28, 2010
    • I’m a pretty awesome dude Wednesday, April 28, 2010
    • I’m a pretty awesome dude • Electrical Engineer by education Wednesday, April 28, 2010
    • I’m a pretty awesome dude • Electrical Engineer by education • Ex-EMC’er Wednesday, April 28, 2010
    • I’m a pretty awesome dude • Electrical Engineer by education • Ex-EMC’er • Founded MessageSling.com Wednesday, April 28, 2010
    • I’m a pretty awesome dude • Electrical Engineer by education • Ex-EMC’er • Founded MessageSling.com • Entered the deadpool in January Wednesday, April 28, 2010
    • I’m a pretty awesome dude • Electrical Engineer by education • Ex-EMC’er • Founded MessageSling.com • Entered the deadpool in January • Senior developer @ MyPunchbowl.com Wednesday, April 28, 2010
    • I’m a pretty awesome dude • Electrical Engineer by education • Built several web apps • Ex-EMC’er • Founded MessageSling.com • Entered the deadpool in January • Senior developer @ MyPunchbowl.com Wednesday, April 28, 2010
    • I’m a pretty awesome dude • Electrical Engineer by education • Built several web apps • Ex-EMC’er • All Ruby. Mostly Rails • Founded MessageSling.com • Entered the deadpool in January • Senior developer @ MyPunchbowl.com Wednesday, April 28, 2010
    • I’m a pretty awesome dude • Electrical Engineer by education • Built several web apps • Ex-EMC’er • All Ruby. Mostly Rails • Founded • All SQL MessageSling.com • Entered the deadpool in January • Senior developer @ MyPunchbowl.com Wednesday, April 28, 2010
    • I’m a pretty awesome dude • Electrical Engineer by education • Built several web apps • Ex-EMC’er • All Ruby. Mostly Rails • Founded • All SQL MessageSling.com • No experience w/ NoSQL/Mongo as of • Entered the deadpool in January September ’09 • Senior developer @ MyPunchbowl.com Wednesday, April 28, 2010
    • I’m a pretty awesome dude • Electrical Engineer by education • Built several web apps • Ex-EMC’er • All Ruby. Mostly Rails • Founded • All SQL MessageSling.com • No experience w/ NoSQL/Mongo as of • Entered the deadpool in January September ’09 • Senior developer @ • Want to provide a beginner’s perspective MyPunchbowl.com Wednesday, April 28, 2010
    • MyPunchbowl is a pretty awesome company Wednesday, April 28, 2010
    • MyPunchbowl is a pretty awesome company • Leader in start to finish party planning Wednesday, April 28, 2010
    • MyPunchbowl is a pretty awesome company • Leader in start to finish party planning • Connecting party planners to party vendors -- subject of this talk Wednesday, April 28, 2010
    • MyPunchbowl is a pretty awesome company • Leader in start to finish party planning • Connecting party planners to party vendors -- subject of this talk • Currently employs 11 people Wednesday, April 28, 2010
    • MyPunchbowl is a pretty awesome company • Leader in start to finish party planning • Connecting party planners to party vendors -- subject of this talk • Currently employs 11 people • Funded over 2 rounds by Intel & Contour Wednesday, April 28, 2010
    • MyPunchbowl is a pretty awesome company • Leader in start to finish party planning • Connecting party planners to party vendors -- subject of this talk • Currently employs 11 people • Funded over 2 rounds by Intel & Contour • Strategic partnership w/ OTC among others Wednesday, April 28, 2010
    • MyPunchbowl is a pretty awesome company • Leader in start to finish party planning • Connecting party planners to party vendors -- subject of this talk • Currently employs 11 people • Funded over 2 rounds by Intel & Contour • Strategic partnership w/ OTC among others • Amount of traffic, users, events, invites, vendors, etc... make our data sets large enough to be interesting :) Wednesday, April 28, 2010
    • MyPunchbowl is a pretty awesome company • Leader in start to finish party planning • Connecting party planners to party vendors -- subject of this talk • Currently employs 11 people • Funded over 2 rounds by Intel & Contour • Strategic partnership w/ OTC among others • Amount of traffic, users, events, invites, vendors, etc... make our data sets large enough to be interesting :) • We’re talking many millions of ‘things’ Wednesday, April 28, 2010
    • Our engineers play with everything Wednesday, April 28, 2010
    • Our engineers play with everything Wednesday, April 28, 2010
    • Our engineers play with everything Wednesday, April 28, 2010
    • Our engineers play with everything Wednesday, April 28, 2010
    • Our engineers play with everything Wednesday, April 28, 2010
    • Our engineers play with everything Wednesday, April 28, 2010
    • Our engineers play with everything Wednesday, April 28, 2010
    • Our engineers play with everything Wednesday, April 28, 2010
    • Our engineers play with everything Wednesday, April 28, 2010
    • Our engineers play with everything Wednesday, April 28, 2010
    • Our engineers play with everything Wednesday, April 28, 2010
    • Our engineers play with everything Wednesday, April 28, 2010
    • Our engineers play with everything Wednesday, April 28, 2010
    • Our engineers play with everything Wednesday, April 28, 2010
    • Wednesday, April 28, 2010
    • Search by category and location Wednesday, April 28, 2010
    • Search by business name Search by category and and location location Wednesday, April 28, 2010
    • Wednesday, April 28, 2010
    • Track searches Wednesday, April 28, 2010
    • Track searches Track vendor impressions Wednesday, April 28, 2010
    • MongoDB + MyPunchbowl A tale in conversations Wednesday, April 28, 2010
    • Tracking requirement a good excuse to finally use MongoDB Wednesday, April 28, 2010
    • Tracking requirement a good excuse to finally use MongoDB Me: Hey Blake I’m gonna use MongoDB to keep track of all this search stuff. Wednesday, April 28, 2010
    • Tracking requirement a good excuse to finally use MongoDB Me: Hey Blake I’m gonna use MongoDB to keep track of all this search stuff. Blake: Cool. Wednesday, April 28, 2010
    • Tracking requirement a good excuse to finally use MongoDB Me: Hey Blake I’m gonna use MongoDB to keep track of all this search stuff. Blake: Cool. Matt (CEO): What’s that? Wednesday, April 28, 2010
    • Tracking requirement a good excuse to finally use MongoDB Me: Hey Blake I’m gonna use MongoDB to keep track of all this search stuff. Blake: Cool. Matt (CEO): What’s that? Blake & Me: Something cool. Wednesday, April 28, 2010
    • Tracking requirement a good excuse to finally use MongoDB Me: Hey Blake I’m gonna use MongoDB to keep track of all this search stuff. Blake: Cool. Matt (CEO): What’s that? Blake & Me: Something cool. Matt (CEO): K Wednesday, April 28, 2010
    • 6 reasons to use MongoDB • Easy to get running (~5 minutes on OSX) • Open Source • Support in multiple (computer) languages • Prototype in Ruby, move to Java if needed • Very active development • Full featured • Great ecosystem Wednesday, April 28, 2010
    • MongoDB feels right Wednesday, April 28, 2010
    • MongoDB feels right Me: MongoDB gives me the warm fuzzies that Rails did. Wednesday, April 28, 2010
    • MongoDB feels right Me: MongoDB gives me the warm fuzzies that Rails did. Darren: Just like that Nunemaker post. Wednesday, April 28, 2010
    • MongoDB feels right Me: MongoDB gives me the warm fuzzies that Rails did. Darren: Just like that Nunemaker post. Me: ? *runs to google* Wednesday, April 28, 2010
    • MongoDB feels right Me: MongoDB gives me the warm fuzzies that Rails did. Darren: Just like that Nunemaker post. Me: ? *runs to google* Wednesday, April 28, 2010
    • Support is INSANE 11:35pm. Wednesday. Founder. 60 second response time. Wednesday, April 28, 2010
    • 2.5 days? Really? Wednesday, April 28, 2010
    • 2.5 days? Really? Yes Wednesday, April 28, 2010
    • 2.5 days? Really? Yes* Wednesday, April 28, 2010
    • 2.5 days to MongoDB 0 Days 1 Day 2 Days 200+ Days and on... Wednesday, April 28, 2010
    • 2.5 days to MongoDB Decision to use MongoDB 0 Days 1 Day 2 Days 200+ Days and on... Wednesday, April 28, 2010
    • 2.5 days to MongoDB Decision to use MongoDB 0 Days 1 Day 2 Days 200+ Days and on... Play around with available OSS Wednesday, April 28, 2010
    • 2.5 days to MongoDB Decision to use MongoDB Figure out document ‘schema’ 0 Days 1 Day 2 Days 200+ Days and on... Play around with available OSS Wednesday, April 28, 2010
    • 2.5 days to MongoDB Decision to use MongoDB Figure out document ‘schema’ 0 Days 1 Day 2 Days 200+ Days and on... Play around with available OSS Write some stuff on top of mongo-ruby-driver Wednesday, April 28, 2010
    • 2.5 days to MongoDB Decision Build test rig to use MongoDB Figure out document ‘schema’ 0 Days 1 Day 2 Days 200+ Days and on... Play around with available OSS Write some stuff on top of mongo-ruby-driver Wednesday, April 28, 2010
    • 2.5 days to MongoDB Decision Build test rig to use MongoDB Figure out document ‘schema’ 0 Days 1 Day 2 Days 200+ Days and on... Play around with Setup Configuration available OSS Management Write some stuff on top of mongo-ruby-driver Wednesday, April 28, 2010
    • 2.5 days to MongoDB Decision Build test rig to use MongoDB Figure out document Get production ready ‘schema’ 0 Days 1 Day 2 Days 200+ Days and on... Play around with Setup Configuration available OSS Management Write some stuff on top of mongo-ruby-driver Wednesday, April 28, 2010
    • 2.5 days to MongoDB Decision Build test rig to use MongoDB Figure out document Get production ready ‘schema’ 0 Days 1 Day 2 Days 200+ Days and on... Play around with Setup Configuration available OSS Management Write some stuff on top Deploy of mongo-ruby-driver Wednesday, April 28, 2010
    • A stroll through Mongo’s OSS ecosystem Wednesday, April 28, 2010
    • A stroll through Mongo’s OSS ecosystem • mongodb (C++) Wednesday, April 28, 2010
    • A stroll through Mongo’s OSS ecosystem • mongodb (C++) • mongo shell (JS via SpiderMonkey) Wednesday, April 28, 2010
    • A stroll through Mongo’s OSS ecosystem • mongodb (C++) • mongo shell (JS via SpiderMonkey) • mongo-ruby-driver Wednesday, April 28, 2010
    • A stroll through Mongo’s OSS ecosystem • mongodb (C++) • mongo shell (JS via SpiderMonkey) • mongo-ruby-driver • mongo-java-driver Wednesday, April 28, 2010
    • A stroll through Mongo’s OSS ecosystem • mongodb (C++) • mongo shell (JS via SpiderMonkey) • mongo-ruby-driver • mongo-java-driver • mongo_record (Ruby) Wednesday, April 28, 2010
    • A stroll through Mongo’s OSS ecosystem • mongodb (C++) • mongo shell (JS via SpiderMonkey) • mongo-ruby-driver • mongo-java-driver • mongo_record (Ruby) • mongo_mapper (Ruby) Wednesday, April 28, 2010
    • A stroll through Mongo’s OSS ecosystem • mongodb (C++) • mongo shell (JS via SpiderMonkey) • mongo-ruby-driver • mongo-java-driver • mongo_record (Ruby) • mongo_mapper (Ruby) • Integration w/ Rails Wednesday, April 28, 2010
    • Document ‘schema’ Wednesday, April 28, 2010
    • Document ‘schema’ • What do documents look like? Wednesday, April 28, 2010
    • Document ‘schema’ • What do documents look like? • Example Document { city: “Boston”, state: “MA”, date: 1272153600, occurrences: 1, category_id: 1 } Wednesday, April 28, 2010
    • Document ‘schema’ • What do documents look like? • How do you update the documents? • Example Document { city: “Boston”, state: “MA”, date: 1272153600, occurrences: 1, category_id: 1 } Wednesday, April 28, 2010
    • Document ‘schema’ • What do documents look like? • How do you update the documents? • Example Document • Mongo shell command { coll.update({‘city’: ‘Boston’, city: “Boston”, ‘state’: ‘MA’, state: “MA”, ‘date’: 1272153600, date: 1272153600, ‘category_id’: 1}, occurrences: 1, {‘$inc’: {‘occurrences’: 1}}, category_id: 1 true) } Wednesday, April 28, 2010
    • Document ‘schema’ • What do documents look like? • How do you update the documents? • Example Document • Mongo shell command { coll.update({‘city’: ‘Boston’, city: “Boston”, ‘state’: ‘MA’, state: “MA”, ‘date’: 1272153600, date: 1272153600, ‘category_id’: 1}, occurrences: 1, {‘$inc’: {‘occurrences’: 1}}, category_id: 1 true) } Document to match Wednesday, April 28, 2010
    • Document ‘schema’ • What do documents look like? • How do you update the documents? • Example Document • Mongo shell command { coll.update({‘city’: ‘Boston’, city: “Boston”, ‘state’: ‘MA’, state: “MA”, ‘date’: 1272153600, date: 1272153600, ‘category_id’: 1}, occurrences: 1, {‘$inc’: {‘occurrences’: 1}}, category_id: 1 true) } Document to match Operation to perform Wednesday, April 28, 2010
    • Document ‘schema’ • What do documents look like? • How do you update the documents? • Example Document • Mongo shell command { coll.update({‘city’: ‘Boston’, city: “Boston”, ‘state’: ‘MA’, state: “MA”, ‘date’: 1272153600, date: 1272153600, ‘category_id’: 1}, occurrences: 1, {‘$inc’: {‘occurrences’: 1}}, category_id: 1 true) } Document to match Operation to perform ‘upsert’: update if it’s there insert if it’s not Wednesday, April 28, 2010
    • Cuhhraazy Indexes Wednesday, April 28, 2010
    • Cuhhraazy Indexes • Original Document { city: “Boston”, state: “MA”, date: 1272153600, occurrences: 1, category_id: 1 } Wednesday, April 28, 2010
    • Cuhhraazy Indexes • Original Document { city: “Boston”, state: “MA”, date: 1272153600, occurrences: 1, category_id: 1 } • 40k cities, 200 days, 60 categories Wednesday, April 28, 2010
    • Cuhhraazy Indexes • Original Document { city: “Boston”, state: “MA”, date: 1272153600, occurrences: 1, category_id: 1 } • 40k cities, 200 days, 60 categories • 480M potential documents Wednesday, April 28, 2010
    • Cuhhraazy Indexes • Original Document { city: “Boston”, state: “MA”, date: 1272153600, occurrences: 1, category_id: 1 } • 40k cities, 200 days, 60 categories • 480M potential documents • Composite indexes • state_1_city_1 • state_1_date_1_city_1 • category_id_1_date_1 • date_1 Wednesday, April 28, 2010
    • Cuhhraazy Indexes • Original Document { city: “Boston”, state: “MA”, date: 1272153600, occurrences: 1, category_id: 1 } • 40k cities, 200 days, 60 categories • 480M potential documents • Composite indexes • state_1_city_1 • state_1_date_1_city_1 • category_id_1_date_1 • date_1 • Flexibility & performance in querying and aggregating Wednesday, April 28, 2010
    • Cuhhraazy Indexes • Original Document • More Complex Document w/ Embedded Document { { city: “Boston”, city: “Boston”, state: “MA”, state: “MA”, date: 1272153600, date: 1272153600, occurrences: 1, occurrences: 9581555, category_id: 1 category_id: 1, } mobile_sources: { • 40k cities, 200 days, 60 categories browsers: { windows_mobile: 1, • 480M potential documents palm_os: 0, • Composite indexes }, iphone_4g: 9481554 • state_1_city_1 zip_codes: [“01518”] • state_1_date_1_city_1 } • category_id_1_date_1 } • date_1 • Flexibility & performance in querying and aggregating Wednesday, April 28, 2010
    • Cuhhraazy Indexes • Original Document • More Complex Document w/ Embedded Document { { city: “Boston”, city: “Boston”, state: “MA”, state: “MA”, date: 1272153600, date: 1272153600, occurrences: 1, occurrences: 9581555, category_id: 1 category_id: 1, } mobile_sources: { • 40k cities, 200 days, 60 categories browsers: { windows_mobile: 1, • 480M potential documents palm_os: 0, • Composite indexes }, iphone_4g: 9481554 • state_1_city_1 zip_codes: [“01518”] • state_1_date_1_city_1 } • category_id_1_date_1 } • date_1 • Additional, complex data down the road? • Flexibility & performance in querying and aggregating Wednesday, April 28, 2010
    • Cuhhraazy Indexes • Original Document • More Complex Document w/ Embedded Document { { city: “Boston”, city: “Boston”, state: “MA”, state: “MA”, date: 1272153600, date: 1272153600, occurrences: 1, occurrences: 9581555, category_id: 1 category_id: 1, } mobile_sources: { • 40k cities, 200 days, 60 categories browsers: { windows_mobile: 1, • 480M potential documents palm_os: 0, • Composite indexes }, iphone_4g: 9481554 • state_1_city_1 zip_codes: [“01518”] • state_1_date_1_city_1 } • category_id_1_date_1 } • date_1 • Additional, complex data down the road? • Flexibility & performance in • Add deep embedded indexes! querying and aggregating Wednesday, April 28, 2010
    • Cuhhraazy Indexes • Original Document • More Complex Document w/ Embedded Document { { city: “Boston”, city: “Boston”, state: “MA”, state: “MA”, date: 1272153600, date: 1272153600, occurrences: 1, occurrences: 9581555, category_id: 1 category_id: 1, } mobile_sources: { • 40k cities, 200 days, 60 categories browsers: { windows_mobile: 1, • 480M potential documents palm_os: 0, • Composite indexes }, iphone_4g: 9481554 • state_1_city_1 zip_codes: [“01518”] • state_1_date_1_city_1 } • category_id_1_date_1 } • date_1 • Additional, complex data down the road? • Flexibility & performance in • Add deep embedded indexes! querying and aggregating • ‘mobile_sources.zip_codes’ Wednesday, April 28, 2010
    • Cuhhraazy Indexes • Original Document • More Complex Document w/ Embedded Document { { city: “Boston”, city: “Boston”, state: “MA”, state: “MA”, date: 1272153600, date: 1272153600, occurrences: 1, occurrences: 9581555, category_id: 1 category_id: 1, } mobile_sources: { • 40k cities, 200 days, 60 categories browsers: { windows_mobile: 1, • 480M potential documents palm_os: 0, • Composite indexes }, iphone_4g: 9481554 • state_1_city_1 zip_codes: [“01518”] • state_1_date_1_city_1 } • category_id_1_date_1 } • date_1 • Additional, complex data down the road? • Flexibility & performance in • Add deep embedded indexes! querying and aggregating • ‘mobile_sources.zip_codes’ • ‘mobile_sources.browsers.iphone_4g’ Wednesday, April 28, 2010
    • Let’s write some Ruby Wednesday, April 28, 2010
    • Let’s write some Ruby class SearchStats cattr_accessor :db, :collection self.collection = $mongo_db.collection ‘searches‘ def self.record_search(city, state, category_id, vendor_id) collection.update({:vendor_id => vendor_id, :city => city, :state => state, :category_id => category_id, :date => Time.now.utc.beginning_of_day.to_i}, {'$inc' => {:occurrence => 1}}, {:upsert => true}) end end Wednesday, April 28, 2010
    • Let’s write some Ruby class SearchStats cattr_accessor :db, :collection self.collection = $mongo_db.collection ‘searches‘ def self.record_search(city, state, category_id, vendor_id) collection.update({:vendor_id => vendor_id, :city => city, :state => state, :category_id => category_id, :date => Time.now.utc.beginning_of_day.to_i}, {'$inc' => {:occurrence => 1}}, {:upsert => true}) end end irb> SearchStats.record_search(‘Boston’, ‘MA’, 9, 42) Wednesday, April 28, 2010
    • Let’s write some Ruby class SearchStats cattr_accessor :db, :collection self.collection = $mongo_db.collection ‘searches‘ def self.record_search(city, state, category_id, vendor_id) collection.update({:vendor_id => vendor_id, :city => city, :state => state, :category_id => category_id, :date => Time.now.utc.beginning_of_day.to_i}, {'$inc' => {:occurrence => 1}}, {:upsert => true}) end end irb> SearchStats.record_search(‘Boston’, ‘MA’, 9, 42) mongo> db.searches.findOne() { "city" : "Boston", "date" : 1272326400, "impression" : 1, "category_id" : 9, "state" : "MA", "vendor_id" : 42 } Wednesday, April 28, 2010
    • TESTING Wednesday, April 28, 2010
    • TESTING • Pretty much the same as anything else Wednesday, April 28, 2010
    • TESTING • Pretty much the same as anything else • test/unit/search_stats_test.rb Wednesday, April 28, 2010
    • TESTING • Pretty much the same as anything else • test/unit/search_stats_test.rb • config/environments/test.rb Wednesday, April 28, 2010
    • TESTING • Pretty much the same as anything else • test/unit/search_stats_test.rb • config/environments/test.rb • $mongo_db = Mongo::Connection.new.db ‘mongo-sf-db’ Wednesday, April 28, 2010
    • TESTING • Pretty much the same as anything else • test/unit/search_stats_test.rb • config/environments/test.rb • $mongo_db = Mongo::Connection.new.db ‘mongo-sf-db’ • SearchStatsTest#setup & #teardown Wednesday, April 28, 2010
    • TESTING • Pretty much the same as anything else • test/unit/search_stats_test.rb • config/environments/test.rb • $mongo_db = Mongo::Connection.new.db ‘mongo-sf-db’ • SearchStatsTest#setup & #teardown • No transactions Wednesday, April 28, 2010
    • TESTING • Pretty much the same as anything else • test/unit/search_stats_test.rb • config/environments/test.rb • $mongo_db = Mongo::Connection.new.db ‘mongo-sf-db’ • SearchStatsTest#setup & #teardown • No transactions • ActiveSupport spoils us Wednesday, April 28, 2010
    • TESTING • Pretty much the same as anything else • test/unit/search_stats_test.rb • config/environments/test.rb • $mongo_db = Mongo::Connection.new.db ‘mongo-sf-db’ • SearchStatsTest#setup & #teardown • No transactions • ActiveSupport spoils us • Gotta clean up after yourself Wednesday, April 28, 2010
    • Other developers gotta use it too Wednesday, April 28, 2010
    • Other developers gotta use it too • Rails’ database.yml takes care of MySQL Wednesday, April 28, 2010
    • Other developers gotta use it too • Rails’ database.yml takes care of MySQL • Gotta switch up MongoDBs on your own Wednesday, April 28, 2010
    • Other developers gotta use it too • Rails’ database.yml takes care of MySQL • Gotta switch up MongoDBs on your own • Simple config system file inspired by another Nunemaker post. Wednesday, April 28, 2010
    • Other developers gotta use it too • Rails’ database.yml takes care of MySQL • Gotta switch up MongoDBs on your own • Simple config system file inspired by another Nunemaker post. • No database migrations! Wednesday, April 28, 2010
    • Other developers gotta use it too • Rails’ database.yml takes care of MySQL • Gotta switch up MongoDBs on your own • Simple config system file inspired by another Nunemaker post. • No database migrations! • Must write rake tasks to create indexes unless your ORM takes care of it for you Wednesday, April 28, 2010
    • Making production ready Wednesday, April 28, 2010
    • Making production ready • Replication was iffy in 1.0.0 (solid now) Wednesday, April 28, 2010
    • Making production ready • Replication was iffy in 1.0.0 (solid now) • Nightly dumps w/ mongoexport Wednesday, April 28, 2010
    • Making production ready • Replication was iffy in 1.0.0 (solid now) • Nightly dumps w/ mongoexport • Upload to S3 Wednesday, April 28, 2010
    • Making production ready • Replication was iffy in 1.0.0 (solid now) • Nightly dumps w/ mongoexport • Upload to S3 • Monitor process w/ monit Wednesday, April 28, 2010
    • Deploy Wednesday, April 28, 2010
    • Deploy • Original deployment uneventful Wednesday, April 28, 2010
    • Deploy • Original deployment uneventful • Just worked Wednesday, April 28, 2010
    • Deploy • Original deployment uneventful • Just worked • Future deployments that required a database upgrade required downtime (bummer) Wednesday, April 28, 2010
    • Deploy • Original deployment uneventful • Just worked • Future deployments that required a database upgrade required downtime (bummer) • We wanted to play with the geospatial search Wednesday, April 28, 2010
    • Day 200 Wednesday, April 28, 2010
    • Day 200 • One production process Wednesday, April 28, 2010
    • Day 200 • One production process • One database Wednesday, April 28, 2010
    • Day 200 • One production process • One database • Daily dumps using mongoexport + upload to S3 Wednesday, April 28, 2010
    • Day 200 • One production process • One database • Daily dumps using mongoexport + upload to S3 • Replication not running Wednesday, April 28, 2010
    • Day 200 • One production process • One database • Daily dumps using mongoexport + upload to S3 • Replication not running • Several collections Wednesday, April 28, 2010
    • Day 200 • One production process • One database • Daily dumps using mongoexport + upload to S3 • Replication not running • Several collections • 100k to >10M documents Wednesday, April 28, 2010
    • Day 200 • One production process • One database • Daily dumps using mongoexport + upload to S3 • Replication not running • Several collections • 100k to >10M documents • Scores of deep, composite indexes Wednesday, April 28, 2010
    • What tripped me up? Wednesday, April 28, 2010
    • Simple Test should "get impressions_by_date" do record1 = {:date => 100, :impression => 10} record2 = {:date => 90, :impression => 8} Mongo::Vendor.collection.insert record1 Mongo::Vendor.collection.insert record2 assert_same_elements [record1, record2], Mongo::Vendor.impressions_by_date end Wednesday, April 28, 2010
    • Crazy Failure 1) Failure: test: Mongo::Vendor should get impressions_by_date. (Mongo::VendorTest) ... -<{{"date"=>100.0, "csum"=>10.0}=>1, {"date"=>90.0, "csum"=>8.0}=>1}> +<{{"date"=>90.0, "csum"=>8.0}=>1, {"date"=>100.0, "csum"=>10.0}=>1}> Wednesday, April 28, 2010
    • Wednesday, April 28, 2010
    • Mongo’s BSON OrderedHash doesn’t behave like ActiveSupport::OrderedHash Wednesday, April 28, 2010
    • Mongo’s BSON OrderedHash doesn’t behave like ActiveSupport::OrderedHash irb(main):001:0> require 'active_support' irb(main):002:0> include ActiveSupport irb(main):003:0> oh = OrderedHash.new irb(main):004:0> oh[:b] = 2 irb(main):005:0> oh[:c] = 3 irb(main):006:0> oh[:a] = 1 irb(main):007:0> a = {:a => 1, :b => 2, :c => 3} irb(main):008:0> a == oh => true irb(main):009:0> oh == a => true Wednesday, April 28, 2010
    • Mongo’s BSON OrderedHash doesn’t behave like ActiveSupport::OrderedHash irb(main):001:0> require 'active_support' irb(main):001:0> require 'mongo' irb(main):002:0> include ActiveSupport irb(main):002:0> oh = OrderedHash.new irb(main):003:0> oh = OrderedHash.new irb(main):003:0> oh[:b] = 2 irb(main):004:0> oh[:b] = 2 irb(main):004:0> oh[:c] = 3 irb(main):005:0> oh[:c] = 3 irb(main):005:0> oh[:a] = 1 irb(main):006:0> oh[:a] = 1 irb(main):006:0> a = {:a => 1, :b => 2, :c irb(main):007:0> a = {:a => 1, :b => 2, :c => 3} => 3} irb(main):008:0> a == oh irb(main):007:0> a == oh => true => true irb(main):009:0> oh == a => true Wednesday, April 28, 2010
    • Mongo’s BSON OrderedHash doesn’t behave like ActiveSupport::OrderedHash irb(main):001:0> require 'active_support' irb(main):001:0> require 'mongo' irb(main):002:0> include ActiveSupport irb(main):002:0> oh = OrderedHash.new irb(main):003:0> oh = OrderedHash.new irb(main):003:0> oh[:b] = 2 irb(main):004:0> oh[:b] = 2 irb(main):004:0> oh[:c] = 3 irb(main):005:0> oh[:c] = 3 irb(main):005:0> oh[:a] = 1 irb(main):006:0> oh[:a] = 1 irb(main):006:0> a = {:a => 1, :b => 2, :c irb(main):007:0> a = {:a => 1, :b => 2, :c => 3} => 3} irb(main):008:0> a == oh irb(main):007:0> a == oh => true => true irb(main):009:0> oh == a irb(main):008:0> oh == a => true => false Wednesday, April 28, 2010
    • Wednesday, April 28, 2010
    • And it turns out... Wednesday, April 28, 2010
    • And it turns out... MongoDB will not add indexes to your collections. Wednesday, April 28, 2010
    • And it turns out... MongoDB will not add indexes to your collections. Unless you tell it to. Wednesday, April 28, 2010
    • add indexes add indexes add indexes add indexes add indexes add indexes add indexes add indexes add indexes add indexes add indexes add indexes add indexes Wednesday, April 28, 2010
    • But it’ll help pick up the pieces Before index > db.vendors.findOne({'date': {'$gt': 1271895200}}) { ... } Sun Apr 25 01:19:51 query v_production.vendors ntoreturn:1 reslen:114 nscanned:~10M { date: { $gt: 1271895200.0 } } nreturned:1 3492ms During index > db.vendors.ensureIndex({'date': 1}) Sun Apr 25 01:20:17 building new index on { date: 1.0 } for v_production.vendors Sun Apr 25 01:20:17 Buildindex v_production.vendors idxNo:2 { _id: ObjId(4bd398d1dbf523027368d566), ns: "v_production.vendors", key: { date: 1.0 }, name: "date_1" } ! ! ... Sun Apr 25 01:21:35 ! done building bottom layer, going to commit Sun Apr 25 01:21:35 done for ~10M records 78.56secs Sun Apr 25 01:21:35 insert v_production.system.indexes 78560ms After Index > db.vendors.find({'date': 1272153600}).explain() { ! ... ! "nscanned" : 11220, ! "millis" : 12, ! ... } Wednesday, April 28, 2010
    • But it’ll help pick up the pieces Before index > db.vendors.findOne({'date': {'$gt': 1271895200}}) { ... } Sun Apr 25 01:19:51 query v_production.vendors ntoreturn:1 reslen:114 nscanned:~10M { date: { $gt: 1271895200.0 } } nreturned:1 3492ms Full collection scan ~3.5s During index > db.vendors.ensureIndex({'date': 1}) Sun Apr 25 01:20:17 building new index on { date: 1.0 } for v_production.vendors Sun Apr 25 01:20:17 Buildindex v_production.vendors idxNo:2 { _id: ObjId(4bd398d1dbf523027368d566), ns: "v_production.vendors", key: { date: 1.0 }, name: "date_1" } ! ! ... Sun Apr 25 01:21:35 ! done building bottom layer, going to commit Sun Apr 25 01:21:35 done for ~10M records 78.56secs Sun Apr 25 01:21:35 insert v_production.system.indexes 78560ms After Index > db.vendors.find({'date': 1272153600}).explain() { ! ... ! "nscanned" : 11220, ! "millis" : 12, ! ... } Wednesday, April 28, 2010
    • But it’ll help pick up the pieces Before index > db.vendors.findOne({'date': {'$gt': 1271895200}}) { ... } Sun Apr 25 01:19:51 query v_production.vendors ntoreturn:1 reslen:114 nscanned:~10M { date: { $gt: 1271895200.0 } } nreturned:1 3492ms Full collection scan ~3.5s During index > db.vendors.ensureIndex({'date': 1}) Sun Apr 25 01:20:17 building new index on { date: 1.0 } for v_production.vendors Sun Apr 25 01:20:17 Buildindex v_production.vendors idxNo:2 { _id: ObjId(4bd398d1dbf523027368d566), ns: "v_production.vendors", key: { date: 1.0 }, name: "date_1" } ! ! ... Sun Apr 25 01:21:35 ! done building bottom layer, going to commit Sun Apr 25 01:21:35 done for ~10M records 78.56secs Sun Apr 25 01:21:35 insert v_production.system.indexes 78560ms Index added ~79s No locks. After Index > db.vendors.find({'date': 1272153600}).explain() { ! ... ! "nscanned" : 11220, ! "millis" : 12, ! ... } Wednesday, April 28, 2010
    • But it’ll help pick up the pieces Before index > db.vendors.findOne({'date': {'$gt': 1271895200}}) { ... } Sun Apr 25 01:19:51 query v_production.vendors ntoreturn:1 reslen:114 nscanned:~10M { date: { $gt: 1271895200.0 } } nreturned:1 3492ms Full collection scan ~3.5s During index > db.vendors.ensureIndex({'date': 1}) Sun Apr 25 01:20:17 building new index on { date: 1.0 } for v_production.vendors Sun Apr 25 01:20:17 Buildindex v_production.vendors idxNo:2 { _id: ObjId(4bd398d1dbf523027368d566), ns: "v_production.vendors", key: { date: 1.0 }, name: "date_1" } ! ! ... Sun Apr 25 01:21:35 ! done building bottom layer, going to commit Sun Apr 25 01:21:35 done for ~10M records 78.56secs Sun Apr 25 01:21:35 insert v_production.system.indexes 78560ms Index added ~79s No locks. After Index > db.vendors.find({'date': 1272153600}).explain() { ! ... ! "nscanned" : 11220, ! "millis" : 12, 12ms! ! ... } Wednesday, April 28, 2010
    • But it’ll help pick up the pieces Before index > db.vendors.findOne({'date': {'$gt': 1271895200}}) { ... } Sun Apr 25 01:19:51 query v_production.vendors ntoreturn:1 reslen:114 nscanned:~10M { date: { $gt: 1271895200.0 } } nreturned:1 3492ms Full collection scan ~3.5s During index > db.vendors.ensureIndex({'date': 1}) Sun Apr 25 01:20:17 building new index on { date: 1.0 } for v_production.vendors Sun Apr 25 01:20:17 Buildindex v_production.vendors idxNo:2 { _id: ObjId(4bd398d1dbf523027368d566), ns: "v_production.vendors", key: { date: 1.0 }, name: "date_1" } ! ! ... Sun Apr 25 01:21:35 ! done building bottom layer, going to commit Sun Apr 25 01:21:35 done for ~10M records 78.56secs Sun Apr 25 01:21:35 insert v_production.system.indexes 78560ms Index added ~79s No locks. After Index > db.vendors.find({'date': 1272153600}).explain() During index creation, system completely { ! ... CRUD responsive. No impact to ! ! "nscanned" : 11220, "millis" : 12, 12ms! MyPunchbowl.com ! ... } Wednesday, April 28, 2010
    • Up next: Top Secret Project Wednesday, April 28, 2010
    • Up next: Top Secret Project • Project involving Wednesday, April 28, 2010
    • Up next: Top Secret Project • Project involving • SCREAMS for a document-based solution Wednesday, April 28, 2010
    • Up next: Top Secret Project • Project involving • SCREAMS for a document-based solution • mongosphinx • MongoDB’s ad-hoc full text search ok, lacks infix matching and bulk index building • Geo-spatial search not yet ellipsoidal (or even spherical) Wednesday, April 28, 2010
    • Up next: Top Secret Project • Project involving • SCREAMS for a document-based solution • mongosphinx • MongoDB’s ad-hoc full text search ok, lacks infix matching and bulk index building • Geo-spatial search not yet ellipsoidal (or even spherical) • paperclip+mongomapper Wednesday, April 28, 2010
    • Up next: Top Secret Project • Project involving • SCREAMS for a document-based solution • mongosphinx • MongoDB’s ad-hoc full text search ok, lacks infix matching and bulk index building • Geo-spatial search not yet ellipsoidal (or even spherical) • paperclip+mongomapper • MyPunchbowl user/event/invite data mining/visualization Wednesday, April 28, 2010
    • Up next: Top Secret Project • Project involving • SCREAMS for a document-based solution • mongosphinx • MongoDB’s ad-hoc full text search ok, lacks infix matching and bulk index building • Geo-spatial search not yet ellipsoidal (or even spherical) • paperclip+mongomapper • MyPunchbowl user/event/invite data mining/visualization • Charding + MapReduce = Nerdstorm Wednesday, April 28, 2010
    • Up next: Top Secret Project • Project involving • SCREAMS for a document-based solution • mongosphinx • MongoDB’s ad-hoc full text search ok, lacks infix matching and bulk index building • Geo-spatial search not yet ellipsoidal (or even spherical) • paperclip+mongomapper • MyPunchbowl user/event/invite data mining/visualization • Charding + MapReduce = Nerdstorm • Building tools to speed up ETL out of MySQL and into MongoDB Wednesday, April 28, 2010
    • The end Questions? Wednesday, April 28, 2010
    • Contact Me ryan@angilly.com http://www.mypunchbowl.com http://ryanangilly.com @angilly Wednesday, April 28, 2010
    • Links Slide 3/4 logos, wget’d from all over the Internets. Dunce Cap http://steynian.files.wordpress.com/2009/09/dunce-cap.jpg http://railstips.org/blog/archives/2009/12/18/why-i-think-mongo-is-to-databases-what-rails-was-to-frameworks/ http://railstips.org/blog/archives/2009/11/10/config-so-simple-your-mama-could-use-it/ Vendors configuration http://gist.github.com/380213 Wednesday, April 28, 2010