• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Scalable Event Analytics with MongoDB & Ruby on Rails
 

Scalable Event Analytics with MongoDB & Ruby on Rails

on

  • 30,548 views

Slides from my talk at RubyConfChina 2010.

Slides from my talk at RubyConfChina 2010.

Statistics

Views

Total Views
30,548
Views on SlideShare
26,418
Embed Views
4,130

Actions

Likes
108
Downloads
222
Comments
7

30 Embeds 4,130

http://nosql.mypopescu.com 1578
http://mobicon.tistory.com 1539
http://blog.yottaa.com 317
http://www.nosqldatabases.com 307
http://www.yottaa.com 85
http://www.mongodb.org 62
http://mobimongo.tistory.com 49
http://www.febryadi.com 46
http://www.twylah.com 38
http://mobibackend.tistory.com 20
http://paper.li 18
http://localhost 13
http://yottaa1.web12.hubspot.com 10
http://translate.googleusercontent.com 7
http://www.linkedin.com 6
http://www.slideshare.net 6
http://blog.naver.com 6
https://twitter.com 4
http://mobifrontend.tistory.com 4
http://webcache.googleusercontent.com 3
http://seekr-artemis.heroku.com 2
http://static.slidesharecdn.com 2
http://fever.leehorrocks.com 1
https://abs.twimg.com 1
http://www.techgig.com 1
http://mongodb.nosqlclub.com 1
http://twitter.com 1
http://us-w1.rockmelt.com 1
http://b.hatena.ne.jp 1
http://a0.twimg.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

CC Attribution-ShareAlike LicenseCC Attribution-ShareAlike License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

17 of 7 previous next Post a comment

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Scalable Event Analytics with MongoDB & Ruby on Rails Scalable Event Analytics with MongoDB & Ruby on Rails Presentation Transcript

    • Scalable Event Analytics with Ruby on Rails & MongoDB
      Ruby Conf China 2010
      Jared Rosoff (@forjared) jrosoff@yottaa.com
    • Yottaa!!!! (www.yottaa.com)
    • Overview
      Ruby at Scale
      What is Event Analytics?
      What are the different ways you could do it?
      How we did it
    • Ruby At Scale?
      http://www.flickr.com/photos/laughingsquid
    • Event Analytics
      Data Source
      Event
      User
      Event
      Event
      Query
      Event Analytics
      Data Source
      Report
      Event
      Event
      Event
      Event
      User
      Query
      Data Source
      Event
      Event
      Report
      Event
      Event
      Event
      Data Source
    • Oh and we are a startup
    • Our requirements:
    • Rails default architecture
      Collection Server
      Data Source
      MySQL
      User
      Reporting Server
    • Rails default architecture
      Collection Server
      Data Source
      MySQL
      User
      Reporting Server
      “Just” a Rails App
    • Rails default architecture
      Performance Bottleneck: Too much load
      Collection Server
      Data Source
      MySQL
      User
      Reporting Server
      “Just” a Rails App
    • Let’s add replication!
      MySQL
      Master
      Collection Server
      Data Source
      Replication
      MySQL
      Master
      User
      Reporting Server
      MySQL
      Master
      MySQL
      Master
    • Let’s add replication!
      MySQL
      Master
      Collection Server
      Data Source
      Replication
      MySQL
      Master
      User
      Reporting Server
      MySQL
      Master
      MySQL
      Master
      Off the shelf!
      Scalable Reads!
    • Let’s add replication!
      Performance Bottleneck: Still can’t scale writes
      MySQL
      Master
      Collection Server
      Data Source
      Replication
      MySQL
      Master
      User
      Reporting Server
      MySQL
      Master
      MySQL
      Master
      Off the shelf!
      Scalable Reads!
    • What about sharding?
      Collection Server
      Data Source
      Sharding
      MySQL
      Master
      MySQL
      Master
      MySQL
      Master
      User
      Reporting Server
      Sharding
    • What about sharding?
      Collection Server
      Data Source
      Sharding
      MySQL
      Master
      MySQL
      Master
      MySQL
      Master
      User
      Reporting Server
      Sharding
      Scalable Writes!
    • What about sharding?
      Development Bottleneck:
      Need to write custom code
      Collection Server
      Data Source
      Sharding
      MySQL
      Master
      MySQL
      Master
      MySQL
      Master
      User
      Reporting Server
      Sharding
      Scalable Writes!
    • Key Value stores to the rescue?
      Collection Server
      Data Source
      MySQL
      Master
      MySQL
      Master
      Cassandra or
      Voldemort
      User
      Reporting Server
    • Key Value stores to the rescue?
      Collection Server
      Data Source
      MySQL
      Master
      MySQL
      Master
      Cassandra or
      Voldemort
      User
      Reporting Server
      Scalable Writes!
    • Key Value stores to the rescue?
      Development Bottleneck:
      Reporting is limited / hard
      Collection Server
      Data Source
      MySQL
      Master
      MySQL
      Master
      Cassandra or
      Voldemort
      User
      Reporting Server
      Scalable Writes!
    • Can I Hadoop my way out of this?
      MySQL
      Master
      MySQL
      Master
      Cassandra or
      Voldemort
      Collection Server
      Data Source
      Hadoop
      MySQL
      Master
      MySQL
      Master
      User
      Reporting Server
      MySQL
      Master
      MySQL
      Slave
    • Can I Hadoop my way out of this?
      MySQL
      Master
      MySQL
      Master
      Cassandra or
      Voldemort
      Collection Server
      Data Source
      Hadoop
      MySQL
      Master
      Scalable Writes!
      MySQL
      Master
      User
      Reporting Server
      MySQL
      Master
      MySQL
      Slave
    • Can I Hadoop my way out of this?
      MySQL
      Master
      MySQL
      Master
      Cassandra or
      Voldemort
      Collection Server
      Data Source
      Hadoop
      MySQL
      Master
      Scalable Writes!
      Flexible Reports!
      MySQL
      Master
      User
      Reporting Server
      MySQL
      Master
      MySQL
      Slave
    • Can I Hadoop my way out of this?
      MySQL
      Master
      MySQL
      Master
      Cassandra or
      Voldemort
      Collection Server
      Data Source
      Hadoop
      MySQL
      Master
      Scalable Writes!
      Flexible Reports!
      “Just” a Rails App
      MySQL
      Master
      User
      Reporting Server
      MySQL
      Master
      MySQL
      Slave
    • Can I Hadoop my way out of this?
      Development Bottleneck:
      Too many systems!
      MySQL
      Master
      MySQL
      Master
      Cassandra or
      Voldemort
      Collection Server
      Data Source
      Hadoop
      MySQL
      Master
      Scalable Writes!
      Flexible Reports!
      “Just” a Rails App
      MySQL
      Master
      User
      Reporting Server
      MySQL
      Master
      MySQL
      Slave
    • MongoDB!
      Collection Server
      Data Source
      MySQL
      Master
      MySQL
      Master
      MongoDB
      User
      Reporting Server
    • MongoDB!
      Collection Server
      Data Source
      MySQL
      Master
      MySQL
      Master
      MongoDB
      User
      Reporting Server
      Scalable Writes!
    • MongoDB!
      Collection Server
      Data Source
      MySQL
      Master
      MySQL
      Master
      MongoDB
      User
      Reporting Server
      Scalable Writes!
      Flexible Reporting!
    • MongoDB!
      Collection Server
      Data Source
      MySQL
      Master
      MySQL
      Master
      MongoDB
      User
      Reporting Server
      Scalable Writes!
      “Just” a rails app
      Flexible Reporting!
    • MongoD
      App Server
      Data Source
      Collection
      MongoD
      Load
      Balancer
      Passenger
      Nginx
      Mongos
      Reporting
      User
      MongoD
    • MongoD
      App Server
      Data Source
      Collection
      MongoD
      Load
      Balancer
      Passenger
      Nginx
      Mongos
      Reporting
      User
      MongoD
      Sharding!
    • MongoD
      App Server
      Data Source
      Collection
      MongoD
      Load
      Balancer
      Passenger
      Nginx
      Mongos
      Reporting
      User
      MongoD
      Sharding!
      High Concurrency
    • MongoD
      App Server
      Data Source
      Collection
      MongoD
      Load
      Balancer
      Passenger
      Nginx
      Mongos
      Reporting
      User
      MongoD
      Sharding!
      High Concurrency
      Scale-Out
    • MongoDBSharding
    • MongoDBSharding
      Replica Sets let us scale storage & transaction capacity for each shard
    • MongoDBSharding
      Replica Sets let us scale storage & transaction capacity for each shard
      Mongos routes transactions to shards based on “shard key”
    • MongoDBSharding
      Replica Sets let us scale storage & transaction capacity for each shard
      Mongos routes transactions to shards based on “shard key”
      Config servers store information about which shards exist
    • Inserting
      3
      2
      Shard key == name
      bob  Shard 2
      1
      insert { ‘name’ : bob }
      Insert { ‘name’ : bob }
    • Querying
      3
      2
      Shard key == name
      bob  Shard 2
      1
      Query { ‘name’ : bob }
      Query { ‘name’ : bob }
    • Map Reduce
      2
      2
      2
      2
      1
      Map-reduce( … )
      Map-reduce(…)
    • Working with Mongo
      MongoMapper makes it look like ActiveRecord
      Documents are more natural than rows in many cases
      Map-Reduce rocks (but needs better support in rails)
      http://www.flickr.com/photos/elhamalawy/2526783078/
    • Ruby
      Mongo
    • Runs over all the objects in the views table, counting how many times a page was viewed
      Adds up all the counts for a unique url / date combination
      Run the map reduce job and return a collection containing the results
    • Results
      Version 1 of our analytics system took 2 weeks with 1 engineer
      We have since added a lot more complexity, but we did it incrementally
      We replaced MySQL entirely with MongoDB
      No need for joins, transactions
      Every table is now a document collection
      It’s fast!
      63ms – Average response time for sending data to server
      93ms – Average response time for displaying reports