Scalable Event Analytics with MongoDB & Ruby on Rails

Scalable Event Analytics with Ruby on Rails & MongoDB
Ruby Conf China 2010
Jared Rosoff (@forjared) jrosoff@yottaa.com

Yottaa!!!! (www.yottaa.com)

Overview
Ruby at Scale
What is Event Analytics?
What are the different ways you could do it?
How we did it

Ruby At Scale?
http://www.flickr.com/photos/laughingsquid

Event Analytics
Data Source
Event
User
Event
Event
Query
Event Analytics
Data Source
Report
Event
Event
Event
Event
User
Query
Data Source
Event
Event
Report
Event
Event
Event
Data Source

Oh and we are a startup

Our requirements:

Rails default architecture
Collection Server
Data Source
MySQL
User
Reporting Server

Collection Server
Data Source
MySQL
User
Reporting Server
“Just” a Rails App

Performance Bottleneck: Too much load
Collection Server
Data Source
MySQL
User
Reporting Server

Let’s add replication!
MySQL
Master
Collection Server
Data Source
Replication
MySQL
Master
User
Reporting Server
MySQL
Master
MySQL
Master

MySQL
Master
Collection Server
Data Source
Replication
MySQL
Master
User
Reporting Server
MySQL
Master
MySQL
Master
Off the shelf!
Scalable Reads!

Performance Bottleneck: Still can’t scale writes
MySQL
Master
Collection Server
Data Source
Replication
MySQL
Master
User
Reporting Server
MySQL
Master
MySQL
Master
Off the shelf!
Scalable Reads!

What about sharding?
Collection Server
Data Source
Sharding
MySQL
Master
MySQL
Master
MySQL
Master
User
Reporting Server
Sharding

Collection Server
Data Source
Sharding
MySQL
Master
MySQL
Master
MySQL
Master
User
Reporting Server
Sharding
Scalable Writes!

Development Bottleneck:
Need to write custom code
Collection Server
Data Source
Sharding
MySQL
Master
MySQL
Master
MySQL
Master
User
Reporting Server
Sharding
Scalable Writes!

Key Value stores to the rescue?
Collection Server
Data Source
MySQL
Master
MySQL
Master
Cassandra or
Voldemort
User
Reporting Server

Collection Server
Data Source
MySQL
Master
MySQL
Master
Cassandra or
Voldemort
User
Reporting Server
Scalable Writes!

Reporting is limited / hard
Collection Server
Data Source
MySQL
Master
MySQL
Master
Cassandra or
Voldemort
User
Reporting Server
Scalable Writes!

Can I Hadoop my way out of this?
MySQL
Master
MySQL
Master
Cassandra or
Voldemort
Collection Server
Data Source
Hadoop
MySQL
Master
MySQL
Master
User
Reporting Server
MySQL
Master
MySQL
Slave

MySQL
Master
MySQL
Master
Cassandra or
Voldemort
Collection Server
Data Source
Hadoop
MySQL
Master
Scalable Writes!
MySQL
Master
User
Reporting Server
MySQL
Master
MySQL
Slave

MySQL
Master
MySQL
Master
Cassandra or
Voldemort
Collection Server
Data Source
Hadoop
MySQL
Master
Scalable Writes!
Flexible Reports!
MySQL
Master
User
Reporting Server
MySQL
Master
MySQL
Slave

Too many systems!
MySQL
Master
MySQL
Master
Cassandra or
Voldemort
Collection Server
Data Source
Hadoop
MySQL
Master
Scalable Writes!
Flexible Reports!
MySQL
Master
User
Reporting Server
MySQL
Master
MySQL
Slave

MongoDB!
Collection Server
Data Source
MySQL
Master
MySQL
Master
MongoDB
User
Reporting Server

MongoDB!
Collection Server
Data Source
MySQL
Master
MySQL
Master
MongoDB
User
Reporting Server
Scalable Writes!

MongoDB!
Collection Server
Data Source
MySQL
Master
MySQL
Master
MongoDB
User
Reporting Server
Scalable Writes!
Flexible Reporting!

MongoDB!
Collection Server
Data Source
MySQL
Master
MySQL
Master
MongoDB
User
Reporting Server
Scalable Writes!
“Just” a rails app
Flexible Reporting!

MongoD
App Server
Data Source
Collection
MongoD
Load
Balancer
Passenger
Nginx
Mongos
Reporting
User
MongoD

MongoD
App Server
Data Source
Collection
MongoD
Load
Balancer
Passenger
Nginx
Mongos
Reporting
User
MongoD
Sharding!

MongoD
App Server
Data Source
Collection
MongoD
Load
Balancer
Passenger
Nginx
Mongos
Reporting
User
MongoD
Sharding!
High Concurrency

MongoD
App Server
Data Source
Collection
MongoD
Load
Balancer
Passenger
Nginx
Mongos
Reporting
User
MongoD
Sharding!
High Concurrency
Scale-Out

MongoDBSharding

MongoDBSharding
Replica Sets let us scale storage & transaction capacity for each shard

MongoDBSharding
Mongos routes transactions to shards based on “shard key”

MongoDBSharding
Mongos routes transactions to shards based on “shard key”
Config servers store information about which shards exist

Inserting
3
2
Shard key == name
bob  Shard 2
1
insert { ‘name’ : bob }
Insert { ‘name’ : bob }

Querying
3
2
Shard key == name
bob  Shard 2
1
Query { ‘name’ : bob }
Query { ‘name’ : bob }

Map Reduce
2
2
2
2
1
Map-reduce( … )
Map-reduce(…)

Working with Mongo
MongoMapper makes it look like ActiveRecord
Documents are more natural than rows in many cases
Map-Reduce rocks (but needs better support in rails)
http://www.flickr.com/photos/elhamalawy/2526783078/

Ruby
Mongo

Runs over all the objects in the views table, counting how many times a page was viewed
Adds up all the counts for a unique url / date combination
Run the map reduce job and return a collection containing the results

Results
Version 1 of our analytics system took 2 weeks with 1 engineer
We have since added a lot more complexity, but we did it incrementally
We replaced MySQL entirely with MongoDB
No need for joins, transactions
Every table is now a document collection
It’s fast!
63ms – Average response time for sending data to server
93ms – Average response time for displaying reports

http://nosql.mypopescu.com	1244
http://mobicon.tistory.com	475
http://blog.yottaa.com	317
http://www.nosqldatabases.com	291
http://www.mongodb.org	62
http://mobimongo.tistory.com	49
http://www.yottaa.com	36
http://www.febryadi.com	34
http://mobibackend.tistory.com	20
http://paper.li	18
http://www.twylah.com	11
http://yottaa1.web12.hubspot.com	10
http://translate.googleusercontent.com	6
http://www.linkedin.com	6
http://www.slideshare.net	6
http://mobifrontend.tistory.com	4
http://webcache.googleusercontent.com	3
http://static.slidesharecdn.com	2
http://seekr-artemis.heroku.com	2
http://www.techgig.com	1
http://a0.twimg.com	1
http://b.hatena.ne.jp	1
http://us-w1.rockmelt.com	1
http://twitter.com	1
http://mongodb.nosqlclub.com	1
http://fever.leehorrocks.com	1
https://abs.twimg.com	1

Scalable Event Analytics with MongoDB & Ruby on Rails

by Jared Rosoff on Jun 28, 2010

Accessibility

Categories

Upload Details

Usage Rights

27 Embeds 2,604

Statistics

Scalable Event Analytics with MongoDB & Ruby on Rails Presentation Transcript