MongoDB Real-time Data Collection and Stats Generation

MongoDB for real-time data collection and stats generation Damon Cortesi (@dacort) damon@rowfeeder.com

The Past • Lots of data • 2M rows/day • Post-computation is slow • TweetStats == 1B tweets

mysql> select count(*) from gnip_activity_2_1 WHERE created_at BETWEEN '2010-03-01' AND '2010-03-02'; +----------+ | count(*) | +----------+ | 1480294 | +----------+ 1 row in set (6.32 sec) mysql> SELECT app,(count(*)/(select count(*) from gnip_activity_2_1 WHERE created_at BETWEEN '2010-03-01' AND '2010-03-02'))*100 AS percent, count(*) as count -> FROM gnip_activity_2_1 -> WHERE created_at BETWEEN '2010-03-01' AND '2010-03-02' -> GROUP BY app -> ORDER BY count DESC -> LIMIT 10; +-------------+---------+--------+ | app | percent | count | +-------------+---------+--------+ | web | 31.1618 | 461286 | | TweetDeck | 12.4685 | 184570 | | UberTwitter | 7.3333 | 108555 | This is NOT real-time | twitterfeed | 7.1350 | 105619 | | API | 5.3761 | 79582 | | Echofon | 4.2635 | 63112 | | Tweetie | 3.4734 | 51416 | | Seesmic | 2.1913 | 32438 | | mobile web | 1.8382 | 27211 | | HootSuite | 1.7951 | 26573 | +-------------+---------+--------+ 10 rows in set (1 min 27.07 sec)

So now what? Tokyo Riak CouchDB MongoDB http://www.ﬂickr.com/photos/bob_august/4307291275/

• Initially simple - Tweets -> Spreadsheet • “Maybe we should save this data...” • Real-time updates > db.serverStatus() • Minimal stats generation "opcounters" : { "insert" : 36687455, "query" : 857059, • Writes > Reads (200x) "update" : 189207744, "delete" : 0, "getmore" : 4176334, "command" : 36734580 }

Streaming Twitter curl http://stream.twitter.com/1/statuses/sample.json - u<user>:<pass> | mongoimport -c twitter_live Courtesy @eliothorowitz ...

Slightly more complex ;) { Twitter/Facebook/etc Redis/Resque Queues Process MongoDB { Process Process Save/Update/Stats

Modiﬁer Operations No query/retrieve Just $set or $inc

Other Beneﬁts • New ﬁeld? No multi-day ALTER statement. • Auto-sharding in 1.6 • --notablescan • aka, don’t pull a Twitter • “On Monday, our users database, where we store millions of user records, got hung up running a long-running query” -- 7/21

So...that earlier SQL? Incrementers Much Better

Simple. Pre-Computed. Wait, can’t I pre-compute in MySQL? Mongo == Fire/forget and async

But sometimes... • Aggregation might still be necessary • This works...but what if we need more?

lnkby.me ‣ Problem statement: 1. Aggregate stats on shortened links 2. Top domains based on # clicks Easy Slightly more difﬁcult 3. Top users driving trafﬁc to those top domains for the past seven days

Solution • Gather stats • Map/Reduce! • Server-side JavaScript • Temporary collection to hold output • Can be written to permanent collection • temp collection renamed atomically

aggregate Get Top 10 Domains, then...

Caveat • Indexes, indexes, indexes • Compound indexes • a,b,c • Query on a; a,b; or a,b,c • Sort on last ﬁeld

Mongo++ • High-volume updates -- win • Stats generation -- win • Mutable schema && json -- win

http://dcortesi.com	3066
http://paper.li	17
http://webcache.googleusercontent.com	12
http://translate.googleusercontent.com	5
http://admin.totalmarketing.com	4
https://twitter.com	3
http://us-w1.rockmelt.com	3
http://207.46.192.232	1
http://twitter.com	1
http://www.slideshare.net	1
http://content.wildfiresocial.com	1
http://static.slidesharecdn.com	1
http://cache.baidu.com	1

MongoDB Real-time Data Collection and Stats Generation

by Damon Cortesi on Jul 27, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

13 Embeds 3,116

Statistics

MongoDB Real-time Data Collection and Stats Generation — Presentation Transcript