Realtime Analytics using MongoDB, Python,
Gevent, and ZeroMQ Rick Copeland @rick446 [email_address]

SourceForge s MongoDB <ul><li>Tried CouchDB
– liked the dev model, not so much the performance </li></ul><ul><li>Migrated consumer-facing pages (summary, browse, download) to MongoDB and it worked great (on MongoDB 0.8 no less!) </li></ul><ul><li>Built an entirely new tool platform around MongoDB (Allura) </li></ul>

The Problem We’re Trying to
Solve <ul><li>We have lots of users (good) </li></ul><ul><li>We have lots of projects (good) </li></ul><ul><li>We don’t know what those users and projects are doing (not so good) </li></ul><ul><li>We have tons of code in PHP, Perl, and Python (not so good) </li></ul>

Introducing Zarkov 0.0.1 <ul><li>Asynchronous TCP
server for event logging with gevent </li></ul><ul><li>Turn OFF “safe” writes, turn OFF Ming validation (or do it in the client) </li></ul><ul><li>Incrementally calculate aggregate stats based on event log using mapreduce with {‘out’:’reduce’} </li></ul>

Zarkov Architecture MongoDB BSON over
ZeroMQ Journal Greenlet Commit Greenlet Write-ahead log Write-ahead log Aggregation Greenlet

Technologies <ul><li>MongoDB </li></ul><ul><ul><li>Fast (10k+ inserts/s
single-threaded) </li></ul></ul><ul><li>ZeroMQ </li></ul><ul><ul><li>Built-in buffering </li></ul></ul><ul><ul><li>PUSH/PULL sockets (push never blocks, easy to distribute work) </li></ul></ul><ul><li>BSON </li></ul><ul><ul><li>Fast Python/C implementation </li></ul></ul><ul><ul><li>More types than JSON </li></ul></ul><ul><li>Gevent </li></ul><ul><ul><li>“ green threads” for Python </li></ul></ul>

“Wow, it’s really fast; can
it replace…” <ul><li>Download statistics? </li></ul><ul><li>Google Analytics? </li></ul><ul><li>Project realtime statistics? </li></ul>“Probably, but it’ll take some work….”

Moving towards production.... <ul><li>MongoDB MapReduce:
convenient, but not so fast </li></ul><ul><ul><li>Global JS Interpreter Lock per mongod </li></ul></ul><ul><ul><li>Lots of writing to temp collections (high lock %) </li></ul></ul><ul><ul><li>Javascript without libraries (ick!) </li></ul></ul><ul><li>Hadoop? Painful to configure, high latency, non-seamless integration with MongoDB </li></ul>

Zarkov’s already doing a lot…
<ul><li>So we added a lightweight map/reduce framework </li></ul><ul><li>Write your map/reduce jobs in Python </li></ul><ul><li>Input/Output is MongoDB </li></ul><ul><li>Intermediate files are local .bson files </li></ul><ul><li>Use ZeroMQ for job distribution </li></ul>

Quick Map/reduce Refresher <ul><li>def map_reduce
(input_collection, query, output_collection, </li></ul><ul><li>map , reduce ): </li></ul><ul><li>objects = input_collection . find(query) </li></ul><ul><li>map_results = list ( map (objects)) </li></ul><ul><li>map_results . sort(key = operator . itemgetter( 0 )) </li></ul><ul><li>for key, kv_pairs in itertools . groupby( </li></ul><ul><li>(map_results, operator . itemgetter( 0 )): </li></ul><ul><li>value = reduce (key, [ v for k,v in kv_pairs ]) </li></ul><ul><li>output_collection . save( </li></ul><ul><li>{ &quot;_id&quot; :key, &quot;value&quot; :value}) </li></ul>

Quick Map/reduce Refresher <ul><li>def map_reduce
(input_collection, query, output_collection, </li></ul><ul><li>map , reduce ): </li></ul><ul><li>objects = input_collection . find(query) </li></ul><ul><li>map_results = list(map(objects)) </li></ul><ul><li>map_results . sort(key = operator . itemgetter( 0 )) </li></ul><ul><li>for key, kv_pairs in itertools . groupby( </li></ul><ul><li>(map_results, operator . itemgetter( 0 )): </li></ul><ul><li>value = reduce (key, [ v for k,v in kv_pairs ]) </li></ul><ul><li>output_collection . save( </li></ul><ul><li>{ &quot;_id&quot; :key, &quot;value&quot; :value}) </li></ul>Parallel

Zarkov Map/Reduce Architecture map_in_#.bson Query
Map Sort Reduce Commit map_out_#.bson reduce_in.bson Job Mgr

Zarkov Map/Reduce <ul><li>Phases managed by
greenlets </li></ul><ul><li>Map and reduce jobs parceled out to remote workers via zmq PUSH/PULL </li></ul><ul><li>Adaptive timeout/retry to support dead workers </li></ul><ul><li>Sort phase is local (big mergesort) but still done in worker processes </li></ul>

Zarkov Web Service <ul><li>We’ve got
the data in, now how do we get it out? </li></ul><ul><li>Zarkov includes a tiny HTTP server </li></ul><ul><ul><li>$ curl -d foo='{&quot;c&quot;:&quot;sfweb&quot;, &quot;b&quot;:&quot;date/2011-07-01/&quot;, &quot;e&quot;:&quot;date/2011-07-04&quot;}' http://localhost:8081/q </li></ul></ul><ul><ul><li>{&quot;foo&quot;: {&quot;sflogo&quot;: [[1309579200000.0, 12774], [1309665600000.0, 13458], [1309752000000.0, 13967]], &quot;hits&quot;: [[1309579200000.0, 69357], [1309665600000.0, 68514], [1309752000000.0, 68494]]}} </li></ul></ul><ul><li>Values come out tweaked for use in flot </li></ul>

Zarkov Deployment at SF.net

Lessons learned at

MongoDB Tricks <ul><li>Autoincrement integers are
harder than in MySQL but not impossible </li></ul><ul><li>Unsafe writes, insert > update </li></ul>class IdGen ( object): @classmethod def get_ids(cls, inc = 1): obj = cls.query.find_and_modify( query={ '_id': 0}, update ={ '$inc': dict(inc =inc), }, upsert= True, new = True) return range(obj .inc - inc, obj.inc)

MongoDB Pitfalls <ul><li>$addToSet is nice
but nothing beats an integer range query </li></ul><ul><li>Avoid Javascript like the plague (mapreduce, group, $where) </li></ul><ul><li>Indexing is nice, but slows things down; use _id when you can </li></ul><ul><li>mongorestore is fast, but locks a lot </li></ul>

Open Source Ming http://sf.net/projects/merciless/ MIT
License Allura http://sf.net/p/allura/ Apache License Zarkov http://sf.net/p/zarkov/ Apache License

Future Work <ul><li>Remove SPoF </li></ul><ul><li>Better
way of expressing aggregates </li></ul><ul><ul><li>Suggestions? </li></ul></ul><ul><li>Better web integration </li></ul><ul><ul><li>WebSockets/Socket.io </li></ul></ul><ul><li>Maybe trigger aggs based on event activity? </li></ul>

Rick Copeland @rick446 [email_address]

Credits <ul><li>http://www.flickr.com/photos/jprovost/5733297977/in/photostream/ </li></ul>