Real-Time Django

Presented for your enjoyment at DjangoCon US 2011Real-Time Django with Ben Slavin and Adam Miskiewicz @benslavin @skevy

The web isRead / Write

Read / The web is Write

GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET POST GET GET GET GET GET GET GGET GET GET GET GET POST GET POST GET GET GET GET GET POST GET GET GET GET GET GET GET GET GET GET GET GET GETGET GET GET GET GET POST GET GET GET GET GET GET GET GET POST GET GET POST GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET POST GET GET GET GET GET GET GET POST GET GET GET G GET POST GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GGET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET POST GET GET GET GET GET GET GET GET GET GET GET GET GET G GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET POST GET GET G GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET POST GET GET PO GET GET GET GET GET GET GET POST GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET G GET GET GET GET GET GET GET GET GET GET GET GET GET POST GET GET GET GET GET GET GET GET GET GET GET GET GET G GET GET GET POST GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET POST GET G POST GET GET GET POST GET GET GET GET GET GET GET POST GET POST GET GET GET GET GET GET GET GET GET GET GET GGET GET GET GET GET POST GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET POST GET GET GET POST GETGET GET GET GET GET GET GET GET GET GET POST GET GET POST GET GET GET GET GET GET GET GET GET GET GET GET GET P GET GET GET GET POST GET GET GET GET GET GET GET GET GET GET GET GET POST GET GET GET GET GET GET GET GET GET GGET GET GET GET GET GET POST GET GET GET GET GET GET GET GET POST GET GET GET GET GET GET GET GET POST GET GET GET GET GET GET GET GET GET GET POST GET GET GET GET GET GET GET GET POST GET GET GET GET GET GET GET GET GET G

1 / second

Django Just Works(with intelligent application design and proper caching)

50 / second

500 / second

5,000 / second

Beyonce!!! 8,868 / secondhttp://twitter.com/#!/twitterglobalpr/status/108285017792331776

Superbowl XLVhttp://blog.twitter.com/2011/02/superbowl.html

4,064 at peak

>2,000 sustained

Django wasn’tbuilt for this.

... but that doesn’tmean we need to use J2EE or Erlang.

Using the techniques discussed today, we have:Processed > 4k pieces of data/second Tracked >50k live datapoints Run live events Served award-show sized audiences

You may not deal with this scale, but hopefully you canLearn from our techniques

Under-documented

A lot to cover

A play. In three parts. Retrieval ProcessingPresentation

RetrievalPolling

Retrieval / PollingWidely usedTwitter, Facebook, Foursquare, etc.

RetrievalContinuous Polling the naïve approach

Retrieval / Continuous Polling SlowSynchronously blocks the request/response cycle

Retrieval / Continuous Polling Not neighborlyAdds undue burden on the upstream service

Retrieval / Continuous PollingFailure model sucks If the upstream service goes down, so do you

RetrievalCached Polling a slightly less-awful approach

Retrieval / Cached Polling Dog pileSame as ‘continuous polling’ in the degenerate case

Retrieval / Cached PollingFailure model sucks If the upstream service goes down, so do you

Retrieval / PollingDON’T BREAK THE CYCLE Don’t do this in the request/response cycle

Retrieval / Pollingmanage.py poll_stuff + crontab -e

Retrieval / PollingStill not enough

Retrieval / PollingRate limits ex. 500 requests / hour

Retrieval / Rate LimitsBatched requests http://api.twitter.com/1/users/lookup.json?screen_name=bolsterlabs,benslavin,skevy

Retrieval / Rate LimitsMultiple clients Use a pool of workers with different IPs and API keys

Retrieval / Rate LimitsSpecial access Ask the upstream provider.

RetrievalWeb hooks No, you come to me.

Retrieval / Web Hooks Out of bandAsynchronous from the user’s perspective.

Retrieval / Web HooksPubSubHubbub Used by Gowalla, Myspace, Google

Retrieval / Web HooksThe data comes to you True ‘push’.

Retrieval / Web Hooks Just handle itClass based views or plain-old methods. It’s just Django w/ different auth.

Retrieval / Web HooksSetup can be complex Or worse, completely manual.

Retrieval StreamingLong-lived, open-socket communication.

Retrieval / StreamingLive updatesbut only when you’re connected.

Retrieval / StreamingSingle clientcan be a significant bottleneck

Retrieval / Streaming“Site Streams may deliver hundreds of messages per second to a client, and each stream may consume significant (> 1 Mbit/sec) bandwidth. Your processing of tweets should be asynchronous,with appropriate buffers in place to handle spikes of 3x normal throughput. Note that slow reading clients are automatically terminated.” https://dev.twitter.com/docs/streaming-api/site-streams

Retrieval / Streaming Hot potatoPass data off as quickly as possible

RetrievalSTORE IT. LOG IT. SAVE IT. This data is ephemeral, and there may be no good way to recreate it once it’s gone.

Processing

ProcessingDenormalization

Processing / DenormalizationYour DB is slow. * Unless you know Frank Wiles.

Processing / Denormalization db_index=Trueis not the answer

Processing / Denormalization Tweet.objects.filter(screenname=”aplusk”) .count()

Processing / Denormalization TweetCount.objects.get(screenname=”aplusk”) .tweet_count Also consider, memcached, Redis, etc.

Processing / Denormalization pre_save, post_save,post_delete and F objects Use these.

Processing / DenormalizationBe careful These only work in Django

ProcessingWorkers

Processing / WorkersDeconstruct the problem

Check for profanity then Retrieve an avatar then Geo-locate the author thenAdd as input for trending terms thenRetrieve author’s social graph then Adjust the leaderboard

Retrieve an avatarCheck for profanity Geo-locate the author Add as input for trending terms then Retrieve author’s social graph Adjust the leaderboard

Processing / Workersdjango-celeryOr manage yourself with any queue

Processingmap + reduce It’s not that scary

Processing / map + reduceGenerationalGet data. Process. Cache results.

Processing / map + reduce / GenerationalGood for many problems. Especially where the intermediate working set is large.

Processing / map + reduce / GenerationalSolutions exist. CouchDB, Mongo, Hadoop

Processing / map + reduceSometimes we can be smarter

Processing / map + reduce / incrementalConsider averages. I mean the mean.

Processing / map + reduce / incremental n 1 n Σi=1 ai

Processing / map + reduce / incremental = ( ) (Σ) n n-11 1n Σ i=1 ai n i=1 ai + a n From O(n) to O(1)

Processing / map + reduce / incremental This example was trivial, but you can oftenStore a partial solution

Presentation(Of the data, not the thing we’re doing now.)

PresentationPartial Caching

Presentation / Partial Caching{% cache 500 my_stuff %} Template fragment caching

Presentation / Partial Cachingclass MyModel(models.Model): as_html = models.TextField()

Presentation / Partial Cachingserialized = json.dumps(my_stuff) cache.set(‘my_stuff’, serialized) Don’t be afraid of low-level caching.

PresentationContinuous Caching

Presentation / Continuous Cachingwhile True: cache_page()

Presentation / Continuous CachingOut-of-band caching. Works when the number of pages is relatively small. Similar to proxy_cache, but more resilient.

Presentation / Continuous Caching[Watch this space.]

PresentationReal-Time Updates

Presentation / Real-Time Updatesgevent, eventlet,tornado, twisted

Presentation / Real-Time UpdatesDjango plays well with others

Presentation / Real-Time Updates Django + RabbitMQ + node.js + socket.io

Presentation / Real-Time Updates[Watch this space.]

PresentationFailure Models

Presentation / Failure ModelsThis isn’t good for anyone.

Presentation / Failure Modelsproxy_cache_use_stale andproxy_next_upstream. For use with nginx

Presentation / Failure ModelsBuild a small backup app. It can serve pre-cached content. Anything is better than a 404, 500 or 502 (usually)

Follow @bolsterlabs for slides and lively discussion.

Thank you.

Don’t be a stranger. Ben Slavin Adam Miskiewicz @benslavin @skevyben@bolsterlabs.com adam@bolsterlabs.com @bolsterlabs

http://lanyrd.com	836
http://aamirhussain.com	572
http://twitter.com	6
https://twitter.com	5
http://zootool.com	3
http://us-w1.rockmelt.com	1
http://localhost	1
http://a0.twimg.com	1

Real-Time Django

by Bolster Labs on Sep 06, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

8 Embeds 1,425

Statistics

Real-Time Django — Presentation Transcript