The web is live. APIs give us access to continuously changing data. We discuss ways to get real-time data into your app, how to handle data processing and what to do when you get thousands of updates per second.

1. Presented for your enjoyment
at DjangoCon US 2011
Real-Time Django
with Ben Slavin and Adam Miskiewicz
@benslavin @skevy

2. The web is
Read / Write

3. Read /
The web is
Write

4. GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET POST GET GET GET GET GET GET G
GET GET GET GET GET POST GET POST GET GET GET GET GET POST GET GET GET GET GET GET GET GET GET GET GET GET GET
GET GET GET GET GET POST GET GET GET GET GET GET GET GET POST GET GET POST GET GET GET GET GET GET GET GET GET
GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET POST GET GET GET GET GET GET GET POST GET GET GET G
GET POST GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET G
GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET
GET GET GET GET GET GET GET GET GET GET GET GET GET POST GET GET GET GET GET GET GET GET GET GET GET GET GET G
GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET POST GET GET G
GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET POST GET GET PO
GET GET GET GET GET GET GET POST GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET G
GET GET GET GET GET GET GET GET GET GET GET GET GET POST GET GET GET GET GET GET GET GET GET GET GET GET GET G
GET GET GET POST GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET POST GET G
POST GET GET GET POST GET GET GET GET GET GET GET POST GET POST GET GET GET GET GET GET GET GET GET GET GET G
GET GET GET GET GET POST GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET POST GET GET GET POST GET
GET GET GET GET GET GET GET GET GET GET POST GET GET POST GET GET GET GET GET GET GET GET GET GET GET GET GET P
GET GET GET GET POST GET GET GET GET GET GET GET GET GET GET GET GET POST GET GET GET GET GET GET GET GET GET G
GET GET GET GET GET GET POST GET GET GET GET GET GET GET GET POST GET GET GET GET GET GET GET GET POST GET GET
GET GET GET GET GET GET GET GET POST GET GET GET GET GET GET GET GET POST GET GET GET GET GET GET GET GET GET G

5. 1 / second

6. Django Just Works
(with intelligent application design and proper caching)

7. 50 / second

8. 500 / second

9. 5,000 / second

10. Beyonce!!!
8,868 / second
http://twitter.com/#!/twitterglobalpr/status/108285017792331776

11. Superbowl XLV
http://blog.twitter.com/2011/02/superbowl.html

12. 4,064 at peak

13. >2,000 sustained

14. Django wasn’t
built for this.

15. ... but that doesn’t
mean we need to use
J2EE or Erlang.

16. Using the techniques discussed today, we have:
Processed > 4k pieces of data/second
Tracked >50k live datapoints
Run live events
Served award-show sized audiences

17. You may not deal with this scale, but hopefully you can
Learn from our techniques

18. Under-documented

19. A lot to cover

20. A play. In three parts.
Retrieval
Processing
Presentation

21. Retrieval
Polling

22. Retrieval / Polling
Widely used
Twitter, Facebook, Foursquare, etc.

23. Retrieval
Continuous Polling
the naïve approach

24. Retrieval / Continuous Polling
Slow
Synchronously blocks the request/response cycle

25. Retrieval / Continuous Polling
Not neighborly
Adds undue burden on the upstream service

26. Retrieval / Continuous Polling
Failure model sucks
If the upstream service goes down, so do you

27. Retrieval
Cached Polling
a slightly less-awful approach

28. Retrieval / Cached Polling
Dog pile
Same as ‘continuous polling’ in the degenerate case

29. Retrieval / Cached Polling
Failure model sucks
If the upstream service goes down, so do you

30. Retrieval / Polling
DON’T BREAK THE CYCLE
Don’t do this in the request/response cycle

31. Retrieval / Polling
manage.py poll_stuff
+
crontab -e

32. Retrieval / Polling
Still not enough

33. Retrieval / Polling
Rate limits
ex. 500 requests / hour

34. Retrieval / Rate Limits
Batched requests
http://api.twitter.com/1/users/lookup.json
?screen_name=bolsterlabs,benslavin,skevy

35. Retrieval / Rate Limits
Multiple clients
Use a pool of workers with
different IPs and API keys

36. Retrieval / Rate Limits
Special access
Ask the upstream provider.

37. Retrieval
Web hooks
No, you come to me.

38. Retrieval / Web Hooks
Out of band
Asynchronous from the user’s perspective.

39. Retrieval / Web Hooks
PubSubHubbub
Used by Gowalla, Myspace, Google

40. Retrieval / Web Hooks
The data comes to you
True ‘push’.

41. Retrieval / Web Hooks
Just handle it
Class based views or plain-old methods.
It’s just Django w/ different auth.

42. Retrieval / Web Hooks
Setup can be complex
Or worse, completely manual.

43. Retrieval
Streaming
Long-lived, open-socket communication.

44. Retrieval / Streaming
Live updates
but only when you’re connected.

45. Retrieval / Streaming
Single client
can be a significant bottleneck

46. Retrieval / Streaming
“Site Streams may deliver hundreds of messages per second to a client, and each
stream may consume significant (> 1 Mbit/sec) bandwidth.
Your processing of tweets should be asynchronous,
with appropriate buffers in place to handle spikes of 3x normal throughput. Note
that slow reading clients are automatically terminated.”
https://dev.twitter.com/docs/streaming-api/site-streams

47. Retrieval / Streaming
Hot potato
Pass data off as quickly as possible

48. Retrieval
STORE IT. LOG IT. SAVE IT.
This data is ephemeral, and there may be
no good way to recreate it once it’s gone.

49. Processing

50. Processing
Denormalization

51. Processing / Denormalization
Your DB is slow.
* Unless you know Frank Wiles.

52. Processing / Denormalization
db_index=True
is not the answer

53. Processing / Denormalization
Tweet.objects
.filter(screenname=”aplusk”)
.count()

54. Processing / Denormalization
TweetCount.objects
.get(screenname=”aplusk”)
.tweet_count
Also consider, memcached, Redis, etc.

55. Processing / Denormalization
pre_save, post_save,
post_delete and F objects
Use these.

56. Processing / Denormalization
Be careful
These only work in Django

57. Processing
Workers

58. Processing / Workers
Deconstruct the problem

59. Check for profanity
then
Retrieve an avatar
then
Geo-locate the author
then
Add as input for trending terms
then
Retrieve author’s social graph
then
Adjust the leaderboard

60. Retrieve an avatar
Check for profanity
Geo-locate the author
Add as input for trending terms
then
Retrieve author’s social graph
Adjust the leaderboard

61. Processing / Workers
django-celery
Or manage yourself with any queue

62. Processing
map + reduce
It’s not that scary

63. Processing / map + reduce
Generational
Get data. Process. Cache results.

64. Processing / map + reduce / Generational
Good for many problems.
Especially where the intermediate working set is large.

65. Processing / map + reduce / Generational
Solutions exist.
CouchDB, Mongo, Hadoop

66. Processing / map + reduce
Sometimes we can
be smarter

67. Processing / map + reduce / incremental
Consider averages.
I mean the mean.

68. Processing / map + reduce / incremental
n
1
n Σi=1
ai

69. Processing / map + reduce / incremental
= ( )
(Σ)
n n-1
1 1
n Σ
i=1
ai n
i=1
ai + a n
From O(n) to O(1)

70. Processing / map + reduce / incremental
This example was trivial, but you can often
Store a partial solution

71. Presentation
(Of the data, not the thing we’re doing now.)

72. Presentation
Partial Caching

73. Presentation / Partial Caching
{% cache 500 my_stuff %}
Template fragment caching

74. Presentation / Partial Caching
class MyModel(models.Model):
as_html = models.TextField()

75. Presentation / Partial Caching
serialized = json.dumps(my_stuff)
cache.set(‘my_stuff’, serialized)
Don’t be afraid of low-level caching.

76. Presentation
Continuous Caching

77. Presentation / Continuous Caching
while True:
cache_page()

78. Presentation / Continuous Caching
Out-of-band caching.
Works when the number of pages is relatively small.
Similar to proxy_cache, but more resilient.

79. Presentation / Continuous Caching
[Watch this space.]

80. Presentation
Real-Time Updates

81. Presentation / Real-Time Updates
gevent, eventlet,
tornado, twisted

82. Presentation / Real-Time Updates
Django plays well
with others

83. Presentation / Real-Time Updates
Django + RabbitMQ
+
node.js + socket.io

84. Presentation / Real-Time Updates
[Watch this space.]

85. Presentation
Failure Models

86. Presentation / Failure Models
This isn’t good for anyone.

87. Presentation / Failure Models
proxy_cache_use_stale
and
proxy_next_upstream
.
For use with nginx

88. Presentation / Failure Models
Build a small backup app.
It can serve pre-cached content.
Anything is better than a 404, 500 or 502 (usually)

89. Follow @bolsterlabs
for slides and
lively discussion.

90. Thank you.

91. Don’t be a stranger.
Ben Slavin Adam Miskiewicz
@benslavin @skevy
ben@bolsterlabs.com adam@bolsterlabs.com
@bolsterlabs