Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Real-Time Django

8,688 views

Published on

The web is live. APIs give us access to continuously changing data. We discuss ways to get real-time data into your app, how to handle data processing and what to do when you get thousands of updates per second.

Published in: Technology
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download Full EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download Full doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THIS can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THIS is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THIS Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THIS the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THIS Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download Full EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download Full doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THIS can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THIS is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THIS Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THIS the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THIS Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Dating direct: ❤❤❤ http://bit.ly/369VOVb ❤❤❤
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Follow the link, new dating source: ❶❶❶ http://bit.ly/369VOVb ❶❶❶
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Real-Time Django

  1. 1. Presented for your enjoyment at DjangoCon US 2011Real-Time Django with Ben Slavin and Adam Miskiewicz @benslavin @skevy
  2. 2. The web isRead / Write
  3. 3. Read / The web is Write
  4. 4. GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET POST GET GET GET GET GET GET GGET GET GET GET GET POST GET POST GET GET GET GET GET POST GET GET GET GET GET GET GET GET GET GET GET GET GETGET GET GET GET GET POST GET GET GET GET GET GET GET GET POST GET GET POST GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET POST GET GET GET GET GET GET GET POST GET GET GET G GET POST GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GGET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET POST GET GET GET GET GET GET GET GET GET GET GET GET GET G GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET POST GET GET G GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET POST GET GET PO GET GET GET GET GET GET GET POST GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET G GET GET GET GET GET GET GET GET GET GET GET GET GET POST GET GET GET GET GET GET GET GET GET GET GET GET GET G GET GET GET POST GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET POST GET G POST GET GET GET POST GET GET GET GET GET GET GET POST GET POST GET GET GET GET GET GET GET GET GET GET GET GGET GET GET GET GET POST GET GET GET GET GET GET GET GET GET GET GET GET GET GET GET POST GET GET GET POST GETGET GET GET GET GET GET GET GET GET GET POST GET GET POST GET GET GET GET GET GET GET GET GET GET GET GET GET P GET GET GET GET POST GET GET GET GET GET GET GET GET GET GET GET GET POST GET GET GET GET GET GET GET GET GET GGET GET GET GET GET GET POST GET GET GET GET GET GET GET GET POST GET GET GET GET GET GET GET GET POST GET GET GET GET GET GET GET GET GET GET POST GET GET GET GET GET GET GET GET POST GET GET GET GET GET GET GET GET GET G
  5. 5. 1 / second
  6. 6. Django Just Works(with intelligent application design and proper caching)
  7. 7. 50 / second
  8. 8. 500 / second
  9. 9. 5,000 / second
  10. 10. Beyonce!!! 8,868 / secondhttp://twitter.com/#!/twitterglobalpr/status/108285017792331776
  11. 11. Superbowl XLVhttp://blog.twitter.com/2011/02/superbowl.html
  12. 12. 4,064 at peak
  13. 13. >2,000 sustained
  14. 14. Django wasn’tbuilt for this.
  15. 15. ... but that doesn’tmean we need to use J2EE or Erlang.
  16. 16. Using the techniques discussed today, we have:Processed > 4k pieces of data/second Tracked >50k live datapoints Run live events Served award-show sized audiences
  17. 17. You may not deal with this scale, but hopefully you canLearn from our techniques
  18. 18. Under-documented
  19. 19. A lot to cover
  20. 20. A play. In three parts. Retrieval ProcessingPresentation
  21. 21. RetrievalPolling
  22. 22. Retrieval / PollingWidely usedTwitter, Facebook, Foursquare, etc.
  23. 23. RetrievalContinuous Polling the naïve approach
  24. 24. Retrieval / Continuous Polling SlowSynchronously blocks the request/response cycle
  25. 25. Retrieval / Continuous Polling Not neighborlyAdds undue burden on the upstream service
  26. 26. Retrieval / Continuous PollingFailure model sucks If the upstream service goes down, so do you
  27. 27. RetrievalCached Polling a slightly less-awful approach
  28. 28. Retrieval / Cached Polling Dog pileSame as ‘continuous polling’ in the degenerate case
  29. 29. Retrieval / Cached PollingFailure model sucks If the upstream service goes down, so do you
  30. 30. Retrieval / PollingDON’T BREAK THE CYCLE Don’t do this in the request/response cycle
  31. 31. Retrieval / Pollingmanage.py poll_stuff + crontab -e
  32. 32. Retrieval / PollingStill not enough
  33. 33. Retrieval / PollingRate limits ex. 500 requests / hour
  34. 34. Retrieval / Rate LimitsBatched requests http://api.twitter.com/1/users/lookup.json?screen_name=bolsterlabs,benslavin,skevy
  35. 35. Retrieval / Rate LimitsMultiple clients Use a pool of workers with different IPs and API keys
  36. 36. Retrieval / Rate LimitsSpecial access Ask the upstream provider.
  37. 37. RetrievalWeb hooks No, you come to me.
  38. 38. Retrieval / Web Hooks Out of bandAsynchronous from the user’s perspective.
  39. 39. Retrieval / Web HooksPubSubHubbub Used by Gowalla, Myspace, Google
  40. 40. Retrieval / Web HooksThe data comes to you True ‘push’.
  41. 41. Retrieval / Web Hooks Just handle itClass based views or plain-old methods. It’s just Django w/ different auth.
  42. 42. Retrieval / Web HooksSetup can be complex Or worse, completely manual.
  43. 43. Retrieval StreamingLong-lived, open-socket communication.
  44. 44. Retrieval / StreamingLive updatesbut only when you’re connected.
  45. 45. Retrieval / StreamingSingle clientcan be a significant bottleneck
  46. 46. Retrieval / Streaming“Site Streams may deliver hundreds of messages per second to a client, and each stream may consume significant (> 1 Mbit/sec) bandwidth. Your processing of tweets should be asynchronous,with appropriate buffers in place to handle spikes of 3x normal throughput. Note that slow reading clients are automatically terminated.” https://dev.twitter.com/docs/streaming-api/site-streams
  47. 47. Retrieval / Streaming Hot potatoPass data off as quickly as possible
  48. 48. RetrievalSTORE IT. LOG IT. SAVE IT. This data is ephemeral, and there may be no good way to recreate it once it’s gone.
  49. 49. Processing
  50. 50. ProcessingDenormalization
  51. 51. Processing / DenormalizationYour DB is slow. * Unless you know Frank Wiles.
  52. 52. Processing / Denormalization db_index=Trueis not the answer
  53. 53. Processing / Denormalization Tweet.objects.filter(screenname=”aplusk”) .count()
  54. 54. Processing / Denormalization TweetCount.objects.get(screenname=”aplusk”) .tweet_count Also consider, memcached, Redis, etc.
  55. 55. Processing / Denormalization pre_save, post_save,post_delete and F objects Use these.
  56. 56. Processing / DenormalizationBe careful These only work in Django
  57. 57. ProcessingWorkers
  58. 58. Processing / WorkersDeconstruct the problem
  59. 59. Check for profanity then Retrieve an avatar then Geo-locate the author thenAdd as input for trending terms thenRetrieve author’s social graph then Adjust the leaderboard
  60. 60. Retrieve an avatarCheck for profanity Geo-locate the author Add as input for trending terms then Retrieve author’s social graph Adjust the leaderboard
  61. 61. Processing / Workersdjango-celeryOr manage yourself with any queue
  62. 62. Processingmap + reduce It’s not that scary
  63. 63. Processing / map + reduceGenerationalGet data. Process. Cache results.
  64. 64. Processing / map + reduce / GenerationalGood for many problems. Especially where the intermediate working set is large.
  65. 65. Processing / map + reduce / GenerationalSolutions exist. CouchDB, Mongo, Hadoop
  66. 66. Processing / map + reduceSometimes we can be smarter
  67. 67. Processing / map + reduce / incrementalConsider averages. I mean the mean.
  68. 68. Processing / map + reduce / incremental n 1 n Σi=1 ai
  69. 69. Processing / map + reduce / incremental = ( ) (Σ) n n-11 1n Σ i=1 ai n i=1 ai + a n From O(n) to O(1)
  70. 70. Processing / map + reduce / incremental This example was trivial, but you can oftenStore a partial solution
  71. 71. Presentation(Of the data, not the thing we’re doing now.)
  72. 72. PresentationPartial Caching
  73. 73. Presentation / Partial Caching{% cache 500 my_stuff %} Template fragment caching
  74. 74. Presentation / Partial Cachingclass MyModel(models.Model): as_html = models.TextField()
  75. 75. Presentation / Partial Cachingserialized = json.dumps(my_stuff) cache.set(‘my_stuff’, serialized) Don’t be afraid of low-level caching.
  76. 76. PresentationContinuous Caching
  77. 77. Presentation / Continuous Cachingwhile True: cache_page()
  78. 78. Presentation / Continuous CachingOut-of-band caching. Works when the number of pages is relatively small. Similar to proxy_cache, but more resilient.
  79. 79. Presentation / Continuous Caching[Watch this space.]
  80. 80. PresentationReal-Time Updates
  81. 81. Presentation / Real-Time Updatesgevent, eventlet,tornado, twisted
  82. 82. Presentation / Real-Time UpdatesDjango plays well with others
  83. 83. Presentation / Real-Time Updates Django + RabbitMQ + node.js + socket.io
  84. 84. Presentation / Real-Time Updates[Watch this space.]
  85. 85. PresentationFailure Models
  86. 86. Presentation / Failure ModelsThis isn’t good for anyone.
  87. 87. Presentation / Failure Modelsproxy_cache_use_stale andproxy_next_upstream. For use with nginx
  88. 88. Presentation / Failure ModelsBuild a small backup app. It can serve pre-cached content. Anything is better than a 404, 500 or 502 (usually)
  89. 89. Follow @bolsterlabs for slides and lively discussion.
  90. 90. Thank you.
  91. 91. Don’t be a stranger. Ben Slavin Adam Miskiewicz @benslavin @skevyben@bolsterlabs.com adam@bolsterlabs.com @bolsterlabs

×