Tricks & challenges developing a large Django application


Published on

Presented at EuroPython 2011 on the 22nd of June.

Tricks & challenges developing a large Django application

  1. Tricks & challengesdeveloping a largeDjango application Simon Willison EuroPython 2011, 22nd June
  3. Definitive database of Professional Events and Speakers
  4. Definitive database Social event recommendation of Professional Events Event networking tools and Speakers Archive of slides, notes and video
  5. Part 1: Tricks
  6. Trick #1: Signing
  7. Lanyrd.comPass data through anuntrusted source withconfidence that it hasntbeen tampered with
  8. Lanyrd.comSigning uses Unsubscribe" links in Signed cookies "You are logged in as simonw" withouthitting the database
  9. Lanyrd.comSigning in Django 1.4 from django.core import signing signing.dumps({"foo": "bar"}) signing.loads(signed_string) response.set_signed_cookie(key, value...) request.get_signed_cookie(key)
  10. Trick #2: cache_version
  11. add a conference you are signed in as simonw, do you want to sign out? calendar conferences coverage profile searchDjango conferences Django Django events looking for participants coverage 1 Django event is looking for participants 52 videos Most recent added 3 weeks agoON NOW EuroPython 2011 52 slide decks 19 Most recent added 4 Italy / Florence hours ago 19th–26th June 2011 Django Plone Pyramid Python Twisted 3 audio clips Most recent added 1 week ago 27 write-upsSEPTEMBER DjangoCon US 2011 6 Most recent added 12011 United States / Portland 6th–8th September 2011 week ago 11 handouts Django Open Source Python Most recent added 18 hours ago 17 PyCON FR 2011 3 notes France / Rennes Most recent added 10 17th–18th September 2011 hours ago Django Python By countryOCTOBER PyCon DE 2011 Ireland 1 4
  12. Lanyrd.comclass Conference(models.Model): ... cache_version = models.IntegerField(default = 0) def save(self, *args, **kwargs): self.cache_version += 1 super(Conference, self).save(*args, **kwargs) def touch(self): Conference.objects.filter( pk = ).update( cache_version = F(cache_version) + 1 )
  13.{% cache 36000 conf-topics conference.cache_version %} <ul class="tags inline-tags meta"> {% for topic in conference.topics.all %} <li><a href="{{ topic.get_absolute_url }}">{{ topic }}</a></li> {% endfor %} </ul>{% endcache %}
  14. Lanyrd.comBulk invalidationfrom django.models import Ftopic.conferences.all().update( cache_version = F(cache_version) + 1)
  15. Trick #3: NoSQL for denormalisation
  16. Lanyrd.comNoSQL = Not Only SQL
  17. Lanyrd.comBut we like joins! Slides from a session on Lanyrd show up... Under "coverage of this conference" Under "slides by Simon Willison" Under "slides about Django"
  18. Lanyrd.comSolution: denormalisedata to Redis and Solr
  19. Commands Clients Documentation Community Download Issues Redis is an open source, advanced key-value store. It is often What people are saying referred to as a data structure server since keys can contain Comparison of CouchDB, Redis, MongoDB, Casandra, Neo4J & strings, hashes, lists, sets and sorted sets. strings hashes lists sets others via @DZone Learn more → @__NeverGiveup Oh YAY, oui tu me redis ! *-* Hm, on srejoint à Try it Download it 14h au bahut ? :o Ready for a test drive? Check this interactive Redis 2.2.10 is the latest stable version. JE L REDIS JE FOLLOW BACK SUR @Fuckement_TL tutorial that will walk you through the most Interested in legacy or unstable versions? important features of Redis. Check the downloads page. une question : "How to use ServiceStack Redis in a web application to take advantage of pub / sub paradigm" #redis #web Nice - Cassandra vs MongoDB vs CouchDB vs Redis vs Riak vs HBase vs Membase vs Neo4j comparison from @kkovacs More... Sponsored by This website is open source software developed by Citrusbyte. The Redis logo was designed by Carlos Prioglio.
  20. Lanyrd.comOur redis sets... simonw-follows:{144,21345,12328...} europython-attendees:{344,21345,787...} redis.sinter( simonw-follows, europython-attendees )
  21. add a conference you are signed in as simonw, do you want to sign out? calendar conferences coverage profile search EuroPython 2011 Youre speaking The European Python Conference AT THIS EVENT 19 –26 JUNE 2011 Florence in Italy 97 attending @europython PEOPLE View the schedule on Lanyrd #europython 80 tracking PEOPLE Save to iCal / iPhone / Outlook / (short URL) GCal TELL YOUR FRIENDS! Tweet about this event 119 speakers Andreas Alan Anna Schreiber Franzoni Ravenscroft Topics @onyame @franzeur Django Andrew Alessandro Anselm Kruis Godwin Dentella Plone @andrewgodwin Pyramid Andrii Alex Martelli Antonio Cuni @antocuni Python Mishkovskyi @mishok13 Twisted Ali Afshar Armin Rigo Armin Edit topics
  22. Lanyrd.comapache > lucene > solr Search the site with Solr Search Main Wiki Powered by Lucid Imagination Last Published: Sat, 04 Jun 2011 12:23:42 GMT About Welcome Who We Are Welcome to Solr Documentation PDF Resources What Is Solr? Related Projects Get Started News May 2011 - Solr 3.2 Released March 2011 - Solr 3.1 Released 25 June 2010 - Solr 1.4.1 Released 7 May 2010 - Apache Lucene Eurocon 2010 Coming to Prague May 18-21 10 November 2009 - Solr 1.4 Released 20 August 2009 - Solrs first book is published! 18 August 2009 - Lucene at US ApacheCon 09 February 2009 - Lucene at ApacheCon Europe 2009 in Amsterdam 19 December 2008 - Solr Logo Contest Results 03 October 2008 - Solr Logo Contest 15 September 2008 - Solr 1.3.0 Available 28 August 2008 - Lucene/Solr at ApacheCon New Orleans 03 September 2007 - Lucene at ApacheCon Atlanta 06 June 2007: Release 1.2 available 17 January 2007: Solr graduates from Incubator 22 December 2006: Release 1.1.0 available 15 August 2006: Solr at ApacheCon US 21 April 2006: Solr at ApacheCon
  23. add a conference you are signed in as simonw, do you want to sign out? calendar conferences coverage profile search Search We found 467 results for “django” FILTER BY django Search type Sessions 209 Topic: Django Coverage 190 ON NOW 1 conference FUTURE 5 conferences Conferences 60 PAST 51 conferences Books 5 Topic: GeoDjango Topics 2 PAST 1 conference People 1 Practical Django Projects By James Bennett PUBLISHED June 2009 FILTER BY past or future Django Geek Past events 55 @djangeek Future events 5 SLIDES
  24. add a conference you are signed in as simonw, do you want to sign out? calendar conferences coverage profile search Your contacts calendar yours 24 contacts 182 Simon Weve found 182 conferences your Twitter contacts are Willison interested in. Your profile page TODAY Café Scientifique: Exploring Attend 21 the dark side of star Track formation with the Herschel From our blog Space Observatory Welcoming Sophie United Kingdom / Brighton Barrett to team 21st June 2011 Lanyrd Astronomy Science Today we have a very special announcement (and for once, 4 contacts tracking its not a new feature!) We would like to welcome the super-wonderful Sophie Barrett to the Lanyrd team. 21 Usability Professionals Attend Session schedules in Association – International Track your calendar Conference You can now subscribe to event schedules in your calendar of United States / Atlanta
  25. CSS Maintainable CSS Maintainability @Natbat on Twitter 1 write-up Design Process Internet Explorer Added 7 months ago 16 CONFERENCES spoken at Show and tell Personal Projects 2 links Most recent added 7 14 CONFERENCES Web Standards Crowdsourcing months ago involved with SEE ALL COVERAGE WildlifeNearYou RSI Community 11 items in total Event Planning Geek Night User Experience Browse by year Usability Usability Testing JavaScript FAVOURITE COVERAGE See 6 items that Natalie Downe has 2011 ! 2010 ! 2009 ! Progressive Enhancement favourited 2008 ! 2007 ! 2006 ! 2005 Unobtrusive JavaScript Flexible design Planning Artifacts Social Software Book credits Appears with TECHNICAL REVIEWER Motivation CSS3 CSS Layout The Ultimate CSS Simon Willison Reference Project Management Dev Fort 9 times TECHNICAL EDITOR Rapid Application Development CSS Mastery Nigel Crawley 5 times Reinier Zwitserloot Full speaking history for Natalie Downe 4 times PAST CONFERENCES FUTURE CONFERENCES Neil Crosby 4 times spoken at speaking at Cristiano Betta Five Pound App #22 London Insites Tour 4 times Brighton, United Kingdom Evening Matt Westcott 30th March 2010 London, United Kingdom 4 times 19th July 2011 CSS3 Wizardry Gareth Rushgrove Workshop 4 times FUTURE CONFERENCES Brighton, United Kingdom 29th January 2010 attending
  26. Trick #4: Hashed static asset filenames in CloudFront
  28. Lanyrd.comBenefits Far futures expiry headers Cache-Control: max-age=315360000 Expires: Fri, 18 Jun 2021 06:45:00 -0000 GMT Guaranteed updated CSS in IE Deploy new assets in advance of application Old versions stick around for rollbacks
  29. push_static Minifies JavaScript and CSS Renames files to include sha1(contents)[:8] Pushes all assets to S3
  30. Part 2: Challenges
  31. Challenge #1: HTTP requests
  32. Lanyrd.comTalking to an API? What if it fails to load? What if it takes 30 seconds to load?
  33. Lanyrd.comA user gives you a URL...urllib.urlopen(url).read() ? what if its a huge file? what if its a slow loading tarpit? what if its the URL to your private, firewalled Solr instance?
  34. Lanyrd.comSafe URL consumption Connection timeouts Logging and Profiling Host validation HTTP caching / if-none-match / etc django.httpclient ?
  35. Challenge #2: Profiling anddebugging production systems
  36. Lanyrd.comDebugging indevelopment rocks! django_debug_toolbar = awesome assert False = instant useful 500 page import pdb; pdb.set_trace() adds a command line debugger to your running application!
  37. Lanyrd.comDEBUG = FalseDEBUG = "BLIND"
  38. Lanyrd.comfrom django.views.debug import technical_500_responseimport sysclass UserBasedExceptionMiddleware(object): def process_exception(self, request, exception): if request.user.is_superuser: return technical_500_response(request, *sys.exc_info())
  39. Lanyrd.commysql-proxy Very handy lua-customisable proxy for all of your MySQL traffic Worst documented software ever log.lua - logs out ALL queries
  40. Lanyrd.comdjango_instrumented (Unreleased) code I wrote for Lanyrd Collects various runtime stats about the current request, stashes a profile JSON in memcached Writes out the profile UUID as part of the HTML A bookmarklet to view the profile
  42. Lanyrd.comDjango improvements DEBUG as a global setting is an anti- pattern More low-level hooks for measuring, well, pretty much everything Live profiling tools (as a third-party product)
  43. MVC Mini Profiler
  44. Challenge #3: Zero downtime deploys
  45. Lanyrd.comChallenge:deploy upgrades,including database upgrades,without downtime
  46. Lanyrd.comDeploy DB changesseparately from code changes Make backwards compatible schema changes We deploy code to a none-web server just so we can run "./ migrate" (you are using South, right?) Use symlinks for instant rollback to previous code
  47. Lanyrd.comRead-only modeIf you can flip your site in to read-only mode,upgrades become a lot easier Flip to read only mode Replicate DB to a new machine Run migrations, then test Switch app servers to new database (On EC2, fire up an entire new web cluster)Feature flags: finely grained version of this
  48. The lesson we keep on learning
  49. Lanyrd.comGlobal settings are BAD Database settings => multi-db (1.2) Cache settings => multi cache backends (1.3) Haystack backends => multi backends (v2.0)
  50. Lanyrd.comGlobal settings are BAD Database settings => multi-db (1.2) Cache settings => multi cache backends (1.3) Haystack backends => multi backends (v2.0) DEBUG TIME_ZONE Middleware?
  51. Lanyrd.comSettings shouldbe modifiableat run-time
  52. Thank you![ [Were hiring a developer/web operations person in London, come and talk to us!