Your SlideShare is downloading. ×
Tricks & challenges developing a large Django application
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Tricks & challenges developing a large Django application


Published on

Presented at EuroPython 2011 on the 22nd of June.

Presented at EuroPython 2011 on the 22nd of June.

  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Tricks & challengesdeveloping a largeDjango application Simon Willison EuroPython 2011, 22nd June
  • 2.
  • 3. Definitive database of Professional Events and Speakers
  • 4. Definitive database Social event recommendation of Professional Events Event networking tools and Speakers Archive of slides, notes and video
  • 5. Part 1: Tricks
  • 6. Trick #1: Signing
  • 7. Lanyrd.comPass data through anuntrusted source withconfidence that it hasntbeen tampered with
  • 8. Lanyrd.comSigning uses Unsubscribe" links in Signed cookies "You are logged in as simonw" withouthitting the database
  • 9. Lanyrd.comSigning in Django 1.4 from django.core import signing signing.dumps({"foo": "bar"}) signing.loads(signed_string) response.set_signed_cookie(key, value...) request.get_signed_cookie(key)
  • 10. Trick #2: cache_version
  • 11. add a conference you are signed in as simonw, do you want to sign out? calendar conferences coverage profile searchDjango conferences Django Django events looking for participants coverage 1 Django event is looking for participants 52 videos Most recent added 3 weeks agoON NOW EuroPython 2011 52 slide decks 19 Most recent added 4 Italy / Florence hours ago 19th–26th June 2011 Django Plone Pyramid Python Twisted 3 audio clips Most recent added 1 week ago 27 write-upsSEPTEMBER DjangoCon US 2011 6 Most recent added 12011 United States / Portland 6th–8th September 2011 week ago 11 handouts Django Open Source Python Most recent added 18 hours ago 17 PyCON FR 2011 3 notes France / Rennes Most recent added 10 17th–18th September 2011 hours ago Django Python By countryOCTOBER PyCon DE 2011 Ireland 1 4
  • 12. Lanyrd.comclass Conference(models.Model): ... cache_version = models.IntegerField(default = 0) def save(self, *args, **kwargs): self.cache_version += 1 super(Conference, self).save(*args, **kwargs) def touch(self): Conference.objects.filter( pk = ).update( cache_version = F(cache_version) + 1 )
  • 13.{% cache 36000 conf-topics conference.cache_version %} <ul class="tags inline-tags meta"> {% for topic in conference.topics.all %} <li><a href="{{ topic.get_absolute_url }}">{{ topic }}</a></li> {% endfor %} </ul>{% endcache %}
  • 14. Lanyrd.comBulk invalidationfrom django.models import Ftopic.conferences.all().update( cache_version = F(cache_version) + 1)
  • 15. Trick #3: NoSQL for denormalisation
  • 16. Lanyrd.comNoSQL = Not Only SQL
  • 17. Lanyrd.comBut we like joins! Slides from a session on Lanyrd show up... Under "coverage of this conference" Under "slides by Simon Willison" Under "slides about Django"
  • 18. Lanyrd.comSolution: denormalisedata to Redis and Solr
  • 19. Commands Clients Documentation Community Download Issues Redis is an open source, advanced key-value store. It is often What people are saying referred to as a data structure server since keys can contain Comparison of CouchDB, Redis, MongoDB, Casandra, Neo4J & strings, hashes, lists, sets and sorted sets. strings hashes lists sets others via @DZone Learn more → @__NeverGiveup Oh YAY, oui tu me redis ! *-* Hm, on srejoint à Try it Download it 14h au bahut ? :o Ready for a test drive? Check this interactive Redis 2.2.10 is the latest stable version. JE L REDIS JE FOLLOW BACK SUR @Fuckement_TL tutorial that will walk you through the most Interested in legacy or unstable versions? important features of Redis. Check the downloads page. une question : "How to use ServiceStack Redis in a web application to take advantage of pub / sub paradigm" #redis #web Nice - Cassandra vs MongoDB vs CouchDB vs Redis vs Riak vs HBase vs Membase vs Neo4j comparison from @kkovacs More... Sponsored by This website is open source software developed by Citrusbyte. The Redis logo was designed by Carlos Prioglio.
  • 20. Lanyrd.comOur redis sets... simonw-follows:{144,21345,12328...} europython-attendees:{344,21345,787...} redis.sinter( simonw-follows, europython-attendees )
  • 21. add a conference you are signed in as simonw, do you want to sign out? calendar conferences coverage profile search EuroPython 2011 Youre speaking The European Python Conference AT THIS EVENT 19 –26 JUNE 2011 Florence in Italy 97 attending @europython PEOPLE View the schedule on Lanyrd #europython 80 tracking PEOPLE Save to iCal / iPhone / Outlook / (short URL) GCal TELL YOUR FRIENDS! Tweet about this event 119 speakers Andreas Alan Anna Schreiber Franzoni Ravenscroft Topics @onyame @franzeur Django Andrew Alessandro Anselm Kruis Godwin Dentella Plone @andrewgodwin Pyramid Andrii Alex Martelli Antonio Cuni @antocuni Python Mishkovskyi @mishok13 Twisted Ali Afshar Armin Rigo Armin Edit topics
  • 22. Lanyrd.comapache > lucene > solr Search the site with Solr Search Main Wiki Powered by Lucid Imagination Last Published: Sat, 04 Jun 2011 12:23:42 GMT About Welcome Who We Are Welcome to Solr Documentation PDF Resources What Is Solr? Related Projects Get Started News May 2011 - Solr 3.2 Released March 2011 - Solr 3.1 Released 25 June 2010 - Solr 1.4.1 Released 7 May 2010 - Apache Lucene Eurocon 2010 Coming to Prague May 18-21 10 November 2009 - Solr 1.4 Released 20 August 2009 - Solrs first book is published! 18 August 2009 - Lucene at US ApacheCon 09 February 2009 - Lucene at ApacheCon Europe 2009 in Amsterdam 19 December 2008 - Solr Logo Contest Results 03 October 2008 - Solr Logo Contest 15 September 2008 - Solr 1.3.0 Available 28 August 2008 - Lucene/Solr at ApacheCon New Orleans 03 September 2007 - Lucene at ApacheCon Atlanta 06 June 2007: Release 1.2 available 17 January 2007: Solr graduates from Incubator 22 December 2006: Release 1.1.0 available 15 August 2006: Solr at ApacheCon US 21 April 2006: Solr at ApacheCon
  • 23. add a conference you are signed in as simonw, do you want to sign out? calendar conferences coverage profile search Search We found 467 results for “django” FILTER BY django Search type Sessions 209 Topic: Django Coverage 190 ON NOW 1 conference FUTURE 5 conferences Conferences 60 PAST 51 conferences Books 5 Topic: GeoDjango Topics 2 PAST 1 conference People 1 Practical Django Projects By James Bennett PUBLISHED June 2009 FILTER BY past or future Django Geek Past events 55 @djangeek Future events 5 SLIDES
  • 24. add a conference you are signed in as simonw, do you want to sign out? calendar conferences coverage profile search Your contacts calendar yours 24 contacts 182 Simon Weve found 182 conferences your Twitter contacts are Willison interested in. Your profile page TODAY Café Scientifique: Exploring Attend 21 the dark side of star Track formation with the Herschel From our blog Space Observatory Welcoming Sophie United Kingdom / Brighton Barrett to team 21st June 2011 Lanyrd Astronomy Science Today we have a very special announcement (and for once, 4 contacts tracking its not a new feature!) We would like to welcome the super-wonderful Sophie Barrett to the Lanyrd team. 21 Usability Professionals Attend Session schedules in Association – International Track your calendar Conference You can now subscribe to event schedules in your calendar of United States / Atlanta
  • 25. CSS Maintainable CSS Maintainability @Natbat on Twitter 1 write-up Design Process Internet Explorer Added 7 months ago 16 CONFERENCES spoken at Show and tell Personal Projects 2 links Most recent added 7 14 CONFERENCES Web Standards Crowdsourcing months ago involved with SEE ALL COVERAGE WildlifeNearYou RSI Community 11 items in total Event Planning Geek Night User Experience Browse by year Usability Usability Testing JavaScript FAVOURITE COVERAGE See 6 items that Natalie Downe has 2011 ! 2010 ! 2009 ! Progressive Enhancement favourited 2008 ! 2007 ! 2006 ! 2005 Unobtrusive JavaScript Flexible design Planning Artifacts Social Software Book credits Appears with TECHNICAL REVIEWER Motivation CSS3 CSS Layout The Ultimate CSS Simon Willison Reference Project Management Dev Fort 9 times TECHNICAL EDITOR Rapid Application Development CSS Mastery Nigel Crawley 5 times Reinier Zwitserloot Full speaking history for Natalie Downe 4 times PAST CONFERENCES FUTURE CONFERENCES Neil Crosby 4 times spoken at speaking at Cristiano Betta Five Pound App #22 London Insites Tour 4 times Brighton, United Kingdom Evening Matt Westcott 30th March 2010 London, United Kingdom 4 times 19th July 2011 CSS3 Wizardry Gareth Rushgrove Workshop 4 times FUTURE CONFERENCES Brighton, United Kingdom 29th January 2010 attending
  • 26. Trick #4: Hashed static asset filenames in CloudFront
  • 27.
  • 28. Lanyrd.comBenefits Far futures expiry headers Cache-Control: max-age=315360000 Expires: Fri, 18 Jun 2021 06:45:00 -0000 GMT Guaranteed updated CSS in IE Deploy new assets in advance of application Old versions stick around for rollbacks
  • 29. push_static Minifies JavaScript and CSS Renames files to include sha1(contents)[:8] Pushes all assets to S3
  • 30. Part 2: Challenges
  • 31. Challenge #1: HTTP requests
  • 32. Lanyrd.comTalking to an API? What if it fails to load? What if it takes 30 seconds to load?
  • 33. Lanyrd.comA user gives you a URL...urllib.urlopen(url).read() ? what if its a huge file? what if its a slow loading tarpit? what if its the URL to your private, firewalled Solr instance?
  • 34. Lanyrd.comSafe URL consumption Connection timeouts Logging and Profiling Host validation HTTP caching / if-none-match / etc django.httpclient ?
  • 35. Challenge #2: Profiling anddebugging production systems
  • 36. Lanyrd.comDebugging indevelopment rocks! django_debug_toolbar = awesome assert False = instant useful 500 page import pdb; pdb.set_trace() adds a command line debugger to your running application!
  • 37. Lanyrd.comDEBUG = FalseDEBUG = "BLIND"
  • 38. Lanyrd.comfrom django.views.debug import technical_500_responseimport sysclass UserBasedExceptionMiddleware(object): def process_exception(self, request, exception): if request.user.is_superuser: return technical_500_response(request, *sys.exc_info())
  • 39. Lanyrd.commysql-proxy Very handy lua-customisable proxy for all of your MySQL traffic Worst documented software ever log.lua - logs out ALL queries
  • 40. Lanyrd.comdjango_instrumented (Unreleased) code I wrote for Lanyrd Collects various runtime stats about the current request, stashes a profile JSON in memcached Writes out the profile UUID as part of the HTML A bookmarklet to view the profile
  • 41.
  • 42. Lanyrd.comDjango improvements DEBUG as a global setting is an anti- pattern More low-level hooks for measuring, well, pretty much everything Live profiling tools (as a third-party product)
  • 43. MVC Mini Profiler
  • 44. Challenge #3: Zero downtime deploys
  • 45. Lanyrd.comChallenge:deploy upgrades,including database upgrades,without downtime
  • 46. Lanyrd.comDeploy DB changesseparately from code changes Make backwards compatible schema changes We deploy code to a none-web server just so we can run "./ migrate" (you are using South, right?) Use symlinks for instant rollback to previous code
  • 47. Lanyrd.comRead-only modeIf you can flip your site in to read-only mode,upgrades become a lot easier Flip to read only mode Replicate DB to a new machine Run migrations, then test Switch app servers to new database (On EC2, fire up an entire new web cluster)Feature flags: finely grained version of this
  • 48. The lesson we keep on learning
  • 49. Lanyrd.comGlobal settings are BAD Database settings => multi-db (1.2) Cache settings => multi cache backends (1.3) Haystack backends => multi backends (v2.0)
  • 50. Lanyrd.comGlobal settings are BAD Database settings => multi-db (1.2) Cache settings => multi cache backends (1.3) Haystack backends => multi backends (v2.0) DEBUG TIME_ZONE Middleware?
  • 51. Lanyrd.comSettings shouldbe modifiableat run-time
  • 52. Thank you![ [Were hiring a developer/web operations person in London, come and talk to us!