Tricks & challenges developing a large Django application

Tricks & challengesdeveloping a largeDjango application Simon Willison EuroPython 2011, 22nd June http://lanyrd.com/sfwcq

Lanyrd.com

Lanyrd.com Deﬁnitive database of Professional Events and Speakers

Lanyrd.com Deﬁnitive database Social event recommendation of Professional Events Event networking tools and Speakers Archive of slides, notes and video

Part 1: Tricks

Trick #1: Signing

Lanyrd.comPass data through anuntrusted source withconﬁdence that it hasntbeen tampered with

Lanyrd.comSigning uses Unsubscribe" links in emailslanyrd.com/un/ImN6VyI.ii0Hwm7p71DEcGfaVzziQaxeuu Signed cookies "You are logged in as simonw" withouthitting the database

Lanyrd.comSigning in Django 1.4 from django.core import signing signing.dumps({"foo": "bar"}) signing.loads(signed_string) response.set_signed_cookie(key, value...) request.get_signed_cookie(key)

Trick #2: cache_version

add a conference you are signed in as simonw, do you want to sign out? calendar conferences coverage profile searchDjango conferences Django Django events looking for participants coverage 1 Django event is looking for participants 52 videos Most recent added 3 weeks agoON NOW EuroPython 2011 52 slide decks 19 Most recent added 4 Italy / Florence hours ago 19th–26th June 2011 Django Plone Pyramid Python Twisted 3 audio clips Most recent added 1 week ago 27 write-upsSEPTEMBER DjangoCon US 2011 6 Most recent added 12011 United States / Portland 6th–8th September 2011 week ago 11 handouts Django Open Source Python Most recent added 18 hours ago 17 PyCON FR 2011 3 notes France / Rennes Most recent added 10 17th–18th September 2011 hours ago Django Python By countryOCTOBER PyCon DE 2011 Ireland 1 4

Lanyrd.comclass Conference(models.Model): ... cache_version = models.IntegerField(default = 0) def save(self, *args, **kwargs): self.cache_version += 1 super(Conference, self).save(*args, **kwargs) def touch(self): Conference.objects.filter( pk = self.pk ).update( cache_version = F(cache_version) + 1 )

Lanyrd.com{% cache 36000 conf-topics conference.pk conference.cache_version %} <ul class="tags inline-tags meta"> {% for topic in conference.topics.all %} <li><a href="{{ topic.get_absolute_url }}">{{ topic }}</a></li> {% endfor %} </ul>{% endcache %}

Lanyrd.comBulk invalidationfrom django.models import Ftopic.conferences.all().update( cache_version = F(cache_version) + 1)

Trick #3: NoSQL for denormalisation

Lanyrd.comNoSQL = Not Only SQL

Lanyrd.comBut we like joins! Slides from a session on Lanyrd show up... Under "coverage of this conference" Under "slides by Simon Willison" Under "slides about Django"

Lanyrd.comSolution: denormalisedata to Redis and Solr

Lanyrd.com Commands Clients Documentation Community Download Issues Redis is an open source, advanced key-value store. It is often What people are saying referred to as a data structure server since keys can contain Comparison of CouchDB, Redis, MongoDB, Casandra, Neo4J & strings, hashes, lists, sets and sorted sets. strings hashes lists sets others http://j.mp/l32SqM via @DZone Learn more → @__NeverGiveup Oh YAY, oui tu me redis ! *-* Hm, on srejoint à Try it Download it 14h au bahut ? :o Ready for a test drive? Check this interactive Redis 2.2.10 is the latest stable version. JE L REDIS JE FOLLOW BACK SUR @Fuckement_TL tutorial that will walk you through the most Interested in legacy or unstable versions? important features of Redis. Check the downloads page. une question : "How to use ServiceStack Redis in a web application to take advantage of pub / sub paradigm" http://t.co/EOgyLU1 #redis #web Nice - Cassandra vs MongoDB vs CouchDB vs Redis vs Riak vs HBase vs Membase vs Neo4j comparison http://bit.ly/l32SqM from @kkovacs More... Sponsored by This website is open source software developed by Citrusbyte. The Redis logo was designed by Carlos Prioglio.

Lanyrd.comOur redis sets... simonw-follows:{144,21345,12328...} europython-attendees:{344,21345,787...} redis.sinter( simonw-follows, europython-attendees )

add a conference you are signed in as simonw, do you want to sign out?Lanyrd.com calendar conferences coverage profile search EuroPython 2011 Youre speaking The European Python Conference AT THIS EVENT 19 –26 JUNE 2011 Florence in Italy 97 attending http://ep2011.europython.eu/ @europython PEOPLE View the schedule on Lanyrd #europython 80 tracking PEOPLE Save to iCal / iPhone / Outlook / lanyrd.com/ccdpc (short URL) GCal TELL YOUR FRIENDS! Tweet about this event 119 speakers Andreas Alan Anna Schreiber Franzoni Ravenscroft Topics @onyame @franzeur Django Andrew Alessandro Anselm Kruis Godwin Dentella Plone @andrewgodwin Pyramid Andrii Alex Martelli Antonio Cuni @antocuni Python Mishkovskyi @mishok13 Twisted Ali Afshar Armin Rigo Armin Edit topics

Lanyrd.comapache > lucene > solr Search the site with Solr Search Main Wiki Powered by Lucid Imagination Last Published: Sat, 04 Jun 2011 12:23:42 GMT About Welcome Who We Are Welcome to Solr Documentation PDF Resources What Is Solr? Related Projects Get Started News May 2011 - Solr 3.2 Released March 2011 - Solr 3.1 Released 25 June 2010 - Solr 1.4.1 Released 7 May 2010 - Apache Lucene Eurocon 2010 Coming to Prague May 18-21 10 November 2009 - Solr 1.4 Released 20 August 2009 - Solrs first book is published! 18 August 2009 - Lucene at US ApacheCon 09 February 2009 - Lucene at ApacheCon Europe 2009 in Amsterdam 19 December 2008 - Solr Logo Contest Results 03 October 2008 - Solr Logo Contest 15 September 2008 - Solr 1.3.0 Available 28 August 2008 - Lucene/Solr at ApacheCon New Orleans 03 September 2007 - Lucene at ApacheCon Atlanta 06 June 2007: Release 1.2 available 17 January 2007: Solr graduates from Incubator 22 December 2006: Release 1.1.0 available 15 August 2006: Solr at ApacheCon US 21 April 2006: Solr at ApacheCon

Lanyrd.com add a conference you are signed in as simonw, do you want to sign out? calendar conferences coverage profile search Search We found 467 results for “django” FILTER BY django Search type Sessions 209 Topic: Django Coverage 190 ON NOW 1 conference FUTURE 5 conferences Conferences 60 PAST 51 conferences Books 5 Topic: GeoDjango Topics 2 PAST 1 conference People 1 Practical Django Projects By James Bennett PUBLISHED June 2009 FILTER BY past or future Django Geek Past events 55 @djangeek Future events 5 SLIDES

Lanyrd.com add a conference you are signed in as simonw, do you want to sign out? calendar conferences coverage profile search Your contacts calendar yours 24 contacts 182 Simon Weve found 182 conferences your Twitter contacts are Willison interested in. Your profile page TODAY Café Scientifique: Exploring Attend 21 the dark side of star Track formation with the Herschel From our blog Space Observatory Welcoming Sophie United Kingdom / Brighton Barrett to team 21st June 2011 Lanyrd Astronomy Science Today we have a very special announcement (and for once, 4 contacts tracking its not a new feature!) We would like to welcome the super-wonderful Sophie Barrett to the Lanyrd team. 21 Usability Professionals Attend Session schedules in Association – International Track your calendar Conference You can now subscribe to event schedules in your calendar of United States / Atlanta

CSS Maintainable CSS Maintainability @Natbat on Twitter 1 write-up Design Process Internet Explorer DebuggingLanyrd.com Added 7 months ago 16 CONFERENCES spoken at Show and tell Personal Projects 2 links Most recent added 7 14 CONFERENCES Web Standards Crowdsourcing months ago involved with SEE ALL COVERAGE WildlifeNearYou RSI Community 11 items in total Event Planning Geek Night User Experience Browse by year Usability Usability Testing JavaScript FAVOURITE COVERAGE See 6 items that Natalie Downe has 2011 ! 2010 ! 2009 ! Progressive Enhancement favourited 2008 ! 2007 ! 2006 ! 2005 Unobtrusive JavaScript Flexible design Planning Artifacts Social Software Book credits Appears with TECHNICAL REVIEWER Motivation CSS3 CSS Layout The Ultimate CSS Simon Willison Reference Project Management Dev Fort 9 times TECHNICAL EDITOR Rapid Application Development CSS Mastery Nigel Crawley 5 times Reinier Zwitserloot Full speaking history for Natalie Downe 4 times PAST CONFERENCES FUTURE CONFERENCES Neil Crosby 4 times spoken at speaking at Cristiano Betta Five Pound App #22 London Insites Tour 4 times Brighton, United Kingdom Evening Matt Westcott 30th March 2010 London, United Kingdom 4 times 19th July 2011 CSS3 Wizardry Gareth Rushgrove Workshop 4 times FUTURE CONFERENCES Brighton, United Kingdom 29th January 2010 attending

Trick #4: Hashed static asset ﬁlenames in CloudFront

Lanyrd.comcdn.lanyrd.net/js/global.ed81d119.jsglobal.jsglobal.ed81d119.js

Lanyrd.comBeneﬁts Far futures expiry headers Cache-Control: max-age=315360000 Expires: Fri, 18 Jun 2021 06:45:00 -0000 GMT Guaranteed updated CSS in IE Deploy new assets in advance of application Old versions stick around for rollbacks

Lanyrd.com./manage.py push_static Miniﬁes JavaScript and CSS Renames ﬁles to include sha1(contents)[:8] Pushes all assets to S3

Part 2: Challenges

Challenge #1: HTTP requests

Lanyrd.comTalking to an API? What if it fails to load? What if it takes 30 seconds to load?

Lanyrd.comA user gives you a URL...urllib.urlopen(url).read() ? what if its a huge ﬁle? what if its a slow loading tarpit? what if its the URL to your private, ﬁrewalled Solr instance? http://10.0.1.1/solr/select/?q=secrets

Lanyrd.comSafe URL consumption Connection timeouts Logging and Proﬁling Host validation HTTP caching / if-none-match / etc django.httpclient ?

Challenge #2: Proﬁling anddebugging production systems

Lanyrd.comDebugging indevelopment rocks! django_debug_toolbar = awesome assert False = instant useful 500 page import pdb; pdb.set_trace() adds a command line debugger to your running application!

Lanyrd.comDEBUG = FalseDEBUG = "BLIND"

Lanyrd.comfrom django.views.debug import technical_500_responseimport sysclass UserBasedExceptionMiddleware(object): def process_exception(self, request, exception): if request.user.is_superuser: return technical_500_response(request, *sys.exc_info())

Lanyrd.commysql-proxy Very handy lua-customisable proxy for all of your MySQL traﬃc Worst documented software ever log.lua - logs out ALL queries https://gist.github.com/1039751

Lanyrd.comdjango_instrumented (Unreleased) code I wrote for Lanyrd Collects various runtime stats about the current request, stashes a profile JSON in memcached Writes out the profile UUID as part of the HTML A bookmarklet to view the profile

Lanyrd.com

Lanyrd.comDjango improvements DEBUG as a global setting is an anti- pattern More low-level hooks for measuring, well, pretty much everything Live proﬁling tools (as a third-party product)

MVC Mini Proﬁlerhttp://www.codinghorror.com/blog/2011/06/performance-is-a-feature.html

Challenge #3: Zero downtime deploys

Lanyrd.comChallenge:deploy upgrades,including database upgrades,without downtime

Lanyrd.comDeploy DB changesseparately from code changes Make backwards compatible schema changes We deploy code to a none-web server just so we can run "./manage.py migrate" (you are using South, right?) Use symlinks for instant rollback to previous code

Lanyrd.comRead-only modeIf you can flip your site in to read-only mode,upgrades become a lot easier Flip to read only mode Replicate DB to a new machine Run migrations, then test Switch app servers to new database (On EC2, fire up an entire new web cluster)Feature flags: finely grained version of this

Lanyrd.com The lesson we keep on learning

Lanyrd.comGlobal settings are BAD Database settings => multi-db (1.2) Cache settings => multi cache backends (1.3) Haystack backends => multi backends (v2.0)

Lanyrd.comGlobal settings are BAD Database settings => multi-db (1.2) Cache settings => multi cache backends (1.3) Haystack backends => multi backends (v2.0) DEBUG TIME_ZONE Middleware?

Lanyrd.comSettings shouldbe modiﬁableat run-time

Thank you!http://lanyrd.com/sfwcq[ [Were hiring a developer/web operations person in London, come and talk to us!

http://lanyrd.com	251
url_unknown	3
http://www.slideshare.net	1
http://academy.optaros.com	1
http://paper.li	1
http://a0.twimg.com	1
http://zootool.com	1
http://127.0.0.1	1
http://localhost:3000	1

Tricks & challenges developing a large Django application

by Simon Willison

Accessibility

Categories

Upload Details

Usage Rights

9 Embeds 261

Statistics

Tricks & challenges developing a large Django application Presentation Transcript