0
django

Django in the Real World

James Bennett ● Jacob Kaplan-Moss
http://b-list.org/            http://jacobian.org/



...
So you’ve written a
    web app...



                      2
Now what?



            3
API Metering                                       Distributed Log storage, analysis
Backups & Snapshots                  ...
What’s on the plate
  Structuring for deployment
  Testing
  Production environments
  Deployment
  The “rest” of the web ...
Writing applications
you can deploy, and
deploy, and deploy...


                        6
The extended extended
        remix!



                        7
The fivefold path
  Do one thing, and do it well.
  Don’t be afraid of multiple apps.
  Write for flexibility.
  Build to ...
1



    9
Do one thing, and do it
        well.



                          10
Application ==
encapsulation



                 11
Keep a tight focus
  Ask yourself: “What does this application
  do?”
  Answer should be one or two short
  sentences




...
Good focus
  “Handle storage of users and
  authentication of their identities.”
  “Allow content to be tagged, del.icio.u...
Bad focus
  “Handle entries in a weblog, and users
  who post them, and their authentication,
  and tagging and categoriza...
Warning signs
  A lot of very good Django applications are
  very small: just a few files
  If your app is getting big eno...
Warning signs
  Even a lot of “simple” Django sites
  commonly have a dozen or more
  applications in INSTALLED_APPS
  If ...
Approach features
   skeptically



                    17
Should I add this feature?
  What does the application do?
  Does this feature have anything to do with
  that?
  No? Gues...
2



    19
Don’t be afraid of
 multiple apps



                     20
The monolith mindset
  The “application” is the whole site
  Re-use is often an afterthought
  Tend to develop plugins tha...
The Django mindset
  Application == some bit of functionality
  Site == several applications
  Tend to spin off new applic...
Django encourages this
  Instead of one “application”, a list:
  INSTALLED_APPS
  Applications live on the Python path, no...
Should this be its own application?

   Is it completely unrelated to the app’s
   focus?
   Is it orthogonal to whatever ...
Unrelated features
  Feature creep is tempting: “but wouldn’t
  it be cool if...”
  But it’s the road to Hell
  See also: ...
I’ve learned this the
      hard way



                        26
djangosnippets.org
  One application
  Includes bookmarking features
  Includes tagging features
  Includes rating feature...
Should be about four
   applications



                       28
Orthogonality
  Means you can change one thing without
  affecting others
  Almost always indicates the need for a
  separ...
Reuse
 Lots of cool features actually aren’t
 specific to one site
 See: bookmarking, tagging, rating...
 Why bring all th...
Advantages
 Don’t keep rewriting features
 Drop things into other sites easily




                                       ...
Need a contact form?



                       32
urlpatterns += (‘’,
    (r’^contact/’, include(‘contact_form.urls’)),
)




                                              ...
And you’re done



                  34
But what about...



                    35
Site-specific needs
  Site A wants a contact form that just
  collects a message.
  Site B’s marketing department wants a
...
3



    37
Write for flexibility



                        38
Common sense
 Sane defaults
 Easy overrides
 Don’t set anything in stone




                               39
Form processing
  Supply a form class
  But let people specify their own if they
  want




                              ...
Templates
  Specify a default template
  But let people specify their own if they
  want




                             ...
Form processing
  You want to redirect after successful
  submission
  Supply a default URL
  But let people specify their...
URL best practices
  Provide a URLConf in the application
  Use named URL patterns
  Use reverse lookups: reverse(),
  per...
Working with models
  Whenever possible, avoid hard-coding a
  model class
  Use get_model() and take an app label/
  mode...
Working with models
  Don’t hard-code fields or table names;
  introspect the model to get those
  Accept lookup arguments...
Learn to love managers
  Managers are easy to reuse.
  Managers are easy to subclass and
  customize.
  Managers let you e...
Advanced techniques
  Encourage subclassing and use of
  subclasses
  Provide a standard interface people can
  implement ...
The API your
 application exposes is
just as important as the
   design of the sites
     you’ll use it in.

             ...
In fact, it’s more
    important.



                     49
Good API design
  “Pass in a value for this argument to
  change the behavior”
  “Change the value of this setting”
  “Sub...
Bad API design
  “API? Let me see if we have one of
  those...” (AKA: “we don’t”)
  “It’s open source; fork it to do what ...
4



    52
Build to distribute



                      53
So you did the tutorial
  from mysite.polls.models
  import Poll
  mysite.polls.views.vote
  include(‘mysite.polls.urls’)
...
Project coupling kills
       re-use



                         55
Why (some) projects suck
  You have to replicate that directory
  structure every time you re-use
  Or you have to do gymn...
A good “project”
  A settings module
  A root URLConf module
  And that’s it.




                          57
Advantages
 No assumptions about where things live
 No tricky bits
 Reminds you that it’s just another Python
 module




...
It doesn’t even have to
    be one module



                          59
ljworld.com
  worldonline.settings.ljworld
  worldonline.urls.ljworld
  And a whole bunch of reused apps in
  sensible loc...
Configuration is
  contextual



                   61
What reusable apps look like
  Single module directly on Python path
  (registration, tagging, etc.)
  Related modules und...
And now it’s easy
  You can build a package with distutils
  or setuptools
  Put it on the Cheese Shop
  People can downlo...
Make it “packageable”
even if it’s only for your
            use


                             64
General best practices
  Be up-front about dependencies
  Write for Python 2.3 when possible
  Pick a release or pick trun...
I usually don’t do
default templates



                     66
Be obsessive about documentation

   It’s Python: give stuff docstrings
   If you do, Django will generate
   documentatio...
5



    68
Embracing and
  extending



                69
Don’t touch!
  Good applications are extensible without
  hacking them up.
  Take advantage of everything an
  application...
But this application
wasn’t meant to be
    extended!


                       71
Use the Python
(and the Django)



                   72
Want to extend a view?
  If possible, wrap the view with your own
  code.
  Doing this repetitively? Just write a
  decora...
Want to extend a model?
  You can relate other models to it.
  You can write subclasses of it.
  You can create proxy subc...
Model inheritance is
    powerful.
 With great power
   comes great
  responsibility.

                       75
Proxy models
  New in Django 1.1.
  Lets you add methods, managers, etc.
  (you’re extending the Python side, not the
  DB...
Extending a form
  Just subclass it.
  No really, that’s all :)




                             77
Other tricks
  Using signals lets you fire off customized
  behavior when particular events happen.
  Middleware offers fu...
But if you must make
changes to someone
    else’s code...


                       79
Keep changes to a minimum
  If possible, instead of adding a feature, add
  extensibility.
  Then keep as much changed cod...
Stay up-to-date
  You don’t want to get out of sync with the
  original version of the code.
  You might miss bugfixes.
  ...
Make sure your VCS is
   up to the job of
merging from upstream


                        82
Be a good citizen
  If you change someone else’s code, let
  them know.
  Maybe they’ll merge your changes in and
  you wo...
What if it’s my own
      code?



                      84
Same principles apply
  Maybe the original code wasn’t sufficient.
  Or maybe you just need a new application.
  Be just a...
Further reading




                  86
Testing



          87
“   Tests are the
Programmer’s stone,
transmuting fear into
     boredom.


                    ”
            — Kent Beck
...
Hardcore TDD



               89
“
   I don’t do test driven
     development. I do
stupidity driven testing… I
 wait until I do something
stupid, and then...
Whatever happens, don’t let
your test suite break thinking,
“I’ll go back and fix this later.”



                        ...
unittest
Unit testing



                              doctest
Functional/behavior
testing
                      django.te...
You need them all.



                     93
Unit tests
  “Whitebox” testing
  Verify the small functional units of your
  app
  Very fine-grained
  Familier to most p...
from django.test import TestCase
from django.http import HttpRequest
from django.middleware.common import CommonMiddleware...
django.test.TestCase
  Fixtures.
  Test client.
  Email capture.
  Database management.
  Slower than unittest.TestCase.

...
Doctests
  Easy to write & read.
  Produces self-documenting code.
  Great for cases that only use assertEquals.
  Somewhe...
class Template(object):
    quot;quot;quot;
    Deal with a URI template as a class::
    
        >>> t = Template(quot;h...
****************************************************
File quot;uri.pyquot;, line 150, in __main__.parse_expansion
Failed e...
Functional tests
  a.k.a “Behavior Driven Development.”
  “Blackbox,” holistic testing.
  All the hardcore TDD folks look ...
Functional testing tools
  django.test.Client
  webunit
  Twill
  ...




                           101
django.test.Client
  Test the whole request path without
  running a web server.
  Responses provide extra information
  a...
def testBasicAddPost(self):
    quot;quot;quot;
    A smoke test to ensure POST on add_view works.
    quot;quot;quot;
   ...
Web browser testing
  The ultimate in functional testing for
  web applications.
  Run test in a web browser.
  Can verify...
Browser testing tools
  Selenium
  Windmill




                        105
Exotic testing
  Static source analysis.
  Smoke testing (crawlers and spiders).
  Monkey testing.
  Load testing.
  ...

...
107
Further resources
  Talks here at PyCon!
  http://bit.ly/pycon2009-testing

  Don’t miss the testing tools panel
  (Sunday...
Deployment



             109
Deployment should...
  Be automated.
  Automatically manage dependencies.
  Be isolated.
  Be repeatable.
  Be identical i...
Dependency
                Isolation    Automation
management

apt/yum/...    virtualenv    Capistrano


easy_install   zc...
Let the live demo begin
          (gulp)




                          112
Building your stack



                      113
net.
                            LiveJournal Backend: Today
                                                           (Ro...
django

database

 media
 server




           115
Application servers
  Apache + mod_python
  Apache + mod_wsgi
  Apache/lighttpd + FastCGI
  SCGI, AJP, nginx/mod_wsgi, ......
Use mod_wsgi



               117
WSGIScriptAlias / /home/mysite/mysite.wsgi




                                             118
import os, sys

# Add to PYTHONPATH whatever you need
sys.path.append('/usr/local/django')

# Set DJANGO_SETTINGS_MODULE
o...
A brief digression
regarding the question
       of scale


                         120
Does this scale?

       django

      database

       media
       server




    Maybe!
                   121
122
Real-world example


Database A   Database B

175 req/s     75 req/s




                          123
Real-world example




             http://tweakers.net/reviews/657/6

                                                 124
django

    media
  web server



  database

database server




                  125
Why separate hardware?
  Resource contention
  Separate performance concerns
  0 → 1 is much harder than 1 → N




       ...
DATABASE_HOST = '10.0.0.100'




            FAIL               127
Connection middleware
  Proxy between web and database layers
  Most implement hot fallover and
  connection pooling
   So...
Connection middleware
  PostgreSQL: pgpool
  MySQL: MySQL Proxy
  Database-agnostic: sqlrelay
  Oracle: ?




            ...
django           media
  web server      media server




  database
database server




                                 ...
Media server traits
  Fast
  Lightweight
  Optimized for high concurrency
  Low memory overhead
  Good HTTP citizen




  ...
Media servers
  Apache?
  lighttpd
  nginx




                132
The absolute minimum


        django           media
       web server      media server




       database
     databas...
The absolute minimum


      django                 media




     database


                web server




             ...
proxy                media

           load balancer         media server




django        django          django
       ...
Why load balancers?



                      136
Load balancer traits
  Low memory overhead
  High concurrency
  Hot fallover
  Other nifty features...




               ...
Load balancers
  Apache + mod_proxy
  perlbal
  nginx




                       138
CREATE POOL mypool
    POOL mypool ADD 10.0.0.100
    POOL mypool ADD 10.0.0.101

CREATE SERVICE mysite
    SET listen = m...
you@yourserver:~$ telnet localhost 60000

pool mysite add 10.0.0.102
OK

nodes 10.0.0.101
10.0.0.101 lastresponse 12379874...
proxy             proxy              proxy     media           media

           load balancing cluster                med...
“Shared nothing”



                   142
BALANCE = None

def balance_sheet(request):
    global BALANCE
    if not BALANCE:
        bank = Bank.objects.get(...)
  ...
Global variables are
    right out



                       144
from django.cache import cache

def balance_sheet(request):
    balance = cache.get('bank_balance')
    if not balance:
  ...
def generate_report(request):
    report = get_the_report()
    open('/tmp/report.txt', 'w').write(report)
    return redi...
Filesystem?
What filesystem?



                   147
Further reading
  Cal Henderson, Building Scalable Web Sites
  (O’Reilly, 2006)
  John Allspaw, The Art of Capacity Planni...
Monitoring



             149
Goals
  When the site goes down, know it immediately.
  Automatically handle common sources of
  downtime.
  Ideally, hand...
Availability monitoring principles
   Check services for availability.
   More then just “ping yoursite.com.”
   Have some...
Availability monitoring tools
  Internal tools
    Nagios
    Monit
    Zenoss
    ...
  External monitoring tools



    ...
Usage monitoring
  Keep track of resource usage over time.
  Spot and identify trends.
  Aid in capacity planning and mana...
Usage monitoring tools
  RRDTool
  Munin
  Cacti
  Graphite




                         154
155
156
Logging and log analysis
  Record information about what’s
  happening right now.
  Analyze historical data for trends.
  ...
Logging tools
  print

  Python’s logging module
  syslogd




                            158
Log analysis
  grep | sort | uniq ‐c | sort ‐rn
  Load log data into relational databases,
  then slice & dice.
  OLAP/OLT...
Performance (and
when to care about it)



                         160
Ignore performance
  First, get the application written.
  Then, make it work.
  Then get it running on a server.
  Then, ...
Code isn’t “fast” or
“slow” until it’s been
      written.


                         162
Code isn’t “fast” or
“slow” until it works.



                         163
Code isn’t “fast” or
“slow” until it’s actually
  running on a server.


                             164
Optimizing code
  Most of the time, “bad” code is obvious as
  soon as you write it. So don’t write it.




              ...
Low-hanging fruit
  Look for code doing lots of DB queries --
  consider caching, or using select_related()
  Look for com...
The DB is the bottleneck
  And if it’s not the DB, it’s I/O.
  Everything else is typically negligible.




              ...
Find out what “slow” means
  Do testing in the browser.
  Do testing with command-line tools like
  wget.
  Compare the re...
Sometimes, perceived
“slowness” is actually
   on the front end.


                         169
Read Steve Souders’ book




                           170
YSlow
http://developer.yahoo.com/yslow/




                                    171
What to do on the server side
  First, try caching.
  Then try caching some more.




                                172
The secret weapon
  Caching turns less hardware into more.
  Caching puts off buying a new DB server.




                ...
But caching is a trade-off



                             174
Things to consider
  Cache for everybody? Or only for people
  who aren’t logged in?
  Cache everything? Or only a few com...
Not all users are the same
  Most visitors to most sites aren’t logged
  in.
  CACHE_MIDDLEWARE_ANONYMOUS
  _ONLY




    ...
Not all views are the same
  You probably already know where your
  nasty DB queries are.
  cache_page on those particular...
Site-wide caches
  You can use Django’s cache middleware to
  do this...
  Or you can use a proper caching proxy
  (e.g., ...
External caches
  Work fine with Django, because Django
  just uses HTTP’s caching headers.
  Take the entire load off Dja...
When caching doesn’t cut it




                              180
Throw money at your DB first




                               181
Web server improvements
  Simple steps first: turn off Keep-Alive,
  etc.
  Consider switching to a lighter-weight
  web s...
Database tuning
  Whole books can be written on DB
  performance tuning




                                     183
Using MySQL?




               184
Using PostgreSQL?
http://www.revsys.com/writings/postgresql-performance.html




                                         ...
Learn how to diagnose
  If things are slow, the cause may not be
  obvious.
  Even if you think it’s obvious, that may not...
Build a toolkit
  Python profilers: profile and cProfile
  Generic “spy on a process” tools: strace,
  SystemTap, and dtra...
Shameless plug




   http://revsys.com/



                        188
Fin.
Jacob Kaplan-Moss <jacob@jacobian.org>


   James Bennett <james@b-list.org>




                                    ...
Upcoming SlideShare
Loading in...5
×

Django in the Real World

20,208

Published on

There's plenty of material (documentation, blogs, books) out there that'll help
you write a site using Django... but then what? You've still got to test,
deploy, monitor, and tune the site; failure at deployment time means all your
beautiful code is for naught.

Published in: Technology
3 Comments
80 Likes
Statistics
Notes
No Downloads
Views
Total Views
20,208
On Slideshare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
774
Comments
3
Likes
80
Embeds 0
No embeds

No notes for slide

Transcript of "Django in the Real World"

  1. 1. django Django in the Real World James Bennett ● Jacob Kaplan-Moss http://b-list.org/ http://jacobian.org/ PyCon 2009 http://jacobian.org/speaking/2009/real-world-django/ 1
  2. 2. So you’ve written a web app... 2
  3. 3. Now what? 3
  4. 4. API Metering Distributed Log storage, analysis Backups & Snapshots Graphing Counters HTTP Caching Cloud/Cluster Management Tools Input/Output Filtering Instrumentation/Monitoring Memory Caching Failover Non-relational Key Stores Node addition/removal and hashing Rate Limiting Autoscaling for cloud resources Relational Storage CSRF/XSS Protection Queues Data Retention/Archival Rate Limiting Deployment Tools Real-time messaging (XMPP) Multiple Devs, Staging, Prod Search Data model upgrades Ranging Rolling deployments Geo Multiple versions (selective beta) Sharding Bucket Testing Smart Caching Rollbacks Dirty-table management CDN Management Distributed File Storage http://randomfoo.net/2009/01/28/infrastructure-for-modern-web-sites 4
  5. 5. What’s on the plate Structuring for deployment Testing Production environments Deployment The “rest” of the web stack Monitoring Performance & tuning 5
  6. 6. Writing applications you can deploy, and deploy, and deploy... 6
  7. 7. The extended extended remix! 7
  8. 8. The fivefold path Do one thing, and do it well. Don’t be afraid of multiple apps. Write for flexibility. Build to distribute. Extend carefully. 8
  9. 9. 1 9
  10. 10. Do one thing, and do it well. 10
  11. 11. Application == encapsulation 11
  12. 12. Keep a tight focus Ask yourself: “What does this application do?” Answer should be one or two short sentences 12
  13. 13. Good focus “Handle storage of users and authentication of their identities.” “Allow content to be tagged, del.icio.us style, with querying by tags.” “Handle entries in a weblog.” 13
  14. 14. Bad focus “Handle entries in a weblog, and users who post them, and their authentication, and tagging and categorization, and some flat pages for static content, and...” The coding equivalent of a run-on sentence 14
  15. 15. Warning signs A lot of very good Django applications are very small: just a few files If your app is getting big enough to need lots of things split up into lots of modules, it may be time to step back and re- evaluate 15
  16. 16. Warning signs Even a lot of “simple” Django sites commonly have a dozen or more applications in INSTALLED_APPS If you’ve got a complex/feature-packed site and a short application list, it may be time to think hard about how tightly- focused those apps are 16
  17. 17. Approach features skeptically 17
  18. 18. Should I add this feature? What does the application do? Does this feature have anything to do with that? No? Guess I shouldn’t add it, then. 18
  19. 19. 2 19
  20. 20. Don’t be afraid of multiple apps 20
  21. 21. The monolith mindset The “application” is the whole site Re-use is often an afterthought Tend to develop plugins that hook into the “main” application Or make heavy use of middleware-like concepts 21
  22. 22. The Django mindset Application == some bit of functionality Site == several applications Tend to spin off new applications liberally 22
  23. 23. Django encourages this Instead of one “application”, a list: INSTALLED_APPS Applications live on the Python path, not inside any specific “apps” or “plugins” directory Abstractions like the Site model make you think about this as you develop 23
  24. 24. Should this be its own application? Is it completely unrelated to the app’s focus? Is it orthogonal to whatever else I’m doing? Will I need similar functionality on other sites? Yes? Then I should break it out into a separate application. 24
  25. 25. Unrelated features Feature creep is tempting: “but wouldn’t it be cool if...” But it’s the road to Hell See also: Part 1 of this talk 25
  26. 26. I’ve learned this the hard way 26
  27. 27. djangosnippets.org One application Includes bookmarking features Includes tagging features Includes rating features 27
  28. 28. Should be about four applications 28
  29. 29. Orthogonality Means you can change one thing without affecting others Almost always indicates the need for a separate application Example: changing user profile workflow doesn’t affect user signup workflow. Make them two different applications. 29
  30. 30. Reuse Lots of cool features actually aren’t specific to one site See: bookmarking, tagging, rating... Why bring all this crap about code snippets along just to get the extra stuff? 30
  31. 31. Advantages Don’t keep rewriting features Drop things into other sites easily 31
  32. 32. Need a contact form? 32
  33. 33. urlpatterns += (‘’, (r’^contact/’, include(‘contact_form.urls’)), ) 33
  34. 34. And you’re done 34
  35. 35. But what about... 35
  36. 36. Site-specific needs Site A wants a contact form that just collects a message. Site B’s marketing department wants a bunch of info. Site C wants to use Akismet to filter automated spam. 36
  37. 37. 3 37
  38. 38. Write for flexibility 38
  39. 39. Common sense Sane defaults Easy overrides Don’t set anything in stone 39
  40. 40. Form processing Supply a form class But let people specify their own if they want 40
  41. 41. Templates Specify a default template But let people specify their own if they want 41
  42. 42. Form processing You want to redirect after successful submission Supply a default URL But let people specify their own if they want 42
  43. 43. URL best practices Provide a URLConf in the application Use named URL patterns Use reverse lookups: reverse(), permalink, {% url %} 43
  44. 44. Working with models Whenever possible, avoid hard-coding a model class Use get_model() and take an app label/ model name string instead Don’t rely on objects; use the default manager 44
  45. 45. Working with models Don’t hard-code fields or table names; introspect the model to get those Accept lookup arguments you can pass straight through to the database API 45
  46. 46. Learn to love managers Managers are easy to reuse. Managers are easy to subclass and customize. Managers let you encapsulate patterns of behavior behind a nice API. 46
  47. 47. Advanced techniques Encourage subclassing and use of subclasses Provide a standard interface people can implement in place of your default implementation Use a registry (like the admin) 47
  48. 48. The API your application exposes is just as important as the design of the sites you’ll use it in. 48
  49. 49. In fact, it’s more important. 49
  50. 50. Good API design “Pass in a value for this argument to change the behavior” “Change the value of this setting” “Subclass this and override these methods to customize” “Implement something with this interface, and register it with the handler” 50
  51. 51. Bad API design “API? Let me see if we have one of those...” (AKA: “we don’t”) “It’s open source; fork it to do what you want” (AKA: “we hate you”) def application(environ, start_response) (AKA: “we have a web service”) 51
  52. 52. 4 52
  53. 53. Build to distribute 53
  54. 54. So you did the tutorial from mysite.polls.models import Poll mysite.polls.views.vote include(‘mysite.polls.urls’) mysite.mysite.bork.bork.bork 54
  55. 55. Project coupling kills re-use 55
  56. 56. Why (some) projects suck You have to replicate that directory structure every time you re-use Or you have to do gymnastics with your Python path And you get back into the monolithic mindset 56
  57. 57. A good “project” A settings module A root URLConf module And that’s it. 57
  58. 58. Advantages No assumptions about where things live No tricky bits Reminds you that it’s just another Python module 58
  59. 59. It doesn’t even have to be one module 59
  60. 60. ljworld.com worldonline.settings.ljworld worldonline.urls.ljworld And a whole bunch of reused apps in sensible locations 60
  61. 61. Configuration is contextual 61
  62. 62. What reusable apps look like Single module directly on Python path (registration, tagging, etc.) Related modules under a package (ellington.events, ellington.podcasts, etc.) No project cruft whatsoever 62
  63. 63. And now it’s easy You can build a package with distutils or setuptools Put it on the Cheese Shop People can download and install 63
  64. 64. Make it “packageable” even if it’s only for your use 64
  65. 65. General best practices Be up-front about dependencies Write for Python 2.3 when possible Pick a release or pick trunk, and document that But if you pick trunk, update frequently 65
  66. 66. I usually don’t do default templates 66
  67. 67. Be obsessive about documentation It’s Python: give stuff docstrings If you do, Django will generate documentation for you And users will love you forever 67
  68. 68. 5 68
  69. 69. Embracing and extending 69
  70. 70. Don’t touch! Good applications are extensible without hacking them up. Take advantage of everything an application gives you. You may end up doing something that deserves a new application anyway. 70
  71. 71. But this application wasn’t meant to be extended! 71
  72. 72. Use the Python (and the Django) 72
  73. 73. Want to extend a view? If possible, wrap the view with your own code. Doing this repetitively? Just write a decorator. 73
  74. 74. Want to extend a model? You can relate other models to it. You can write subclasses of it. You can create proxy subclasses (in Django 1.1) 74
  75. 75. Model inheritance is powerful. With great power comes great responsibility. 75
  76. 76. Proxy models New in Django 1.1. Lets you add methods, managers, etc. (you’re extending the Python side, not the DB side). Keeps your extensions in your code. Avoids many problems with normal inheritance. 76
  77. 77. Extending a form Just subclass it. No really, that’s all :) 77
  78. 78. Other tricks Using signals lets you fire off customized behavior when particular events happen. Middleware offers full control over request/response handling. Context processors can make additional information available if a view doesn’t. 78
  79. 79. But if you must make changes to someone else’s code... 79
  80. 80. Keep changes to a minimum If possible, instead of adding a feature, add extensibility. Then keep as much changed code as you can out of the original app. 80
  81. 81. Stay up-to-date You don’t want to get out of sync with the original version of the code. You might miss bugfixes. You might even miss the feature you needed. 81
  82. 82. Make sure your VCS is up to the job of merging from upstream 82
  83. 83. Be a good citizen If you change someone else’s code, let them know. Maybe they’ll merge your changes in and you won’t have to fork anymore. 83
  84. 84. What if it’s my own code? 84
  85. 85. Same principles apply Maybe the original code wasn’t sufficient. Or maybe you just need a new application. Be just as careful about making changes. If nothing else, this will highlight ways in which your code wasn’t extensible to begin with. 85
  86. 86. Further reading 86
  87. 87. Testing 87
  88. 88. “ Tests are the Programmer’s stone, transmuting fear into boredom. ” — Kent Beck 88
  89. 89. Hardcore TDD 89
  90. 90. “ I don’t do test driven development. I do stupidity driven testing… I wait until I do something stupid, and then write tests ” to avoid doing it again. — Titus Brown 90
  91. 91. Whatever happens, don’t let your test suite break thinking, “I’ll go back and fix this later.” 91
  92. 92. unittest Unit testing doctest Functional/behavior testing django.test.Client, Twill Windmill, Selenium Browser testing 92
  93. 93. You need them all. 93
  94. 94. Unit tests “Whitebox” testing Verify the small functional units of your app Very fine-grained Familier to most programmers (JUnit, NUnit, etc.) Provided in Python by unittest 94
  95. 95. from django.test import TestCase from django.http import HttpRequest from django.middleware.common import CommonMiddleware from django.conf import settings class CommonMiddlewareTest(TestCase):     def setUp(self):         self.slash = settings.APPEND_SLASH; self.www = settings.PREPEND_WWW     def tearDown(self):         settings.APPEND_SLASH = self.slash; settings.PREPEND_WWW = self.www     def _get_request(self, path):         request = HttpRequest()         request.META = {'SERVER_NAME':'testserver', 'SERVER_PORT':80}         request.path = request.path_info = quot;/middleware/%squot; % path         return request              def test_append_slash_redirect(self):         settings.APPEND_SLASH = True         request = self._get_request('slash')         r = CommonMiddleware().process_request(request)         self.assertEquals(r.status_code, 301)         self.assertEquals(r['Location'], 'http://testserver/middleware/slash/')      95
  96. 96. django.test.TestCase Fixtures. Test client. Email capture. Database management. Slower than unittest.TestCase. 96
  97. 97. Doctests Easy to write & read. Produces self-documenting code. Great for cases that only use assertEquals. Somewhere between unit tests and functional tests. Difficult to debug. Don’t always provide useful test failures. 97
  98. 98. class Template(object):     quot;quot;quot;     Deal with a URI template as a class::              >>> t = Template(quot;http://example.com/{p}?{‐join|&|a,b,c}quot;)         >>> t.expand(p=quot;fooquot;, a=quot;1quot;)         'http://example.com/foo?a=1'         >>> t.expand(p=quot;barquot;, b=quot;2quot;, c=quot;3quot;)         'http://example.com/bar?c=3&b=2'     quot;quot;quot; def parse_expansion(expansion):     quot;quot;quot;     Parse an expansion ‐‐ the part inside {curlybraces} ‐‐ into its component     parts. Returns a tuple of (operator, argument, variabledict)::         >>> parse_expansion(quot;‐join|&|a,b,c=1quot;)         ('join', '&', {'a': None, 'c': '1', 'b': None})              >>> parse_expansion(quot;c=1quot;)         (None, None, {'c': '1'})     quot;quot;quot; def percent_encode(values):     quot;quot;quot;     Percent‐encode a dictionary of values, handling nested lists correctly::              >>> percent_encode({'company': 'AT&T'})         {'company': 'AT%26T'}         >>> percent_encode({'companies': ['Yahoo!', 'AT&T']})         {'companies': ['Yahoo%21', 'AT%26T']}     quot;quot;quot; 98
  99. 99. **************************************************** File quot;uri.pyquot;, line 150, in __main__.parse_expansion Failed example:     parse_expansion(quot;c=1quot;) Expected:     (None, None, {'c': '2'}) Got:     (None, None, {'c': '1'}) **************************************************** 99
  100. 100. Functional tests a.k.a “Behavior Driven Development.” “Blackbox,” holistic testing. All the hardcore TDD folks look down on functional tests. But it keeps your boss happy. Easy to find problems, harder to find the actual bug. 100
  101. 101. Functional testing tools django.test.Client webunit Twill ... 101
  102. 102. django.test.Client Test the whole request path without running a web server. Responses provide extra information about templates and their contexts. 102
  103. 103. def testBasicAddPost(self):     quot;quot;quot;     A smoke test to ensure POST on add_view works.     quot;quot;quot;     post_data = {         quot;namequot;: uquot;Another Sectionquot;,         # inline data         quot;article_set‐TOTAL_FORMSquot;: uquot;3quot;,         quot;article_set‐INITIAL_FORMSquot;: uquot;0quot;,     }     response = self.client.post('/admin/admin_views/section/add/', post_data)     self.failUnlessEqual(response.status_code, 302) def testCustomAdminSiteLoginTemplate(self):     self.client.logout()     request = self.client.get('/test_admin/admin2/')     self.assertTemplateUsed(request, 'custom_admin/login.html')     self.assertEquals(request.context['title'], 'Log in') 103
  104. 104. Web browser testing The ultimate in functional testing for web applications. Run test in a web browser. Can verify JavaScript, AJAX; even design. Test your site across supported browsers. 104
  105. 105. Browser testing tools Selenium Windmill 105
  106. 106. Exotic testing Static source analysis. Smoke testing (crawlers and spiders). Monkey testing. Load testing. ... 106
  107. 107. 107
  108. 108. Further resources Talks here at PyCon! http://bit.ly/pycon2009-testing Don’t miss the testing tools panel (Sunday, 10:30am) Django testing documentation http://bit.ly/django-testing Python Testing Tools Taxonomy http://bit.ly/py-testing-tools 108
  109. 109. Deployment 109
  110. 110. Deployment should... Be automated. Automatically manage dependencies. Be isolated. Be repeatable. Be identical in staging and in production. Work the same for everyone. 110
  111. 111. Dependency Isolation Automation management apt/yum/... virtualenv Capistrano easy_install zc.buildout Fabric pip Puppet zc.buildout 111
  112. 112. Let the live demo begin (gulp) 112
  113. 113. Building your stack 113
  114. 114. net. LiveJournal Backend: Today (Roughly.) BIG-IP Global Database perlbal (httpd/proxy) bigip1 mod_perl bigip2 proxy1 master_a master_b web1 proxy2 web2 proxy3 Memcached web3 slave1 slave2 ... slave5 proxy4 djabberd mc1 web4 djabberd proxy5 mc2 ... User DB Cluster 1 djabberd mc3 uc1a uc1b webN mc4 User DB Cluster 2 ... uc2a uc2b gearmand mcN Mogile Storage Nodes gearmand1 User DB Cluster 3 sto2 sto1 gearmandN uc3a uc3b Mogile Trackers sto8 ... tracker1 tracker3 User DB Cluster N ucNa ucNb “workers” MogileFS Database gearwrkN Job Queues (xN) mog_a mog_b theschwkN jqNa jqNb Brad Fitzpatrik, http://danga.com/words/2007_06_usenix/ slave1 slaveN http://danga.com/words/ 3 114
  115. 115. django database media server 115
  116. 116. Application servers Apache + mod_python Apache + mod_wsgi Apache/lighttpd + FastCGI SCGI, AJP, nginx/mod_wsgi, ... 116
  117. 117. Use mod_wsgi 117
  118. 118. WSGIScriptAlias / /home/mysite/mysite.wsgi 118
  119. 119. import os, sys # Add to PYTHONPATH whatever you need sys.path.append('/usr/local/django') # Set DJANGO_SETTINGS_MODULE os.environ['DJANGO_SETTINGS_MODULE'] = 'mysite.settings' # Create the application for mod_wsgi import django.core.handlers.wsgi application = django.core.handlers.wsgi.WSGIHandler() 119
  120. 120. A brief digression regarding the question of scale 120
  121. 121. Does this scale? django database media server Maybe! 121
  122. 122. 122
  123. 123. Real-world example Database A Database B 175 req/s 75 req/s 123
  124. 124. Real-world example http://tweakers.net/reviews/657/6 124
  125. 125. django media web server database database server 125
  126. 126. Why separate hardware? Resource contention Separate performance concerns 0 → 1 is much harder than 1 → N 126
  127. 127. DATABASE_HOST = '10.0.0.100' FAIL 127
  128. 128. Connection middleware Proxy between web and database layers Most implement hot fallover and connection pooling Some also provide replication, load balancing, parallel queries, connection limiting, &c DATABASE_HOST = '127.0.0.1' 128
  129. 129. Connection middleware PostgreSQL: pgpool MySQL: MySQL Proxy Database-agnostic: sqlrelay Oracle: ? 129
  130. 130. django media web server media server database database server 130
  131. 131. Media server traits Fast Lightweight Optimized for high concurrency Low memory overhead Good HTTP citizen 131
  132. 132. Media servers Apache? lighttpd nginx 132
  133. 133. The absolute minimum django media web server media server database database server 133
  134. 134. The absolute minimum django media database web server 134
  135. 135. proxy media load balancer media server django django django web server cluster database database server 135
  136. 136. Why load balancers? 136
  137. 137. Load balancer traits Low memory overhead High concurrency Hot fallover Other nifty features... 137
  138. 138. Load balancers Apache + mod_proxy perlbal nginx 138
  139. 139. CREATE POOL mypool     POOL mypool ADD 10.0.0.100     POOL mypool ADD 10.0.0.101 CREATE SERVICE mysite     SET listen = my.public.ip     SET role = reverse_proxy     SET pool = mypool     SET verify_backend = on     SET buffer_size = 120k ENABLE mysite 139
  140. 140. you@yourserver:~$ telnet localhost 60000 pool mysite add 10.0.0.102 OK nodes 10.0.0.101 10.0.0.101 lastresponse 1237987449 10.0.0.101 requests 97554563 10.0.0.101 connects 129242435 10.0.0.101 lastconnect 1237987449 10.0.0.101 attempts 129244743 10.0.0.101 responsecodes 200 358 10.0.0.101 responsecodes 302 14 10.0.0.101 responsecodes 207 99 10.0.0.101 responsecodes 301 11 10.0.0.101 responsecodes 404 18 10.0.0.101 lastattempt 1237987449 140
  141. 141. proxy proxy proxy media media load balancing cluster media server cluster django django django cache cache web server cluster cache cluster database database database database server cluster 141
  142. 142. “Shared nothing” 142
  143. 143. BALANCE = None def balance_sheet(request):     global BALANCE     if not BALANCE:         bank = Bank.objects.get(...)         BALANCE = bank.total_balance()     ... FAIL 143
  144. 144. Global variables are right out 144
  145. 145. from django.cache import cache def balance_sheet(request):     balance = cache.get('bank_balance')     if not balance:         bank = Bank.objects.get(...)         balance = bank.total_balance()         cache.set('bank_balance', balance) WIN     ... 145
  146. 146. def generate_report(request):     report = get_the_report()     open('/tmp/report.txt', 'w').write(report)     return redirect(view_report) def view_report(request):     report = open('/tmp/report.txt').read()     return HttpResponse(report) FAIL 146
  147. 147. Filesystem? What filesystem? 147
  148. 148. Further reading Cal Henderson, Building Scalable Web Sites (O’Reilly, 2006) John Allspaw, The Art of Capacity Planning (O’Reilly, 2008) http://kitchensoap.com/ http://highscalability.com/ 148
  149. 149. Monitoring 149
  150. 150. Goals When the site goes down, know it immediately. Automatically handle common sources of downtime. Ideally, handle downtime before it even happens. Monitor hardware usage to identify hotspots and plan for future growth. Aid in postmortem analysis. Generate pretty graphs. 150
  151. 151. Availability monitoring principles Check services for availability. More then just “ping yoursite.com.” Have some understanding of dependancies (if the db is down, I don’t need to also hear that the web servers are down.) Notify the “right” people using the “right” methods, and don’t stop until it’s fixed. Minimize false positives. Automatically take action against common sources of downtime. 151
  152. 152. Availability monitoring tools Internal tools Nagios Monit Zenoss ... External monitoring tools 152
  153. 153. Usage monitoring Keep track of resource usage over time. Spot and identify trends. Aid in capacity planning and management. Look good in reports to your boss. 153
  154. 154. Usage monitoring tools RRDTool Munin Cacti Graphite 154
  155. 155. 155
  156. 156. 156
  157. 157. Logging and log analysis Record information about what’s happening right now. Analyze historical data for trends. Provide postmortem information after failures. 157
  158. 158. Logging tools print Python’s logging module syslogd 158
  159. 159. Log analysis grep | sort | uniq ‐c | sort ‐rn Load log data into relational databases, then slice & dice. OLAP/OLTP engines. Splunk. Analog, AWStats, ... Google Analytics, Mint, ... 159
  160. 160. Performance (and when to care about it) 160
  161. 161. Ignore performance First, get the application written. Then, make it work. Then get it running on a server. Then, maybe, think about performance. 161
  162. 162. Code isn’t “fast” or “slow” until it’s been written. 162
  163. 163. Code isn’t “fast” or “slow” until it works. 163
  164. 164. Code isn’t “fast” or “slow” until it’s actually running on a server. 164
  165. 165. Optimizing code Most of the time, “bad” code is obvious as soon as you write it. So don’t write it. 165
  166. 166. Low-hanging fruit Look for code doing lots of DB queries -- consider caching, or using select_related() Look for complex DB queries, and see if they can be simplified. 166
  167. 167. The DB is the bottleneck And if it’s not the DB, it’s I/O. Everything else is typically negligible. 167
  168. 168. Find out what “slow” means Do testing in the browser. Do testing with command-line tools like wget. Compare the results, and you may be surprised. 168
  169. 169. Sometimes, perceived “slowness” is actually on the front end. 169
  170. 170. Read Steve Souders’ book 170
  171. 171. YSlow http://developer.yahoo.com/yslow/ 171
  172. 172. What to do on the server side First, try caching. Then try caching some more. 172
  173. 173. The secret weapon Caching turns less hardware into more. Caching puts off buying a new DB server. 173
  174. 174. But caching is a trade-off 174
  175. 175. Things to consider Cache for everybody? Or only for people who aren’t logged in? Cache everything? Or only a few complex views? Use Django’s cache layer? Or an external caching system? 175
  176. 176. Not all users are the same Most visitors to most sites aren’t logged in. CACHE_MIDDLEWARE_ANONYMOUS _ONLY 176
  177. 177. Not all views are the same You probably already know where your nasty DB queries are. cache_page on those particular views. 177
  178. 178. Site-wide caches You can use Django’s cache middleware to do this... Or you can use a proper caching proxy (e.g., Squid, Varnish). 178
  179. 179. External caches Work fine with Django, because Django just uses HTTP’s caching headers. Take the entire load off Django -- requests never even hit the application. 179
  180. 180. When caching doesn’t cut it 180
  181. 181. Throw money at your DB first 181
  182. 182. Web server improvements Simple steps first: turn off Keep-Alive, etc. Consider switching to a lighter-weight web server (e.g., nginx) or lighter-weight system (e.g., from mod_python to mod_wsgi). 182
  183. 183. Database tuning Whole books can be written on DB performance tuning 183
  184. 184. Using MySQL? 184
  185. 185. Using PostgreSQL? http://www.revsys.com/writings/postgresql-performance.html 185
  186. 186. Learn how to diagnose If things are slow, the cause may not be obvious. Even if you think it’s obvious, that may not be the cause. 186
  187. 187. Build a toolkit Python profilers: profile and cProfile Generic “spy on a process” tools: strace, SystemTap, and dtrace. Django debug toolbar                                (http://bit.ly/django-debug-toolbar) 187
  188. 188. Shameless plug http://revsys.com/ 188
  189. 189. Fin. Jacob Kaplan-Moss <jacob@jacobian.org> James Bennett <james@b-list.org> 189
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×