Your SlideShare is downloading. ×
0
Large Problems
                                 Mostly Solved



                                              Eric Holsch...
What this talk isn’t



                 » In depth
                 » Overly technical
                 » Conclusive




...
What this talk is



                 » Providing current best-of-class
                   solutions to common problems
  ...
Overview



                 » Patterns among good apps
                 » The problems that are mostly solved
           ...
A good application




Wednesday, September 8, 2010
Leverages previous
                                knowledge/APIs




Wednesday, September 8, 2010
Easy Setup




Wednesday, September 8, 2010
Good upgrade path




Wednesday, September 8, 2010
Good documentation




Wednesday, September 8, 2010
Well Tested




Wednesday, September 8, 2010
Solves a real problem




Wednesday, September 8, 2010
Mostly Solved Problems




Wednesday, September 8, 2010
Search




Wednesday, September 8, 2010
Haystack


                 » 9782 lines of docs
                 » 8882 lines of tests
                 » http://haystack...
Line counts


            find . |egrep "rst|txt" |xargs wc -l |cut -c1-8 |
            awk '{s+=$1} END {print s}'

      ...
Setup
                        » pip install django-haystack
                        » Add ‘haystack’ to Installed Apps
   ...
Upgrade Path



                 » Simple backend
                 » Whoosh
                 » Solr




Wednesday, Septemb...
Beautiful APIs



            unfriendly_results = SearchQuerySet().exclude
            (content='hello').filter(content='w...
Beautiful APIs


            class NoteIndex(SearchIndex):
               text = CharField(document=True, use_template=Tru...
Awesome Features


                 » Faceting
                 » Highlighting
                 » More Like This
         ...
Wednesday, September 8, 2010
Amazing full-text search




Wednesday, September 8, 2010
In an hour.




Wednesday, September 8, 2010
Documentation




Wednesday, September 8, 2010
Sphinx


                 » 16598 lines of docs
                 » 13272 lines of tests
                 » http://sphinx.p...
Setup


                 » pip install Sphinx
                 » sphinx-quickstart
                 » Edit/Create Restruct...
Awesome features


                 » Uses ReStructuredText
                 » Easily generate PDFs
                 » Lin...
Shameless Self Promotion

                 » Use Read The Docs
                 » Host Sphinx docs, or create them
       ...
Wednesday, September 8, 2010
Make Beautiful
 Documentation With Tools
       You Know

Wednesday, September 8, 2010
Database Migrations




Wednesday, September 8, 2010
South


                 » 5662 lines of docs
                 » 4328 lines of tests
                 » http://south.aerac...
Setup

                        » pip install south

                        » south in Installed Apps

                   ...
Beautiful API



            def forwards(self, orm):
              for adopter in orm.Adopter.objects.all():
            ...
Boring




Wednesday, September 8, 2010
Safe, Painless
                               Data Migration


Wednesday, September 8, 2010
Delayed Execution




Wednesday, September 8, 2010
Run commands out of
                            process




Wednesday, September 8, 2010
Celery


                 » 19610 lines of docs
                 » 13822 lines of tests
                 » http://celerypr...
Setup


                        » pip install django-celery
                        » Add ‘djcelery’ to Installed Apps
   ...
When and Why


                 » Makes your site fast
                 » Use for tasks that users don’t
                 ...
Examples



                 » Activity
                 » Sending Email
                 » Complex DB queries




Wednesd...
Wednesday, September 8, 2010
Path



                 » Start with ALWAYS_EAGER
                 » Use a ghetto queue
                 » Rabbit MQ




...
Awesome features



                 » Run on multiple machines
                 » Cron replacement
                 » Goo...
Magically make your code
             async


Wednesday, September 8, 2010
Remote Execution




Wednesday, September 8, 2010
Run commands on remote
            servers (for deployment)




Wednesday, September 8, 2010
Fabric


                 » 7272 lines of docs
                 » 4572 lines of tests
                 » http://fabfile.org...
Setup



                 » pip install fabric
                 » Add commands to a fabfile.py
                 » fab <comm...
Simple example

            #Fabfile
            from fabric.api import run

            def host_type():
              run...
Write python to deploy
                   your code


Wednesday, September 8, 2010
Deployment




Wednesday, September 8, 2010
Gunicorn


                 » 1898 lines of docs
                 » 2172 lines of tests
                 » http://gunicorn...
Setup



                        » pip install gunicorn
                        » add gunicorn to your Installed Apps
    ...
Need to install Nginx in
                    front



Wednesday, September 8, 2010
Nginx setup

            server {
                 listen 80;
                 server_name example.com;
                 a...
Awesome Features



                 » Async backend workers
                 » Simple configuration
                 » Pyt...
Stupidly Simple,
                               Production Ready
                                  Deployment


Wednesday,...
Packaging




Wednesday, September 8, 2010
Pip & Virtualenv



                 » Written by Ian Bicking
                 » http://pip-installer.org
                ...
Set up


                 » sudo easy_install pip # :D
                 » sudo pip install virtualenv
                 » v...
Awesome Features


                 » Separate environments for each
                   application or deployment
        ...
Adds community standards
to something everyone was
    doing their own way

Wednesday, September 8, 2010
No more PYTHONPATH
                  hacking



Wednesday, September 8, 2010
Competing Applications




Wednesday, September 8, 2010
APIs




Wednesday, September 8, 2010
APIs




                 » TastyPie
                 » Piston




Wednesday, September 8, 2010
Tagging




Wednesday, September 8, 2010
Tagging




                 » Django Taggit *
                 » Django Tagging




Wednesday, September 8, 2010
Continuous Integration




Wednesday, September 8, 2010
Continuous Integration




                 » Hudson *
                 » Buildbot




Wednesday, September 8, 2010
Smaller Problems




Wednesday, September 8, 2010
Debugging in Development




Wednesday, September 8, 2010
django-debug-toolbar



                 » http://github.com/robhudson/
                   django-debug-toolbar/
         ...
Wednesday, September 8, 2010
Profiles & Registration




Wednesday, September 8, 2010
django-[profiles|registration]


                 » http://bitbucket.org/ubernostrum/
                   django-profiles/
  ...
Filtering




Wednesday, September 8, 2010
django-filter




                 » http://github.com/alex/django-filter
                 » Written by Alex Gaynor




Wedn...
Wednesday, September 8, 2010
Finding packages




Wednesday, September 8, 2010
Djangopackages.com




Wednesday, September 8, 2010
Unsolved Problems




Wednesday, September 8, 2010
Template Tags




Wednesday, September 8, 2010
Logging




Wednesday, September 8, 2010
Model Introspection




Wednesday, September 8, 2010
Class Based Views




Wednesday, September 8, 2010
Source code arrangement




Wednesday, September 8, 2010
Notifications/Email




Wednesday, September 8, 2010
Debugging in Production




Wednesday, September 8, 2010
OpenID/OAuth




Wednesday, September 8, 2010
VCS Abstraction




Wednesday, September 8, 2010
Positive Note




Wednesday, September 8, 2010
Cal’s list of problems ’08
                 » Multiple Databases
                 » Multiple multi-keyed cache requests
  ...
Cal’s list of problems ’08
                 » Multiple Databases
                 » Multiple multi-keyed cache requests
  ...
Remember


                 » Leverage good applications in the
                   community
                 » Take previ...
Questions?




                 » eric@ericholscher.com
                 » twitter.com/ericholscher




Wednesday, Septemb...
Upcoming SlideShare
Loading in...5
×

Large problems, Mostly Solved

10,580

Published on

0 Comments
25 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
10,580
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
205
Comments
0
Likes
25
Embeds 0
No embeds

No notes for slide

Transcript of "Large problems, Mostly Solved"

  1. 1. Large Problems Mostly Solved Eric Holscher http://ericholscher.com Djangocon 2010 Wednesday, September 8, 2010
  2. 2. What this talk isn’t » In depth » Overly technical » Conclusive Wednesday, September 8, 2010
  3. 3. What this talk is » Providing current best-of-class solutions to common problems » Identifying problems that aren’t currently solved well Wednesday, September 8, 2010
  4. 4. Overview » Patterns among good apps » The problems that are mostly solved » Unsolved Problems Wednesday, September 8, 2010
  5. 5. A good application Wednesday, September 8, 2010
  6. 6. Leverages previous knowledge/APIs Wednesday, September 8, 2010
  7. 7. Easy Setup Wednesday, September 8, 2010
  8. 8. Good upgrade path Wednesday, September 8, 2010
  9. 9. Good documentation Wednesday, September 8, 2010
  10. 10. Well Tested Wednesday, September 8, 2010
  11. 11. Solves a real problem Wednesday, September 8, 2010
  12. 12. Mostly Solved Problems Wednesday, September 8, 2010
  13. 13. Search Wednesday, September 8, 2010
  14. 14. Haystack » 9782 lines of docs » 8882 lines of tests » http://haystacksearch.org/ » Written by Daniel Lindsley Wednesday, September 8, 2010
  15. 15. Line counts find . |egrep "rst|txt" |xargs wc -l |cut -c1-8 | awk '{s+=$1} END {print s}' find . |egrep "py" |xargs wc -l |cut -c1-8 | awk '{s+=$1} END {print s}' #FORE! Wednesday, September 8, 2010
  16. 16. Setup » pip install django-haystack » Add ‘haystack’ to Installed Apps » Settings for siteconf and backend » Create Search Site » Create Search Indexes » Add views » Reindex Wednesday, September 8, 2010
  17. 17. Upgrade Path » Simple backend » Whoosh » Solr Wednesday, September 8, 2010
  18. 18. Beautiful APIs unfriendly_results = SearchQuerySet().exclude (content='hello').filter(content='world') unfriendly_results.order_by('-pub_date')[:5] Wednesday, September 8, 2010
  19. 19. Beautiful APIs class NoteIndex(SearchIndex): text = CharField(document=True, use_template=True) author = CharField(model_attr='user') pub_date = DateTimeField(model_attr='pub_date') def get_queryset(self): return Note.objects.filter (pub_date__lte=datetime.datetime.now()) Wednesday, September 8, 2010
  20. 20. Awesome Features » Faceting » Highlighting » More Like This » Easy Customization Wednesday, September 8, 2010
  21. 21. Wednesday, September 8, 2010
  22. 22. Amazing full-text search Wednesday, September 8, 2010
  23. 23. In an hour. Wednesday, September 8, 2010
  24. 24. Documentation Wednesday, September 8, 2010
  25. 25. Sphinx » 16598 lines of docs » 13272 lines of tests » http://sphinx.pocoo.org/ » Written by Georg Brandl Wednesday, September 8, 2010
  26. 26. Setup » pip install Sphinx » sphinx-quickstart » Edit/Create Restructured Text files » make html Wednesday, September 8, 2010
  27. 27. Awesome features » Uses ReStructuredText » Easily generate PDFs » Link between your own and other docs » Themes Wednesday, September 8, 2010
  28. 28. Shameless Self Promotion » Use Read The Docs » Host Sphinx docs, or create them there. » http://rtfd.org » http://media.readthedocs.org/ django.pdf Wednesday, September 8, 2010
  29. 29. Wednesday, September 8, 2010
  30. 30. Make Beautiful Documentation With Tools You Know Wednesday, September 8, 2010
  31. 31. Database Migrations Wednesday, September 8, 2010
  32. 32. South » 5662 lines of docs » 4328 lines of tests » http://south.aeracode.org/ » Written by Andrew Godwin Wednesday, September 8, 2010
  33. 33. Setup » pip install south » south in Installed Apps » manage.py schemamigration app -- initial » manage.py schemamigratoin app name --auto » manage.py migrate Wednesday, September 8, 2010
  34. 34. Beautiful API def forwards(self, orm): for adopter in orm.Adopter.objects.all(): try: adopter.first_name, adopter.last_name = adopter.name.split() except ValueError: adopter.first_name, adopter.last_name = adopter.name, "" adopter.save() Wednesday, September 8, 2010
  35. 35. Boring Wednesday, September 8, 2010
  36. 36. Safe, Painless Data Migration Wednesday, September 8, 2010
  37. 37. Delayed Execution Wednesday, September 8, 2010
  38. 38. Run commands out of process Wednesday, September 8, 2010
  39. 39. Celery » 19610 lines of docs » 13822 lines of tests » http://celeryproject.org/ » Written by Ask Solem Wednesday, September 8, 2010
  40. 40. Setup » pip install django-celery » Add ‘djcelery’ to Installed Apps » Syncdb » Configure your settings backend » Add @task decorator to function Wednesday, September 8, 2010
  41. 41. When and Why » Makes your site fast » Use for tasks that users don’t immediately care about » Even if they do, return fast to a waiting page Wednesday, September 8, 2010
  42. 42. Examples » Activity » Sending Email » Complex DB queries Wednesday, September 8, 2010
  43. 43. Wednesday, September 8, 2010
  44. 44. Path » Start with ALWAYS_EAGER » Use a ghetto queue » Rabbit MQ Wednesday, September 8, 2010
  45. 45. Awesome features » Run on multiple machines » Cron replacement » Good error reporting Wednesday, September 8, 2010
  46. 46. Magically make your code async Wednesday, September 8, 2010
  47. 47. Remote Execution Wednesday, September 8, 2010
  48. 48. Run commands on remote servers (for deployment) Wednesday, September 8, 2010
  49. 49. Fabric » 7272 lines of docs » 4572 lines of tests » http://fabfile.org » Written by Christian Vest Hansen & Jeff Forcier Wednesday, September 8, 2010
  50. 50. Setup » pip install fabric » Add commands to a fabfile.py » fab <command> Wednesday, September 8, 2010
  51. 51. Simple example #Fabfile from fabric.api import run def host_type(): run('uname -s') # Command $ fab -H example.com host_type Wednesday, September 8, 2010
  52. 52. Write python to deploy your code Wednesday, September 8, 2010
  53. 53. Deployment Wednesday, September 8, 2010
  54. 54. Gunicorn » 1898 lines of docs » 2172 lines of tests » http://gunicorn.org/ » Created by Benoit Chesneau Wednesday, September 8, 2010
  55. 55. Setup » pip install gunicorn » add gunicorn to your Installed Apps » manage.py run_gunicorn Wednesday, September 8, 2010
  56. 56. Need to install Nginx in front Wednesday, September 8, 2010
  57. 57. Nginx setup server { listen 80; server_name example.com; access_log /var/log/nginx/example.log; location / { proxy_pass http://127.0.0.1:8080; } } Wednesday, September 8, 2010
  58. 58. Awesome Features » Async backend workers » Simple configuration » Python Wednesday, September 8, 2010
  59. 59. Stupidly Simple, Production Ready Deployment Wednesday, September 8, 2010
  60. 60. Packaging Wednesday, September 8, 2010
  61. 61. Pip & Virtualenv » Written by Ian Bicking » http://pip-installer.org » http://virtualenv.openplans.org Wednesday, September 8, 2010
  62. 62. Set up » sudo easy_install pip # :D » sudo pip install virtualenv » virtualenv awesome_env » source awesome_env/bin/activate » pip install Django Wednesday, September 8, 2010
  63. 63. Awesome Features » Separate environments for each application or deployment » Install code from repos » Doesn’t use eggs » User-owned environment Wednesday, September 8, 2010
  64. 64. Adds community standards to something everyone was doing their own way Wednesday, September 8, 2010
  65. 65. No more PYTHONPATH hacking Wednesday, September 8, 2010
  66. 66. Competing Applications Wednesday, September 8, 2010
  67. 67. APIs Wednesday, September 8, 2010
  68. 68. APIs » TastyPie » Piston Wednesday, September 8, 2010
  69. 69. Tagging Wednesday, September 8, 2010
  70. 70. Tagging » Django Taggit * » Django Tagging Wednesday, September 8, 2010
  71. 71. Continuous Integration Wednesday, September 8, 2010
  72. 72. Continuous Integration » Hudson * » Buildbot Wednesday, September 8, 2010
  73. 73. Smaller Problems Wednesday, September 8, 2010
  74. 74. Debugging in Development Wednesday, September 8, 2010
  75. 75. django-debug-toolbar » http://github.com/robhudson/ django-debug-toolbar/ » Written by Rob Hudson Wednesday, September 8, 2010
  76. 76. Wednesday, September 8, 2010
  77. 77. Profiles & Registration Wednesday, September 8, 2010
  78. 78. django-[profiles|registration] » http://bitbucket.org/ubernostrum/ django-profiles/ » http://bitbucket.org/ubernostrum/ django-registration/ » Written by James Bennett Wednesday, September 8, 2010
  79. 79. Filtering Wednesday, September 8, 2010
  80. 80. django-filter » http://github.com/alex/django-filter » Written by Alex Gaynor Wednesday, September 8, 2010
  81. 81. Wednesday, September 8, 2010
  82. 82. Finding packages Wednesday, September 8, 2010
  83. 83. Djangopackages.com Wednesday, September 8, 2010
  84. 84. Unsolved Problems Wednesday, September 8, 2010
  85. 85. Template Tags Wednesday, September 8, 2010
  86. 86. Logging Wednesday, September 8, 2010
  87. 87. Model Introspection Wednesday, September 8, 2010
  88. 88. Class Based Views Wednesday, September 8, 2010
  89. 89. Source code arrangement Wednesday, September 8, 2010
  90. 90. Notifications/Email Wednesday, September 8, 2010
  91. 91. Debugging in Production Wednesday, September 8, 2010
  92. 92. OpenID/OAuth Wednesday, September 8, 2010
  93. 93. VCS Abstraction Wednesday, September 8, 2010
  94. 94. Positive Note Wednesday, September 8, 2010
  95. 95. Cal’s list of problems ’08 » Multiple Databases » Multiple multi-keyed cache requests » Slow HTTP requests while we're doing other stuff » ORM creates dumb SQL » Verbose template syntax » No query debugger » Ugly SQL syntax » Commit all fields at once » No mascot » No deployment system » Migrating Models Wednesday, September 8, 2010
  96. 96. Cal’s list of problems ’08 » Multiple Databases » Multiple multi-keyed cache requests » Slow HTTP requests while we're doing other stuff » ORM creates dumb SQL » Verbose template syntax » No query debugger » Ugly SQL syntax (Table aliases) » Commit all fields at once » No mascot » No deployment system » Extending models Wednesday, September 8, 2010
  97. 97. Remember » Leverage good applications in the community » Take previous knowledge and apply it to new situations » You can create an awesome application used by other people Wednesday, September 8, 2010
  98. 98. Questions? » eric@ericholscher.com » twitter.com/ericholscher Wednesday, September 8, 2010
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×