Large problems, Mostly Solved

10,928 views
10,804 views

Published on

Large problems, Mostly Solved

  1. 1. Large Problems Mostly Solved Eric Holscher http://ericholscher.com Djangocon 2010 Wednesday, September 8, 2010
  2. 2. What this talk isn’t » In depth » Overly technical » Conclusive Wednesday, September 8, 2010
  3. 3. What this talk is » Providing current best-of-class solutions to common problems » Identifying problems that aren’t currently solved well Wednesday, September 8, 2010
  4. 4. Overview » Patterns among good apps » The problems that are mostly solved » Unsolved Problems Wednesday, September 8, 2010
  5. 5. A good application Wednesday, September 8, 2010
  6. 6. Leverages previous knowledge/APIs Wednesday, September 8, 2010
  7. 7. Easy Setup Wednesday, September 8, 2010
  8. 8. Good upgrade path Wednesday, September 8, 2010
  9. 9. Good documentation Wednesday, September 8, 2010
  10. 10. Well Tested Wednesday, September 8, 2010
  11. 11. Solves a real problem Wednesday, September 8, 2010
  12. 12. Mostly Solved Problems Wednesday, September 8, 2010
  13. 13. Search Wednesday, September 8, 2010
  14. 14. Haystack » 9782 lines of docs » 8882 lines of tests » http://haystacksearch.org/ » Written by Daniel Lindsley Wednesday, September 8, 2010
  15. 15. Line counts find . |egrep "rst|txt" |xargs wc -l |cut -c1-8 | awk '{s+=$1} END {print s}' find . |egrep "py" |xargs wc -l |cut -c1-8 | awk '{s+=$1} END {print s}' #FORE! Wednesday, September 8, 2010
  16. 16. Setup » pip install django-haystack » Add ‘haystack’ to Installed Apps » Settings for siteconf and backend » Create Search Site » Create Search Indexes » Add views » Reindex Wednesday, September 8, 2010
  17. 17. Upgrade Path » Simple backend » Whoosh » Solr Wednesday, September 8, 2010
  18. 18. Beautiful APIs unfriendly_results = SearchQuerySet().exclude (content='hello').filter(content='world') unfriendly_results.order_by('-pub_date')[:5] Wednesday, September 8, 2010
  19. 19. Beautiful APIs class NoteIndex(SearchIndex): text = CharField(document=True, use_template=True) author = CharField(model_attr='user') pub_date = DateTimeField(model_attr='pub_date') def get_queryset(self): return Note.objects.filter (pub_date__lte=datetime.datetime.now()) Wednesday, September 8, 2010
  20. 20. Awesome Features » Faceting » Highlighting » More Like This » Easy Customization Wednesday, September 8, 2010
  21. 21. Wednesday, September 8, 2010
  22. 22. Amazing full-text search Wednesday, September 8, 2010
  23. 23. In an hour. Wednesday, September 8, 2010
  24. 24. Documentation Wednesday, September 8, 2010
  25. 25. Sphinx » 16598 lines of docs » 13272 lines of tests » http://sphinx.pocoo.org/ » Written by Georg Brandl Wednesday, September 8, 2010
  26. 26. Setup » pip install Sphinx » sphinx-quickstart » Edit/Create Restructured Text files » make html Wednesday, September 8, 2010
  27. 27. Awesome features » Uses ReStructuredText » Easily generate PDFs » Link between your own and other docs » Themes Wednesday, September 8, 2010
  28. 28. Shameless Self Promotion » Use Read The Docs » Host Sphinx docs, or create them there. » http://rtfd.org » http://media.readthedocs.org/ django.pdf Wednesday, September 8, 2010
  29. 29. Wednesday, September 8, 2010
  30. 30. Make Beautiful Documentation With Tools You Know Wednesday, September 8, 2010
  31. 31. Database Migrations Wednesday, September 8, 2010
  32. 32. South » 5662 lines of docs » 4328 lines of tests » http://south.aeracode.org/ » Written by Andrew Godwin Wednesday, September 8, 2010
  33. 33. Setup » pip install south » south in Installed Apps » manage.py schemamigration app -- initial » manage.py schemamigratoin app name --auto » manage.py migrate Wednesday, September 8, 2010
  34. 34. Beautiful API def forwards(self, orm): for adopter in orm.Adopter.objects.all(): try: adopter.first_name, adopter.last_name = adopter.name.split() except ValueError: adopter.first_name, adopter.last_name = adopter.name, "" adopter.save() Wednesday, September 8, 2010
  35. 35. Boring Wednesday, September 8, 2010
  36. 36. Safe, Painless Data Migration Wednesday, September 8, 2010
  37. 37. Delayed Execution Wednesday, September 8, 2010
  38. 38. Run commands out of process Wednesday, September 8, 2010
  39. 39. Celery » 19610 lines of docs » 13822 lines of tests » http://celeryproject.org/ » Written by Ask Solem Wednesday, September 8, 2010
  40. 40. Setup » pip install django-celery » Add ‘djcelery’ to Installed Apps » Syncdb » Configure your settings backend » Add @task decorator to function Wednesday, September 8, 2010
  41. 41. When and Why » Makes your site fast » Use for tasks that users don’t immediately care about » Even if they do, return fast to a waiting page Wednesday, September 8, 2010
  42. 42. Examples » Activity » Sending Email » Complex DB queries Wednesday, September 8, 2010
  43. 43. Wednesday, September 8, 2010
  44. 44. Path » Start with ALWAYS_EAGER » Use a ghetto queue » Rabbit MQ Wednesday, September 8, 2010
  45. 45. Awesome features » Run on multiple machines » Cron replacement » Good error reporting Wednesday, September 8, 2010
  46. 46. Magically make your code async Wednesday, September 8, 2010
  47. 47. Remote Execution Wednesday, September 8, 2010
  48. 48. Run commands on remote servers (for deployment) Wednesday, September 8, 2010
  49. 49. Fabric » 7272 lines of docs » 4572 lines of tests » http://fabfile.org » Written by Christian Vest Hansen & Jeff Forcier Wednesday, September 8, 2010
  50. 50. Setup » pip install fabric » Add commands to a fabfile.py » fab <command> Wednesday, September 8, 2010
  51. 51. Simple example #Fabfile from fabric.api import run def host_type(): run('uname -s') # Command $ fab -H example.com host_type Wednesday, September 8, 2010
  52. 52. Write python to deploy your code Wednesday, September 8, 2010
  53. 53. Deployment Wednesday, September 8, 2010
  54. 54. Gunicorn » 1898 lines of docs » 2172 lines of tests » http://gunicorn.org/ » Created by Benoit Chesneau Wednesday, September 8, 2010
  55. 55. Setup » pip install gunicorn » add gunicorn to your Installed Apps » manage.py run_gunicorn Wednesday, September 8, 2010
  56. 56. Need to install Nginx in front Wednesday, September 8, 2010
  57. 57. Nginx setup server { listen 80; server_name example.com; access_log /var/log/nginx/example.log; location / { proxy_pass http://127.0.0.1:8080; } } Wednesday, September 8, 2010
  58. 58. Awesome Features » Async backend workers » Simple configuration » Python Wednesday, September 8, 2010
  59. 59. Stupidly Simple, Production Ready Deployment Wednesday, September 8, 2010
  60. 60. Packaging Wednesday, September 8, 2010
  61. 61. Pip & Virtualenv » Written by Ian Bicking » http://pip-installer.org » http://virtualenv.openplans.org Wednesday, September 8, 2010
  62. 62. Set up » sudo easy_install pip # :D » sudo pip install virtualenv » virtualenv awesome_env » source awesome_env/bin/activate » pip install Django Wednesday, September 8, 2010
  63. 63. Awesome Features » Separate environments for each application or deployment » Install code from repos » Doesn’t use eggs » User-owned environment Wednesday, September 8, 2010
  64. 64. Adds community standards to something everyone was doing their own way Wednesday, September 8, 2010
  65. 65. No more PYTHONPATH hacking Wednesday, September 8, 2010
  66. 66. Competing Applications Wednesday, September 8, 2010
  67. 67. APIs Wednesday, September 8, 2010
  68. 68. APIs » TastyPie » Piston Wednesday, September 8, 2010
  69. 69. Tagging Wednesday, September 8, 2010
  70. 70. Tagging » Django Taggit * » Django Tagging Wednesday, September 8, 2010
  71. 71. Continuous Integration Wednesday, September 8, 2010
  72. 72. Continuous Integration » Hudson * » Buildbot Wednesday, September 8, 2010
  73. 73. Smaller Problems Wednesday, September 8, 2010
  74. 74. Debugging in Development Wednesday, September 8, 2010
  75. 75. django-debug-toolbar » http://github.com/robhudson/ django-debug-toolbar/ » Written by Rob Hudson Wednesday, September 8, 2010
  76. 76. Wednesday, September 8, 2010
  77. 77. Profiles & Registration Wednesday, September 8, 2010
  78. 78. django-[profiles|registration] » http://bitbucket.org/ubernostrum/ django-profiles/ » http://bitbucket.org/ubernostrum/ django-registration/ » Written by James Bennett Wednesday, September 8, 2010
  79. 79. Filtering Wednesday, September 8, 2010
  80. 80. django-filter » http://github.com/alex/django-filter » Written by Alex Gaynor Wednesday, September 8, 2010
  81. 81. Wednesday, September 8, 2010
  82. 82. Finding packages Wednesday, September 8, 2010
  83. 83. Djangopackages.com Wednesday, September 8, 2010
  84. 84. Unsolved Problems Wednesday, September 8, 2010
  85. 85. Template Tags Wednesday, September 8, 2010
  86. 86. Logging Wednesday, September 8, 2010
  87. 87. Model Introspection Wednesday, September 8, 2010
  88. 88. Class Based Views Wednesday, September 8, 2010
  89. 89. Source code arrangement Wednesday, September 8, 2010
  90. 90. Notifications/Email Wednesday, September 8, 2010
  91. 91. Debugging in Production Wednesday, September 8, 2010
  92. 92. OpenID/OAuth Wednesday, September 8, 2010
  93. 93. VCS Abstraction Wednesday, September 8, 2010
  94. 94. Positive Note Wednesday, September 8, 2010
  95. 95. Cal’s list of problems ’08 » Multiple Databases » Multiple multi-keyed cache requests » Slow HTTP requests while we're doing other stuff » ORM creates dumb SQL » Verbose template syntax » No query debugger » Ugly SQL syntax » Commit all fields at once » No mascot » No deployment system » Migrating Models Wednesday, September 8, 2010
  96. 96. Cal’s list of problems ’08 » Multiple Databases » Multiple multi-keyed cache requests » Slow HTTP requests while we're doing other stuff » ORM creates dumb SQL » Verbose template syntax » No query debugger » Ugly SQL syntax (Table aliases) » Commit all fields at once » No mascot » No deployment system » Extending models Wednesday, September 8, 2010
  97. 97. Remember » Leverage good applications in the community » Take previous knowledge and apply it to new situations » You can create an awesome application used by other people Wednesday, September 8, 2010
  98. 98. Questions? » eric@ericholscher.com » twitter.com/ericholscher Wednesday, September 8, 2010

×