Large problems, Mostly Solved

Large Problems
Mostly Solved

Eric Holscher
http://ericholscher.com
Djangocon 2010

Wednesday, September 8, 2010

What this talk isn’t

» In depth
» Overly technical
» Conclusive


What this talk is

» Providing current best-of-class
solutions to common problems
» Identifying problems that aren’t
currently solved well


Overview

» Patterns among good apps
» The problems that are mostly solved
» Unsolved Problems


A good application


Leverages previous
knowledge/APIs


Easy Setup


Good upgrade path


Good documentation


Well Tested


Solves a real problem


Mostly Solved Problems


Search


Haystack

» 9782 lines of docs
» 8882 lines of tests
» http://haystacksearch.org/
» Written by Daniel Lindsley


Line counts

ﬁnd . |egrep "rst|txt" |xargs wc -l |cut -c1-8 |
awk '{s+=$1} END {print s}'

ﬁnd . |egrep "py" |xargs wc -l |cut -c1-8 | awk
'{s+=$1} END {print s}'

#FORE!


Setup
» pip install django-haystack
» Add ‘haystack’ to Installed Apps
» Settings for siteconf and backend
» Create Search Site
» Create Search Indexes
» Add views
» Reindex

Upgrade Path

» Simple backend
» Whoosh
» Solr


Beautiful APIs

unfriendly_results = SearchQuerySet().exclude
(content='hello').ﬁlter(content='world')

unfriendly_results.order_by('-pub_date')[:5]


Beautiful APIs

class NoteIndex(SearchIndex):
text = CharField(document=True, use_template=True)
author = CharField(model_attr='user')
pub_date = DateTimeField(model_attr='pub_date')

def get_queryset(self):
return Note.objects.ﬁlter
(pub_date__lte=datetime.datetime.now())


Awesome Features

» Faceting
» Highlighting
» More Like This
» Easy Customization


Amazing full-text search


In an hour.


Documentation


Sphinx

» http://sphinx.pocoo.org/
» Written by Georg Brandl


Setup

» pip install Sphinx
» sphinx-quickstart
» Edit/Create Restructured Text ﬁles
» make html


Awesome features

» Uses ReStructuredText
» Easily generate PDFs
» Link between your own and other
docs
» Themes


Shameless Self Promotion

» Use Read The Docs
» Host Sphinx docs, or create them
there.
» http://rtfd.org
» http://media.readthedocs.org/
django.pdf


Make Beautiful
Documentation With Tools
You Know


Database Migrations


South

» http://south.aeracode.org/
» Written by Andrew Godwin


Setup

» pip install south

» south in Installed Apps

» manage.py schemamigration app --
initial
» manage.py schemamigratoin app
name --auto
» manage.py migrate


Beautiful API

def forwards(self, orm):
for adopter in orm.Adopter.objects.all():
try:
adopter.ﬁrst_name, adopter.last_name = adopter.name.split()
except ValueError:
adopter.ﬁrst_name, adopter.last_name = adopter.name, ""
adopter.save()


Boring


Safe, Painless
Data Migration


Delayed Execution


Run commands out of
process


Celery

» http://celeryproject.org/
» Written by Ask Solem


Setup

» pip install django-celery
» Add ‘djcelery’ to Installed Apps
» Syncdb
» Conﬁgure your settings backend
» Add @task decorator to function


When and Why

» Makes your site fast
» Use for tasks that users don’t
immediately care about
» Even if they do, return fast to a
waiting page


Examples

» Activity
» Sending Email
» Complex DB queries


Path

» Start with ALWAYS_EAGER
» Use a ghetto queue
» Rabbit MQ


Awesome features

» Run on multiple machines
» Cron replacement
» Good error reporting


Magically make your code
async


Remote Execution


Run commands on remote
servers (for deployment)


Fabric

» http://fabﬁle.org
» Written by Christian Vest Hansen &
Jeff Forcier


Setup

» pip install fabric
» Add commands to a fabﬁle.py
» fab <command>


Simple example

#Fabﬁle
from fabric.api import run

def host_type():
run('uname -s')

# Command
$ fab -H example.com host_type


Write python to deploy
your code


Deployment


Gunicorn

» http://gunicorn.org/
» Created by Benoit Chesneau


Setup

» pip install gunicorn
» add gunicorn to your Installed Apps
» manage.py run_gunicorn


Need to install Nginx in
front


Nginx setup

server {
listen 80;
server_name example.com;
access_log /var/log/nginx/example.log;

location / {
proxy_pass http://127.0.0.1:8080;
}
}


Awesome Features

» Async backend workers
» Simple conﬁguration
» Python


Stupidly Simple,
Production Ready
Deployment


Packaging


Pip & Virtualenv

» Written by Ian Bicking
» http://pip-installer.org
» http://virtualenv.openplans.org


Set up

» sudo easy_install pip # :D
» sudo pip install virtualenv
» virtualenv awesome_env
» source awesome_env/bin/activate
» pip install Django


Awesome Features

» Separate environments for each
application or deployment
» Install code from repos
» Doesn’t use eggs
» User-owned environment


Adds community standards
to something everyone was
doing their own way


No more PYTHONPATH
hacking


Competing Applications


APIs


APIs

» TastyPie
» Piston


Tagging


Tagging

» Django Taggit *
» Django Tagging


Continuous Integration


Continuous Integration

» Hudson *
» Buildbot


Smaller Problems


Debugging in Development


django-debug-toolbar

» http://github.com/robhudson/
django-debug-toolbar/
» Written by Rob Hudson


Proﬁles & Registration


django-[proﬁles|registration]

» http://bitbucket.org/ubernostrum/
django-proﬁles/
» http://bitbucket.org/ubernostrum/
django-registration/
» Written by James Bennett


Filtering


django-ﬁlter

» http://github.com/alex/django-ﬁlter
» Written by Alex Gaynor


Finding packages


Djangopackages.com


Unsolved Problems


Template Tags


Logging


Model Introspection


Class Based Views


Source code arrangement


Notiﬁcations/Email


Debugging in Production


OpenID/OAuth


VCS Abstraction


Positive Note


Cal’s list of problems ’08
» Multiple Databases
» Multiple multi-keyed cache requests
» Slow HTTP requests while we're doing other stuff
» ORM creates dumb SQL
» Verbose template syntax
» No query debugger
» Ugly SQL syntax
» Commit all ﬁelds at once
» No mascot
» No deployment system
» Migrating Models

Cal’s list of problems ’08
» Multiple Databases
» Multiple multi-keyed cache requests
» Slow HTTP requests while we're doing other stuff
» ORM creates dumb SQL
» Verbose template syntax
» No query debugger
» Ugly SQL syntax (Table aliases)
» Commit all ﬁelds at once
» No mascot
» No deployment system
» Extending models

Remember

» Leverage good applications in the
community
» Take previous knowledge and apply it
to new situations
» You can create an awesome
application used by other people


Questions?

» eric@ericholscher.com
» twitter.com/ericholscher


Large problems, Mostly Solved

More Related Content

Viewers also liked

Similar to Large problems, Mostly Solved

More from ericholscher

Large problems, Mostly Solved