Large problems, Mostly Solved

Large Problems
Mostly Solved

Eric Holscher
http://ericholscher.com
Djangocon 2010

Wednesday, September 8, 2010

What this talk isn’t

» In depth
» Overly technical
» Conclusive


What this talk is

» Providing current best-of-class
solutions to common problems
» Identifying problems that aren’t
currently solved well


Overview

» Patterns among good apps
» The problems that are mostly solved
» Unsolved Problems


A good application


Leverages previous
knowledge/APIs


Easy Setup


Good upgrade path


Good documentation


Well Tested


Solves a real problem


Mostly Solved Problems


Search


Haystack

» 9782 lines of docs
» 8882 lines of tests
» http://haystacksearch.org/
» Written by Daniel Lindsley


Line counts

ﬁnd . |egrep "rst|txt" |xargs wc -l |cut -c1-8 |
awk '{s+=$1} END {print s}'

ﬁnd . |egrep "py" |xargs wc -l |cut -c1-8 | awk
'{s+=$1} END {print s}'

#FORE!


Setup
» pip install django-haystack
» Add ‘haystack’ to Installed Apps
» Settings for siteconf and backend
» Create Search Site
» Create Search Indexes
» Add views
» Reindex

Upgrade Path

» Simple backend
» Whoosh
» Solr


Beautiful APIs

unfriendly_results = SearchQuerySet().exclude
(content='hello').ﬁlter(content='world')

unfriendly_results.order_by('-pub_date')[:5]


Beautiful APIs

class NoteIndex(SearchIndex):
text = CharField(document=True, use_template=True)
author = CharField(model_attr='user')
pub_date = DateTimeField(model_attr='pub_date')

def get_queryset(self):
return Note.objects.ﬁlter
(pub_date__lte=datetime.datetime.now())


Awesome Features

» Faceting
» Highlighting
» More Like This
» Easy Customization


Amazing full-text search


In an hour.


Documentation


Sphinx

» http://sphinx.pocoo.org/
» Written by Georg Brandl


Setup

» pip install Sphinx
» sphinx-quickstart
» Edit/Create Restructured Text ﬁles
» make html


Awesome features

» Uses ReStructuredText
» Easily generate PDFs
» Link between your own and other
docs
» Themes


Shameless Self Promotion

» Use Read The Docs
» Host Sphinx docs, or create them
there.
» http://rtfd.org
» http://media.readthedocs.org/
django.pdf


Make Beautiful
Documentation With Tools
You Know


Database Migrations


South

» http://south.aeracode.org/
» Written by Andrew Godwin


Setup

» pip install south

» south in Installed Apps

» manage.py schemamigration app --
initial
» manage.py schemamigratoin app
name --auto
» manage.py migrate


Beautiful API

def forwards(self, orm):
for adopter in orm.Adopter.objects.all():
try:
adopter.ﬁrst_name, adopter.last_name = adopter.name.split()
except ValueError:
adopter.ﬁrst_name, adopter.last_name = adopter.name, ""
adopter.save()


Boring


Safe, Painless
Data Migration


Delayed Execution


Run commands out of
process


Celery

» http://celeryproject.org/
» Written by Ask Solem


Setup

» pip install django-celery
» Add ‘djcelery’ to Installed Apps
» Syncdb
» Conﬁgure your settings backend
» Add @task decorator to function


When and Why

» Makes your site fast
» Use for tasks that users don’t
immediately care about
» Even if they do, return fast to a
waiting page


Examples

» Activity
» Sending Email
» Complex DB queries


Path

» Start with ALWAYS_EAGER
» Use a ghetto queue
» Rabbit MQ


Awesome features

» Run on multiple machines
» Cron replacement
» Good error reporting


Magically make your code
async


Remote Execution


Run commands on remote
servers (for deployment)


Fabric

» http://fabﬁle.org
» Written by Christian Vest Hansen &
Jeff Forcier


Setup

» pip install fabric
» Add commands to a fabﬁle.py
» fab <command>


Simple example

#Fabﬁle
from fabric.api import run

def host_type():
run('uname -s')

# Command
$ fab -H example.com host_type


Write python to deploy
your code


Deployment


Gunicorn

» http://gunicorn.org/
» Created by Benoit Chesneau


Setup

» pip install gunicorn
» add gunicorn to your Installed Apps
» manage.py run_gunicorn


Need to install Nginx in
front


Nginx setup

server {
listen 80;
server_name example.com;
access_log /var/log/nginx/example.log;

location / {
proxy_pass http://127.0.0.1:8080;
}
}


Awesome Features

» Async backend workers
» Simple conﬁguration
» Python


Stupidly Simple,
Production Ready
Deployment


Packaging


Pip & Virtualenv

» Written by Ian Bicking
» http://pip-installer.org
» http://virtualenv.openplans.org


Set up

» sudo easy_install pip # :D
» sudo pip install virtualenv
» virtualenv awesome_env
» source awesome_env/bin/activate
» pip install Django


Awesome Features

» Separate environments for each
application or deployment
» Install code from repos
» Doesn’t use eggs
» User-owned environment


Adds community standards
to something everyone was
doing their own way


No more PYTHONPATH
hacking


Competing Applications


APIs


APIs

» TastyPie
» Piston


Tagging


Tagging

» Django Taggit *
» Django Tagging


Continuous Integration


Continuous Integration

» Hudson *
» Buildbot


Smaller Problems


Debugging in Development


django-debug-toolbar

» http://github.com/robhudson/
django-debug-toolbar/
» Written by Rob Hudson


Proﬁles & Registration


django-[proﬁles|registration]

» http://bitbucket.org/ubernostrum/
django-proﬁles/
» http://bitbucket.org/ubernostrum/
django-registration/
» Written by James Bennett


Filtering


django-ﬁlter

» http://github.com/alex/django-ﬁlter
» Written by Alex Gaynor


Finding packages


Djangopackages.com


Unsolved Problems


Template Tags


Logging


Model Introspection


Class Based Views


Source code arrangement


Notiﬁcations/Email


Debugging in Production


OpenID/OAuth


VCS Abstraction


Positive Note


Cal’s list of problems ’08
» Multiple Databases
» Multiple multi-keyed cache requests
» Slow HTTP requests while we're doing other stuff
» ORM creates dumb SQL
» Verbose template syntax
» No query debugger
» Ugly SQL syntax
» Commit all ﬁelds at once
» No mascot
» No deployment system
» Migrating Models

Cal’s list of problems ’08
» Multiple Databases
» Multiple multi-keyed cache requests
» Slow HTTP requests while we're doing other stuff
» ORM creates dumb SQL
» Verbose template syntax
» No query debugger
» Ugly SQL syntax (Table aliases)
» Commit all ﬁelds at once
» No mascot
» No deployment system
» Extending models

Remember

» Leverage good applications in the
community
» Take previous knowledge and apply it
to new situations
» You can create an awesome
application used by other people


Questions?

» eric@ericholscher.com
» twitter.com/ericholscher


Large problems, Mostly Solved

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (18)

Similar to Large problems, Mostly Solved

Similar to Large problems, Mostly Solved (20)

More from ericholscher

More from ericholscher (7)

Large problems, Mostly Solved