Keynote for DjangoCon 2009, presented on the 8th of September 2009. Covers two cowboy projects - WildLifeNearYou.com and MP expenses - and talks about ways of "reigning in the cowboy" and developing in a more sustainable way.

1. Cowboy development
with Django
Simon Willison
DjangoCon 2009

2. http://www.youtube.com/watch?v=nZx9sNXv9h0

3. Just one problem... we
didn’t have cowboys in
England

4. The Napoleonic Wars

5. A Napoleonic Sea Fort
http://en.wikipedia.org/wiki/File:Alderney_-_Fort_Clonque_02.jpg

6. Super Evil Dev Fort

7. http://www.anotherurl.com/travel/fort_clonque/handbook.htm

8. Photos by Cindy Li
http://www.flickr.com/photos/cindyli/sets/72157610369683426/

14. WildLifeNearYou.com
(Built in 1 week and 10 months)

15. DEMO

16. Search uses the geospatial branch of Xapian
Species database comes from Freebase
Photos can be imported from Flickr
“Suggest changes” to our Zoo information uses
model objects representing proposed changes to
other model objects

17. /dev/fort
Cohort 3: Winter 2009 What is /dev/fort?
The trip Imagine a place of no distractions, no
The third /dev/fort will run from 9th to 16th November on the Kintyre IM, no Twitter — in fact, no
Peninsula in Scotland. internet. Within, a group of a dozen
or more developers, designers,
thinkers and doers. And a lot of a
food.
Cohort 2: Summer 2009
Now imagine that place is a fort.
The trip
The second /dev/fort ran from 30th May to 6th June 2009 at Knockbrex
Castle in Scotland. As with the first cohort, we have a few remaining
problems still to iron out (thorny issues inside Django we were hoping to
avoid, that sort of thing). We hope to have the site in alpha by the end of the
summer.
Cohort members
Ryan Alexander, Steven Anderson, James Aylett, Hannah Donovan, Natalie
Downe, Mark Norman Francis, Matthew Hasler, Steve Marshall, Richard
Pope, Gareth Rushgrove, Simon Willison.
The idea behind /dev/fort is to throw
Cohort 1: Winter 2008 a group of people together, cut them
off from the rest of the world, and
http://devfort.com/

18. Cowboy development at work

19. MP expenses

20. Heather Brooke

21. January 2005
The FOI request

22. February 2008
The Information Tribunal

23. “Transparency will
damage democracy”

24. January 2009
The exemption law

27. March 2009
The mole

28. “All of the receipts of 650-odd MPs,
redacted and unredacted, are for sale at a
price of £300,000, so I am told. The price
is going up because of the interest in the
subject.”
Sir Stuart Bell, MP
Newsnight, 30th March

29. 8th May, 2009
The Daily Telegraph

30. At the Guardian...

31. April: “Expenses are due out in
a couple of months, is there
anything we can do?”

32. June: “Expenses have been
bumped forward, they’re out
next week!”

33. Thursday 11th June
The proof-of-concept

34. Monday 15th June
The tentative go-ahead

35. Tuesday 16th June
Designer + client-side engineer

36. Wednesday 17th June
Operations engineer

37. Thursday 18th June
Launch day!

44. How we built it

47. $ convert Frank_Comm.pdf pages.png

49. Frictionless registration

51. Page filters

53. page_filters = (
# Maps name of filter to dictionary of kwargs to doc.pages.filter()
('reviewed', {
'votes__isnull': False
}),
('unreviewed', {
'votes__isnull': True
}),
('with line items', {
'line_items__isnull': False
}),
('interesting', {
'votes__interestingvote__status': 'yes'
}),
('interesting but known', {
'votes__interestingvote__status': 'known'
...
)
page_filters_lookup = dict(page_filters)

54. pages = doc.pages.all()
if page_filter:
kwargs = page_filters_lookup.get(page_filter)
if kwargs is None:
raise Http404, 'Invalid page filter: %s' % page_filter
pages = pages.filter(**kwargs).distinct()
# Build the filters
filters = []
for name, kwargs in page_filters:
filters.append({
'name': name,
'count': doc.pages.filter(**kwargs).distinct().count(),
})

55. Matching names

56. http://github.com/simonw/datamatcher

57. On the day

61. def get_mp_pages():
"Returns list of (mp-name, mp-page-url) tuples"
soup = Soup(urllib.urlopen(INDEX_URL))
mp_links = []
for link in soup.findAll('a'):
if link.get('title', '').endswith("'s allowances"):
mp_links.append(
(link['title'].replace("'s allowances", ''), link['href'])
)
return mp_links

62. def get_pdfs(mp_url):
"Returns list of (description, years, pdf-url, size) tuples"
soup = Soup(urllib.urlopen(mp_url))
pdfs = []
trs = soup.findAll('tr')[1:] # Skip the first, it's the table header
for tr in trs:
name_td, year_td, pdf_td = tr.findAll('td')
name = name_td.string
year = year_td.string
pdf_url = pdf_td.find('a')['href']
size = pdf_td.find('a').contents[-1].replace('(', '').replace(')', '')
pdfs.append(
(name, year, pdf_url, size)
)
return pdfs

65. “Drop Everything”

66. Photoshop + AppleScript
v.s.
Java + IntelliJ

67. Images on our docroot (S3
upload was taking too long)

68. Blitz QA

69. Launch! (on EC2)

71. Crash #1: more Apache
children than MySQL
connections

74. unreviewed_count = Page.objects.filter(
votes__isnull = True
).distinct().count()

75. SELECT
COUNT(DISTINCT `expenses_page`.`id`)
FROM
`expenses_page` LEFT OUTER JOIN `expenses_vote` ON (
`expenses_page`.`id` = `expenses_vote`.`page_id`
) WHERE `expenses_vote`.`id` IS NULL

76. unreviewed_count = cache.get('homepage:unreviewed_count')
if unreviewed_count is None:
unreviewed_count = Page.objects.filter(
votes__isnull = True
).distinct().count()
cache.set('homepage: unreviewed_count', unreviewed_count, 60)

77. With 70,000 pages and a LOT of votes...
DB takes up 135% of CPU
Cache the count in memcached...
DB drops to %35 of CPU

78. unreviewed_count = Page.objects.filter(
votes__isnull = True
).distinct().count()
reviewed_count = Page.objects.filter(
votes__isnull = False
).distinct().count()

79. unreviewed_count = Page.objects.filter(
is_reviewed = False
).count()

80. Migrating to InnoDB on a
separate server

81. ssh mps-live "mysqldump mp_expenses" |
sed 's/ENGINE=MyISAM/ENGINE=InnoDB/g' |
sed 's/CHARSET=latin1/CHARSET=utf8/g' |
ssh mysql-big "mysql -u root mp_expenses"

82. Reigning in the cowboy

83. Reigning in the cowboy
An RSS to JSON proxy service
Pair programming
Comprehensive unit tests, with mocks
Continuous integration (Team City)
Deployment scripts against CI build numbers

84. Points of embarrassment
Database required to run the test suite
Logging? What logging?
Tests get deployed alongside the code (!)
... but generally pretty smooth sailing

85. A final thought

86. Web development in 2005
Relational
Cache
Database
Application Admin tools
Templates XML feeds

87. Web development in 2009
Relational Search Datastructure External web Non-relational
Cache
Database index servers services database
Admin tools
Application Message queue Offline workers
Monitoring and reporting
Templates XML feeds API Webhooks

88. Thank you