This document summarizes the steps taken to optimize the performance of a website for reviewing parliamentary expenses. Initially, querying for unreviewed pages was causing high CPU usage on the database server. To address this, the count of unreviewed pages was cached in Memcached, reducing CPU usage. Other optimizations included migrating the database to InnoDB, adding a "next" button that redirects to a random unreviewed page, and caching the list of unreviewed page IDs in Memcached for faster random access. During a future scale issue, Redis will be used instead of Memcached. Putting the site in read-only mode also helped during a period of high load.

1. ScaleFail
(and MPs’ expenses)

2. Copyright © Steve Bell 2009

9. How we built it

11. Crash #1: more
Apache children than
MySQL connections

14. unreviewed_count = Page.objects.filter(
votes__isnull = True
).distinct().count()

15. SELECT
COUNT(DISTINCT `expenses_page`.`id`)
FROM
`expenses_page` LEFT OUTER JOIN `expenses_vote` ON (
`expenses_page`.`id` = `expenses_vote`.`page_id`
) WHERE `expenses_vote`.`id` IS NULL

16. unreviewed_count = cache.get('homepage:unreviewed_count')
if unreviewed_count is None:
unreviewed_count = Page.objects.filter(
votes__isnull = True
).distinct().count()
cache.set('homepage: unreviewed_count', unreviewed_count, 60)

17. • With 70,000 pages and a LOT of votes...
• DB takes up 135% of CPU
• Cache the count in memcached...
• DB drops to %35 of CPU

18. unreviewed_count = Page.objects.filter(
votes__isnull = True
).distinct().count()
reviewed_count = Page.objects.filter(
votes__isnull = False
).distinct().count()

19. unreviewed_count = Page.objects.filter(
is_reviewed = False
).count()

20. Migrating to InnoDB
on a separate server

21. ssh mps-live "mysqldump mp_expenses" |
sed 's/ENGINE=MyISAM/ENGINE=InnoDB/g' |
sed 's/CHARSET=latin1/CHARSET=utf8/g' |
ssh mysql-big "mysql -u root mp_expenses"

22. “next” button

23. def next_global(request):
# Next unreviewed page from the whole site
all_unreviewed_pages = Page.objects.filter(
is_reviewed = False
).order_by('?')
if all_unreviewed_pages:
return Redirect(
all_unreviewed_pages[0].get_absolute_url()
)
else:
return HttpResponse(
'All pages have been reviewed!'
)

24. def next_global(request):
# Next unreviewed page from the whole site
all_unreviewed_pages = Page.objects.filter(
is_reviewed = False
).order_by('?')
if all_unreviewed_pages:
return Redirect(
all_unreviewed_pages[0].get_absolute_url()
)
else:
return HttpResponse(
'All pages have been reviewed!'
)

25. import random
def next_global_from_cache(request):
page_ids = cache.get('unreviewed_page_ids')
if page_ids:
return Redirect(
'/page/%s/' % random.choice(page_ids)
)
else:
return next_global(request)

26. Next time, I’ll use redis

27. Read-only mode
saved our bacon

28. Copyright © Martin Rowson 2008