A 5 minute presentation for UK ScaleCamp 2009

1. ScaleFail
(and MPs’ expenses)

2. Copyright © Steve Bell 2009

9. How we built it

11. Crash #1: more
Apache children than
MySQL connections

14. unreviewed_count = Page.objects.filter(
votes__isnull = True
).distinct().count()

15. SELECT
COUNT(DISTINCT `expenses_page`.`id`)
FROM
`expenses_page` LEFT OUTER JOIN `expenses_vote` ON (
`expenses_page`.`id` = `expenses_vote`.`page_id`
) WHERE `expenses_vote`.`id` IS NULL

16. unreviewed_count = cache.get('homepage:unreviewed_count')
if unreviewed_count is None:
unreviewed_count = Page.objects.filter(
votes__isnull = True
).distinct().count()
cache.set('homepage: unreviewed_count', unreviewed_count, 60)

17. • With 70,000 pages and a LOT of votes...
• DB takes up 135% of CPU
• Cache the count in memcached...
• DB drops to %35 of CPU

18. unreviewed_count = Page.objects.filter(
votes__isnull = True
).distinct().count()
reviewed_count = Page.objects.filter(
votes__isnull = False
).distinct().count()

19. unreviewed_count = Page.objects.filter(
is_reviewed = False
).count()

20. Migrating to InnoDB
on a separate server

21. ssh mps-live "mysqldump mp_expenses" |
sed 's/ENGINE=MyISAM/ENGINE=InnoDB/g' |
sed 's/CHARSET=latin1/CHARSET=utf8/g' |
ssh mysql-big "mysql -u root mp_expenses"

22. “next” button

23. def next_global(request):
# Next unreviewed page from the whole site
all_unreviewed_pages = Page.objects.filter(
is_reviewed = False
).order_by('?')
if all_unreviewed_pages:
return Redirect(
all_unreviewed_pages[0].get_absolute_url()
)
else:
return HttpResponse(
'All pages have been reviewed!'
)

24. def next_global(request):
# Next unreviewed page from the whole site
all_unreviewed_pages = Page.objects.filter(
is_reviewed = False
).order_by('?')
if all_unreviewed_pages:
return Redirect(
all_unreviewed_pages[0].get_absolute_url()
)
else:
return HttpResponse(
'All pages have been reviewed!'
)

25. import random
def next_global_from_cache(request):
page_ids = cache.get('unreviewed_page_ids')
if page_ids:
return Redirect(
'/page/%s/' % random.choice(page_ids)
)
else:
return next_global(request)

26. Next time, I’ll use redis

27. Read-only mode
saved our bacon

28. Copyright © Martin Rowson 2008