ScaleFail

ScaleFail(and MPs’ expenses)

How we built it

Crash #1: moreApache children thanMySQL connections

unreviewed_count = Page.objects.ﬁlter( votes__isnull = True).distinct().count()

SELECT COUNT(DISTINCT èxpenses_page`.ìd`)FROM èxpenses_page` LEFT OUTER JOIN èxpenses_vote` ON ( èxpenses_page`.ìd` = èxpenses_vote`.`page_id` ) WHERE èxpenses_vote`.ìd` IS NULL

unreviewed_count = cache.get(homepage:unreviewed_count)if unreviewed_count is None: unreviewed_count = Page.objects.ﬁlter( votes__isnull = True ).distinct().count() cache.set(homepage: unreviewed_count, unreviewed_count, 60)

• With 70,000 pages and a LOT of votes... • DB takes up 135% of CPU• Cache the count in memcached... • DB drops to %35 of CPU

unreviewed_count = Page.objects.ﬁlter( votes__isnull = True).distinct().count()reviewed_count = Page.objects.ﬁlter( votes__isnull = False).distinct().count()

unreviewed_count = Page.objects.ﬁlter( is_reviewed = False).count()

Migrating to InnoDBon a separate server

ssh mps-live "mysqldump mp_expenses" |sed s/ENGINE=MyISAM/ENGINE=InnoDB/g | sed s/CHARSET=latin1/CHARSET=utf8/g | ssh mysql-big "mysql -u root mp_expenses"

“next” button

def next_global(request): # Next unreviewed page from the whole site all_unreviewed_pages = Page.objects.ﬁlter( is_reviewed = False ).order_by(?) if all_unreviewed_pages: return Redirect( all_unreviewed_pages[0].get_absolute_url() ) else: return HttpResponse( All pages have been reviewed! )

import randomdef next_global_from_cache(request): page_ids = cache.get(unreviewed_page_ids) if page_ids: return Redirect( /page/%s/ % random.choice(page_ids) ) else: return next_global(request)

Next time, I’ll use redis

Read-only modesaved our bacon

ScaleFail

by Simon Willison

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 9

Statistics

ScaleFail Presentation Transcript