• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
ScaleFail
 

ScaleFail

on

  • 1,788 views

A 5 minute presentation for UK ScaleCamp 2009

A 5 minute presentation for UK ScaleCamp 2009

Statistics

Views

Total Views
1,788
Views on SlideShare
1,779
Embed Views
9

Actions

Likes
0
Downloads
3
Comments
0

1 Embed 9

http://lanyrd.com 9

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    ScaleFail ScaleFail Presentation Transcript

    • ScaleFail(and MPs’ expenses)
    • Copyright © Steve Bell 2009
    • How we built it
    • Crash #1: moreApache children thanMySQL connections
    • unreviewed_count = Page.objects.filter( votes__isnull = True).distinct().count()
    • SELECT COUNT(DISTINCT `expenses_page`.`id`)FROM `expenses_page` LEFT OUTER JOIN `expenses_vote` ON ( `expenses_page`.`id` = `expenses_vote`.`page_id` ) WHERE `expenses_vote`.`id` IS NULL
    • unreviewed_count = cache.get(homepage:unreviewed_count)if unreviewed_count is None: unreviewed_count = Page.objects.filter( votes__isnull = True ).distinct().count() cache.set(homepage: unreviewed_count, unreviewed_count, 60)
    • • With 70,000 pages and a LOT of votes... • DB takes up 135% of CPU• Cache the count in memcached... • DB drops to %35 of CPU
    • unreviewed_count = Page.objects.filter( votes__isnull = True).distinct().count()reviewed_count = Page.objects.filter( votes__isnull = False).distinct().count()
    • unreviewed_count = Page.objects.filter( is_reviewed = False).count()
    • Migrating to InnoDBon a separate server
    • ssh mps-live "mysqldump mp_expenses" |sed s/ENGINE=MyISAM/ENGINE=InnoDB/g | sed s/CHARSET=latin1/CHARSET=utf8/g | ssh mysql-big "mysql -u root mp_expenses"
    • “next” button
    • def next_global(request): # Next unreviewed page from the whole site all_unreviewed_pages = Page.objects.filter( is_reviewed = False ).order_by(?) if all_unreviewed_pages: return Redirect( all_unreviewed_pages[0].get_absolute_url() ) else: return HttpResponse( All pages have been reviewed! )
    • def next_global(request): # Next unreviewed page from the whole site all_unreviewed_pages = Page.objects.filter( is_reviewed = False ).order_by(?) if all_unreviewed_pages: return Redirect( all_unreviewed_pages[0].get_absolute_url() ) else: return HttpResponse( All pages have been reviewed! )
    • import randomdef next_global_from_cache(request): page_ids = cache.get(unreviewed_page_ids) if page_ids: return Redirect( /page/%s/ % random.choice(page_ids) ) else: return next_global(request)
    • Next time, I’ll use redis
    • Read-only modesaved our bacon
    • Copyright © Martin Rowson 2008