• Like
  • Save
guardian.co.uk - AWS Experience Report - MP Expenses Crowd Sourcing
Upcoming SlideShare
Loading in...5

guardian.co.uk - AWS Experience Report - MP Expenses Crowd Sourcing



Experience report given at Amazon Web Services, Cloud for the Enterprise Event: London

Experience report given at Amazon Web Services, Cloud for the Enterprise Event: London

CC attributions:

Traffic light - http://www.flickr.com/photos/ahnmyrrh/3620801348/



Total Views
Views on SlideShare
Embed Views



3 Embeds 5

http://www.linkedin.com 3
http://www.slideshare.net 1
https://www.linkedin.com 1


Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment
  • The act will go in to full force in January 2005
  • In October, they released a summary by category. list of mps and bulk totals - eg staffing allowance. <br /> But with this kind of thing the devil&#x2019;s in the details, so aggregate detail anonymizes raised Heather&#x2019;s antenna. <br /> <br /> In US at Washington State 1992 she did the same (Clerks office, asked to see claims,&#x201D;OK&#x201D; got a big box out filled with receipts - considered public record). Not one incorrect claim
  • in January 2005 she filed a FOI request. This is when FOI came into effect
  • a parliamentary order released at the same time as the government announced proposals to build a third Heathrow runway
  • Daily Telegraph publishing details of MPs expenses
  • One of our Software Architects, Simon Willison had a day or two between projects and got a PoC together
  • Development proper starts
  • Start building up the team
  • I got involved, setup infrastructure.
  • Go live.
  • Provisioning new machines - envisioned as having spike then long tail
  • Look at documentation for running with EBS on EC2
  • 06:00am
  • The &#x201C;Drop Everything&#x201D; school of project management.
  • Business as usual, pull in appropriate resources
  • Mechanical turk, prioritized list, but we couldn&#x2019;t absorb the resources
  • Top 30 MPs done first
  • Lack of good parrallel uploader - hadoop has built in
  • Great tactic for testing things out. <br /> <br /> Private beta, internally, just a simple cli to change the security group to enable the world
  • Dark launched <br /> EC2 security groups enabled a quick switch on <br /> Started on one large instance
  • Elastic IP, switch box - does involve downtime but it may be acceptible for your app.
  • Scripts variable costly, time constraints meant use a &#x201C;trusted&#x201D; ami source
  • Read only mode,
  • Common design practice
  • Role management - AWS gives you a base, but doesn&#x2019;t encapsulate ability to change as system is running.

guardian.co.uk - AWS Experience Report - MP Expenses Crowd Sourcing guardian.co.uk - AWS Experience Report - MP Expenses Crowd Sourcing Presentation Transcript

  • MPs Expenses AWS Experiences Paul Nasrat • paul.nasrat@guardian.co.uk • @nasrat
  • “Web development on journalism deadlines”
  • The back story...
  • November 2000 The Freedom of Information Act
  • Heather Brooke
  • 2004 The request
  • January 2005 The FOI request
  • July 2006 The FOI commissioner
  • February 2008 The Information Tribunal
  • May 2008 The high court
  • March 2009 The mole
  • 8th May, 2009 The Daily Telegraph
  • At the Guardian...
  • April: “Expenses are due out in a couple of months, is there anything we can do?”
  • June: “Expenses have been bumped forward, they’re out next week!”
  • Thursday 11th June The proof-of-concept
  • Monday 15th June The tentative go-ahead
  • Tuesday 16th June Designer + client-side engineer
  • Wednesday 17th June Operations engineer
  • Thursday 18th June Launch day!
  • How we built it
  • Local development environment
  • Django and Git
  • EC2
  • EBS + Snapshotting
  • On the day
  • “Drop Everything”
  • Response Team
  • Embrace change
  • Photoshop + AppleScript v.s. Java + IntelliJ
  • Incremental Delivery
  • Images on our docroot (S3 upload was taking too long)
  • Blitz QA
  • Dark Launched
  • Launch! (on EC2)
  • Migrating to InnoDB on a separate server
  • Refactor infrastructure
  • Downsizing
  • Lessons Learnt
  • AMI Building broken echo "Installing Yum 3.0" wget http://linux.duke.edu/projects/yum/ download/3.0/yum-3.0.5.tar.gz tar -xvzf yum-3.0.5.tar.gz cd yum-3.0.5 make DESTDIR=/ install
  • Software Switches
  • User Data Bootstrap
  • Configuration Management
  • Puppet
  • Final thoughts • Elasticity requires thought • Fire-drill for event driven projects • Long tail of crowd-sourcing • Form teams around projects