MPs Expenses
    AWS Experiences
Paul Nasrat • paul.nasrat@guardian.co.uk • @nasrat
“Web development on
journalism deadlines”
The back story...
November 2000
The Freedom of Information Act
Heather Brooke
2004
The request
January 2005
 The FOI request
July 2006
The FOI commissioner
February 2008
The Information Tribunal
May 2008
The high court
March 2009
  The mole
8th May, 2009
The Daily Telegraph
At the Guardian...
April: “Expenses are due out
  in a couple of months, is
there anything we can do?”
June: “Expenses have been
bumped forward, they’re out
        next week!”
Thursday 11th June
  The proof-of-concept
Monday 15th June
The tentative go-ahead
Tuesday 16th June
Designer + client-side engineer
Wednesday 17th June
   Operations engineer
Thursday 18th June
    Launch day!
How we built it
Local development
   environment
Django and Git
EC2
EBS + Snapshotting
On the day
“Drop Everything”
Response Team
Embrace change
Photoshop + AppleScript
           v.s.
     Java + IntelliJ
Incremental Delivery
Images on our docroot
 (S3 upload was taking
       too long)
Blitz QA
Dark Launched
Launch! (on EC2)
Migrating to InnoDB
on a separate server
Refactor infrastructure
Downsizing
Lessons Learnt
AMI Building broken
	 	 	 echo "Installing Yum 3.0"
	 	 	 wget http://linux.duke.edu/projects/yum/
download/3.0/yum-3.0.5....
Software Switches
User Data Bootstrap
Configuration
Management
Puppet
Final thoughts

• Elasticity requires thought
• Fire-drill for event driven projects
• Long tail of crowd-sourcing
• Form ...
guardian.co.uk - AWS Experience Report - MP Expenses Crowd Sourcing
guardian.co.uk - AWS Experience Report - MP Expenses Crowd Sourcing
guardian.co.uk - AWS Experience Report - MP Expenses Crowd Sourcing
guardian.co.uk - AWS Experience Report - MP Expenses Crowd Sourcing
guardian.co.uk - AWS Experience Report - MP Expenses Crowd Sourcing
guardian.co.uk - AWS Experience Report - MP Expenses Crowd Sourcing
guardian.co.uk - AWS Experience Report - MP Expenses Crowd Sourcing
guardian.co.uk - AWS Experience Report - MP Expenses Crowd Sourcing
guardian.co.uk - AWS Experience Report - MP Expenses Crowd Sourcing
guardian.co.uk - AWS Experience Report - MP Expenses Crowd Sourcing
guardian.co.uk - AWS Experience Report - MP Expenses Crowd Sourcing
guardian.co.uk - AWS Experience Report - MP Expenses Crowd Sourcing
Upcoming SlideShare
Loading in...5
×

guardian.co.uk - AWS Experience Report - MP Expenses Crowd Sourcing

614

Published on

Experience report given at Amazon Web Services, Cloud for the Enterprise Event: London

CC attributions:

Traffic light - http://www.flickr.com/photos/ahnmyrrh/3620801348/

Published in: Technology, News & Politics
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
614
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
19
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • The act will go in to full force in January 2005
  • In October, they released a summary by category. list of mps and bulk totals - eg staffing allowance.
    But with this kind of thing the devil’s in the details, so aggregate detail anonymizes raised Heather’s antenna.

    In US at Washington State 1992 she did the same (Clerks office, asked to see claims,”OK” got a big box out filled with receipts - considered public record). Not one incorrect claim
  • in January 2005 she filed a FOI request. This is when FOI came into effect
  • a parliamentary order released at the same time as the government announced proposals to build a third Heathrow runway
  • Daily Telegraph publishing details of MPs expenses
  • One of our Software Architects, Simon Willison had a day or two between projects and got a PoC together
  • Development proper starts
  • Start building up the team
  • I got involved, setup infrastructure.
  • Go live.
  • Provisioning new machines - envisioned as having spike then long tail
  • Look at documentation for running with EBS on EC2
  • 06:00am
  • The “Drop Everything” school of project management.
  • Business as usual, pull in appropriate resources
  • Mechanical turk, prioritized list, but we couldn’t absorb the resources
  • Top 30 MPs done first
  • Lack of good parrallel uploader - hadoop has built in
  • Great tactic for testing things out.

    Private beta, internally, just a simple cli to change the security group to enable the world
  • Dark launched
    EC2 security groups enabled a quick switch on
    Started on one large instance
  • Elastic IP, switch box - does involve downtime but it may be acceptible for your app.
  • Scripts variable costly, time constraints meant use a “trusted” ami source
  • Read only mode,
  • Common design practice
  • Role management - AWS gives you a base, but doesn’t encapsulate ability to change as system is running.
  • guardian.co.uk - AWS Experience Report - MP Expenses Crowd Sourcing

    1. 1. MPs Expenses AWS Experiences Paul Nasrat • paul.nasrat@guardian.co.uk • @nasrat
    2. 2. “Web development on journalism deadlines”
    3. 3. The back story...
    4. 4. November 2000 The Freedom of Information Act
    5. 5. Heather Brooke
    6. 6. 2004 The request
    7. 7. January 2005 The FOI request
    8. 8. July 2006 The FOI commissioner
    9. 9. February 2008 The Information Tribunal
    10. 10. May 2008 The high court
    11. 11. March 2009 The mole
    12. 12. 8th May, 2009 The Daily Telegraph
    13. 13. At the Guardian...
    14. 14. April: “Expenses are due out in a couple of months, is there anything we can do?”
    15. 15. June: “Expenses have been bumped forward, they’re out next week!”
    16. 16. Thursday 11th June The proof-of-concept
    17. 17. Monday 15th June The tentative go-ahead
    18. 18. Tuesday 16th June Designer + client-side engineer
    19. 19. Wednesday 17th June Operations engineer
    20. 20. Thursday 18th June Launch day!
    21. 21. How we built it
    22. 22. Local development environment
    23. 23. Django and Git
    24. 24. EC2
    25. 25. EBS + Snapshotting
    26. 26. On the day
    27. 27. “Drop Everything”
    28. 28. Response Team
    29. 29. Embrace change
    30. 30. Photoshop + AppleScript v.s. Java + IntelliJ
    31. 31. Incremental Delivery
    32. 32. Images on our docroot (S3 upload was taking too long)
    33. 33. Blitz QA
    34. 34. Dark Launched
    35. 35. Launch! (on EC2)
    36. 36. Migrating to InnoDB on a separate server
    37. 37. Refactor infrastructure
    38. 38. Downsizing
    39. 39. Lessons Learnt
    40. 40. AMI Building broken echo "Installing Yum 3.0" wget http://linux.duke.edu/projects/yum/ download/3.0/yum-3.0.5.tar.gz tar -xvzf yum-3.0.5.tar.gz cd yum-3.0.5 make DESTDIR=/ install
    41. 41. Software Switches
    42. 42. User Data Bootstrap
    43. 43. Configuration Management
    44. 44. Puppet
    45. 45. Final thoughts • Elasticity requires thought • Fire-drill for event driven projects • Long tail of crowd-sourcing • Form teams around projects
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×