Your SlideShare is downloading. ×
  • Like
guardian.co.uk - AWS Experience Report - MP Expenses Crowd Sourcing
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

guardian.co.uk - AWS Experience Report - MP Expenses Crowd Sourcing

  • 582 views
Published

Experience report given at Amazon Web Services, Cloud for the Enterprise Event: London …

Experience report given at Amazon Web Services, Cloud for the Enterprise Event: London

CC attributions:

Traffic light - http://www.flickr.com/photos/ahnmyrrh/3620801348/

Published in Technology , News & Politics
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
582
On SlideShare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
19
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • The act will go in to full force in January 2005
  • In October, they released a summary by category. list of mps and bulk totals - eg staffing allowance.
    But with this kind of thing the devil’s in the details, so aggregate detail anonymizes raised Heather’s antenna.

    In US at Washington State 1992 she did the same (Clerks office, asked to see claims,”OK” got a big box out filled with receipts - considered public record). Not one incorrect claim
  • in January 2005 she filed a FOI request. This is when FOI came into effect
  • a parliamentary order released at the same time as the government announced proposals to build a third Heathrow runway
  • Daily Telegraph publishing details of MPs expenses
  • One of our Software Architects, Simon Willison had a day or two between projects and got a PoC together
  • Development proper starts
  • Start building up the team
  • I got involved, setup infrastructure.
  • Go live.
  • Provisioning new machines - envisioned as having spike then long tail
  • Look at documentation for running with EBS on EC2
  • 06:00am
  • The “Drop Everything” school of project management.
  • Business as usual, pull in appropriate resources
  • Mechanical turk, prioritized list, but we couldn’t absorb the resources
  • Top 30 MPs done first
  • Lack of good parrallel uploader - hadoop has built in
  • Great tactic for testing things out.

    Private beta, internally, just a simple cli to change the security group to enable the world
  • Dark launched
    EC2 security groups enabled a quick switch on
    Started on one large instance
  • Elastic IP, switch box - does involve downtime but it may be acceptible for your app.
  • Scripts variable costly, time constraints meant use a “trusted” ami source
  • Read only mode,
  • Common design practice
  • Role management - AWS gives you a base, but doesn’t encapsulate ability to change as system is running.

Transcript

  • 1. MPs Expenses AWS Experiences Paul Nasrat • paul.nasrat@guardian.co.uk • @nasrat
  • 2. “Web development on journalism deadlines”
  • 3. The back story...
  • 4. November 2000 The Freedom of Information Act
  • 5. Heather Brooke
  • 6. 2004 The request
  • 7. January 2005 The FOI request
  • 8. July 2006 The FOI commissioner
  • 9. February 2008 The Information Tribunal
  • 10. May 2008 The high court
  • 11. March 2009 The mole
  • 12. 8th May, 2009 The Daily Telegraph
  • 13. At the Guardian...
  • 14. April: “Expenses are due out in a couple of months, is there anything we can do?”
  • 15. June: “Expenses have been bumped forward, they’re out next week!”
  • 16. Thursday 11th June The proof-of-concept
  • 17. Monday 15th June The tentative go-ahead
  • 18. Tuesday 16th June Designer + client-side engineer
  • 19. Wednesday 17th June Operations engineer
  • 20. Thursday 18th June Launch day!
  • 21. How we built it
  • 22. Local development environment
  • 23. Django and Git
  • 24. EC2
  • 25. EBS + Snapshotting
  • 26. On the day
  • 27. “Drop Everything”
  • 28. Response Team
  • 29. Embrace change
  • 30. Photoshop + AppleScript v.s. Java + IntelliJ
  • 31. Incremental Delivery
  • 32. Images on our docroot (S3 upload was taking too long)
  • 33. Blitz QA
  • 34. Dark Launched
  • 35. Launch! (on EC2)
  • 36. Migrating to InnoDB on a separate server
  • 37. Refactor infrastructure
  • 38. Downsizing
  • 39. Lessons Learnt
  • 40. AMI Building broken echo "Installing Yum 3.0" wget http://linux.duke.edu/projects/yum/ download/3.0/yum-3.0.5.tar.gz tar -xvzf yum-3.0.5.tar.gz cd yum-3.0.5 make DESTDIR=/ install
  • 41. Software Switches
  • 42. User Data Bootstrap
  • 43. Configuration Management
  • 44. Puppet
  • 45. Final thoughts • Elasticity requires thought • Fire-drill for event driven projects • Long tail of crowd-sourcing • Form teams around projects