Production is a bitch

Uploaded on

Deploying to production at scale is a royal pain in the arse (but worth it) and these are some of my observations

Deploying to production at scale is a royal pain in the arse (but worth it) and these are some of my observations

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. Production is a Bitch
  • 2. Some deployments got very well and wow we’re awesome and feel great and LETS PARTY!
  • 3. Some deployments go poorly and wreck weekends and make us feel bad and lets drown our sorrows
  • 4. We do both, but we’re learning
  • 5. Troublesome deployments usually fail in two ways
  • 6. Your code doesn’t work* * more on this later
  • 7. Your code doesn’t scale* * also more later
  • 8. Good deployments are usually based on two principles
  • 9. Developed with the right mindset
  • 10. Developed with good technique
  • 11. The mindset is the more important part
  • 12. “She was one in a million, yeah. So there’s five more, just in New South Wales” The Whitlams
  • 13. Always expect the worst and you’ll only get happy surprises
  • 14. Play the averages
  • 15. 24 Hours
  • 16. 332 Error Pages Served
  • 17. 1,690,671 Total Requests
  • 18. 0.019%
  • 19. your code doesn’t work
  • 20. “But it passes QA/CI/ Acceptance/whatever”
  • 21. Continuous Integration is a dress rehearsal
  • 22. “It’s some thing wrong with the framework or linux or ruby or something”
  • 23. You’ve probably just written something retarded
  • 24. ... except when you haven’t and there’s some obscure one in a million MySql bug
  • 25. your code doesn’t scale
  • 26. does it even matter?
  • 27. Average Response Time 139ms
  • 28. Maximum Response Time 87,997ms
  • 29. “using an RDBMS means your design is wrong” Ben Schwarz
  • 30. Bullshit.
  • 31. “crying yourself to sleep every night means your design is wrong”
  • 32. The correct answer to every question relating to maintenance, scalability, reliability, availability is:
  • 33. It depends.
  • 34. ...that being said, there are some very good guidelines and techniques that rarely go wrong
  • 35. Write noisy code Fail loudly and fast
  • 36. and make sure you’re listening
  • 37. Hoptoad Get Exceptional New Relic ExceptionNotifer
  • 38. Write unit tests for production and hook up alerting for failures
  • 39. Measure, then change
  • 40. New Relic is ace.
  • 41. ruby-prof railsbench Benchmark (stdlib) ab
  • 42. Write slow, clean code See what’s slow and how Make that fast
  • 43. @johnbarton