Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Rollback: The Impossible Dream


Published on

Roll back doesn’t exist. It’s not real. It’s a fantasy, a dream, a delusion. Any vendor who tells you they have a roll back capability is lying to you. And lying to you in a downright dangerous way that will come back to haunt you at 4am in a war room when someone says:

“We can’t fix this. Let’s roll back the deployment.”

This talk is designed to explain and demonstrate to Operations staff:

Why roll back is a fantasy and explained with a dash of Werner Heisenberg
Why it is dangerous and how you can recognize when you’re about to get trapped
How you can avoid falling into that trap of considering it an appropriate compensating control.
It’ll also explain what you can actually do operationally instead of “rolling back”. This will cover other alternative compensating controls that can help you get running again and resolve your outage whilst still allowing you to find root cause.

Published in: Technology, Spiritual
  • Be the first to comment

Rollback: The Impossible Dream

  1. 1. RollbackThe Impossible Dream by James Turnbull jamtur01 @ github kartar @ twitter jamesturnbull on freenode james @
  2. 2. About MeVP Technical Operations at Puppet Labs Puppet guy Ruby guy Talks funny
  3. 3. A show of hands
  4. 4. Who thinks theyknow what rollback is?
  5. 5. Last set of hands
  6. 6. YMMV
  7. 7. Definitions
  8. 8. Traditional
  9. 9. Modern
  10. 10. Fact or Fiction?
  11. 11. Accept certain constraints
  12. 12. Constraint #1Apply sufficient capital
  13. 13. Constraint #2 Idempotent
  14. 14. Constraint #3Cascade-less failure
  15. 15. Constraint #4 Resources
  16. 16. A Philosophical Digression
  17. 17. If I know where I amI don’t know how I got there If I know how I got there I don’t know where I am
  18. 18. Very few “systems”are truly deterministic
  19. 19. A Mathematical Digression
  20. 20. On system rollback and totalised fields An algebraic approach to system change Mark Burgess and Alva Couch 20th June 2011 df
  21. 21. So what’s wrong with rollback?
  22. 22. Risk
  23. 23. Learning from mistakes
  24. 24. Complex systems are … complex
  25. 25. Human error
  26. 26. What is the problemrollback is trying to solve?
  27. 27. What is the problem YOU are trying to solve?
  28. 28. So how can wemitigate Rollback shortcomings?
  29. 29. Preventative Design
  30. 30. Rollback is (often) anarchitecture problem
  31. 31. Increase Resilience
  32. 32. OperationalIntelligence
  33. 33. A little bit of DevOps in every byte…
  34. 34. Small, iterative changes
  35. 35. Accept that failure happens
  36. 36. “We can’t test that? Okay we can roll it back if it breaks…”
  37. 37. Assumption is themother of all fuckups*
  38. 38. “But the system can’t be{run|upgraded|deployed} like that because…”
  39. 39. Conclusions
  40. 40. Rollback is possible but not probable
  41. 41. If you have to have “rollback” accept constraints
  42. 42. You can mitigate the need for it
  43. 43. Thank you!Questions/Insults? jamtur01 @ github kartar @ twitter jamesturnbull on freenode james @