Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Devops, Dungeons & Dragons

Presentation for Devops downunder, aka devopsdays Sydney 2013. What does it mean to be an expert at the art of operations? How do we learn such things? Can we run firedrills like a D&D campaign?

  • Login to see the comments

Devops, Dungeons & Dragons

  1. 1. Devops, Dungeons & Dragons David Lutz
  2. 2. beginner vs expert HP 500 ATK 20
  3. 3. Scenario 1 Phone goes off at 3am... HP 50 ATK 10
  4. 4. #fail because Johnny didn't ● remain cool under pressure ● make judgement call on severity of problem ● keep track of time ● consider escalating ● make a guess to what the problem might be ● communicate what's going on ● preserve forensic evidence
  5. 5. Scenario 2 Phone goes off at 3am... HP 700 ATK 400
  6. 6. beginner vs expert instinct or experience? how do you level up your skills? SPELL CARD lose 10 HP per turn
  7. 7. Daemon HP 800 ATK 15
  8. 8. How to level up? SPELL CARD Lose 100 HP
  9. 9. Four Stages for Learning Any New Skill Noel Burch SPELL CARD ATK +20 per turn
  10. 10. 1. Unconscious incompetence "I don't know what I don't know" 2. Conscious incompetence "I know what I don't know" 3. Conscious competence "I know it, but it's hard" 4. Unconscious competence "I know it so well, I don't need to think about it"
  11. 11. Training and practice reduce the time between the four stages HP 100 ATK 70
  12. 12. 1. Unconscious incompetence (first 6 days) 2. Conscious incompetence (first 6 weeks) 3. Conscious competence (first 6 months) 4. Unconscious competence (6 months+) David's rule of thumb for new employees
  13. 13. Learning theories Adults can learn in abstract ways. Reading about something. Observing someone else doing something. Children can learn by more direct doing method. Role play. Daemon HP 40 ATK 25
  14. 14. Role-play, fire drills, wargames are a powerful way to learn things. SPELL CARD HP +40 ATK +10
  15. 15. Practice dealing with emergencies Daemon lose a turn HP -5
  16. 16. Responsibilities of devops dungeonmaster 1. Plan the scenario beforehand 2. Explain to the rest of the team what's happened and break stuff 3. Monitor the situation and take notes 4. Time each event during the scenario 5. Postmortem
  17. 17. Pass on knowledge by doing! SPELL CARD ATK +20
  18. 18. The rest of the team 1. Identify the problem 2. Resolve the problem as if it was a real incident at 3am 3. Exercise the alerting systems, monitoring systems, comms systems 4. Learn from each other, talk through what's going on
  19. 19. objective: Positive outcome for the business Reduce MTTR SPELL CARD both lose half ATK
  20. 20. After the firedrill/campaign... Dungeonmaster runs a retro postmortem style 1. Was monitoring and alerting sufficient? 2. Could recovery have been quicker? How? 3. Did we uncover any latent faults or unknown dependencies? 4. Involve the developers. For example, could better kill switches or levers be put in to the apps to aid operating them?
  21. 21. Fantasy Creatures Or, think about how you want your team structured. SPELL CARD HP -10 ATK +10
  22. 22. For example: Party of 4 dwarves wouldn't work well SPELL CARD 2x ATK/round
  23. 23. Dwarves - Slow, very tough and strong, not very smart, like mining Wizards - Good at Magic Elves - Fast, somewhat magical, live in the forest Humans - Not especially good at anything, but adaptable
  24. 24. Creature attributes Strength Speed Magic ability Daemon HP 666 ATK 60
  25. 25. Engineer attributes Programming Operating Systems Data Modelling/Management Networking Metrics, Troubleshooting SPELL CARD Roll -10x1d20 HP
  26. 26. Dwarves == Specialists Wizards == Developers Elves == Sysadmins Humans == Generalists
  27. 27. Developer Programming ✭✭✭✭✭ Operating Systems ✭✭ Data Modelling/Management ✭✭✭ Networking ✭ Metrics, Troubleshooting ✭✭✭ Sysadmin Programming ✭ Operating Systems ✭✭✭✭✭ Data Modelling/Management ✭✭ Networking ✭✭✭ Metrics, Troubleshooting ✭✭✭✭
  28. 28. Specialists DBAs/Network Engineers/QA extremely high skills in one of ● Data Modelling/Management ● Networking ● Metrics, Troubleshooting (and bug finding) Generalists Architects/Automators Wide range of skills, but may not be expert in any area
  29. 29. The end Questions? SPELL CARD invulnerability