Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

2013 Velocity DevOps Metrics -- It's Not Just For WebOps Any More!


Published on

Presentation with Jez Humble, James Turnbull, John Willis

Published in: Technology, Business

2013 Velocity DevOps Metrics -- It's Not Just For WebOps Any More!

  1. 1. DevOps: It’s Not Just For WebOpsAnd We Have The Metrics To Prove ItGene  Kim  (IT  Revolu0on  Press)  Jez  Humble  (ThoughtWorks)  James  Turnbull  (Puppet  Labs)  John  Willis  (Enstra0us  /  Dell)  
  2. 2. PresentersJames Turnbull, VP of Community, Evangelism, and BusinessDevelopment•  Prominent voice in DevOps community•  Author of several technical books including The LogStashBookGene Kim, IT researcher and author•  Founder and former CTO of Tripwire for 13 years•  Author "The Phoenix Project: A Novel About IT, DevOps, andHelping Your Business Win” and “Visible Ops Handbook”Jez Humble, Principal, ThoughtWorks•  Co-author of Continuous Delivery•  Loud mouth on effective practices for software delivery
  3. 3. The Big Why?• Desire  to  understand  the  health  and  habits  of  DevOps  community  • Understand  the  link  between  behaviors  and  performance  • Mi0gate  the  “echo  chamber”  effect:  what  is  the  evidence  of  improved  performance  • Prove  that  DevOps  isn’t  just  for  unicorns…    it’s  for  the  horses,  too.  
  4. 4. Goals For This Presentation• Show  you  how  to  do  cross-­‐sec0on  popula0on  studies  • Survey  Design  • Survey  Execu0on  • Survey  Analysis  • Share  our  top  findings  • Get  your  help  for  our  2013-­‐2014  study  
  5. 5. Where Did The High Performers Come From?
  6. 6. Over Ten Years, We Benchmarked 1500+ IT Orgs
  7. 7. Intuitive AnalyticalAcademicStudyPopulationSamplePopulationSampleAcademicStudyPopulationSampleWhat Is Cross-Section Population StudyPopulationSampleAcademicStudyPopulationStudyBenchmarkROI StoryBestPracticePeerRecommendationAnecdotePrior ExperienceWe AreHereVendorsConsultantsIndustryAnalystTradeshowsIndustryPressBlogshere
  8. 8. High Performing IT Organizations§  High  performers  maintain  a  posture  of  compliance  § Fewest  number  of  repeat  audit  findings  § One-­‐third  amount  of  audit  prepara0on  effort  §  High  performers  find  and  fix  security  breaches  faster  § 5  /mes  more  likely  to  detect  breaches  by  automated  control  § 5  /mes  less  likely  to  have  breaches  result  in  a  loss  event  §  When  high  performers  implement  changes…  § 14  /mes  more  changes  § One-­‐half  the  change  failure  rate    § One-­‐quarter  the  first  fix  failure  rate    § 10x  faster  MTTR  for  Sev  1  outages  §  When  high  performers  manage  IT  resources…  § One-­‐third  the  amount  of  unplanned  work  § 8  /mes  more  projects  and  IT  services  § 6  /mes  more  applica0ons    Source: IT Process Institute, 2008
  9. 9. Visible Ops: Playbook of High Performers• The IT Process Institutehas been studying high-performing organizationssince 1999• What is common to all thehigh performers?• What is different betweenthem and average and lowperformers?• How did they become great?
  10. 10. The Story Begins…Picture credit: Flickr: Floralgal
  11. 11. I Know!12  
  12. 12. Lesson:You are only as smart as theaverageof the top 5 people you hang out with13  
  13. 13. 2012 DevOps Survey•  Written by Puppet Labs, Gene Kim, and Jez Humble•  Ran through December of 2012•  Goal: Get a snapshot of the current state of the healthand habits of the DevOps community
  14. 14. How To Do A Cross Section Population Study• Survey  design  • Survey  execu0on  • Survey  analysis  
  15. 15. The Vision: DevOps Survey Of Practice II• Performance  • Behaviors  • Demographics  • Psychographics  • Planned  target  was  30  ques0ons  • We  asked  for  a  budget  of  8  ques0ons  16  
  16. 16. Survey DesignIn  the  ideal,  we  want  to  be  able  to  say,  “the  more  you  do  [A,  B,  C],  the  more  performance  improves”  X axis: independent variablesDevOps behaviors hypothesized to improve performanceY axis: dependent variablesDevOps performance measures
  17. 17. Y-Axis: What Performance Looks Like• How  ofen  do  you  deploy  code?  (e.g.,  daily,  weekly,  monthly,  yearly)  • What  is  the  average  lead  0me  of  a  change?    (i.e.,  how  long  does  it  take  to  go  from  “code  commihed”  to  “code  successfully  running  in  produc0on?”)    • What  percentage  of  your  changes  require  rollbacks  or  hoiixes?  • On  average,  how  long  does  it  take  to  restore  service  when  something  goes  wrong?  (e.g.,  a  day,  a  week,  a  month,  a  year)  18  
  18. 18. X-Axis: What Behaviors Result In Performance• Are  environment  and  infrastructure  changes  (i.e.,  everything  except  for  code)  checked  into  revision  control?  • Is  there  an  automated  process  to  deploy  environment  and  infrastructure  changes?  • Who  performs  code  deployments?    (e.g.,  Dev,  Ops,  both)  • Who  is  on  the  hook  for  produc0on  support  (e.g.,  Dev,  Ops,  both)  19  
  19. 19. Next Step: Survey Execution•  Once  the  survey  instrument  is  ready,  then  you  need  people  to  take  the  survey  •  Michelle  Carroll  and  the  Puppet  Labs  marke0ng  machine  went  to  work  • Mailing  lists  • Twiher  • Reddit  
  20. 20. 21  
  21. 21. The Results• 4039  completed  survey  responses  in  30  days!!!  • This  is  amazing.    With  a  primary  research  firm,  this  type  of  popula0on  size  usually  would  cost  $100-­‐200K!  22  
  22. 22. Lesson: Do Not Change Answers Mid-Survey
  23. 23. Uh, Oh… Now what?
  24. 24. Lesson:Get data analyst on the team BEFORE youdo survey design!25  
  25. 25. Where Do You Find Your Own Wally?• Steal  a  consultant  ;)  • R  User  Groups  • University  students  in  search  of  a  thesis  • Economics  students  a  plus  (sta0s0cs,  applied)  • Ex-­‐Wall  Street  investors  (not  recommended)  Wally Zabaglio, Puppet Labs Data Analyst–  Former specialized analytics consultant at The NielsenCompany; analyst for the US Department of EnergysOffice of Environmental Management–  Primary analyst on DevOps survey results
  26. 26. 27  
  27. 27. Data Cleanup• Used  Google  Refine  to  clean  up  data  and  compensate  for  “Gene  screwup”  • Used  R  to  do  analysis  • Look  for  signal  • Explore  data  • Run  correla0ons  
  28. 28. Key Findings
  29. 29. High Performing DevOps Teams• They’re  more  agile  • 30x  more  frequent  deployments  • 8,000x  shorter  lead  0me    (minutes/hours  vs.  months/quarters)  • They’re  more  reliable  • 2x  the  change  success  rate  • 12x  faster  MTTR  
  30. 30. 31Technical Practices Matter•  89% are using infrastructure version control•  82% are using automated code deployments
  31. 31. Deliberate Practice MattersOrganizations that implemented DevOps practices over 12months ago were 5x more likely to be high performing thanorganizations that weren’t implementing DevOps at all.
  32. 32. The Lost Hypotheses *o  Who performs code deployments?o  Who is on the hook for production support?
  33. 33. Measuring Culture“I’ll tell you EXACTLY what devops means.Devops means giving a shit about your job enough to not passthe buck. Devops means giving a shit about your job enough towant to learn all the parts and not just your little world.Developers need to understand infrastructure. Operationspeople need to understand code. People need to fucking workwith each other and not just occupy space next to each other.”John Vincent | @lusis |
  34. 34. Measuring CultureTrust (and Verify)Continuous Improvement vs Risk Management Theater“Human Error”Heroes / "high performers"
  35. 35. The tribe is operationally strong *    70%  Opera0ons  people      
  36. 36. The tribe is operationally strong    This  is  not  a  Dev  takeover      
  37. 37. The tribe is enterprise too!  26%  -­‐  500  to  9999  16%  -­‐  10K  plus      
  38. 38. Overcoming barriers to DevOps adoptionBiggest barriers were cultural•  Lack of manager buy-in•  Lack of team buy-in•  Value of DevOps not understood outside mygroup
  39. 39. What are organizations looking for in DevOps skills?•  Coding/scripting (84%)•  People skills (60%)•  Process re-engineering skills (56%)•  Experience with specific tools (19%)
  40. 40. Challenges to culture  Its  easier  to  teach  tools  than  teach  people  skills      
  41. 41. Preparing For DevOps Survey 2013-2014• How  do  we  measure  culture?  • High  management  vs.  low  trust  management  styles  (and  link  to  team  size)  • Industry  code  • Beher  define  performance  variables  to  enable  regression  analysis  (Likert-­‐type  scale)  • Business  cri0cality:  system  of  engagement  vs.  system  of  record  • What  else?    We  want  to  hear  your  thoughts!  
  42. 42. If you … Then …… want to learn more about the findings •  Read the full-length report:•  Share the infographic:•  Get these slides:… have ideas on the DevOps Survey OfPractice 2013-2014•  Come talk with us!•  Join our Google+ Community:… share your stories of transformation •  Go to FlowCon!… get a free copy of The Phoenix Project:A Novel About IT, DevOps, and HelpingYour Business Win•  Go to the AppFirst booth at 2:40pm. Genewill be signing books!