Successfully reported this slideshow.
Your SlideShare is downloading. ×

2013 Velocity DevOps Metrics -- It's Not Just For WebOps Any More!

Loading in …3

Check these out next

1 of 47 Ad

More Related Content

Slideshows for you (20)

Viewers also liked (20)


Similar to 2013 Velocity DevOps Metrics -- It's Not Just For WebOps Any More! (20)


Recently uploaded (20)

2013 Velocity DevOps Metrics -- It's Not Just For WebOps Any More!

  1. 1. DevOps: It’s Not Just For WebOps And We Have The Metrics To Prove It Gene  Kim  (IT  Revolu0on  Press)   Jez  Humble  (ThoughtWorks)   James  Turnbull  (Puppet  Labs)   John  Willis  (Enstra0us  /  Dell)  
  2. 2. Presenters James Turnbull, VP of Community, Evangelism, and Business Development •  Prominent voice in DevOps community •  Author of several technical books including The LogStash Book Gene Kim, IT researcher and author •  Founder and former CTO of Tripwire for 13 years •  Author "The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win” and “Visible Ops Handbook” Jez Humble, Principal, ThoughtWorks •  Co-author of Continuous Delivery •  Loud mouth on effective practices for software delivery
  3. 3. The Big Why? • Desire  to  understand  the  health  and  habits  of   DevOps  community   • Understand  the  link  between  behaviors  and   performance   • Mi0gate  the  “echo  chamber”  effect:  what  is   the  evidence  of  improved  performance   • Prove  that  DevOps  isn’t  just  for  unicorns…    it’s   for  the  horses,  too.  
  4. 4. Goals For This Presentation • Show  you  how  to  do  cross-­‐sec0on  popula0on   studies   • Survey  Design   • Survey  Execu0on   • Survey  Analysis   • Share  our  top  findings   • Get  your  help  for  our  2013-­‐2014  study  
  5. 5. Where Did The High Performers Come From?
  6. 6. Over Ten Years, We Benchmarked 1500+ IT Orgs
  7. 7. Intuitive Analytical Academic Study Population Sample Population Sample Academic Study Population Sample What Is Cross-Section Population Study Population Sample Academic Study Population Study Benchmark ROI Story Best Practice Peer Recommendation Anecdote Prior Experience We Are Here Vendors Consultants Industry Analyst Tradeshows Industry Press Blogshere
  8. 8. High Performing IT Organizations §  High  performers  maintain  a  posture  of  compliance   § Fewest  number  of  repeat  audit  findings   § One-­‐third  amount  of  audit  prepara0on  effort   §  High  performers  find  and  fix  security  breaches  faster   § 5  /mes  more  likely  to  detect  breaches  by  automated  control   § 5  /mes  less  likely  to  have  breaches  result  in  a  loss  event   §  When  high  performers  implement  changes…   § 14  /mes  more  changes   § One-­‐half  the  change  failure  rate     § One-­‐quarter  the  first  fix  failure  rate     § 10x  faster  MTTR  for  Sev  1  outages   §  When  high  performers  manage  IT  resources…   § One-­‐third  the  amount  of  unplanned  work   § 8  /mes  more  projects  and  IT  services   § 6  /mes  more  applica0ons     Source: IT Process Institute, 2008
  9. 9. Visible Ops: Playbook of High Performers • The IT Process Institute has been studying high- performing organizations since 1999 • What is common to all the high performers? • What is different between them and average and low performers? • How did they become great?
  10. 10. The Story Begins… Picture credit: Flickr: Floralgal
  11. 11. I Know! 12  
  12. 12. Lesson: You are only as smart as the average of the top 5 people you hang out with 13  
  13. 13. 2012 DevOps Survey •  Written by Puppet Labs, Gene Kim, and Jez Humble •  Ran through December of 2012 •  Goal: Get a snapshot of the current state of the health and habits of the DevOps community
  14. 14. How To Do A Cross Section Population Study • Survey  design   • Survey  execu0on   • Survey  analysis  
  15. 15. The Vision: DevOps Survey Of Practice II • Performance   • Behaviors   • Demographics   • Psychographics   • Planned  target  was  30  ques0ons   • We  asked  for  a  budget  of  8  ques0ons   16  
  16. 16. Survey Design In  the  ideal,  we  want  to  be  able  to  say,  “the  more  you  do  [A,  B,   C],  the  more  performance  improves”   X axis: independent variables DevOps behaviors hypothesized to improve performance Y axis: dependent variables DevOps performance measures
  17. 17. Y-Axis: What Performance Looks Like • How  ofen  do  you  deploy  code?  (e.g.,  daily,   weekly,  monthly,  yearly)   • What  is  the  average  lead  0me  of  a  change?    (i.e.,   how  long  does  it  take  to  go  from  “code   commihed”  to  “code  successfully  running  in   produc0on?”)     • What  percentage  of  your  changes  require   rollbacks  or  hoiixes?   • On  average,  how  long  does  it  take  to  restore   service  when  something  goes  wrong?  (e.g.,  a  day,   a  week,  a  month,  a  year)   18  
  18. 18. X-Axis: What Behaviors Result In Performance • Are  environment  and  infrastructure  changes   (i.e.,  everything  except  for  code)  checked  into   revision  control?   • Is  there  an  automated  process  to  deploy   environment  and  infrastructure  changes?   • Who  performs  code  deployments?    (e.g.,  Dev,   Ops,  both)   • Who  is  on  the  hook  for  produc0on  support   (e.g.,  Dev,  Ops,  both)   19  
  19. 19. Next Step: Survey Execution •  Once  the  survey   instrument  is  ready,  then   you  need  people  to  take   the  survey   •  Michelle  Carroll  and  the   Puppet  Labs  marke0ng   machine  went  to  work   • Mailing  lists   • Twiher   • Reddit  
  20. 20. 21  
  21. 21. The Results • 4039  completed  survey  responses  in  30   days!!!   • This  is  amazing.    With  a  primary  research  firm,   this  type  of  popula0on  size  usually  would  cost   $100-­‐200K!   22  
  22. 22. Lesson: Do Not Change Answers Mid-Survey
  23. 23. Uh, Oh… Now what?
  24. 24. Lesson: Get data analyst on the team BEFORE you do survey design! 25  
  25. 25. Where Do You Find Your Own Wally? • Steal  a  consultant  ;)   • R  User  Groups   • University  students  in  search  of  a  thesis   • Economics  students  a  plus  (sta0s0cs,  applied)   • Ex-­‐Wall  Street  investors  (not  recommended)   Wally Zabaglio, Puppet Labs Data Analyst –  Former specialized analytics consultant at The Nielsen Company; analyst for the US Department of Energy's Office of Environmental Management –  Primary analyst on DevOps survey results
  26. 26. 27  
  27. 27. Data Cleanup • Used  Google  Refine  to  clean  up  data  and   compensate  for  “Gene  screwup”   • Used  R  to  do  analysis   • Look  for  signal   • Explore  data   • Run  correla0ons  
  28. 28. Key Findings
  29. 29. High Performing DevOps Teams • They’re  more  agile   • 30x  more  frequent  deployments   • 8,000x  shorter  lead  0me     (minutes/hours  vs.  months/quarters)   • They’re  more  reliable   • 2x  the  change  success  rate   • 12x  faster  MTTR  
  30. 30. 31 Technical Practices Matter •  89% are using infrastructure version control •  82% are using automated code deployments
  31. 31. Deliberate Practice Matters Organizations that implemented DevOps practices over 12 months ago were 5x more likely to be high performing than organizations that weren’t implementing DevOps at all.
  32. 32. The Lost Hypotheses * o  Who performs code deployments? o  Who is on the hook for production support?
  33. 33. Measuring Culture “I’ll tell you EXACTLY what devops means. Devops means giving a shit about your job enough to not pass the buck. Devops means giving a shit about your job enough to want to learn all the parts and not just your little world. Developers need to understand infrastructure. Operations people need to understand code. People need to fucking work with each other and not just occupy space next to each other.” John Vincent | @lusis |
  34. 34. Measuring Culture Trust (and Verify) Continuous Improvement vs Risk Management Theater “Human Error” Heroes / "high performers"
  35. 35. The tribe is operationally strong *     70%  Opera0ons  people      
  36. 36. The tribe is operationally strong     This  is  not  a  Dev  takeover      
  37. 37. The tribe is enterprise too!   26%  -­‐  500  to  9999   16%  -­‐  10K  plus      
  38. 38. Overcoming barriers to DevOps adoption Biggest barriers were cultural •  Lack of manager buy-in •  Lack of team buy-in •  Value of DevOps not understood outside my group
  39. 39. What are organizations looking for in DevOps skills? •  Coding/scripting (84%) •  People skills (60%) •  Process re-engineering skills (56%) •  Experience with specific tools (19%)
  40. 40. Challenges to culture   It's  easier  to  teach  tools   than  teach  people  skills      
  41. 41. Preparing For DevOps Survey 2013-2014 • How  do  we  measure  culture?   • High  management  vs.  low  trust  management   styles  (and  link  to  team  size)   • Industry  code   • Beher  define  performance  variables  to  enable   regression  analysis  (Likert-­‐type  scale)   • Business  cri0cality:  system  of  engagement  vs.   system  of  record   • What  else?    We  want  to  hear  your  thoughts!  
  42. 42. If you … Then … … want to learn more about the findings •  Read the full-length report: •  Share the infographic: infographic •  Get these slides: DevOpsSurveyOfPractice … have ideas on the DevOps Survey Of Practice 2013-2014 •  Come talk with us! •  Join our Google+ Community: 104363243715545285555 … share your stories of transformation •  Go to FlowCon! … get a free copy of The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win •  Go to the AppFirst booth at 2:40pm. Gene will be signing books!