2013 Velocity DevOps Metrics -- It's Not Just For WebOps Any More!
DevOps: It’s Not Just For WebOpsAnd We Have The Metrics To Prove ItGene Kim (IT Revolu0on Press) Jez Humble (ThoughtWorks) James Turnbull (Puppet Labs) John Willis (Enstra0us / Dell)
PresentersJames Turnbull, VP of Community, Evangelism, and BusinessDevelopment• Prominent voice in DevOps community• Author of several technical books including The LogStashBookGene Kim, IT researcher and author• Founder and former CTO of Tripwire for 13 years• Author "The Phoenix Project: A Novel About IT, DevOps, andHelping Your Business Win” and “Visible Ops Handbook”Jez Humble, Principal, ThoughtWorks• Co-author of Continuous Delivery• Loud mouth on effective practices for software delivery
The Big Why?• Desire to understand the health and habits of DevOps community • Understand the link between behaviors and performance • Mi0gate the “echo chamber” eﬀect: what is the evidence of improved performance • Prove that DevOps isn’t just for unicorns… it’s for the horses, too.
Goals For This Presentation• Show you how to do cross-‐sec0on popula0on studies • Survey Design • Survey Execu0on • Survey Analysis • Share our top ﬁndings • Get your help for our 2013-‐2014 study
Intuitive AnalyticalAcademicStudyPopulationSamplePopulationSampleAcademicStudyPopulationSampleWhat Is Cross-Section Population StudyPopulationSampleAcademicStudyPopulationStudyBenchmarkROI StoryBestPracticePeerRecommendationAnecdotePrior ExperienceWe AreHereVendorsConsultantsIndustryAnalystTradeshowsIndustryPressBlogshere
High Performing IT Organizations§ High performers maintain a posture of compliance § Fewest number of repeat audit ﬁndings § One-‐third amount of audit prepara0on eﬀort § High performers ﬁnd and ﬁx security breaches faster § 5 /mes more likely to detect breaches by automated control § 5 /mes less likely to have breaches result in a loss event § When high performers implement changes… § 14 /mes more changes § One-‐half the change failure rate § One-‐quarter the ﬁrst ﬁx failure rate § 10x faster MTTR for Sev 1 outages § When high performers manage IT resources… § One-‐third the amount of unplanned work § 8 /mes more projects and IT services § 6 /mes more applica0ons Source: IT Process Institute, 2008
Visible Ops: Playbook of High Performers• The IT Process Institutehas been studying high-performing organizationssince 1999• What is common to all thehigh performers?• What is different betweenthem and average and lowperformers?• How did they become great?www.ITPI.org
The Story Begins…Picture credit: Flickr: Floralgal
Lesson:You are only as smart as theaverageof the top 5 people you hang out with13
2012 DevOps Survey• Written by Puppet Labs, Gene Kim, and Jez Humble• Ran through December of 2012• Goal: Get a snapshot of the current state of the healthand habits of the DevOps community
How To Do A Cross Section Population Study• Survey design • Survey execu0on • Survey analysis
The Vision: DevOps Survey Of Practice II• Performance • Behaviors • Demographics • Psychographics • Planned target was 30 ques0ons • We asked for a budget of 8 ques0ons 16
Survey DesignIn the ideal, we want to be able to say, “the more you do [A, B, C], the more performance improves” X axis: independent variablesDevOps behaviors hypothesized to improve performanceY axis: dependent variablesDevOps performance measures
Y-Axis: What Performance Looks Like• How ofen do you deploy code? (e.g., daily, weekly, monthly, yearly) • What is the average lead 0me of a change? (i.e., how long does it take to go from “code commihed” to “code successfully running in produc0on?”) • What percentage of your changes require rollbacks or hoiixes? • On average, how long does it take to restore service when something goes wrong? (e.g., a day, a week, a month, a year) 18
X-Axis: What Behaviors Result In Performance• Are environment and infrastructure changes (i.e., everything except for code) checked into revision control? • Is there an automated process to deploy environment and infrastructure changes? • Who performs code deployments? (e.g., Dev, Ops, both) • Who is on the hook for produc0on support (e.g., Dev, Ops, both) 19
Next Step: Survey Execution• Once the survey instrument is ready, then you need people to take the survey • Michelle Carroll and the Puppet Labs marke0ng machine went to work • Mailing lists • Twiher • Reddit
Lesson:Get data analyst on the team BEFORE youdo survey design!25
Where Do You Find Your Own Wally?• Steal a consultant ;) • R User Groups • University students in search of a thesis • Economics students a plus (sta0s0cs, applied) • Ex-‐Wall Street investors (not recommended) Wally Zabaglio, Puppet Labs Data Analyst– Former specialized analytics consultant at The NielsenCompany; analyst for the US Department of EnergysOffice of Environmental Management– Primary analyst on DevOps survey results
High Performing DevOps Teams• They’re more agile • 30x more frequent deployments • 8,000x shorter lead 0me (minutes/hours vs. months/quarters) • They’re more reliable • 2x the change success rate • 12x faster MTTR
31Technical Practices Matter• 89% are using infrastructure version control• 82% are using automated code deployments
Deliberate Practice MattersOrganizations that implemented DevOps practices over 12months ago were 5x more likely to be high performing thanorganizations that weren’t implementing DevOps at all.
The Lost Hypotheses *o Who performs code deployments?o Who is on the hook for production support?
Measuring Culture“I’ll tell you EXACTLY what devops means.Devops means giving a shit about your job enough to not passthe buck. Devops means giving a shit about your job enough towant to learn all the parts and not just your little world.Developers need to understand infrastructure. Operationspeople need to understand code. People need to fucking workwith each other and not just occupy space next to each other.”John Vincent | @lusis | http://bit.ly/12DkRhf
The tribe is operationally strong * 70% Opera0ons people
The tribe is operationally strong This is not a Dev takeover
The tribe is enterprise too! 26% -‐ 500 to 9999 16% -‐ 10K plus
Overcoming barriers to DevOps adoptionBiggest barriers were cultural• Lack of manager buy-in• Lack of team buy-in• Value of DevOps not understood outside mygroup
What are organizations looking for in DevOps skills?• Coding/scripting (84%)• People skills (60%)• Process re-engineering skills (56%)• Experience with speciﬁc tools (19%)
Challenges to culture Its easier to teach tools than teach people skills
Preparing For DevOps Survey 2013-2014• How do we measure culture? • High management vs. low trust management styles (and link to team size) • Industry code • Beher deﬁne performance variables to enable regression analysis (Likert-‐type scale) • Business cri0cality: system of engagement vs. system of record • What else? We want to hear your thoughts!
If you … Then …… want to learn more about the findings • Read the full-length report:http://puppetlabs.com/devops• Share the infographic:http://puppetlabs.com/2013-state-of-devops-infographic• Get these slides:http://slideshare.net/realgenekim/DevOpsSurveyOfPractice… have ideas on the DevOps Survey OfPractice 2013-2014• Come talk with us!• Join our Google+ Community:https://plus.google.com/communities/104363243715545285555… share your stories of transformation • Go to FlowCon! http://flowcon.org… get a free copy of The Phoenix Project:A Novel About IT, DevOps, and HelpingYour Business Win• Go to the AppFirst booth at 2:40pm. Genewill be signing books!