Devops Metrics

DEVOPS METRICS ANDREW SHAFER - PATRICK DEBOIS - ISRAEL GATThursday 16 June 2011

Patrick Debois Jedi @patrickdebois http://jedi.be/blog http://www.vectronicsappleworld.com/collection/articlepics/joysticks/snap23.jpgThursday 16 June 2011

Israel Gat Cutter Consortium @agileexec http://theagileexecutive.comThursday 16 June 2011

Andrew Shafer - Rabble Rouser - Cloudscaling @littleidea - http://stochasticresonance.wordpress.comThursday 16 June 2011

http://www.ﬂickr.com/photos/quinnanya/3588989219Thursday 16 June 2011

http://www.ﬂickr.com/photos/zen/3233023066/Thursday 16 June 2011

To Compare http://www.ﬂickr.com/photos/thebusybrain/2492945625/Thursday 16 June 2011

http://www.ﬂickr.com/photos/topshampatti/219953012/Thursday 16 June 2011

To UnderstandThursday 16 June 2011

To ProtectThursday 16 June 2011

To Improve http://www.ﬂickr.com/photos/vizzzual-dot-com/2655969483/Thursday 16 June 2011

Describing a Metric http://www.ﬂickr.com/photos/chandramarsono/4324373384/Thursday 16 June 2011

Describing a Metric •basis of measurement http://www.ﬂickr.com/photos/chandramarsono/4324373384/Thursday 16 June 2011

Describing a Metric •basis of measurement •assumptions http://www.ﬂickr.com/photos/chandramarsono/4324373384/Thursday 16 June 2011

Describing a Metric •basis of measurement •assumptions •level and usage http://www.ﬂickr.com/photos/chandramarsono/4324373384/Thursday 16 June 2011

Describing a Metric •basis of measurement •assumptions •level and usage •desired trend http://www.ﬂickr.com/photos/chandramarsono/4324373384/Thursday 16 June 2011

Describing a Metric •basis of measurement •assumptions •level and usage •desired trend •when to use http://www.ﬂickr.com/photos/chandramarsono/4324373384/Thursday 16 June 2011

Describing a Metric •basis of measurement •assumptions •level and usage •desired trend •when to use •when to stop http://www.ﬂickr.com/photos/chandramarsono/4324373384/Thursday 16 June 2011

Describing a Metric •basis of measurement •assumptions •level and usage •desired trend •when to use •when to stop •how to game http://www.ﬂickr.com/photos/chandramarsono/4324373384/Thursday 16 June 2011

Describing a Metric •basis of measurement •assumptions •level and usage •desired trend •when to use •when to stop •how to game •warnings http://www.ﬂickr.com/photos/chandramarsono/4324373384/Thursday 16 June 2011

Beware of Vanity Metrics UNACTIONABLE INACCESSIBLE UNAUDITABLE http://www.ﬂickr.com/photos/centralasian/5829549813/Thursday 16 June 2011

Red is good , Green is worthless • http://www.reliableplant.com/Read/21305/lean-metrics-red-is-good,-green-is-worthlessThursday 16 June 2011

Thursday 16 June 2011

“In truth, a good case could be made that if your knowledge is meagre and unsatisfactory, the last thing in the world you should do is make measurements; the chance is negligible that you will measure the right things accidentally.” George Miller http://stochasticresonance.wordpress.com/2008/08/03/miles-per-gallon/Thursday 16 June 2011

What we can learn from monitoring http://www.experimentalistsanonymous.com/stuff/paralleluniverse/images/2edmoneyshot.jpgThursday 16 June 2011

Level 0 - Individual ResourceThursday 16 June 2011

Availability Processes, Port,SSH Login Network, Memory, CPUThursday 16 June 2011

Capacity Processes, Port,SSH Login Network, Memory, CPUThursday 16 June 2011

Health Thresholds http://www.smiley-faces.org/wallpaper/smiley-face-wallpaper-001.jpg http://www.geocities.jp/nikonikocalendar/index_en.htmlThursday 16 June 2011

Metric versus Diagnostic Does it contribute to Yes Measurement the Metric bottom line? No Diagnostic Pete Behrens,Trail Ridge Consulting http://www.slideshare.net/krivitsky/agile-metrics-presentation-65425Thursday 16 June 2011

Level 1 - Services LevelThursday 16 June 2011

Performance SSH, HTTP, NTPThursday 16 June 2011

Performance SSH, HTTP, NTP # Servers/AdminThursday 16 June 2011

Performance SSH, HTTP, NTP # Servers/Admin # Line of Code/DeveloperThursday 16 June 2011

“Gameable” http://kithicor.org/wp/wp-content/uploads/2010/01/gamerEvolution.jpgThursday 16 June 2011

Level 2 - Team LevelThursday 16 June 2011

Aggregation LB-WEB WEB WEB WEB 1 2 3Thursday 16 June 2011

Aggregation DEV Project Team DEV DEV DEVThursday 16 June 2011

Burn up chartThursday 16 June 2011

Team UtilizationThursday 16 June 2011

“Velocity also has a direction component” http://www.randomstew.com/wp-content/uploads/mooneye.jpgThursday 16 June 2011

What about efﬁciency? http://www.ﬂickr.com/photos/jdiggans/92981578/Thursday 16 June 2011

http://sievertschreiber.ﬁles.wordpress.com/2009/12/good-code-is-measured-in-wtf-per-minute.jpgThursday 16 June 2011

Incentives & Behaviors http://www.ﬂickr.com/photos/turtlemadness/2439050717/Thursday 16 June 2011

Development Operations Level 3 - Integration Levelhttp://www.xgn.nl/images/upload/20081020225036.jpg http://www.kawaiikakkoiisugoi.com/wp-content/uploads/2011/03/pacman-screen.jpgThursday 16 June 2011

Integration WEB APP DBThursday 16 June 2011

Collaboration DEV QA OPS # of deploys / dayhttp://www.sage.org/lists/sage-members-archive/2001/msg01208.htmlhttp://www.datacenterknowledge.com/archives/2009/12/30/how-many-servers-can-one-admin-manage/http://fc04.deviantart.net/fs70/f/2010/108/2/3/Pac_Man_Widescreen_by_molotov_arts.jpgThursday 16 June 2011

Counting Collaboration More interaction doesn’t mean a better party • http://www.nature.com/news/2010/101013/full/news.2010.538.htmlThursday 16 June 2011

Measuring Collaboration? “Play Nice Study” http:///www.healthylarimer.org/documents/PlayNice.pptThursday 16 June 2011

Levels Collaboration Collaboration Coalition Coordination CooperationThursday 16 June 2011

Cooperation provide information somewhat deﬁned roles formal communication decisions made independentlyThursday 16 June 2011

•share information & resources •deﬁned roles •frequent communication Coordination •some shared decision makingThursday 16 June 2011

•share ideas •share resources Coalition •frequent & prioritized communication •all members have a vote on decisionsThursday 16 June 2011

Collaboration •members belong to one system •frequent communication characterized by mutual trust •consensus on all decisionsThursday 16 June 2011

Knowledge sharing http://www.au.af.mil/au/awc/awcgate/navy/onr_metrics_on_collab.pdfThursday 16 June 2011

Team Resilience 10th Floor Test Truck Factor to take a random machine in your infrastructure ,drop it from the 10th ﬂoor, and be able to restore your infrastructure to working within 5-10 minute The number of people on your team who have to be hit with a truck before the project is in serious trouble Resilience should be a 4th management objective, alongside Better/Faster/Cheaper John Allspaw http://www.kitchensoap.com/ Thursday 16 June 2011

Black Hole Theory Absorbs light, to see the effects you need to look at nearby objects http://www.collaborate.com/blog/2010/01/28/musing-metrics-collaborationThursday 16 June 2011

Quest for proof • Innovating and/or providing decision support • Saving time or Money (tangible but less than (tangible) quality) • Increase Quality • Easing access to and (tangible... but less so) interactions with subject- matter experts (intangible) http://communityandbusiness.groupsite.com/group/Collaboration20/post/measuring-the-value-of-collaboationThursday 16 June 2011

More vs Better Widgets Add more servers Tune existing servers Process Improvement Quality Improvement Easier More Difﬁcult!! http://socialconsulting.wordpress.com/2009/02/24/measuring-collaboration/Thursday 16 June 2011

Collaboration Challenge “Maximize Team Utility”Thursday 16 June 2011

Level 4 - Business LevelThursday 16 June 2011

AHA KA-CHING!Thursday 16 June 2011

Return on Investment http://www.ﬂickr.com/photos/hanekomu/4481674104/Thursday 16 June 2011

Inventory (WIP)Thursday 16 June 2011

Time to Market http://stefanroock.wordpress.com/2010/03/02/kanban-deﬁnition-of-lead-time-and-cycle-time/Thursday 16 June 2011

Mean Time Between Failure http://museumdosgames.blogspot.com/2010/04/game-over.htmlThursday 16 June 2011

Mean Time Between ResolutionThursday 16 June 2011

Technical DebtThursday 16 June 2011

LeaksThursday 16 June 2011

Debt Metaphor not all debt is created equal http://www.ﬂickr.com/photos/shoot-art/4536509717/Thursday 16 June 2011

Customer Responsiveness Jim HighsmithThursday 16 June 2011

Types of Tech debt “oops” “yeah I know” unintentional intentional “we’ll ﬁx it the next minor “we’ll ﬁx it the next major revision” revision” short term (tactical) long term (strategical) http://forums.construx.com/blogs/stevemcc/archive/2007/11/01/technical-debt-2.aspxThursday 16 June 2011

Gartner • http://www.castsoftware.com/resources/materials/analyst/Gartner-How_to_Monetize_Application_Technical_Debt.pdfThursday 16 June 2011

Vicious Cycle of The Vicious Cycle of TechnicalDebt Technical Debt (More) Relentless Pressure Take Diminished Technical Dev Velocity Debt Technical Fail to Pay Debt Debt Back Accrues Neglect Maintenance http://theagileexecutive.com/2010/09/20/how-to-break-the-vicious-cycle-of-technical-debt/ Israel Gat 1Thursday 16 June 2011

Symptoms - Measurable • Poor Customer Responsiveness • Long Delivery Times • Late deliveries • Lots of defects (Toxic Code) • Rising Development costs • Frustrated & Poor performing Teams http://www.theagileengineer.com/public/Home/Home_ﬁles/TechnicalDebt_published.pdfThursday 16 June 2011

Secret Sauce http://radar.oreilly.com/archives/2007/10/Ops-ROI-jesserobbins.htmlThursday 16 June 2011

The Team Knows http://kingsinsight.com/2010/07/31/ask-the-team-about-technical-debt-the-creeping-doom-graph/Thursday 16 June 2011

Failure Value Demand % Demand As a Team http://farm4.static.ﬂickr.com/3167/2769773425_16c235dc28.jpgThursday 16 June 2011

Level 5 - End User LevelThursday 16 June 2011

End User Perspective • END to END monitoring • User Experience (browser testing) • External parties (Email, ISP, Plugins)Thursday 16 June 2011

Participatory Design http://www.informaworld.com/smpp/content~db=all~content=a790786652Thursday 16 June 2011

“Crowd” - Monitoring Sudden Change in User Activity Pattern Detection of problem? A/B TestingThursday 16 June 2011

Spend more time Automation => creatively , early in the process Traditional Engineering http://www.informaworld.com/smpp/content~db=all~content=a790786652Thursday 16 June 2011

Design Strategies To Design is Human To Design does not require super powers Design used to be a collective or shared ability http://www.amazon.com/Research-Design-Thinking-Nigel-Cross/dp/9062757960 p 4 http://www.psych.lancs.ac.uk/people/lindenball.htmlThursday 16 June 2011

Open Innovation Wikipedia Open Source SharedConﬁg Management Shared Infrastructures http://en.wikipedia.org/wiki/Open_innovation http://www.clickadvisor.com/downloads/Bughin_Next_Step_in_Open_Innovation.pdfThursday 16 June 2011

Novice vs Expert • depth ﬁrst • breath ﬁrst , depth dives • vertical oriented • horizontal oriented • can not deal with • uncertainty = options uncertainty Linden Ballhttp://www.amazon.com/Research-Design-Thinking-Nigel-Cross/dp/9062757960 p 147 http://www.psych.lancs.ac.uk/people/lindenball.htmlThursday 16 June 2011

Continuous Evolution “Problem space evolves with the solution” http://www.amazon.com/Research-Design-Thinking-Nigel-Cross/dp/9062757960 http://www.psych.lancs.ac.uk/people/lindenball.htmlThursday 16 June 2011

Cultural TaxonomyThursday 16 June 2011

tl;dr • avoid vanity metrics • insight > data • don`t hate the player • hate the game • business value good • beware of technical debtThursday 16 June 2011

Questions?Thursday 16 June 2011

Thursday 16 June 2011

Thank you!Thursday 16 June 2011

http://www.jedi.be	2032
http://www.planetdevops.net	670
http://jedi.be	353
http://agile.dzone.com	163
http://localhost:4331	61
http://feeds.feedburner.com	54
http://zicht.nl	16
http://fivas2.etp5.etp.no	9
http://translate.googleusercontent.com	8
http://us-w1.rockmelt.com	6
http://www.linkedin.com	3
http://ossos.a2u.com.br	3
http://lef99.jedi.be	2
http://webcache.googleusercontent.com	2
http://a0.twimg.com	2
http://reviewbacktest	2
http://www.onlydoo.com	2
http://jirapame.wordpress.com	1
http://development.interact-intranet.com	1
http://www.dzone.com	1
http://www.rackspacecloud.com	1
http://laz.jedi.be	1

Devops Metrics

by Patrick Debois on Jun 16, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

22 Embeds 3,393

Statistics

Devops Metrics — Presentation Transcript