Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Metrics to Power DevOps

4,947 views

Published on

Tech Mahindra and CollabNet have worked together on a number of mission-critical projects, and over the course of their partnership have developed unique expertise in lifecycle, development-to-production metrics. Gain an understanding not only of what metrics are important, but also practical approaches to building reports and dashboards that deliver a single-pane view of all your delivery pipelines across the enterprise.

Participants will learn:

KPI’s of end-to-end dashboard driven development and delivery
Best practices for metrics in Agile / DevOps environments
Role of technology frameworks for integrated planning and reporting

Published in: Technology
  • Be the first to comment

Metrics to Power DevOps

  1. 1. 1 Copyright ©2014 CollabNet, Inc. All Rights Reserved. Metrics to Power DevOps Toward making decisions based on data – not opinion Sujoy Sen Sr. Director, Field Engineering CollabNet, Inc. Kevin Hancock Vice President Practice Head – IT Transformation Consulting
  2. 2. 2 Copyright ©2014 CollabNet, Inc. All Rights Reserved. Kevin Hancock, Senior Director Worldwide Field Operations In his ten plus year at CollabNet, Kevin has led successful implementations of CollabNet products and services at some of the world’s largest enterprises. Kevin works with organizations to transform their globally distributed application development teams into modern, agile organizations, helping to save millions of dollars while accelerating software delivery. Kevin holds a Bachelor of Science degree from Carnegie Mellon University, and a Masters degree from the University of Pittsburgh. Sujoy Sen, Practice Head – IT Transformation Consulting Sujoy is part of the leadership team in the Tech Advisory group of TechMahindra. He is heading a practice which helps customers improve the success rate of their critical IT projects. Sujoy is a Certified Six Sigma Black Belt & Manager of Organizational Excellence from American Society for Quality, a PMP, a CISA and an Agile Trainer & Coach. Sujoy has more than 19 years of experience across various domains and functions. He has been part of Tech Mahindra’s IT & Quality consulting group for the last 10 years. Today’s Presenters
  3. 3. 3 Copyright ©2014 CollabNet, Inc. All Rights Reserved. • Where are we now? • What’s the problem? • What do we need? • How do we get there? Agenda
  4. 4. 4 Copyright ©2014 CollabNet, Inc. All Rights Reserved.4 Copyright ©2014 CollabNet, Inc. All Rights Reserved. Where are we now?
  5. 5. 5 Copyright ©2014 CollabNet, Inc. All Rights Reserved. IT Priorities are changing McKinsey Global Survey, 2014 • IT is expected to improve business process effectiveness • IT spending on innovation is expected to increase by 50% and spending on transaction activities is expected to go down by 15% • Further current spending on next gen infrastructure is at it highest in 5 years • To address the concerns and make effectives use of additional infrastructure spending, IT should be more agile and responsive resulting in more innovation • To enable innovation, a culture of experimentation where IT is allowed to test new innovations quickly with customer, capture feedback and continuously improve • This requires the ability of continuous delivery resulting in better, faster and cheaper innovation The focus has shifted from reducing cost to improving the effectiveness of business processes
  6. 6. 6 Copyright ©2014 CollabNet, Inc. All Rights Reserved. Agile Upstream 32%* Source: February 5, 2014, “How Can You Scale Your Agile Adoption?” Forrester report *Do standup, burndown/burnup, and short iterations; product owner; and constant feedback. Agile upstream • Agile Upstream are those practices usually associated with planning, managing and executing a Software project. • Scrum, Kanban, XP are all used upstream in the process. • But what about Downstream?
  7. 7. 7 Copyright ©2014 CollabNet, Inc. All Rights Reserved. 13%† Source: February 5, 2014, “How Can You Scale Your Agile Adoption?” Forrester report †Do upstream plus continuous integration, continuous delivery, and test-driven development. Agile downstream • Agile Downstream refers to Test automation (Test-driven Development). Continuous Integration, Continuous Delivery. • Test automation is a key part of any Continuous Integration process • Successful Continuous Integration requires Build Management • Successful Continuous Delivery requires Release Management
  8. 8. 8 Copyright ©2014 CollabNet, Inc. All Rights Reserved. • Agile Software Development • Continuous Integration • Continuous Delivery • Visible and accessible Metrics • Access for ALL Stakeholders (RBAC permitting) We need an Agile System
  9. 9. 9 Copyright ©2014 CollabNet, Inc. All Rights Reserved. Few Companies Have Achieved Enterprise Agility Plan (Upstream) Deploy (Downstream) WorkgroupEnterprise Perform Upstream Agile: standup, burndown/burnup, and short iterations; product owner; and constant feedback 13% <20% A few % 32% Perform Upstream + Downstream Agile: Perform upstream plus continuous integration, continuous delivery, and test-driven development Have Scaled Agile across their organization Have achieved Enterprise Agility
  10. 10. 10 Copyright ©2014 CollabNet, Inc. All Rights Reserved.10 Copyright ©2014 CollabNet, Inc. All Rights Reserved. What’s the Problem?
  11. 11. 11 Copyright ©2014 CollabNet, Inc. All Rights Reserved. • Highly optimized silos or work centers make sense for Waterfall – Trust each work center to build the most “perfect product” to pass to the next work center – We know this rarely works • Agile Enterprises optimize for the System – Identify and fix bottlenecks – Only improve a work center if it improves the system Silos of Data and Automation
  12. 12. 12 Copyright ©2014 CollabNet, Inc. All Rights Reserved. • The first way: – Systems Thinking • The second way: – Amplify feedback loops • The third way: – Culture of continual experimentation and learning A model for DevOps Empowerment
  13. 13. 13 Copyright ©2014 CollabNet, Inc. All Rights Reserved. The First Way: Systems Thinking • Focus on value streams • Think globally • Understand the system as a whole • Stop problems early and often Operations (customer) Development (business) • Centralised source code and environment configuration. • Consistent and automated release processes. • Decreased cycle times, faster release cadence.
  14. 14. 14 Copyright ©2014 CollabNet, Inc. All Rights Reserved. • Protect the integrity of the entire system of work, versus completion of tasks • Expose visual data so everyone can see how their decisions affect the entire system The second way: Amplify Feedback Loops Operations (customer) Development (business) • Find and fix defects, performance and availability issues faster. • Enable specialised functions such as Ops and InfoSec to collaborate more effectively. • Increase communication across the entire business. • Get more work done
  15. 15. 15 Copyright ©2014 CollabNet, Inc. All Rights Reserved. • Continual experimentation • Tasking risks and learning from failure • Repetition and practice - pre-requisites to mastery • Create rituals that reward risk taking • Introduce faults to increase resilience The Third way: Culture of continual experimentation and learning Operations (customer) Development (business)
  16. 16. 16 Copyright ©2014 CollabNet, Inc. All Rights Reserved.16 Copyright ©2014 CollabNet, Inc. All Rights Reserved. What Do We Need?
  17. 17. 17 Copyright ©2014 CollabNet, Inc. All Rights Reserved. Key Metrics that Power DevOps 1. Change Volume 2. Lead Time (from dev to deploy) 3. Deployment Frequency 4. Percentage of Failed Deployments 5. Build Count By Day 6. File Churn 7. Build duration 8. Cycle time Traditional agile metrics have focused here Enterprise DevOps Metrics require consistency across projects, enterprise-wide visibility, and the ability to drill to project data and people Open ALM Platform
  18. 18. 18 Copyright ©2014 CollabNet, Inc. All Rights Reserved. • As an executive, I need to see across projects to determine the relative success of different teams. Portfolio Dashboards Any metric is more valuable when it is put into context, and compared with other teams.
  19. 19. 19 Copyright ©2014 CollabNet, Inc. All Rights Reserved. • How many user stories and new lines of code are being deployed? Change Volume The key is to understand the risk of a given deployment – more user stories and more lines of code adds risk.
  20. 20. 20 Copyright ©2014 CollabNet, Inc. All Rights Reserved. • How long does it take for a new requirement/story/task tp go from the time when it hits development to when it successfully gets deployed into production. Average Age (from development to deployment) Average age is an important indicator of efficiency in the process – when tracked using value stream mapping, it can help the team to visualize areas in the process which need improvement. Cycle time should reduce as the team gets a better hold of the lifecycle.
  21. 21. 21 Copyright ©2014 CollabNet, Inc. All Rights Reserved. • How often is the team deploying new code? Team Deployments For an Agile team delivering code in 2 or 4-week sprints, this metric should become predictable, and correlate closely, if not exactly to those sprints.
  22. 22. 22 Copyright ©2014 CollabNet, Inc. All Rights Reserved. • On average, how long are builds taking to complete? Build Duration This metrics should remain stable over time. Spikes in build duration should be red-flagged and investigated.
  23. 23. 23 Copyright ©2014 CollabNet, Inc. All Rights Reserved. • How is the team trending in terms of cycle time, over the lifecycle of a project? Cycle Time Technical debt will add to cycle time, but so will general complexity. Most projects start out strong, but over time degrade. Choose the level that is acceptable and seek stability at that level.
  24. 24. 24 Copyright ©2014 CollabNet, Inc. All Rights Reserved. Velocity and Estimation • How are teams doing? Will they finish what is promised? Velocity will let us know how much work a team can get done, if they are consistent in their estimation and improving; while the burnup will let us know if we are finishing work quicker than it is being added to the queue
  25. 25. 25 Copyright ©2014 CollabNet, Inc. All Rights Reserved.25 Copyright ©2014 CollabNet, Inc. All Rights Reserved. How Do We Get There? The Cloud Framework to Power DevOps
  26. 26. 26 Copyright ©2014 CollabNet, Inc. All Rights Reserved. Connect Teams, Processes, and Tools
  27. 27. 27 Copyright ©2014 CollabNet, Inc. All Rights Reserved. TechM ADOPT Platform with TeamForge Your data center Public clouds Dashboards Control center for delivery pipelines, with social activity streams and open pluggable API’s GitEye Code Search Code Review Git + Gerrit Jenkins Hudson Tracker Discussions Wiki Documents Reports File Releases JIRA Selenium PPM One Platform, Multiple Possibilities QTP Platform 4
  28. 28. 28 Copyright ©2014 CollabNet, Inc. All Rights Reserved. Tool Snapshot Project Dashboard Agile Backlogs Configuration Management Unit Test / Code quality tool integration Jenkins integration Collaboration Tools Code Review Tool Customized Reports Release Planning Defect Tracker Defect Tracker/Impediment Tracker Infrastructure Automation Traceability for application lifecycle activities Task Tracking / Managing Sprint Platform 4
  29. 29. 29 Copyright ©2014 CollabNet, Inc. All Rights Reserved. Scrum Team 2 Test (T) QA (Q) Scrum Team 1 Dev (D) T A Prod Prod (P) D Sprint Build? Build? Unit tests Unit tests Unit tests Build? Continuous Integration Continuous Testing Automate d Tests Automate d Tests Unit tests Continuous Deployment Continuous Release & Monitoring ContinuousPlanning Continuous Delivery Build (B) Daily Daily Daily/Release point • Devops should create a flow that results in faster through put time • Our approach ensures that we improve the overall throughput by bringing in process, tools, automation and training interventions where the flow gets stuck • TechM proposes a holistic approach to enable Devops in an organization by slowing eliminating legacy practices which don’t promote collaboration Focus on Cadence and Orchestration
  30. 30. 30 Copyright ©2014 CollabNet, Inc. All Rights Reserved. Simplified process for continuous delivery Task to Package Idea to task Package to Deployment Business vision Products Identification Product Roadmap Product Epics (product backlog) Release Plans Sprint Plans (Sprint backlog) Tasks Design Code Automated Unit test Automated Code quality check Trigger Integration Automated Build Package Check-in to repository Deploy to QA Env* Manual/ Automated SIT Testing Deploy to UAT Env* Manual / Automated UAT Testing Deploy to Production Env* S E S EStart End Automated monitoring in production I2T T2P P2D Processes 1 Processes 1
  31. 31. 31 Copyright ©2014 CollabNet, Inc. All Rights Reserved. Phased Approach Isolated Teams SME Shared Teams Continuous Planning Continuous Release and Monitoring Continuous Integration Continuous Testing Continuous Delivery Product Teams Multi Layered Teams Multidisciplinary Project Teams Demand pipeline established; Release plans are developed and maintained for individual applications; Projects are able to establish traceability across requirements, design, development, testing and defect management is maintained through out the lifecycle Projects are able demonstrate the behavior of continuous feedback; as opposed to a periodic schedule, builds are triggered by every commit to the single code repository Project is able to align itself with enterprise release cadence; environments are provisioned ‘on demand’; releases deployed across multiple environments including data migration Continuous regression reduces the number of defects; most integrated bugs are fixed regularly to avoid last minute changes; test cases reused; ‘non-functional’ code quality measured and technical debt improved Dev and ops team work together seamlessly delivering both projects and managing operations. Metrics are captured to monitor process performance and continuous improvement. Level 1 Level 2 Level 3 Level 4 Level 5 VALUE COLLABORATION Capability Model 2 • Our assessment framework is built on this staged approach. Staged approach does necessarily mean that it is a sequence • TechM would evaluate organization as-is status against these stages • As-is assessment would cover people, process, tools, policies, KPIs and flow • As an outcome of the assessment we would provide the gaps and the transformation roadmap to cover these gaps • The Devops consultant/ Program manager would work towards implementing this transformation roadmap
  32. 32. 32 Copyright ©2014 CollabNet, Inc. All Rights Reserved. Transformation Phases
  33. 33. 33 Copyright ©2014 CollabNet, Inc. All Rights Reserved. iSPEED tm Methodology Study Plan Enable and Expand Diffuse • Review historical data, documents, tools and processes • Understand the organization structure • Interview key stakeholders using diagnostic tool • Prepare business case • Identify multiple solutions blueprints • Prepare transformation roadmap • Identify Enterprise Change Enablement team • Develop workflows as per selected blueprint • Review the workflows with all stakeholders • Tune up workflows • Develop detailed process, templates, checklists, guidelines • Modify tool workflows • Facilitate deployment planning workshops, prioritize process activities , prepare deployment plans • Enable teams by training and support • Identify projects and deploy • Aggregate data • Enable Continuous Improvement • Quantify benefits and Showcase projects • Collate and analyze performance data • Benchmark and improve • Repot • Establish improvement log • Continue deployment support Interview • Interview key customer stakeholders • Understand customer context • Abstract key business drivers • Enlist the top priorities 2 to 3 WKS 4 to 6 WKS 8 to 12 WKS OngoingMax 4hrs Feasibility Decision • Business case • Solution blueprint • Transformation roadmap • Process Workflows • Processes, templates, checklist • Trainings documents • Project data • Audit & Management reports • Management reports • Improvement log • Fixed Price • Fixed Price • Time and Hire • Outcome based pricing • Fixed Price • Time and Hire • Outcome based pricing • Time and Hire Deliverables Engagement model
  34. 34. 34 Copyright ©2014 CollabNet, Inc. All Rights Reserved. DevOps Transformation Assessment (DOT) ADCM provides a structured and tested approach for any organizations Devops strategy. Outcome of the assessment provides a roadmap that can be used for Devops strategy and rollout
  35. 35. 35 Copyright ©2014 CollabNet, Inc. All Rights Reserved. DOT Analytics 0 0.5 1 1.5 2 2.5 3 3.5 4 Business IT Alignment Portfolio planning Release planning Capability planning Capacity planning Infrastructure planning Agile delivery planning Task planning Value addition Collaboration Institutionalization • Each stage has multiple focus areas. The depiction here is for continuous planning • To arrive at the score, TechM consultants would speak to various stakeholders and take them through a series of questions • Each focus area is evaluates for • the value addition that it is currently bringing, • The collaboration it facilitates • Level of implementation that has been achieved • Focus areas where today nothing exists is given a score of zero • Further for each area a detailed improvement plan is prepared Continuous planning
  36. 36. 36 Copyright ©2014 CollabNet, Inc. All Rights Reserved. DOT Roadmap Sample Continuous planning W1 W2 W3 W4 W5 W6 W7 Business IT alignment People Process Tooling Flow & integration Performance & outcomes Deliverables Portfolio Planning People Process Tooling Flow & integration Performance & outcomes Deliverables Release Planning People Process Tooling Flow & integration Performance & outcomes Deliverables Capability Planning People Process Tooling Flow & integration Performance & outcomes Deliverables Capacity Planning People Process Tooling Flow & integration Performance & outcomes Deliverables Infrastructure planning People Process Tooling Flow & integration Performance & outcomes Deliverables Agile delivery planning People Process Tooling Flow & integration Performance & outcomes Deliverables Task planning People Process Tooling Flow & integration Performance & outcomes Deliverables
  37. 37. 37 Copyright ©2014 CollabNet, Inc. All Rights Reserved.37 Copyright ©2014 CollabNet, Inc. All Rights Reserved. Thank You Contact Details: Kevin Hancock - CollabNet kevin@collab.net Sujoy Sen – Tech Mahindra Sujoy.Sen@TechMahindra.com
  38. 38. 38 Copyright ©2014 CollabNet, Inc. All Rights Reserved.38 Copyright ©2014 CollabNet, Inc. All Rights Reserved. © 2014 CollabNet, Inc., All rights reserved. CollabNet is a trademark or registered trademark of CollabNet Inc., in the US and other countries. All other trademarks, brand names, or product names belong to their respective holders. CollabNet, Inc. 8000 Marina Blvd., Suite 600 Brisbane, CA 94005 www.collab.net +1-650-228-2500 +1-888-778-9793 blogs.collab.net twitter.com/collabnet www.facebook.com/collabnet www.linkedin.com/company/collabnet-inc

×