Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Evidence Based Management - Measuring value to enable improvement and agility

58 views

Published on

by Mia Horrigan

Organisations invest in agile processes, tools, training, and coaching, but how much are they getting back?

Has product delivery improved?

How much happier are users and the business customers?

Are employees empowered and enabled?

Traditional metrics might give you insight into improvements of operational efficiency, but the real conversation is about the value created for your organisation by the improved processes. Without measuring value, the success of any agile initiative is based on nothing more than intuition and assumption.

Mia will discuss Evidence based management and how this empirical process can help agile transformations measure and manage the value derived from the transformation initiative. Mia will focus on the 4 Key Value Areas: Current Value, Ability to Innovate, Unrealised Value and time to market, and how these contribute to an organisation’s ability to deliver business value.

Published in: Business
  • Be the first to comment

Evidence Based Management - Measuring value to enable improvement and agility

  1. 1. Evidenced Based Management (EBM) for Business Agility Scrum Australia 24 October 2018 Mia Horrigan Partner ZXM and Enterprise Agile Coach page 01 Mia Horrigan@zenexmachina.com Blog – www.zenexmachina.wordpress.com Twitter - @miahorri www.zenexmachina.com
  2. 2. page 02 Has our investment in Agile been worth it? 2 Organisations invest in agile processes, tools, training, and coaching, but how much are they getting back? Has product delivery improved? How much happier are users and the business customers? Are employees empowered and enabled?
  3. 3. page 03 Successful Project ? Success = • All requirements delivered… • By agreed-upon date… • For an agreed- upon cost “On track” = project follows plan, hits milestones
  4. 4. page 04 Traditional MeasuresActivity: reflect actions taken e.g. No. of meetings, No. of projects, No. people in team, or cost per period Output: reflect things that were produced, No. of releases, No. of defects fixed, No. of features delivered, or even velocity Using progress as main guide can be misleading and misalign business and delivery organisations Activity Output
  5. 5. page 05 Waterfall Triangle Traditional Iron Triangle assumes Minimal change “Plan — Do” Problem: focus on activity and output, not outcomes and value Measuring activity just tells where time was spent, not whether value was produced
  6. 6. page 06 Agile TriangleAgile Triangle enables inevitable changes “Envision — evolve” Value is the reason why we do things; activity and output do not tell you whether what you are doing is worthwhile or successful
  7. 7. page 07 How is Value Measured?Success = value is maximised Quality and capability are sustainable Focus on outcomes, not activity and output Focus on Business Value, improvements and helps accountability Only 20% Useful 64% Rarely or Never Used Chaos Report 2015 Standish Group
  8. 8. page 08 Agile Measures Outcome: measures that reflect the change to customers or users. Impact: Affect on the company itself Be value maximisers What value did we deliver and what was the behaviour change or increased outcome? Outcome Impact
  9. 9. page 09 Evidenced Based Management (EBM) Key Value Areas (KVAs) Market Value (Customer Value) Organisational Capability EBM is an empirical approach that measures value delivered as evidence of organisational agility Gives organisations the ability to measure the value they deliver to customers and the means by which they deliver that value, and to use those measures to guide improvements in both Unrealised Value (UV) Current Value (CV) Ability to Innovate (A2I) Time to Market (T2M) Agility Business Value Source Evidenced Based Management Guide https://www.scrum.org/resources/evidence-based-management
  10. 10. page 010 Use of a practice’s effectiveness does mean we are delivering value 10 Organisations adopting agile product delivery practices can easily lose sight of their real goal of improving the value, by focusing on improving activities and outputs instead of business outcomes Agile is a means to an end, not the end itself The whole point of adopting agile practices is to improve business performance When organisations lose sight of this, managers ask questions that seem sensible, but might create unintended and undesirable consequences Is build automation present? Are code standards being met? Are test first practices being used? Is the team velocity increasing? Are developers integrating code frequently? How frequently?What is the quality of the code? Source Evidenced Based Management Guide https://www.scrum.org/resources/evidence-based-management
  11. 11. page 011 What we Measure is Important What you measure translates into what we think is important Typically teams are used to metrics being used against them Velocity is data to help aid team’s own forecasting not a measure to compare teams Counting lines of code is easy but doesn’t tell anything about quality, the functionality provided or even the effectiveness Simply observing a situation necessarily changes that situation Human tendency to look for answers where it’s easy to look
  12. 12. page 012 Measurement is Strategic Without measuring value, the success of any agile initiative is based on nothing more than intuition and assumption Helps us know if we are moving closer to the goals of the organisation Understand what actually will be valuable that we’re not currently delivering
  13. 13. page 013 EBM Approach to Improving Value 13 Evidence Based Decision-Making is a process for making decisions about a program, practice, or policy based on Evidence and informed by experiential evidence EBM approach suggests four phases that enable organisations to constantly learn and improve the value derived from software investments 1. Measure KVMs (Key Value Metrics) 2. Select KVAs (Key Value Areas) to Improve 3. Conduct Experiments to Improve Value 4. Evaluate Outcome Results
  14. 14. page 014 If you had one wish: What is one measure/s you would want to improve? page 014
  15. 15. page 015 Current Value page 015
  16. 16. page 016 Reveals the value that the organisation delivers to customers, today Goal: to maximise the value an organisation delivers at the present time; it considers only what exists right now, not what it might do in the future Reveals organisation’s actual value in the marketplace but has no relevance on an organisation’s ability to sustain value in the future Current Value (CV)
  17. 17. page 017 How happy are our stakeholders?• How happy are customers today? Is their happiness improving or declining? • • How happy are employees? Is their happiness improving or declining? • How happy are investors and other stakeholders? Is their happiness improving or declining? Leading indicators Lagging indicators • Employee Satisfaction • Customer Satisfaction • Usage Index • Revenue per Employee • Product Cost Ratio
  18. 18. page 018 Personas • Who are our customers? • What is their story? • What are their goals, motivations and pain points? • Why do they want to work with us? • What outcomes do they want from our products? How would we measure that! Understand our Customers
  19. 19. page 019 Customers ValueWhat is the Organisations Value Proposition? Why do our customers work with us? What outcomes do they want from our products? How would we measure that!
  20. 20. page 020 Current Value and Goal Driven Prioritisation Use Impact Mapping to: • Be goal driven when coming up with and prioritising features • Prioritise features based on impacts • Show progress towards impacts. E.g. how many features are finished relate to each impact? Guides measurements of impacts being realised • Validate an existing list of features back to a goal to see if they are relevant
  21. 21. page 021 Visualise goal driven prioritisation for your customer Think Outcome rather than Deliverable Helps User Stories/PBIs to be outcome focused Modified Impact Map Goal – Persona – Impact – Outcome – PBI
  22. 22. page 022 Hypothesis Driven Delivery - Express Hypotheses Adapted from Gothelf and Seiden: Lean UX We believe [doing this] for [these people] will achieve [this outcome] We will know that this is true when we see [this measurement] changed Feature Persona Outcome Measure
  23. 23. page 023 Current Value – Key Value Measures KVM Measuring: Revenue per Employee The ratio (gross revenue / # of employees) is a key competitive indicatorwithin anindustry. Thisvaries significantlybyindustry. Product Cost Ratio Total expenses and costs for the product(s)/system(s) being measured, including operational costs compared to revenue. Employee Satisfaction Some form of sentiment analysis to help gauge employee engagement, energy, and enthusiasm. Customer Satisfaction Some form of sentiment analysis to help gauge customer engagement and happiness with the product. Usage Index Measurement of usage, by feature, to help infer the degree to which customers find the product useful and whether actual usage meets expectations on how long users should be taking with a feature.
  24. 24. page 024 Unrealised Value page 024
  25. 25. page 025 The potential future value that could be realised if the organization was able to perfectly meet the needs of all potential customers. Goal: to maximise the value that the organization realises from a product over time Unrealised Value (UV)
  26. 26. page 026 What potential is to be gained?Can any additional money be made in this market? Is it worth the effort and risk to pursue further returns in this market? Should further investments be made to capture additional Unrealised Value? Leading indicators Lagging indicators • Competitor strength/weakness • Customer acquisition or defection • Market share trends • Overall market growth/decline relative to market share trends
  27. 27. page 027 Balance of Current and Future Benefits Decision to invest in one product means not investing in others Considering both CV and UV provides a way to balance present and possible future benefits Current Value UnrealisedValue Early version used to test the market, but there is great market potential so investment warranted High HighLow Low Current “Cash Cow” with limited competitors may not warrant new investment unless reinventing
  28. 28. page 028 When you test a hypothesis, what is success? How long would it take for you to know whether you’ve actually achieved.. • the outcome you wanted to achieve, and • the impact you wanted to create?
  29. 29. page 029 Unrealised Value – Key Value Measures KVM Measuring: Market Share The relative percentage of the market controlled by the product. Customer or user satisfaction gap The difference between a customer or user’s desired experience and their current experience.
  30. 30. page 030 Time to Market (T2M) page 030
  31. 31. page 031 Time to Market Organisation’s ability to quickly deliver new capabilities, services, or products Goal: to minimise the amount of time it takes for the organisation to deliver value Without actively managing T2M, the ability to sustain delivering value in the future remains uncertain • How fast can the organisation learn from new experiments? • How fast can we learn from new information and adapt? • How fast can we deliver new value to customers?
  32. 32. page 032 What does it take to go from idea to value and improvement opportunities? How can you improve your time to market and reduce the time it takes you to get feedback? What are the steps needed from forming an idea to delivering value to customers?
  33. 33. page 033 Visualise the delivery process Create a value stream map that represents your delivery process Start with the idea and end with when the customer or user realises value • What is the current time to market? • What bottlenecks are there? • Identify areas to improve delivery pipeline automation, improve maintainability or remove technical debt to reduce waste
  34. 34. page 034 T2M MeasuresHow long would it take for to know whether we’ve actually achieved the outcome we wanted to achieve and the impact we wanted to achieve? Leading indicators Lagging indicators • Frequency of Build Success • Build pass/fail trends • Release Stabilisation trends • MTTR – mean time to Repair • Cycle Time • Release Frequency • Lead Time • Time to Learn
  35. 35. page 035 T2M – Key Value Measures KVM Measuring: Build and integration frequency The number of integrated and tested builds per time period. For a team that is releasing frequently or continuously, this measure is superseded by actual release measures. Release Frequency The number of releases per time period, e.g. continuously, daily, weekly, monthly, quarterly, etc. This helps reflect the time needed to satisfy the customer with new and competitive products. Release Stabilization Period The time spent correcting product problems between the point the developers say it is ready to release and the point where it is actually released to customers. This helps represent the impact of poor development practices and underlying design and code base. Mean Time to Repair The average amount of time it takes from when an error is detected and when it is fixed. This helps reveal the efficiency of an organization to fix an error. Cycle Time The amount of time from when work starts on a release until the point where it is actually released. This measure helps reflect an organization’s ability to reach its customer. Lead Time The amount of time from when an idea is proposed or a hypothesis is formed until a customer can benefit from that idea. This measure may vary based on customer and product. It is a contributing factor for customer satisfaction. Time-to-Learn The total time needed to sketch an idea or improvement, build it, deliver it to users, and learn from their usage.
  36. 36. page 036 Ability to innovate (T2I) page 036
  37. 37. page 037 Ability to Innovate (A2I)Ability of a organisation to deliver new capabilities that might better meet customer needs Goal: to maximise ability to deliver new products and innovative solutions Ability to Innovate helps avoid software that is overloaded by low value features Continually Re-evaluate A2I by asking: • What prevents the organisation from delivering new value? • What prevents customers or users from benefiting from that innovation?
  38. 38. page 038 Impediments to delivering value Reduces A2I As low-value features accumulate, more of the budget and time is consumed maintaining the product, not increasing capacity to innovate Anything that prevents users from benefitting from innovation, such as hard to install software or lack of capabilities, will also reduce A2I •Maintaining multiple code branches or product versions •Complex or monolithic application architecture •Insufficient product-like environments to test on •Lack of operational excellence •Lack of decentralised decision- making •Spending too much time fixing defects or reducing technical debt •Inability to hire and inspire talented, passionate team-members
  39. 39. page 039 A2I MeasuresHow long would it take for to know whether we’ve actually achieved the outcome we wanted to achieve and the impact we wanted to achieve? Leading indicators Lagging indicators • Technical Debt trends • Architectural Coupling • Defect trends • Production incident trends • Downtime trends • Number of active branches, time spent merging • Time spent context switching • Velocity trends • Innovation Rate • Installed Version Index • Usage Index
  40. 40. page 040 Defect Reduction 12 months into Agile Transition – Large scale 32 delivery areas and 45 teams across the Organisation • 18Q2, defects significantly down from 17Q2, a release of comparable scope and volume • Previously 6 teams would be taken offline to fix defects for 3 months pre and 1 month post deployment • Robust Function testing meant defects discovered earlier by agile teams • Integration of components became part of DoD 17Q2 18Q2
  41. 41. page 041 Innovation RateWhat percentage of your product budget is spent on: • Innovation and building new functionality vs. • Incremental business change to expand capacity? vs. • Maintaining business operations? Source: Forrester, October 2010, 2011 IT Budget Planning Guide For CIOs Source: 2016-2017 Global CIO Survey N=1081 29% Incremental business change Business innovation 18% 58% 2010 2016-17
  42. 42. page 042 Innovation rate – Which is Better? 52% Business innovation 10% Incremental business change 38% Business Operations 29% Business innovation 53% Incremental business change 18% Business Operations What is good depends on: • What stage of Product Development Life Cycle • What type of Industry • Pace of change and competition for users • Risk appetite vs Opportunity Enablement value 29% is the cross-industry average across products and systems, roughly the same for SMB and Enterprise
  43. 43. page 043 What can we do to Improve? Support your Scrum Team in creating great products, with no defects, that people actually use The lower the cost of supporting your products, the more resources there will be for satisfying your customer’s needs Building software is largely a one time cost, support and maintenance goes on forever
  44. 44. page 044 A2I – Key Value Measures KVM Measuring: Usage Index Measurement of features in the product that are frequently used. This helps capture features that are rarely or never used. Innovation Rate The percentage of effort or cost spent on new product capabilities, divided by total product effort or cost. This provides insight into the capacity of the organization to deliver new product capabilities. Defect trends Measurement of change in defects since last measurement. A defect is anything that reduces the value of the product to a customer, user, or to the organization itself. Defects are generally things that don’t work as intended. On-Product Index The percentage of the total user base that is using the current version of the product. Installed Version Index The number of versions of a product that are currently being supported. This reflects the effort the organization spends supporting and maintaining older versions of software. Technical Debt A concept in programming that reflects the extra development and testing work that arises when “quick and dirty” solutions result in later remediation. Production Incident Trends The number of times the Development Team was interrupted to fix a problem in an installed product. The number and frequency of Production Incidents can help indicate the stability of the product. Active code branches, time spent merging code between branches These measures are similar to the Installed Version Index, since different deployed versions usually have separate code branches. Time spent context- switching Number of meetings per day per person, and the number of times a day team members are interrupted to help people outside the team can give simple insight into the magnitude of the problem.
  45. 45. page 045 Conclusions page 045
  46. 46. page 046 Deciding what to Measure is critical
  47. 47. page 047 EBM Key Value Measures Current Value Revenue per Employee Product Cost Ratio Employee Satisfaction Customer Satisfaction Usage Index Time to Market T2M Build and integration frequency Release Frequency Release Stabilization Period Mean Time to Repair Cycle Time Lead Time Time-to-Learn Ability to Innovate A2I Usage Index Innovation Rate Defect trends On-Product Index Installed Version Index Technical Debt Production Incident Trends Active code branches, time spent merging code between branches Time spent context- switching Unrealised Value Market Share Customer or user satisfaction gap
  48. 48. page 048 Inspect and AdaptMeasurement helps us inspect so that we can adapt and improve our products and also the way we work The spirit of improvement helps creates innovation Understanding the changes of the KVMs prepares the organization for its next learning loop and track changes across time learn from patterns that emerge Incremental changes performed in small learning loops are the most effective method for increasing an organisation’s overall agility
  49. 49. page 049 Experiment and Constantly Learn to Improve Value Derived Visualise relative strengths and weaknesses Observe the impact of these practices to overall organisational value Assess the results and impact of an experiment to monitor the trend of value over time
  50. 50. page 050 EBM provides a wholistic view of Value to enhance Business Agility Measure success in terms of value and outcomes, not output and activity Celebrate success and learn what helped create it Focus on improving results Run experiments to understand what is valued, to build innovative products users will love Maximise learning to build knowledge and capability Source Evidenced Based Management Guide https://www.scrum.org/resources/evidence-based-management
  51. 51. page 051 51 “If you can't measure it, you can't improve it.” - Peter Drucker • Current Value is most important - a product that offers no value to its users won’t last long • Customer/user experience is only part of the picture; sustaining and improving value to customers requires engaged employees and happy investors • Improving value requires frequent delivery of new value, i.e improving the Time-to-Market • Understanding and removing impediments to faster delivery is essential • Faster Time-to-Market is not the whole story - Fast release cycles delivering only very small improvements do little to rapidly improve the value delivered by a product • Ability to innovate is determined by ability to deliver significant innovation in each release • Measuring this ability gives organisations insights needed to remove barriers • Improving organisational performance is a cyclic, iterative process • Measure current conditions, set performance goals, form small improvement experiments, and measure again to gauge the effect, then repeating, continuously
  52. 52. page 052 Fin! Mia Horrigan@zenexmachina.com Blog – www.zenexmachina.wordpress.com Twitter - @miahorri www.zenexmachina.com Source and Thanks to Patricia Kong- Co Author of Evidenced Based Management https://www.scrum.org/resources/evidence- based-management

×