Downloaded 19 times
Uploaded byBen Linders
Agile Requirements
The document discusses agile requirements and measuring quality in software projects. It introduces two models for estimating quality: a defect estimation model and a quality factor model. It also discusses measuring quality by looking at where defects are inserted into the design and code and where they are detected, with the goal of reducing fault slip through and improving customer satisfaction. Quality is influenced by management factors as well as the performance of design, testing and other project phases.
More Related Content
Agile Requirements
- 1. Agile Requirements Agile Consortium Benelux, sep 30, 2009 Ben Linders Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 1
- 2. Contents / Agenda Introduction Business Quality Cases Factors Agile Requirements Conclusions Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 2 2
- 3. Problem Statement Introduction Quality improvement needed in many organizations Business case • Identification of problem areas • Selected improvement • Decision Quantified • Costs & benefits • Lead time to result Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 3 3
- 4. Quantification problems Introduction Much time needed to gather data Difficult to measure things Hard to keep management commitment Expensive Required: Business case, with limited but sufficient measurement effort, to gain management commitment and funding Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 4 4
- 5. Affiliate Collaboration Introduction SEI Pittsburgh, PA: Software Engineering Measurement & Analysis Group Ericsson Netherlands: Market Unit Northern Europe & Main R&D Center The Software Engineering Institute Affiliate Program provides sponsoring organizations with an opportunity to contribute their best ideas and people to a uniquely collaborative peer group who combine their technical knowledge and experience to help define superior software engineering practices. Affiliates: http://www.sei.cmu.edu/collaborating/affiliates/affiliates.html Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 5 5
- 6. Two models Introduction Defect Estimation Model Resident Defects in Design Base • Data, tuned with expert opinion Design Process Competence, skills Defects Inserted (documentation, Defect Density Tools, environment code) Estimate Fault Slip Through Detection Rate • Test Process Competence, skills Test Capacity Defects Detected (Inspection, test) Fault Slip Through Tools, environment Defect Classification • Project/Product Quality Resident Defects in Delivered Product (Un)happy customers Process Inputs and outputs Influencing factors Defect Level Measurement Quality Factor Model • Expert opinion, extend with data • Quick Quality Scan • Prediction Fault Slip Through • Improvement Areas Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 6 6
- 7. Measuring quality Business Cases Insertion: Where are defects made? How to prevent? Detection: Where are defects found? Early/economic removal? Quality: How many defect are left in the product at release? Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 7 7
- 8. Process View Business Cases Resident Defects in Design Base Design Process Defects Inserted Defect Density Competence, skills (documentation, Tools, environment code) Detection Rate Test Process Competence, skills Defects Detected Fault Slip Through Test Capacity (Inspection, test) Tools, environment Defect Classification Resident Defects in (Un)happy customers Process Delivered Product Inputs and outputs Influencing factors Defect Level Measurement Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 8 8
- 9. Fault Slip Through Business Cases Lead ??? Time Cost ??? FST Quality ??? Fault Slip Through = Number of defects detected in integration & customer test that should have been detected earlier “Should” implies that the defect is more cost effective to find earlier. Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 9 9
- 10. Quality Phase Performance Quality Factors Defect Insertion Management Factors Quality Factor: Influencing quality of the delivered product Defect Detection Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 10 10
- 11. Management Factors Quality Factors Management Context for Technical Activities Direct: • Project Management • Process Management Indirect: • Strategic & Operational Line Management Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 11 11
- 12. Defect Insertion Quality Factors Technical Activities where defects inserted • Root Cause Analysis • Defect Prevention Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 12 12
- 13. Defect Detection Quality Factors Technical Activities where defects detected • Early Detection • Economy of Test • Release Quality Reduce Defect Slippage Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 13 13
- 14. Quality Factors -Requirements Quality Factors Purpose Process Requirement Maturity Management • Predict Quality Capability Requirement • Leading indicator Stability Commitment Sources Requirements • Research Performance Roadmap Quality • Expert opinion • Experience Root Scope Requirement Cause Stability Definition Analysis Capability Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 14 14
- 15. Quality performance assessment Agile Req. Survey based upon Quality Factors • 34 respondents from management & technical roles • 4 management areas & 7 technical areas 2 sub questions for each quality factor: • How relevant is the factor when we want to improve quality? “little if any,” “moderate,” “substantial,” or “extensive,” • How well are we doing currently? “poor,” “fair,” “good,” and “excellent.” Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 15 15
- 16. Findings Requirements Agile Req. Process Requirement Maturity Management Capability Requirement Stability Commitment Requirements Performance Roadmap Quality Root Scope Requirement Cause Stability Definition Analysis Capability Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 16 16
- 17. Pilot “Business Casefor Quality” Agile Req. Context: • Process management • Quality steering • Starting with Agile Pilot: Agile for Requirements • Calculate value of process change • Run the pilot • Evaluate the result Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 17 17
- 18. Improve: Requirements Stability Agile Req. Requirements Stability – Inverse of the amount of requirement changes over time. (The less changes, the higher stability.) Agile deployment • Backlog with Prioritized User Stories • Product manager as Product Owner • (Pre-) Planning game • Architecture team • Stand up meetings Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 18 18
- 19. Improve: Scope Stability Agile Req. Scope Stability – Impact of major changes in projects that are related to changes in the product roadmap, including stability of the products to be developed, development teams involved in projects, and major changes in project funding or delivery dates. Agile deployment • Backlog • Responsibility of Agile teams and Product Owner • (Pre-) Planning game • Retrospectives Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 19 19
- 20. Improve: Requirement Definition Agile Req. Capability Requirements Definition Capability – The skill and experience level of the people doing requirements definition (e.g., product managers). Agile deployment • (Pre-) Planning game • Stand up meetings • Collaborative Culture • Retrospectives Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 20 20
- 21. Steering Agile Quality Agile Req. • Estimate latent defects after demo (planning game) • Collect defects during test (after demo). • Classify defects: • “introduction phase“ • “should have been detected phase” • Root cause analysis: Prevention • Decide improvement actions and communicate • Re-estimate and predict release quality. Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 21 21
- 22. Results Agile forRequirements Agile Req. • Very low number of requirement defects • Previous projects also had a low number • Based upon the data no conclusion could be drawn Root Cause Analysis: • understanding requirements increased: planning game & stand-up meetings. • Improvements from retrospectives increased cooperation between development team and product owner. Requirements quality performance increased! Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 22 22
- 23. Conclusions Conclusions Quicker Business Case: • Quality Factors/Performance • Fault Slip Through • Combining data and expert opinion Improved Requirements Performance • Agile increased requirements quality • Less defects after release • Increased flexibility and collaboration Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 23 23
- 24. More information Conclusions Publications: • Building Process Improvement Business Cases SEI Technical Note: http://www.sei.cmu.edu/library/abstracts/reports/09tn017.cfm • Controlling Project Performance by Using the Project Defect Model in proceedings PSQT West Conference 2005 • The Business Benefit of Root Cause Analysis in proceedings SM/ASM conference 2003 • SPI, the agile way! To be presented at the SPIder conference, october 2009 www.spiderconferentie.nl Contact: • Email: benlinders@gmail.com • http://www.linkedin.com/in/benlinders Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 24 24
- 25. Backup Slides Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 25 25
- 26. Solution Introduction Technologies • Bayesian Belief Networks (BBN) • Monte Carlo Simulation • Root Cause Analysis • Cost of Quality, Defect Slippage Six Sigma DMAIC Approach • Modeling Business Cases • Research Quality Factors & quantify Quality Improvement • Validate “Business Case for Quality” Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 26 26
- 27. Building a businesscase Business Cases Quality Quality Fault Quality Factor Slip BBN Quality Factor Phase Through Performance Factor Quality Factor Historical Industry Project Data Data Monte Current Improved Quality Phase Quality Phase Carlo Performance Performance Subjective Expert Opinion Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 27 27
- 28. Bayes Belief Network(BBN) Business Cases • Probabilistic graphical model, to model uncertainty • Diagnose and explain why an outcome happened • Predict outcomes based on insight to one or more factors Used: • Modeling Quality Factors • Predicting Quality Phase Performance • What if Scenario Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 28 28
- 29. Monte Carlo Simulation Business Cases • Compute a result based on random sampling • Modeling distributions of data • Can make uncertainty visible Used: • Calculate value of process changes Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 29 29
- 30. Quality Prediction Business Cases Current Model: Estimation • Extrapolate past performance • Based on inserted/detected defects • Plan & track Wanted: Prediction • Causes of defects • What if Scenarios All models are wrong • Decision taking Some models are useful Deming Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 30 30
- 31. Step 2: DefectPrediction Business Cases Fault Slip Through Defect found in a (later) test phase that should have been found earlier “Should”: More Cost effective (economical) Predict Defect Reduction • Determine process impact • Simulate quality change • Predict savings Pilots • Agile • Model Driven Development Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 31 31
- 32. Quantify Quality Improvement Quality Factors Connect defect data with Quality performance • Maximum quality factor => Industry best in class Published industry data from various sources • Distribution: Linear (keep it simple) Extend BBN to calculate remaining defects after each phase Result: Model for “what if scenario’s” • Calculate defects in release products, when quality performance improves • Cost of Quality data to calculate savings Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 32 32
- 33. Monte Carlo: Qualityperformance Quality Factors Monte Carlo simulation • Input from 5 experts • Estimated chance of occurrence and impact on FST (1-5 scale) • Simulation done to calculate impact on quality factors • Result used in BBN model to calculate effect on defect slippage Expected result: • Reduced number of requirement defects introduced • Increased effectiveness of late testing phases • Less defects in products shipped to customers • Cost saving: — Limited saving in the project — Major saving during maintenance Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 33 33
Editor's Notes
- #2 After an agile pilot, ETM decided to fully roll out the agile way of working. All projects have setup cross functional teams, some including I&V. Stakeholders like product managers, system managers and projects managers enable that teams can produce the right software. Support functions, like CM, QA, CPI, and managers, and process responsibles are also using agile techniques, to prioritize work and deliver value to the projects. So how does this impact process management? This presentation shows that processes are still important at ETM, but in a different way: agile integrated into enhanced streamline development. The main goal here is to get an "every employee improving our value stream and efficiency continuously" environment/culture.
- #3 Color coded agenda, so that the audience can see where we are. Intro: Problem statement, approach as SE affiliate, overview of technologies used Business Cases: Defect slippage as target, overview of the BBN/Monte Carlo model Quality Factors: Details of the model, what influences quality Pilot: Survey to determine improvement area (BBN), agile improvement (Monte Carlo) Conclusions: Is this approach useful, what did we learn?
- #4 Link between improvement and a business case: Investment/benefit
- #5 Why the current approach doesn’t work. What do we need to solve it?
- #6 Show quickly to let the audience know that this was a collaboration (maybe too much text?)
- #8 © 2006 Carnegie Mellon University 01/31/13
- #9 © 2006 Carnegie Mellon University 01/31/13
- #10 Position Fault Slip Through as main indicator of Quality, since: Accepted within Ericsson Indicates significant savings Can also be used to get cost/time savings
- #11 First sheet of the Quality Factors part: Zoom into factors that determine quality. This is the BBN model. Overview of the Quality phases: Management factors Defect Insertion Defect detection Background. This model has been made to investigate quality improvement at Ericsson R&D The Netherlands. It includes quality factors that most probably will have an impact on quality at that development site. It is by no means intended to be a complete model, and different factors may be applicable for other companies.
- #12 Zoom into the 4 areas of management factors Explain shortly why the influence of strategic and operational line management is indirect (via project/process)
- #13 Shortly show the main phases in software development where potentially defects are included.
- #14 Show the main phases where defects are detected. Explain that defects left is input for the defects slippage (main indicator of quality).
- #15 Show some detail on how the quality phase performance is build up, example on code inspection.
- #16 How the assessment was done: Survey based SEI survey tool Questions reviewed with the SEI Representation of compete R&D flow 2 axes: Relevant & performance
- #17 Show some detail on how the quality phase performance is build up, example on code inspection.
- #18 First sheet of the pilot part of the presentation. Explain the 2 steps: Assessment to determine potential areas of improvement (BBN) Actual improvement (agile requirements), calculate & validate results
- #23 Very low number of requirement defects, from which Some on not yet release functionality (implemented later) Some due to changes in platform software, could not be prevented Only 1 defect which could have been prevented in planning game
- #27 Mention the technologies used, and that the project was run in a Six Sigma way.
- #28 Animated sheet! BBN technology is used to model the quality phase performance, based up Quality factors. This is used to models the target that we focus upon: defect slippage. Historical and industry data have been used to quantify the relationship between quality phase performance and fault slippage. The BBN models current performance. Based upon expert opinion, using Monte Carlo, a calculation is done of the quality phase performance after the improvement. This is fed into the BBN to calculate the impact on defect slippage, and the potential savings.
- #29 © 2006 Carnegie Mellon University 01/31/13 Wikipedia: A Bayesian network (or a belief network ) is a probabilistic graphical model that represents a set of variables and their probabilistic independencies. For example, a Bayesian network could represent the probabilistic relationships between diseases and symptoms. Given symptoms, the network can be used to compute the probabilities of the presence of various diseases. Formally, Bayesian networks are directed acyclic graphs whose nodes represent variables, and whose missing edges encode conditional independencies between the variables. Nodes can represent any kind of variable, be it a measured parameter, a latent variable or a hypothesis. They are not restricted to representing random variables , which represents another " Bayesian " aspect of a Bayesian network. Efficient algorithms exist that perform inference and learning in Bayesian networks. Bayesian networks that model sequences of variables ( e.g. speech signals or protein sequences ) are called dynamic Bayesian networks . Generalizations of Bayesian networks that can represent and solve decision problems under uncertainty are called influence diagrams .
- #30 © 2006 Carnegie Mellon University 01/31/13 Wikipedia: Monte Carlo methods are a class of computational algorithms that rely on repeated random sampling to compute their results. Monte Carlo methods are often used when simulating physical and mathematical systems. Because of their reliance on repeated computation and random or pseudo-random numbers, Monte Carlo methods are most suited to calculation by a computer . Monte Carlo methods tend to be used when it is infeasible or impossible to compute an exact result with a deterministic algorithm.[1] Monte Carlo simulation methods are especially useful in studying systems with a large number of coupled degrees of freedom, such as fluids, disordered materials, strongly coupled solids, and cellular structures (see cellular Potts model). More broadly, Monte Carlo methods are useful for modeling phenomena with significant uncertainty in inputs, such as the calculation of risk in business. These methods are also widely used in mathematics: a classic use is for the evaluation of definite integrals, particularly multidimensional integrals with complicated boundary conditions.
- #33 Explain how the Quality Phase performance is linked to defect slippage, though the use of industry and historical data. This will probably trigger questions, propose to discuss them after the presentation (BoF or …)?