Nesma, profile picture
Uploaded byNesma
PPTX, PDF494 views

Iwsm2014 defect density measurements using cosmic (thomas fehlmann)

This document discusses measuring defect density in software projects using the COSMIC functional size measurement method. It proposes: 1) Counting the number of defects found based on the number of affected data movements, as defined by the COSMIC model, rather than just defect repository entries. 2) Adjusting the test size measurement to match the number of data movements executed in tests. 3) This allows defect density to be calculated more precisely and supports benchmarking across projects.

Embed presentation

Downloaded 12 times
1 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Defect Density Measurements Using COSMIC Thomas M. Fehlmann, Zürich Eberhard Kranich, Duisburg Euro Project Office AG E: info@e-p-o.com H: www.e-p-o.com
2 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Dr. Thomas Fehlmann  1981: Dr. Math. ETHZ  1991: Six Sigma for Software Black Belt  1999: Euro Project Office AG, Zürich  2001: Akao Price 2001 for original contributions to QFD  2003: SwissICT Expert for Software Metrics, ICTscope.ch  2004: Member of the Board QFD Institute Deutschland – QFD Architect  2007: CMMI for Software – Level 4 & 5  2011: Net Promoter® Certified Associate  2012: Member of the DASMA Board  2013: Vice-President ISBSG         ® Certified Associate   Dr. Thomas Fehlmann
Eberhard Kranich 3 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions  Mathematics and Computer Science  Emphasis on Mathematical Statistics  Mathematical Optimization  Theory of Polynomial Complexity of Algorithms  Worked at T-Systems International GmbH in Bonn, Germany Working at T-Systems International GmbH in Bonn, Germany  Six Sigma Black Belt for Software Development  Software Quality Assurance Manager  Member of the DASMA Board
4 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions What is a Defect?  Defect = Behavior impacting expected or required functionality of software  How many bugs?  By counting the size of defect repositories?  By number of entries???
5 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Software Testing as a Game  Tester sees selected sequences in the UML sequence diagram  Tester can “walk” the data movements when planning or executing tests  Functionality becomes visible to the agile team  Defects impacting functionality become visible to testers Functional Process Other Application Some Device 8.// Move some data 9.// Move some data 10.// Move some data 11.// Move some data Other Device
6 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Functionality, Defect Size, and Defect Density  What happens if data movements have defects?  Testers mark the data movement where a defect has been detected  Same Metric:  ISO/IEC 19761 COSMIC Functional Process Other Application Some Device 8.// Move some data Move some data 10.// Move some data 11.// Move some data Other Device  Functional Size  Number of Data Movements needed to implement all FUR  Test Size  Number of Data Movements executed in Tests  Test Story  Collection of Test Cases aiming at certain FURs  Defect Count  Number of Data Movements affected by some defect detected in a test story
7 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Defects Density Prediction?  Now he counts the defects!  And counts and adjusts test size  By ISO/IEC 19761 COSMIC Functional Process Other Application Some Device 8.// Move some data 9.// Move some data 10.// Move some data 11.// Move some data Other Device How does he know that he found all the defects?
8 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Research Questions  What is Defect Density?  Defects per KDLOC?  What is Test Coverage?  Code lines executed by some test case?
9 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions SW Testing and SW Metrics  Counting practices for defect counting are undocumented  “Number of Defects Found” per Stages / with Tests / etc.  How do you count “Number of Defects”?  Is it simply the number of entries in a defect repository?  How can you avoid double reporting?  Or make sure two defects are reported twice and not in a single report?  A successor to the “Defect Measurement Manual” published by UKSMA in October 2000 is under review: “Defect Measurement and Analysis Handbook”  By European cooperation  Important enhancement for ISBSG’s Data Collection!
10 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions SW Testing and SW Metrics  Counting practices for defect counting are undocumented  “Number of Defects Found” per Stages / with Tests / etc.  How do you count “Number of Defects”?  Is it simply the number of entries in a defect repository?  How can you avoid double reporting?  Or make sure two defects are reported twice and not in a single report?  A successor to the “Defect Measurement Manual” published by UKSMA in October 2000 is under review: “Defect Measurement and Analysis Handbook”  By European cooperation  Important enhancement for ISBSG’s Data Collection!
11 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Learn & Remember Count Defects based on Reference Model Software Development the Six Sigma Way Use COSMIC ISO/IEC 19761 Benchmark!
12 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions As a … [Functional User] I want to … [get something done] Such that …[quality characteristic] So that … [value or benefit] 1) Q001 Propose Standard Destinations User of public transportation be able to store my preferred destinations they are valid for the Ticket Shop I no longer have to pay fees when catched without tickets 2) Q002 Find Nearest Boarding Station User of public transportation locate nearest station with GPS that's being served right now I immediately can see whether it's right 3) Q003 Process Payment Provider of transportation services give user access to their preferred payment options all payments are traceable in Ticket Shop they can manage spending 4) Q004 Issue Ticket User of public transportation get a valid ticket with settings from Ticket Shop I no longer have to pay fees when catched without tickets 5) Q005 Show Ticket Ticket controller see the validated ticket I can check validity period and travel range I don't need to go into a dispute with a client Functional User Requirements Example: The Ticket Apps  Customer’s Voice:  Give me a ticket subito! Without much ado and questions if I simply want to go home
13 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions The Ticket Apps 12 Entry (E) + 9 eXit (X) + 1 Read (R) + 2 Write (W) = 24 CFP App User Home Destinations Saved Stations Ticket Shop Ticket Purchase Phone's GPS GIS Application Timetable Service Local 1.// Add New Destination Traveler 2.// Check whether Destination Exists 3.// Collect Matching Destinations 4.// Show Matching Destinations 5.// Select Exact Destination 6.// Record Selected Destinations 7.// Ask App for a Ticket 8.// Propose Destinations 9.// Proposed Destinations 10.// Select Destination Prepare 11.// GPS Coordinates 12.// Date & Time 13.// Ask GIS 14.// Nearest Boarding Station 15.// Search Connection
14 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions The Ticket Apps – Objects of Interest Description Type Wants to buy a ticket to ride Device User Store a list of destinations that are valid for "getting home" Functional Process List of standard stations eligible as travel destination Persistent Data Finds nearest origin station Other Application Buy the ticket needed for that route Functional Process External ticket shop able to identify the App’s user Other Application Finds connections between stations at a given time & day Other Application Provides GPS coordinates where phone is located Device User Synchronized time service located in smartphone Device User App User Home Destinations Saved Stations GIS Application Ticket Purchase Ticket Shop Timetable Service Phone's GPS Local Time
15 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions The Ticket Apps – Prepare Destinations App User Ticket Shop 1.// Add New Destination Prepare 2.// Check whether Destination Exists 3.// Collect Matching Destinations 4.// Show Matching Destinations 5.// Select Exact Destination Home Destinations Saved Stations 6.// Record Selected Destination  Preparation:  Store standard destination stations on Smartphone  After checking with the Ticket Shop whether they exist or which stop to select  Involves Ticket Shop and local data ‘Standard Stations”
Show Matching Destinations 16 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions The Ticket App in the IFPUG Model Confirm Purchase Select Destination Enter New Destination Request Ticket Boarding Station Mobile Ticket Boundary IFP=51 EI EQ EO ILF EIF EQ 1 / 2 EI 1 / 3 EI 2 / 2 EI 2 / 1 EIF 1 / 5 Timetable Service EIF 1 / 1 Local Time EIF 1 / 3 GIS Application EIF 2 / 12 Ticket Shop EO 2 / 2 EO 3 / 3 EO 1 / 2 ILF 1 / 2 Home Destinations  Less complicated than UML Sequence Diagrams  Suits business better  Better boundary identification between different layers
17 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Prepare Destination in the IFPUG Model  It is not obvious what happens  Two elementary process seem unrelated  Nevertheless, they update an ILF  [Uh, they forgot the elementary process for deleting obsolete home destinations!] Show Matching Destinations Enter New Destination Boundary IFP=19 EI EQ EO ILF EIF ILF 1 / 2 Home Destinations EO 1 / 2 EI 1 / 3 EIF 2 / 12 Ticket Shop
18 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Buy Ticket in the IFPUG Model  Elementary processes stay  The ILF ‘Home Destinations’ becomes an EIF  Adding the two counts yields 19 + 42 > 51  IFPUG is useless for functional sizing when developing Apps or counting defects  IFPUG 4.3 is great for identifying system boundaries splitting apps! Request Ticket Confirm Purchase Boarding Station Mobile Ticket Select Destination Boundary IFP=42 EI EQ EO ILF EIF EIF 1 / 5 Timetable Service EIF 1 / 1 Local Time EIF 1 / 3 GIS Application EIF 1 / 2 Home Destinations EIF 2 / 12 Ticket Shop EI 2 / 2 EI 1 / 1 EO 2 / 2 EO 2 / 3 EQ 1 / 2
19 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Requirements for a Defect Measurement Reference Model  It must be additive  The total size must be the sum of the components’ sizes.  It must be understandable  Support UML Sequence Diagrams  Works well with UML Use Cases  Also with Business Process Model & Notation (BPMN) 2.0  It must fit into agile delivery  Buglione-Trudel Matrix for managing agile projects  Story Cards for Sprints Story Card for Prepare Destinations Test is Ready Draft is Ready Review Done Final-ized Appro-ved Func-tional R001-01: Prepare Destinations Refactoring Count: 5 6 x reworked 0 0 Story Points: Functional Size: Business Impact: App User Prepare Destinations Standard Stations Ticket Shop 1.// Add New Destination 2.// Check whether Destination Exists 3.// Collect Matching Destinations 4.// Show Matching Destinations 5.// Select Exact Destination 6.// Record Selected Destinations The user prepares his ticket app by entering standard destinations. The ticket shop is involved for resolving station names matches and avoid spelling errors
20 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Learn & Remember Count Defects based on Reference Model Software Development the Six Sigma Way Use COSMIC ISO/IEC 19761 Benchmark!
21 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions What is a Test?  A Software Test has  Several Test Stories • Weighted by Customer’s Priority for the Test Story, reflecting the value for the customer  Each Test Story has many Test Cases • For various kind of test data  A Test Size attribute • Number of data movements executed by test cases  A Test Coverage attribute • Percentage of data movements covered with test cases  An Outcome • Passed or Failed – Passed: All responses according expectations – Failed: at least one test case didn’t yield the expected response
CT-B Ticketing CT-B.1 Select Destination CT-B.1.1 Station of origin known Asks for destination CT-B.1.2 No destination selected Stops without contacting Ticket Shop CT-B.2 Get Ticket CT-B.2.1 Both stations known Asks to confirm price CT-B.2.2 Boarding station undefined Doesn't ask for destination CT-B.3 Price Calculation CT-B.3.1 Both stations known Presents price of next actual connection CT-B.3.2 Both stations known Display intermediate change stations CT-B.4 Issue Ticket CT-B.4.1 Issue successful, ticket shown All credentials visible CT-B.4.2 Ticket Shop blocked Explanation given CT-B.5 Payment Tests CT-B.5.1 Credit limit exceeded Returns exceed notice CT-B.5.2 Payment service out of order Returns warning 22 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Sample Test Stories Test Story Case 1 Test Data Expected Response Case 2 Test Data Expected Response CT-A Prepare CT-A.1 Find Nearest Station CT-A.1.1 Enter GPS exactly Returns correct station CT-A.1.2 Enter GPS nearby Returns nearest station CT-A.2 Served Stations only CT-A.2.1 Select time of service Returns next available connection CT-A.2.2 Select time out of service Return next best served station CT-A.3 Enter New Destination CT-A.3.1 Enter valid station name Destination stored CT-A.3.2 Enter invalid station name Destination rejected  Eight Test Stories for the Ticket Apps  Each Test Story has several Test Cases  Each Test Case has defined Test Data  The Expected Response is known as per Test Case
23 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions What is a Test Case?  A Test Cases has  Entry Data (“Test Data”) • Explaining the environment for the test case • Typically valid, invalid, borderline data • Normal and disturbed communication services  A known Expected Response • The response of the system is known in advance  A known sequence of data movements executed • Defining Test Coverage • Each Test Case has a Size
What is a Test Case? 24 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions  The sequence of data movements can be visualized in the sequence diagram per test case  By double-clicking the respective data movements Find Route GIS Application Phone's GPS 8.// GPS Coordinates 9.// Date & Time 10.// Ask GIS 11.// Nearest Boarding Station Local Time Test Story No. 1 User Stories Test Case Measurements for Test Story CT-A.1 CT-A.1 Find Nearest Station Q001: Propose Standard Destinations Q002: Find Nearest Boarding Station Q003: Process Payment Q004: Issue Ticket Q005: Show Ticket Expected Response CT-A.1.1 Enter GPS exactly E012 E012,X009 Returns correct station CT-A.1.2 Enter GPS nearby E012,X009 Returns nearest station CT-A.1.3 Enter malformatted GPS E006 Returns no station CT-A.1.4 Enter far away GPS position E006,E007,X004,E008 E012,X009 Returns nearest station plus warning CT-A.1.5 GPS not working E005,E006 E006 Returns warning Test Story Contribution (CFP): 2 6 0 1 6 Test Size
Defects Observed Data Movements Affected 25 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Defect Reporting  When a defect has been identified in a test case, it can be recorded by double-clicking the suspect sequence of data movements User Stories Test Case Measurements for Test Story CT-B.1 CT-B.1 Select Destination Q001: Propose Standard Destinations CT-B.1.1 Station of origin known R001,X001,E005 CT-B.1.2 No destination selected R001,X001 CT-B.1.3 Valid boarding & destination W001,E005,R001,X001 CT-B.1.4 List of destinations X003,R001,W001 Test Story Contribution (CFP): 12 Test Story No. 4 Expected Response CFP Name Label Description Name Label Asks for destination 6 Stops without contacting Ticket Shop 2 Returns next available connection 15 All readable and visible 3 #003 Multiple References If a station contains more than one transport mode, e.g., bus and train, both are recorded as separate destinations W001 Record Selected Destinations Test Size 26 1 Defect Count
26 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions 12 Entry (E) + 9 eXit (X) + 1 Read (R) + 2 Write (W) = 24 CFP Extract Report App User Home Destinations Saved Stations Ticket Shop Ticket Purchase Phone's GPS GIS Application Timetable Service 1.// Add New Destination Traveler 2.// Check whether Destination Exists 3.// Collect Matching Destinations 4.// Show Matching Destinations 5.// Select Exact Destination 6.// Record Selected Destinations 7.// Ask App for a Ticket 8.// Propose Destinations 9.// Proposed Destinations 10.// Select Destination Prepare 11.// GPS Coordinates Select Data Movements for Test Case CT-B.1.4: List of destinations when executed in view of Q001: Propose Standard Destinations This Test Case identifies Fault #003: Multiple References affecting Data Movement 6.// Record Selected Destinations 12.// Date 13.// Ask GIS 14.// Nearest Boarding Station Finish Record Defects
27 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Count Defects  You can count One Defect  Per Data Movement  Identified by One Test Story And never more… ©2012-2014 Rhafiel
28 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Compare  This means  The more test stories you have, the more defects you can uncover  Test size is equally important as functional size  Defect density depends from test size  Test Benchmarking Test Status Summary Total CFP: 24 Defects Pending for Removal: 3 Test Size in CFP: 185 Defects Found in Total: 3 Test Intensity in CFP: 7.7 Defect Density: 13% Test Coverage: 79%
Measuring Test Coverage 29 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Test Stories Number of Response 풚 Goal Test Coverage Find Nearest Station Served Stations only Enter New Destination Select Destination Get Ticket Price Calculation Issue Ticket Payment Tests Achieved Coverage CT-A.1 CT-A.2 CT-A.3 CT-B.1 CT-B.2 CT-B.3 CT-B.4 CT-B.5 Test Stories Deployment Combinator User Stories Q001 Propose Standard Destinations 0.52 2 9 12 8 9 7 0.55 Q002 Find Nearest Boarding Station 0.45 6 9 7 10 3 5 4 0.45 Q003 Process Payment 0.62 3 4 4 4 10 7 16 0.59 Q004 Issue Ticket 0.24 1 3 1 3 6 2 3 0.23 Q005 Show Ticket 0.29 6 3 2 1 9 3 3 0.31 Ideal Profile for Test Stories 0.18 0.20 0.26 0.40 0.38 0.51 0.36 0.40 Convergence Gap 0.18 0.2 0.2 0.4 0.4 0.5 0.4 0.4 0.06 0.10 Convergence Range 0.20 Convergence Limit Toggle Measured Controls Expected Achieved Response 풚퐸 data movements executed
30 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Learn & Remember Count Defects based on Reference Model Software Development the Six Sigma Way Use COSMIC ISO/IEC 19761 Benchmark!
31 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Conclusions  You need more than one model when measuring modern software  ISO/IEC 19761 COSMIC is most appropriate for measuring defects  Other methods might be suitable for cost estimations  However, in these days, what drives cost more than faulty software?  Stop the nonsense with counting defects in repositories  Include defect density goals in contracts  Do independent auditing as long as you don’t trust your measurement equipment  Start Measuring Defects. Now.
32 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Questions?

More Related Content

PDF
Automated functional size measurement for three tier object relational mappin...
byIWSM Mensura
 
PDF
How to estimate the cost of a Maximo migration project with a high level of c...
byMariano Zelaya Feijoo
 
PPTX
Cost xpert
byMadeha Arif
 
PDF
Automatic measurements of use cases with cosmic thomas fehlmann
byIWSM Mensura
 
PPTX
Lesson 1 navigating lab view
byRaymundo Vazquez
 
PPTX
industrial internship presentation
byShreyaBhatt23
 
PPT
Conversion Detail
byKiya Aliyar, PMP
 
PPT
Cocomo model
byBala Ganesh
 
Automated functional size measurement for three tier object relational mappin...
byIWSM Mensura
 
How to estimate the cost of a Maximo migration project with a high level of c...
byMariano Zelaya Feijoo
 
Cost xpert
byMadeha Arif
 
Automatic measurements of use cases with cosmic thomas fehlmann
byIWSM Mensura
 
Lesson 1 navigating lab view
byRaymundo Vazquez
 
industrial internship presentation
byShreyaBhatt23
 
Conversion Detail
byKiya Aliyar, PMP
 
Cocomo model
byBala Ganesh
 

Viewers also liked

PDF
Nesma autumn conference - Outsourcing needs software measurement - Werner Hei...
byNesma
 
PPT
Iwsm2014 sizing the entire development process (mauricio aguiar & luigi bug...
byNesma
 
PPTX
IWSM2014 IT confidence - How to ensure that valid and current industry data ...
byNesma
 
PDF
NESMA 2020
byNesma
 
PPTX
IWSM2014 MEGSUS14 - GQM on energy for SaaS - CETIC
byNesma
 
PPTX
IWSM2014 MEGSUS14 - A general overview of software sustainability measureme...
byNesma
 
PPTX
Iwsm2014 conference diner (eric van der vliet)
byNesma
 
PPTX
Iwsm2014 identifying and managing complex modules in executable software de...
byNesma
 
PPT
Iwsm2014 analysis of the per-unit work effort and per-unit work cost of the...
byNesma
 
PPTX
Iwsm2014 verifying the accuracy of automation tools for cosmic measurement ...
byNesma
 
PPTX
Estimation in the Tendering Process - Frank Vogelezang
byNesma
 
PPT
Converteren van functionele omvang van FPA naar COSMIC
byNesma
 
PPTX
Estimating packaged software - Eric van der Vliet - NESMA najaarsbijeenkomst ...
byNesma
 
PPTX
Posters presented at the NESMA fall conference
byNesma
 
PPTX
Iwsm2014 measuring the functional size of mobile apps with cosmic (harold v...
byNesma
 
PDF
Status van de brede review gebruiksgids SOA - Martin Jacobs - NESMA najaarsbi...
byNesma
 
PDF
Nesma autum conference 2015 - Measuring & improving different dimensions - Ni...
byNesma
 
PPTX
Iwsm2014 the effect of highlighting error categories in fsm training on the...
byNesma
 
PDF
Nesma autumn conference 2015 - Agile x FPA - Reflections about solution in a ...
byNesma
 
PPTX
Metrics to improve organisational performance - NESMA najaarsconferentie 2013
byNesma
 
Nesma autumn conference - Outsourcing needs software measurement - Werner Hei...
byNesma
 
Iwsm2014 sizing the entire development process (mauricio aguiar & luigi bug...
byNesma
 
IWSM2014 IT confidence - How to ensure that valid and current industry data ...
byNesma
 
NESMA 2020
byNesma
 
IWSM2014 MEGSUS14 - GQM on energy for SaaS - CETIC
byNesma
 
IWSM2014 MEGSUS14 - A general overview of software sustainability measureme...
byNesma
 
Iwsm2014 conference diner (eric van der vliet)
byNesma
 
Iwsm2014 identifying and managing complex modules in executable software de...
byNesma
 
Iwsm2014 analysis of the per-unit work effort and per-unit work cost of the...
byNesma
 
Iwsm2014 verifying the accuracy of automation tools for cosmic measurement ...
byNesma
 
Estimation in the Tendering Process - Frank Vogelezang
byNesma
 
Converteren van functionele omvang van FPA naar COSMIC
byNesma
 
Estimating packaged software - Eric van der Vliet - NESMA najaarsbijeenkomst ...
byNesma
 
Posters presented at the NESMA fall conference
byNesma
 
Iwsm2014 measuring the functional size of mobile apps with cosmic (harold v...
byNesma
 
Status van de brede review gebruiksgids SOA - Martin Jacobs - NESMA najaarsbi...
byNesma
 
Nesma autum conference 2015 - Measuring & improving different dimensions - Ni...
byNesma
 
Iwsm2014 the effect of highlighting error categories in fsm training on the...
byNesma
 
Nesma autumn conference 2015 - Agile x FPA - Reflections about solution in a ...
byNesma
 
Metrics to improve organisational performance - NESMA najaarsconferentie 2013
byNesma
 

Similar to Iwsm2014 defect density measurements using cosmic (thomas fehlmann)

PPTX
Iwsm2014 defect density measurements using cosmic (thomas fehlmann)
byCOSMIC - Common Software Measurement International Consortium
 
PDF
Agile Agency: State of the Art 2018
byRoger Brown
 
PDF
Nesma autumn conference 2015 - A QFD based tool for managing agile requiremen...
byNesma
 
PDF
Nesma autumn conference 2015 - Is FPA a valuable addition to predictable agil...
byNesma
 
PDF
An Agile Testing Dashboard: Metrics that Matter
byTechWell
 
PPT
Agile methods and safety critical software - Peter Gardner
byAdaCore
 
PPT
Agile Application Modernization Project
byLawrence Wilkes
 
PDF
Automatic measurements of use cases with cosmic thomas fehlmann
byIWSM Mensura
 
PPTX
Initializing new project
byLai Ha
 
PPT
Agile QA presentation
byCarl Bruiners
 
PDF
Agile UX in an Agency Environment
byDan Kalafus
 
PDF
Practical Application of Agile Techniques in Developing Safety Related Systems
byAdaCore
 
PPTX
Module 1 - SE.pptx
byDrJayashreeNair
 
PDF
Quality culture transition guide model - full
byMelissa Eaden
 
PDF
Agile is the New Black
byFred George
 
PDF
ROI Driven Digital Development
byRobbie Burns
 
PPTX
Agile Development and Continuous Testing
byTestinium
 
PDF
Agile Testing: The Key to Faster, Higher-Quality Releases
byJace Reed
 
PDF
Introduction to Investigation And Utilizing Lean Test Metrics In Agile Softwa...
byIJERA Editor
 
PDF
Lean or agile, software architecture is fragile
byBoyan Mihaylov
 
Iwsm2014 defect density measurements using cosmic (thomas fehlmann)
byCOSMIC - Common Software Measurement International Consortium
 
Agile Agency: State of the Art 2018
byRoger Brown
 
Nesma autumn conference 2015 - A QFD based tool for managing agile requiremen...
byNesma
 
Nesma autumn conference 2015 - Is FPA a valuable addition to predictable agil...
byNesma
 
An Agile Testing Dashboard: Metrics that Matter
byTechWell
 
Agile methods and safety critical software - Peter Gardner
byAdaCore
 
Agile Application Modernization Project
byLawrence Wilkes
 
Automatic measurements of use cases with cosmic thomas fehlmann
byIWSM Mensura
 
Initializing new project
byLai Ha
 
Agile QA presentation
byCarl Bruiners
 
Agile UX in an Agency Environment
byDan Kalafus
 
Practical Application of Agile Techniques in Developing Safety Related Systems
byAdaCore
 
Module 1 - SE.pptx
byDrJayashreeNair
 
Quality culture transition guide model - full
byMelissa Eaden
 
Agile is the New Black
byFred George
 
ROI Driven Digital Development
byRobbie Burns
 
Agile Development and Continuous Testing
byTestinium
 
Agile Testing: The Key to Faster, Higher-Quality Releases
byJace Reed
 
Introduction to Investigation And Utilizing Lean Test Metrics In Agile Softwa...
byIJERA Editor
 
Lean or agile, software architecture is fragile
byBoyan Mihaylov
 

More from Nesma

PDF
Viktor Clerc - Presentatie Nesma over NPR5333.pdf
byNesma
 
PDF
Harold van Heeringen - Nesma FP in Cost Estimation.pdf
byNesma
 
PDF
Frank Vogelezang - EFS presentation 6-6-2024.pdf
byNesma
 
PDF
Eric van der Vliet - ICEAA SW - Status.pdf
byNesma
 
PDF
The Use of Functional Size in the Industry.pdf
byNesma
 
PDF
2024-04 - Nesma webinar - Benchmarking.pdf
byNesma
 
PDF
Agile Team Performance Measurement webinar
byNesma
 
PDF
Software Cost Estimation webinar January 2024.pdf
byNesma
 
PDF
Nesma event June '23 - How to use objective metrics as a basis for agile cost...
byNesma
 
PDF
Nesma event June '23 - NEN Practice Guideline - NPR.pdf
byNesma
 
PDF
Nesma event June '23 - Easy Function Sizing - Introduction.pdf
byNesma
 
PPTX
Automotive Software Cost Estimation - The UCE Approach - Emmanuel Mary
byNesma
 
PPTX
The COSMIC battle between David and Goliath - Paul Hussein
byNesma
 
PPTX
Succesful Estimating - It's how you tell the story - Amritpal Singh Agar
byNesma
 
PPTX
(Increasing) Predictability of large Government ICT Projects - Koos Veefkind
byNesma
 
PPTX
CEBoK for Software Past Present Future - Megan Jones
byNesma
 
PPTX
Agile Development and Agile Cost Estimation - A return to basic principles - ...
byNesma
 
PPTX
Resolving Cost Management and Key Pitfalls of Agile Software Development - Da...
byNesma
 
PPTX
Project Succes is a Choice - Joop Schefferlie
byNesma
 
PDF
Afrekenen met functiepunten
byNesma
 
Viktor Clerc - Presentatie Nesma over NPR5333.pdf
byNesma
 
Harold van Heeringen - Nesma FP in Cost Estimation.pdf
byNesma
 
Frank Vogelezang - EFS presentation 6-6-2024.pdf
byNesma
 
Eric van der Vliet - ICEAA SW - Status.pdf
byNesma
 
The Use of Functional Size in the Industry.pdf
byNesma
 
2024-04 - Nesma webinar - Benchmarking.pdf
byNesma
 
Agile Team Performance Measurement webinar
byNesma
 
Software Cost Estimation webinar January 2024.pdf
byNesma
 
Nesma event June '23 - How to use objective metrics as a basis for agile cost...
byNesma
 
Nesma event June '23 - NEN Practice Guideline - NPR.pdf
byNesma
 
Nesma event June '23 - Easy Function Sizing - Introduction.pdf
byNesma
 
Automotive Software Cost Estimation - The UCE Approach - Emmanuel Mary
byNesma
 
The COSMIC battle between David and Goliath - Paul Hussein
byNesma
 
Succesful Estimating - It's how you tell the story - Amritpal Singh Agar
byNesma
 
(Increasing) Predictability of large Government ICT Projects - Koos Veefkind
byNesma
 
CEBoK for Software Past Present Future - Megan Jones
byNesma
 
Agile Development and Agile Cost Estimation - A return to basic principles - ...
byNesma
 
Resolving Cost Management and Key Pitfalls of Agile Software Development - Da...
byNesma
 
Project Succes is a Choice - Joop Schefferlie
byNesma
 
Afrekenen met functiepunten
byNesma
 

Iwsm2014 defect density measurements using cosmic (thomas fehlmann)

  • 1.
    1 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Defect Density Measurements Using COSMIC Thomas M. Fehlmann, Zürich Eberhard Kranich, Duisburg Euro Project Office AG E: info@e-p-o.com H: www.e-p-o.com
  • 2.
    2 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Dr. Thomas Fehlmann  1981: Dr. Math. ETHZ  1991: Six Sigma for Software Black Belt  1999: Euro Project Office AG, Zürich  2001: Akao Price 2001 for original contributions to QFD  2003: SwissICT Expert for Software Metrics, ICTscope.ch  2004: Member of the Board QFD Institute Deutschland – QFD Architect  2007: CMMI for Software – Level 4 & 5  2011: Net Promoter® Certified Associate  2012: Member of the DASMA Board  2013: Vice-President ISBSG         ® Certified Associate   Dr. Thomas Fehlmann
  • 3.
    Eberhard Kranich 3 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions  Mathematics and Computer Science  Emphasis on Mathematical Statistics  Mathematical Optimization  Theory of Polynomial Complexity of Algorithms  Worked at T-Systems International GmbH in Bonn, Germany Working at T-Systems International GmbH in Bonn, Germany  Six Sigma Black Belt for Software Development  Software Quality Assurance Manager  Member of the DASMA Board
  • 4.
    4 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions What is a Defect?  Defect = Behavior impacting expected or required functionality of software  How many bugs?  By counting the size of defect repositories?  By number of entries???
  • 5.
    5 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Software Testing as a Game  Tester sees selected sequences in the UML sequence diagram  Tester can “walk” the data movements when planning or executing tests  Functionality becomes visible to the agile team  Defects impacting functionality become visible to testers Functional Process Other Application Some Device 8.// Move some data 9.// Move some data 10.// Move some data 11.// Move some data Other Device
  • 6.
    6 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Functionality, Defect Size, and Defect Density  What happens if data movements have defects?  Testers mark the data movement where a defect has been detected  Same Metric:  ISO/IEC 19761 COSMIC Functional Process Other Application Some Device 8.// Move some data Move some data 10.// Move some data 11.// Move some data Other Device  Functional Size  Number of Data Movements needed to implement all FUR  Test Size  Number of Data Movements executed in Tests  Test Story  Collection of Test Cases aiming at certain FURs  Defect Count  Number of Data Movements affected by some defect detected in a test story
  • 7.
    7 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Defects Density Prediction?  Now he counts the defects!  And counts and adjusts test size  By ISO/IEC 19761 COSMIC Functional Process Other Application Some Device 8.// Move some data 9.// Move some data 10.// Move some data 11.// Move some data Other Device How does he know that he found all the defects?
  • 8.
    8 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Research Questions  What is Defect Density?  Defects per KDLOC?  What is Test Coverage?  Code lines executed by some test case?
  • 9.
    9 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions SW Testing and SW Metrics  Counting practices for defect counting are undocumented  “Number of Defects Found” per Stages / with Tests / etc.  How do you count “Number of Defects”?  Is it simply the number of entries in a defect repository?  How can you avoid double reporting?  Or make sure two defects are reported twice and not in a single report?  A successor to the “Defect Measurement Manual” published by UKSMA in October 2000 is under review: “Defect Measurement and Analysis Handbook”  By European cooperation  Important enhancement for ISBSG’s Data Collection!
  • 10.
    10 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions SW Testing and SW Metrics  Counting practices for defect counting are undocumented  “Number of Defects Found” per Stages / with Tests / etc.  How do you count “Number of Defects”?  Is it simply the number of entries in a defect repository?  How can you avoid double reporting?  Or make sure two defects are reported twice and not in a single report?  A successor to the “Defect Measurement Manual” published by UKSMA in October 2000 is under review: “Defect Measurement and Analysis Handbook”  By European cooperation  Important enhancement for ISBSG’s Data Collection!
  • 11.
    11 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Learn & Remember Count Defects based on Reference Model Software Development the Six Sigma Way Use COSMIC ISO/IEC 19761 Benchmark!
  • 12.
    12 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions As a … [Functional User] I want to … [get something done] Such that …[quality characteristic] So that … [value or benefit] 1) Q001 Propose Standard Destinations User of public transportation be able to store my preferred destinations they are valid for the Ticket Shop I no longer have to pay fees when catched without tickets 2) Q002 Find Nearest Boarding Station User of public transportation locate nearest station with GPS that's being served right now I immediately can see whether it's right 3) Q003 Process Payment Provider of transportation services give user access to their preferred payment options all payments are traceable in Ticket Shop they can manage spending 4) Q004 Issue Ticket User of public transportation get a valid ticket with settings from Ticket Shop I no longer have to pay fees when catched without tickets 5) Q005 Show Ticket Ticket controller see the validated ticket I can check validity period and travel range I don't need to go into a dispute with a client Functional User Requirements Example: The Ticket Apps  Customer’s Voice:  Give me a ticket subito! Without much ado and questions if I simply want to go home
  • 13.
    13 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions The Ticket Apps 12 Entry (E) + 9 eXit (X) + 1 Read (R) + 2 Write (W) = 24 CFP App User Home Destinations Saved Stations Ticket Shop Ticket Purchase Phone's GPS GIS Application Timetable Service Local 1.// Add New Destination Traveler 2.// Check whether Destination Exists 3.// Collect Matching Destinations 4.// Show Matching Destinations 5.// Select Exact Destination 6.// Record Selected Destinations 7.// Ask App for a Ticket 8.// Propose Destinations 9.// Proposed Destinations 10.// Select Destination Prepare 11.// GPS Coordinates 12.// Date & Time 13.// Ask GIS 14.// Nearest Boarding Station 15.// Search Connection
  • 14.
    14 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions The Ticket Apps – Objects of Interest Description Type Wants to buy a ticket to ride Device User Store a list of destinations that are valid for "getting home" Functional Process List of standard stations eligible as travel destination Persistent Data Finds nearest origin station Other Application Buy the ticket needed for that route Functional Process External ticket shop able to identify the App’s user Other Application Finds connections between stations at a given time & day Other Application Provides GPS coordinates where phone is located Device User Synchronized time service located in smartphone Device User App User Home Destinations Saved Stations GIS Application Ticket Purchase Ticket Shop Timetable Service Phone's GPS Local Time
  • 15.
    15 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions The Ticket Apps – Prepare Destinations App User Ticket Shop 1.// Add New Destination Prepare 2.// Check whether Destination Exists 3.// Collect Matching Destinations 4.// Show Matching Destinations 5.// Select Exact Destination Home Destinations Saved Stations 6.// Record Selected Destination  Preparation:  Store standard destination stations on Smartphone  After checking with the Ticket Shop whether they exist or which stop to select  Involves Ticket Shop and local data ‘Standard Stations”
  • 16.
    Show Matching Destinations 16 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions The Ticket App in the IFPUG Model Confirm Purchase Select Destination Enter New Destination Request Ticket Boarding Station Mobile Ticket Boundary IFP=51 EI EQ EO ILF EIF EQ 1 / 2 EI 1 / 3 EI 2 / 2 EI 2 / 1 EIF 1 / 5 Timetable Service EIF 1 / 1 Local Time EIF 1 / 3 GIS Application EIF 2 / 12 Ticket Shop EO 2 / 2 EO 3 / 3 EO 1 / 2 ILF 1 / 2 Home Destinations  Less complicated than UML Sequence Diagrams  Suits business better  Better boundary identification between different layers
  • 17.
    17 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Prepare Destination in the IFPUG Model  It is not obvious what happens  Two elementary process seem unrelated  Nevertheless, they update an ILF  [Uh, they forgot the elementary process for deleting obsolete home destinations!] Show Matching Destinations Enter New Destination Boundary IFP=19 EI EQ EO ILF EIF ILF 1 / 2 Home Destinations EO 1 / 2 EI 1 / 3 EIF 2 / 12 Ticket Shop
  • 18.
    18 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Buy Ticket in the IFPUG Model  Elementary processes stay  The ILF ‘Home Destinations’ becomes an EIF  Adding the two counts yields 19 + 42 > 51  IFPUG is useless for functional sizing when developing Apps or counting defects  IFPUG 4.3 is great for identifying system boundaries splitting apps! Request Ticket Confirm Purchase Boarding Station Mobile Ticket Select Destination Boundary IFP=42 EI EQ EO ILF EIF EIF 1 / 5 Timetable Service EIF 1 / 1 Local Time EIF 1 / 3 GIS Application EIF 1 / 2 Home Destinations EIF 2 / 12 Ticket Shop EI 2 / 2 EI 1 / 1 EO 2 / 2 EO 2 / 3 EQ 1 / 2
  • 19.
    19 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Requirements for a Defect Measurement Reference Model  It must be additive  The total size must be the sum of the components’ sizes.  It must be understandable  Support UML Sequence Diagrams  Works well with UML Use Cases  Also with Business Process Model & Notation (BPMN) 2.0  It must fit into agile delivery  Buglione-Trudel Matrix for managing agile projects  Story Cards for Sprints Story Card for Prepare Destinations Test is Ready Draft is Ready Review Done Final-ized Appro-ved Func-tional R001-01: Prepare Destinations Refactoring Count: 5 6 x reworked 0 0 Story Points: Functional Size: Business Impact: App User Prepare Destinations Standard Stations Ticket Shop 1.// Add New Destination 2.// Check whether Destination Exists 3.// Collect Matching Destinations 4.// Show Matching Destinations 5.// Select Exact Destination 6.// Record Selected Destinations The user prepares his ticket app by entering standard destinations. The ticket shop is involved for resolving station names matches and avoid spelling errors
  • 20.
    20 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Learn & Remember Count Defects based on Reference Model Software Development the Six Sigma Way Use COSMIC ISO/IEC 19761 Benchmark!
  • 21.
    21 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions What is a Test?  A Software Test has  Several Test Stories • Weighted by Customer’s Priority for the Test Story, reflecting the value for the customer  Each Test Story has many Test Cases • For various kind of test data  A Test Size attribute • Number of data movements executed by test cases  A Test Coverage attribute • Percentage of data movements covered with test cases  An Outcome • Passed or Failed – Passed: All responses according expectations – Failed: at least one test case didn’t yield the expected response
  • 22.
    CT-B Ticketing CT-B.1Select Destination CT-B.1.1 Station of origin known Asks for destination CT-B.1.2 No destination selected Stops without contacting Ticket Shop CT-B.2 Get Ticket CT-B.2.1 Both stations known Asks to confirm price CT-B.2.2 Boarding station undefined Doesn't ask for destination CT-B.3 Price Calculation CT-B.3.1 Both stations known Presents price of next actual connection CT-B.3.2 Both stations known Display intermediate change stations CT-B.4 Issue Ticket CT-B.4.1 Issue successful, ticket shown All credentials visible CT-B.4.2 Ticket Shop blocked Explanation given CT-B.5 Payment Tests CT-B.5.1 Credit limit exceeded Returns exceed notice CT-B.5.2 Payment service out of order Returns warning 22 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Sample Test Stories Test Story Case 1 Test Data Expected Response Case 2 Test Data Expected Response CT-A Prepare CT-A.1 Find Nearest Station CT-A.1.1 Enter GPS exactly Returns correct station CT-A.1.2 Enter GPS nearby Returns nearest station CT-A.2 Served Stations only CT-A.2.1 Select time of service Returns next available connection CT-A.2.2 Select time out of service Return next best served station CT-A.3 Enter New Destination CT-A.3.1 Enter valid station name Destination stored CT-A.3.2 Enter invalid station name Destination rejected  Eight Test Stories for the Ticket Apps  Each Test Story has several Test Cases  Each Test Case has defined Test Data  The Expected Response is known as per Test Case
  • 23.
    23 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions What is a Test Case?  A Test Cases has  Entry Data (“Test Data”) • Explaining the environment for the test case • Typically valid, invalid, borderline data • Normal and disturbed communication services  A known Expected Response • The response of the system is known in advance  A known sequence of data movements executed • Defining Test Coverage • Each Test Case has a Size
  • 24.
    What is aTest Case? 24 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions  The sequence of data movements can be visualized in the sequence diagram per test case  By double-clicking the respective data movements Find Route GIS Application Phone's GPS 8.// GPS Coordinates 9.// Date & Time 10.// Ask GIS 11.// Nearest Boarding Station Local Time Test Story No. 1 User Stories Test Case Measurements for Test Story CT-A.1 CT-A.1 Find Nearest Station Q001: Propose Standard Destinations Q002: Find Nearest Boarding Station Q003: Process Payment Q004: Issue Ticket Q005: Show Ticket Expected Response CT-A.1.1 Enter GPS exactly E012 E012,X009 Returns correct station CT-A.1.2 Enter GPS nearby E012,X009 Returns nearest station CT-A.1.3 Enter malformatted GPS E006 Returns no station CT-A.1.4 Enter far away GPS position E006,E007,X004,E008 E012,X009 Returns nearest station plus warning CT-A.1.5 GPS not working E005,E006 E006 Returns warning Test Story Contribution (CFP): 2 6 0 1 6 Test Size
  • 25.
    Defects Observed DataMovements Affected 25 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Defect Reporting  When a defect has been identified in a test case, it can be recorded by double-clicking the suspect sequence of data movements User Stories Test Case Measurements for Test Story CT-B.1 CT-B.1 Select Destination Q001: Propose Standard Destinations CT-B.1.1 Station of origin known R001,X001,E005 CT-B.1.2 No destination selected R001,X001 CT-B.1.3 Valid boarding & destination W001,E005,R001,X001 CT-B.1.4 List of destinations X003,R001,W001 Test Story Contribution (CFP): 12 Test Story No. 4 Expected Response CFP Name Label Description Name Label Asks for destination 6 Stops without contacting Ticket Shop 2 Returns next available connection 15 All readable and visible 3 #003 Multiple References If a station contains more than one transport mode, e.g., bus and train, both are recorded as separate destinations W001 Record Selected Destinations Test Size 26 1 Defect Count
  • 26.
    26 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions 12 Entry (E) + 9 eXit (X) + 1 Read (R) + 2 Write (W) = 24 CFP Extract Report App User Home Destinations Saved Stations Ticket Shop Ticket Purchase Phone's GPS GIS Application Timetable Service 1.// Add New Destination Traveler 2.// Check whether Destination Exists 3.// Collect Matching Destinations 4.// Show Matching Destinations 5.// Select Exact Destination 6.// Record Selected Destinations 7.// Ask App for a Ticket 8.// Propose Destinations 9.// Proposed Destinations 10.// Select Destination Prepare 11.// GPS Coordinates Select Data Movements for Test Case CT-B.1.4: List of destinations when executed in view of Q001: Propose Standard Destinations This Test Case identifies Fault #003: Multiple References affecting Data Movement 6.// Record Selected Destinations 12.// Date 13.// Ask GIS 14.// Nearest Boarding Station Finish Record Defects
  • 27.
    27 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Count Defects  You can count One Defect  Per Data Movement  Identified by One Test Story And never more… ©2012-2014 Rhafiel
  • 28.
    28 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Compare  This means  The more test stories you have, the more defects you can uncover  Test size is equally important as functional size  Defect density depends from test size  Test Benchmarking Test Status Summary Total CFP: 24 Defects Pending for Removal: 3 Test Size in CFP: 185 Defects Found in Total: 3 Test Intensity in CFP: 7.7 Defect Density: 13% Test Coverage: 79%
  • 29.
    Measuring Test Coverage 29 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Test Stories Number of Response 풚 Goal Test Coverage Find Nearest Station Served Stations only Enter New Destination Select Destination Get Ticket Price Calculation Issue Ticket Payment Tests Achieved Coverage CT-A.1 CT-A.2 CT-A.3 CT-B.1 CT-B.2 CT-B.3 CT-B.4 CT-B.5 Test Stories Deployment Combinator User Stories Q001 Propose Standard Destinations 0.52 2 9 12 8 9 7 0.55 Q002 Find Nearest Boarding Station 0.45 6 9 7 10 3 5 4 0.45 Q003 Process Payment 0.62 3 4 4 4 10 7 16 0.59 Q004 Issue Ticket 0.24 1 3 1 3 6 2 3 0.23 Q005 Show Ticket 0.29 6 3 2 1 9 3 3 0.31 Ideal Profile for Test Stories 0.18 0.20 0.26 0.40 0.38 0.51 0.36 0.40 Convergence Gap 0.18 0.2 0.2 0.4 0.4 0.5 0.4 0.4 0.06 0.10 Convergence Range 0.20 Convergence Limit Toggle Measured Controls Expected Achieved Response 풚퐸 data movements executed
  • 30.
    30 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Learn & Remember Count Defects based on Reference Model Software Development the Six Sigma Way Use COSMIC ISO/IEC 19761 Benchmark!
  • 31.
    31 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Conclusions  You need more than one model when measuring modern software  ISO/IEC 19761 COSMIC is most appropriate for measuring defects  Other methods might be suitable for cost estimations  However, in these days, what drives cost more than faulty software?  Stop the nonsense with counting defects in repositories  Include defect density goals in contracts  Do independent auditing as long as you don’t trust your measurement equipment  Start Measuring Defects. Now.
  • 32.
    32 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Questions?

Editor's Notes

  • #2 This is a short introduction to the paper “Measuring Defects in a Six Sigma Context” being published in the proceedings. Dr. Thomas Fehlmann is addicted to Six Sigma for Software since 1991. In various positions in the software industry he developed new ways to adopt Six Sigma and Lean to software development. Basis for adopting Six Sigma is the ability to measure software. Because of this, he plays a major role in the software measurement community. He received the Akao price for contributions to Quality Function Deployment in 2001, and is member of the board of QFD-ID Germany, DASMA Germany, ICTscope.ch Switzerland, and the International Software Benchmarking Standards Group ISBSG. Eberhard Kranich studied Mathematics and Computer Science, with an emphasis on Mathematical Statistics, Mathematical Optimization, and on Theory of Polynomial Complexity of Algorithms. He worked at T-Systems International GmbH in Bonn until 2013, Germany, as a Six Sigma Black Belt and Quality Assurance Manager in the context of software development.
  • #3 October 22, 2014
  • #4 Eberhard Kranich studied Mathematics and Computer Science, with an emphasis on Mathematical Statistics, Mathematical Optimization, and on Theory of Polynomial Complexity of Algorithms. He worked at T-Systems International GmbH in Bonn until 2013, Germany, as a Six Sigma Black Belt and Quality Assurance Manager, mainly in the context of software development.
  • #5 When developing software, defects are common. Developing software is a knowledge acquisition process, and from requirements to implementation many things can go wrong. Six Sigma is not usually applied to software, although safety and security of software would urgently call for our techniques. The reason is measurements. Functional size measurement has a long history, but always misused for commercial matters, namely effort estimation. Defect size usually is determined by counting the number of entries in a defect repository, usually weighted for severity. This is not a measurements, and you cannot apply Six Sigma to such stuff (although some people did).
  • #6 The basic interface is the sequence diagram. Although sequence diagrams can become large, you should use a tool that allows focusing on a selection of data movements only. Here only four objects of interest are displayed and only four out of 23 data movements. The tester should be able to step through an App by halting execution when “visiting” an object of interest, e.g., before executing a functional process. This can be achieved by test stubs inserted in the code and connected to the sequence diagram shown on the SharePoint site.
  • #7 When a defect has been identified, the respective data movement can be visually marked, e.g., by being blocked by a bug. However, such a defect might exist only under defined test data conditions. If test management confirms the existence of such a defect, it is possible to block that data movement for this particular test data or environment. Now we can define Test Size an Defect Density based on the ISO/IEC 19761 COSMIC international standard, now available in version 4.0
  • #8 However, there is a caveat: How does our software knowledge developer know whether he found all of the defects? Is test coverage sufficient? How do we determine that?
  • #9 Lines of Code definitely doesn’t address today’s needs in ICT, maybe with the exception of unit testing, when writing new code. However, you often have neither code nor exact specification when you use some services on the web.
  • #10 Counting practices on testing metrics seem missing. Despite the fact that there are some sensible questions such as “What is a Defect?”
  • #11 There has been issued a defect measurement manual by UKSMA in October 2000. The only document I found addressing the aforementioned questions how you count defects, avoiding multiple counts and loosing some of them. The Italian Software Metrics association currently translates the document in Italian. Otherwise, I didn’t found any sign of life.
  • #12 Remember from this talk one thing: Count defects based on a reference model, never as anything related to code, or to a defect repository
  • #13 As an example, consider a software for a ticket app. It closes a gap, namely between the ticket shop that can do everything, including tickets and tickets for the whole family or office outing, but rather takes a long time and doesn’t fit on a smartphone’s screen, and a small fine mobile app that does what you need in short time, with very few clicks, producing a valid ticket on the screen of your phone – but limited to the mobile-bearing person only and to a few, previously defined destinations. So you can just before the train arrives, when all the offices are closed and the vending machines are out of order, buy a valid ticket and enjoy the ride home without fear of being checked for a valid ticket and fined because you have none. We select five user stories and validate with a transfer function that they meet the Customer’s Needs found by the NPS Gemba. As a benefit, we get a priorization profile for the user stories, not by sponsors, but by the customers themselves.
  • #14 This sequence diagram depicts this very simple application software. Sequence diagrams are used as part of the UML method to specify a software or an ICT service. For us as a Six Sigma Professionals it is important that the same sequence diagrams allow to measure a software based on their functional scope. If you can measure, you can also use the usual methods of Six Sigma for construction and operation of software solutions - because you can hardly measure code, especially not software services. Services you only can assess on the basis of their functionality. The rules for measuring data movements are stated in the International Standard ISO/IEC 19’761, known as COSMIC. Basically, you count data movements between functional processes, devices, and persistent data stores, as required by some set of Functional User Requirements (FURs). Here, we chose the User Stories as FURs.
  • #15 The “Objects of Interest” represent the Functional Processes needed to execute the App; here two of them. Device Users using the App; here the interactive device, the holder of the smartphone, and its GPS device. Persistent Data resides on the smartphone. Other Applications behave as any other device users; however, they might reside on some other system or service. Note that Androids and iOS phones need an external GIS Application, Google Maps, to interpret GPS coordinates delivered by the phone. The Ticket Shop application is always external. The phone’s system clock synchronizes with local train times and thus is also considered an (external) application. All other applications are assumed to be operational and already tested; not part of the game.
  • #16 The first Functional Process prepares the list of standard destinations, what our customer might call “go home” after the party. Valid destination stations must be prepared in advance, and might not be served by transportation services at all times of the day or night.
  • #17 Looking at the IFPUG model, it immediately looks more attractive and more easily understandable than the UML sequence diagram – for business people and possibly analysts. Not for developers.
  • #18 The usual advantage when counting apps before implementing is that you detect missing requirements; see Sylvie Trudel’s doctoral thesis about “Using the COSMIC Functional Size Measurement Method (ISO 19761) as a Software Requirements Improvement Mechanism” from 2012.
  • #19 The IFPUG count is not capable to deal with modern software development, such as needed when combining services with native apps, because the count is not additive. Not only when writing apps, also in general when using an agile methodology splitting work into piece that fit into a sprint, or any other fixed-length time slot, the need for a sizing adding work units correctly is predominant. Also, defects must be counted in an additive model and thus IFPUG cannot be used. Nevertheless, the IFPUG method is not obsolete as it still provides a model easily understandable by business people, and allows for comparisons with older benchmarks.
  • #20 We can summarize our requirements for a defect counting method. Note that defects are no longer related to code; defect counting even makes sense when looking at Business Process Model notated with BPMN 2.0
  • #21 Remember from this talk two things: Count defects based on a reference model, never as anything related to code, or to a defect repository Use an additive functional sizing model; always relate defects to some function, even if quality is missing
  • #22 Testing Apps is a great deal more demanding than testing in traditional environments. One challenge is that the sheer number of platforms needed to run tests outnumbers capacity even for large organizations. Another problem is that mobile platforms works under unstable conditions. A communication once established can break down within short and be unavailable for quite some time, without neither the server nor the phone device having a particular problem.
  • #23 For our Ticket App we created eight test stories – something similar to use cases, but with a number of test cases yielding both favorable and unfavorable conditions.
  • #24 A Test Case creates a predictable environment for executing a sequence of data movements with known expected response. The test data must be reproducible as well as the environment. Because ISO/IEC 19761 COSMIC is additive, each test case has a well-defined size, and these sizes can be added up to the size of test story or the total software test.
  • #25 The tools provided actually allow recording such test cases, including test data and environment. For instance, when entering a “valid” GPS data, a table of GPS data representing public transportation stations must be available to the tester. Today it’s not.
  • #26 Similar to the above, when detecting a defect, it can be recorded directly with respect to the test case that uncovered it. Most often, a defect is uncovered by more than one test case within the same test story. It will be counted once, unless there is another test story, relating to different value to the customer that uncovers another defect.
  • #27 You can identify defects in UML sequence diagrams. The author distributes a tool based on Excel (Office 2013) that allows visually locate and record defects.
  • #28 Now, defects counting is no longer open to infinity. It ends when all test stories find one defect in all data movements. There will be never more defects than test size times functional size. That allows to use Six Sigma techniques for software & services.
  • #29 Thus, the total functional size is only one aspect – more important is the total of defects found. That allows calculating defect density but must be understood in relation to test size – and test size is much higher than functional size, reflecting the total number of times that data movements are executed in tests. This also allows for calculating an average test intensity that must be understood together with test coverage, i.e., the percentage of data movements covered by tests. This is much more meaningful than anything like code coverage when integrating services and mobile apps in your ICT solution.
  • #30 This is a sample test coverage transfer function. The measurements in the matrix cells count the number of data movements executed by test cases, and supporting one or more FUR. Thus one data movement can add to the count of several cells, although it usually doesn’t for too many. The matrix defines a transfer function 𝑨𝒙= 𝒚 𝐸 ; the “Eigensolution” 𝒙 describes the priority per Test Story with regard to customer needs; i.e., the importance of the FUR 𝒚 by eliminating measurement inconsistencies; see Thomas Saaty’s Analytic Hierarchy Process (AHP).
  • #31 Remember from this talk three things: Count defects based on a reference model, never as anything related to code, or to a defect repository Use an additive functional sizing model; always relate defects to some function, even if quality is missing Defect counting must allow for benchmarking across teams and even industries, else it’s useless for contractual quality
  • #32 Now it’s time for a change… a huge change, I suppose…
  • #33 The author has published quite a bit on the subject together with Eberhard Kranich from T-Systems in Bonn – e.g., in QFD symposia, at SW metrics conferences like MetriKon or IWSM / Mensura; also at Lean Six Sigma Conference Glasgow, Strathclyde and in Zurich. Currently, there is a book in the works: "Managing Complexity"