Iwsm2014 defect density measurements using cosmic (thomas fehlmann)
•Download as PPTX, PDF•
1 like•333 views
This document discusses measuring defects in software development using the COSMIC functional size measurement method. It provides background on COSMIC and how it can be used to count defects based on the number of data movements affected. The document examines what constitutes a defect and how defect density and test coverage can be measured. It advocates using COSMIC to benchmark defects as it is an additive method that is understandable and fits with agile delivery.
Recommended
More Related Content
Viewers also liked
Viewers also liked (19)
Similar to Iwsm2014 defect density measurements using cosmic (thomas fehlmann)
Similar to Iwsm2014 defect density measurements using cosmic (thomas fehlmann) (20)
More from COSMIC - Common Software Measurement International Consortium
More from COSMIC - Common Software Measurement International Consortium (16)
Recently uploaded
Recently uploaded (20)
Iwsm2014 defect density measurements using cosmic (thomas fehlmann)
- 1. 1 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Defect Density Measurements Using COSMIC Thomas M. Fehlmann, Zürich Eberhard Kranich, Duisburg Euro Project Office AG E: info@e-p-o.com H: www.e-p-o.com
- 2. 2 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Dr. Thomas Fehlmann 1981: Dr. Math. ETHZ 1991: Six Sigma for Software Black Belt 1999: Euro Project Office AG, Zürich 2001: Akao Price 2001 for original contributions to QFD 2003: SwissICT Expert for Software Metrics, ICTscope.ch 2004: Member of the Board QFD Institute Deutschland – QFD Architect 2007: CMMI for Software – Level 4 & 5 2011: Net Promoter® Certified Associate 2012: Member of the DASMA Board 2013: Vice-President ISBSG 1981: Dr. Math. ETHZ 1991: Six Sigma for Software Black Belt 1999: Euro Project Office AG, Zürich 2001: Akao Price 2001 for original contributions to QFD 2003: SwissICT Expert for Software Metrics, ICTscope.ch 2004: Member of the Board QFD Institute Deutschland – QFD Architect 2007: CMMI for Software – Level 4 & 5 2011: Net Promoter® Certified Associate 2012: Member of the DASMA Board 2013: Vice-President ISBSG Dr. Thomas Fehlmann
- 3. 3 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Mathematics and Computer Science Emphasis on Mathematical Statistics Mathematical Optimization Theory of Polynomial Complexity of Algorithms Working at T-Systems International GmbH in Bonn, Germany Six Sigma Black Belt for Software Development Software Quality Assurance Manager Member of the DASMA Board Eberhard KranichEberhard Kranich Mathematics and Computer Science Emphasis on Mathematical Statistics Mathematical Optimization Theory of Polynomial Complexity of Algorithms Worked at T-Systems International GmbH in Bonn, Germany Six Sigma Black Belt for Software Development Software Quality Assurance Manager Member of the DASMA Board
- 4. 4 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions What is a Defect? Defect = Behavior impacting expected or required functionality of software How many bugs? By counting the size of defect repositories? By number of entries???
- 5. 5 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Software Testing as a Game Tester sees selected sequences in the UML sequence diagram Tester can “walk” the data movements when planning or executing tests Functionality becomes visible to the agile team Defects impacting functionality become visible to testers Functional Process Other Application Some Device 8.// Move some data 9.// Move some data 10.// Move some data 11.// Move some data Other Device
- 6. 6 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Functionality, Defect Size, and Defect Density What happens if data movements have defects? Testers mark the data movement where a defect has been detected Same Metric: ISO/IEC 19761 COSMIC Functional Process Other Application Some Device 8.// Move some data Move some data 10.// Move some data 11.// Move some data Other Device Functional Size Number of Data Movements needed to implement all FUR Test Size Number of Data Movements executed in Tests Test Story Collection of Test Cases aiming at certain FURs Defect Count Number of Data Movements affected by some defect detected in a test story
- 7. 7 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Defects Density Prediction? Now he counts the defects! And counts and adjusts test size By ISO/IEC 19761 COSMIC Functional Process Other Application Some Device 8.// Move some data 9.// Move some data 10.// Move some data 11.// Move some data Other Device How does he know that he found all the defects?
- 8. 8 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Research Questions What is Defect Density? Defects per KDLOC? What is Test Coverage? Code lines executed by some test case?
- 9. 9 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions SW Testing and SW Metrics Counting practices for defect counting are undocumented “Number of Defects Found” per Stages / with Tests / etc. How do you count “Number of Defects”? Is it simply the number of entries in a defect repository? How can you avoid double reporting? Or make sure two defects are reported twice and not in a single report? A successor to the “Defect Measurement Manual” published by UKSMA in October 2000 is under review: “Defect Measurement and Analysis Handbook” By European cooperation Important enhancement for ISBSG’s Data Collection!
- 10. 10 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions SW Testing and SW Metrics Counting practices for defect counting are undocumented “Number of Defects Found” per Stages / with Tests / etc. How do you count “Number of Defects”? Is it simply the number of entries in a defect repository? How can you avoid double reporting? Or make sure two defects are reported twice and not in a single report? A successor to the “Defect Measurement Manual” published by UKSMA in October 2000 is under review: “Defect Measurement and Analysis Handbook” By European cooperation Important enhancement for ISBSG’s Data Collection!
- 11. 11 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Learn & Remember Software Development the Six Sigma Way Use COSMIC ISO/IEC 19761 Benchmark! Count Defects based on Reference Model
- 12. 12 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions As a … [Functional User] I want to … [get something done] Such that …[quality characteristic] So that … [value or benefit] 1) Q001 Propose Standard Destinations User of public transportation be able to store my preferred destinations they are valid for the Ticket Shop I no longer have to pay fees when catched without tickets 2) Q002 Find Nearest Boarding Station User of public transportation locate nearest station with GPS that's being served right now I immediately can see whether it's right 3) Q003 Process Payment Provider of transportation services give user access to their preferred payment options all payments are traceable in Ticket Shop they can manage spending 4) Q004 Issue Ticket User of public transportation get a valid ticket with settings from Ticket Shop I no longer have to pay fees when catched without tickets 5) Q005 Show Ticket Ticket controller see the validated ticket I can check validity period and travel range I don't need to go into a dispute with a client Functional User Requirements Example: The Ticket Apps Customer’s Voice: Give me a ticket subito! Without much ado and questions if I simply want to go home
- 13. 13 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions The Ticket Apps 12 Entry (E) + 9 eXit (X) + 1 Read (R) + 2 Write (W) = 24 CFP App User Home Destinations Saved Stations Ticket Shop Ticket Purchase Phone's GPS GIS Application Timetable Service Loca 1.// Add New Destination Traveler 2.// Check whether Destination Exists 3.// Collect Matching Destinations 4.// Show Matching Destinations 5.// Select Exact Destination 6.// Record Selected Destinations 7.// Ask App for a Ticket 8.// Propose Destinations 9.// Proposed Destinations 10.// Select Destination Prepare 11.// GPS Coordinates 12.// Date & Time 13.// Ask GIS 14.// Nearest Boarding Station
- 14. 14 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions The Ticket Apps – Objects of Interest Description Type Wants to buy a ticket to ride Device User Store a list of destinations that are valid for "getting home" Functional Process List of standard stations eligible as travel destination Persistent Data Finds nearest origin station Other Application Buy the ticket needed for that route Functional Process External ticket shop able to identify the App’s user Other Application Finds connections between stations at a given time & day Other Application Provides GPS coordinates where phone is located Device User Synchronized time service located in smartphone Device User App User Home Destinations Ticket Purchase Ticket Shop Phone's GPS Timetable Service Saved Stations GIS Application Local Time
- 15. 15 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions The Ticket Apps – Prepare Destinations App User Ticket Shop 1.// Add New Destination Prepare 2.// Check whether Destination Exists 3.// Collect Matching Destinations 4.// Show Matching Destinations 5.// Select Exact Destination Home Destinations Saved Stations 6.// Record Selected Destination Preparation: Store standard destination stations on Smartphone After checking with the Ticket Shop whether they exist or which stop to select Involves Ticket Shop and local data ‘Standard Stations”
- 16. 16 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions The Ticket App in the IFPUG Model Boundary IFP=51 EI EO EQ ILF EIF EQ 1 / 2 Select Destination EI 1 / 3 Enter New Destination EI 2 / 2 Request Ticket EI 2 / 1 Confirm Purchase EIF 2 / 12 Ticket Shop EIF 1 / 3 GIS Application EIF 1 / 5 Timetable Service EIF 1 / 1 Local Time EO 2 / 2 Boarding Station EO 3 / 3 Mobile Ticket EO 1 / 2 Show Matching Destinations ILF 1 / 2 Home Destinations Less complicated than UML Sequence Diagrams Suits business better Better boundary identification between different layers
- 17. 17 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Prepare Destination in the IFPUG Model It is not obvious what happens Two elementary process seem unrelated Nevertheless, they update an ILF [Uh, they forgot the elementary process for deleting obsolete home destinations!] Boundary IFP=19 EI EO EQ ILF EIF ILF 1 / 2 Home Destinations EO 1 / 2 Show Matching Destinations EI 1 / 3 Enter New Destination EIF 2 / 12 Ticket Shop
- 18. 18 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Buy Ticket in the IFPUG Model Elementary processes stay The ILF ‘Home Destinations’ becomes an EIF Adding the two counts yields 19 + 42 > 51 IFPUG is useless for functional sizing when developing Apps or counting defects IFPUG 4.3 is great for identifying system boundaries splitting apps! Boundary IFP=42 EI EO EQ ILF EIF EIF 2 / 12 Ticket Shop EIF 1 / 3 GIS Application EIF 1 / 5 Timetable Service EIF 1 / 1 Local Time EIF 1 / 2 Home Destinations EI 2 / 2 Request Ticket EI 1 / 1 Confirm Purchase EO 2 / 2 Boarding Station EO 2 / 3 Mobile Ticket EQ 1 / 2 Select Destination
- 19. 19 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Requirements for a Defect Measurement Reference Model It must be additive The total size must be the sum of the components’ sizes. It must be understandable Support UML Sequence Diagrams Works well with UML Use Cases Also with Business Process Model & Notation (BPMN) 2.0 It must fit into agile delivery Buglione-Trudel Matrix for managing agile projects Story Cards for Sprints Story Card for Prepare Destinations Test is Ready Draft is Ready Review Done Final- ized Appro- ved Func- tional R001-01: Prepare Destinations 6 x reworked Refactoring Count: 5 0 0 Business Impact: Functional Size: Story Points: App User Prepare Destinations Standard Stations Ticket Shop 1.// Add New Destination 2.// Check whether DestinationExists 3.// Collect Matching Destinations 4.// Show Matching Destinations 5.// Select Exact Destination 6.// Record Selected Destinations The user prepares his ticket app by entering standard destinations. The ticket shop is involved for resolving station names matches and avoid spelling errors
- 20. 20 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Learn & Remember Software Development the Six Sigma Way Use COSMIC ISO/IEC 19761 Benchmark! Count Defects based on Reference Model
- 21. 21 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions What is a Test? A Software Test has Several Test Stories • Weighted by Customer’s Priority for the Test Story, reflecting the value for the customer Each Test Story has many Test Cases • For various kind of test data A Test Size attribute • Number of data movements executed by test cases A Test Coverage attribute • Percentage of data movements covered with test cases An Outcome • Passed or Failed – Passed: All responses according expectations – Failed: at least one test case didn’t yield the expected response
- 22. 22 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Sample Test Stories Eight Test Stories for the Ticket Apps Each Test Story has several Test Cases Each Test Case has defined Test Data The Expected Response is known as per Test Case Test Story Case 1 Test Data Expected Response Case 2 Test Data Expected Response CT-A Prepare CT-A.1 Find Nearest Station CT-A.1.1 Enter GPS exactly Returns correct station CT-A.1.2 Enter GPS nearby Returns nearest station CT-A.2 Served Stations only CT-A.2.1 Select time of service Returns next available connection CT-A.2.2 Select time out of service Return next best served station CT-A.3 Enter New Destination CT-A.3.1 Enter valid station name Destination stored CT-A.3.2 Enter invalid station name Destination rejected CT-B Ticketing CT-B.1 Select Destination CT-B.1.1 Station of origin known Asks for destination CT-B.1.2 No destination selected Stops without contacting Ticket Shop CT-B.2 Get Ticket CT-B.2.1 Both stations known Asks to confirm price CT-B.2.2 Boarding station undefined Doesn't ask for destination CT-B.3 Price Calculation CT-B.3.1 Both stations known Presents price of next actual connection CT-B.3.2 Both stations known Display intermediate change stations CT-B.4 Issue Ticket CT-B.4.1 Issue successful, ticket shown All credentials visible CT-B.4.2 Ticket Shop blocked Explanation given CT-B.5 Payment Tests CT-B.5.1 Credit limit exceeded Returns exceed notice CT-B.5.2 Payment service out of order Returns warning
- 23. 23 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions What is a Test Case? A Test Cases has Entry Data (“Test Data”) • Explaining the environment for the test case • Typically valid, invalid, borderline data • Normal and disturbed communication services A known Expected Response • The response of the system is known in advance A known sequence of data movements executed • Defining Test Coverage • Each Test Case has a Size
- 24. 24 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions The sequence of data movements can be visualized in the sequence diagram per test case By double- clicking the respective data movements Find Route GIS Application Phone's GPS 8.// GPS Coordinates 9.// Date & Time 10.// Ask GIS 11.// Nearest Boarding Station Local Time What is a Test Case? Test Story No. 1 User Stories CT-A.1 Find Nearest Station Q001: Propose Standard Destinations Q002: Find Nearest Boarding Station Q003: Process Payment Q004: Issue Ticket Q005: Show Ticket Expected Response CT-A.1.1 Enter GPS exactly E012 E012,X009 Returns correct station CT-A.1.2 Enter GPS nearby E012,X009 Returns nearest station CT-A.1.3 Enter malformatted GPS E006 Returns no station CT-A.1.4 Enter far away GPS position E006,E007,X004,E008 E012,X009 Returns nearest station plus warning CT-A.1.5 GPS not working E005,E006 E006 Returns warning Test Story Contribution (CFP): 2 6 0 1 6 Test Size Test Case Measurements for Test Story CT-A.1
- 25. 25 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Defect Reporting When a defect has been identified in a test case, it can be recorded by double-clicking the suspect sequence of data movements User Stories CT-B.1 Select Destination Q001: Propose Standard Destinations CT-B.1.1 Station of origin known R001,X001,E005 CT-B.1.2 No destination selected R001,X001 CT-B.1.3 Valid boarding & destination W001,E005,R001,X001 CT-B.1.4 List of destinations X003,R001,W001 Test Story Contribution (CFP): 12 Test Case Measurements for Test Story CT-B.1 Test Story No. 4 Expected Response CFP Name Label Description Name Label Asks for destination 6 Stops without contacting Ticket Shop 2 Returns next available connection 15 All readable and visible 3 #003 Multiple References If a station contains more than one transport mode, e.g., bus and train, both are recorded as separate destinations W001 Record Selected Destinations Test Size 26 1 Defects Observed Data Movements Affected Defect Count
- 26. 26 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions 12 Entry (E) + 9 eXit (X) + 1 Read (R) + 2 Write (W) = 24 CFP App User Home Destinations Saved Stations Ticket Shop Ticket Purchase Phone's GPS GIS Application Timetable Service 1.// Add New Destination Traveler 2.// Check whether Destination Exists 3.// Collect Matching Destinations 4.// Show Matching Destinations 5.// Select Exact Destination 6.// Record Selected Destinations 7.// Ask App for a Ticket 8.// Propose Destinations 9.// Proposed Destinations 10.// Select Destination Prepare 11.// GPS Coordinates 12.// Da 13.// Ask GIS 14.// Nearest Boarding Station Extract Report Select Data Movements for Test Case CT-B.1.4: List of destinations when executed in view of Q001: Propose Standard Destinations This Test Case identifies Fault #003: Multiple References affecting Data Movement 6.// Record Selected Destinations Finish Record Defects
- 27. 27 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Count Defects You can count One Defect Per Data Movement Identified by One Test Story And never more… ©2012-2014 Rhafiel
- 28. 28 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Compare This means The more test stories you have, the more defects you can uncover Test size is equally important as functional size Defect density depends from test size Test Benchmarking Test Status Summary Total CFP: 24 Defects Pending for Removal: 3 Test Size in CFP: 185 Defects Found in Total: 3 Test Intensity in CFP: 7.7 Defect Density: 13% Test Coverage: 79%
- 29. 29 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Test Stories GoalTestCoverage FindNearestStation ServedStationsonly EnterNewDestination SelectDestination GetTicket PriceCalculation IssueTicket PaymentTests AchievedCoverage CT-A.1 CT-A.2 CT-A.3 CT-B.1 CT-B.2 CT-B.3 CT-B.4 CT-B.5 Q001 Propose Standard Destinations 0.52 2 9 12 8 9 7 0.55 Q002 Find Nearest Boarding Station 0.45 6 9 7 10 3 5 4 0.45 Q003 Process Payment 0.62 3 4 4 4 10 7 16 0.59 Q004 Issue Ticket 0.24 1 3 1 3 6 2 3 0.23 Q005 Show Ticket 0.29 6 3 2 1 9 3 3 0.31 Ideal Profile for Test Stories 0.18 0.20 0.26 0.40 0.38 0.51 0.36 0.40 Convergence Gap 0.18 0.2 0.2 0.4 0.4 0.5 0.4 0.4 0.06 0.10 Convergence Range 0.20 Convergence Limit Test Stories Deployment Combinator User Stories Toggle Measured Controls Measuring Test Coverage Achieved Response 𝒚 𝐸 Expected Response 𝒚Number of data movements executed
- 30. 30 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Learn & Remember Software Development the Six Sigma Way Use COSMIC ISO/IEC 19761 Benchmark! Count Defects based on Reference Model
- 31. 31 Customer Orientation Lean Six Sigma Agile Processes Project Estimations Transfer Functions Conclusions You need more than one model when measuring modern software ISO/IEC 19761 COSMIC is most appropriate for measuring defects Other methods might be suitable for cost estimations However, in these days, what drives cost more than faulty software? Stop the nonsense with counting defects in repositories Include defect density goals in contracts Do independent auditing as long as you don’t trust your measurement equipment Start Measuring Defects. Now.
Editor's Notes
- This is a short introduction to the paper “Measuring Defects in a Six Sigma Context” being published in the proceedings. Dr. Thomas Fehlmann is addicted to Six Sigma for Software since 1991. In various positions in the software industry he developed new ways to adopt Six Sigma and Lean to software development. Basis for adopting Six Sigma is the ability to measure software. Because of this, he plays a major role in the software measurement community. He received the Akao price for contributions to Quality Function Deployment in 2001, and is member of the board of QFD-ID Germany, DASMA Germany, ICTscope.ch Switzerland, and the International Software Benchmarking Standards Group ISBSG. Eberhard Kranich studied Mathematics and Computer Science, with an emphasis on Mathematical Statistics, Mathematical Optimization, and on Theory of Polynomial Complexity of Algorithms. He worked at T-Systems International GmbH in Bonn until 2013, Germany, as a Six Sigma Black Belt and Quality Assurance Manager in the context of software development.
- October 22, 2014
- Eberhard Kranich studied Mathematics and Computer Science, with an emphasis on Mathematical Statistics, Mathematical Optimization, and on Theory of Polynomial Complexity of Algorithms. He worked at T-Systems International GmbH in Bonn until 2013, Germany, as a Six Sigma Black Belt and Quality Assurance Manager, mainly in the context of software development.
- When developing software, defects are common. Developing software is a knowledge acquisition process, and from requirements to implementation many things can go wrong. Six Sigma is not usually applied to software, although safety and security of software would urgently call for our techniques. The reason is measurements. Functional size measurement has a long history, but always misused for commercial matters, namely effort estimation. Defect size usually is determined by counting the number of entries in a defect repository, usually weighted for severity. This is not a measurements, and you cannot apply Six Sigma to such stuff (although some people did).
- The basic interface is the sequence diagram. Although sequence diagrams can become large, you should use a tool that allows focusing on a selection of data movements only. Here only four objects of interest are displayed and only four out of 23 data movements. The tester should be able to step through an App by halting execution when “visiting” an object of interest, e.g., before executing a functional process. This can be achieved by test stubs inserted in the code and connected to the sequence diagram shown on the SharePoint site.
- When a defect has been identified, the respective data movement can be visually marked, e.g., by being blocked by a bug. However, such a defect might exist only under defined test data conditions. If test management confirms the existence of such a defect, it is possible to block that data movement for this particular test data or environment. Now we can define Test Size an Defect Density based on the ISO/IEC 19761 COSMIC international standard, now available in version 4.0
- However, there is a caveat: How does our software knowledge developer know whether he found all of the defects? Is test coverage sufficient? How do we determine that?
- Lines of Code definitely doesn’t address today’s needs in ICT, maybe with the exception of unit testing, when writing new code. However, you often have neither code nor exact specification when you use some services on the web.
- Counting practices on testing metrics seem missing. Despite the fact that there are some sensible questions such as “What is a Defect?”
- There has been issued a defect measurement manual by UKSMA in October 2000. The only document I found addressing the aforementioned questions how you count defects, avoiding multiple counts and loosing some of them. The Italian Software Metrics association currently translates the document in Italian. Otherwise, I didn’t found any sign of life.
- Remember from this talk one thing: Count defects based on a reference model, never as anything related to code, or to a defect repository
- As an example, consider a software for a ticket app. It closes a gap, namely between the ticket shop that can do everything, including tickets and tickets for the whole family or office outing, but rather takes a long time and doesn’t fit on a smartphone’s screen, and a small fine mobile app that does what you need in short time, with very few clicks, producing a valid ticket on the screen of your phone – but limited to the mobile-bearing person only and to a few, previously defined destinations. So you can just before the train arrives, when all the offices are closed and the vending machines are out of order, buy a valid ticket and enjoy the ride home without fear of being checked for a valid ticket and fined because you have none. We select five user stories and validate with a transfer function that they meet the Customer’s Needs found by the NPS Gemba. As a benefit, we get a priorization profile for the user stories, not by sponsors, but by the customers themselves.
- This sequence diagram depicts this very simple application software. Sequence diagrams are used as part of the UML method to specify a software or an ICT service. For us as a Six Sigma Professionals it is important that the same sequence diagrams allow to measure a software based on their functional scope. If you can measure, you can also use the usual methods of Six Sigma for construction and operation of software solutions - because you can hardly measure code, especially not software services. Services you only can assess on the basis of their functionality. The rules for measuring data movements are stated in the International Standard ISO/IEC 19’761, known as COSMIC. Basically, you count data movements between functional processes, devices, and persistent data stores, as required by some set of Functional User Requirements (FURs). Here, we chose the User Stories as FURs.
- The “Objects of Interest” represent the Functional Processes needed to execute the App; here two of them. Device Users using the App; here the interactive device, the holder of the smartphone, and its GPS device. Persistent Data resides on the smartphone. Other Applications behave as any other device users; however, they might reside on some other system or service. Note that Androids and iOS phones need an external GIS Application, Google Maps, to interpret GPS coordinates delivered by the phone. The Ticket Shop application is always external. The phone’s system clock synchronizes with local train times and thus is also considered an (external) application. All other applications are assumed to be operational and already tested; not part of the game.
- The first Functional Process prepares the list of standard destinations, what our customer might call “go home” after the party. Valid destination stations must be prepared in advance, and might not be served by transportation services at all times of the day or night.
- Looking at the IFPUG model, it immediately looks more attractive and more easily understandable than the UML sequence diagram – for business people and possibly analysts. Not for developers.
- The usual advantage when counting apps before implementing is that you detect missing requirements; see Sylvie Trudel’s doctoral thesis about “Using the COSMIC Functional Size Measurement Method (ISO 19761) as a Software Requirements Improvement Mechanism” from 2012.
- The IFPUG count is not capable to deal with modern software development, such as needed when combining services with native apps, because the count is not additive. Not only when writing apps, also in general when using an agile methodology splitting work into piece that fit into a sprint, or any other fixed-length time slot, the need for a sizing adding work units correctly is predominant. Also, defects must be counted in an additive model and thus IFPUG cannot be used. Nevertheless, the IFPUG method is not obsolete as it still provides a model easily understandable by business people, and allows for comparisons with older benchmarks.
- We can summarize our requirements for a defect counting method. Note that defects are no longer related to code; defect counting even makes sense when looking at Business Process Model notated with BPMN 2.0
- Remember from this talk two things: Count defects based on a reference model, never as anything related to code, or to a defect repository Use an additive functional sizing model; always relate defects to some function, even if quality is missing
- Testing Apps is a great deal more demanding than testing in traditional environments. One challenge is that the sheer number of platforms needed to run tests outnumbers capacity even for large organizations. Another problem is that mobile platforms works under unstable conditions. A communication once established can break down within short and be unavailable for quite some time, without neither the server nor the phone device having a particular problem.
- For our Ticket App we created eight test stories – something similar to use cases, but with a number of test cases yielding both favorable and unfavorable conditions.
- A Test Case creates a predictable environment for executing a sequence of data movements with known expected response. The test data must be reproducible as well as the environment. Because ISO/IEC 19761 COSMIC is additive, each test case has a well-defined size, and these sizes can be added up to the size of test story or the total software test.
- The tools provided actually allow recording such test cases, including test data and environment. For instance, when entering a “valid” GPS data, a table of GPS data representing public transportation stations must be available to the tester. Today it’s not.
- Similar to the above, when detecting a defect, it can be recorded directly with respect to the test case that uncovered it. Most often, a defect is uncovered by more than one test case within the same test story. It will be counted once, unless there is another test story, relating to different value to the customer that uncovers another defect.
- You can identify defects in UML sequence diagrams. The author distributes a tool based on Excel (Office 2013) that allows visually locate and record defects.
- Now, defects counting is no longer open to infinity. It ends when all test stories find one defect in all data movements. There will be never more defects than test size times functional size. That allows to use Six Sigma techniques for software & services.
- Thus, the total functional size is only one aspect – more important is the total of defects found. That allows calculating defect density but must be understood in relation to test size – and test size is much higher than functional size, reflecting the total number of times that data movements are executed in tests. This also allows for calculating an average test intensity that must be understood together with test coverage, i.e., the percentage of data movements covered by tests. This is much more meaningful than anything like code coverage when integrating services and mobile apps in your ICT solution.
- This is a sample test coverage transfer function. The measurements in the matrix cells count the number of data movements executed by test cases, and supporting one or more FUR. Thus one data movement can add to the count of several cells, although it usually doesn’t for too many. The matrix defines a transfer function 𝑨𝒙= 𝒚 𝐸 ; the “Eigensolution” 𝒙 describes the priority per Test Story with regard to customer needs; i.e., the importance of the FUR 𝒚 by eliminating measurement inconsistencies; see Thomas Saaty’s Analytic Hierarchy Process (AHP).
- Remember from this talk three things: Count defects based on a reference model, never as anything related to code, or to a defect repository Use an additive functional sizing model; always relate defects to some function, even if quality is missing Defect counting must allow for benchmarking across teams and even industries, else it’s useless for contractual quality
- Now it’s time for a change… a huge change, I suppose…
- The author has published quite a bit on the subject together with Eberhard Kranich from T-Systems in Bonn – e.g., in QFD symposia, at SW metrics conferences like MetriKon or IWSM / Mensura; also at Lean Six Sigma Conference Glasgow, Strathclyde and in Zurich. Currently, there is a book in the works: "Managing Complexity"