Test Process Maturity Measurement and Related Measurements

3,075 views

Published on

Test process from different perspective -
Test Maturity Model (TMMSM)
TMM goals and symptoms
Metrics system architecture
Goal focused measurements

Published in: Technology, Business
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,075
On SlideShare
0
From Embeds
0
Number of Embeds
143
Actions
Shares
0
Downloads
163
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Test Process Maturity Measurement and Related Measurements

  1. 1. TEST PROCESS MATURITY AND RELATED MEASUREMENTSM C Nagarajnagaraj@stagsoftware.com in.linkedin.com/in/mcnagaraj
  2. 2. Contents Test process from different perspective Test Maturity Model (TMMSM ) TMM goals and symptoms Metrics system architecture Goal focused measurements TMM SM is a registered service mark of Illinois Institute of Technology, ChicagoCopyright 2011. STAG Software Pvt Ltd. All rights reserved. 2
  3. 3. Test Process looked from different perspective Methodology Technology and tools PeopleCopyright 2011. STAG Software Pvt Ltd. All rights reserved. 3
  4. 4. Test Process looked from different perspective Methodology What are the key testing methods? When do we use them? How do we implement them? How do we prevent defect migration and improve defect-finding capabilities at all levels? How can we adapt standard terminologies and test standards? How do we create meaningful test plans and other critical test deliverables? How do we identify and prioritize potential software test process improvements?Copyright 2011. STAG Software Pvt Ltd. All rights reserved. 4
  5. 5. Test Process looked from different perspective Technology and tools • How can we use risk analysis techniques to decide on goals for testing? • How can we design adequate test cases? • How can we decide on what types of test required to find what types of bugs? • What measurements to collect and analyze to control test process? • How can we select test tools and vendors? • How to implement an effective tools program? • What are the most significant opportunities for automating test process? • How to get better ROI for investment made on tools?Copyright 2011. STAG Software Pvt Ltd. All rights reserved. 5
  6. 6. Test Process looked from different perspective People • Do we understand the attitude required in people involved in testing? • Do we know what training required for people involved in testing? • Do people in the organization believe that the quality of test process determines the success of overall test process? • Do we understand the development team also has role in testing? • Do people in the organization feel that test engineers pursue faults not the people? • Do we understand the business value of effective testing? • Does management team support continuous test process improvement?Copyright 2011. STAG Software Pvt Ltd. All rights reserved. 6
  7. 7. Effective test process! • Is there a way through which we can prove test assets created are indeed complete and effective? • Is there a defect detection technology one can use at each stage of SDLC to uncover defects effectively? • How can we assess accurately where we stand with respect to quality goals? • Can we guarantee clean software to our customers? • Is it possible to assess how much to retest when code change? • Which, where, when and how much to automate in test engineering activities? Have a look at HBT (Hypothesis Based Testing) at www.stagsoftware.comCopyright 2011. STAG Software Pvt Ltd. All rights reserved. 7
  8. 8. Internal structure of TMM Levels Indicate Contain Maturity Goals Testing capability Supported by Maturity sub-goals Achieved by Address Activities/tasks/resp. Implementation Organized by and organization adaptation Critical views Manager Dev/tester User/clientCopyright 2011. STAG Software Pvt Ltd. All rights reserved. 8
  9. 9. TMM levels • Test process optimization Level 5: Optimization/Defect • Quality control Prevention and Quality Control • Application of process data for defect prevention • Software quality evaluation Level 4: • Establish a test measurement program Management and Measurement • Establish an organization wide review program • Control and monitor the testing process Level 3: • Integrate testing into the software life cycle Integration • Establish a technical training program • Establish a software test organization • Institutionalize basic testing techniques & methods Level 2: • Initiate a test planning process Phase Definition • Develop testing and debugging goals Level 1: InitialCopyright 2011. STAG Software Pvt Ltd. All rights reserved. #
  10. 10. Metrics system architecture GOALS Questions Metrics Measurements Process-1 Process-2 Process-n Customer Basili’s GQM Paradigm is goal centered frameworkCopyright 2011. STAG Software Pvt Ltd. All rights reserved. 10
  11. 11. TMM Level 1 • Testing is a chaotic process • Not distinguished from debugging • Documented set of specification not available • Tests are developed in ad-hoc way after coding is completed • The objective of testing is to show that software worksCopyright 2011. STAG Software Pvt Ltd. All rights reserved. 11
  12. 12. Measurements at L1 As there is no specific goals at this maturity level we can see whether we have answers for these questions 1. How do you know the size of software tested? 2. How do you categorize projects as big, medium and small projects? 3. Do you know how many defects were found in the project? 4. How much effort spent for activity coming under heading “testing”? Size: Defects: Size of code in KLOC High, medium, low severity Number of requirements or features defects count Number of test cases developed Defects/KLOC Number of test cases executed Cost: Total effort spent for testingCopyright 2011. STAG Software Pvt Ltd. All rights reserved. 12
  13. 13. TMM Level 2 TMM Level 2 goals Characteristics 2.1 Develop Testing and There is a clear separation Debugging Goals and Policies between debugging and testing phase 2.2 Initiate a Test Planning It is a planned activity in project Process plan Plan starts after coding is 2.3 Institutionalize basic complete testing techniques and Basic testing techniques in place methods Testing is multi-leveledCopyright 2011. STAG Software Pvt Ltd. All rights reserved. 13
  14. 14. Possible questions to check on goals status • How much time planned for all levels of testing? • How much time spent in testing activities? • How much time spent in debugging and fixing different types of bugs? • What percentage of code coverage targeted and achieved? • Did we uncover different types of defects at right stage in the SDLC?Copyright 2011. STAG Software Pvt Ltd. All rights reserved. 14
  15. 15. Measurements at L2 Time/effort related measurement: Time/effort spent is test planning Time/effort spent in unit, integration, system, regression testing Total time/effort spent in testing activities (Granularity in above measurements also possible like time/effort spent in test design for unit / integration/ system tests) Number of planned test cases, unplanned test cases Planned/actual degree of statement coverage Defects: Number of defects in each phase (SDLC Phases) Number of defects found in each level of testing (UT, IT, ST, UAT) Number of each type of defects found Time taken to fix and re-test each defects typeCopyright 2011. STAG Software Pvt Ltd. All rights reserved. 15
  16. 16. TMM Level 3 TMM Level 3 goals Characteristics 3.1 Establish a Test There is an established test Organization organization Testing is integrated to SDLC 3.2 Establish a technical Test plan is developed, tracked training program and controlled (Integrated with project plan) 3.3 Integrate testing into the Test engineers drive test process software life cycle improvement Users/clients attend milestone 3.4 Control and monitor the meeting testing process User/clients support in developing usability test plansCopyright 2011. STAG Software Pvt Ltd. All rights reserved. 16
  17. 17. Possible questions to check on goals status • How many people are in testing team at different levels? • How many hrs of training the team go through for competency development for different levels? • How much time spent in all levels of testing against plan? • How many projects using tools invested effectively? • Do you know what percentage of defects escaping UT/IT (internal) and ST (External)? • Are we under control for planned budgets for testing in all projects? • Is team productivity assumed matching with actual observed?Copyright 2011. STAG Software Pvt Ltd. All rights reserved. 17
  18. 18. Measurements at L3 • Coverage related measurement Requirement, statement, branch coverage Types of test planned and completed • Productivity related measurement Number of Test cases written / Per unit time Test cases executed /Per unit time • Training related measurement Number of training hrs attended / year (for all test professionals) • ROI on tools initiatives Total cost saved by automation/ Total tool program cost • Defect escapes at different levels of testing Unit Test escape = (Total UT bugs found in ST/ Total ST bugs) External Test escape = (Total bugs found in UAT / Total bugs found in all levels • Number of Users/Clients formal interactionsCopyright 2011. STAG Software Pvt Ltd. All rights reserved. 18
  19. 19. TMM Level 4 TMM Level 4 goals Characteristics 4.1 Establish an organization Review program is effective wide review program Reviews are planned activity in project plan 4.2 Establish a test Measurements collection and measurement program analysis process effective Quality attributes of a product 4.3 Software quality are well-defined and measured evaluationCopyright 2011. STAG Software Pvt Ltd. All rights reserved. 19
  20. 20. Possible questions to check on goals status • Is review activities conducted formally? • How many are trained on inspection process? • Is formal inspection effective? • Hope measurements to be collected in all projects are agreed upfront and collected? • Are measurements collected are analyzed? • Is there an organization critical metrics defined with baseline value? • Do we get clear visibility on quality at each review?Copyright 2011. STAG Software Pvt Ltd. All rights reserved. 20
  21. 21. Measurements at L4 • Number of inspection leaders available • Number of people trained on inspection • Size of the item inspected • Time spent on inspection activities • Number of defects found per hour of review time • Effort spent on different types of test • Quality levels/quality index reached at each milestone review against plan • Measurement collections against plan • # of corrective action if required based on analysis of measurements collectedCopyright 2011. STAG Software Pvt Ltd. All rights reserved. 21
  22. 22. TMM Level 5 TMM Level 5 goals Characteristics 5.1 Defect prevention Data from all projects are collectively analyzed Critical defects types are 5.2 Quality control analyzed thoroughly Quality control concepts are adopted Right tools are inserted 5.3 Test process optimization progressively in the organizationCopyright 2011. STAG Software Pvt Ltd. All rights reserved. 22
  23. 23. Possible questions to check on goals status • Is organization critical measurements are coming from all projects? • Do we understand various types of defects we uncover and where we need to act for prevention? • Do we have clear control limits defined for critical metrics in the organization for key processes? • Are you happy with ROI on tools invested? • Is senior management support for spending effort process improvements evident?Copyright 2011. STAG Software Pvt Ltd. All rights reserved. 23
  24. 24. Measurements at L5 • Time/Effort spent in defect causal analysis • Number of actions suggested • Effort/cost for implementing action plans • Costs of statistical testing • Effort/costs of training SEPG team in process control • Effort/costs on quantitative process analysis • Number of process changes • Number of new tools introduced in the organizationCopyright 2011. STAG Software Pvt Ltd. All rights reserved. 24
  25. 25. References 1. Practical Software Testing by Ilene Burnstein 2. Software Metrics by C. R. PandianCopyright 2011. STAG Software Pvt Ltd. All rights reserved. 25
  26. 26. Thank you! Web: www.stagsoftware.com Blog: www.stagsoftware.com/blog Twitter: @stagsoftCopyright 2011. STAG Software Pvt Ltd. All rights reserved. 26

×