Overview What we can measure using metrics Common metrics to evaluate test process why do we need to use metrics Test metrics life cycle (TMLC) Type of metrics Fundamental testing metrics

1. SOFTWARE
TESTING METRICS
“A METRIC IS A QUANTITATIVE MEASURE OF THE DEGREE TO WHICH A SYSTEM,
SYSTEM COMPONENT, OR PROCESS POSSESSES A GIVEN ATTRIBUTE”
DAVID TZEMACH
WWW.MACHTESTED.COM
MAY 12 2017

2. AGENDA
• OVERVIEW
• WHAT WE CAN MEASURE USING METRICS
• COMMON METRICS TO EVALUATE TEST PROCESS
• WHY DO WE NEED TO USE METRICS
• TEST METRICS LIFE CYCLE (TMLC)
• TYPE OF METRICS
• FUNDAMENTAL TESTING METRICS

3. METRICS AND WHAT IT
ACTUALLY MEANS?

4. OVERVIEW
ONE OF THE MOST IMPORTANT THINGS IN ANY PROJECT IS TO DETERMINE THE FACTORS OF SUCCESS, THESE FACTORS CAN BE
BASED ON QUALITY, COSTS AND TIME.
IN SOFTWARE TESTING PROJECTS, WE WILL NEED TO MEASURE THE QUALITY THAT CAN BE BASED ON MANY CRITERIA INCLUDING
THE 3 DESCRIBED ABOVE.
EXAMPLES
• SPRINT QUALITY IN AGILE PROJECTS (A VELOCITY IS A BUILT-IN TOOL THAT WE CAN USE TO DETERMINE IT).
• THE PROGRESS OF A PROJECT (% OF THE WORK THAT ALREADY COMPLETED/REMAINED).
• THE QUALITY OF TESTING ARTIFACTS (DEFECTS AND TEST DOCUMENTATION).
• SYNCHRONIZATION AMONG TEST AND DEVELOPMENT TEAMS.
• THE QUALITY OF THE PROCESS USED DURING THE PROJECT
• DEVIATIONS FROM THE ORIGINAL PLAN.

5. COMMON TEST METRICS

6. WHAT ARE THE MOST COMMON TEST METRICS
• HOW LONG WOULD IT TAKE TO COMPLETE THE TESTING PROCESS?
• HOW MANY DEFECTS ARE OPENED FOR THE SAME ROOT CAUSE?
• HOW MUCH EFFORT WENT INTO THE TESTING OF THE PRODUCT?
• HOW MANY TEST CASES ARE DESIGNED PER REQUIREMENT?
• HOW MANY TEST CASES ARE DESIGNED PER RISK (RBT)?
• TEST COVERAGE COMPARED TO THE ORIGINAL ESTIMATIONS.
• THE MEMBER OF TEST CASES THAT WERE EXECUTED.
• DEFECTS DIVIDED BY SEVERITY.
• DEFECTS DIVIDED BY PRIORITY.
• ANY TESTING DELAYS.
• TEST PRODUCTIVITY.

7. WHY DO WE NEED TO USE
METRICS?

8. SO WHY SHOULD WE THINK ABOUT TEST
METRICS?
TESTING METRICS ARE THE MOST EFFICIENT AND IMPORTANT WAY TO MEASURE THE QUALITY OF A TESTING PROJECT, IN
ADDITION, WE CAN SPECIFY A FEW MORE REASONS SUCH AS:
• WITHOUT USING METRICS, YOU CANNOT MEASURE THE QUALITY OF THE IMPLEMENTED PROCESS.
• USING TEST METRICS, MANAGERS CAN TAKE DECISIONS ON DIFFERENT PHASES OF THE TESTING PROJECTS AND
CONTINUALLY IMPROVE THE PROCESS (NO MATTER WHAT IS THE TESTING METHODOLOGY).
• MANAGERS CAN JUSTIFY HARD DECISIONS THAT THEY NEED TO TAKE DURING THE PROJECT.
• MANAGERS CAN EVALUATE THE RISKS AND DETERMINE FUTURE PLANS TO REMOVE THEM.
• TEST METRICS WILL HELP TO MONITOR THE SUCCESS FACTORS OF THE TESTING PROJECT (I ALREADY REVIEWED SOME
OF THEM IN THE PREVIOUS PARAGRAPH).

9. TEST METRICS
LIFE CYCLE

10. PHASES OF TMLC
ID Phase name
1 Analysis The analysis phase is the first phase of the TMLC were we determine the metrics that
will use to measure the project/process/testing success.
2 Communicate The first thing that we will need to perform during this phase, is to explain the
metrics (And which data should be captured to support them) to the resources
in the testing projects (Product managers, team leaders and off course the testing
teams).
3 Evaluation We will analysis the data captured by the test team and once it’s approved we will use
to as an input to the identified metrics.
4 Report During the last phase of the TMLC, we will generate a report that summarizes the
results and conclusions of the test metrics, publish it to all relevant resources and take
feedback that we will use to improve the process.

11. TYPES OF METRICS

12. BASE METRICS (DIRECT MEASURE)
BASE METRICS ARE THE METRICS WHICH ARE DETERMINED FROM THE RAW DATA
GENERATED BE THE TESTERS THROUGHOUT THE TEST LIFE CYCLE (TLC), THESE DATA
INCLUDE INFORMATION ABOUT DEFECTS STATUSES, TEST COVERAGE AND MORE.
IN ADDITION, USING THIS RAW DATA WE CAN PROVIDE INPUTS TO THE FORMULAS
USED TO DERIVE CALCULATED METRICS.

13. CALCULATED METRICS (INDIRECT MEASURE)
CALCULATED METRICS ARE BASED ON THE DATA GENERATED IN BASE METRICS
(CONVERSION OF BASE METRICS DATA INTO MORE USEFUL INFORMATION).
USING THIS TYPE OF METRIC, THE TEST LEAD CAN TRACK AND MEASURE THE TESTING
PROGRESS AT DIFFERENT PHASES OF THE TESTING PROCESS.

14. TYPES OF METRICS

15. OVERVIEW
LET’S REVIEW SOME OF THE MOST COMMON METRICS THAT WE CAN USE TO DETERMINE
DIFFERENT ASPECTS OF THE TESTING QUALITY, AS YOU WILL SEE, EACH METRIC IS DESCRIBED AS A
SIMPLE MATHEMATICAL FUNCTION.
PRIOR TO USING THIS MATHEMATICAL FUNCTIONS, WE NEED TO UNDERSTAND THE IMPORTANCE OF
THE ABSOLUT NUMBERS DERIVED FROM THE ACTUAL TESTING PROCESS, THIS NUMBERS ARE BASED
ON A FEW BASIC CRITERIA (TOTAL NUMBER OF TEST, NUMBER OF FAILED/PASSED TESTS ETC.) THAT
WE WILL USE AS THE METRIC INPUT, SO THERE IS A HUGH IMPORTANT TO USE THE REAL NUMBERS.

16. GROUP A: TEST EXECUTION BASED METRICS
METRIC 1: MEASURING THE % OF PASSED TESTS
(PASSED TESTS) / (TOTAL NUMBER OF TEST EXECUTED) *100 = % OF THE PASSED TEST CASES
METRIC 2: MEASURING THE % OF FAILED TESTS
(FAILED TESTS) / (TOTAL NUMBER OF TEST EXECUTED) *100 = % OF THE FAILED TEST CASES
METRIC 3: MEASURING THE % OF THE REMAINING TESTS
(TESTS THAT ARE NOT EXECUTED) / (TOTAL NUMBER OF TESTS) *100 = % OF THE REMAINING TEST CASES
METRIC 4: MEASURING THE % OF THE EXECUTED TESTS
(TESTS THAT ARE EXECUTED) / (TOTAL NUMBER OF TESTS) *100 = % OF THE EXECUTED TEST CASES

17. GROUP B: METRICS USED TO MEASURE DEFECTS STATUSES
METRIC 5: MEASURING THE % OF FIXED DEFECTS
(NUMBER OF FIXED DEFECTS) / (TOTAL NUMBER OF DEFECTS REPORTED PER SPRINT/PHASE) *100 = % OF THE FIXED DEFECTS
METRIC 6: MEASURING THE % OF INVALID DEFECTS
(NUMBER OF INVALID DEFECTS) / (TOTAL NUMBER OF DEFECTS REPORTED PER SPRINT/PHASE) *100 = % OF THE INVALID DEFECTS
METRIC 7: MEASURING THE % OF WONTFIX DEFECTS
(NUMBER OF WONTFIX DEFECTS) / (TOTAL NUMBER OF DEFECTS REPORTED PER SPRINT/PHASE) *100 = % OF THE WONTFIX DEFECTS
METRIC 8: MEASURING THE % OF CANNOT REPRODUCE DEFECTS
(NUMBER OF CANNOT REPRODUCE DEFECTS) / (TOTAL NUMBER OF DEFECTS REPORTED PER SPRINT/PHASE) *100 = % OF THE CANNOT
REPRODUCE DEFECTS
METRIC 9: MEASURING THE % OF DEFERRED DEFECTS
(NUMBER OF DEFERRED DEFECTS) / (TOTAL NUMBER OF DEFECTS REPORTED PER SPRINT/PHASE) *100 = % OF THE DEFERRED DEFECTS

18. GROUP C: METRICS USED TO MEASURE DEFECTS SEVERITY
METRIC 11: MEASURING THE % OF BLOCKER DEFECTS
(BLOCKER DEFECTS) / (TOTAL NUMBER OF DEFECTS REPORTED PER SPRINT/PHASE) *100 = % OF THE BLOCKERDEFECTS
METRIC 12: MEASURING THE % OF CRITICAL DEFECTS
(CRITICAL DEFECTS) / (TOTAL NUMBER OF DEFECTS REPORTED PER SPRINT/PHASE) *100 = % OF THE CRITICALDEFECTS
METRIC 13: MEASURING THE % OF MAJOR DEFECTS
(MAJOR DEFECTS) / (TOTAL NUMBER OF DEFECTS REPORTED PER SPRINT/PHASE) *100 = % OF THE MAJORDEFECTS

19. GROUP D: METRICS USED FOR TIME MEASUREMENT
METRIC 14: AVERAGE TIME FOR DEVELOPMENT TO FIX DEFECTS
(TOTAL TIME USED FOR BUG FIXES) / (TOTAL NUMBER OF DEFECTS REPORTED PER SPRINT/PHASE)
METRIC 15: NUMBER OF TEST EXECUTION FOR A SPECIFIC TIME PERIOD
(NUMBER OF TEST RUN) / (TOTAL TIME)

20. FOR ADDITIONAL KB’S PLEASE
VISIT MY BLOG
WWW.MACHTESTED.COM