Istqb ctal tm

ISTQB CTAL Test Manager (TM)
A Subject Overview of the
ISTQB CTAL TM Certification
Petro Porchuk
Lohika 2010
Copyright Petro Porchuk 2010. Lohika

My Goal
 Present topics due to the Syllabus CTAL
 90% of the info have the ISTQB colour
 Give some references
 10% - Express my own Vision
http://istqb.org/download/attachments/2326555/CTAL_Syllabus_V_2007.pdf

My Main Source
Software Testing Practice:
Test Management:
A Study Guide for the Certified
Tester Exam Istqb Advanced
Level
Andreas Spillner (Author)
http://www.amazon.com/Software-Testing-Practice-Management-Certified/dp/193395213X/ref=sr_1_6?ie=UTF8&s=books&qid=1275570124&sr=8-6

The ISTQB Framework, May 2010
http://istqb.org/display/ISTQB/Certification?atl_token=CrArtYjmmm

CTAL TM – Part 1
Part 1
1. Introduction
2. Test Process and Test Tools
3. Testing in the software Life Cycle
4. Test Policy and Test Handbook
5. The Test Plan
6. Test Control
7. Assessing and Improving the Development and Test Processes
Part 2
8. Deviation Management
9. Risk Management and Risk-Oriented Testing
10. Staff Qualification and Skills
11. Test Metrics
12. Selecting and Implementing Test Tools
13. Standards Copyright Petro Porchuk 2010. Lohika

(1) The Introduction
TL should be:
 Technically Skilled
 Communication and
Language Skilled
 Good Analyst
 Pro-Active
 Brave Organizer
http://www.lestnizza.com.ua/upload/image/it_range.jpg

(2)
(2) Test process and Test Tools
Test process and Test Tools
 The Fundamental Test Process
 Different Types of Test Tools
 Tools for Management and Test Control
 Tools for Test Data and Test Script Specific.
 Tools for Static Testing
 Tools for Dynamic Testing

2.1. The Fundamental Test Process

2.2. The Test Tools
 CASE Tools (Computer Aided Software Engineering)
 CAST Tools (Computer Aided Software Testing)
 Tools for Management and Test Control
(Planning, Defect Management, Requirement Management, Test Spec. Management)
 Tools for Test Data and Test Script Specific
(Test Data Generators, Test Generators)
 Tools for Static Testing
(Static analizers, model analize spesifications, reviews tools)
 Tools for Dynamic Testing
(Debugger, drivers, stubs, test robots, L/P test tools, monitors)

”A Fool with a Tool is still a Fool!”
http://www.sungsblog.com/labels/world.php

2.3. Summary
✔ Testing must be divided into individual process steps
✔ For each phase of the Test Process Tools are available
✔ The use of the Test Tools is only of advantage if a Test
Process is controlled and defined process
✔ ”A Fool with a Tool is still a Fool”

We All Do Work Process Oriented
We All Do Work Process Oriented

(3)
(3) Testing in the Software Life Cycle
Testing in the Software Life Cycle
 Test and Development Process
 The General V-Model
 The W-Model
 Rational Unified Process (RUP)
 V-Model XT
 Extreme Programming
 Rapid Application Development (RAD)
 Dynamic System Development Method (DSDM)

3.1. Test and Development Process
QA Process should have strong interaction with the
Development Process
 Reaction to the Requirement Change
 Configuration Management
 Project Management
 The General Test Process should be adjusted to the
specific activities applied Development Processes

3.2. The General V-Model
http://www.testingexcellence.com/v-model/

3.3. The V-Model for QA
Pros:
 Emphasizes Test
Levels
 Verification
 Validation
Cons:
 Preparatory Test
Activities are missing

3.4. The W-Model
http://www.testingthefuture.net/2009/09/the-w-model/

3.5. The W-Model for QA
Pros:
 Preparatory Test
Activities are here
 Concurrency of Test
and Dev. Processes
 PREviews
 Debugging
Cons:
 Needs several Test
Managers for large
projects
 Refers only to a
System Development,
not Process Oriented

3.6. The Rational Unified Process
http://www.wittmannclan.de/ptr/cs/slcycles.html

3.7. QA in RUP
Pros:
 Test Activities begin
early and accompany
other processes
Cons:
 Which Test Activities
to be eprformed when
is not made clear

3.8. V-Model XT (eXtreme Tailoring)
https://sosa.ucsd.edu/teaching/cse294/fall2004/pres/vmxt.htm
 The current version of the V-Model is
the V-Model 97. Since 1997, no
changes or improvements were made.
In 2002, a project was initiated to
redesign and improve the existing
process model. The resulting V-Model
XT should reflect new standards and
technologies; it also should expose
significantly enhanced quality
properties such as usability,
adaptability, changeability and
scalability.
 Available since 2005.

3.9. QA in V-Model XT
Pros:
 Tailoring
 Particular attention to
Customer-Supplier
communication
 Quality Management
Cons:
 No Test Manager
role, but Quality
Manager

3.10. The Extreme Programming
http://www.extremeprogramming.org/map/project.html

3.11. QA in XP
Pros:
 Testing is at the
center of the
development effort
(Test First)
 Advanced Unit
Testing
Cons:
 Missing systematic
Test Specification
creation
 QA misses
Aceptance Testing,
it's been put on a
customer

3.12. Rapid Application Development
http://www.antrix.com/services/service_list/11_validation_computer_systems.htm

3.13. QA in RAD
Pros:
 Extensive use of
Tools
 Invloving Customer to
evaluate
Cons:
 Roles of Testing or
Test Management are
not defined, there is
”SWAT Team”
instead.
”SWAT” = Skilled Workers and Advanced Tools

3.14. Dynamic System Develpment Method
 DSDM = SCRUM
 SCRUM = RAD + (precise process steps and roles)
 Focus on Teamwork
 Close Cooperation with a Customer
 Iterative System Development
 Time Boxing
 ”MoSCoW” Principle

3.15. QA in SCRUM
Pros:
 Team driven
evolutionary process
Cons:
 Test Manager role is
not here.
 Tester is responsible
for component tests
mostly.

(3) Summary
✔TM should adopt Test Process to the chosen
Development Model
✔TM should make sure the docs are available to
have enough time for the test preparation
✔TM should make sure Test Activities start early
in the Process
✔TM should make sure the needed Test Levels
are planned and executed

(4)
(4) Test Policy and Test Handbook
Test Policy and Test Handbook
 Quality Policy and Test Policy
 Bring the Test Policy to Life
 Test Handbook

4.1. Quality Policy and Test Policy
4.1. Quality Policy and Test Policy
 Quality Policy
Expresses the particular importance that an organization
attaches to ”quality” and the demands it makes on the quality of
it's products, services and processes.
 Test Policy
A high level document describing the principles, approach and
major objectives of the organization regarding testing.
 Test Handbook
A concretization of the Test Policy
http://istqb.org/download/attachments/2326555/ISTQB+Glossary+of+Testing+Terms+2+1.pdf

4.3. Bring the Test Policy to Life
4.3. Bring the Test Policy to Life
 Relevance to Business Objective
 Be Realistic
 Adequate Maturity
 Measurability
 Liveliness

(4)
(4) Summary
Summary

A Question
A Question
What is the most Common Question for the QA
interview
?
http://profiles.friendster.com/98111238

A Question
A Question
What is the most Common Question for the QA
interview
?
What is the Test Plan and
what does it consist of?
http://profiles.friendster.com/98111238

(5)
(5) The Test Plan
The Test Plan
 General Test Plan Structure
 Defining a Test Strategy
 Test Effort Estimation
 Flat Models
 Detailed Models baced on Test Activities
 Models baced on Functional Volume
 Organization of Test Teams and Test Levels

5.1. The General Test Plan Structure
5.1. The General Test Plan Structure
Chapter According to IEEE 829
1. Test plan identifier
2. Introduction
3. Test items
4. Features to be tested
5. Features not to be tested
6. Approach
7. Item pass/fail criteria (test exit criteria)
8. Suspension criteria and resumption requirements
9. Test deliverables
10. Testing tasks
11. Environmental needs
12. Responsibilities
13. Staffing and training needs
14. Schedule
15. Risk and contingencies
16. Approvals

5.2. Defining the Test Strategy
5.2. Defining the Test Strategy

5.3. Test Effort Estimation – Flat Models
5.3. Test Effort Estimation – Flat Models
Adding a percentage flat rate to the
development effort
QA Effort = 0.5*(DEV Effort)
 Assuming development estimation is accurate
 Take into an account experience from similar
projects
 It makes sence to use several sources
 But:
Cannot be used for big projects

5.4. Test Effort Estimation – Detailed Models
5.4. Test Effort Estimation – Detailed Models
Concatenation of individual activities and
different estimation methods must be used for
each activity. Consider:
 Manpower
 Time
 HW and SW for the Test Environment
 Testware (test automation tools, etc)
 Different influencing factors
 Keep estimated tasks small
 Involve the team in estimation
 Use experience

5.5.1. Test Effort Estimation – Model Based
Estimation of the Scope or Functionality of the
test objects and establishing a mathematical
model
 FPA (Function Point Analysis)
 Count the number of functions to be developed
 Add the non-functional values
(14*(1..5)+0.65 * func)
 Transform to man*hours
The Test Effort is a result of estimates from function
points.
 But: Used only for simply numbers (of test cases)

Test Point Analysis Uses Quality Atributes (ISO 9126)
 Dynamic Test Points (FP; Quality Attributes; Function priority,
compexity, influence etc) Are multiplied with each other per
function and summed up. This reflects the dynamic test efforts.
 Static Test Points (flexibility, testability, tracebility) The total
number of FP is multiplied by a factor and added up. Static test
efforts
 Dynamic and Static test points are summed up and subsequently
weighted with environment factors (docs, testware, tools,
organization culture etc)
 The result – is the number of test hours required

(5) Summary
✔ The Test Plan is the concrete application of the Test
Handbook, mapping the strategic requirement of the
Test Policy into the Project
✔ Essential Test Plan components are:
Test Strategy
Test Effort Estimation
Test Organisation
✔ For Test Effort Estimation may be applied:
Intuition, experience, detailed itemization, formulas
✔ The WORSE estimation is NO ESTIMATION

(6)
(6) Test Control
Test Control
✔Initializing the Test Tasks
✔Monitoring the Test Process
✔Reacting to Test Results
✔Reacting to Changed Circumstances
✔Evaluating the Test Completion
✔Test Report

Standish Group Chaos Report 2009
Standish Group Chaos Report 2009
Successful Projects
32%
Late, Overbudget, Miss features
44%
Failed Projects
24%
http://www.standishgroup.com/newsroom/chaos_2009.php
"These numbers represent a downtick in the success rates from the previous study, as
well as a significant increase in the number of failures", says Jim Crear, Standish Group
CIO, "They are low point in the last five study periods. This year's results represent the
highest failure rate in over a decade"

How to Fix This?
How to Fix This?
?

How to Fix This?
How to Fix This?
?
 Total Quality Management (TQM)
 Test and Development Processes Improvement

(7)
(7) Assessing and Improving the
Assessing and Improving the
Development and Test Processes
Development and Test Processes
 General Techniques and Approaches
 Total Quality Management
 Kaizen
 Six Sigma
 Improving the Software Development Processes
 Capability Maturity Model Integration (CMMI)
 SPICE
 Evaluating the Test Processes
 Testing Maturity Model (TMM)
 Test Process Improvement (TPI)

7.1 Total Quality Management (TQM)
7.1 Total Quality Management (TQM)
QUALITY is in the center. No instructions – it's an attitude.
http://www.edrawsoft.com/TQM-Diagrams.php

7.2 Kaizen
7.2 Kaizen
(Japanese for
(Japanese for Kai "
Kai "improvement" or
improvement" or Zen
Zen "change for the better")
"change for the better")
 Intensive use of the employee suggestion system
 Appreciation and regard for employees' striving for improvement
 Established small group discussion circles addressing defects and improvement
suggestions
 ”Just in time” production to eliminate wasted effort
 5 Ss process for the improvement of workspaces:
 Seiri – tidiness; old and useless items have no place at the workspace
 Seiton – orderliness; ”everything” has its place for quick retrieval and storage
 Seiso – cleanliness; keep the workspace clean and tidy at all times
 Seiketsu – standartization of all practices and processes
 Shitsuke – discipline; all activities are performed in a discipline way
 ”Total Productive Maintenance” for maintenance and servicing of all means of
production
 BUT: difficult to apply in european culture such strong Jananese rules

7.3 Six Sigma
7.3 Six Sigma
(by Motorola, USA in 1981)
(by Motorola, USA in 1981)
 Six Sigma seeks to improve the quality of process outputs
by identifying and removing the causes of defects (errors)
and minimizing variability in manufacturing and business
processes.
 The term six sigma originated from terminology associated
with manufacturing, specifically terms associated with
statistical modelling of manufacturing processes.
 The maturity of a manufacturing process can be described
by a sigma rating indicating its yield, or the percentage of
defect-free products it creates.
 A six-sigma process is one in which 99.99966% of the
products manufactured are free of defects (), compared to a
one-sigma process in which only 31% are free of defects.
 BUT: Requires a lot of statistics from the Test Manager
http://en.wikipedia.org/wiki/Six_Sigma

7.4 CMMI
7.4 CMMI
(Software Engineering Institute (SEI, 2002))
(Software Engineering Institute (SEI, 2002))
http://en.wikipedia.org/wiki/Capability_Maturity_Model_Integration

7.5 SPICE
7.5 SPICE ("Software Process Improvement and Capability Evaluation")
("Software Process Improvement and Capability Evaluation")
(ISO/IEC 15504 standard)
(ISO/IEC 15504 standard)
http://www.kuglermaagusa.com/tools_spice15504_com.html
0 ”Incomplete process” | General Failure to perform the bace practices in the Process

7.6 Testing Maturity Model (TMM)
7.6 Testing Maturity Model (TMM)
(1996 Illinois Institute of Technology, Chicago)
(1996 Illinois Institute of Technology, Chicago)
Based on CMM:
Levels Process Areas
1 – Initial No process identified
2 – Phase Definition Test Policy and Goals
Test Planning
Test Techniques and Methods
Test Environment
3 - Integration Test Organization
Test Training Program
Test Life Cycle and Integration
Control and Monitor
4 – Management and Measurement Peer Reviews
Test Measurement
Software Quality Evaluation
5 – Optimization Defect Prevention
Quality Control
Test Process Optimization

7.7 Test Process Improvement (TPI)
7.7 Test Process Improvement (TPI)
(Sogeti, Netherlands. http://www.sogeti.com)
(Sogeti, Netherlands. http://www.sogeti.com)
Key / Scale 0 1 2 3 4 5 6 7 8 9 10 11 12 13
1 Test strategy A B C D
2 Life-cycle model A B
3 Moment of involvement A B C D
4 Estimating and planning A B
5 Test Specification techniques A B
6 Static test techniques A B
7 Metrics A B C D
8 Test tools A B C
9 Test environment A B C
10 Office environment A
11 Commitment and motivation A B C
12 Test functions and training A B C
13 Scope of methodology A B C
14 Communication A B C
15 Reporting A B C D
16 Defect management A B C
17 Testware management A B C D
18 Test process management A B C
19 Evaluation A B
20 Low-level testing A B C

7.8 TPI Assessment
7.8 TPI Assessment
Key / Scale 0 1 2 3 4 5 6 7 8 9 10 11 12 13
...
Key / Scale 0 1 2 3 4 5 6 7 8 9 10 11 12 13
...
Current..
Required..

(7) Summary
✔ It is up to the individual company to decide how well a
particular model is suited toi it's own specific
circumstances.

CTAL TM – Part 2
Part 1
1. Introduction
2. Test Process and Test Tools
3. Testing in the software Life Cycle
4. Test Policy and Test Handbook
5. The Test Plan
6. Test Control
7. Assessing and Improving the Development and Test Processes
Part 2
8. Deviation Management
9. Risk Management and Risk-Oriented Testing
10. Staff Qualification and Skills
11. Test Metrics
12. Selecting and Implementing Test Tools
13. Standards Copyright Petro Porchuk 2010. Lohika

Deviation Terminology
Deviation Terminology
Error (Human Action)
Bug, Fault (Wrong Code)
Failure (Wrong Behavior)
http://estonovaamisa.files.wordpress.com/2009/02/human-error1.jpg

(8)
(8) Deviation Management
Deviation Management
✔Terminology
✔ Error
✔ Fault
✔ Failure
✔Documenting Incidents
✔Incident Handling (Bug Tracking Systems
Workflow)
✔IEEE 1044/1044.1 Standard (Chapter 13)

What is Risk?
What is Risk?
?

What is Risk?
What is Risk?
?
IT Project = Risk = Probability * Damage

(9)
(9) Risk Management and Risk-Oriented Testing
Risk Management and Risk-Oriented Testing
 Risk Management
 Identification of the risk context
 Risk identification
 Risk analysis and risk evaluation
 Risk control
 Risk verification and monitoring
 Risk-Oriented Test Plan Creation and Test Prioritization
 Failure Modes and Effect Analysis (FMEA)

9.1.1 Identification of the Risk Context
9.1.1 Identification of the Risk Context

9.1.2 Risk Identification. Risk Categories
9.1.2 Risk Identification. Risk Categories
Risk Category Influence Prediction Example
External risk
Cannot be influenced Difficult to predict •Lightning Damage
•Out of Power
Strategic risk
Difficult to influence Easy to predict •Changing a Corporate
standard etc
Project risk
Easy to influence Easy to predict •Sales Problems
•Technical Risks
•Servers unavailability
•Defect Correction
Risks
Product risk
Can be influenced Easy to predict •Customer suffering
from product failures
•Poor Quality affects
reputation

9.1.3 Risk Identification Techniques
9.1.3 Risk Identification Techniques
 Expert Interviews and Questionnaires
 Independent Estimations
 Risk Workshops
 Risk Brainstorming (Worst Case Scenarios)
 Use of risk templates and checklists
 Experiences from completed projects

9.1.4 Risk Evaluation
9.1.4 Risk Evaluation
 F – the frequency of execution of use (a function)
 C – criticality. Represents the worst possible effects of a function not working well (at all)
 Rp – The Project risks
 Rt – Technical Product risks
 Rb – The Commercial Product risks
Risk Factor:
RF=
RtRbRp
3
F∗RbC∗Rt

9.2 Risk Oriented Test Plan Creation
9.2 Risk Oriented Test Plan Creation
 Target-oriented test plan creation:
applying appropriately different test techniques and test
intensities to cover system functions with different risks
 Prioritized testing:
giving areas with higher risks higher priority and testing them
earlier
 Residual risk awareness:
identifying the residual risks that remain in the delivered
software and are due to reductions in test or nonperformance of
planned tests

9.3.1 Risk Based Testing.
9.3.1 Risk Based Testing.
Failure Modes and Effect Analysis (FMEA)
 FMEA is a Method used to identify potential error
types and their effect on a system.
FMEA Procedure:
1. Generate a list of all the risk factors.
2. Specify the type and probability of falures for each factor.
3. Investigate the effect of failures on other factors, applying, e.g., simulations.
4. Investigate the effect on Project Planning.
5. Identify possibilities for detecting the failure.
6. Identify possibilities for compensating the failure.
7. Identify possibilities for preventing the failure.
8. Define measures to avoid defects.
9. Evaluate the effect of the proposed measures.
10. Document the results.

9.3.2 Risk Priority Number.
9.3.2 Risk Priority Number.
 FMEA defined the Risk Priority Number (RPN).
O – Probability of Occurrence
E – Expected Damage
D – Probability of Detection
RPN =O∗E∗D

(9)
(9) Summary
Summary
✔ Risk = Rrobability * Damage
✔ Testing is a preventive measure, reducing risks
✔ Based on the most important risks, suitable test
techniques are to be selected, and tests are to
prioritized to allow tests covering high risks to be
executed early and with the appropriate effort
✔ Prioritization and risk estimation criteria, defined in the
Test Plan, can be applied to test objects and test
cases

Skilled people as the key success factor
Skilled people as the key success factor
TOYAKO, JAPAN - JULY 08: (L-R) German Chancellor Angela Merkel, U.S. President George W. Bush, Japanese Prime Minister
Yasuo Fukuda, French President Nicolas Sarkozy and Russian President Dmitry Medvedev plant the memorial tree after a working
session at the Windsor Hotel Toya on July 8, 2008 in Toyako, Hokkaido, Japan.
http://www.life.com/image/81853290

(10)
(10) Staff Qualification and Skills
Staff Qualification and Skills
 Individual Skills
 Functional Team Roles
 Social Team Roles
 The Communication Factor
 The Motivation Factor

10.1 Individual Skills
10.1 Individual Skills
 The ”perfect” test team member has exellent technical knowledge and high social competency.
 There is NO PERFECT Individual from the scratch. Need to develop.
 IT Expert as QA:
 Great technical competency
 DANGER: In love with technology → loses focus
 Application Expert as QA:
 Great domain expert. Knows non-func. requirements
 DANGER: Blinkered atitude → doesn't seek for improvements
 Social Competence
 The ability to familiarize quickly with complex domains
 The ability to detect defects. (”professional ressimism”)
 The ability to cope with criticism adequately. Diplomacy
 Courage to leave some gaps
 Discipline, exactitude, patience, tolerance
 The ability to work in a team and to communicate

10.2 Functional Team Roles
10.2 Functional Team Roles
 Test Manager
 Leads the Test Team
 Test Planning and Control, Test Reporting
 Human Resource Management and Test Process Improvement
 Test Designer (QA Analyst)
 Creation and Maintenance of Test Specification. Prioritizing
 Defining and Apply Test Design Techniques
 Test Automator
 The Test Team's Programmer
 Developing Automation Tests, Test frameworks and Tools
 Test Administrator
 Installation, Administration and Maintenance of the Test Environment
 Installing and set up of the System Software (Operating Systems, Databases, Application Servers)
 Tester
 Execution the tests and documenting the test results
 Accuracy and Creativity
 Expert
 Supporting L/P testing, database or network testing etc

10.3 Social Team Roles
10.3 Social Team Roles
Type Descriptors Strengths Weakness
Monitor Evaluator Strategic Power of judgment Lacks drive and the ability to
inspire others
Shaper Dynamic, open-minded,
result oriented
Lot's of drive, fight
inefficiency, experts pressure
Prone to irritate and to hurt
others
Planter Individualistic Creative. Lot's of intellectual
power
Ignores practical details and
instructions
Completer Straight, anxiuos Ability to complete things,
perfectionism
Inclined to worry undully,
reluctant to delegate
Team Worker Cooperative, mild, sensitive Ability to cope with different
situation and people,
promotes team spirit
Can be easily influenced
Implementer Conservative, dutiful Hard-working, disciplined Inflexible. Rejects unproved
ideas
Co-ordinator Self-confident, mature Recognizes individual
talents. Good at clarifying
goals
Has limited creativity
Resource Investigator Extrovert, enthusiastic,
communicative
Likes to develop contacts,
picks up new ideas, reacts to
challenges
Too optimistic, tend to lose
interest after initial
enthusiasm
Specialist Single-minded, self-starting,
dedicated
Provides knowledge and
skills in rare supply
Contributes only a narrow
front, overlooks the ”big
picture”

10.4 The Communication Factor
10.4 The Communication Factor
 Communicate problems accurately
 Test Team External Communication
 Tester → Developer
 Developer → Tester
 Test Manager → Projects Manager
 Project Manager → Test Manager
 Tester → User
 Test Team Internal Communication
 Regular Team Meetings
 Knowledge Transition

10.5.1 The Motivation Factor
10.5.1 The Motivation Factor
http://media.photobucket.com/image/funny workers/geet_kunal/happy-best-wishes-monday/rg272945.jpg

10.5.2 The Motivation Checklist
10.5.2 The Motivation Checklist
 The schedule is a motivating factor
 The resources provided are a motivating factor
 The test process is a motivating factor
 The task is a motivating factor
 The team will feel valued, recognised and
motivated
http://www.imbus.de/fileadmin/imbus_repository/Downloads/imbus_Allgemein/Dokumente/Checkliste-Motivation_E.zip

(10)
(10) Summary
Summary
✔ A member of the test team should be IT professional, have
knowledge from the application domain and have social competence
✔ The different tasks in the test process are reflected in the different
roles within the test team
✔ Becides professional role, each team member should play a social
role in the team
✔ Detected problems should be communicated to the external parties
with relevan communication style
✔ Test Manager should organize regular internal meetings
✔ The Test Manager should provide a healthy motivation environment
and professional working condition for the team. Part of this task is to
adequately represent the role and contribution of the test team within
the organization

Why do the People change job?
Why do the People change job?
Stupid Manager
Far from Home
Low Salary
Not Allowed to use Facebook

(11)
(11) Test Metrics
Test Metrics
 Why do we need Metrics?
 Presenting Measurements
 Test Metrics Types
 Residual Defect Estimations and Reliability

11.1 Why do we need Metrics?
11.1 Why do we need Metrics?
 The Test Metrics is the Tool for Control
Tom DeMarco on Measurement:
„You can not control what you can not measure.
Measurement is the prerequisite to management
control.“
from Tom DeMarco
American Consultant, 1982

11.2 Presenting Measurements
11.2 Presenting Measurements
 Diagrams help you to visualize measurement values and their progression

11.3.1 Test Metrics Types
 Test Case Based Metrics
 Number of Planned, Specified Test Cases
 Number of Created Test Procedures
 Number of Changed, Deleted Test Cases
 Number of Run, Passed, Failed, Blocked TC
 Test Object Based Metrics
 Number of tested functions / Total number of functions
 Number of executed TC / Number od specified TC epr function
 Percentage of all requirements covered by Test Cases
 Number of platforms covered by test
 Percentage of interfaces tested
 Number of lines of codes or executable instructions (kilo lines of code – KLOC)
 Number of the covered paths in the control flows graph
 Percentage of branches in the control flow graph that were covered

 Defect - Baced Metrics
 Bug Detection Percentage
 Number of defects relative to criticality (defect severity)
 Number of defects per status (defect correction progress)
 Defect report status
 Cost and Effort-Based Metrics
 Number of person-days for test planning or specification
 Number of person-days for the creation of test procedures
 Evaluating Test Effectiveness
 Degree of Defect Detection
 Confidence Level
 Test Effectiveness
DDD=
numberof weighted defects during test
total numberof defects found
CL=1−
numberof weighted defects
number of executed test cases
∗degree of test coverage
TE=DDD∗CL

11.4 Residual Defect Estimations and Reliability
11.4 Residual Defect Estimations and Reliability
 Three options to evaluate Test Process based on defects found
 Estimate number of defects inherent in the system prior to test and estimate the number of
defects to be found
 Monitor the Defect Detection Percentage
 Inject bugs into the program code and monitor the proportion to be found during testing
 Reliability Growth Model
Target numberof Defects=
residual numberof defects
effectiveness percentage

(11)
(11) Summary
Summary
✔ For each Test Level suitable Test Metric should be
applied
✔ Ensure compliance, if necessary through audits

(12)
(12) Standards
Standards
 General Understanding of Standards
 Standards related to Software Testing
 Standards in the phases of the Test Process

12.1 General Understanding
12.1 General Understanding
 Standards are developed and maintained by national and
international organizations
 ISO – International Organization od Standardization
 ANSI – American National Standard Institute
 BSI – British Standard Institution
 IEEE – Institute of Electrical and Electronics Engineers
 Standard Types
 Corporate Standards
 Best Practices and Technical specifications
 Domain Specific
 Generally Applicable Standards

12.2 Main Standards related to Software Testing
12.2 Main Standards related to Software Testing
Standard Description
BS 7925-1 British Standard: Software Testing, Part 1: Vocabulary, 1998
BS 7925-2 British Standard: Software Testing, Part 1: Software Component Testing, 1998
ISO 9000 A family of standards for quality management systems
IEEE 610.12 1990 IEEE Standard Glossary of Software Engineering Terminology
ISO 90003 Software Engineering Quality Management Standard
IEEE 730 IEEE Standard for Software Quality Assurance Plans, 2002
IEEE 1008 ANSI/IEEE 1008-1987 IEEE Standard for Software Unit Testing
IEEE 1012 IEEE Std 1012-1998 IEEE Standard for Software Verification and Validation
ISO 9126-x IEEE Std 1044-1993 IEEE Standard Classification for Software Anomalies
ISO 9241 ISO 9241-11:1998. Ergonomic requirements for office work with visual display terminals
IEEE 1044 IEEE Std 1044-1993 IEEE Standard Classification for Software Anomalies
IEEE 829 IEEE 829-1998, Standard for Software Test Documentation
IEEE 1059 IEEE Std 1059-1993 IEEE Guide for Software Verification and Validation Plans

12.3 Standards in the phases of the Test Process
12.3 Standards in the phases of the Test Process
Type of
Standard
Test Phase
Terminology
and contracts
Processes Products Documentation Methods and
techniques
Test Planning
Test Control
BS 7925-1
ISO 2382
ISO 9000
ISO 12207-0
IEEE 610.12
ISO 12207-0
ISO 9000-3
IEEE 730
IEEE 1008
IEEE 1012
IEEE 1061
ISO 9126-x
ISO 12119
ISO 12207-1
ISO 15026
IEEE 982.1
IEEE 1228
IEEE 730
IEEE 829
IEEE 1012
IEEE 1228
ISO 12207-2
ISO 16085
IEEE 730.1
IEEE 982.2
IEEE 1059
Test analysis
Test design
ISO 14598-4
ISO 15408
IEEE 1062
IEEE 1228
IEEE 1008
IEEE 1012
ISO 15939
ISO 9241
ISO 12119
IEEE 1044
IEEE 982.2
IEEE 829
IEEE 1063
BS 7925-2
IEEE 1028
ISO 60812
EN 61508-3
EN 50128
IEC 61025
ISO 5806
Test realization
Test execution
IEEE 1008 ISO 9241
IEEE 828
IEEE 1209
IEEE 1348
IEEE 829
IEEE 1209
IEEE 1348
IEEE 1028
ETSI 201-873
IEEE 1042
IEEE 1209
IEEE 1348
Test evaluation
Reporting
IEEE 1008 IEEE 982.2
IEEE 1044
IEEE 829 IEEE 1044.1
ISO 12119

(13)
(13) Summary
Summary
✔ Test Manager shouldn't reinvent the wheel, use
standards for Testing
✔ Ensure compliance, if necessary through audits

Istqb ctal tm

More Related Content

What's hot

Viewers also liked

Similar to Istqb ctal tm

More from Petro Porchuk

Recently uploaded

Istqb ctal tm