ISTQB Advanced Test Manager Training 2012 - Testing Process

ISTQB Advanced Level 2012 –
Test Manager

Objectives
 Build on ISTQB CTFL in the area of test manager
 Prepare for the ISTQB CTAL – Test Manager exam
24-Sep-15ISTQB Advanced Level 2012 - Test Manager 2

Prerequisites
 ISTQB CTFL or equivalent
 Practical experience in SW testing

Notes
 Ask any time.
 Turn your cell silent.

References
 ISTQB CTAL – TM syllabus version 2012

Outline
 Testing Process
 Test Management
 Reviews
 Defect Management
 Improving the Test Process
 Test Tools and Automation
 People Skills – Team Composition

Testing Process
 Introduction
 Test Planning, Monitoring and Control
 Test Analysis
 Test Design
 Test Implementation
 Test Execution
 Evaluating Exit Criteria and Reporting
 Test Closure Activities

Testing Process
 Introduction
 Test Analysis
 Test Design
 Test Execution

Fundamental Test Process in ISTQB
CTFL Syllabus
 Can be executed:
 Sequentially
 In parallel (exploratory)
 Or overlapping
P
A
&
D
I
&
E
E
&
R
C
C

Refinement of Fundamental Test
Process in ISTQB CTAL Syllabi
 Activities are considered separately in order:
 To provide additional refinement and optimization of the processes
 To better fit the software development lifecycle
 To facilitate effective test monitoring and control
 Tailoring these main activities within the context of the system and the
project is usually required.

Refinement of Fundamental Test
Process in ISTQB CTAL Syllabi cont’d
 It is important to understand the other steps in the test process.
 Majority of the TM’s work usually is done during the planning, monitoring,
and control of the testing project.
P
A D I E E & R
C
M & C

Testing Process
 Introduction
 Test Analysis
 Test Design
 Test Execution

Learning Objectives
 TM-1.2.1 (K4) Analyze the test needs for a system in order to plan test
activities and work products that will achieve the test objectives

Test Planning
 Identification of:
 Activities and resources to meet mission and
objectives defined in test strategy
 Methods for gathering and tracking metrics that:
 Guide the project
 Determine adherence to the plan
 Assess achievement of objectives
 Tools, training and documentation guidelines based on
me metrics defined in the planning
 For each level, starts @ test process initiation and
continues till closure activities

Test Strategy Affects Test Planning
 Strategy(ies) determine tasks that should occur during the planning.
 In risk-based testing strategy , risk analysis guides planning regarding
mitigating activities of identified product risks and help contingency
planning.
 Test effort allocation
 Testing levels needed
 Test activities prioritization
 Test techniques selection
 Need for static testing
 In a reactive strategy, planning for the creation of test charters and tools
for dynamic testing techniques such as exploratory testing may be
warranted.

Examples: Risk-Analysis Guiding Test
Planning
 If a number of likely and serious potential security defects identified, a
significant amount of effort should be spent developing and executing
security tests.
 If serious defects are usually found in the design specification, the test
planning process could result in additional static testing (reviews) of the
design specification.
 If performance is a high risk, performance testing may be conducted as
soon as integrated code is available.

Example: DO-178 B Needed Coverage
 Used in an airborne environment
Failure Level Description Testing Technique
A: Catastrophic
Failure may cause lack of critical function
needed to safely fly or land the plane
MC/DC
B: Hazardous
Failure may have a large –ve impact on safety
or performance
Decision Testing
and MC/DC
optional
C: Major
Failure is significant, but less serious than A or
B
Statement testing
@ minimum
D: Minor
Failure is noticeable, but with less impact than
C
E: No effect Failure has no impact on safety

Test Management PoV
Test Basis, Test Conditions, and Test
Cases Complex Relationship
 Need to be understood for effective
planning, monitoring and control
 Affects tool decisions
Test Development PoV
Test Object
Test Items
Test Basis
Test Conditions
Test Cases
Test Procedures
Test
Analysis
Test
Design
Test
Implem
entation

Relation with the Development Team
Impacts Test Planning
 Development work products affect traceability and traceability matrix
requirements.
 Development work products must be approved before specific testing
activities can start.
 Development life cycle model determines which and how information
should be exchanged between development and testing teams.
 Test plan lists SW features both in and outside testing scope w/
appropriate project formality and documentation.
 TM defines initial test environment specification

Test Planning Considers Dependencies
 External dependencies and associated SLA’s must be identified and initial
contact should be made.
 Example dependencies:
 Outside groups
 Other projects
 External vendors or development partners
 Deployment team
 Database administrators

Example: Test Planning in Sequential
Models
 Has many test planning issues
that must be considered:
1. Quality and delivery trade-off
due to schedule compression @
the end
2. Unstable or untestable SW
delivered to test team
3. Failure to include all activities
shown in the model

Example: Test Planning in Iterative
Models
1. Regression testing
2. Planning for bugs
3. Lack of rigorous testing

Example: Test Planning in Agile
Models (Iterative Models ++)
Challenges
 Volume and speed o change
 Keeping w/ the pace
 Increased regression risk
 Inadequate unit testing
 Poor test oracles and test basis
 Scrum or scrums
 Over expectation
Opportunities
 Automated unit testing
 Static code analysis
 Code coverage
 Continuous integration
 Automated functional tests
 Informal reviews and power o 3
 Control of technical debt

Example: Test Planning in Spiral
Models
1. Testing must be very flexible
2. Testing objective varies along
the development lifecycle (early
testing is for knowing the
unknown not for finding
defects).
3. Unpredictable schedule

Test Monitoring
 Providing feedback and visibility about all test activities
 Efficient test control requires defining a schedule and monitoring
framework.
 This framework should include detailed measures and targets needed to
relate test work products and activities to plan and strategic objectives.
 Easy for small or less complex projects

Defining Monitoring Objectives and
Targets
 Test monitoring framework must be defined a manner that is
understandable and relevant to the project and business stakeholders
(residual risks vs. defects and tests cases for example).
 Test conditions are a key player as they link test basis to test work
products.
 Properly configured traceability wit properly defined test conditions make
the complex relationships more transparent and comprehensible.

Involvement of Stakeholders in Defining
Monitoring Frameworks
 Sometimes, stakeholders require targets that are not related to system
functionality or specifications (common if no or lacking formal
documentation).
 Involvement stakeholders @ early stage help define measures and targets
that provide better control and influence testing activities

Example: Quality Objectives as Entry
Criteria

Test Control
 Ongoing comparing of actual progress against plan and implementing
corrective actions when needed
 Guides testing to fulfill mission, strategies, and objectives
 Appropriate reactions to the control data depend (decision taking, test re-
prioritizing, test rescheduling or changing entry criteria) on detailed
planning information (traceability between test basis, test conditions and
test work products).

Exercise: Test Planning, Monitoring
and Control
 Assume, you are a TM who is planning test for a project that will follow
iterative SDLC.
 Your requirements are prioritized into 5 levels (VH, H, M, L, VL).
 Follow the ISTQB fundamental test process and outline the test activities
across each iteration.

Testing Process
 Introduction
 Test Analysis
 Test Design
 Test Execution

Learning Objectives
 TM-1.3.1 (K3) Use traceability to check completeness and consistency of
defined test conditions with respect to the test objectives, test strategy,
and test plan
 TM-1.3.2 (K2) Explain the factors that might affect the level of detail at
which test conditions may be specified and the advantages and
disadvantages for specifying test conditions at a detailed level

Test Analysis
 Test analysis = defining “what” to test (test conditions)
 Test design = defining “how” to test (test cases)
 Test analysis and design can be implemented as parallel, integrated, or
iterative activities.

Test Conditions
 Identified by analysis of the test basis, test objectives, and product risks
 Viewed as detailed measures and targets for success (exit criteria)
 Test conditions coverage, test conditions achievement % ...
 Should bi-directionally traceable
 To test basis, test objectives, risks or any other project or stakeholder criteria
for success
 To test design and resulting test work products

When and How Test Analysis?
 As soon as test basis is established for a test level (test basis and test
conditions will differ according to test level)
 Formal test techniques and analytical techniques can identify test
conditions
 Test conditions may vary in level o detail.
 Specifying test conditions in a detailed fashion will tend to result in a
larger number of test conditions.

How much Detail in Test Conditions?
 Answer depends on some factors:
 Testing level
 Detail and quality of test basis
 System/software complexity
 Risks
 The relationship between the test basis, what is to be tested and how it is to
be tested
 SDLC in use
 Test management tool being utilized
 Level at which test design and other test work products are to be specified
and documented
 Skills and knowledge of the test analysts
 The level of maturity of the test process and the organization itself
 Availability of other project stakeholders for consultation

Exercise: Test Analysis
 Perform a complete test analysis for the given screenshot.

Detailed Test Conditions Advantages
 Facilitates more flexibility in relating other test work products to test basis
and test objectives yielding better monitor and control eventually
 Contributes to defects prevention
 Relates testing work products to stakeholders in terms that they can
understand
 Helps influence and direct testing and development activities
 More efficient coverage of detailed measures and targets
 Basis for clearer horizontal traceability within a test level

Detailed Test Conditions Usage
 Effective in the following situations:
 Lightweight test design documentation methods are being used to
accommodate SDLC, cost and/or time constraints or other factors
 Little or no formal requirements or other development work products are
available as the test basis
 The project is large-scale, complex or high risk

Disadvantages of Detailed Test
Conditions
 Potentially time-consuming
 Maintainability can become difficult in a changing environment.
 Level of formality needs to be defined and implemented across the team.

General Test Conditions Usage
 Effective in the following situations:
 CT
 Less complex projects where simple hierarchical relationships exist between
what is to be tested and how it is to be tested
 AT where use cases can be utilized to help define tests

Testing Process
 Introduction
 Test Analysis
 Test Design
 Test Execution

Learning Objectives
 TM-1.4.1 (K3) Use traceability to check completeness and consistency of
designed test cases with respect to the defined test conditions

Test Implementation
 Fulfillment of the test design
 Checking against explicit and implicit entry criteria
 Creating automated tests
 Organizing tests (both manual and automated) into execution order
 Finalizing test data and test environment
 Forming a test execution schedule, including resource allocation

Test Design
 Test design = defining “how” to test (test cases) by stepwise elaboration of
the identified test conditions or test basis using test techniques defined
during test planning
 Test design includes test data design and test environment design as well.
 After the TA’s do their work, TM must check the design exit criteria before
moving to the next step.
 Test design exit criteria include review and approval from relevant
stakeholders, coverage, cost, time ...

Test Design Traceability
 Test cases can be related directly to test basis or indirectly through test
conditions (preferably detailed ones).
 Test basis items could be individual requirements (atomic?!), risk item,
strategic objectives, test objectives or project or stakeholder success
criterion.
 Traceability resolution loss should be avoided.
 Must be bi-directional to allow TM to relate test results in terms of project
aspects

Inputs and Outputs for Test Design
Test Plan
Test
Conditions
Test
strategy
Quality
Risks
Test
Oracles
Concrete
or
Logical?
Test Cases w/
Traceability
Test Data
Design
Test
Environment
Design
TA’s

Concrete or Logical Test Cases?
Concrete Test Cases
 Specific information
 Preconditions, inputs, outputs, post
conditions, and procedures
 Useful when:
 Requirements are well-defined.
 Testers are inexperienced
 External verification like audits is
needed.
 Benefit is excellent reproducibility.
 Maintenance is a significant issue and
limits testers creativity.
Logical Test Cases
 Guidelines for what should be tested
 TAs can vary data and/or procedure
 Useful when:
 Requirements are not well-defined.
 Testers are experienced in testing and
the product.
 Formality is not needed.
 Benefits are better coverage due to
variation and lower costs of creation
and maintenance.
 Reproducibility is a significant issue.

Sequential or Iterative Test
Development Process?
 Decision is influenced by the test level and test strategy.
 In ST, SIT and AT, it is more sequential as test basis are available much
earlier than the test execution start, while in CIT and CT, it is more
iterative.
 Analytical strategies are sequential while reactive tend are iterative by
nature.
 Even in sequential development, some overlap may be beneficial
especially test data generation after test design and before test
implementation can reveal problems in the test design.

Example: IEC 61508 Test Implications
 Risk-analysis is required.
 Risk-based testing is a mandatory risk mitigation technique.
 All risk-relevant test work products must be formally documented.
 SIL mandates selection of test techniques, required structural coverage
and test levels to be used.
 Reviews are mandatory even for test work products.

Exercise: Test Design
 Derive minimal number of test cases from previously defined test
conditions.

Testing Process
 Introduction
 Test Analysis
 Test Design
 Test Execution

Learning Objectives
 TM-1.5.1 (K3) Use risks, prioritization, test environment and data
dependencies, and constraints to develop a test execution schedule which
is complete and consistent with respect to the test objectives, test
strategy, and test plan

Test Implementation

Test Execution Schedule
 Tests must be scheduled and sequenced during test implementation
taking into considerations:
 Order of automated and manual test execution
 Prioritization assigned to tests from requirements or risks
 Execution constraints like tester or environment availability or test
environment reconfiguration
 TM should carefully review schedule to avoid any test execution blockage.

Ready for Test Execution?
 Trust, but verify the test environment and test data.
 If the test environment to evolve during execution, double check all changes
that will occur in accordance with the test execution schedule.
 Contingencies should be defined if these anticipated changes did not occur for
a reason or another.
 Tests must be reviewed and verified before running.
 Double check execution schedule to make sure every thing will be in place.
 Make sure SW is ready for testing.
 Smoke tests are effective in checking the test execution readiness.

Example: Test Execution Entry Criteria

Detail and Complexity
 Same argument considered for test conditions apply for test cases, test
data description and test environment description.
 Using testware as a way of system documentation, testware reuse or
having testers w/ different skill set mandates documenting more details.
 Regulatory rules may also mandates documentation details.
 Project complexity can also determine level of details needed in
implementation.
 TM must defined implementation level of details to guide testers during
implementation.

Early Test Implementation Pros
 Concrete tests provide worked examples of how SW should behave, if
written in accordance with test basis.
 Business domain experts verify concrete tests easier than abstract
business rules, and thereby identify further weaknesses in SW
specifications.
 Verified tests may provide illuminating illustrations of required behavior
for SW designers and developers.

Early Test Implementation Cons
 Changing test basis may lead to change in the testware already developed.
 In Agile SDLC, embracing changes is the norm!.
 In sequential SDLC, poorly managed projects can lead to unhappy
surprises.
 Before embarking on an extensive test implementation, it is wise to
understand SDLC and the predictability of SW features that will be
available for testing.
 In conclusion, late implementation may be favored.

Testing Process
 Introduction
 Test Analysis
 Test Design
 Test Execution

Learning Objectives
 TM-1.6.1 (K3) Use traceability to monitor test progress for completeness
and consistency with the test objectives, test strategy, and test plan

Test Execution
 Begins once test object is delivered and entry criteria to test execution are
satisfied (or waived carefully)
 Criteria examples are:
 Tests should be designed
 Tools should be in place
 Test results tracking should be working and tracked data is understandable by
all team members.
 Standards for test logging and defect reporting should be available and
published.
 Bottom line, do whatever it takes to make execution proceeds efficiently.

Blending Reactive Test Strategies
 10-20% of testing should be reactive (even more in case formal testing is
automated).
 Blending offsets weaknesses in test strategies as they complement each other.
 Reactive testing blend is done by giving permissions to explore other options
during formal execution.
 Time boxing can help.
 Experienced testers should do experience- and defect-based testing more.
 Time boxing and chartered testing can help.
 Reproducibility for unscripted test should be considered.
 Capture-play back tools can help.

TM Role in Test Execution
 Being there all day during test execution
when needed unless he is in meetings
 Daily team debriefing of progress with the
entire team
 Take proper control actions if needed
 Checks and ensures proper bi-directional
traceability carried out by the team
Pass 2 Defects Status Count
Verified 47
To be verified 1
Needs approval 5
Deferred 21
Failed verification 3
Not a defect 8
New 13
Total 103

Exercise: Metrics
 Describe one or more metric that you can use to report residual risks
during test execution.

Testing Process
 Introduction
 Test Analysis
 Test Design
 Test Execution

Learning Objectives
 TM-1.7.1 (K2) Explain the importance of accurate and timely information
collection during the test process to support accurate reporting and
evaluation against exit criteria

Testing as a Source of Information
 Testing produces lots of information (relevant or not).
 TM must ensure that right information is captured and delivered to
appropriate stakeholders.
 TM should be able to relate the test results to the exit criteria.
 Test management tools must be properly selected and in place to support
accurately and timely gather information so as to facilitate effective
evaluation and reporting.
 TM should never use gathered info for personal appraisals.

Proper Information Collection
 Requires clear definition of what to be collected, by whom and when
 Should be documented during test planning
 Frequent check on gathered information is a must (never assume they are
OK).
 Team should be trained and aware of how to gather/report information w/
or w/o tools.

Reporting Frequency and Detail
 There is no one size fits all reporting approach.
 Close monitoring of exit criteria does not necessitate detailed reporting.
 Close monitoring = proper control
 Reporting = providing supporting information
 Details and type of reporting should differ according to recipient
stakeholders.
 They need different information to do their work.

Testing Process
 Introduction
 Test Analysis
 Test Design
 Test Execution

Learning Objectives
 TM-1.8.1 (K2) Summarize the four groups of test closure activities
 TM-1.8.2 (K3) Implement a project retrospective to evaluate processes
and discover areas to improve

Test Closure Activities
 Once execution is complete, key outputs should be captured and either
passed to the relevant person or archived.
Test Closure
Activities
Test
Completion
Check
Test Artifacts
Handover
Lessons
Learned
Archiving

Test Closure Activities cont’d
 These tasks are important, often missed, and should be explicitly included
as part of the test plan.
 It is common for one or more of these tasks to be omitted due to
premature reassignment or dismissal of project team members, resource
or schedule pressures on subsequent projects, or team burnout.
 On projects carried out under contract, such as custom development, the
contract should specify the tasks required.

Example: Test Closure Checklists
Test Completion
• All planned tests
run or
deliberately
skipped?
• All known
defects fixed,
deferred or
accepted?
Test Artifacts
Handover
• Known defects
communicated
to operation
and support
team?
• Test and
environment to
maintenance
team?
• Regression tests
documented?
Lessons Learned
• Were relevant
stakeholders
involved in risk
analysis?
• Were estimates
accurate?
• RCA done and
actions defined?
• Process
improvement?
• Any
unanticipated
deviation?
Archiving
• All test work
product
archived in CM?
• If the project to
be reopened, do
we have every
thing we need?

Project Retrospective
 A meeting held at the end of each iteration/increment
 To discuss:
 What went well?
 What can be improved?
 How to improve?

Exercise: Retrospective
 Assume that 25% of defects opened on a certain feature were rejected as
not actual problems.
 If rejection rate is more than 5%, is unacceptable.
 List 5 actions, that could investigate the root cause of the high defect
rejection rate.

Summary
 Introduction
 Test Analysis
 Test Design
 Test Execution

ISTQB Advanced Test Manager Training 2012 - Testing Process

More Related Content

What's hot

Viewers also liked

Similar to ISTQB Advanced Test Manager Training 2012 - Testing Process

More from Amr Ali (ISTQB CTAL Full, CSM, ITIL Foundation)

Recently uploaded

ISTQB Advanced Test Manager Training 2012 - Testing Process

Editor's Notes