Prevent the surprise, become a pro-active test manager. Too often projects suddenly seem to spin out of control. Challenges and risks keep stacking up and the defect count grows exponentially. At the same time, management can put pressure on you, asking when testing will be completed. A surprise? Not really, defects only paint half the picture. The test effort, after all, is primarily determined by the number of tests that need to be completed. For an on the spot status of testing and accurate view on the quality and risks of the entire project we need to organize the test process to provide flexible, up-to-date metrics and trends on a daily basis. E.g. we need a view on baseline vs. actuals and ETC’s on test cases. Advanced metrics will provide answers on what needs to be done tomorrow to stay on track, the location and root cause of issues and who is required to take action. Also the test effort remaining for an acceptable product (or a specific risk level) can be estimated fairly accurately. In addition early involvement and preparation in the development life cycle, performing test intakes rather than reviews, will help you bridge the gap between different development teams and allows you to verify consistency between business requirements, the integration model, functional specifications and technical specifications. It facilitates knowledge transfer and provides you with the “story” behind the specifications. This will help prevent structural issues in an early stage and avoid blocking issues during test execution. This presentation combines daily test metrics and trends with test process dynamics and shows you how to become a “pro-active” test manager. Even better you can apply it tomorrow and take your test process to a distinct higher maturity level.

1. Rien van Vugt, Maurice Siteur
Eurostar Manchester, 23 November 2011
Houston we have a problem

2. And suddenly the project is Out of Control
© 2011 Eurostar Vugt-Siteur. All rights reserved.

3. AGENDA
 What is the problem?
 What do we need?
 Demo
 Future
© 2011 Eurostar Vugt-Siteur. All rights reserved.

4.  Testing seems so easy:
We just need to perform X tests in Y amount of time
 But when we detect defects:
• We need to redo a fair amount of testing => high impact on required capacity
• We do not know how many defects we will find !
WHAT IS THE PROBLEM?
© 2011 Capgemini. All rights reserved.
 Testing seems so easy:
We just need to perform X tests in Y amount of time
 But when we detect defects:
• We need to redo a fair amount of testing => high impact on required capacity
• We do not know how many defects we will find !
Just a single hill to climb: or an endless struggle of hill after hill:
• Planning based on Baseline, ETC and Actuals does not work for testing :
Cycle 1 Cycle 2 Extra Cycle
Original deadlineStart

5. © 2011 Capgemini. All rights reserved.
WHAT IS THE PROBLEM?

6. What is The Problem?
 Large project
• Testing at multiple locations
• How do I get the complete status at a moment?
 Tracking takes all day
 Lack of time for analysis of the collected information
 Delayed monitoring & reporting of progress: e.g. on a weekly basis (or longer)
 Figures made-up manually based on subjective information
(intentionally or unintentionally)
© 2011 Eurostar Vugt-Siteur. All rights reserved.

7. Spreadsheets
 Spreadsheets are generally not good enough for registration of the progress
 (Test) Tools are good in registration but not so good in advanced graphic,
presentation and managing the list of tests or defects
 Presentation is the issue – a bowl of defects is confusing
A table with amounts is limited, a good picture paints a thousand words
Trends over time can provide valuable information but are often unavailable
 We often resort back to e.g. Excel to make the results “presentable” on the fly
or run into our defect management meeting with a print-out of tests or defects
© 2011 Eurostar Vugt-Siteur. All rights reserved.
“Houston we have a problem”; Rien van Vugt, Maurice Siteur; EuroStar 22 nov 2011

8. Tools for Monitoring Test & Defects
 Defect management tools are in place most of the time
ClearQuest, Mantis, Bugzilla, HP mercury, SourceForge, …..
 Often set-up from a development point of view
oStatus only open/closed (out of the box)
oLimited detail, more information is needed to monitor & control the workflow
oPriority and severity often confused
 Test (monitoring) tools are less abundant and may be expensive
From a test coordination & management point of view we just need “something” to
manage a list of tests, much like a list of defects
© 2011 Eurostar Vugt-Siteur. All rights reserved.

9. What Do We Need?
 Statistics & trends on tests as well as defects
 Easy to set-up and generate with flexibility for presentation
much like a spreadsheet can do
 Dashboards for:
o Test coordination (typical daily)
o Progress reporting at project level (typical weekly)
o Management reporting/summaries (weekly or monthly)
o Test evaluation (by test level)
 Capability & flexibility to present the information at a higher aggregated level
 Embed them into the test process so that they:
o Are up to date
o Objective
o Can be retrieved at a moments notice
© 2011 Eurostar Vugt-Siteur. All rights reserved.

10. Monitoring Tests - Examples
1© 2011 Eurostar Vugt-Siteur. All rights reserved.

11. Monitoring Defects - Examples
0
5
10
15
20
25
30
0 1 2 3 4 5 10 15 20 40 60 80 >
#Defects
Defect Age (max. days)
Open Defect Aging
Tested OK
In Test
Resolved
Additional
Low
Medium
High
Critical
Blocking
0
50
100
150
200
250
300
350
400
450
#Defects
Open Issue Trend
On hold
In Test
Resolved
Additional
Open prio 5
Open prio 4
Open prio 3
Open prio 2
Open prio 1
5 13
20
2 0
0
51
37
226
Open Issues
Open prio 1
Open prio 2
Open prio 3
Open prio 4
Open prio 5
Additional
Resolved
In Test
On hold
© 2011 Eurostar Vugt-Siteur. All rights reserved. 1

12. Get Into The Game Early
1© 2011 Eurostar Vugt-Siteur All rights reserved.
Quality Gate
Delivery
Test execution
Build
Business
Requirements
Functional
Design
Technical
Design
Integration
Design
Component
Integration
Test
Unit Test
T
e
s
t
D
e
v
e
l
o
p
m
e
n
t
Test
Responsibility
High level test design Detailed test design
High
Level
Test
Design
Detailed
Level
Test
Design
Technical
aspects
Interfaces
Functionality
System
Integration Test
(SIT)
System Test
(ST)
Acceptance
Test (UAT)
Production
Review, Intake
&
Knowledge Transfer
Requirements
B
u
s
s
s
e
n
i

13. Get Into The Game Early
1© 2011 Eurostar Vugt-Siteur All rights reserved.
Early Involvement of testing in the development life cycle:
1. Review and intake of:
o Requirements
o Integration and interfaces
o Specifications (e.g. use cases)
2. Do not review just for the sake of it
Actual use of the information drives out issues and inconsistencies and can be used to:
oPrevent defects
oComplete test designs early: “conform specs” & “fit for business”
oSet-up traceability
oDefine structure for metrics and different views for monitoring & control
3. Combine this with knowledge transfer on the business background

14. End Of Day Demonstration
1© 2011 Eurostar Vugt-Siteur. All rights reserved.
Test cases
Defects
Imported Information
End Of Day (Excel)Test Tooling
Extra info
Manual or
Derived from
Import
Special EOD formulas
Trend tables
Traceability tables
Shortlists
CSV, XLS or API
Excel Graphics &
Tables
Dashboard(s) for
Coordination
&
Reporting

15. The Future
Integrated tooling?
1© 2011 Eurostar Vugt-Siteur. All rights reserved.

16. Conclusions
Get involved early
 Through effective reviews and test intakes
 Set-up different angles to view/monitor tests progress & defects
Embed advanced metrics in the test & defect process
 Instant grip on your testing
 Set-up and monitor trends in test execution
 Improvements in Test Tooling are needed
Act today to solve tomorrow’s problems
1© 2011 Eurostar Vugt-Siteur. All rights reserved.
Pro-Active Test Manager

17. Questions?
The Tool is obtainable – Leave your card