This webinar presentation briefly covered TPI and TMMi and their usefulness as process assessment frameworks. However, the main thrust of the talk was to present a proven and pragmatic way to do process improvement that gets senior management commitment AND buy-in from project practitioners.

1. Sogeti Ireland
Effective Test Process Improvement
Fran O’Hara
fran.ohara@sogeti.com
22nd July 2009
© Sogeti Group 2009 Page 1
Contents
• Introduction
• TPI
• TMMi
• Process improvement challenges and
sample solutions
• Summary
• Q&A
© Sogeti Group 2009 Page 2

2. SOGETI Overview: A Worldwide Leader in IT Services
• Sogeti Group Part of
20,000 professionals • Leader in consulting, IT &
200 locations in 14 countries outsourcing
• Providing worldwide standards • 88,000 staff
• Revenue €8.7 billion
Copyright TMap®
and TPI®
brands
• Sogeti:
Dedicated R&D on new developments
• £1.6 billion revenues
Test thought leadership
• Testing revenues
USA £150M (2007), £190M
(2008)
Sweden
Test Centre of Excellence
provides for
UK Denmark
Ireland • Reduced cost (>25% over 2 yrs)
NL
Germany
• Improved quality
Belux
Switzerland • Improved flexibility
France
• Reduced time-to-test/market
• Reduced risk
Spain
• Increased capacity
India
3
© Sogeti Group 2009 Page 3
Test Process Improvement - learning to work smarter,
to test software better, cheaper, faster
Process focus improves World Class
predictability of
performance (reduces
variation), as well as
performance itself
Products/services
(difference between
desired and actual)
Capability
People Process Technology
© Sogeti Group 2009 Page 4

3. Triggers
• Efficiency/cost?
• Effectiveness?
• Growing Complexity?
• Different levels of Test Maturity?
• Time Pressure?
• Are we doing the right thing(s)?
Development Business
Automation Time-to-market
COTS CBD Standard. Reuse
ty Growin
ali
I’s Agile
g
Qu Complex
GU
ity
Cheap
ic esXML Error
serv
free
We b User-friendliness
Integration
Tester Maintainability
© Sogeti Group 2009 Page 5
The TPI® Model
Key areas
Levels Test
Maturity
Matrix
Checkpoints Improvement suggestions
© Sogeti Group 2009 Page 6

4. 20 Key Areas
• Test strategy • Commitment and motivation
• Life-cycle model • Testing functions and
• Moment of involvement training
• Estimating and planning • Scope of methodology
• Test specification • Communication
techniques • Reporting
• Static test techniques • Defect management
• Metrics • Testware management
• Test automation • Test process management
• Test environment • Evaluation
• Office environment • Low-level testing
© Sogeti Group 2009 Page 7
Current Situation (example)
Key area / Scale 0 1 2 3 4 5 6 7 8 9 10 11 12 13
1 Test strategy A B C D
2 Life - cycle model A B
3 Moment of involvement A B C D
4 Estimating and planning A B
5 Test specification A B
techniques Dependencies!
6 Static test techniques A B
7 Metrics A B C D
8 Test automation A B C
9 Test environment A B C
10 Office environment A
11 Commitment and A B C
motivation
12 Test functions and training A B C
13 Scope of methodology A B C
14 Communication A B C
15 Reporting A B C D
16 Defect management A B C
17 Testware management A B C D
18 Test process management A B C
19 Evaluation A B
20 Low - level testing A B C
© Sogeti Group 2009 Page 8

5. Process Improvement ….
Key area / Scale 0 1 2 3 4 5 6 7 8 9 10 11 12 13
1 Test strategy A B C D
2 Life - cycle model A B
3 Moment of involvement A B C D
4 Estimating and planning A B
5 Test specification A B
techniques
6 Static test techniques A B
7 Metrics A B C D
8 Test automation A B C
9 Test environment A B C
10 Office environment A Opti
ControlledB Efficient
11 Commitment and
motivation
A Cmizing
12 Test functions and training A B C
13 Scope of methodology A B C
14 Communication A B C
15 Reporting A B C D
16 Defect management A B C
17 Testware management A B C D
18 Test process management A B C
19 Evaluation A B
20 Low - level testing A B C
© Sogeti Group 2009 Page 9
Testing Maturity Model Integration (TMMi®)
• Currently under development by the TMMi Foundation
See www.tmmifoundation.org
Levels 2 & 3 published
• The TMMi® Foundation will provide:
A standard TMMi® Model that can be used in isolation or
in support of other process improvement models and is
either staged or continuous.
An independently managed data repository to support
TMMi® assessment method accreditation, assessor and
assessment certification/validation and validated
assessment data and certificates.
Assessment Method Accreditation/Audit Framework for
TMMi® in accordance with ISO15504 and the process to
certify commercial assessment methods against the
standard model.
Certification and training/examination process, procedures
and standards for formal, public accreditation of
Assessors and Lead Assessors and the on-going
management.
© Sogeti Group 2009 Page 10

6. TMMi® Model
Level Process Areas
5 Optimisation Defect Prevention
Test Process Optimisation
Quality Control
4 Management & Measurement Test Measurement
Software Quality Evaluation
Advanced Peer Reviews
3 Integration Test Organisation
Test Training Program
Test Life Cycle and Integration
Non-Functional Testing
Peer Reviews
2 Managed Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
1 Initial
© Sogeti Group 2009 Page 11
Problems with Process Improvement
Typical Issues with Process Results from Sogeti ‘Worldwide survey on TPI’ :
Improvement:
Lack of management commitment **
Lack of management
commitment
Unrealistic expectations *
Lack of prioritised goals to Unclear scope and target (i.e. goals) *
focus/prioritise Unclear communication
Only 17% had measurable results
Excessive emphasis on
documenting and assessing
process Results unknown for too long a time
Difficulty in defining and Lack of change process
deploying the process Change team without influence
Inadequate buy-in from staff Involvement of people within the test process*
Top down approach (‘Thou shalt..’)
Lack of process architecture
44% combined with other improvement projects
© Sogeti Group 2009 Page 12

7. Improvement Strategy - 1
Lack of management commitment
Problems
- Lack of management commitment**
Lack of prioritised goals to
focus/prioritise
- Unrealistic expectations*
- Unclear scope and target (i.e. goals)*
- Unclear communication
- Only 17% had measurable results - NB
Solutions
Achieve sign-off on documented requirements
and constraints and associated improvement
(project) plan
Align and prioritise improvement efforts with
business goals through a goal-problem tree
Focus on early and measurable results
© Sogeti Group 2009 Page 13
Problem Scenario - 1
Customer
satisfaction
Reported Production Defects rating dropped
20% in June
80
survey
70
60 compared to 6
50 months
40
previously.
30 # Severity 1 & 2 defects
20
10 Cost reduction
0
1 2 3 4 5 6
of 10% is also
Months
a department
target
© Sogeti Group 2009 Page 14

8. Problem Scenario -2
Origin of Production Defects Data also
shows that
rework of
production
User requirements
System requirements
defects is
Architecture
costing 40% of
Detailed design development
Code & Build budget
Integration
© Sogeti Group 2009 Page 15
Examples of test-related improvement actions
• Perform formal peer reviews of System requirements
Involve system testers
• Use a risk-based approach to system test involving business
users in risk workshops
• Use a test driven development approach to unit test (unit
test cases developed and automated before code written)
• Design system tests as soon as system requirements
become available
• Resource system test with some dedicated test
professionals
• Perform formal planning of system test
• Measure DDP for peer reviews, unit test and system test
• Some ways to identify improvement actions:
Brainstorm process problems and vote
Brainstorm improvement ideas/actions
Model gap analysis – assessment results
© Sogeti Group 2009 Page 16

9. Goal Tree & Cost/Benefit
Business Goal Dev/Test Test Process Est. Cost Est.
Department Goal Goals of Benefit
required
actions
Reduce IT Reduce rework Improve 200k 1m
Appn. costs of production effectiveness of (10% of IT
Development defects by 50% Peer reviews, budget)
costs by 10% Unit test, and
System Test by
75% (Currently
running at 50%
DDP)
Improve Reduce number of As above 0 Satisfied
Customer production defects customers
satisfaction per month by 50%
ratings by 20% (based on
equivalent appn.
throughput)
© Sogeti Group 2009 Page 17
Goal Metric Scenario – Sample Results
Defect Detection Percentage per Release
100
80
60
DDP
40
20
0
1 2 3 4 5 6
Release number (or subsequent projects)
© Sogeti Group 2009 Page 18

10. Improvement Strategy - 2
Excessive emphasis on documenting and
Problems assessing process
- Results unknown for too long a time
Use a more agile approach to PI - Iterate in
Solutions short improvement cycles (2-3 months) – how
do you eat an elephant?
Focus on lightweight, rapid process modelling
for early deployment (80/20 rule for any gap
analysis)
Perform formal assessments to confirm
process maturity results
© Sogeti Group 2009 Page 19
Improvement Strategy - 3
Difficulty in defining and deploying the
Problems process
Lack of change process
Change team without influence
Inadequate buy-in from staff
Involvement of people within the test process*
Top down approach (‘Thou shalt..’)
Use a facilitated workshop approach to process
Solutions modelling/improvement
Process owners involved in workshops, ‘change team’
become facilitators
Goal–problem driven focus
Use a continuously improving well-defined change
process
© Sogeti Group 2009 Page 20

11. Example Modeling Session
•Picture of process-level modeling (Image2)
•session inserted here.
© Sogeti Group 2009 Page 21
Example “Mapped” Element
© Sogeti Group 2009 Page 22

12. When a standard framework is required –
sample process level template
Level of Abstraction of tasks is Activity Controls or ‘Work Aids’ provide options
key to a common framework for supporting templates, tools, procedures, etc.
Activity Name Activity Purpose Activity Controls
Action Artifact
Artifact
Inputs Entry Tasks Exit Outputs
Artifact Condition ActionAction Condition Artifact
Artifact
Artifact Condition Action Condition
Action
Artifact
Issues Roles Measurements
Planning, Name Name Costs, Efforts,
Tailoring, … Time, Quality
Tailoring can also be used to highlight
key differences of approach required
© Sogeti Group 2009 Page 23
Sample Test Process Element
Define the Test Strategy
Entry Criteria Controls / Work Aids Exit Criteria
1. Inputs are under 1. Project Lifecycles with defined Test Stages 1. Test-ware is placed under
Configuration Management 2. Test Methodology (identifying Test Methods & Test Configuration Management
control Techniques.) control.
3. Review Meeting Minutes Template
4. Risk List Template
5. Test Strategy Template
Inputs Tasks Outputs
1. Project Requirements 1. Determine level of Requirements definition for the 1. (Risk Based) Test Strategy.
Document(s) [User, project 2. Prioritised Risk List.
System, 2. Divide the System under Test into its components, if
Functional/Technical] relevant. Prioritise component parts
2. Statement of Work 3. Identify and analyse Product Risks, taking Project
3. Test Project Scope Requirements, Statement of Work, Prioritized Quality
Characteristics and System Components into
consideration
4. Identify and Define Test Levels.
5. Identify and Define Test Level Attributes - Methods and
Test Techniques, exit criteria, coverage, automation
approach, etc.
6. Document the Testing Strategy.
7. Review and sign-off the Testing Strategy.
Tailorings Roles Metrics
1. Test Manager (T.1-T.7) 1. Planned versus actual effort
2. Test Designer (T.5)
3. Business Analyst (T.3, T.7)
4. System Expert (T.3-T.7)
5. End User (T.3, T.7)
© Sogeti Group 2009 Page 24

13. Process of change
Obtain
awareness
Determine target,
area of consideration
and approach
Define
Execute improvement
assessment actions
Perform Formulate
evaluation plan
Implement
improvement
actions
© Sogeti Group 2009 Page 25
Suggestion: Use a performance-based ‘agile’
process improvement approach
Phase 2: Phase 1: Selected
Documented goals and problems
baselines of for process
current practice improvement
and performance
Phase 3: Process Phase 4: Revised
documents to be process and
deployed performance
baselines
© Sogeti Group 2009 Page 26

14. Do’s – from TPI® survey
• Clear scope and target 16%
• Clear communication 9%
• Results periodically visible 8%
• Champions in change team 8%
• Use of a change process 7%
• Other ... 5%
Biggest DO’s:
• Management commitment 24%
• Involvement of “own” people 23%
© Sogeti Group 2009 Page 27
Summary – key suggestions
• Use an agile approach to test process
improvement – incremental with
80/20 prioritisation
Driven by and linked to measurable
business goals
Addressing key staff process problems
• Involvement and ownership by test
staff
• Lightweight, usable process
definition (80/20 again!)
© Sogeti Group 2009 Page 28

15. Sogeti Support
• Capability building
test process and how to improve it
seminars, workshops and training
• Process assessments
Model based (e.g. TPI®) and/or benchmarking
• Consulting/mentoring during process
improvement
• fran.ohara@sogeti.com
• www.sogeti.ie
© Sogeti Group 2009 Page 29