Anton muzhailo - formal test process improvement. how to invest to the test process improvement and win

1. Formal approach to Test Process
Improvement. How to invest & win?
Anton Muzhailo, GlobalLogic, Ukraine

2. About the speaker
Anton Muzhailo
• Senior Automation Engineer,
GlobalLogic
• ISTQB Certified Test Manager
• ISTQB Certified Trainings Coach,
CodeSpace
• 3 years of mentoring experience,
250+ students

3. Should you improve your testing process?
A. Yes B. No
C. Maybe D. What?

4. The truth are always in details
Nothing should be improved for the sake of improvement!
_____ you improve your test process?
Can Should

5. Return of investment (ROI)
ROI – is a difference between the size of investment and a profit that it
brings.
Always consider ROI in scope of time:
• Immediate benefit
• Short-term benefit
• Long-term benefit
𝑅𝑂𝐼 % =
𝑃𝑟𝑜𝑓𝑖𝑡 − 𝐼𝑛𝑣𝑒𝑠𝑡𝑚𝑒𝑛𝑡 𝑐𝑜𝑠𝑡
𝐼𝑛𝑣𝑒𝑠𝑡𝑚𝑒𝑛𝑡 𝑐𝑜𝑠𝑡
∗ 100%
-10
-5
0
5
10
15
20
25
Month 1 Month 2 Month 3 Month 4
Profit %

6. Why not to improve?
No money – no honeyToo late
High improvement
implementation risks
Little or no ROI
People doesn’t want
changes

7. Product
Quality
Development
technology
Process
quality
Money, scope,
schedule
People
quality
So, you decided to improve…
.. but what?

8. Improving development technology
1. Find something to improve. (HINT: Easy to find, pick anything)
2. Raise the question to developers. Explain how they are responsible for quality.
3. Involve project/product management if needed
4. Respond on questions why current technology is bad
5. Expect severe resistance even if you have strong arguments
6. Fight to the death or leave with nothing
Сollect the evidences of a
poor technology.
Introduce the metrics that will
reveal the problemsOR
Always play in terms of ROI

9. Improving people quality
Build the
team
Fire & hire
another
Do the job
for them

10. Improving money/schedule/scope?

11. Improving the test process
Test process
Improvement types
Model-based
approaches
Process referenced
Content referenced
Analytical
approaches
Casual analysis
GQM (goal,
question, metric)
Retrospectives

12. Existing Improvement Models
TMM – Test Maturity Model
STEP – Systematic Test Evaluation Process
CTP – Critical Test Processes
TPI – Test Process Improvement
TMap – Test Management approach
TSM – Testability Support Model
TAP - Testing Assessment Programme
TIM – Test Improvement Model
TOM – Test Organization Maturity
SQR – Software Quality Rank
TMM
TPI
CTP
STEP
TOM
TIM
SQR
TMap
TAP TSM

13. TPI Next Areas and Levels
Stakeholder
Relations
1 Stakeholder commitment
2 Degree of involvement
3 Test Strategy
4 Test Organization
5 Communication
6 Reporting
Test Profession
12 Methodology Practice
13 Tester Professionalism
14 Test Case Design
15 Test Tools
16 Test Environment
Test
Management
7 Test Process Management
8 Estimating and Planning
9 Metrics
10 Defect Management
11 Testware Management
This is what your test
process consists of

14. TPI NEXT Levels
# Level Meaning
1 Initial Chaotic, ad-hoc processes
2 Controlled Doing the right activities
3 Efficient Doing the activities right (effective)
4 Optimizing Continuously improving the activities
This is how “cool” is
every part of your
testing process

15. General Picture
I
(Initial)
II
(Controlled)
III
(Efficient)
IV
(Optimizing)
16 key areas
key areas
maturity levels
checkpoints
16
4
157

16. TPI Next Maturity Matrix example

17. TPI Next Implementation Process
Evaluate current
situation
Plan this
improvement cycle
Implement
according to the
plan
Evaluate the
results
Plan another
improvement cycle

18. Excel workbook that:
• has 16 sheets for each key area
• Each area has maturity levels
• Each maturity level has its
checkpoints
• Each checkpoint has Y/N flag to
select
User Manual with all the magic
explained deeper
http://www.tmap.net/tpi-downloads

21. Those unfilled checkpoints
are our first goal to satisfy!
For now, assume that all
unfilled checkpoints are equal
within one level!
We can’t deal with this
one as we are not passed
the Controlled level yet!

22. The principal stakeholder
agrees with the
documented test strategy
Each test case is related to
a test basis document in a
transparent way
Test activities are started early, timely
before test execution, with the goal to
keep the test activities of the project’s
critical path
A tester is involved in the
analysis and mitigation of
project risks
Test environment
requirements are
documented

23. The reporting contains trends
and recommendations
concerning test goals and
product risks.
The test policy is followed.
To be as accurate as possible, at
least two estimating techniques
are used.
Defects are analyzed for common
properties and recommendations
are made to avoid future defects.
The test team has an undisputed
part in all relevant development
activities, being accepted and
valued.

24. P Checkpoint Implementation way Owner
1
The principal stakeholder agrees
with the documented test
strategy
Conduct formal walkthrough review of a test strategy.
Explain the test strategy is project-dependent. May
require the involve of upper management.
Nilson
1
Each test case is related to a test
basis document in a transparent
way
Evaluate if test management tool is able to create the
traceability matrix. Make sure there is a formal
management of test basis.
Nilson
2
A tester is involved in the analysis
and mitigation of project risks
Ask if there are any objections to involve tester to
project risks analysis. Get an aggreement.
McGower
2
Test activities are started early,
timely before test execution, with
the goal to keep the test activities
of the project’s critical path
Communicate with Nilson. Clear and aggreed test
strategy should solve this problem. Check the test plan
too.
Frattia
3
Test environment requirements
are documented
Document all the test environments and get the
approval. Establish the formal process of test
environments documentation
Alvarez

25. 4 of 5 improvements were
implemented successfully
Was not implemented due to severe resistance
from management as they think that project risks
should only be handled by management. May
require audit or independent assessment to solve

26. ?
in/muzhailo
Anton Muzhailo