The document discusses the role of testing in ensuring an organization's product and service value. It covers the history of software testing and how the focus has shifted from quality assurance to value assurance. The presentation examines different value drivers for testing like user experience testing, testing in production, crowd testing and more. It acknowledges challenges in measuring value and discusses moving from objective to more subjective metrics to better understand user perception of value.

1. The role of testing in
the organisation’s
value assurance
itSMF Belgium – 22 April 2022
Wim Demey – Solution Architect Testing (CTG)

2. The facets of value
“ someone’s perception of the worth
of a product or a service ”
Product
Value
User
Experience
Customer
Experience

3. The facets of value
“ someone’s perception of the worth
of a product or a service ”

4. IT Service Management
Software Testing
Value… priceless like a diamond

5. IT Service Management
ITIL 4 Service Value System

6. Software Testing
The impact of its history
Quality assurance vs value assurance
How does testing deal with subjectivity?
History of testing
Triangle of testing
Value drivers
How does testing relate to value?

7. “application of a patch for a particular bug as a phase
within the stage of software development”
1947 – 1956 | Debugging
History of
testing

8. 1957 – 1978 | Demonstration
“demonstrate that the program
did what had previously been
said to be done, using expected
and recognisable parameters”
(after Charles Baker)
History of
testing

9. 1979 – 1982 | Destruction
“testing as the process of
executing a program with the
intent of finding errors”
History of
testing

10. 1983 – 1987 | Evaluation
“testing as a
methodology which
includes analysis,
review, and testing
activities”
History of
testing

11. 1988 – now | Prevention
“testing becomes focused
on preventing program
failures before they occur”
History of
testing

12. Triangle of
testing
Quality
Assurance
Quality
Control
Value
Assurance
“Activities focused on providing confidence that quality
requirements are fulfilled”
(definition according to ISTQB - https://glossary.istqb.org)

13. Quality
Assurance
Quality
Control
Value
Assurance
“A set of activities designed to evaluate the quality of a component or a
system ”
(definition according to ISTQB - https://glossary.istqb.org)
Triangle of
testing

14. Quality
Assurance
Quality
Control
Value
Assurance
“ The process of checking that projects make the right (value)
commitments to the business and deliver on these objectives ”
Triangle of
testing

15. REQ 1: The washing machine has multiple programs
REQ 2: Any program can be selected by switching the button to the left
or to the right
REQ 3: Each program has a unique pictogram
PROMO TITLE : Even a baby can use our washing machine
Triangle of
testing

16. Value
Drivers
*Extract of World Quality Report 2020-2021, page 11
…
Be part of the business.
IT can also help itself by getting closer to the business,
learning its skills and taking time to understand its
objectives.
…
Listen more to users.
”Is this working for you? Tell us what you think.” Asking
and listening will make things better, and will make
customers happier.
…
QA orchestration in agile and DevOps

17. Value
Drivers
(Source: https://agilemanifesto.org)
Agile Manifesto (2001)

18. Value
Drivers
Behaviour Driven Development (BDD)
Business
Domain Driven Design
“ A collaborative approach to development in which the team is
focusing on delivering expected behaviour of a component or
system for the customer, which forms the basis for testing “
Technology
Test Driven Design
BDD
(Source: https://glossary.istqb.org/en/term/behavior-driven-development)

19. Value
Drivers
Behaviour Driven Development (BDD)
(Source: https://www.testbytes.net/blog/behavior-driven-development)

20. Value
Drivers
Behaviour Driven Development (BDD)
(Source: https://www.testbytes.net/blog/behavior-driven-development)
Feature : returned items go back to the stock
In order to keep track of the stock
As a store owner
I want to add items back to the stock when they are returned
Scenario: refunded items should be returned to the stock
Given a customer previously bought a pair of shoes from me
And I currently have 50 pairs left in the stock
When the customer returns his pair of shoes for a refund
Then I should have 51 pair of shoes in the stock

21. Value
Drivers
User (Acceptance) Testing
User
Interface
User
Experience
Customer
Experience
“ We have to understand that we don’t inherently
understand other people.
Even if another person is like us in every obvious
way, they are unlike us in one very special way “
(1993, Dan Norman)

22. Value
Drivers
User (Acceptance) Testing

23. Value
Drivers
User (Acceptance) Testing

24. Value
Drivers
User (Acceptance) Testing

25. Value
Drivers
Testing in production
Modern QA is not about if you test
in production, but when you do it
Complex
environments
Performance &
scalability
Usability over
functionality
Chaos
Engineering

26. Value
Drivers
Testing in production
50% users
50% users
25% conversion
40% conversion
A/B Testing
85% users
Version A
Version B
15% users
Canary release

27. Value
Drivers
Crowd testing
(Link: https://we-are-testers.com)

28. Value
Drivers
Exploratory testing
(Link: https://medium.com/agile-outside-the-box/exploratory-testing-quick-start-942bbf6a738a)

29. Challenges in value measurement
IT Service Management
Software Testing
Watermelon
effect
Subjectivity

30. Impact of
user testing
Traditional metrics:
- Total number of test cases
- Number of test cases passed/failed/blocked
- Number of defects found
- Number of defects according to their status
(e.g. new/fixed/closed/rejected)
- Number of defects according to their severity
(e.g. cosmetic – minor – major – blocking)
- ….

31. Shrink
the effect
- User acceptance testing
- User acceptance testing
- Exploratory testing
- Crowd testing
- User acceptance testing
- (Value based) exploratory testing
- (Value based) crowd testing

32. Shrink
the effect
(Source: https://www.happysignals.com/)

33. Shrink
the effect
Platform that increase level of
self-servicing for employees
Do the employees use more self-services
with the new platform?
How do you rate the
user interface of the
new platform?
Are the services self-
explaining?
How is the interaction
with the Service Desk
when you have a
question?
UX Design Team Customer Team
How do you rate the
available features of
the new platform?
Service Desk Team Application Team
# usability issues
# completed customer
surveys
# performance issues
# calls with CT
# calls with SDT
Success rate of
provisioning self-
services
CSAT – NPS score
Talk time
# First time right
answers
# positive SDT
reviews
# application issues
# updates/fixes
# releases
Test related metrics
Desired Experience XLA question X- data
SLA related team
SLA/KPI data

34. The new
normal
Soft
Subjective
Data
Hard
Objective
Data

35. The new
normal
Soft
Subjective
Data
Hard
Objective
Data

36. The new
normal
ISO 25010 – Product quality model

37. The new
normal
Machine Intelligence quality characteristics
(Source: https://www.sogeti.com/explore/blog/machine-intelligence-quality-characteristics-for-
better-ai-solutions//)

38. Testing doesn’t create value
but supports confidence in the value
pursued by an organisation
Wim Demey – Solution Architect Testing (CTG)
wim.demey@ctg.com
https://www.linkedin.com/in/wimdemey/

39. References to icons used in this presentation:
• Diamond icons created by Freepik – Flaticon
• Product icons created by Freepik – Flaticon
• User experience icons created by Freepik – Flaticon
• Diamond icons created by Freepik – Flaticon
• Facts icons created by Freepik - Flaticon
• Diamond icons created by bqlqn – Flaticon
• Lightning icons created by Good Ware – Flaticon
• Summer icons created by Smashicons - Flaticon
Wim Demey – Solution Architect Testing (CTG)
wim.demey@ctg.com
https://www.linkedin.com/in/wimdemey/