This document discusses developing a test strategy based on understanding risk perception. It conducted a risk questionnaire that showed differing views of acceptable risk levels. This identified a need for greater testing rigor than currently applied. The strategy established clear development, product owner and testing responsibilities. It positioned testing as a consultancy to the business to provide expertise needed based on the risk appetite identified.

1. The Challenge

2. A Test Strategy
What is a test strategy?
- A Test Strategy Used to be a
document
- No-one wanted to read it
- Only testers knew what was in it
- Only testers cared what was in it

3. A Test Strategy
What is a test strategy?
- It’s not a document.
- It is embodied in the people that we
employ and the testing responsibilities
that we give them

4. The Problem
No clear test strategy?
- At the point of joining there was no
clear understanding of testing
responsibilities
- This was related to a lack of clarity on
the business expectation in testing
- There was no consensus over what
constituted acceptable risk in the
development process

5. Understanding Risk

6. Individual Risk Perception
What influences our perception of risk
- System One Thinking
- Availability Heuristic
- Recency Biases

7. Driving in Sweden
How we behave in the face of changes in perceived risk
- 3 September 1967
- Level of road fatalities
dropped
- After 2 years returned
to ‘normal’

8. Post 9/11
How we behave in the face of changes in perceived risk
- 11 September 2011
- Level of road fatalities
increased
- After 1 year returned
to ‘normal’

9. Risk Homeostasis Theory
How we behave in the face of changes in perceived risk
- Risk Compensation
- Changes to perceived
risk
- Further Evidence
- Evidence in Software
Security

10. Testers and Risk Perception
What influences testers’ perception of risk
- Stories = Bugs
- Personal Experience =
Bugs
- Availability = Bugs

11. The Business

12. The Business and
Risk Perception
What influences Business Owners’ perception of risk
- Lack of ‘Availability’
- Underestimation Bias
- Not just considering
software risks

13. The Business and
Risk Perception
What influences Business Owners’ perception of risk
- Risk of missing market
- Cost to develop vs
cost of failure
- Risk of upset
customers

14. The Business and
Risk Perception
Do Businesses Exhibit Risk Homeostasis?
Acceptable Risk
Time
Too Risky
Too Slow/Costly

15. Learning from Finance

16. Assessing Investor Risk
- Financial Advisors need to
assess individual risk
appetite
- Risk questionnaires are a
commonly used approach to
this

17. Assessing Investor Risk
If you could increase your chances of a return by
taking a higher risk would you:
a) Take more risk with all of my money
b) Take more risk with half of my money
c) Take more risk with quarter of my money
d) Not take the risk

18. Assessing Investor Risk
Assume you had an initial investment portfolio
worth £100,000. If, due to market conditions, your
portfolio fell would you:
a) Sell all of the investments. You do not intend to
take risks
b) Sell to cut your losses and reinvest into more
secure investment sectors
c) Hold the investment and sell nothing, expecting
performance to improve
d) Invest more funds to lower your average
investment price

19. Assessing Investor Risk
- The approach relies on an
understandable consistent
level of investor risk
- i.e. Homeostasis

20. The Risk Questionnaire

21. The Risk Questionnaire
Understanding acceptable level of business risk
is a critical element of a successful test
strategy
• Adopt too high a level of risk and you
impact customer relationships and
reputation and expose the business to
potentially legal consequences
• Adopt too rigourous an approach and you
impact overall cost-effectiveness of the
development process
Why did we do a risk questionnaire?

22. The Risk Questionnaire
Why did we do a risk questionnaire?
The aim of the risk questionnaire was to
• Establish whether there was an
understood level of risk across the
business
• Provide a discussion point if there was
not a consistent understanding of
acceptable risk
• Drive the understanding of acceptable
risk to help formulate a strategy

23. The Risk Questionnaire
What was the approach?
A series of questions was presented to all
respondents with answers ranging from 1 to 5
• 1 was always the option relating to the highest
risk position
• 5 was always the option relating to the lowest
risk position
4 Categories
1. Business Risk
2. Development Risk
3. Perceived/Target Status
4. Time Spent on Testing Activities

24. The Risk Questionnaire
The Questions
Questions needed careful crafting
• Not too generic
• Relevant to our customer and release context
• Not biased towards our own opinions or agenda

25. Business Risk Questions
Interaction between the Software and Customers
Rate the following questions on 1 - 5 from 1=Strongly agree 5=Strongly disagree
1. On time delivery is more important than taking longer to deliver a higher quality
product
2. I am happy to accept the need for later effort in maintaining a product if we can
deliver that product at a lower up-front cost
3. Our customers would knowingly accept a reduced level of rigour in
development compared to other products in order to keep the cost of the
software down
4. Putting the software in front of customers and responding to the issues they
encounter is a cost effective way to prioritise fixing software problems
5. The cost of fixing issues in production software is now reduced to the point that
this is an economically viable approach
6. Our product context is one in which we can adopt a relatively low level of rigour
compared to other business facing software development organisations
7. I would be reluctant to see an entire sprint given over entirely to testing and
bug fixing unless this was driven by issues encountered by the customer

26. The Risk Questionnaire – Business Risk
Min/Max/Average Ranges show a wide
range of perception on business risk
questions
• Wide range of opinion on the
acceptability of putting software
out early and fixing in production
• Apparent consensus on a low level
of rigour to achieve low up front
cost or meet delivery targets being
inappropriate for our context

27. Development Risk Questions
Focus on the Risks inherent in our development activity
Rate the following questions on 1 - 5 from 1=Strongly agree 5=Strongly disagree
1. The effective application of developer/unit testing can eliminate the need for
further devoted testing activity
2. Appropriate software design can eliminate the need for devoted performance
and stability testing
3. Adding further development skills in our agile teams provides more value in our
context than devoted testers
4. The testing of our products does not require specialist testing knowledge and
could be performed by individuals with limited training in software testing
5. I would be reluctant to schedule specific testing tasks on a team’s backlog
without any associated development

28. The Risk Questionnaire – Development Risk
Min/Max/Average Ranges show greater consensus on
development oriented risk questions
• General agreement around the value of specialist testing
and targeted test activities
• Wider range of responses around relative value of
testing to developer/unit testing

29. Testing Commitment
Looking at the Time Spent on Testing Activities
Rate the following in terms of time
1 = Less than 1 hour;
2 = 2 - 4 hours;
3 = 0.5 - 1 days ;
4 = 1-2 days;
5 = more than 2 days
1. How much time out of a typical user story development taking 1 person-
week in total should be given over to the creation of unit tests?
2. How much time out of a typical user story development taking 1 person-
week in total should be given over to automated acceptance testing?
3. How much time out of a typical user story development taking 1 person-
week in total should be given over to human exploratory testing?

30. The Risk Questionnaire – Development Risk
Consensus on time spent unit testing
• Then diverging opinion on time to spend on testing
activities
• Decreasing expectation as we move away from
automation and into exploration

31. Perception Questions
How do we perceive our current and target status in testing
• With 1 being lowest and 5 highest rate how you think the
company currently stands in its typical level of rigour in
software quality and testing?
• With 1 being lowest and 5 highest rate how you think the
company should stand in its typical level of rigour in software
quality and testing?

32. The Risk Questionnaire – Perception
The questions on company perception indicated an acknowledgement of a need for
greater rigour in its quality and testing
• Average for the perception of current level of rigour was 2.6 / 5
• Average for the target level of rigour was 4 / 5

33. Crafting a Strategy

34. The Risk Questionnaire – Conclusions
• General pattern of responses was towards the lower end of the risk
options indicating a desire for greater rigour than currently applied
• Increasing the level of rigour in development work is rarely achievable
without cost
• The cost must be factored in to our project estimates
• These costs must be justified by potential savings in other areas such as
support

35. Establish Clear Responsibilities
- Developers
- Focus on the “Solution” domain i.e. make sure what we have
delivered works as we designed
- “Build the thing right”
- PO
- Focus on the “Problem” domain i.e. make sure what we have
designed meets the business need and adds value
- “Build the right thing.
- Testing
- Identifying Testing needs
- Providing expertise and assistance to deliver these
We need to provide clear expectation for the roles in the business?

36. Testing as a Consultancy

37. The Risk Questionnaire – Learning
• The questionnaire helped us to progress along the ‘testing wave’
• Questions weren’t calibrated for consistency
• Result allowed the establishment of a test strategy appropriate to the
risk appetite of the business – a testing consultancy
• Provided justification for hiring

38. Thank You