This document provides an overview of creating a testing strategy. It begins with explaining why a testing strategy is important, as testing accounts for a large portion of IT budgets. It then discusses the key questions a testing strategy should answer: what to test, where to test, when to test, how to test, and who will test. The document outlines a process for creating a testing strategy, including assessing the current state, defining a future vision, and creating a roadmap to get from the current to future state. It provides examples of what to include under each section of the strategy, such as system architecture under "what to test" and test environments under "where to test". Overall, the document provides guidance on developing a

1. NOTICE: PROPRIETARY AND CONFIDENTIAL
This material is proprietary to and contains trade secrets and information which is solely the property of Centric Consulting, LLC. It is solely for the Client’s internal use and shall not be used, reproduced, copied, disclosed, transmitted, in whole or in part,
without the express consent of Centric Consulting, LLC. © 2016 Centric Consulting, LLC. All rights reserved.
Testing
Strategy:
The Silver Bullet
Presented by: Matthew Eakin, Test Strategist
Matt.Eakin@centricconsulting.com
@MatthewEakin
MatthewEakin

2. Agenda
•Why a Testing Strategy?
•What is a Testing Strategy?
•What – needs to be tested?
•Where – is it going to be tested?
•When – is it going to be tested?
•Who – is going to test it?
•How – are they going to test it?
•Steps to creating a solid Test Strategy:
1. Assessment of current state
2. Vision of where you want to be
3. Create Testing Strategy
4. Create Roadmap (based on Strategy) to get from current
to future state
2

3. Why a Testing Strategy?
• 29% of an IT’s budget is typically spent on testing. This is expected to grow to 34%
in the next several years
• CIO’s are transitioning organizations towards Bi-Modal IT shops, maturing agile into
DevOps. Integration of quality activities and the promotion of whole team
accountability of quality is paramount to successful transformations.
• Meanwhile, 40% of organizations lack sufficient transparency into QA activities and
QA evaluations
• Agile and DevOps requires a strategic focus on automation to reduce feedback
times while improving coverage
• Better defect detection (72%)
• Reduced cycle times (69%)
• Reduced testing costs (67%)
• Better test re-use (66%)

4. Driving Forces in IT today
Historic Testing Response:
4
With Less
Do More
Historic Testing Response:
“Add more Testers”
“Buy more tools”

5. What is a Test Strategy
What needs to be tested

6. Strategy vs. Plan
Test Strategy Test Plan
Written once Re-Written for each Project
Written for each application or system Re-Written for each Project
Explains WHY? Why use a specific
tool? Why do you need x resources?
Why you need a dedicated QA env?
Details Who tests, What they test,
When they test and Where they do the
testing
6
“Planning is everything. Plans are nothing.”
–Field Marshal Helmuth Graf von Moltke (“Moltke the Younger”)
–Often attributed to Dwight D. Eisenhower

7. 5 Basic Questions
•What – needs to be tested?
System Architecture
•Where – is it going to be tested?
Environment
•When – is it going to be tested?
Cadence
•How – is it going to be tested?
Tools
•Who – is going to test it?
Resources
7

8. System Architecture
What needs to be tested

9. 9

10. User Behavior Tests
10
•Acceptance Tests
•Functional Tests
•End-to-End Tests
•UI Smoke Tests
•Multi-browsers
•Mobile

11. Middle-Tier Tests
11
•API Tests
•Security Tests
•Some Performance
Tests
•Some Integration
Tests

12. Server-Side Tests
12
•Database Tests
•Some Integration
Tests
•Unit Tests

13. Test Automation Pyramid
13

14. Brian Marick’s Testing Quadrants

15. Environment
Where is it going to be tested?

16. 16

17. Environment Purpose
What test(s) are best suited for which environment?
17
DEV CI Test/QA UAT Perf
Unit X
Smoke X X X X
Integration X
Functional X X
API X X
E2E X
Behavior X X
Mobile X X
Multi-
browser
X X
Look &
Feel
X

18. Deeper Dive
When talking about environments, numerous other discussions will happen:
•Databases
•Test data needed
•Clients needed in each environment
•Continuous Integration (CI)
•How will code, configuration and databases get migrated from
environment to environment?
•Smoke tests
•Code versioning – which code is in which environment?
•Virtual machines (VM’s)?
•Integration with external systems
18

19. Cadence
When is it going to be tested?

20. •Run test when Functionality is ready? Daily? End of Sprint?
Prior to Release?
20

21. Tools
How are the tests going to be executed?

22. Needs Checklist
KEY: you don’t need 1 tool that “does it all.” But you are going to need
tool(s) to do what you need done.
What have you learned thus far?
•Types of tests needed to be executed (API tests, DB validation, UI tests,
Integration tests, etc.)
•Will you be using CI to migrate code?
•When will the tests be executed during a Sprint (cadence)?
•Will you need to run a single test? Group of tests? Smoke tests?
•Collaboration/Review on scripts?
22

23. 23

24. Resources
Who is going to be executing the tests?

25. Skillsets & Ownership
•Do you have the skillset internally to execute what needs tested?
•Current team, potential contactors, external teams
•Dunning-Kruger Effect -> you need to be honest and objective
•Who owns testing?
•Development? Architects? Business? Testing?
KEY: It doesn’t matter who “owns” testing. Everyone has some testing
which needs done to ensure the product is great.
You need to own the coordination of execution (a.k.a. the Strategy)
25

26. Additional Strategies
Important areas which need to be addressed

27. What else…
Out of these initial discussions, quite a few additional topics will come up:
•Reporting – when is the best time to report your testing findings?
•Metrics – what are the best metrics?
•Internal (team), external (product owner)
•Defect management – best way to get defects triaged and in the hands of
those who can fix them
•Training – testers, analysts, developers, product owners
•Use of contractors
•Coaching, training
•Skill & resource gaps fill-in
•Use of Off-shore
•Is there a place for it?
27

28. Where to start

29. 3 Things you need to figure out…
1.Where are you?
Current state…
2.Where do you want to go?
Future state…
3.How are you going to get from Current
to Future state?
Roadmap…
29

30. Current State
How to determine where you are…

31. Assessments
•Many available on internet
•Google “software testing assessment”, “Testing Maturity Model”, “test
process assessment”
•Many consulting companies have canned assessments
•All good, but can be costly
Key: Do the right kind of assessment to identify your problems
•Manual testing assessment, test automation assessment, performance
assessment, etc..
Key: Be Objective/honest
•Very difficult to do but critical
31

32. What is an assessment?
32

33. Breakdown of a Category
33

34. Future State
Where do you want to be…

35. Again, Look to the Assessment
•You know where you are. How high up do you want to go?
•How high up can you realistically go? In 1 year? In 2 years? In 5
years?
35

36. Roadmap
How to get from Current to Future State

37. Again, Look to the Assessment
•If you want to attain Optimizing maturity, and are currently at Controlled, the
Assessment can provide a roadmap to get from Current to Future state.
37

38. Test Strategy Document
Putting it all together

39. Putting it all together
•Putting a Strategy together
is like putting together a
1,000 piece puzzle…
…Without a picture
39

40. What to consider
•Pain Points
•What is causing the most testing pain?
•Budget (testing and department)
•What do you have the $$ to do?
•Department Priorities
•What does everyone else on the team feel is important?
•Department Politics & Culture
•What can you realistically get done?
•What will people realistically do?
•What is completely new?
40

41. Each Criteria
•Documentation of current state
•State the facts (good, bad, ugly)
•Document what you found
•Recommendations
•How do you plan to improve?
•Tools?
•Personnel?
•Training?
•System resources?
•When do you plan to improve?
•Immediate (now), Short-Term (over the next few months), Medium-
Term (1-2 years), Long-Term (future)
41

42. Create a Timeline (Backlog)
•Immediate (what can be done now)
•List out all the first steps in each criteria
•Some criteria may have more than 1
•Prioritize what you want to tackle 1st, 2nd, etc..
•Short Term (what can be done in the near future)
•List out all the “2nd” steps in each criteria, and any leftover 1st steps
•Prioritize what you want to tackle 1st, 2nd, etc..
•Mid Term (what can be done in 1-2 years)
•These take time to plan, get funding, get resources, etc..
•Long Term (completed will into the future)
•Don’t waste too much time dreaming these up as they will most likely
change by the time you get there
42

43. Start with the Biggest Pain Points
•What is causing the most
pain?
Focus on it first
•Teams can only handle
change in small doses
Focus on 2-3 things at
a time
43

44. Create a KanBan Board
44

45. How does this enable
you to do more,
with less?

46. Doing more
•Entire stack/team testing
•Focus testing efforts on high-risk/high-pain areas
•Better view of all tests -> remove duplicates
•Make a case for test automation
•Make a case for better tools
•Make a case for training
46

47. With Less
•Make a case for open source tools
•vs. current high-cost licensed tools
•Better view of all tests -> remove duplicates
•Make a case for consultants
•More resources at reduced cost
•Make a case for test automation
•Make a case for off-shore resources
47