A test strategy brings teams together to establish a foundational set of principles of quality into their apps together. Teams vary, and products and applications vary, so how do we build a testing strategy tailored to your needs that helps teams build quality products? In this session, we will look at the role of automation within a test strategy. I’ll provide heuristics to help guide you in identifying how many you need and what types of tests to automate. Then we will investigate the personas involved within automation projects. The creators, executors, and consumers of automation to help you tune your strategy to fit your team. Finally, I’ll share a model that teams can use to help guide them in crafting and discussing their own test strategy. A strategy that sets a foundation for quality, identifies where the app needs support, and how that risk is being managed through the software development lifecycle.

1. I
ARCHITECTING
A TEST
STRATEGY
Brendan Connolly
@theBConnolly
http://brendanconnolly.net

2. TEST STRATEGY
Guiding principles that
provide focus, context,
and drive actions
of the team to manage risk
@theBConnolly

3. Testing isn’t always well
understood
That leads to feelings of fear
When things go wrong scared
people often look to blame.
AM
BIGUITY
UN
CERTAIN
TY
BLAME
WHY STRATEGY
MATTERS
@theBConnolly

4. People love knowing there
is a plan
Testing means different
things to different people
Move together towards
quality
CLARITY
ALIGN
M
EN
T
QUALITY
WHY STRATEGY
MATTERS
@theBConnolly

5. ROLE
OF
AUTOMATION
I
@theBConnolly

6. How much time should
we spend writing,
maintaining, debugging
automation?
How many tests should
we automate?
How long should it take
to run?
What tests should we
automate?
What isn’t a good fit for
automation?
LOOKING FOR ANSWERS
TIME QUANTITY CHOICE
@theBConnolly

7. TEST
MONUMENTS
What can all these
Test Automation Pyramids tell us?
@theBConnolly

8. TEST PYRAMID
MATHEMATICS
Treating the Test Pyramid as a
triangle
Use geometry & trigonometry to
find the total area of each level
@theBConnolly

9. For the Original 3 Layer Test Pyramid
Assuming Each Layer Is Equally Tall
Using Different Dimensions the
Results Are Uniform
@theBConnolly
TEST PYRAMID MATHEMATICS
RESULTS
3 levels UI Service Unit
Great Pyramid 11.1% 33.3% 55.6%
Equilateral 11.1% 33.4% 55.5%
45-45-90 11.17% 33.21% 55.62%

10. Same Assumptions...
More Levels
@theBConnolly
TEST PYRAMID MATHEMATICS
RESULTS
4 levels UI Component Integration Unit
Great Pyramid 6.3% 18.7% 31.3% 43.7%
Equilateral 6.3% 18.8% 31.3% 43.7%
45-45-90 6.18% 18.79% 31.15% 43.88%
5 levels UI API Component Integration Unit
Great
Pyramid
4% 12% 19.8% 28.2% 36%

11. We can agree about the Top
We can agree about the Bottom
We undervalue the Delicious
Goodness in the Middle
@theBConnolly
THE GOOEY
CENTER Unit Gooey Center UI
36-55% 33-60% 4-11%

12. A QUESTION OF
QUANTITY
Per the Test Pyramid for every
100 tests added you would
have:
- 35-55 unit tests,
- 33-60 service tests
- 4-11 UI / E2E tests.
Does that fit with the practices
on your team?
@theBConnolly

13. @theBConnolly
A QUESTION OF
TIME
Per the Test Pyramid for a 40 hour
work week:
- 18 hours writing & maintaining
unit level tests
- 18 hours writing & maintaining
service level tests
- 4 hours writing & maintaining
UI level tests.

14. A QUESTION OF
EFFORT
Using Effort for estimation brings
automation in line with the feature
development.
Point Stories Inclusively but use
automation effort a metric for
comparison.
@theBConnolly

15. THE HANDFUL
HEURISTIC
For a given area of your
product you only need a
handful of end to end tests
There is great power in a
handful of meaningful &
reliable tests
@theBConnolly

16. CONFIRM &
CHARACTERIZE
Write the Tests that:
- Confirm that the Application is
Releasable
- Characterize Key Behaviors
@theBConnolly

17. ROLE
OF
THE TEAM
@theBConnolly

18. ROLE OF THE TEAM
PERSONAS
Your Team is Unique
To uncover the best
automation strategy
Understand your team first
@theBConnolly

19. The person or group
responsible for writing
test automation
The person or system
responsible for running
the automated test suite
The person or system
that derives value from
the results of test
execution
AUTOMATION
PERSONAS
CREATOR EXECUTOR CONSUMER
@theBConnolly

20. Writes tests while
developing new
functionality
Runs tests as changes
are made
Test results inform and
guide decisions through
development process
Developer Developer Developer
AUTOMATION PERSONAS
UNIT TESTS CASE #1
EXECUTOR CONSUMERCREATOR
@theBConnolly

21. Writes tests while
developing new
functionality
When code is checked in
tests are run and results
collected
Team knows their
individual changes are
not unintentionally
affecting others
Developer CI/CD Pipeline Dev Team
AUTOMATION PERSONAS
UNIT TESTS CASE #2
EXECUTORCREATOR CONSUMER
@theBConnolly

22. Writes tests while
developing new
functionality
When code is checked in
tests are run and results
collected
Tooling consumes or
analyzes to derive
actionable information
Developer CI/CD Pipeline Metrics
AUTOMATION PERSONAS
UNIT TESTS CASE #3
EXECUTORCREATOR CONSUMER
@theBConnolly

23. Contextual Benefit - Direct & Meaningful Impact
on Day to Day Work
Collective Ownership - Team comes together to support
quality outcomes
Multiple Consumers - Value increased and costs reduced as
tests provide multiple streams of value.
@theBConnolly
UNIT TEST
VALUE CHARACTERISTICS

24. QA / Tester writes
automated tests
Tests are run on some
cadence. Before release,
nightly build etc.
Who is this for? How Do
they benefit?
QA Schedule ???
AUTOMATION PERSONAS
UI / E2E TESTS
EXECUTORCREATOR CONSUMER
@theBConnolly

25. QA / Tester writes
automated tests
Tests are run on some
cadence. Before release,
nightly build etc.
QA / Tester reviews the
test results focusing on
tests they own
QA Schedule QA
AUTOMATION PERSONAS
UI / E2E TESTS CASE #1
EXECUTORCREATOR CONSUMER
@theBConnolly

26. A dedicated team of
automators writes tests
QA / Tester runs the
tests as needed based
on release schedule
Managers look at test
result reports tracking
trends and metrics.
Automation
Team
QA Management
AUTOMATION PERSONAS
UI / E2E TESTS CASE #2
EXECUTORCREATOR CONSUMER
@theBConnolly

27. Contextual Benefit - Ensure Tests Provide Direct & Meaningful
Value
Collective Ownership - It Takes a Village to Support
Automation
Multiple Consumers - Identify Questions That Automation Can
Answer for Consumers
@theBConnolly
UI / E2E TEST
SUCCESS CRITERIA

28. RE-MODELING
I
@theBConnolly

29. A MODEL OF
ISOLATION
THERE IS ALWAYS A BROADER
CONTEXT
WE TEST TO UNDERSTAND
THE REALITY OF THAT
CONTEXT
@theBConnolly

30. QUALITY
CATHEDRALI
@theBConnolly

31. Just like the Test Pyramid a
Quality Cathedral is built
upon a solid foundation of
unit tests
QUALITY CATHEDRAL
FOUNDATION
@theBConnolly

32. Focused On Building A
Quality Application that
Inspires & Engages Our
Users
QUALITY CATHEDRAL
APPLICATION
CENTRIC
@theBConnolly

33. Higher Level Tests Stand
Alongside the Application
Supporting the It Exactly
Where It Needs it
QUALITY CATHEDRAL
TARGETED
SUPPORTS
@theBConnolly

34. Free Standing Supports
that reinforce lower
level tests
QUALITY CATHEDRAL
BUTTRESSING
QUALITY
@theBConnolly

35. I
STRUCTURED
TO SUPPORT
THE WHOLE
@theBConnolly

36. A Network of Tests that
work together to withstand
the demand & stress your
product faces
QUALITY CATHEDRAL
SCALE &
WITHSTAND
@theBConnolly

37. You know your product
You know your team
Target where your application
needs support with your
resources to efficiently
manage risk
QUALITY CATHEDRAL
YOUR TEAM
YOUR SUPPORTS
@theBConnolly

38. CHARTERING
A
STRATEGY
I
@theBConnolly

39. WATCHING OUT
FOR WATERFALLS
Heavy Documentation
Explicit Processes
Bureaucracy
These do not guarantee better
outcomes
Strategy needs to be Agile too
@theBConnolly

40. EXPLORATORY TEST
CHARTERS
@theBConnolly

41. TEST STRATEGY
CHARTERS
Use (Testing / Automation Technique)
Led By (Owner / Creator)
To Support (Desired Outcomes)
@theBConnolly

42. TEST STRATEGY
CHARTER EXAMPLES
Use Unit Testing
Led By Developers
To Support a healthy codebase and easier
refactoring
Use End 2 End UI Tests
Led By QA with Dev support
To Support Exploratory testing on stable
builds
@theBConnolly

43. STRATEGY EVALUATION
EVENTS
When:
- Projects Kickoff / End
- Team Members
Change
- Incidents / Outages
- Team Retro’s
Discuss, Evolve, Reflect
@theBConnolly

44. TEST STRATEGY
RETROSPECTIVE
Even without a documented
test strategy, people will have their
own ideas about what types of
testing activities should or should
not be occurring.
It is worth periodically checking that
everyone is on the same page.
- Katrina Clokie
https://leanpub.com/testingindevops
@theBConnolly

45. BUILDING YOUR
STRATEGY
Your App, Your Team, Your Needs
Empower Quality
@theBConnolly

46. BE WELL,
CREATE,
SHARE
Connect With Me
@theBConnolly
http://brendanconnolly.net