We call them automation frameworks, but typically they are a single purpose software full of assumptions around that purpose, tightly coupled to executing complete end to end test cases. Inspired by the Richardson Maturity Model for REST Api’s, I will share a heuristic model of automated testing maturity consisting of four levels: Level 0: Tests exist, but framework does not. Level 1: UI Frame work exists Level 2: Multi Layer framework, Beyond UI, api or other components included for arranging or manipulating, only consumer is tests. Level 3: Testing SDK, Separate libraries/packages, consumable outside of tests and multiple consumers.

1. Automation Maturity
Heuristic

2. PONDERING
the automation universe
2

3. 3
Hopes
Dreams

4. Maturity
HEURISTIC
4
https://martinfowler.com/articles/richardsonMaturityModel.html
A Tool for Comparison & Conversation
Expectations & Experiences Vary Broadly

5. 5
Level Zero
TESTS WITHOUT A
FRAMEWORK

6. 6
Tool Focused
○ Have Problem, Want It Gone
○ Someone found, read about, was
sold on just the tool for the job
Level ZERO
Tests Without A Framework

7. Dream
7

8. 8
Like A Kid in a Candy Store
○ Sugared Up & Ready to Conquer the World
Level ZERO
Tests Without A Framework

9. Reality
9

10. 10
Let’s Not Be Quick To Judge
○ We All Start Here
○ It’s Not The Tools Fault
○ Direct Impact For Testers
Level ZERO
Tests Without A Framework

11. 11
Level ONE
A FRAMEWORK
EXISTS

12. 12
The Honeymoon is Over
○ Test Start to Fail
○ Maintenance Costs
○ It’s More Difficult Keeping a Test Suite than
Getting One
Level ONE
A Framework Exists

13. 13
Unifying Automation
○ Portable ( Internally Consistent)
○ Maintainable
○ Reliable
Level ONE
A Framework Exists

14. 14
Pattern Focused
○ Design Patterns
○ Conventions
○ Centralization
Level ONE
A Framework Exists

15. 15
It’s Harder to Unwind, than Introduce
○ Patterns & Conventions are process
for your code
○ Increased Bureaucracy, Complexity & Barrier to
Entry
Pumping the BRAKES

16. 16
Level TWO
A MULTI-LAYER
FRAMEWORK

17. Level TWO
A Multi-Layer Framework
17
So Many Tests, So Little Time
○ A Victim of Your Own Success
○ Continually Growing
○ Need to Reduce “Time to Feedback”

18. Level TWO
A Multi-Layer Framework
18
Seeking Efficiencies
○ Reliability & Frequency
○ Time & Cost

19. 19
Architecture Focused
○ Leveraging Application Layers
○ Carving Out the Excess UI from Tests
○ Finding Seams to Stage UI Interactions
Level TWO
A Multi-Layer Framework

20. Hybrid
Maturity
20
As we turn our attention to our
applications architecture
We must also turn inward &
investigate the architecture of
our automation

21. Existential
TESTING CRISIS
21

22. 22
What Are We Testing Here?
○ Accepting Approximations
○ Understanding Motivations
○ Test Complexity, Nuance & Strategy
Existential TESTING CRISIS

23. 23
Automation Bragging Rights
○ How Big Is Your Test Suite?
○ How Long Does it Take to Run All Your Tests?
○ How Many Servers Do You Have to Spin Up?
Level TWO
A Multi-Layer Framework

24. A RIFT IN QUALITY
24

25. A Rift in QUALITY
25
We are all on TEAM QUALITY
Unfortunately our Automation does not
always reflect that

26. 26
Niche MARKET
Framework Value is Constrained by:
○ Number of Good Candidates for Automation
○ How Fast Tests Can Execute

27. 27
Handful Heuristic
Each “Area” of an Application Only Needs a
Handful of End to End Tests.

28. 28
Handful Heuristic
○ Quantity is a path to Complexity
○ Higher Quantities May Reveal Gaps In
Coverage/Understanding of Lower Level Tests
○ The Need to Scale/Parallelize Could Be a Tech
Solution to a Strategy Problem

29. 29
Level THREE
A TESTING SDK

30. 30
Software Development Kit
A set of software tools and programs used by
developers to create applications for specific
platforms.
Level THREE
A Testing SDK
Source: https://clevertap.com/blog/what-is-an-sdk/

31. 31
Platform Focused
○ A Set of Tools To Compose Solutions
○ Separate Libraries / Packages
○ Used Outside of Traditional Tests
Level THREE
A Testing SDK

32. 32
Reframing
AUTOMATION

33. INCREASES FLEXIBILITY
33

34. Why Build a Monument
TO AUTOMATED TESTS?
34

35. Building a
QUALITY CATHEDRAL
35
The Quality of our Application
is the Core.
Create & Use libraries, tools &
tests to counter-balance risk
with efficient supports
.

36. RELIEVES PAIN
36

37. Relieves Pain
37
Lifting Everyone Up
○ Serve More People
○ Solve More Problems
○ Total Addressable Market GROWS
○ Same Code, Lower Cost

38. Bundles of JOY
38
Joy for Testers & Automators Alike
○ Scripts, Apps, Chatbots to Help Your Testing
○ External Integrations Have API’s Too
○ Open Sourcing Projects

39. RAISES CONSCIOUSNESS
39

40. Raises CONSCIOUSNESS
40
Expose > Engage > Contribute > Evangelize
○ Successful automation needs an engaged
audience.
○ People need to feel its impact.

41. REDUCED ANXIETY
41

42. 42
Automation Is Exclusionary
✘ Tests That Don’t Fit
✘ External Integrations
✘ People That Don’t Code
✘ The Tester
Reduces ANXIETY

43. 43
Inclusive AUTOMATION
We Can Perpetuate Exclusion in Our Automation,
Or We can Build an Inclusive Platform for Quality.
We Don't Lose Anything,
But We Have So Much Quality to Gain.

44. A FEELING OF PEACE
44
Level 0: Tests Without A Framework
Level 1: Framework Exists
Level 2: Multi-Layer Framework
Level 3: Testing SDK
Holistic automation enabled strategy

45. Be Well,
Create
Share
45
Connect With Me