The Test Automation Pyramid is a useful model to help us understand and discuss automated testing efforts. Generally speaking it is a good practice to have lots of unit tests, fewer component integration tests, fewer API tests, and even fewer UI tests.
2. Test Automation
Why?
Allows you to test more with less effort
Allows you to deploy/release more quickly and more frequently
Higher quality software
How?
As a natural part of the development process
A story is not done until appropriate automated tests are passing
3. Test Automation Pyramid
Shows different technical levels on which
you can have automated tests
A model
Going up
Scope increases
Execution time increases
Effort increases
Guideline for relative distribution of your
testing effort
4. Unit Tests
Testing smallest possible parts of the system
in isolation
Use of test doubles to replace dependencies
Dummies, stubs, spies, fakes, mocks
Smallest scope
Easier to locate and understand errors
Best performance
Execute early and often
Won’t catch all defects
5. Component Integration Tests
● Testing that multiple components work
together
● Test doubles are not used for the integration(s)
you want to test
● Varying scope
● Good performance
● Won’t catch all defects
6. API Tests
Testing that your APIs behave as expected
REST
SOAP
...
Big scope
Involves many components
Decent performance
Won’t catch all defects
7. UI Tests
End-to-end testing
Maximum scope
Can catch many problems
Worst performance
Don't use UI for setup and teardown
Parallelize
Can be brittle, flaky and a lot of work to maintain
Test critical user workflows
Won’t catch all defects
8. Exploratory Testing
Manual testing
Not regression testing
Experience + creativity
Used to learn about the system, discover defects
and to improve automated testing
Can be based on a mission/test charter or persona
Test for accurate and helpful error messages when registering
an expense incorrectly
Test for unexpected results
when configuring company
details in two different tabs
Impatient manager new to the system
9. Where is…?
● Acceptance testing
● BDD
● Smoke testing
● Security testing
● Performance testing
● System testing
● System integration testing
● …
10. What should I aim for?
Depends, discuss in your team
As an example, Google suggests
70% unit tests
20% integration tests
10% end-to-end tests
14. Testing Week Task
Confluence: Test Automation Pyramid - Testing Week 2015
How does your team's test automation effort fit into the Test Automation Pyramid,
how does your pyramid look today, and how can it improve going forward?
17. Example workflow
Refinement process
3 amigos: PO/BA + dev +
QA/tester
Acceptance criteria finalized
Acceptance tests finalized
Risks and mitigation
What should be tested, on
which level
19. Example workflow: Test execution
Balance between quick feedback
and discovering problems early
During implementation
After implementation
At the beginning of your delivery
pipeline (CI server)
After deployments
Internal Test
User Acceptance
Production
Exploratory testing
20. Summary
The test automation pyramid is a useful thinking tool and discussion reference
Make conscious decisions about what to test on which level, and also where you
run which tests
Try to distribute your tests optimally
High quality
High velocity
Medium effort
Discuss with your team
Do the Testing Week task
My name is Alexander, and I’m usually one floor above, working on Quality Assurance stuff, although I am here quite often as well
I want to give an introduction to the test automation pyramid, a concept which I think it is useful for everyone involved in the development process to know about, so hopefully we have many different roles in the audience
Im going to start off with some observations about test automation in general
When you boil it all down, we usually do test automation because it allows us to test more with less effort
You invest some time into designing, implementing and maintaining your automated tests, and from that investment you get certain benefits:
Don’t need to spend time and effort to do manual testing, at least not regression testing
We can test things that are not possible or practical to do manually
In some ways, automated tests are more accurate and more reliable
Run tests all the time, continuous verification and quick feedback on that your service works as expect while we are constantly changing it
When more and more of your test effort is automated, you can release in a shorter amount of time, and also more frequently which should be a goal in itself
Result hopefully, is higher software quality
Test automation should be done as a natural part of the development process, and not as an afterthought or something to be done when you have time
If you do that, you will accumulate technical debt, that you will suffer for later
A story is not done until appropriate automated tests are passing
What is appropriate? That depends on many different factors, which is why it should be discussed in the team prior to implementation of a user story
In that discussion you can use the test automation pyramid
Shows different technical levels on which you can have automated tests: Unit Tests, Component Integration Tests, ...
An important thing to mention is that this is a model. It is a thinking tool, something to be used as a reference, and it is imperfect by definition. You can create a pyramid like this in many different ways, and I will show you some examples of that later.
As you move up in the pyramid, the scope of the tests increase, and their execution time, and the effort needed to create/run/maintain them
Model argues that you should probably have more unit tests than component integration tests,
more component integration tests than API tests, more UI tests than API tests, and so on.
We’ll get into why as we go through the layers.
With unit tests you want to test the smallest possible parts of the system in isolation
To achieve that you replace dependencies with test doubles
Dummies, stubs, spies, fakes, mocks
An example for those who are not developers: If you’re creating a unit test for unit A, and unit A depends on unit B. In your unit test, you replace unit B with a test double that you create and control. So your unit test for unit A is not relying on unit B at all.
Because we have replaced the dependencies that to unit B, our unit test has a very small scope. If it fails, we know exactly where the problem is - in unit A.
The small scope is also why unit tests have the best performance in the pyramid. When you replace all the dependencies, so you are not writing to hard drive, talking to a database or integrating with an external system, and so on
But even with a great unit test suite, you need another line of defense
We already know that our units work as we expect them to, in isolation, now we want to make sure that they work in combination
Which is why we don’t use test doubles, at least not for the integrations that we are interesting in testing
So in my previous example, we would test unit A and B together, without using a test double
You can have small component integration test like that, testing only two components, or you have bigger component integration tests, involving a bigger part of the system
So the scope of these tests vary, but in general the performance is still pretty good, even though unit tests are faster
Most services expose a lot of important functionality through APIs
As an example, you might have a REST API that is used by your front-end, used by your mobile app, and used by external systems integrating with your system
Now, you could consider API tests as being very large component integration tests, because the API relies on many different components on lower levels
But when you are testing APIs, you have a different focus
You are testing the system from more of a business and user perspective, thinking about how your API will be used
You want to guarantee that your API can be used in a certain way, and that it behaves as it should
Generally, API tests have a large scope, but they are still pretty fast, at least much faster than UI Tests
If you know that your UI relies on a solid API, you don’t need a huge number of UI tests, which brings us to the next level
Here I want to stress that I mean using the UI to test the application end-to-end, possibly with integrations to other systems. We are not about unit tests or component integration tests in JavaScript - that would be the bottom layer.
By definition, these end-to-end tests have the largest scope compared to the other layers, which is
Good because they can catch many different types of problems
But you need to spend a little bit of time to figure out where the error actually is
The large scope is also one reason for UI tests having the worst performance
As an example, let's say you are using Selenium WebDriver which allows you to test web applications by automating the browser. In that case you automatically start a browser, download resources, execute JavaScript, the browser makes multiple calls to your REST API, then the backend makes multiple calls to the database, and possibly to other systems
But to improve the performance of your UI tests you can
Prepare or clean up test data with other means that perform better
Run the tests in parallel, but doing that can be a big challenge in itself (tests, application, test infrastructure)
Brittle, flaky and a lot of work to maintain: Because the scope of the test is so large.
Brittle, meaning that they fail easily. Many things can impact the test, even though you didn’t intend to.
Flakey, meaning that they fail inconsistently, for instance because you have a bigger test infrastructure involved that could randomly crash
Rant: A flakey test is worse than having no test at all. If a test can fail when nothing is wrong, you will stop taking that test seriously. You will also waste a lot of time, running and re-running the test, investigating the results and so on. If you have flakey tests, address the root cause, and if you can't, delete the test, or switch to another technology.
To get back on track: To avoid having too many UI tests you want to identify the most important user workflows in your application, and create tests that emulate a user using going through those workflows
In general: Very valuable, but costly (for the same reason)
Not a part of the pyramid itself, because it is manual testing
but not regression testing
Combine experience with the system, and experience as a tester, with creativity to explore the system,
looking at the system from new angles, discovering new defects, and using what you learn to improve your automated tests
Often, exploratory testing is based on a mission, test charter or persona
<some examples>
Example: Let’s say you act as this impatient manager, not filling out all the input fields you should and double and triple clicking all the buttons, and you discover a problem when clicking the Save button repeatedly
Then you can investigate that, see if you have similar problems elsewhere and if you should have automated tests for catching these problems.
So we have covered all the layers, but I am sure someone has been thinking where is this and that form of testing?
The pyramid concerns itself with different technical levels of testing, while these examples here are either different motivations for testing, or certain properties you want to test for, and can be done on all or several of these technical levels
I wanted to illustrate that there is no silver bullet. You can’t use one single tool for test automation. If you do, you might get this →
This is what you don’t want.
Ice cream cone:
Mostly manual testing, started with Selenium WebDriver.
Too many UI tests, so you start with SoapUI to create API tests.
Completely opposite of what you want.
Hour-glass:
Developers write unit tests. Other people who write UI tests. No cooperation.
Something to be cognizant of, and to avoid.
Here are the test automation pyramids I promised you earlier.
As you can see, there are many ways of creating a test automation pyramid. Maybe your system doesn’t have a UI. Or maybe you have some good arguments for why you should have more API tests than component integration tests, which then of course would be fine. It’s not really possible to have one ideal test automation pyramid for everyone, which is why there is a special Testing Week task for this
Go to this page on Confluence.
Use my example, or create your own pyramid.
Make sure you have a common understanding within your team.
Try figure how your test effort is distributed today, and how you can improve
I want to talk about where the test automation pyramid is relevant in the development process
Example workflow from someone having an idea until the idea has been implemented and deployed to Production
QP: Where the team builds quality into the product
DP: Where you verify quality
QP: Quality increases, maximum quality when you push to mainline
DP: Confidence increases, maximum confidence when you deploy to Production
When the PO or BA is designing a user story they might draft some acceptance criteria - when A, B, and C are true, I consider the story to be fully implemented
Then, during refinement/grooming/breakout, different roles collaborate closely, sometimes referred to as the 3 amigos
Of course you can have more than 3 people, but all roles should be represented.
So you might have a PO/BA, the person with the vision for the user story, a developer, and a QA/tester
They make sure they have a common understanding of the story, they finalize the acceptance criteria, they finalize the acceptance tests, and they discuss risks and how to mitigate them
In that discussion they plan what should be tested, on which level, and the concept of the pyramid can be very useful
If you have done a good job in the refinement phase, developers should have a pretty clear picture of what is expected both in terms of functionality and automated tests
Automated tests are implemented as part of the user story
Assuming we have a good process for creating automated tests, where and when do we execute the tests?
Typically unit tests and component integration tests are executed by developers during implementation, and before they push their changes to the mainline
Then, you run the same tests on a CI server, in case someone forgot to run them locally, ‘
to make sure the tests pass on other machines as well,
and as the last line of defence before you deploy that new change to the first test environment
But then, which tests do you run in the different environments
In this example we have 3 test environments, and they have different objectives
In the internal test environment, all integrations are mocked, so our system is completely isolated. Because all the integrations are mocked, we are able to run all our API tests and our UI tests, verifying that this version works as expected and won’t cause problems when we promote it to the User Acceptance environment which is where we have real integrations
In the User Acceptance environment can run the API and UI tests again, to make sure the system still works with but now with real integrations
In the Production environment you probably only run a smoke test
Exploratory testing
In all environments
Developers can do exploratory testing during/after implementation in their development environment
QA/testers, or PO/BA can do it in any test environment. You can put a manual step anywhere in this automated delivery pipeline, so it is really up to you to see what fits you
Some people are even doing exploratory testing in Production, using feature toggles to turn on a new feature only for internal users who do exploratory testing in Production
A quick summary, to wrap up.
The test automation pyramid is a useful thinking tool and discussion reference
Make conscious decisions about what to test on which level, and also where you run which tests
Try to distribute your tests optimally
High quality
High velocity
Medium effort
Discuss with your team
Do the Testing Week task
That’s it for me, do you have any questions or comments?
