Agile Testing Days 2018: Our journey to a software Testing Ice-Cream Cone Approach Presenters: Karla Silva & Larissa Soares

1. Our Journey to a Software
Testing Ice-Cream Cone
Approach
Agile Testing Days, 2018
Karla Silva & Larissa Soares
Our slides are available for you at: bit.ly/atd-2018-ice-cream

2. Karla Silva
Software Test Engineer
Larissa Soares
Software Test Engineer

3. CESAR
www.cesar.org.br
- Private innovation institute;
- Clients in several domains;
- Over 500 employees;
- Based in the city of Recife;
- Branches in the cities of Manaus,
Sorocaba and Curitiba;

4. 1
Motivation

5. “The test pyramid is a way of thinking about different kinds of automated tests
should be used to create a balanced portfolio. "
Martin Fowler
Author
Alister Scott
Quality Engineer
“ 100,000 end-to-end selenium tests and success in the same sentence? WTF?
Sounds like a nightmare to me!"

6. What about your test pyramid?
(Or other geometric shape)

7. Level A
Level B
Level C
Draft

9. Background
2

10. Tech stack
- Behat (BDD framework for PHP );
- Test scenarios written in natural
language (Gherkin);
- Development of our UI test
automation framework;
UX designer
UI designer
DEV
QA
4 Projects
International
client

11. Development process (the plan)
UX
UX designer creates a new
feature file in the Product
backlog
from 2º sprint
day forward
Dev team considers the
feature file during coding
and testing
Refinement
UX designer adds the
acceptance criteria into the
feature file
Planning
Team plans the sprint
according to the features
files selected to the Sprint
backlog
2º sprint day
QA team brainstorms and writes test
cases using the test automation
framework.
Feature is delivered
QA team uses the feature file
as a script to manual testing
Automation sprint
QA Automation team
implements the test cases

12. Feature: Log in to the application
As a user
I want to be able to log in to the application
so that I can successfully use my account
Scenario: Log in to the application
Given The user accesses the “login” page
When The user fills the “login field” with “martin.scott”
And The user fills the “password field” with “123@abc”
Then The user should be on “home” page
Scenario: Log out of the application
…
Login.feature

13. Scope
The user does the following on the “speakers" table
The user does the following on the “confirm changes modal" area
The user does the following on the page
Interaction
The user clicks on the “save button”
The user clicks and fills the “user name"
field with the “martin.scott” value
The user hovers over the “info icon”
Assert
The user is on the “home” page
The “welcome banner” text is “visible"
The “delete account button“ is “enabled"
The test automation framework

14. Development process (the reality)
UX
UX designer creates a new
feature file in the Product
backlog
from 2º sprint
day forward
Dev team considers the
feature file during coding
and testing
Planning
Team plans the sprint
according to the features
files selected to the Sprint
backlog
2º sprint day
QA team brainstorms and writes test
cases using the test automation
framework.
Feature is delivered
QA team uses the feature file
as a script to manual testing
Automation sprint
QA Automation team
implements the test cases
Refinement
UX designer adds the
acceptance criteria into the
feature file

15. What about our test strategy
shape?

16. Manual
Scripted Testing. Done by QA team.
Integration & Unit automation
Scripts were not planned. Done by DEV team.
UI automation
End-to-end testing. Done by QA team.
Ice-cream Cone

17. Lessons Learned
3

18. Easy identification of bugs that affect the end user
A lot of flows simulating a real user scenario;
Test scenarios as software documentation
Test scenarios were automated, but using a
natural language (Gherkin);
1
Went Well
2
3
4
Project rotation
Four projects were using and contributing to
the codebase;
Reduce time in regression test
In some projects, a full regression could take 5
days. With the E2E suite, those tests took less
than 24 hours to execute.

19. Went Wrong
Sub-teams not integrated
In the first two sprints, the UX team (including
business) decided to not follow the process.
Test strategy designed exclusively by QA
QA was in charge of the brainstorming and the test
design;
1
2

20. Went Wrong
3
4
Automated suite takes too long to execute
On two projects, the execution could take more
than 12 hours;
Analysis of output execution was too expensive
Due to the amount of scenarios, the execution
output was too long, and it will require so much
effort to analyze, even if using some tools;
High maintenance costs
E2E are know as “flaky tests”. Changes on the page
components were part of our routine, and it made
our tests fail.
5

21. 3
2
1
Understand the needs of each project
Accept that our projects are not equal, and we
should understand theirs needs, so they might have
different pyramids;
Test strategy definition should be done by the
team
Other sub-teams should be included in the
brainstorming and/or tests classification into levels
sessions;
Consider other levels of testing
Starting some study and proof of concepts (PoC)
about acceptance, components, and API tests.
Action Items

22. 6
5
4
Add different levels of tests to the CI
Components, acceptance, and API tests were
added to the CI. Other levels (performance, E2E)
are trigged automatically each deploy;
Reduce the amount of E2E tests
E2E should be reduced to focus on sanity scenarios;
Analyze unit tests
QAs should help reviewing the unit tests.
Action Items

23. 4
Present and Future

24. Have we reached the pyramid pattern?
Manual
Exploratory testing done by QA team.
UI
Front-end acceptance and end-to-end done by
QA/DEV.
Integration
API and integration done by QA/DEV.
Unit
Safety net. Done by DEV team with QA help.

25. Contact info
lsls@cesar.org.br
twitter.com/lari_lsls
linkedin.com/in/larissasls
karla.silva@cesar.org.br
twitter.com/karlambsilva
linkedin.com/in/karlambsilva
Larissa SoaresKarla Silva

26. THANKS FOR
YOUR ATTENTION!
Feedback welcome! Evaluate my
session and get the chance to win a free
ticket for the Agile Testing Days 2019.
Go to agiletestingdays.com/session-ratings
and give your rating until 30th of November 2018!
Our slides are available for you at: bit.ly/atd-2018-ice-cream

27. Get a sticker!