1. Test Pyramid vs ROI
Anton Semenchenko

2. Anton Semenchenko
Creator of communities www.COMAQA.BY
and www.CoreHard.by, founder of company
www.DPI.Solutions, «tricky» manager at
EPAM Systems. Almost 15 years of
experience in IT, main specialization:
Automation, С++ and lower development,
management, sales.
Anton Semenchenko
EPAM Systems, Software Testing Manager

3. Agenda
1. Test Pyramid: general information
2. ROI: general information
3. Two questions
4. Test Pyramid: detailed information
5. Sources of information
6. Two the same questions
7. Solution: high level information

4. Test Pyramid: definition
• «An effective test automation strategy calls for automating tests at three different
levels, as shown in Figure, which depicts the test automation pyramid»

5. Test Pyramid with percentage #1
1. UI - 5%
2. Acceptance – 5%
3. Integration – 10%
4. Unit - 80%

6. Test Pyramid with percentage #2
1. UI – 1%
2. End to End Flow – 4%
3. Workflow API – 6%
4. Integration – 9%
5. Domain Logic Acceptance – 10%
6. Unit – 70%

7. ROI: general information
Answer three simple questions:
1. How many hours do we spend currently?
2. How many hours will we spend after automation?
3. When automation will bring value?
What information should be gathered?
1. Number of test cases (for estimations)
2. Number of executions per release
3. Effort for manual execution of the test cases

8. ROI: general information
What should be estimated?
Implementation effort:
1. Effort to create/adopt automation framework
2. Effort to automate test cases
Maintenance effort:
1. Execution of automated scripts
2. Results analysis
3. Bug reporting
4. Fixing “broken” scripts

9. ROI: calculation
How many hours do we spend currently?
Manual effort per release = Effort for TC execution * number of runs per release
How many hours will we spend after automation?
Effort after automation per release = Maintenance efforts +
Effort for TC execution (non-automatable) * number of runs per release
When automation effort will bring value?
Number of Releases to invest in automation = Automation implementation effort
/ (Manual effort per release - Effort after automation per release)

10. Two questions (or 4 )
• Percent ??? Time Money or amount of Test Cases?
• Pyramid Triangle angels ???
• How to “calculate” 2 pyramids for exact project context?
• How to “calculate” exact percentage for exact project context?

11. Test Pyramid: definition
• «An effective test automation strategy calls for automating tests at three different
levels, as shown in Figure, which depicts the test automation pyramid»

12. Test Pyramid: Unit Testing
• «Unit testing should be the foundation of a solid test automation strategy and as
such represents the largest part of the pyramid. Automated unit tests are wonderful
because they give specific data to a programmer—there is a bug and it’s on line 47.»
• «Unit tests are usually written in the same language as the system, programmers are
often most comfortable writing them.»

13. Test Pyramid: Service
• «Although I refer to the middle layer of the test automation pyramid as the service
layer, I am not restricting us to using only a service-oriented architecture. All
applications are made up of various services. In the way I’m using it, a service is
something the application does in response to some input or set of inputs.»
• «Service-level testing is about testing the services of an application separately from its
user interface.»

14. Test Pyramid: Service - Calculator example
• «What’s needed next is a simple program that can read the rows of this spreadsheet,
pass the data columns to the right service within your application, and verify that the
right results occur. Despite this simplistic example where the result is simple
calculation, the result could be anything—data updated in the database, an e-mail
sent to a specific recipient, money transferred between bank accounts, and so on.»

15. Test Pyramid: UI
• «Automated user interface testing is placed at the top of the test automation
pyramid because we want to do as little of it as possible.»
• «User interface tests often have the following negative attributes:
• Brittle. A small change in the user interface can break many tests. When this is repeated many
times over the course of a project, teams simply give up and stop correcting tests every time
the user interface changes.
• Expensive to write. A quick capture-and-playback approach to recording user interface tests
can work, but tests recorded this way are usually the most brittle. Writing a good user interface
test that will remain useful and valid takes time.
• Time consuming. Tests run through the user interface often take a long time to run. I’ve seen
numerous teams with impressive suites of automated user interface tests that take so long to
run they cannot be run every night, much less multiple times per day.»

16. The remaining role of user Interface Tests
• “But don’t we need to do some user interface testing? Absolutely, but far less of it than any
other test type.
• Instead, we run the majority of tests (such as boundary tests) through the service layer,
invoking the methods (services) directly to confirm that the functionality is working properly.
At the user interface level what’s left is testing to confirm that the services are hooked up to
the right buttons and that the values are displaying properly in the result field. To do this we
need a much smaller set of tests to run through the user interface layer.
• Where many organizations have gone wrong in their test automation efforts over the years
has been in ignoring this whole middle layer of service testing. Although automated unit
testing is wonderful, it can cover only so much of an application’s testing needs. Without
service-level testing to fill the gap between unit and user interface testing, all other testing
ends up being performed through the user interface, resulting in tests that are expensive to
run, expensive to write, and brittle”

17. Automate Within the Sprint
Find a balance 

18. Sources of information: 2 poles
• “Succeeding with Agile: Software Development Using Scrum” by Mike Cohn
• “Хабрахабр”
• Игорь Хрол: “Автоматизация тестирования: отбрасываем лишнее и проверяем
суть” (#3)
• Игорь Хрол: “Автоматизация тестирования: отбрасываем лишнее и проверяем
суть, SQA Days-15” (#2)
• “Хабрахабр”
• Игорь Хрол: “Автоматизация тестирования: отбрасываем лишнее и проверяем
суть” (#1)

19. Two the same questions
• Percent ??? Time Money or amount of Test Cases?
• Pyramid Triangle angels ???
• How to “calculate” 2 pyramids for exact project context?
• How to “calculate” exact percentage for exact project context?

20. Solution
• Adapt ROI Calculator for you project context
• Calculate ROI to “build” 2 Test pyramids

21. ROI Factors?
ROI factors ???

22. ROI factors
Name Definition Impact
“run” per Release How many times TC should be “run” per Release +
“envs” How many “envs” should be covered +
“updated” per Release How many times TC should be “updated” per Release -
“write” t$ to “write” TC -
“develop” t$ to “develop” TC -
“update” t$ to “update” TC -
“prepare” env t$ to “prepare” env for TC run -
“run” t$ to “run” TC -
“analyze” t$ to “analyze” Test Result -
“locate” t$ to “locate” defect -
“fix” t$ to “fix” defect -

23. Haven't we forgotten anything?
+1 more factor

24. Business Needs
Needs 

25. Examples
• Facebook, Bamboo
• Head hunter
• Microsoft
• Military related software (Data Protection solution for US Army)

26. Haven't we forgotten anything?
+1 more factor

27. Business Risks
Risks 

28. Examples
• A huge set of projects for Siemens
• Data Protection Systems: granular Back-up Restore plug-ins
• Back-end “oriented” software

29. Automation tests types #1
1. UI - 5%
2. Acceptance – 5%
3. Integration – 10%
4. Unit - 80%

30. ROI factors
Name Definition Impact
“run” per Release How many times TC should be “run” per Release +
“envs” How many “envs” should be covered +
“updated” per Release How many times TC should be “updated” per Release -
“write” t$ to “write” TC -
“develop” t$ to “develop” TC -
“update” t$ to “update” TC -
“prepare” env t$ to “prepare” env for TC run -
“run” t$ to “run” TC -
“analyze” t$ to “analyze” Test Result -
“locate” t$ to “locate” defect -
“fix” t$ to “fix” defect -

31. Testing types “comparative analysis”
Factor Unit Integration Acceptance UI
“run” per Release + +- - -
“envs” - +- + +
“updated” per
Release
- +- -+ -
“write” + +- - -
“develop” + +- - -
“update” + +- - -
“prepare” env + +- - -
“run” + +- +- -
“analyze” + +- -+ -
“locate” + +- - -
“fix” + +- - -

32. Automation tests types #2
1. UI – 1%
2. End to End Flow – 4%
3. Workflow API – 6%
4. Integration – 9%
5. Domain Logic Acceptance – 10%
6. Unit – 70%

33. Sources of information: refresh
• “Succeeding with Agile: Software Development Using Scrum” by Mike Cohn
• “Хабрахабр”
• Игорь Хрол: “Автоматизация тестирования: отбрасываем лишнее и проверяем
суть” (#3)
• Игорь Хрол: “Автоматизация тестирования: отбрасываем лишнее и проверяем
суть, SQA Days-15” (#2)
• “Хабрахабр”
• Игорь Хрол: “Автоматизация тестирования: отбрасываем лишнее и проверяем
суть” (#1)

34. ROI Calculator: refresh
• ISKRA Антон Семенченко «Метрики в тестировании»
• SQA Days 16. Доклад Антона Семенченко о продаже автоматизированного тестирования
• SQA Days 15 Антон Семенченко. Как эффективно организовать Автоматизацию, если у вас недостаточно
времени, ресурсов и денег
• Solit 2014. Антон Семенченко. "Как эффективно продавать Automation Service“
• Solit 2014. Антон Семенченко. "Как эффективно организовать Автоматизацию”
• Антон Семенченко: «Эволюция тестового окружения»
• Антон Семенченко: «Автоматизированное тестирование переживает взрывное развитие»
• Антон Семенченко: «Как автоматизатору не оказаться за бортом эволюции»
• TMPA-2015. Антон Семенченко. Automated testing: yesterday, today, tomorrow – the vectors of
development
• SECR 2016. Антон Семенченко. Метрики в тестировании – практика использования в СНГ

35. CONTACT ME
semenchenko@dpi.solutions
dpi.semenchenko
https://www.linkedin.com/in/anton-semenchenko-612a926b
https://www.facebook.com/semenchenko.anton.v
https://twitter.com/comaqa

36. www.COMAQA.BY
Community’s audience
Testing specialists (manual and automated)
Automation tools developers
Managers and sales specialists in IT
IT-specialists, thinking about migrating to automation
Students looking for perspective profession.
Community goals
Create unified space for effective communication for all IT-specialists in the context of automated testing.
Your profit
Ability to listen to reports from leading IT-specialists and share your experience.
Take part in «promo»-versions of top IT-conferences in CIS for free.
Meet regularly, at different forums, community «offices», social networks and messengers.

37. www.COMAQA.BY
info@comaqa.by
https://www.facebook.com/comaqa.by/
http://vk.com/comaqaby
+375 33 33 46 120
+375 44 74 00 385

38. www.CoreHard.by
Community’s audience
«Harsh» С++ developers & co, IoT, BigData, High Load, Parallel Computing
Automation tools developers
Managers and sales specialists in IT
Students looking for perspective profession.
Community goals
Create unified space for effective communication for all IT-specialists in the context of «harsh»
development.
Your profit
Ability to listen to reports from leading IT-specialists and share your experience.
Take part in «promo»-versions of top IT-conferences in CIS for free.
Meet regularly, at different forums, community «offices», social networks and messengers.

39. www.CoreHard.by
info@corehard.by
https://www.facebook.com/corehard.by/
+375 33 33 46 120
+375 44 74 00 385