Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Test Pyramid vs ROI
Anton Semenchenko
Anton Semenchenko
Creator of communities www.COMAQA.BY
and www.CoreHard.by, founder of company
www.DPI.Solutions, «tricky»...
Agenda
1. Test Pyramid: general information
2. ROI: general information
3. Two questions
4. Test Pyramid: detailed informa...
Test Pyramid: definition
• «An effective test automation strategy calls for automating tests at three different
levels, as...
Test Pyramid with percentage #1
1. UI - 5%
2. Acceptance – 5%
3. Integration – 10%
4. Unit - 80%
Test Pyramid with percentage #2
1. UI – 1%
2. End to End Flow – 4%
3. Workflow API – 6%
4. Integration – 9%
5. Domain Logi...
ROI: general information
Answer three simple questions:
1. How many hours do we spend currently?
2. How many hours will we...
ROI: general information
What should be estimated?
Implementation effort:
1. Effort to create/adopt automation framework
2...
ROI: calculation
How many hours do we spend currently?
Manual effort per release = Effort for TC execution * number of run...
Two questions (or 4 )
• Percent ??? Time  Money or amount of Test Cases?
• Pyramid  Triangle angels ???
• How to “calcula...
Test Pyramid: definition
• «An effective test automation strategy calls for automating tests at three different
levels, as...
Test Pyramid: Unit Testing
• «Unit testing should be the foundation of a solid test automation strategy and as
such repres...
Test Pyramid: Service
• «Although I refer to the middle layer of the test automation pyramid as the service
layer, I am no...
Test Pyramid: Service - Calculator example
• «What’s needed next is a simple program that can read the rows of this spread...
Test Pyramid: UI
• «Automated user interface testing is placed at the top of the test automation
pyramid because we want t...
The remaining role of user Interface Tests
• “But don’t we need to do some user interface testing? Absolutely, but far les...
Automate Within the Sprint
Find a balance 
Sources of information: 2 poles
• “Succeeding with Agile: Software Development Using Scrum” by Mike Cohn
• “Хабрахабр”
• И...
Two the same questions
• Percent ??? Time  Money or amount of Test Cases?
• Pyramid  Triangle angels ???
• How to “calcula...
Solution
• Adapt ROI Calculator for you project  context
• Calculate ROI to “build” 2 Test pyramids
ROI Factors?
ROI factors ???
ROI factors
Name Definition Impact
“run” per Release How many times TC should be “run” per Release +
“envs” How many “envs...
Haven't we forgotten anything?
+1 more factor
Business Needs
Needs 
Examples
• Facebook, Bamboo
• Head hunter
• Microsoft
• Military related software (Data Protection solution for US Army)
Haven't we forgotten anything?
+1 more factor
Business Risks
Risks 
Examples
• A huge set of projects for Siemens
• Data Protection Systems: granular Back-up  Restore plug-ins
• Back-end “or...
Automation tests types #1
1. UI - 5%
2. Acceptance – 5%
3. Integration – 10%
4. Unit - 80%
ROI factors
Name Definition Impact
“run” per Release How many times TC should be “run” per Release +
“envs” How many “envs...
Testing types “comparative analysis”
Factor Unit Integration Acceptance UI
“run” per Release + +- - -
“envs” - +- + +
“upd...
Automation tests types #2
1. UI – 1%
2. End to End Flow – 4%
3. Workflow API – 6%
4. Integration – 9%
5. Domain Logic Acce...
Sources of information: refresh
• “Succeeding with Agile: Software Development Using Scrum” by Mike Cohn
• “Хабрахабр”
• И...
ROI Calculator: refresh
• ISKRA Антон Семенченко «Метрики в тестировании»
• SQA Days 16. Доклад Антона Семенченко о продаж...
CONTACT ME
semenchenko@dpi.solutions
dpi.semenchenko
https://www.linkedin.com/in/anton-semenchenko-612a926b
https://www.fa...
www.COMAQA.BY
Community’s audience
Testing specialists (manual and automated)
Automation tools developers
Managers and sal...
www.COMAQA.BY
info@comaqa.by
https://www.facebook.com/comaqa.by/
http://vk.com/comaqaby
+375 33 33 46 120
+375 44 74 00 385
www.CoreHard.by
Community’s audience
«Harsh» С++ developers & co, IoT, BigData, High Load, Parallel Computing
Automation t...
www.CoreHard.by
info@corehard.by
https://www.facebook.com/corehard.by/
+375 33 33 46 120
+375 44 74 00 385
Upcoming SlideShare
Loading in …5
×

Пирамида Тестирования через призму ROI калькулятора и прочая геометрия

1,335 views

Published on

Доклад Антона Семенченко на конференции SQA Days-20. 24-26 ноября 2016. Минск
www.sqadays.com

Published in: Education
  • Be the first to comment

  • Be the first to like this

Пирамида Тестирования через призму ROI калькулятора и прочая геометрия

  1. 1. Test Pyramid vs ROI Anton Semenchenko
  2. 2. Anton Semenchenko Creator of communities www.COMAQA.BY and www.CoreHard.by, founder of company www.DPI.Solutions, «tricky» manager at EPAM Systems. Almost 15 years of experience in IT, main specialization: Automation, С++ and lower development, management, sales. Anton Semenchenko EPAM Systems, Software Testing Manager
  3. 3. Agenda 1. Test Pyramid: general information 2. ROI: general information 3. Two questions 4. Test Pyramid: detailed information 5. Sources of information 6. Two the same questions 7. Solution: high level information
  4. 4. Test Pyramid: definition • «An effective test automation strategy calls for automating tests at three different levels, as shown in Figure, which depicts the test automation pyramid»
  5. 5. Test Pyramid with percentage #1 1. UI - 5% 2. Acceptance – 5% 3. Integration – 10% 4. Unit - 80%
  6. 6. Test Pyramid with percentage #2 1. UI – 1% 2. End to End Flow – 4% 3. Workflow API – 6% 4. Integration – 9% 5. Domain Logic Acceptance – 10% 6. Unit – 70%
  7. 7. ROI: general information Answer three simple questions: 1. How many hours do we spend currently? 2. How many hours will we spend after automation? 3. When automation will bring value? What information should be gathered? 1. Number of test cases (for estimations) 2. Number of executions per release 3. Effort for manual execution of the test cases
  8. 8. ROI: general information What should be estimated? Implementation effort: 1. Effort to create/adopt automation framework 2. Effort to automate test cases Maintenance effort: 1. Execution of automated scripts 2. Results analysis 3. Bug reporting 4. Fixing “broken” scripts
  9. 9. ROI: calculation How many hours do we spend currently? Manual effort per release = Effort for TC execution * number of runs per release How many hours will we spend after automation? Effort after automation per release = Maintenance efforts + Effort for TC execution (non-automatable) * number of runs per release When automation effort will bring value? Number of Releases to invest in automation = Automation implementation effort / (Manual effort per release - Effort after automation per release)
  10. 10. Two questions (or 4 ) • Percent ??? Time Money or amount of Test Cases? • Pyramid Triangle angels ??? • How to “calculate” 2 pyramids for exact project context? • How to “calculate” exact percentage for exact project context?
  11. 11. Test Pyramid: definition • «An effective test automation strategy calls for automating tests at three different levels, as shown in Figure, which depicts the test automation pyramid»
  12. 12. Test Pyramid: Unit Testing • «Unit testing should be the foundation of a solid test automation strategy and as such represents the largest part of the pyramid. Automated unit tests are wonderful because they give specific data to a programmer—there is a bug and it’s on line 47.» • «Unit tests are usually written in the same language as the system, programmers are often most comfortable writing them.»
  13. 13. Test Pyramid: Service • «Although I refer to the middle layer of the test automation pyramid as the service layer, I am not restricting us to using only a service-oriented architecture. All applications are made up of various services. In the way I’m using it, a service is something the application does in response to some input or set of inputs.» • «Service-level testing is about testing the services of an application separately from its user interface.»
  14. 14. Test Pyramid: Service - Calculator example • «What’s needed next is a simple program that can read the rows of this spreadsheet, pass the data columns to the right service within your application, and verify that the right results occur. Despite this simplistic example where the result is simple calculation, the result could be anything—data updated in the database, an e-mail sent to a specific recipient, money transferred between bank accounts, and so on.»
  15. 15. Test Pyramid: UI • «Automated user interface testing is placed at the top of the test automation pyramid because we want to do as little of it as possible.» • «User interface tests often have the following negative attributes: • Brittle. A small change in the user interface can break many tests. When this is repeated many times over the course of a project, teams simply give up and stop correcting tests every time the user interface changes. • Expensive to write. A quick capture-and-playback approach to recording user interface tests can work, but tests recorded this way are usually the most brittle. Writing a good user interface test that will remain useful and valid takes time. • Time consuming. Tests run through the user interface often take a long time to run. I’ve seen numerous teams with impressive suites of automated user interface tests that take so long to run they cannot be run every night, much less multiple times per day.»
  16. 16. The remaining role of user Interface Tests • “But don’t we need to do some user interface testing? Absolutely, but far less of it than any other test type. • Instead, we run the majority of tests (such as boundary tests) through the service layer, invoking the methods (services) directly to confirm that the functionality is working properly. At the user interface level what’s left is testing to confirm that the services are hooked up to the right buttons and that the values are displaying properly in the result field. To do this we need a much smaller set of tests to run through the user interface layer. • Where many organizations have gone wrong in their test automation efforts over the years has been in ignoring this whole middle layer of service testing. Although automated unit testing is wonderful, it can cover only so much of an application’s testing needs. Without service-level testing to fill the gap between unit and user interface testing, all other testing ends up being performed through the user interface, resulting in tests that are expensive to run, expensive to write, and brittle”
  17. 17. Automate Within the Sprint Find a balance 
  18. 18. Sources of information: 2 poles • “Succeeding with Agile: Software Development Using Scrum” by Mike Cohn • “Хабрахабр” • Игорь Хрол: “Автоматизация тестирования: отбрасываем лишнее и проверяем суть” (#3) • Игорь Хрол: “Автоматизация тестирования: отбрасываем лишнее и проверяем суть, SQA Days-15” (#2) • “Хабрахабр” • Игорь Хрол: “Автоматизация тестирования: отбрасываем лишнее и проверяем суть” (#1)
  19. 19. Two the same questions • Percent ??? Time Money or amount of Test Cases? • Pyramid Triangle angels ??? • How to “calculate” 2 pyramids for exact project context? • How to “calculate” exact percentage for exact project context?
  20. 20. Solution • Adapt ROI Calculator for you project context • Calculate ROI to “build” 2 Test pyramids
  21. 21. ROI Factors? ROI factors ???
  22. 22. ROI factors Name Definition Impact “run” per Release How many times TC should be “run” per Release + “envs” How many “envs” should be covered + “updated” per Release How many times TC should be “updated” per Release - “write” t$ to “write” TC - “develop” t$ to “develop” TC - “update” t$ to “update” TC - “prepare” env t$ to “prepare” env for TC run - “run” t$ to “run” TC - “analyze” t$ to “analyze” Test Result - “locate” t$ to “locate” defect - “fix” t$ to “fix” defect -
  23. 23. Haven't we forgotten anything? +1 more factor
  24. 24. Business Needs Needs 
  25. 25. Examples • Facebook, Bamboo • Head hunter • Microsoft • Military related software (Data Protection solution for US Army)
  26. 26. Haven't we forgotten anything? +1 more factor
  27. 27. Business Risks Risks 
  28. 28. Examples • A huge set of projects for Siemens • Data Protection Systems: granular Back-up Restore plug-ins • Back-end “oriented” software
  29. 29. Automation tests types #1 1. UI - 5% 2. Acceptance – 5% 3. Integration – 10% 4. Unit - 80%
  30. 30. ROI factors Name Definition Impact “run” per Release How many times TC should be “run” per Release + “envs” How many “envs” should be covered + “updated” per Release How many times TC should be “updated” per Release - “write” t$ to “write” TC - “develop” t$ to “develop” TC - “update” t$ to “update” TC - “prepare” env t$ to “prepare” env for TC run - “run” t$ to “run” TC - “analyze” t$ to “analyze” Test Result - “locate” t$ to “locate” defect - “fix” t$ to “fix” defect -
  31. 31. Testing types “comparative analysis” Factor Unit Integration Acceptance UI “run” per Release + +- - - “envs” - +- + + “updated” per Release - +- -+ - “write” + +- - - “develop” + +- - - “update” + +- - - “prepare” env + +- - - “run” + +- +- - “analyze” + +- -+ - “locate” + +- - - “fix” + +- - -
  32. 32. Automation tests types #2 1. UI – 1% 2. End to End Flow – 4% 3. Workflow API – 6% 4. Integration – 9% 5. Domain Logic Acceptance – 10% 6. Unit – 70%
  33. 33. Sources of information: refresh • “Succeeding with Agile: Software Development Using Scrum” by Mike Cohn • “Хабрахабр” • Игорь Хрол: “Автоматизация тестирования: отбрасываем лишнее и проверяем суть” (#3) • Игорь Хрол: “Автоматизация тестирования: отбрасываем лишнее и проверяем суть, SQA Days-15” (#2) • “Хабрахабр” • Игорь Хрол: “Автоматизация тестирования: отбрасываем лишнее и проверяем суть” (#1)
  34. 34. ROI Calculator: refresh • ISKRA Антон Семенченко «Метрики в тестировании» • SQA Days 16. Доклад Антона Семенченко о продаже автоматизированного тестирования • SQA Days 15 Антон Семенченко. Как эффективно организовать Автоматизацию, если у вас недостаточно времени, ресурсов и денег • Solit 2014. Антон Семенченко. "Как эффективно продавать Automation Service“ • Solit 2014. Антон Семенченко. "Как эффективно организовать Автоматизацию” • Антон Семенченко: «Эволюция тестового окружения» • Антон Семенченко: «Автоматизированное тестирование переживает взрывное развитие» • Антон Семенченко: «Как автоматизатору не оказаться за бортом эволюции» • TMPA-2015. Антон Семенченко. Automated testing: yesterday, today, tomorrow – the vectors of development • SECR 2016. Антон Семенченко. Метрики в тестировании – практика использования в СНГ
  35. 35. CONTACT ME semenchenko@dpi.solutions dpi.semenchenko https://www.linkedin.com/in/anton-semenchenko-612a926b https://www.facebook.com/semenchenko.anton.v https://twitter.com/comaqa
  36. 36. www.COMAQA.BY Community’s audience Testing specialists (manual and automated) Automation tools developers Managers and sales specialists in IT IT-specialists, thinking about migrating to automation Students looking for perspective profession. Community goals Create unified space for effective communication for all IT-specialists in the context of automated testing. Your profit Ability to listen to reports from leading IT-specialists and share your experience. Take part in «promo»-versions of top IT-conferences in CIS for free. Meet regularly, at different forums, community «offices», social networks and messengers.
  37. 37. www.COMAQA.BY info@comaqa.by https://www.facebook.com/comaqa.by/ http://vk.com/comaqaby +375 33 33 46 120 +375 44 74 00 385
  38. 38. www.CoreHard.by Community’s audience «Harsh» С++ developers & co, IoT, BigData, High Load, Parallel Computing Automation tools developers Managers and sales specialists in IT Students looking for perspective profession. Community goals Create unified space for effective communication for all IT-specialists in the context of «harsh» development. Your profit Ability to listen to reports from leading IT-specialists and share your experience. Take part in «promo»-versions of top IT-conferences in CIS for free. Meet regularly, at different forums, community «offices», social networks and messengers.
  39. 39. www.CoreHard.by info@corehard.by https://www.facebook.com/corehard.by/ +375 33 33 46 120 +375 44 74 00 385

×