Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
TEST AUTOMATION.
А ЧТО НА ВЫХОДЕ?
QA FEST
PAVLO BASIUK, ITERA
What is test automation outcome?
System Under
Test
Tests inputs Tests results
What is test automation outcome?
System Under
Test
Tests inputs Tests results
Precondition data
Environmental status
Progr...
Point of view
Automation engineers
QA team
Developers
Customer, QA Manager
Test Report should be:
Requirements for report
• Structured
– Test-suites, test-cases, test-steps
– Filters for results (to see only pass/fail/ot...
Analyzing Results
• test environment
• application changes
• test errors
False failure
• test defect
• missed error
False ...
Examples
Real-life reports
Hand-made Html Report Overview
Hand-made Html Report Details
ReportNG
ReportNG sample overview
ReportNG sample details
ReportNG real-case overview
ReportNG real-case details
Thucydides
Thucydides overview
04.11.2014 / 18
Thucydides details
04.11.2014 / 20
Ranorex
04.11.2014
Allure
Allure overview
Failures are groupped by assertion
Test suites
Test case details
Even more details on test steps
Nice charts
Possible tweaks
Test statuses
Input data
Special modes
Resource utilization
Known bugs during test run
Semi-automatic defe...
Possible improvements
Review (again) who is target audience
What are main goals for report
Were do you spend most time
References
1. AT.info http://automated-testing.info
2. ReportNG http://reportng.uncommons.org
3. QA Automation Notes http:...
Thank you!
Any questions?
QA Fest 2014. Павел Басюк. Test automation: а что на выходе?
QA Fest 2014. Павел Басюк. Test automation: а что на выходе?
Upcoming SlideShare
Loading in …5
×

QA Fest 2014. Павел Басюк. Test automation: а что на выходе?

899 views

Published on

Атоматизация бывает разной: большой и маленькой, скучной и веселой. Но каждый раз когда запускаються автотесты - есть какие-то результаты. Это могут быть красивые или не очень html отчеты, либо банальный текстовый файл с большым логом. Именно о всём разнообразии отчётов мы и пообщаемся: кто и когда их создает, а кто и зачем их читает. Рассмотрим несколько примеров реальных отчетов, и обсудим как их можно улучшить.

Published in: Technology
  • Be the first to comment

QA Fest 2014. Павел Басюк. Test automation: а что на выходе?

  1. 1. TEST AUTOMATION. А ЧТО НА ВЫХОДЕ? QA FEST PAVLO BASIUK, ITERA
  2. 2. What is test automation outcome? System Under Test Tests inputs Tests results
  3. 3. What is test automation outcome? System Under Test Tests inputs Tests results Precondition data Environmental status Program start state Environmental changes Program end state Post condition data Program end state Post condition data Environmental changes
  4. 4. Point of view Automation engineers QA team Developers Customer, QA Manager
  5. 5. Test Report should be:
  6. 6. Requirements for report • Structured – Test-suites, test-cases, test-steps – Filters for results (to see only pass/fail/other) • Summarized – High-level overview of results, so it should take less time to understand how good results are – Summary per feature/tag/component • Rich – Screenshots, additional info (e.g. attachments, system configuration) • Detailed – With all input data used – With all actions done and all expected/actual results – Execution time (for test-suites, test-cases) • Easy to create – We don’t want to spend extra time to prepare any reports
  7. 7. Analyzing Results • test environment • application changes • test errors False failure • test defect • missed error False success • same error reported • dependent tests failed Duplicate failures
  8. 8. Examples Real-life reports
  9. 9. Hand-made Html Report Overview
  10. 10. Hand-made Html Report Details
  11. 11. ReportNG
  12. 12. ReportNG sample overview
  13. 13. ReportNG sample details
  14. 14. ReportNG real-case overview
  15. 15. ReportNG real-case details
  16. 16. Thucydides
  17. 17. Thucydides overview 04.11.2014 / 18
  18. 18. Thucydides details 04.11.2014 / 20
  19. 19. Ranorex
  20. 20. 04.11.2014
  21. 21. Allure
  22. 22. Allure overview
  23. 23. Failures are groupped by assertion
  24. 24. Test suites
  25. 25. Test case details
  26. 26. Even more details on test steps
  27. 27. Nice charts
  28. 28. Possible tweaks Test statuses Input data Special modes Resource utilization Known bugs during test run Semi-automatic defect report
  29. 29. Possible improvements Review (again) who is target audience What are main goals for report Were do you spend most time
  30. 30. References 1. AT.info http://automated-testing.info 2. ReportNG http://reportng.uncommons.org 3. QA Automation Notes http://qa-automation-notes.blogspot.com 4. Thucydides http://thucydides.info 5. Ranorex http://ranorex.com 6. Allure http://allure.qatools.ru
  31. 31. Thank you! Any questions?

×