Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

ReportPortal.io: пример CI/CD c метриками на основе продукта с Открытым Кодом

525 views

Published on

Доклад Дмитрия Гуменюка на конференции SQA Days-20. 24-26 ноября 2016. Минск
www.sqadays.com

Published in: Education
  • Be the first to comment

  • Be the first to like this

ReportPortal.io: пример CI/CD c метриками на основе продукта с Открытым Кодом

  1. 1. Software quality assurance days 20 Международная конференция по вопросам качества ПО sqadays.com Минск. 24–26 ноября 2016 Дмитрий Гуменюк EPAM Systems. Минск, Беларусь ReportPortal.io: пример CI/CD c метриками на основе продукта с Открытым Кодом
  2. 2. http://ReportPortal.io пример CI/CD с метриками, на основе продукта с Открытым Кодом Проект Менеджер 10 лет с компанией EPAM Systems. Разработчик в прошлом Возглавляю разработку инициатив внутри Центра Компетенций Тестирования. Report Portal – Product Owner. DZMITRY HUMIANIUK ДМИТРИЙ ГУМЕНЮК
  3. 3. http://ReportPortal.io пример CI/CD с метриками, на основе продукта с Открытым Кодом Проект Менеджер 10 лет с компанией EPAM Systems. Разработчик в прошлом Возглавляю разработку инициатив внутри Центра Компетенций Тестирования. Report Portal – Product Owner. DZMITRY HUMIANIUK ДМИТРИЙ ГУМЕНЮК Проект Менеджер 10 лет с компанией EPAM Systems. Разработчик в прошлом Возглавляю разработку инициатив внутри Центра Компетенций Тестирования. Report Portal – Product Owner.
  4. 4. http://ReportPortal.io пример CI/CD с метриками, на основе продукта с Открытым Кодом
  5. 5. http://ReportPortal.io пример CI/CD с метриками, на основе продукта с Открытым Кодом OPEN SOURCED GPL v3.0 GITHUB.COM/ REPORTPORTAL
  6. 6. http://ReportPortal.io пример CI/CD с метриками, на основе продукта с Открытым Кодом
  7. 7. http://ReportPortal.io пример CI/CD с метриками, на основе продукта с Открытым Кодом МЕТРИКИ
  8. 8. МЕТРИКИ “ПОЖАРНЫЕ ИЗВЕЩАТЕЛИ” Name Why it is important to monitor this What decision can be made Periodi city Calculation Execution Frequenc y To understand and evaluate how often test cases executed. Goal: Increase execution ratio per day/build. Configure automated execution of TA after commits (smoke, acceptance) and regressions right after nightly builds. Triggers 1 execution per week or less (???) Target 1 per commit Interpretation code change should be verified not to affect functionality or other code base. Each day Measures 1. test executions count Formula Unit of measure Number % of results analyzed To understand and evaluate percentage of reviewed fail reports. If fail report not reviewed - there is not any value Triggers <60% Target 100% Interpretation failed report(test cases) should be review Each day Measures 1. No of not analyzed failed test cases (TI) 2. No of failed due to product bug (PB) 3. No of failed due to automation issues (AB) 4. No of failed due to system issue (SI) Formula 100% - To_Investigate / (PB+AB+SI+TI) Unit of measure % http://ReportPortal.io пример CI/CD с метриками, на основе продукта с Открытым Кодом
  9. 9. МЕТРИКИ “ПОЖАРНЫЕ ИЗВЕЩАТЕЛИ” Name Why it is important to monitor this What decision can be made Period icity Calculation % of product issues (stability of automatio n) To understand and evaluate percentage of failed test which corresponds to production issues Triggers trend not being positive Target >90% Interpretation trend of fails due to test automation instability should decrease (automation issues, system issue, not analyzed failed test cases) Measures 1. No of not analyzed failed test cases (TI) 2. No of failed due to product bug (PB) 3. No of failed due to automation issues (AB) 4. No of failed due to system issue (SI) Formula PB / (PB+AB+SI+TI) Unit of measure % % of automatio n issues To understand and evaluate percentage of failed test which corresponds to test automation issues (code issues, test case invalid) Triggers trend not being positive Target <5% Interpretation trend of fails due to test automation instability should decrease Measures 1. No of not analyzed failed test cases (TI) 2. No of failed due to product bug (PB) 3. No of failed due to automation issues (AB) 4. No of failed due to system issue (SI) Formula AB / (PB+AB+SI+TI) Unit of measure % http://ReportPortal.io пример CI/CD с метриками, на основе продукта с Открытым Кодом
  10. 10. МЕТРИКИ “ПОЖАРНЫЕ ИЗВЕЩАТЕЛИ” Name Why it is important to monitor this What decision can be made Perio dicity Calculation % of system issues To understand and evaluate percentage of failed test which corresponds to system issues (no connectivity, lack of resources and etc.) Triggers trend not being positive Target <5% Interpretation trend of fails due to system issue instability should decrease Measures 1. No of not analyzed failed test cases (TI) 2. No of failed due to product bug (PB) 3. No of failed due to automation issues (AB) 4. No of failed due to system issue (SI) Formula SI / (PB+AB+SI+TI) Unit of measure % Execution Time To understand and evaluate time of each execution cycle. Triggers Regression cycles > 15h acceptance cycles >1h Target regression <8h acceptance <1h Interpretation time for execution should not go over described limits Measures 1. Execution duration of regression, hours (Reg) 2. Execution duration of acceptance, hours (Acp) Formula Reg < 15h Acp < 1h Unit of measure Hours http://ReportPortal.io пример CI/CD с метриками, на основе продукта с Открытым Кодом
  11. 11. http://ReportPortal.io пример CI/CD с метриками, на основе продукта с Открытым Кодом ReportPortal.io Community M eet-up 14 December 2016 19:00 Imaguru, M insk
  12. 12. http://ReportPortal.io пример CI/CD с метриками, на основе продукта с Открытым Кодом http:/ / ReportPortal.io GITHUB.COM / REPORTPORTAL Report Portal Community ReportPortal.io Community ReportPortal_io

×