SlideShare a Scribd company logo
Building Quality Gates
and automated decisions in CI/CD pipelines.
What we learned and built on the top of a regression
of 200,000 test cases
Dmitriy Gumeniuk
EPAM Systems
Head of Testing Products
CEO ReportPortal.io
Dmitriy Gumeniuk
Head of Testing Products @ EPAM Systems
Based in New York Metro Area
Leads development of a portfolio of accelerators
focused on the usage of Machine Learning and Neural
Networks in test automation.
While having 16 years of experience in software development,
has been contributing to DevTestOps communities by actively
speaking at events and organizing the DelEx Conference,
aimed to inspire DevTestOps practices.
CEO ReportPortal.io
/ghdmitry /ghdmitry
Dm
itr
iy
Gu
me
niu
k
ReportPortal.io
DelEx
Conference
Dmitriy
Gumeniuk
Drill4J.github.io
Healenium.io
TDSpora.ai
QASpace JIRA
plugin
ReportPortal.io – AI-powered Test Automation
Dashboard
Drill4J – Test Gap and Impact analysis, regression
minimization tool
Healenium – self-healing capabilities for Selenium-
based test cases
TDSpora.ai – ML-based synthetic data generation
based on production data
QASpace – JIRA plugin for neat Test Case
Management
Quality Gates?
CODE BUILD TEST REALESE DEPLOY
Quality Gates
Quality gates are checkpoints at which the
project's quality is assessed.
They involve a set of predefined criteria that must
be met before a software product can progress
through the development lifecycle.
These gates are used to ensure that the project is
meeting the established quality standards and to
identify and address any issues that may arise during
the development process.
REALESE DEPLOY
Quality Gates Benefits of establishing
1. Improved quality: Quаlity Gates maintain high standards, resulting in reliable and
stable products with better user satisfaction.
2. Early identification of issues: Detect potential problems early in the development
process, reducing the cost and effort involved in fixing defects.
3. Increased efficiency: Streamline the development process by enforcing clear criteria
of acceptance, leading to more efficient use of resources.
4. Better communication: Promote a shared understanding of quality expectations
among team members at all stages, fostering improved teamwork.
5. Enhanced risk management: Mitigate risks by ensuring software meets predefined
quality criteria before release, reducing the likelihood of issues impacting users.
6. Faster time-to-market: Identify and address issues early, shortening the time it takes
to bring a high-quality product to market.
7. Informed decision-making: Quality Gates provide data-driven insights on software
quality, enabling stakeholders to make informed decisions regarding product
releases.
CODE BUILD TEST REALESE DEPLOY
Quality Gates
Quality gates
Non-functional:
- Execution time
- Test Flakiness
- Code quality (static code analysis)
- Code Reviews
- Environment stability
- Performance testing
- Security testing
- Accessibility testing
Functional:
- Pass/fail ratio
- Feature/component coverage
- User flows health
- Defect detection rate
What’s in reality?
Test Report – build 34.5.2
Testing_Team_Lead
QA Manager
Test Report – build 34.5.2
Dear Manager,
The most popular
Quality Gates form:
What’s in reality?
“Oh Jeez….
I supposed to get it on Wednesday…”
MANAGER AT THIS TIME:
What’s in reality?
TEAM LEAD AT THIS TIME:
What’s in reality?
“Okay Guys, let’s run our Quality Gates check-list…”
What’s in reality?
The most common Quality Gates form:
- Emailable reports
- Check-list calls
PROS:
- They do exist
CONS:
- People factor:
- People are lazy
- People make mistakes
- Time factor
- No automated decision
RELEASE REJECT
CODE BUILD TEST REALESE DEPLOY
TEST
200 000test cases
63 days of execution time
in a single thread
21 VMs
x 6 threads
x 18 hours
550 CI/CD jobs to run
UNRELIABLE RESULTS
AUTOMATION MAINTENANCE EFFORT
AUTOMATION STATUS TRACKING
ANALYSIS AND MANAGING EFFORTS
Triaging failed test reports with a goal to identify product, automation or
system issues
Aggregation of all automated testing results and visibility to the actual
status
Ability to track the effort to stabilize and maintain the automated tests
High risk of sequence failure and processing time makes TA results
unreliable source of data
Regression
Component1
Component2
Component3
Component4
Component5
Component6
Component7
Intergration1
Intergration2
Intergration3
Intergration4
Intergration5
Intergration6
Intergration7
E2E Special
Bugs and issues
This Gate fails:
IF Percent of failure for ALL test > 0%
OR the amount of all test cases for ALL tests < X
#1 Consolidation of results
age of failed
#1 Consolidation of results
#2 Triage of failed results
#2 Triage of failed results
Product
Bugs Automation
issues
Environment
issues
#2 Triage of failed results
Product Bugs – actual value of test automation in finding product bugs
Automation issues – technical dept of QA automation team
(outdated tests, broken locator, incorrect test data, waiters, etc.)
Environment issues – infrastructure limitations and inconsistency
(configuration issues, limited budget, unstable connection,
version mismatches, etc.)
#2 Triage of failed results
Goal:
- categorize each and every failed test case
Expectations Reality
#2 Triage of failed results
Improved visibility returns as investments into the QA infrastructure
Goal:
- categorize each and every failed test case
Why?
- Known issues
- Multiply failures because of the one specific issue
- Just boring
Solution
- ML algorithms to triage known issues
Result of ML categorization:
- Known issues categorized automatically
- Unknown issues are grouped for bulk investigation
- Team focuses on new failures
- Released team effort is focused on stabilization and improvements.
This Gate fails:
IF Percent of failure for ALL test > 0%
OR the amount of all test cases for ALL tests < X
OR the percent of defect type for ALL test > 3%
Total AB
OR the percent of defect type for ALL test > 0%
Critical PB
#3 Test Case attributes enrichment
@Test
@TestCaseId("TNG.1.0.1")
@Attributes(attributes = {
@Attribute(key = ”Priority", value = ”critical") ,
@Attribute(key = ”Role", value = ”Admin") ,
@Attribute(key = ”Flow", value = ”configuration") ,
@Attribute(key = ”Component", value = ”DCHP"),
@Attribute(key = ”IP-Type", value = ”IPv6") })
public void testDHCPconfigurationForIPv6() {
...
Assert.assertEquals(1, 1);
}
Goal:
- Establish traceability between tests and requirements.
How:
- Adopt `Documentation as a Code` principals – tag test cases with attributes
- Establish the best practice of test case implementation
- DOD: test attributes in Acceptance Criteria
#3 Test Case attributes enrichment
#3 Test Case attributes enrichment
#3 Test Case attributes enrichment
#3 Test Case attributes enrichment
This Gate fails:
IF Percent of failure for ALL test > 0%
OR the amount of all test cases for ALL tests < X
OR the percent of defect type for test
with > 1%
Total AB
component:payment
OR the percent of defect type for test
with > 2%
Product Bugs
severity:trivial
OR the percent of defect type for ALL test > 3%
Total AB
OR the percent of defect type for ALL test > 0%
Critical PB
OR new failures for exist
layer:API
#3 Test Case attributes enrichment
This Gate fails:
IF Percent of failure for ALL test > 0%
OR the amount of all test cases for ALL tests < X
OR the percent of defect type for test
with > 1%
Total AB
component:payment
OR the percent of defect type for test
with > 2%
Product Bugs
severity:trivial
OR the percent of defect type for ALL test > 3%
Total AB
OR the percent of defect type for ALL test > 0%
Critical PB
OR new failures for exist
layer:API
Automation
Product
Environment
#4 Results
severity:trivial layer:API
Automation
Product
Environment
component:payment
#1 Consolidation of results
webhook
http://ReportPortal.io
/ghdmitry /ghdmitry
Dmitriy Gumeniuk

More Related Content

What's hot

Agile Testing Process
Agile Testing ProcessAgile Testing Process
Agile Testing Process
Intetics
 
Software Testing Maturity Model and Assessment by Abstracta
Software Testing Maturity Model and Assessment by AbstractaSoftware Testing Maturity Model and Assessment by Abstracta
Software Testing Maturity Model and Assessment by Abstracta
Kalei White
 
QA process Presentation
QA process PresentationQA process Presentation
QA process Presentation
Nadeeshani Aththanagoda
 
Easy Setup for Parallel Test Execution with Selenium Docker
Easy Setup for Parallel Test Execution with Selenium DockerEasy Setup for Parallel Test Execution with Selenium Docker
Easy Setup for Parallel Test Execution with Selenium Docker
Sargis Sargsyan
 
Módulo 6. Agile Testing
Módulo 6. Agile TestingMódulo 6. Agile Testing
Módulo 6. Agile Testing
Johnny Ordóñez
 
New trends in testing automation
New trends in testing automationNew trends in testing automation
New trends in testing automation
Eran Kinsbrunner
 
Automation testing introduction for FujiNet
Automation testing introduction for FujiNetAutomation testing introduction for FujiNet
Automation testing introduction for FujiNet
Hai Tran Son
 
Transforming Your QA and Test Team
Transforming Your QA and Test TeamTransforming Your QA and Test Team
Transforming Your QA and Test Team
TechWell
 
Role Of Qa And Testing In Agile 1225221397167302 8
Role Of Qa And Testing In Agile 1225221397167302 8Role Of Qa And Testing In Agile 1225221397167302 8
Role Of Qa And Testing In Agile 1225221397167302 8
a34sharm
 
Agile QA presentation
Agile QA presentationAgile QA presentation
Agile QA presentation
Carl Bruiners
 
Introduction & Manual Testing
Introduction & Manual TestingIntroduction & Manual Testing
Introduction & Manual Testing
VenkateswaraRao Siddabathula
 
Agile testing - Testing From Day 1
Agile testing - Testing From Day 1Agile testing - Testing From Day 1
Agile testing - Testing From Day 1
Kaizenko
 
6 Traits of a Successful Test Automation Architecture
6 Traits of a Successful Test Automation Architecture6 Traits of a Successful Test Automation Architecture
6 Traits of a Successful Test Automation Architecture
Erdem YILDIRIM
 
Chapter 5 - Improving the Testing Process
Chapter 5 -  Improving the Testing ProcessChapter 5 -  Improving the Testing Process
Chapter 5 - Improving the Testing Process
Neeraj Kumar Singh
 
Xray for Jira - How to automate your QA process
Xray for Jira - How to automate your QA processXray for Jira - How to automate your QA process
Xray for Jira - How to automate your QA process
Xpand IT
 
Cypress e2e automation testing - day1 intor by: Hassan Hameed
Cypress e2e automation testing -  day1 intor by: Hassan HameedCypress e2e automation testing -  day1 intor by: Hassan Hameed
Cypress e2e automation testing - day1 intor by: Hassan Hameed
Hassan Muhammad
 
Automated Mobile Testing using Appium.pdf
Automated Mobile Testing using Appium.pdfAutomated Mobile Testing using Appium.pdf
Automated Mobile Testing using Appium.pdf
Anand722237
 
Agile QA process
Agile QA processAgile QA process
Agile QA process
Ashish Agrawal
 
Introduction to Agile software testing
Introduction to Agile software testingIntroduction to Agile software testing
Introduction to Agile software testing
KMS Technology
 
Load and performance testing
Load and performance testingLoad and performance testing
Load and performance testing
Qualitest
 

What's hot (20)

Agile Testing Process
Agile Testing ProcessAgile Testing Process
Agile Testing Process
 
Software Testing Maturity Model and Assessment by Abstracta
Software Testing Maturity Model and Assessment by AbstractaSoftware Testing Maturity Model and Assessment by Abstracta
Software Testing Maturity Model and Assessment by Abstracta
 
QA process Presentation
QA process PresentationQA process Presentation
QA process Presentation
 
Easy Setup for Parallel Test Execution with Selenium Docker
Easy Setup for Parallel Test Execution with Selenium DockerEasy Setup for Parallel Test Execution with Selenium Docker
Easy Setup for Parallel Test Execution with Selenium Docker
 
Módulo 6. Agile Testing
Módulo 6. Agile TestingMódulo 6. Agile Testing
Módulo 6. Agile Testing
 
New trends in testing automation
New trends in testing automationNew trends in testing automation
New trends in testing automation
 
Automation testing introduction for FujiNet
Automation testing introduction for FujiNetAutomation testing introduction for FujiNet
Automation testing introduction for FujiNet
 
Transforming Your QA and Test Team
Transforming Your QA and Test TeamTransforming Your QA and Test Team
Transforming Your QA and Test Team
 
Role Of Qa And Testing In Agile 1225221397167302 8
Role Of Qa And Testing In Agile 1225221397167302 8Role Of Qa And Testing In Agile 1225221397167302 8
Role Of Qa And Testing In Agile 1225221397167302 8
 
Agile QA presentation
Agile QA presentationAgile QA presentation
Agile QA presentation
 
Introduction & Manual Testing
Introduction & Manual TestingIntroduction & Manual Testing
Introduction & Manual Testing
 
Agile testing - Testing From Day 1
Agile testing - Testing From Day 1Agile testing - Testing From Day 1
Agile testing - Testing From Day 1
 
6 Traits of a Successful Test Automation Architecture
6 Traits of a Successful Test Automation Architecture6 Traits of a Successful Test Automation Architecture
6 Traits of a Successful Test Automation Architecture
 
Chapter 5 - Improving the Testing Process
Chapter 5 -  Improving the Testing ProcessChapter 5 -  Improving the Testing Process
Chapter 5 - Improving the Testing Process
 
Xray for Jira - How to automate your QA process
Xray for Jira - How to automate your QA processXray for Jira - How to automate your QA process
Xray for Jira - How to automate your QA process
 
Cypress e2e automation testing - day1 intor by: Hassan Hameed
Cypress e2e automation testing -  day1 intor by: Hassan HameedCypress e2e automation testing -  day1 intor by: Hassan Hameed
Cypress e2e automation testing - day1 intor by: Hassan Hameed
 
Automated Mobile Testing using Appium.pdf
Automated Mobile Testing using Appium.pdfAutomated Mobile Testing using Appium.pdf
Automated Mobile Testing using Appium.pdf
 
Agile QA process
Agile QA processAgile QA process
Agile QA process
 
Introduction to Agile software testing
Introduction to Agile software testingIntroduction to Agile software testing
Introduction to Agile software testing
 
Load and performance testing
Load and performance testingLoad and performance testing
Load and performance testing
 

Similar to Building functional Quality Gates with ReportPortal

Software Quality and Test Strategies for Ruby and Rails Applications
Software Quality and Test Strategies for Ruby and Rails ApplicationsSoftware Quality and Test Strategies for Ruby and Rails Applications
Software Quality and Test Strategies for Ruby and Rails Applications
Bhavin Javia
 
Resume 2 year
Resume  2 yearResume  2 year
Resume 2 year
pawan kumar
 
ISTQB Foundation - Chapter 2
ISTQB Foundation - Chapter 2ISTQB Foundation - Chapter 2
ISTQB Foundation - Chapter 2
Chandukar
 
Software engineering quality assurance and testing
Software engineering quality assurance and testingSoftware engineering quality assurance and testing
Software engineering quality assurance and testing
Bipul Roy Bpl
 
Enhancing Software Quality
Enhancing Software QualityEnhancing Software Quality
Enhancing Software Quality
Anand Prabhala
 
CV_SyedShoeb_2015
CV_SyedShoeb_2015CV_SyedShoeb_2015
CV_SyedShoeb_2015
Syed Shoeb
 
JagSinghupdated
JagSinghupdatedJagSinghupdated
JagSinghupdated
jag singh
 
ISTQB / ISEB Foundation Exam Practice - 2
ISTQB / ISEB Foundation Exam Practice - 2ISTQB / ISEB Foundation Exam Practice - 2
ISTQB / ISEB Foundation Exam Practice - 2
Yogindernath Gupta
 
Priti_Mule_ETL and ManualTesting(7.6+yrs Exp)
Priti_Mule_ETL and ManualTesting(7.6+yrs Exp)Priti_Mule_ETL and ManualTesting(7.6+yrs Exp)
Priti_Mule_ETL and ManualTesting(7.6+yrs Exp)
Priti Mule
 
Testing Interview Questions.pdf
Testing Interview Questions.pdfTesting Interview Questions.pdf
Testing Interview Questions.pdf
PradeepaKannan6
 
Product quality in agile project
Product quality in agile projectProduct quality in agile project
Product quality in agile project
Nhan Nguyen
 
Lightning Talks by Globant - Automation (This app runs by itself )
Lightning Talks by Globant -  Automation (This app runs by itself ) Lightning Talks by Globant -  Automation (This app runs by itself )
Lightning Talks by Globant - Automation (This app runs by itself )
Globant
 
ISTQB, ISEB Lecture Notes- 2
ISTQB, ISEB Lecture Notes- 2ISTQB, ISEB Lecture Notes- 2
ISTQB, ISEB Lecture Notes- 2
onsoftwaretest
 
Soumya sree Sridharala
Soumya sree SridharalaSoumya sree Sridharala
Soumya sree Sridharala
Soumya Sree Sridharala
 
Neha_Maggu
Neha_MagguNeha_Maggu
Neha_Maggu
Neha Maggu
 
IT8076 – Software Testing Intro
IT8076 – Software Testing IntroIT8076 – Software Testing Intro
IT8076 – Software Testing Intro
JohnSamuel280314
 
ISTQBCH2.ppt
ISTQBCH2.pptISTQBCH2.ppt
ISTQBCH2.ppt
RppsKumar1
 
ISTQBCH2.ppt
ISTQBCH2.pptISTQBCH2.ppt
ISTQBCH2.ppt
ghkadous
 
Neeraj_Kumar
Neeraj_KumarNeeraj_Kumar
Neeraj_Kumar
NEERAJ KUMAR
 
Software testing ppt
Software testing pptSoftware testing ppt
Software testing ppt
Poonkodi Jayakumar
 

Similar to Building functional Quality Gates with ReportPortal (20)

Software Quality and Test Strategies for Ruby and Rails Applications
Software Quality and Test Strategies for Ruby and Rails ApplicationsSoftware Quality and Test Strategies for Ruby and Rails Applications
Software Quality and Test Strategies for Ruby and Rails Applications
 
Resume 2 year
Resume  2 yearResume  2 year
Resume 2 year
 
ISTQB Foundation - Chapter 2
ISTQB Foundation - Chapter 2ISTQB Foundation - Chapter 2
ISTQB Foundation - Chapter 2
 
Software engineering quality assurance and testing
Software engineering quality assurance and testingSoftware engineering quality assurance and testing
Software engineering quality assurance and testing
 
Enhancing Software Quality
Enhancing Software QualityEnhancing Software Quality
Enhancing Software Quality
 
CV_SyedShoeb_2015
CV_SyedShoeb_2015CV_SyedShoeb_2015
CV_SyedShoeb_2015
 
JagSinghupdated
JagSinghupdatedJagSinghupdated
JagSinghupdated
 
ISTQB / ISEB Foundation Exam Practice - 2
ISTQB / ISEB Foundation Exam Practice - 2ISTQB / ISEB Foundation Exam Practice - 2
ISTQB / ISEB Foundation Exam Practice - 2
 
Priti_Mule_ETL and ManualTesting(7.6+yrs Exp)
Priti_Mule_ETL and ManualTesting(7.6+yrs Exp)Priti_Mule_ETL and ManualTesting(7.6+yrs Exp)
Priti_Mule_ETL and ManualTesting(7.6+yrs Exp)
 
Testing Interview Questions.pdf
Testing Interview Questions.pdfTesting Interview Questions.pdf
Testing Interview Questions.pdf
 
Product quality in agile project
Product quality in agile projectProduct quality in agile project
Product quality in agile project
 
Lightning Talks by Globant - Automation (This app runs by itself )
Lightning Talks by Globant -  Automation (This app runs by itself ) Lightning Talks by Globant -  Automation (This app runs by itself )
Lightning Talks by Globant - Automation (This app runs by itself )
 
ISTQB, ISEB Lecture Notes- 2
ISTQB, ISEB Lecture Notes- 2ISTQB, ISEB Lecture Notes- 2
ISTQB, ISEB Lecture Notes- 2
 
Soumya sree Sridharala
Soumya sree SridharalaSoumya sree Sridharala
Soumya sree Sridharala
 
Neha_Maggu
Neha_MagguNeha_Maggu
Neha_Maggu
 
IT8076 – Software Testing Intro
IT8076 – Software Testing IntroIT8076 – Software Testing Intro
IT8076 – Software Testing Intro
 
ISTQBCH2.ppt
ISTQBCH2.pptISTQBCH2.ppt
ISTQBCH2.ppt
 
ISTQBCH2.ppt
ISTQBCH2.pptISTQBCH2.ppt
ISTQBCH2.ppt
 
Neeraj_Kumar
Neeraj_KumarNeeraj_Kumar
Neeraj_Kumar
 
Software testing ppt
Software testing pptSoftware testing ppt
Software testing ppt
 

More from Dmitriy Gumeniuk

Self healing test automation with Healenium and Minimization of regression su...
Self healing test automation with Healenium and Minimization of regression su...Self healing test automation with Healenium and Minimization of regression su...
Self healing test automation with Healenium and Minimization of regression su...
Dmitriy Gumeniuk
 
Test Gap Analysis and regression minimization with Drill4j. Observability on ...
Test Gap Analysis and regression minimization with Drill4j. Observability on ...Test Gap Analysis and regression minimization with Drill4j. Observability on ...
Test Gap Analysis and regression minimization with Drill4j. Observability on ...
Dmitriy Gumeniuk
 
What's new in selenium grid 4.0 expected
What's new in selenium grid 4.0 expectedWhat's new in selenium grid 4.0 expected
What's new in selenium grid 4.0 expected
Dmitriy Gumeniuk
 
How IntelliJ IDEA plugin can help you in QA Automation
How IntelliJ IDEA plugin can help you in QA AutomationHow IntelliJ IDEA plugin can help you in QA Automation
How IntelliJ IDEA plugin can help you in QA Automation
Dmitriy Gumeniuk
 
Automation of Security scanning easy or cheese?
Automation of Security scanning easy or cheese?Automation of Security scanning easy or cheese?
Automation of Security scanning easy or cheese?
Dmitriy Gumeniuk
 
Device manager server for android farm or running more than 30k tests every day
Device manager server for android farm or running more than 30k tests every dayDevice manager server for android farm or running more than 30k tests every day
Device manager server for android farm or running more than 30k tests every day
Dmitriy Gumeniuk
 
Migrating to the serverless mindset
Migrating to the serverless mindsetMigrating to the serverless mindset
Migrating to the serverless mindset
Dmitriy Gumeniuk
 
Building a self-service marketplace for Test Data (Dzmitry Humianiuk, Belarus)
Building a self-service marketplace for Test Data (Dzmitry Humianiuk, Belarus)Building a self-service marketplace for Test Data (Dzmitry Humianiuk, Belarus)
Building a self-service marketplace for Test Data (Dzmitry Humianiuk, Belarus)
Dmitriy Gumeniuk
 
ReportPortal use cases presentation
 ReportPortal use cases presentation ReportPortal use cases presentation
ReportPortal use cases presentation
Dmitriy Gumeniuk
 
Applicabilitity of Machine Learning in Test Automation @ Delex Conf 2018, Minsk
Applicabilitity of Machine Learning in Test Automation @ Delex Conf 2018, MinskApplicabilitity of Machine Learning in Test Automation @ Delex Conf 2018, Minsk
Applicabilitity of Machine Learning in Test Automation @ Delex Conf 2018, Minsk
Dmitriy Gumeniuk
 
ReportPortal.io - how to make machine learning categorize your test fails
ReportPortal.io - how to make machine learning categorize your test failsReportPortal.io - how to make machine learning categorize your test fails
ReportPortal.io - how to make machine learning categorize your test fails
Dmitriy Gumeniuk
 

More from Dmitriy Gumeniuk (11)

Self healing test automation with Healenium and Minimization of regression su...
Self healing test automation with Healenium and Minimization of regression su...Self healing test automation with Healenium and Minimization of regression su...
Self healing test automation with Healenium and Minimization of regression su...
 
Test Gap Analysis and regression minimization with Drill4j. Observability on ...
Test Gap Analysis and regression minimization with Drill4j. Observability on ...Test Gap Analysis and regression minimization with Drill4j. Observability on ...
Test Gap Analysis and regression minimization with Drill4j. Observability on ...
 
What's new in selenium grid 4.0 expected
What's new in selenium grid 4.0 expectedWhat's new in selenium grid 4.0 expected
What's new in selenium grid 4.0 expected
 
How IntelliJ IDEA plugin can help you in QA Automation
How IntelliJ IDEA plugin can help you in QA AutomationHow IntelliJ IDEA plugin can help you in QA Automation
How IntelliJ IDEA plugin can help you in QA Automation
 
Automation of Security scanning easy or cheese?
Automation of Security scanning easy or cheese?Automation of Security scanning easy or cheese?
Automation of Security scanning easy or cheese?
 
Device manager server for android farm or running more than 30k tests every day
Device manager server for android farm or running more than 30k tests every dayDevice manager server for android farm or running more than 30k tests every day
Device manager server for android farm or running more than 30k tests every day
 
Migrating to the serverless mindset
Migrating to the serverless mindsetMigrating to the serverless mindset
Migrating to the serverless mindset
 
Building a self-service marketplace for Test Data (Dzmitry Humianiuk, Belarus)
Building a self-service marketplace for Test Data (Dzmitry Humianiuk, Belarus)Building a self-service marketplace for Test Data (Dzmitry Humianiuk, Belarus)
Building a self-service marketplace for Test Data (Dzmitry Humianiuk, Belarus)
 
ReportPortal use cases presentation
 ReportPortal use cases presentation ReportPortal use cases presentation
ReportPortal use cases presentation
 
Applicabilitity of Machine Learning in Test Automation @ Delex Conf 2018, Minsk
Applicabilitity of Machine Learning in Test Automation @ Delex Conf 2018, MinskApplicabilitity of Machine Learning in Test Automation @ Delex Conf 2018, Minsk
Applicabilitity of Machine Learning in Test Automation @ Delex Conf 2018, Minsk
 
ReportPortal.io - how to make machine learning categorize your test fails
ReportPortal.io - how to make machine learning categorize your test failsReportPortal.io - how to make machine learning categorize your test fails
ReportPortal.io - how to make machine learning categorize your test fails
 

Recently uploaded

Things to Consider When Choosing a Website Developer for your Website | FODUU
Things to Consider When Choosing a Website Developer for your Website | FODUUThings to Consider When Choosing a Website Developer for your Website | FODUU
Things to Consider When Choosing a Website Developer for your Website | FODUU
FODUU
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
shyamraj55
 
20240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 202420240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 2024
Matthew Sinclair
 
Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
ssuserfac0301
 
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAUHCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
panagenda
 
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing DaysClimate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
Kari Kakkonen
 
AI-Powered Food Delivery Transforming App Development in Saudi Arabia.pdf
AI-Powered Food Delivery Transforming App Development in Saudi Arabia.pdfAI-Powered Food Delivery Transforming App Development in Saudi Arabia.pdf
AI-Powered Food Delivery Transforming App Development in Saudi Arabia.pdf
Techgropse Pvt.Ltd.
 
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
panagenda
 
Infrastructure Challenges in Scaling RAG with Custom AI models
Infrastructure Challenges in Scaling RAG with Custom AI modelsInfrastructure Challenges in Scaling RAG with Custom AI models
Infrastructure Challenges in Scaling RAG with Custom AI models
Zilliz
 
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracyGraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
Tomaz Bratanic
 
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AIMind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
Kumud Singh
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Safe Software
 
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial IntelligenceAI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
IndexBug
 
OpenID AuthZEN Interop Read Out - Authorization
OpenID AuthZEN Interop Read Out - AuthorizationOpenID AuthZEN Interop Read Out - Authorization
OpenID AuthZEN Interop Read Out - Authorization
David Brossard
 
Microsoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdfMicrosoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdf
Uni Systems S.M.S.A.
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Malak Abu Hammad
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
DianaGray10
 
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practicesNational Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
Quotidiano Piemontese
 
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and MilvusGenerating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
Zilliz
 
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems S.M.S.A.
 

Recently uploaded (20)

Things to Consider When Choosing a Website Developer for your Website | FODUU
Things to Consider When Choosing a Website Developer for your Website | FODUUThings to Consider When Choosing a Website Developer for your Website | FODUU
Things to Consider When Choosing a Website Developer for your Website | FODUU
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
 
20240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 202420240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 2024
 
Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
 
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAUHCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
 
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing DaysClimate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
 
AI-Powered Food Delivery Transforming App Development in Saudi Arabia.pdf
AI-Powered Food Delivery Transforming App Development in Saudi Arabia.pdfAI-Powered Food Delivery Transforming App Development in Saudi Arabia.pdf
AI-Powered Food Delivery Transforming App Development in Saudi Arabia.pdf
 
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
 
Infrastructure Challenges in Scaling RAG with Custom AI models
Infrastructure Challenges in Scaling RAG with Custom AI modelsInfrastructure Challenges in Scaling RAG with Custom AI models
Infrastructure Challenges in Scaling RAG with Custom AI models
 
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracyGraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
 
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AIMind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
 
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial IntelligenceAI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
 
OpenID AuthZEN Interop Read Out - Authorization
OpenID AuthZEN Interop Read Out - AuthorizationOpenID AuthZEN Interop Read Out - Authorization
OpenID AuthZEN Interop Read Out - Authorization
 
Microsoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdfMicrosoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdf
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
 
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practicesNational Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
 
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and MilvusGenerating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
 
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
 

Building functional Quality Gates with ReportPortal

  • 1. Building Quality Gates and automated decisions in CI/CD pipelines. What we learned and built on the top of a regression of 200,000 test cases Dmitriy Gumeniuk EPAM Systems Head of Testing Products CEO ReportPortal.io
  • 2. Dmitriy Gumeniuk Head of Testing Products @ EPAM Systems Based in New York Metro Area Leads development of a portfolio of accelerators focused on the usage of Machine Learning and Neural Networks in test automation. While having 16 years of experience in software development, has been contributing to DevTestOps communities by actively speaking at events and organizing the DelEx Conference, aimed to inspire DevTestOps practices. CEO ReportPortal.io /ghdmitry /ghdmitry Dm itr iy Gu me niu k
  • 3. ReportPortal.io DelEx Conference Dmitriy Gumeniuk Drill4J.github.io Healenium.io TDSpora.ai QASpace JIRA plugin ReportPortal.io – AI-powered Test Automation Dashboard Drill4J – Test Gap and Impact analysis, regression minimization tool Healenium – self-healing capabilities for Selenium- based test cases TDSpora.ai – ML-based synthetic data generation based on production data QASpace – JIRA plugin for neat Test Case Management
  • 5. CODE BUILD TEST REALESE DEPLOY Quality Gates Quality gates are checkpoints at which the project's quality is assessed. They involve a set of predefined criteria that must be met before a software product can progress through the development lifecycle. These gates are used to ensure that the project is meeting the established quality standards and to identify and address any issues that may arise during the development process.
  • 6. REALESE DEPLOY Quality Gates Benefits of establishing 1. Improved quality: Quаlity Gates maintain high standards, resulting in reliable and stable products with better user satisfaction. 2. Early identification of issues: Detect potential problems early in the development process, reducing the cost and effort involved in fixing defects. 3. Increased efficiency: Streamline the development process by enforcing clear criteria of acceptance, leading to more efficient use of resources. 4. Better communication: Promote a shared understanding of quality expectations among team members at all stages, fostering improved teamwork. 5. Enhanced risk management: Mitigate risks by ensuring software meets predefined quality criteria before release, reducing the likelihood of issues impacting users. 6. Faster time-to-market: Identify and address issues early, shortening the time it takes to bring a high-quality product to market. 7. Informed decision-making: Quality Gates provide data-driven insights on software quality, enabling stakeholders to make informed decisions regarding product releases.
  • 7. CODE BUILD TEST REALESE DEPLOY Quality Gates Quality gates Non-functional: - Execution time - Test Flakiness - Code quality (static code analysis) - Code Reviews - Environment stability - Performance testing - Security testing - Accessibility testing Functional: - Pass/fail ratio - Feature/component coverage - User flows health - Defect detection rate
  • 8. What’s in reality? Test Report – build 34.5.2 Testing_Team_Lead QA Manager Test Report – build 34.5.2 Dear Manager, The most popular Quality Gates form:
  • 9. What’s in reality? “Oh Jeez…. I supposed to get it on Wednesday…” MANAGER AT THIS TIME:
  • 10. What’s in reality? TEAM LEAD AT THIS TIME:
  • 11. What’s in reality? “Okay Guys, let’s run our Quality Gates check-list…”
  • 12. What’s in reality? The most common Quality Gates form: - Emailable reports - Check-list calls PROS: - They do exist CONS: - People factor: - People are lazy - People make mistakes - Time factor - No automated decision RELEASE REJECT
  • 13. CODE BUILD TEST REALESE DEPLOY
  • 14. TEST 200 000test cases 63 days of execution time in a single thread 21 VMs x 6 threads x 18 hours 550 CI/CD jobs to run UNRELIABLE RESULTS AUTOMATION MAINTENANCE EFFORT AUTOMATION STATUS TRACKING ANALYSIS AND MANAGING EFFORTS Triaging failed test reports with a goal to identify product, automation or system issues Aggregation of all automated testing results and visibility to the actual status Ability to track the effort to stabilize and maintain the automated tests High risk of sequence failure and processing time makes TA results unreliable source of data
  • 17.
  • 18.
  • 19. This Gate fails: IF Percent of failure for ALL test > 0% OR the amount of all test cases for ALL tests < X
  • 20.
  • 21.
  • 22.
  • 23.
  • 25. age of failed #1 Consolidation of results
  • 26. #2 Triage of failed results
  • 27. #2 Triage of failed results Product Bugs Automation issues Environment issues
  • 28. #2 Triage of failed results Product Bugs – actual value of test automation in finding product bugs Automation issues – technical dept of QA automation team (outdated tests, broken locator, incorrect test data, waiters, etc.) Environment issues – infrastructure limitations and inconsistency (configuration issues, limited budget, unstable connection, version mismatches, etc.)
  • 29. #2 Triage of failed results Goal: - categorize each and every failed test case Expectations Reality
  • 30. #2 Triage of failed results Improved visibility returns as investments into the QA infrastructure Goal: - categorize each and every failed test case Why? - Known issues - Multiply failures because of the one specific issue - Just boring Solution - ML algorithms to triage known issues Result of ML categorization: - Known issues categorized automatically - Unknown issues are grouped for bulk investigation - Team focuses on new failures - Released team effort is focused on stabilization and improvements.
  • 31. This Gate fails: IF Percent of failure for ALL test > 0% OR the amount of all test cases for ALL tests < X OR the percent of defect type for ALL test > 3% Total AB OR the percent of defect type for ALL test > 0% Critical PB
  • 32. #3 Test Case attributes enrichment @Test @TestCaseId("TNG.1.0.1") @Attributes(attributes = { @Attribute(key = ”Priority", value = ”critical") , @Attribute(key = ”Role", value = ”Admin") , @Attribute(key = ”Flow", value = ”configuration") , @Attribute(key = ”Component", value = ”DCHP"), @Attribute(key = ”IP-Type", value = ”IPv6") }) public void testDHCPconfigurationForIPv6() { ... Assert.assertEquals(1, 1); } Goal: - Establish traceability between tests and requirements. How: - Adopt `Documentation as a Code` principals – tag test cases with attributes - Establish the best practice of test case implementation - DOD: test attributes in Acceptance Criteria
  • 33. #3 Test Case attributes enrichment
  • 34.
  • 35. #3 Test Case attributes enrichment
  • 36. #3 Test Case attributes enrichment
  • 37. #3 Test Case attributes enrichment This Gate fails: IF Percent of failure for ALL test > 0% OR the amount of all test cases for ALL tests < X OR the percent of defect type for test with > 1% Total AB component:payment OR the percent of defect type for test with > 2% Product Bugs severity:trivial OR the percent of defect type for ALL test > 3% Total AB OR the percent of defect type for ALL test > 0% Critical PB OR new failures for exist layer:API
  • 38. #3 Test Case attributes enrichment This Gate fails: IF Percent of failure for ALL test > 0% OR the amount of all test cases for ALL tests < X OR the percent of defect type for test with > 1% Total AB component:payment OR the percent of defect type for test with > 2% Product Bugs severity:trivial OR the percent of defect type for ALL test > 3% Total AB OR the percent of defect type for ALL test > 0% Critical PB OR new failures for exist layer:API Automation Product Environment
  • 41.