QA Fest 2019. Анна Чернышова. Self-healing test automation 2.0. The Future

Q
SELF-HEALING TEST
AUTOMATION 2.0
THE FUTURE
• Problem statement
• How sha works
• Demo
• TAF integration
• POS results
• Future plans
ANNA CHERNYSHOVA
Lead Software Test Automation Engineer
Testing Competency Center Expert
Email: anna_chernyshova@epam.com
FB: anna.chernyshova.79
PROBLEMS
Unstable automated E2E tests inhibit CI
~40% team effort spent to fix automations
issues, rather than test product and add new
tests
Web application updates constantly every
sprint, which leads to locator change
PROBLEM STATEMENT
PROBLEMS GOALS FEATURES
Unstable automated E2E tests inhibit CI
~40% team effort spent to fix automations
issues, rather than test product and add new
tests
Web application updates constantly every
sprint, which leads to locator change
Improve E2E automated test stability
PROBLEM STATEMENT
PROBLEMS GOALS FEATURES
Unstable automated E2E tests inhibit CI
~40% team effort spent to fix automations
issues, rather than test product and add new
tests
Web application updates constantly every
sprint, which leads to locator change
Improve E2E automated test stability
Reduce amount of test cases failed due to
non-product issue
PROBLEM STATEMENT
PROBLEMS GOALS FEATURES
Unstable automated E2E tests inhibit CI
~40% team effort spent to fix automations
issues, rather than test product and add new
tests
Web application updates constantly every
sprint, which leads to locator change
Improve E2E automated test stability
Reduce amount of test cases failed due to
non-product issue
Reduces Tests Maintenance efforts
PROBLEM STATEMENT
PROBLEMS GOALS FEATURES
Unstable automated E2E tests inhibit CI
~40% team effort spent to fix automations
issues, rather than test product and add new
tests
Web application updates constantly every
sprint, which leads to locator change
Improve E2E automated test stability
Reduce amount of test cases failed due to
non-product issue
Reduces Tests Maintenance efforts
More time to cover new functionality with
tests
PROBLEM STATEMENT
PROBLEMS GOALS FEATURES
Unstable automated E2E tests inhibit CI
~40% team effort spent to fix automations
issues, rather than test product and add new
tests
Web application updates constantly every
sprint, which leads to locator change
Improve E2E automated test stability
Reduce amount of test cases failed due to
non-product issue
Reduces Tests Maintenance efforts
More time to cover new functionality with
tests
Shipping better products faster
PROBLEM STATEMENT
What we
have on the
market
Dynamic
locators
PROS & CONS
• Dynamic DOM state updates
• Easy to write and support tests
• Tests are stable
• Cross-browser, CI, Jira integrations from the box
• Only for new projects
• Strong dependency from tool ecosystem
• May be not secured
• May lose tests when project finished
Epam Self-healing
Applicable at any stage of project development
Minimal dependency from the TAF
Replace static locators with dynamic ones
Self-healing
library…
Gives ability to find controls (new locator) for updated
WEB pages
Use kinda ML algorithms for Web page changes
identification
Is a standalone Java jar connected to test cases code
base
Helps >2x times reduced test fails because of updated
UI
Provides Html report which handles locators analysis
Provides Intellij Idea plugin to make code updates with
new locator values
HOW SHA WORKS
LCS ALGORITHM
Longest common subsequence
problem of finding the longest subsequence common to all sequences in a set of sequences
*widely used by revision control systems such as Git
LCS ALGORITHM MODIFICATION
Longest common subsequence with weight
Added extra weight for tag, Id, class, value, other attributes
<button id=btn-1>
<button id=btn-1>
Self-healing selenium (jar)
Tree-comparing(jar)
Includes Tree-comparing dependency
Implements Selenium WebDriver
Overwrites findElement() method
Catch NoSuchElementException
Activates LCS algorithm in Tree-comparing library
Save reference element path to storage
Get reference element path from storage
Get current DOM state
Search in current state for the best subsequence
Generate new CSS locator
JAR LIBRARIES
Self-healing selenium (jar) Tree-comparing(jar)
Reference elements path
storage
Test
Driver.findElement(PageAwareBy(…))
Web Page
Target
element
Find Element
Test Automation Framework
Element Found
Self-healing selenium (jar) Tree-comparing(jar)
Reference elements path
storage
Test
Driver.findElement(PageAwareBy(…))
Web Page
Target
element
Find Element
Test Automation Framework
Element Found
Save successful path
Old locator
Self-healing selenium (jar) Tree-comparing(jar)
Reference elements path
storage
Test
Driver.findElement(PageAwareBy(…))
Web Page
Target
element
Find Element
Element Not Found
Test Automation Framework
Self-healing selenium (jar) Tree-comparing(jar)
Reference elements path
storage
Test
Driver.findElement(PageAwareBy(…))
Web Page
Target
element
Find Element
Element Not Found
Test Automation Framework
Old locator
Successful path
Page state
Self-healing selenium (jar) Tree-comparing(jar)
Reference elements path
storage
Test
Driver.findElement(PageAwareBy(…))
Web Page
Target
element
Find Element
Element Not Found
Test Automation Framework
Old locator New locator
Successful path
Page state
New locator
Self-healing selenium (jar) Tree-comparing(jar)
Reference elements path
storage
Test
Driver.findElement(PageAwareBy(…))
Web Page
Target
element
Find Element
Element Not Found
Test Automation Framework
Old locator New locator
Successful path
Page state
New locator
Find New Element
Self-healing selenium (jar) Tree-comparing(jar)
Reference elements path
storage
Test
Driver.findElement(PageAwareBy(…))
Web Page
Target
element
Find Element
Element Not Found
Test Automation Framework
Element Found
Save successful path
Old locator New locator
Successful path
Page state
New locator
Find New Element
QA Fest 2019. Анна Чернышова. Self-healing test automation 2.0. The Future
SelfHealingDriver
findElement
public WebElement findElement(By by) {
if (by instanceof PageAwareBy) {
try {
Trying to find the element and save its path if success
} catch (NoSuchElementException var5) {
Find previous reference element path in storage
Generate new best matches locator
return healedLocator;
}
} else {
return delegate.findElement(by);
}
}
PageAwareBy
extends By
PageAwareBy(String pageName, By by, SelfHealingEngine
engine) {
this.pageName = pageName;
this.by = by;
this.engine = engine;
} PageObject Page
Locator
New locator processor
Functions supported
• By() -> PageAwareBy()
• @FindBy -> @PageAwareFindBy
• WebDriver -> SelfHealingDriver
• Iframe support
• Actions support
• Remote test run
• Parallel test run
• Works with Selenium wrappers like Selenide
Html report
Intellij Idea plugin to make code updates
HTML REPORT
Intellij Idea plugin to make code updates
PLUGIN
HTML REPORT
Update
request
{
filename: MainPageWithFindBy
lineNumber: 49
failedLocatorValue://input[@name='EMAIL’]
healedLocatorValue: input.t186__input.js-tilda-rule.t-input
}
Intellij Idea plugin to make code updates
TAF CODE BASE
PLUGIN
HTML REPORT
Find locator and update
Update
request
{
filename: MainPageWithFindBy
lineNumber: 49
failedLocatorValue://input[@name='EMAIL’]
healedLocatorValue: input.t186__input.js-tilda-rule.t-input
}
TAF INTEGRATION
1. Declare custom EngineConfig with the path to store new locators. For example 'shaselenium'
EngineConfig engineConfig = EngineConfig
.custom()
.setStorage(new FileSystemPathStorage(Paths.get("sha", "selenium")))
.build();
2. Init driver instance of SelfHealingDriver
SelfHealingDriver driver = new SelfHealingDriver(new ChromeDriver(), engineConfig);
3. Locate elements
By buttonBy = PageAwareBy.by("MainPage", By.id(testButtonId));
Or
@PageAwareFindBy(page="MainPage", findBy = @FindBy(id = "markup-generation-button"))
WebElement buttonBy;
4. Interact with elements as usual
driver.findElement(buttonBy).click();
Important! Do not delete data from the folder where files with new
locators are stored. They are used to perform self-healing in next
automation runs
Important! "MainPage" is the name of the page to which the
WebElement belongs
RESULTS
Projects for pilot
TAF based on Java +
Selenium
(WebDriver)
Continuous UI
updates
UI generated by CMS
Projects for pilot
TAF based on Java +
Selenium
(WebDriver)
Continuous UI
updates
UI generated by CMS
>20 replies
6 Web projects for
pilot
3 projects are still
in focus
100% tests healed
Healed localization testing
Issues
Test execution time
increase
Can’t heal single element
from the list
Probability of
false/positive results
Code not updated after
healing
FUTURE PLANS
Future plans
Idea plugin Mobile support
Idea Plugin
✅ @PageAwareFindBy
❌ PageAwareBy(..)
❌ Local declaring
Thanks to
Oleh Khailenko
Lead Software Engineer
Kostiantyn Morozkin
Senior Software Engineer
Nikolai Kobzev
Senior Software Engineer
Vladimir Moshkin
SoftwareTest Automation Engineer
Email: anna_chernyshova@epam.com
FB: anna.chernyshova.79
1 of 44

More Related Content

What's hot(20)

Wax on, wax offWax on, wax off
Wax on, wax off
Bol.com Techlab239 views
Karate, the black belt of HTTP API testing?Karate, the black belt of HTTP API testing?
Karate, the black belt of HTTP API testing?
Bertrand Delacretaz1.3K views
Karate DSLKarate DSL
Karate DSL
anil borse410 views
Unit & integration testingUnit & integration testing
Unit & integration testing
Pavlo Hodysh407 views
BDD with JBehave and SeleniumBDD with JBehave and Selenium
BDD with JBehave and Selenium
Nikolay Vasilev61.7K views
Jbehave- Basics to AdvanceJbehave- Basics to Advance
Jbehave- Basics to Advance
Ravinder Singh1.9K views
Api testingApi testing
Api testing
Keshav Kashyap30.1K views
API Testing following the Test PyramidAPI Testing following the Test Pyramid
API Testing following the Test Pyramid
Elias Nogueira1.7K views
B4USolution_API-TestingB4USolution_API-Testing
B4USolution_API-Testing
b4usolution .1.3K views
Automated tests to a REST APIAutomated tests to a REST API
Automated tests to a REST API
Luís Barros Nóbrega511 views
Testing with laravelTesting with laravel
Testing with laravel
Derek Binkley1.7K views
Fitnesse - Acceptance testingFitnesse - Acceptance testing
Fitnesse - Acceptance testing
vijay_challa6.8K views
Data driven testingData driven testing
Data driven testing
Đăng Minh10.2K views

Similar to QA Fest 2019. Анна Чернышова. Self-healing test automation 2.0. The Future

Testing In JavaTesting In Java
Testing In JavaDavid Noble
29.8K views42 slides

Similar to QA Fest 2019. Анна Чернышова. Self-healing test automation 2.0. The Future(20)

More from QAFest(20)

Recently uploaded(20)

Scope of Biochemistry.pptxScope of Biochemistry.pptx
Scope of Biochemistry.pptx
shoba shoba110 views
Sociology KS5Sociology KS5
Sociology KS5
WestHatch50 views
Education and Diversity.pptxEducation and Diversity.pptx
Education and Diversity.pptx
DrHafizKosar56 views
Dance KS5 BreakdownDance KS5 Breakdown
Dance KS5 Breakdown
WestHatch52 views
SIMPLE PRESENT TENSE_new.pptxSIMPLE PRESENT TENSE_new.pptx
SIMPLE PRESENT TENSE_new.pptx
nisrinamadani2146 views
Class 10 English  lesson plansClass 10 English  lesson plans
Class 10 English lesson plans
Tariq KHAN172 views
Narration  ppt.pptxNarration  ppt.pptx
Narration ppt.pptx
Tariq KHAN62 views
ANATOMY AND PHYSIOLOGY UNIT 1 { PART-1}ANATOMY AND PHYSIOLOGY UNIT 1 { PART-1}
ANATOMY AND PHYSIOLOGY UNIT 1 { PART-1}
DR .PALLAVI PATHANIA156 views
Nico Baumbach IMR Media ComponentNico Baumbach IMR Media Component
Nico Baumbach IMR Media Component
InMediaRes1186 views
Lecture: Open InnovationLecture: Open Innovation
Lecture: Open Innovation
Michal Hron82 views
ICS3211_lecture 08_2023.pdfICS3211_lecture 08_2023.pdf
ICS3211_lecture 08_2023.pdf
Vanessa Camilleri68 views
AI Tools for Business and StartupsAI Tools for Business and Startups
AI Tools for Business and Startups
Svetlin Nakov57 views
Gopal Chakraborty Memorial Quiz 2.0 Prelims.pptxGopal Chakraborty Memorial Quiz 2.0 Prelims.pptx
Gopal Chakraborty Memorial Quiz 2.0 Prelims.pptx
Debapriya Chakraborty221 views
Industry4wrd.pptxIndustry4wrd.pptx
Industry4wrd.pptx
BC Chew153 views
Drama KS5 BreakdownDrama KS5 Breakdown
Drama KS5 Breakdown
WestHatch50 views
ACTIVITY BOOK key water sports.pptxACTIVITY BOOK key water sports.pptx
ACTIVITY BOOK key water sports.pptx
Mar Caston Palacio132 views

QA Fest 2019. Анна Чернышова. Self-healing test automation 2.0. The Future

  • 1. SELF-HEALING TEST AUTOMATION 2.0 THE FUTURE • Problem statement • How sha works • Demo • TAF integration • POS results • Future plans
  • 2. ANNA CHERNYSHOVA Lead Software Test Automation Engineer Testing Competency Center Expert Email: anna_chernyshova@epam.com FB: anna.chernyshova.79
  • 3. PROBLEMS Unstable automated E2E tests inhibit CI ~40% team effort spent to fix automations issues, rather than test product and add new tests Web application updates constantly every sprint, which leads to locator change PROBLEM STATEMENT
  • 4. PROBLEMS GOALS FEATURES Unstable automated E2E tests inhibit CI ~40% team effort spent to fix automations issues, rather than test product and add new tests Web application updates constantly every sprint, which leads to locator change Improve E2E automated test stability PROBLEM STATEMENT
  • 5. PROBLEMS GOALS FEATURES Unstable automated E2E tests inhibit CI ~40% team effort spent to fix automations issues, rather than test product and add new tests Web application updates constantly every sprint, which leads to locator change Improve E2E automated test stability Reduce amount of test cases failed due to non-product issue PROBLEM STATEMENT
  • 6. PROBLEMS GOALS FEATURES Unstable automated E2E tests inhibit CI ~40% team effort spent to fix automations issues, rather than test product and add new tests Web application updates constantly every sprint, which leads to locator change Improve E2E automated test stability Reduce amount of test cases failed due to non-product issue Reduces Tests Maintenance efforts PROBLEM STATEMENT
  • 7. PROBLEMS GOALS FEATURES Unstable automated E2E tests inhibit CI ~40% team effort spent to fix automations issues, rather than test product and add new tests Web application updates constantly every sprint, which leads to locator change Improve E2E automated test stability Reduce amount of test cases failed due to non-product issue Reduces Tests Maintenance efforts More time to cover new functionality with tests PROBLEM STATEMENT
  • 8. PROBLEMS GOALS FEATURES Unstable automated E2E tests inhibit CI ~40% team effort spent to fix automations issues, rather than test product and add new tests Web application updates constantly every sprint, which leads to locator change Improve E2E automated test stability Reduce amount of test cases failed due to non-product issue Reduces Tests Maintenance efforts More time to cover new functionality with tests Shipping better products faster PROBLEM STATEMENT
  • 9. What we have on the market
  • 11. PROS & CONS • Dynamic DOM state updates • Easy to write and support tests • Tests are stable • Cross-browser, CI, Jira integrations from the box • Only for new projects • Strong dependency from tool ecosystem • May be not secured • May lose tests when project finished
  • 12. Epam Self-healing Applicable at any stage of project development Minimal dependency from the TAF Replace static locators with dynamic ones
  • 13. Self-healing library… Gives ability to find controls (new locator) for updated WEB pages Use kinda ML algorithms for Web page changes identification Is a standalone Java jar connected to test cases code base Helps >2x times reduced test fails because of updated UI Provides Html report which handles locators analysis Provides Intellij Idea plugin to make code updates with new locator values
  • 15. LCS ALGORITHM Longest common subsequence problem of finding the longest subsequence common to all sequences in a set of sequences *widely used by revision control systems such as Git
  • 16. LCS ALGORITHM MODIFICATION Longest common subsequence with weight Added extra weight for tag, Id, class, value, other attributes <button id=btn-1> <button id=btn-1>
  • 17. Self-healing selenium (jar) Tree-comparing(jar) Includes Tree-comparing dependency Implements Selenium WebDriver Overwrites findElement() method Catch NoSuchElementException Activates LCS algorithm in Tree-comparing library Save reference element path to storage Get reference element path from storage Get current DOM state Search in current state for the best subsequence Generate new CSS locator JAR LIBRARIES
  • 18. Self-healing selenium (jar) Tree-comparing(jar) Reference elements path storage Test Driver.findElement(PageAwareBy(…)) Web Page Target element Find Element Test Automation Framework Element Found
  • 19. Self-healing selenium (jar) Tree-comparing(jar) Reference elements path storage Test Driver.findElement(PageAwareBy(…)) Web Page Target element Find Element Test Automation Framework Element Found Save successful path Old locator
  • 20. Self-healing selenium (jar) Tree-comparing(jar) Reference elements path storage Test Driver.findElement(PageAwareBy(…)) Web Page Target element Find Element Element Not Found Test Automation Framework
  • 21. Self-healing selenium (jar) Tree-comparing(jar) Reference elements path storage Test Driver.findElement(PageAwareBy(…)) Web Page Target element Find Element Element Not Found Test Automation Framework Old locator Successful path Page state
  • 22. Self-healing selenium (jar) Tree-comparing(jar) Reference elements path storage Test Driver.findElement(PageAwareBy(…)) Web Page Target element Find Element Element Not Found Test Automation Framework Old locator New locator Successful path Page state New locator
  • 23. Self-healing selenium (jar) Tree-comparing(jar) Reference elements path storage Test Driver.findElement(PageAwareBy(…)) Web Page Target element Find Element Element Not Found Test Automation Framework Old locator New locator Successful path Page state New locator Find New Element
  • 24. Self-healing selenium (jar) Tree-comparing(jar) Reference elements path storage Test Driver.findElement(PageAwareBy(…)) Web Page Target element Find Element Element Not Found Test Automation Framework Element Found Save successful path Old locator New locator Successful path Page state New locator Find New Element
  • 26. SelfHealingDriver findElement public WebElement findElement(By by) { if (by instanceof PageAwareBy) { try { Trying to find the element and save its path if success } catch (NoSuchElementException var5) { Find previous reference element path in storage Generate new best matches locator return healedLocator; } } else { return delegate.findElement(by); } }
  • 27. PageAwareBy extends By PageAwareBy(String pageName, By by, SelfHealingEngine engine) { this.pageName = pageName; this.by = by; this.engine = engine; } PageObject Page Locator New locator processor
  • 28. Functions supported • By() -> PageAwareBy() • @FindBy -> @PageAwareFindBy • WebDriver -> SelfHealingDriver • Iframe support • Actions support • Remote test run • Parallel test run • Works with Selenium wrappers like Selenide
  • 30. Intellij Idea plugin to make code updates HTML REPORT
  • 31. Intellij Idea plugin to make code updates PLUGIN HTML REPORT Update request { filename: MainPageWithFindBy lineNumber: 49 failedLocatorValue://input[@name='EMAIL’] healedLocatorValue: input.t186__input.js-tilda-rule.t-input }
  • 32. Intellij Idea plugin to make code updates TAF CODE BASE PLUGIN HTML REPORT Find locator and update Update request { filename: MainPageWithFindBy lineNumber: 49 failedLocatorValue://input[@name='EMAIL’] healedLocatorValue: input.t186__input.js-tilda-rule.t-input }
  • 34. 1. Declare custom EngineConfig with the path to store new locators. For example 'shaselenium' EngineConfig engineConfig = EngineConfig .custom() .setStorage(new FileSystemPathStorage(Paths.get("sha", "selenium"))) .build(); 2. Init driver instance of SelfHealingDriver SelfHealingDriver driver = new SelfHealingDriver(new ChromeDriver(), engineConfig); 3. Locate elements By buttonBy = PageAwareBy.by("MainPage", By.id(testButtonId)); Or @PageAwareFindBy(page="MainPage", findBy = @FindBy(id = "markup-generation-button")) WebElement buttonBy; 4. Interact with elements as usual driver.findElement(buttonBy).click(); Important! Do not delete data from the folder where files with new locators are stored. They are used to perform self-healing in next automation runs Important! "MainPage" is the name of the page to which the WebElement belongs
  • 36. Projects for pilot TAF based on Java + Selenium (WebDriver) Continuous UI updates UI generated by CMS
  • 37. Projects for pilot TAF based on Java + Selenium (WebDriver) Continuous UI updates UI generated by CMS >20 replies 6 Web projects for pilot 3 projects are still in focus
  • 38. 100% tests healed Healed localization testing
  • 39. Issues Test execution time increase Can’t heal single element from the list Probability of false/positive results Code not updated after healing
  • 41. Future plans Idea plugin Mobile support
  • 42. Idea Plugin ✅ @PageAwareFindBy ❌ PageAwareBy(..) ❌ Local declaring
  • 43. Thanks to Oleh Khailenko Lead Software Engineer Kostiantyn Morozkin Senior Software Engineer Nikolai Kobzev Senior Software Engineer Vladimir Moshkin SoftwareTest Automation Engineer