Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Leveraging Visual Testing with Your Functional Tests


Published on

Designing and implementing (or selecting) the right automation strategy, for functional testing, with visual testing, can help your project with greater test coverage while improving test scalability

Published in: Software
  • Be the first to comment

  • Be the first to like this

Leveraging Visual Testing with Your Functional Tests

  1. 1. Peter Kim 2017
  2. 2. INTRO – H20Dragon Peter Kim Email:
  3. 3. Agenda ž  Share how visual testing benefited our functional test automation strategy ž  Share where visual testing successfully provided coverage to other test types ž  Share how to get started with using visual testing ž  Demos ž  Q&A
  4. 4. What is Visual Testing?
  5. 5. What is Visual Testing? ž  Verifies the UI/UX —  Test coverage of what was actually rendered on the UI device —  Test coverage for responsive app. behaviors Verifies as though you are using your eyes.
  6. 6. Automated Visual Testing
  7. 7. A need to change because .... ž  Catch defects earlier in the SDLC ž  Verify the UI/UX across target browsers/UI devices ž  Verify the UI/UX across different resolutions ž  Existing automation strategy didn’t include “visual” coverage. ž  Current process of asserting validations are too static ž  Brittle Selenium locators (e.g. XPATH) ž  Page Objects aren’t scaling quickly enough ž  Customers are reporting defects, in PROD, instead of us ž  Inability to scale functional (Regression) test coverage ž  Uncertainty of what type of assertions are needed ž  Lack domain knowledge in the app. ž  Test reports lack depth ž  Test reports need meaningful visuals ž  Customers feel that your UI/UX is as important as the functionality ž  QA team has limited resources ž  It’s not 2007 anymore, but your test automation design/strategy thinks it is. ž  You want to repurpose your functional tests for other test types – e.g. Localization / Accessibility
  8. 8. You need it because .. Although, we were unable to create assertions to catch potential defects, with visual testing, we now at least have the ability to report and share with others what has changed. Here, you’ve empowered others, or possibly held accountable, that they too were in a position to catch (or see) potential defects or comment on changes (whether intended or not per requirements/user story).
  9. 9. Automation with Selenium
  10. 10. Automation with Selenium ž  Challenges —  Takes time and resources to learn effective Selenium/WebDriver —  Selenium is an automation library. You’re responsible for creating and/or piecing together the framework, designs, and strategies that will leverage it. —  Poor test frameworks/design will mean brittle tests and will most likely hinder your ability to scale – you will be playing “catch-up” with the current versions/releases of your app.
  11. 11. Automation with Selenium ž  Challenges —  Assertions are typically “static”, based on the requirements/user stories/pre-canned queries. E.g. Limited number of assertions used to verify and validate your test case. This means, if your application behaves differently and none of your assertions cover that change .. you are now exposed to risks (defects, false confidence in your findings).
  12. 12. Are You Automating the “Expected”? “Obvious is the most dangerous word in mathematics.” - Eric Temple Bell (Mathematician) 1883-1960
  13. 13. Are You Automating the ‘Expected’?
  14. 14. Are You Automating the ‘Expected’? ž  Pesticide Paradox Over the course of time, your automated tests will diminish it’s ability to catch defects. Especially, the tests that seem to always pass.
  15. 15. Improving automation ž What was our criteria to advance our existing automation strategy?
  16. 16. Automation Requirements ž  Need a technical solution that plays well with Selenium/WebDriver since all our end-to-end tests are Selenium based. ž  Minimal impact, e.g. refactoring, of our existing automation test suites. ž  Avoid re-inventing designs based on old designs, thinking they’re new designs. ž  Be cautious of “best practices” that have been around since 2007. ž  Short and simple learning curve, with powerful results. ž  Technology/tool must be well supported ž  Technology/tool must scale and be secure (e.g. cloud, multiple device support) ž  Powerful and customizable test reports
  17. 17. Lessons Learned ž  .. Be cautious when investing in resources with strategies which ultimately repurpose the same old approach and processes. ž  Consider leveraging a new design/strategy ž  Consider leveraging proven technologies that integrate with your existing testware.
  18. 18. DEMO – Functional Automated Test Filters, for advanced search, returns correct set of Porsche 911s.
  19. 19. DEMO – Functional Automated Test Filters, for advanced search, returns correct set of Porsche 911s.
  20. 20. DEMO I. Create a baseline visual test The visual baseline will be used for validation of future visual test results.
  21. 21. DEMO I. Create a baseline visual test The visual baseline will be used for validation of future visual test results.
  22. 22. Demo I. Re-run the test. Visually compare the new run against the baseline
  23. 23. Demo I. Recap ž  Created a visual baseline ž  Execute end-to-end with visual coverage —  Specified matchType (e.g. Strict, Content, Layout) —  Specified target page/regions ž  Automated coverage scales with more assertions. ž  Easy to “see” visual results ž  Ability to manage/update assertion criteria
  24. 24. Demo II. Visual Defect
  25. 25. Demo II. Visual Defect Analysis
  26. 26. Demo II. View the results
  27. 27. What was actually demonstrated? ž  Little resources are needed to add test coverage (assertions) if you have a simple, yet powerful, ability to leverage visual testing. ž  Coverage is dramatically improved —  Context (text) —  Layout —  Graphics ž  Less brittleness in your automated Selenium tests, while decreasing coding efforts ž  Forgiveness of ones’ lack of insight of “what should be covered?” or “what’s the scope?” ž  UI (Visual) assertions can facilitate functional testing ž  Minimize pesticide paradox ž  Fred Flintstone has an awesome car!
  28. 28. Leveraging VT Localization Test Automation (Regression)
  29. 29. Leveraging VT ž  Localization Testing Ensuring that all targeted and/or served parties, regardless of language, nationality, and culture are equally and fairly respected and presented with the same information and content.
  30. 30. Localization Test Automation ž  Content ž  Images ž  Layout
  31. 31. Localization Test Automation ž  Content —  Headings —  Data formats ○  Date ○  Currency —  Translation Errors ž  Images ž  Layout
  32. 32. Localization Test Automation ž  Content —  Headings —  Data formats ○  Date ○  Currency —  Translation Errors ž  Images ž  Layout Get more details on automating your Localization tests at
  33. 33. Localization Test Auto
  34. 34. Tools –Applitools Eyes ž  Visual Test Automation —  Advanced Visual/Digital Analysis (AI) ○  Mitigates false positives (e.g. floating or shifting of elements) ○  Create/manage regions per inclusion/exclusion —  Applitools Account (It’s free) —  Applitools API Key —  Install the SDK (based on your programming language) for your Selenium tests —  Understanding resolutions – e.g. “viewport” Reference:
  35. 35. Summary ž  Functional testing can be improved by leveraging visual testing. ž  It’s simple to conduct Visual Test Automation E.g. Applitools Eyes with Selenium/WebDriver based framework ž  Heavy reliance, using the pattern of asserting the “obvious”, being the “master of the obvious”, or “happy path” is a great way to ship defects into PROD. ž  Run and validate your tests on the actual devices your end-users are using. ž  Run and validate your tests on the same resolutions (UI devices) based on the “rendered” pages (not just the DOM). ž  Automate responsive apps visually. ž  Create a plan to deal with Pesticide Paradox ž  Advance your test automation framework with new designs and strategies (it’s not 2007) – also, leverage new technologies that are proven to be reliable and scalable, while being simple yet powerful. ž  For QA/QE teams – leverage “CYA”
  36. 36. Thank You Peter Kim Email: