The document discusses visual testing (VT) for localization tests. It introduces VT as a way to automate visual, manual tests through the user interface using computer vision. The document outlines reasons for using VT, such as detecting visual defects that cannot be found through traditional functional testing. It also discusses challenges with the pesticide paradox and brittle selectors. The remainder of the document provides an agenda, introduction of the speaker, and demos of VT using Applitools.

1. S
Concur Technologies
a SAP Company
Visual Testing for Localization Tests
By Peter Kim
Reviewed by Damir Hasic
Dec. 12, 2015

2. Agenda
S Intro
S What is VT?
S Scaling Your Automation Framework
S Functional vs UI Testing
S Why do I need VT?
S When do we use it
S Demo I, II, III
S Three Design Concepts You Should Know
S Why you should use VT right now!
S Q&A

3. Peter Kim
S Senior Software Engineer at Concur (a SAP Company) with 10+ years in
start-ups and Fortune 10 companies
S Passion
S Creating dynamic test automation frameworks that scale
S Creating frameworks that generate executable code (run-time)
S Application/Data Modeling
S Software Design Patterns and Anti-patterns
S Current Interests
S Node.js
S React
S DevOps/CI/CD
S Load/Perf. Testing integrated with Func.
S Adv. Mobile Automation with Products (e.g. Telerik Test Studio)

4. Why QA?
Improving Quality

5. What is VT?

6. What is VT?
Testing applications as a human would, through the UI,
with your eyes.

7. What is VT?
Testing applications as a human would, through the UI,
with your eyes.
Automate your visual (manual) tests .. driving
reliability, scalability, and efficiency in your QA/
Testing resources and processes successfully.

8. Scaling Your Automation
Framework/Strategy
S Easy to refactor existing tests (test-scripts)
S Framework is designed to extend/leverage new technologies
without sacrificing or risking refactoring the existing test-ware
S Reuse of existing automated tests within tests
S Scalable execution across OS/browsers/versions …
S Simple test management (reports/artifacts)
S Simple, Fast, and Powerful

9. Scaling Your Automation
Framework/Strategy
S Easy to refactor existing tests (test-scripts)
Framework is written to support the browser, independent
of the web application under test.

10. Scaling Your Automation
Framework/Strategy
S Framework is designed to extend/leverage new technologies
without sacrificing or risking refactoring the existing test-ware
S Your functional test framework should be able to leverage a visual
test framework with little impact.
S Page Objects is an old design … leverage an application model
based design
S Designing/Implementing frameworks requires the same
disciplines and processes as software development.

11. Scaling Your Automation
Framework/Strategy
S Reuse of existing automated tests within tests
Support composition at the command level (DSL) to support
scalability and reuse. Remove the barriers that are created from
hard-coding and duplication (DRY vs WET) of test steps.
Create and define tests that are requirements driven.
E.g.
Describe “what” the test script should do, rather than writing the details on “how”
to do it.

12. Scaling Your Automation
Framework/Strategy
S Scalable execution across OS/browsers/versions …
S Leverage the power of the cloud for testing (e.g. Sauce Labs)
S Relieve your team from the overhead needed to manage your test
grid.
S Leverage parallel/concurrent test execution
S Strong, yet simple, process to conduct impact analysis

13. Scaling Your Automation
Framework/Strategy
S Simple test management (reports/artifacts)
S Understand why xUNIT/JUNIT/NUNIT frameworks, while
sufficient for unit testing, they’re designed for DEV – they limit
the reporting needs for functional testing conducted by QA.
S Create a test manager that supports run-time (dynamic)
reporting.
S Couple assertions to the scenario or feature (test cases/
scenarios)
S Couple the scenario or feature to a requirement.

14. Scaling Your Automation
Framework/Strategy
S Simple, Fast, and Powerful
You have test cases that need to be automated, why write “code”
when you can generate that same “code” via code-generators.
If you can describe your test steps, based on requirements, then your
framework can translate it to executable code (e.g. meta-data/
dynamic programming).
“I just wrote 10,000 lines of code in 2 minutes”

15. Scaling Your Automation
Framework/Strategy
S Easy to refactor existing tests (test-scripts)
S Framework is designed to extend/leverage new technologies
without sacrificing or risking refactoring the existing test-ware
S Reuse of existing automated tests within tests
S Scalable execution across OS/browsers/versions …
S Simple test management (reports/artifacts)
S Simple, Fast, and Powerful

16. Automation Challenges
An example without VT
S Requirements (User Stories)
Only admins can access manage users accounts page.
S Test cases (Scenarios)
Successful login, with role “admin”, returns users account page.
S Assertions
S Verify “//*[@id=‘update_user’]”
S Verify “css=div#change password a”
S Verify “//title[contains(text(), ‘Manage User’)]”
S Do you really want to pick a selector that’s overly resistant to brittleness for the sake
of uniqueness?

17. Automation Challenges

18. Automation Challenges
Defects that were not detected, but why did your
automated test pass?

19. Automation Challenges
Are you feeding or detecting your bugs?
Pesticide Paradox

20. Automation Challenges
Are you feeding or detecting your bugs?
Warning signs of Pesticide Paradox
“.. we have over 1500 automated
tests, and they all pass.”

21. Automation Challenges
Are you feeding or detecting your bugs?
Warning signs of Pesticide Paradox
“.. we have over 1500 automated
tests, and they all pass.”
How many validations (assertions) were used for testing this feature in the past?
Are we adding/updating assertions as the feature is modified?
Are the assertions written to favor ‘pass’?
How many assertions constitute appropriate coverage?
Is UI and functional testing any different? How about from you’re the POV of
your customers?
How long does it take to validate your test, from the UI? Test suites?
Can you test your entire web site’s UI? Across all localizations?
How much more can a human take from staring at a monitor?

22. Can you find all the differences?

23. .. see the differences?

24. Localization Tests

25. Localization Tests
S Text/Fonts

26. Localization Tests
S Text/Fonts
S Regional/Cultural

27. Localization Tests
S Text/Fonts
S Regional/Cultural
S Salutations
S “Mr. Peter Kim” vs “Mr. Kim, Peter”
S Dates
S Big-endian : (year, month, day)
S Little-endian (day, month, year)
S Middle-endian (month, day, year)
S Theme/Identity

28. Localization Tests
S Text/Fonts
S Regional/Cultural
S Images

29. Localization Tests
S Text/Fonts
S Regional/Cultural
S Images
S Audio/Video Resources (Voice translations)
S Resources (Attachments – e.g. PDF Files across multiple
languages)
S Themes (based on user’s brand/preferences)

30. Why you need automated VT?
Functional testing is conducted by targeting specific assertions, to
verify a requirement, however it’s limited to those specific
enumerated assertions – therefore, finding defects, especially during
regression testing, isn’t sufficient.

31. Why you need automated VT?
S Selenium/Webdriver …
S automation is primarily used for automated functional tests (e.g.
Inspection of HTML/DOM elements ..), not the rendered UI from
the browser.
What you receive may not be what you think is actually
rendered.
S insufficient to detect visual defects

32. Why you need automated VT?
S QA Testers must validate/verify ...
S Web browsers (e.g. rendering behaviors)
S OS
S Screen resolutions
S Devices (iOS/Android)

33. Why you need automated VT?
S QA Testers must validate/verify ...
S Web browsers (e.g. rendering behaviors)
S OS
S Screen resolutions
S Devices (iOS/Android)
S Now, imagine your team must test across dozens of
localizations as well.

34. Why you need automated VT?
S Leverage an automation framework which provides the
ability to execute visual tests quickly and scales …
S Coverage
S Layout, Size
S Colors, Fonts
S Images
S Content

35. Why you need automated VT?

36. Why you need automated VT?

37. When do we use VT?
S Unit Tests (DEV – UI/UX)
S Integrate (Pages/Sections)
S Acceptance Testing (Staging/PROD)
S Deploy to PROD

38. Demo I – Basic use
S Setup the environment variable APPLITOOLS_API_KEY
S Create a baseline for future VT tests
S Review/Accept the baseline from your Applitools account
S Execute ScoutUI
S Review and accept test results (Applitools account)

39. Demo I
Create a baseline for future VT
S Setup the environment variable APPLITOOLS_API_KEY

40. Demo I
Create a baseline for future VT
:Strategy:
S “Scout” the navigation/routes you need to hit target pages/forms for
your automated test.
S Modify the URL
S Navigation via links
S Expose content by user actions (e.g. mouse events/scrolling)
S Page access by filling out a form (partially or fully)
S Page access by submitting a form

41. Demo I
Create a baseline for future VT
:Create the test script:

42. Demo I
Run the test script
scoutui –config demo1.cfg
[–eyes]
[--host <url>]
[--sauce]
[–browser <type>]
[–viewport WidthxHeight]
–app <String> –title <String>
> scoutui –config demo1.cfg –eyes –browser firefox –viewport 800x600
–app Demo1 –title Webinar

43. Demo I
Create a baseline for future VT Tests
S Review the generated results to create the initial baseline
S Log in to your Applitools account and navigate to the test
results.
S Review the results and prep the baseline as needed
S Accept any modifications to the new baseline
S Save your changes

44. Demo II.
UI Regression Tests
S New release with localization updates
S Re-execute our UI regression tests
S Review the results and prep
S Accept or Reject all validations that detected differences
S Update and prime our new baseline as needed
S Adding demarcation points/regions with Eyes*
S Save your changes

45. Demo II.
UI Regression Tests
Update the rendered pages to
ensure we cause
visual validations to fail.

46. Demo III - @Concur
S As a QA/Test Engineer, you must validate the localization
updates for Japanese and French in less than a week.
However, you only know English and also have limited
knowledge of the web application.
S How will you do this?

47. Demo III - @Concur
Test like a scout, with a camera, and navigate to those pages
that were already ‘discovered’ by past scouts using their
“navigational maps”.

48. 3 Design Concepts You Should
Know
S Know Your Software Design Patterns
S Command
Plays nicely across multiple frameworks to ensure actions are well
managed and scalable.
S Composition – Manage and extend your framework and test-ware
S Composite – Promote re-use and clean object management
S Singleton – Centralize metrics and common objects/utils (Reports)
S Observer – Manage your dependencies

49. 3 Design Concepts You Should
Know!
S Your Page Object Design is outdated … use an application
model based design.
S Requirements driven testing
S Code Generators
S BDD
S Scalability in design and execution

50. 3 Design Concepts You Should
Know!
S Manage your assertions to be simple yet scalable.
E.g. Leverage Visual Tests in your framework, with an “on-off”
switch design, while leveraging a secured cloud for testing.
S Facilitate ease of adding more assertions into your functional
tests.
S Prevent Pesticide Paradox
S Improve coverage for you all regression test types
S Reduce the pains involved with managing a test grid/devices

51. Creating assertions should not be a time
consuming task

52. Reliability/Repeatability/Scalability

53. Recognize Outdated Automation
Designs/Strategies

54. Q&A
S References
S http://www.applitools.com
S http://www.saucelabs.com
S http://joecolantonio.com/testtalks/66-peter-kim-building-automation-frameworks-selenium/
S STP Conference Spring 2015 – Model Based Frameworks and Scalability
S Telerik Webinar – YouTube:Telerik Scaling Your Framework (2015)
S Contact me
S Peter.Kim@concur.com
S LinkedIn: https://www.linkedin.com/in/peterkim777
S Meet me at STPCon Spring 2016 Session Speaker – “If Eddie Van Halen was a QA/Test Engineer…”
S Work with me
S https://www.concur.com
S Thanks to ..
S Mosche Milman – VP of Operations@Applitools for technical/customer support
S Damir Hasic@Concur
S Darla Raitt@Concur
S David Lawrence@Concur
S NUI Team@Concur