Visual Automation Framework via Screenshot Comparison powered by Applitools

1. Visual Automation
Framework via
Screenshot Comparison
Mek Srunyu Stittri
Apr 15 2015
Powered by Applitools

2. Visual Automation Testing
What is Visual Test?
“A quality assurance activity aimed to
verify that a Graphical User Interface
appears correctly to users”

3. This
Credit : image from Adam Carmi, co-founder Applitools
A Visual Bug

4. Or this
A Visual Bug

5. Data Layer
Backend
Front end
Top of the stack checkpointing.
Requires the least amount of investment but
huge returns. Visually Perfect
Why Automate Visual Testing

6. Credit : image from Adam Carmi, co-founder Applitools
Why Automate Visual Testing

7. THE TEST MATRIX IS HUGE..
■ Web browsers
■ Operating systems
■ Screen resolutions
■ Responsive designs
■ L10n
Why Automate Visual Testing

8. Many are already doing it (and sharing)…
Credit : image from Adam Carmi, co-founder Applitools
Why Automate Visual Testing

9. Well, what are we waiting for..

10. Credit : image from Adam Carmi, co-founder Applitools
Challenges

11. If it’s so important.
How come it’s not
widely implemented ?
Its not that simple..
Credit : image from Adam Carmi, co-founder Applitools
Challenges

12. Credit : image from Adam Carmi, co-founder Applitools
Anti Aliasing

13. Credit : image from Adam Carmi, co-founder Applitools
Anti Aliasing

14. Credit : image from Adam Carmi, co-founder Applitools
Brightness

15. Credit : image from Adam Carmi, co-founder Applitools
Brightness

16. Credit : image from Adam Carmi, co-founder Applitools
Sub pixel shifts

17. Credit : image from Adam Carmi, co-founder Applitools
Sub pixel shifts

18. ■ Dynamic content
■ 1 pixel offsets in element
positioning
■ Moving elements
■ Images of different size
QE
Credit : image from Adam Carmi, co-founder Applitools
And more...

19. Introducing - APPLITOOLS, A specialized image processing stack designed
to compare computer generated UI images
Handles..
1. Anti-aliasing
2. Partial and full pixel offsets
3. Images of different size
4. Dynamic content
5. Moving elements
6. Layout matching
How did we do this

20. ● Exact - Only windows that are exact, pixel based, replicas of each other will be considered a success.
● Strict - Taking into consideration both layout and content changes while using Applitools Eyes
smart image analysis tools to ignore minor changes that might fail a test when using pixel based
comparison.
● Content - Only identify content differences while ignoring layout based
● Layout - Applitools Eyes will only identify layout differences while ignoring content differences.
Flexible Verification settings built-in

21. STRICT - Same image : Passed

22. CONTENT - Inverted color : Passed

23. STRICT - Inverted color : Failed

24. CONTENT - Actual Bug

25. Testdata (BaseLine) Creation Flow
1. Identify a Panel and create a new Applitools UI testcase to navigate to the Panel under test, do UI
interactions and add Visual Checkpoints
2. Run the test once to generate the BaseLine
3. Go to Applitools to approve the BaseLine and add customizations as necessary
Test Execution Flow
1. Run the test
2. Analyze the test result on differences reported
Workflow

26. Demo

27. Jersey dependency conflicts
■ http://support.applitools.com/customer/portal/questions/9866576-nosuchmethod-error
Integration into existing Framework with our own Implementation of
WebDriver
Applitools API requires passing in the actual Selenium WebDriver instance
■ http://support.applitools.com/customer/portal/questions/9866542-using-own-implementation-of-webdriver
■ A new testbase ApplitoolsTestBase.java is also created for all new validation tests. More in wiki
More Groovy dependency conflicts with the latest RELEASE version
■ http://support.applitools.com/customer/portal/questions/11629509-dependency-conflict-in-the-latest-release-version-getting-
error-when-initgremlingroovyscriptengine-
Challenges

28. public class ApplitoolsTestBase extends TestBase {
protected Eyes eyes;
/**
* Instantiates new {@link Eyes} object and sets Applitools Key and Testname from TestNG injection
*/
@BeforeMethod
public void setupTestMethod(Method method) {
super.setupTestMethod(method);
this.eyes = new Eyes();
eyes.setApiKey(APPLITOOLS_API_KEY);
String testPreFix = method.getDeclaringClass().getCanonicalName();
// shorten down com.mekdev.qe to c.m.q
testPreFix = testPreFix.replace("com.mekdev.qe", "c.m.q");
// Instantiate new driver and set application name and testname
WebDriver driver = eyes.open(getWebDriver(), "Visual Test", testPreFix + "::"+ method.getName());
setWebDriver(driver);
}
}
New Applitools specific TestBase

29. Test Example
@Test
public void testApplitools() throws Exception {
eyes.setMatchLevel(MatchLevel.CONTENT);
// Start the UI portion of the test
Navigation navigator = new Navigation.Builder()
.username(Constants.ADMIN).password(Constants.PASSWORD)
.build();
navigator.run();
LoginPage loginPage = WebDriver.update(navigator, LoginPage.class);
// Click Run
loginPage.clickSignup();
// Check screen #1
loginPage.fillInformation("John doe","password");
eyes.checkWindow("Signup Page");
// Check screen #2
loginPage.clickCancel();
eyes.checkWindow("Back to Login selection");
// finish the test
eyes.close();
loginPage.close();
}

30. Cross Browser / OS tests using the same BaseLine Data
■ Use - eyes.setBaselineName("baselineName");
Evaluate feasibility and stability of Screen Regioning Capture vs whole screen
■ Via - eyes.checkRegion(selector, matchTimeout, "windowName");
Expand to Different Screen Resolutions
■ Requires some tweaking to the framework to specify viewport resolution at the test level -
driver = eyes.open(driver, appName, testName, viewportSize);
Next Phases

31. Q & A
Special Thanks
Adam Carmi - VP Engineering, Co-Founder at Applitools
Adam Carmi: Automated Visual Testing that Doesn't Suck!
http://www.meetup.com/SeleniumSanJose/events/209029032/ - November 17, 2014
http://www.slideshare.net/adamcarmi/visual-test-automation-that-does-not-suck - Adam’s original
slide deck from Selenium Meetup
Moshe Milman - VP Operations, Co-Founder at Applitools
24x7 Continuous Support, late night Join.me sessions from Tel Aviv