Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
BUILDING YOUR OWN NATIVE APP
VISUAL TESTING MODULE WITH APPIUM
Wim Selles, 

Sr. Solutions Architect @Sauce Labs
IT’S NOT ABOUT ME
! :@wswebcreation
" : https://github.com/wswebcreation
IDEA
IDEA
REQUIREMENTS
• Native App support
• Cross platform
• Integrate WebdriverIO
• Paid tools === trial first
• Support blockout...
BLOCKOUTS
Status-/navigation-/

home indicator bar
WHY …?
Animations Dynamic text
RESEARCH
VISUAL REGRESSION / TESTING FOR NATIVE APPS
iOSSnapshotTestCase
Nakal
sightr.io
screenshot-tests-for-android
RESULT
VISUAL REGRESSION / TESTING FOR NATIVE APPS
Requirements
screenshot-tests-

for-android
iOSSnapshot

TestCase
Nakal...
RESULT
VISUAL REGRESSION / TESTING FOR NATIVE APPS
Requirements
screenshot-tests-

for-android
iOSSnapshot

TestCase
Nakal...
RESULT
VISUAL REGRESSION / TESTING FOR NATIVE APPS
Requirements
screenshot-tests-

for-android
iOSSnapshot

TestCase
Nakal...
RESULT
VISUAL REGRESSION / TESTING FOR NATIVE APPS
Requirements
screenshot-tests-

for-android
iOSSnapshot

TestCase
Nakal...
RESULT
VISUAL REGRESSION / TESTING FOR NATIVE APPS
Requirements
screenshot-tests-

for-android
iOSSnapshot

TestCase
Nakal...
CONCLUSION
WHAT TO DO?
LOGIC
WHO SHOULD DO WHAT?
Requirements Module Comparison engine
Actively maintained - Yes
Environment dependencies No No
B...
LOGIC
WHO SHOULD DO WHAT?
Requirements Module Comparison engine
Actively maintained - Yes
Environment dependencies No No
B...
JAVASCRIPT PIXEL-BY-PIXEL LIBRARIES
js-imagediff
image-diff
Rembrandt.JS
LooksSamepixelmatch
node-opencv
img-diff-js
Pixel...
JAVASCRIPT PIXEL-BY-PIXEL LIBRARIES
js-imagediff
image-diff
Rembrandt.JS
LooksSamepixelmatch
node-opencv
img-diff-js
Pixel...
JAVASCRIPT PIXEL-BY-PIXEL LIBRARIES
js-imagediff
image-diff
Rembrandt.JS
LooksSamepixelmatch
node-opencv
img-diff-js
Pixel...
JAVASCRIPT PIXEL-BY-PIXEL LIBRARIES
js-imagediff
image-diff
Rembrandt.JS
LooksSamepixelmatch
node-opencv
img-diff-js
Pixel...
JAVASCRIPT PIXEL-BY-PIXEL LIBRARIES
js-imagediff
image-diff
Rembrandt.JS
LooksSamepixelmatch
node-opencv
img-diff-js
Pixel...
CONCLUSION
CONCLUSION
• saveScreen
WHAT SHOULD IT DO?
• saveScreen
• compareScreen
WHAT SHOULD IT DO?
• saveScreen
• compareScreen
• saveElement
WHAT SHOULD IT DO?
• saveScreen
• compareScreen
• saveElement
• compareElement
WHAT SHOULD IT DO?
SAVESCREEN
WHAT SHOULD IT DO?
COMPARESCREEN
WHERE COULD IT GO WRONG, WHERE COULD IT CAUSE FLAKINESS?
Navigation bar Home bar indicatorStatus bar
STATUS BAR
• Android and iOS differ in height
• iOS OS versions / devices differ
• Android OS versions / devices differ
• ...
DEMO
NAVIGATION BAR
• Android OS versions / devices differ
• Not each device has a tool/navigation bar
• Cross device solution
...
HOME BAR INDICATOR
● Currently 2 sizes
● The behaviour can be adjusted
IPHONE X SERIES
SAVEELEMENT
WHAT SHOULD IT DO?
SAVEELEMENT
WHAT SHOULD IT DO?
SAVEELEMENT
WHAT SHOULD IT DO?
SAVEELEMENT
EXTRA POWER
COMPAREELEMENT
● Comparing (options) the 

same as for compareScreen
● Element block-outs 

more complex
WHAT SHOULD IT DO?
USE CASE
React Native
0.54
React Native
0.57
USE CASE
React Native
0.54
React Native
0.57
USE CASE
DISCLAIMER
Cons
● Basic comparison
● Impact on project size
● Limited blockout options
● No dashboard
Pro’s
● Open Source
...
WANT TO USE IT?
https://github.com/wswebcreation/wdio-native-app-compare
WANT TO USE IT?
THANK YOU.
Building Your Own Native App Visual Testing Module with Appium
Building Your Own Native App Visual Testing Module with Appium
Building Your Own Native App Visual Testing Module with Appium
Building Your Own Native App Visual Testing Module with Appium
Building Your Own Native App Visual Testing Module with Appium
Building Your Own Native App Visual Testing Module with Appium
Building Your Own Native App Visual Testing Module with Appium
Building Your Own Native App Visual Testing Module with Appium
Building Your Own Native App Visual Testing Module with Appium
Building Your Own Native App Visual Testing Module with Appium
Building Your Own Native App Visual Testing Module with Appium
Building Your Own Native App Visual Testing Module with Appium
Upcoming SlideShare
Loading in …5
×

Building Your Own Native App Visual Testing Module with Appium

337 views

Published on

We all know that Appium can do (almost) everything you tell it to do, but in the end we always ask it to do the same thing: to automate our happy and/or our error flow. The result can give us the confirmation that the user can still buy a product, or verify if his bank account still has a positive balance.

But what if we need to know more than that: what if we also need to verify the layout of our app? The answer would be easy, as there are plenty of open-source or paid solutions that can help you with that. However, almost all of these solutions focus on web/hybrid apps, not on native iOS and or Android apps.

Wim faced this challenge when he was automating a React Native app for a customer. There was no image comparison tool that could do what he wanted it to do, so he rolled up his sleeves and started building one. In this talk, Wim will walk down the path he took and he will explain:

- Why he used ResembleJS as a core visual comparison solution
- The things he learned about Appium in the process (pro tips!)
- The differences between iOS and Android and how he managed to solve this in one cross-platform solution
- The pros and cons of the solution

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Building Your Own Native App Visual Testing Module with Appium

  1. 1. BUILDING YOUR OWN NATIVE APP VISUAL TESTING MODULE WITH APPIUM Wim Selles, 
 Sr. Solutions Architect @Sauce Labs
  2. 2. IT’S NOT ABOUT ME ! :@wswebcreation " : https://github.com/wswebcreation
  3. 3. IDEA
  4. 4. IDEA
  5. 5. REQUIREMENTS • Native App support • Cross platform • Integrate WebdriverIO • Paid tools === trial first • Support blockouts • Support element screenshots/bounding box FOR A VISUAL REGRESSION / TESTING SOLUTION
  6. 6. BLOCKOUTS Status-/navigation-/
 home indicator bar WHY …? Animations Dynamic text
  7. 7. RESEARCH VISUAL REGRESSION / TESTING FOR NATIVE APPS iOSSnapshotTestCase Nakal sightr.io screenshot-tests-for-android
  8. 8. RESULT VISUAL REGRESSION / TESTING FOR NATIVE APPS Requirements screenshot-tests-
 for-android iOSSnapshot
 TestCase Nakal sightr.io VisWiz Applitools Native apps support Yes Yes Yes Yes Yes Yes WebdriverIO support
 - JS
 - No extra env
 dependencies No No No No Yes Not for 
 native apps Trial? Open source Open source Open source Beta Yes Yes Image diff Yes, basic Yes, basic Yes Yes Yes, basic Yes, extensive Blockouts No No Yes and no No no Yes Element screenshots No No No No No No Bounding box No No No No No Yes
  9. 9. RESULT VISUAL REGRESSION / TESTING FOR NATIVE APPS Requirements screenshot-tests-
 for-android iOSSnapshot
 TestCase Nakal sightr.io VisWiz Applitools Native apps support Yes Yes Yes Yes Yes Yes WebdriverIO support
 - JS
 - No extra env
 dependencies No No No No Yes Not for 
 native apps Trial? Open source Open source Open source Beta Yes Yes Image diff Yes, basic Yes, basic Yes Yes Yes, basic Yes, extensive Blockouts No No Yes and no No no Yes Element screenshots No No No No No No Bounding box No No No No No Yes
  10. 10. RESULT VISUAL REGRESSION / TESTING FOR NATIVE APPS Requirements screenshot-tests-
 for-android iOSSnapshot
 TestCase Nakal sightr.io VisWiz Applitools Native apps support Yes Yes Yes Yes Yes Yes WebdriverIO support
 - JS
 - No extra env
 dependencies No No No No Yes Not for 
 native apps Trial? Open source Open source Open source Beta Yes Yes Image diff Yes, basic Yes, basic Yes Yes Yes, basic Yes, extensive Blockouts No No Yes and no No no Yes Element screenshots No No No No No No Bounding box No No No No No Yes
  11. 11. RESULT VISUAL REGRESSION / TESTING FOR NATIVE APPS Requirements screenshot-tests-
 for-android iOSSnapshot
 TestCase Nakal sightr.io VisWiz Applitools Native apps support Yes Yes Yes Yes Yes Yes WebdriverIO support
 - JS
 - No extra env
 dependencies No No No No Yes Not for 
 native apps Trial? Open source Open source Open source Beta Yes Yes Image diff Yes, basic Yes, basic Yes Yes Yes, basic Yes, extensive Blockouts No No Yes and no No no Yes Element screenshots No No No No No No Bounding box No No No No No Yes
  12. 12. RESULT VISUAL REGRESSION / TESTING FOR NATIVE APPS Requirements screenshot-tests-
 for-android iOSSnapshot
 TestCase Nakal sightr.io VisWiz Applitools Native apps support Yes Yes Yes Yes Yes Yes WebdriverIO support
 - JS
 - No extra env
 dependencies No No No No Yes Not for 
 native apps Trial? Open source Open source Open source Beta Yes Yes Image diff Yes, basic Yes, basic Yes Yes Yes, basic Yes, extensive Blockouts No No Yes and no No no Yes Element screenshots No No No No No No Bounding box No No No No No Yes ?
  13. 13. CONCLUSION WHAT TO DO?
  14. 14. LOGIC WHO SHOULD DO WHAT? Requirements Module Comparison engine Actively maintained - Yes Environment dependencies No No Blockout(s) Yes, determine and provide Yes Element screenshot Yes - Create baseline Yes - Screenshots per device Yes - Support compare options
 (for example antialiasing) Yes Yes
  15. 15. LOGIC WHO SHOULD DO WHAT? Requirements Module Comparison engine Actively maintained - Yes Environment dependencies No No Blockout(s) Yes, determine and provide Yes Element screenshot Yes - Create baseline Yes - Screenshots per device Yes - Support compare options
 (for example antialiasing) Yes Yes
  16. 16. JAVASCRIPT PIXEL-BY-PIXEL LIBRARIES js-imagediff image-diff Rembrandt.JS LooksSamepixelmatch node-opencv img-diff-js Pixel-Diff Resemble.JS
  17. 17. JAVASCRIPT PIXEL-BY-PIXEL LIBRARIES js-imagediff image-diff Rembrandt.JS LooksSamepixelmatch node-opencv img-diff-js Pixel-Diff Resemble.JS
  18. 18. JAVASCRIPT PIXEL-BY-PIXEL LIBRARIES js-imagediff image-diff Rembrandt.JS LooksSamepixelmatch node-opencv img-diff-js Pixel-Diff Resemble.JS
  19. 19. JAVASCRIPT PIXEL-BY-PIXEL LIBRARIES js-imagediff image-diff Rembrandt.JS LooksSamepixelmatch node-opencv img-diff-js Pixel-Diff Resemble.JS
  20. 20. JAVASCRIPT PIXEL-BY-PIXEL LIBRARIES js-imagediff image-diff Rembrandt.JS LooksSamepixelmatch node-opencv img-diff-js Pixel-Diff Resemble.JS
  21. 21. CONCLUSION
  22. 22. CONCLUSION
  23. 23. • saveScreen WHAT SHOULD IT DO?
  24. 24. • saveScreen • compareScreen WHAT SHOULD IT DO?
  25. 25. • saveScreen • compareScreen • saveElement WHAT SHOULD IT DO?
  26. 26. • saveScreen • compareScreen • saveElement • compareElement WHAT SHOULD IT DO?
  27. 27. SAVESCREEN WHAT SHOULD IT DO?
  28. 28. COMPARESCREEN WHERE COULD IT GO WRONG, WHERE COULD IT CAUSE FLAKINESS? Navigation bar Home bar indicatorStatus bar
  29. 29. STATUS BAR • Android and iOS differ in height • iOS OS versions / devices differ • Android OS versions / devices differ • Cross platform / device solution
  30. 30. DEMO
  31. 31. NAVIGATION BAR • Android OS versions / devices differ • Not each device has a tool/navigation bar • Cross device solution FOR ANDROID
  32. 32. HOME BAR INDICATOR ● Currently 2 sizes ● The behaviour can be adjusted IPHONE X SERIES
  33. 33. SAVEELEMENT WHAT SHOULD IT DO?
  34. 34. SAVEELEMENT WHAT SHOULD IT DO?
  35. 35. SAVEELEMENT WHAT SHOULD IT DO?
  36. 36. SAVEELEMENT EXTRA POWER
  37. 37. COMPAREELEMENT ● Comparing (options) the 
 same as for compareScreen ● Element block-outs 
 more complex WHAT SHOULD IT DO?
  38. 38. USE CASE React Native 0.54 React Native 0.57
  39. 39. USE CASE React Native 0.54 React Native 0.57
  40. 40. USE CASE
  41. 41. DISCLAIMER Cons ● Basic comparison ● Impact on project size ● Limited blockout options ● No dashboard Pro’s ● Open Source ● The (module) logic is easy ● Easy cross platform/device blockout logic driver.compareScreen( 'compareScreen-elementBlockOuts', { elementBlockOuts: [ // block out element 1 { element: LoginScreen.loginButton }, // block out element 2 with margin { element: LoginScreen.email, margin: 50}, ], } );
  42. 42. WANT TO USE IT? https://github.com/wswebcreation/wdio-native-app-compare
  43. 43. WANT TO USE IT?
  44. 44. THANK YOU.

×