Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Selenium Based Visual Test Automation

3,900 views

Published on

Slides from my talk "Selenium Based Visual Test Automation" at the Selenium Israel Meetup on Dec 1st, 2014.

Published in: Software
  • Be the first to comment

Selenium Based Visual Test Automation

  1. 1. SELENIUM BASED VISUAL TEST AUTOMATION Adam Carmi (@carmiadam) Co-Founder and VP R&D at Applitools adam.carmi@applitools.com
  2. 2. YOU CAN AND SHOULD AUTOMATE YOUR VISUAL TESTS!
  3. 3. AGENDA  Why automated visual testing?  Tools & Technology  Where does it fit?
  4. 4. WHAT IS VISUAL TESTING? A quality assurance activity aimed to verify that a Graphical User Interface appears correctly to users
  5. 5. A VISUAL BUG
  6. 6. AND ANOTHER…
  7. 7. WHY SHOULD IT BE AUTOMATED? THE TEST MATRIX IS TOO BIG TO APPROXIMATE MANUALLY  Web browsers  Operating systems  Screen resolutions  Responsive designs  L10n
  8. 8. WHY SHOULD IT BE AUTOMATED?
  9. 9. WHY SHOULD IT BE AUTOMATED? MANUAL VISUAL TESTING IS ERROR PRONE
  10. 10. WHY SHOULD IT BE AUTOMATED? Many are already doing it (and sharing)… PhantomCSS Fighting Layout Bugs CSS Critc Wraith Needle Grunt PhotoBox dpdxt WebdriverCSS Eyes Huxley FBSnapshotTestCase Selenium Visual Diff Gemini VisualCeption Specter Snap And Compare
  11. 11. AGENDA  Why automated visual testing?  Tools & Technology  Where does it fit?
  12. 12. THE WORKFLOW Drive the AUT and take screenshots Compare screenshots with baseline images Report differences Update the baseline
  13. 13. DEMO https://github.com/webdriverio/webdrivercss
  14. 14. THE WORKFLOW Drive the AUT and take screenshots Compare screenshots with baseline images Report differences Update the baseline
  15. 15. DRIVING AND CAPTURING OTHER PhantomCSS (CasperJS) Fighting Layout Bugs (Java) Wraith (URLs) CSS Critic (URLs) Needle (Python) dpdxt (URLs) Grunt PhotoBox (URLs) WebdriverCSS (JS) Eyes (All) Huxley (RP) FBSnapshotTestCase Selenium Visual Diff (Java) VisualCeption (PHP) Specter (JS) Wraith-Selenium (Ruby) Snap And Compare (URLs) Eyes (All) Gemini (JS) Gemini (JS)
  16. 16. SCREENSHOTS  Real browsers?  Full page?  Frames?  Regions?  Viewport size?  Page stabilization?  Page preparation?
  17. 17. THE WORKFLOW Drive the AUT and take screenshots Compare screenshots with baseline images Report differences Update the baseline
  18. 18. WHY NOT? FALSE POSITIVES
  19. 19. ANTI-ALIASING?
  20. 20. ANTI ALIASING 1/2
  21. 21. ANTI ALIASING 2/2
  22. 22. BRIGHTNESS 1/2
  23. 23. BRIGHTNESS 2/2
  24. 24. SUB PIXEL SHIFTS 1/2
  25. 25. SUB PIXEL SHIFTS 2/2
  26. 26. SUB PIXEL SHIFTS 1/2
  27. 27. SUB PIXEL SHIFTS 2/2
  28. 28. ANDMORE…  1 pixel offsets in element positioning  Dynamic content Moving elements  Images of different size
  29. 29. Image Comparison APIs
  30. 30. ImageMagick  A powerful command line tool for image processing.  APIs are available for most programming languages.  Fuzzing is used to eliminate slight color differences  An error ratio is usually used to determine a match $ compare –metric AE –fuzz 5% img1.png img2.png diff.png 2246
  31. 31. Resemble.js  An image comparison tool implemented in Javascript  Used by PhantomCSS and other tools.  Good antialiasing support  An error ratio is usually used to determine a match  http://huddle.github.io/Resemble.js/
  32. 32. Applitools Eyes  A specialized image processing stack designed to compare computer generated UI images  Anti-aliasing  Partial and full pixel offsets  Images of different size  Dynamic content  Moving elements  Layout matching  Available as a cloud service
  33. 33. DEMO
  34. 34. THE WORKFLOW Drive the AUT and take screenshots Compare screenshots with baseline images Report differences Update the baseline
  35. 35. REPORT DIFFERENCES As files on the file system (combined with source control)
  36. 36. REPORT DIFFERENCES As a Gallery (example from Selenium Visual Diff)
  37. 37. THE WORKFLOW Drive the AUT and take screenshots Compare screenshots with baseline images Report differences Update the baseline
  38. 38. UPDATE THE BASELINE  Rename or commit individual image files  Overwrite mode
  39. 39. UPDATE THE BASELINE  GUI (Gemini GUI)  Automatic maintenance (demo)
  40. 40. AGENDA  Why automated visual testing?  Tools & Technology  Where does it fit?
  41. 41. WHERE DOES IT FIT? • Component s • Code review • Developers • Designers • QA
  42. 42. WHERE DOES IT FIT? • Pages • Page sections • Developers • Designers • QA • Others
  43. 43. WHERE DOES IT FIT? • Staging vs. Production • Ops • QA
  44. 44. WHERE DOES IT FIT? • Monitoring • Ops • QA
  45. 45. QUESTIONS?
  46. 46. THANK YOU Adam Carmi (@carmiadam) Co-Founder and VP R&D at Applitools adam.carmi@applitools.com

×