SlideShare a Scribd company logo
1 of 47
Automating visual software testing
Adam Carmi
Co-Founder and CTO
Applitools
INTRODUCTION TO
AUTOMATED VISUAL
TESTING
Automating visual software testing
YOU CAN AND SHOULD
AUTOMATE YOUR
VISUAL TESTS!
Automating visual software testing
Automating visual software testing
AGENDA
• Why automated visual testing?
• Tools & Technology
• Where does it fit?
• Q & A
Automating visual software testing
WHAT IS VISUAL TESTING?
A quality assurance activity aimed to verify that a
Graphical User Interface appears correctly to users
Automating visual software testing
Automating visual software testing
Automating visual software testing
Automating visual software testing
WHY AUTOMATE?
The test matrix is too big to cover manually
• Web browsers
• Devices
• Operating systems
• Screen resolutions
• Responsive design
• L10n
• 3rd Party upgrades
Automating visual software testing
WHY AUTOMATE?
Mobile apps quality is critical
• Harder to roll back changes
• Can’t push daily
• Updates take battery and data
• Updates are not mandatory
• Higher quality bar
Automating visual software testing
WHY AUTOMATE?
Release cycles keep getting shorter
Automating visual software testing
AGENDA
• Why automated visual testing?
• Tools & Technology
• Where does it fit?
• Q & A
Automating visual software testing
HOW DOES IT WORK?
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
Automating visual software testing
DEMOHELLO WORLD
Automating visual software testing
HOW DOES IT WORK?
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
Automating visual software testing
Automating visual software testing
Automating visual software testing
Automating visual software testing
Automating visual software testing
Automating visual software testing
AND MORE…
• 1 pixel offsets in element positioning
• Dynamic content
• Moving elements
• Images of different size
• Performance
Automating visual software testing
Automating visual software testing
IMAGE COMPARISON ENGINES
Automating visual software testing
IMAGE MAGICK
• A command line tool for bitmap processing
• Fuzzing is used to eliminate slight color differences
• An error ratio is usually used to determine a match
$ compare –metric AE –fuzz 5% img1.png img2.png diff.png
2246
Automating visual software testing
THE JAVASCRIPT ENGINES
• Resemble.js
Pixel by pixel + error ratio + anti-aliasing
http://huddle.github.io/Resemble.js
• Blink-Diff
Pixel by pixel + error ratio + anti-aliasing + perceptual color distance computation
https://github.com/yahoo/blink-diff
• Looks-Same
Pixel by pixel + perceptual color distance computation + ignore Caret
https://github.com/gemini-testing/looks-same
Automating visual software testing
APPLITOOLS EYES
• Handles anti-aliasing, pixel offsets, color similarity, and image scaling
• Dynamic and moving content
• Compare images of different sizes
• No error ratio configuration required
• Validates full UI pages
• Fast!
• Supports layout / structure matching
Automating visual software testing
DEMOVISUAL LAYOUT MATCHING
Automating visual software testing
HOW DOES IT WORK?
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
Automating visual software testing
QUICK FEEDBACK TOOLS
MOTIVATION
• Get fast feedback on code changes
• Run tests locally in the background
CONS
• Partial coverage
• Chrome, IE?
• Limited navigation
• Stale browser versions
SETUP
• Render screenshots with a headless
browser
• PhatomJS, SlimerJS
• Configuration file driven tests
Automating visual software testing
VISUAL COVERAGE TOOLS
MOTIVATION
• Verify that the app / site looks right in
all real execution environments
• Test as many UI states as possible
• Execute many tests in parallel
CONS
• Requires test infrastructure
• Implement and maintain test code
SETUP
• Render screenshots using real
browsers on a variety of operating
systems and devices in parallel
• WebDriver, Grid
• Code driven tests
• WebDriver, DSL
Automating visual software testing
SAMPLE TOOLS
CODE / SCRIPT
● Needle (Python WD)
● Gemini (JS DSL)
● Selenium Visual Diff (Java WD)
● VisualCeption (PHP CodeCeption)
● Shoov (JS WebDriverIO)
● Vizregress (.NET WD)
● Rspec Page Regression (Capibara)
● Applitools Eyes (Appium, XCUI, Espresso)
● PhantomCSS (JS CasperJS)
● Specter (JS DSL)
● FBSnapshotTestCase (XCTest)
CONFIGURATION
● Viff
● CSS Visual Test
● Wraith
● Dpdxt
● Grunt PhotoBox
● Snap And Compare
● BackstopJS
● CSSCritic
● Kobold
● DiffCop
● SUCCSS
Automating visual software testing
HOW DOES IT WORK?
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
Automating visual software testing
REPORT DIFFERENCES
As files on the file system
Automating visual software testing
REPORT DIFFERENCES
As a gallery
Automating visual software testing
REPORT DIFFERENCES
In your CI build report
Automating visual software testing
HOW DOES IT WORK?
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
Automating visual software testing
UPDATE THE BASELINE
Gemini
Automating visual software testing
UI VERSION CONTROL
• Keep CI tests green
• Multiple app versions
• Revert to previous baselines
• A visual change log (documentation)
1 1 1
2
1
2
3 3
2
4
4
Automating visual software testing
Automating visual software testing
DEMOAUTOMATED MAINTENANCE
Automating visual software testing
AGENDA
• Why automated visual testing?
• Tools & Technology
• Where does it fit?
• Q & A
Automating visual software testing
WHERE DOES IT FIT?
CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION
Before committing / merging code:
• See how UI changes appear in other execution environments
• Make sure there are no unintended changes
Automating visual software testing
WHERE DOES IT FIT?
CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION
Visual testing of frontend components by frontend developers
Automating visual software testing
WHERE DOES IT FIT?
CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION
Regression, Localization, Accessibility, Cross browser/device, responsive
design testing and more…
All team members can review and maintain
A collaboration tool
Automating visual software testing
WHERE DOES IT FIT?
CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION
Validate your staging deployment using your production
deployment as a baseline
Automating visual software testing
WHERE DOES IT FIT?
CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION
No missing resources in production
No corruption due to 3rd party data, upgrades, hackers, etc.
Automating visual software testing
Adam Carmi (@carmiadam)
Co-Founder and CTO at Applitools
QUESTIONS?

More Related Content

What's hot

Advanced Visual Test Automation With Selenium
Advanced Visual Test Automation With SeleniumAdvanced Visual Test Automation With Selenium
Advanced Visual Test Automation With Seleniumadamcarmi
 
Sauce Labs+Applitools - Automated Visual Testing in the Cloud
Sauce Labs+Applitools - Automated Visual Testing in the CloudSauce Labs+Applitools - Automated Visual Testing in the Cloud
Sauce Labs+Applitools - Automated Visual Testing in the CloudSauce Labs
 
Advanced Automated Visual Testing for Mobile Apps
Advanced Automated Visual Testing for Mobile AppsAdvanced Automated Visual Testing for Mobile Apps
Advanced Automated Visual Testing for Mobile Appsadamcarmi
 
Advanced automated visual testing with Selenium
Advanced automated visual testing with SeleniumAdvanced automated visual testing with Selenium
Advanced automated visual testing with Seleniumadamcarmi
 
Visual regression with applitools eyes
Visual regression with applitools eyesVisual regression with applitools eyes
Visual regression with applitools eyesShama Ugale
 
Building Your Own Native App Visual Testing Module with Appium
Building Your Own Native App  Visual Testing Module with AppiumBuilding Your Own Native App  Visual Testing Module with Appium
Building Your Own Native App Visual Testing Module with AppiumWim Selles
 
Adam carmi
Adam carmiAdam carmi
Adam carmiCodeFest
 
Conference talk trials and triumphs of visual testing
Conference talk trials and triumphs of visual testingConference talk trials and triumphs of visual testing
Conference talk trials and triumphs of visual testingKunwardeep Bedi
 
Learnings from Hybrid App Testing Jijesh Mohan
Learnings from Hybrid App Testing Jijesh MohanLearnings from Hybrid App Testing Jijesh Mohan
Learnings from Hybrid App Testing Jijesh MohanvodQA
 
Visual AI Testing Using Applitools
Visual AI Testing Using ApplitoolsVisual AI Testing Using Applitools
Visual AI Testing Using ApplitoolsMikhail Laptev
 
To Deploy or Not-To-Deploy - decide using TTA's Trend & Failure Analysis
To Deploy or Not-To-Deploy - decide using TTA's Trend & Failure AnalysisTo Deploy or Not-To-Deploy - decide using TTA's Trend & Failure Analysis
To Deploy or Not-To-Deploy - decide using TTA's Trend & Failure AnalysisAnand Bagmar
 
Cross browser testing
Cross browser testingCross browser testing
Cross browser testingaralikatte
 
Practical ideas for getting the most out of your working environment
Practical ideas for getting the most out of your working environmentPractical ideas for getting the most out of your working environment
Practical ideas for getting the most out of your working environmentbluebirdtrans
 
JSFoo 2016 React Native Workshop
JSFoo 2016 React Native WorkshopJSFoo 2016 React Native Workshop
JSFoo 2016 React Native WorkshopChandra Mohan
 
Learnings from Hybrid App Testing - Jijesh Mohan
Learnings from Hybrid App Testing - Jijesh MohanLearnings from Hybrid App Testing - Jijesh Mohan
Learnings from Hybrid App Testing - Jijesh MohanThoughtworks
 
Hangman game - AI Powered and in Python
Hangman game - AI Powered and in PythonHangman game - AI Powered and in Python
Hangman game - AI Powered and in Pythonaiclub_slides
 

What's hot (18)

Advanced Visual Test Automation With Selenium
Advanced Visual Test Automation With SeleniumAdvanced Visual Test Automation With Selenium
Advanced Visual Test Automation With Selenium
 
Sauce Labs+Applitools - Automated Visual Testing in the Cloud
Sauce Labs+Applitools - Automated Visual Testing in the CloudSauce Labs+Applitools - Automated Visual Testing in the Cloud
Sauce Labs+Applitools - Automated Visual Testing in the Cloud
 
Advanced Automated Visual Testing for Mobile Apps
Advanced Automated Visual Testing for Mobile AppsAdvanced Automated Visual Testing for Mobile Apps
Advanced Automated Visual Testing for Mobile Apps
 
Advanced automated visual testing with Selenium
Advanced automated visual testing with SeleniumAdvanced automated visual testing with Selenium
Advanced automated visual testing with Selenium
 
Visual regression with applitools eyes
Visual regression with applitools eyesVisual regression with applitools eyes
Visual regression with applitools eyes
 
Building Your Own Native App Visual Testing Module with Appium
Building Your Own Native App  Visual Testing Module with AppiumBuilding Your Own Native App  Visual Testing Module with Appium
Building Your Own Native App Visual Testing Module with Appium
 
Adam carmi
Adam carmiAdam carmi
Adam carmi
 
Conference talk trials and triumphs of visual testing
Conference talk trials and triumphs of visual testingConference talk trials and triumphs of visual testing
Conference talk trials and triumphs of visual testing
 
Learnings from Hybrid App Testing Jijesh Mohan
Learnings from Hybrid App Testing Jijesh MohanLearnings from Hybrid App Testing Jijesh Mohan
Learnings from Hybrid App Testing Jijesh Mohan
 
AIA2018 - Janno Stern - Intro To Product Sprint
AIA2018 - Janno Stern - Intro To Product SprintAIA2018 - Janno Stern - Intro To Product Sprint
AIA2018 - Janno Stern - Intro To Product Sprint
 
Visual AI Testing Using Applitools
Visual AI Testing Using ApplitoolsVisual AI Testing Using Applitools
Visual AI Testing Using Applitools
 
To Deploy or Not-To-Deploy - decide using TTA's Trend & Failure Analysis
To Deploy or Not-To-Deploy - decide using TTA's Trend & Failure AnalysisTo Deploy or Not-To-Deploy - decide using TTA's Trend & Failure Analysis
To Deploy or Not-To-Deploy - decide using TTA's Trend & Failure Analysis
 
Sexy React Stack
Sexy React StackSexy React Stack
Sexy React Stack
 
Cross browser testing
Cross browser testingCross browser testing
Cross browser testing
 
Practical ideas for getting the most out of your working environment
Practical ideas for getting the most out of your working environmentPractical ideas for getting the most out of your working environment
Practical ideas for getting the most out of your working environment
 
JSFoo 2016 React Native Workshop
JSFoo 2016 React Native WorkshopJSFoo 2016 React Native Workshop
JSFoo 2016 React Native Workshop
 
Learnings from Hybrid App Testing - Jijesh Mohan
Learnings from Hybrid App Testing - Jijesh MohanLearnings from Hybrid App Testing - Jijesh Mohan
Learnings from Hybrid App Testing - Jijesh Mohan
 
Hangman game - AI Powered and in Python
Hangman game - AI Powered and in PythonHangman game - AI Powered and in Python
Hangman game - AI Powered and in Python
 

Viewers also liked

Introduction to Project Development using Visual Basic
Introduction to Project Development using Visual BasicIntroduction to Project Development using Visual Basic
Introduction to Project Development using Visual BasicMuralidharan Radhakrishnan
 
ADO Controls - Database Usage from Exploring MS Visual Basic 6.0 Book
ADO Controls - Database Usage from Exploring MS Visual Basic 6.0 BookADO Controls - Database Usage from Exploring MS Visual Basic 6.0 Book
ADO Controls - Database Usage from Exploring MS Visual Basic 6.0 BookMuralidharan Radhakrishnan
 
Performance Testing for Mobile Apps & Sites using Apache JMeter
Performance Testing for Mobile Apps & Sites using Apache JMeterPerformance Testing for Mobile Apps & Sites using Apache JMeter
Performance Testing for Mobile Apps & Sites using Apache JMeterAlon Girmonsky
 
Data base connectivity and flex grid in vb
Data base connectivity and flex grid in vbData base connectivity and flex grid in vb
Data base connectivity and flex grid in vbAmandeep Kaur
 
Visual basic 6
Visual basic 6Visual basic 6
Visual basic 6Spy Seat
 
Creating a quiz using visual basic 6
Creating a quiz using visual basic 6Creating a quiz using visual basic 6
Creating a quiz using visual basic 6Ella Marie Wico
 
Selenium Basics Tutorial
Selenium Basics TutorialSelenium Basics Tutorial
Selenium Basics TutorialClever Moe
 
Presentation on visual basic 6 (vb6)
Presentation on visual basic 6 (vb6)Presentation on visual basic 6 (vb6)
Presentation on visual basic 6 (vb6)pbarasia
 
Basic controls of Visual Basic 6.0
Basic controls of Visual Basic 6.0Basic controls of Visual Basic 6.0
Basic controls of Visual Basic 6.0Salim M
 
Visual basic ppt for tutorials computer
Visual basic ppt for tutorials computerVisual basic ppt for tutorials computer
Visual basic ppt for tutorials computersimran153
 

Viewers also liked (16)

Introduction to Project Development using Visual Basic
Introduction to Project Development using Visual BasicIntroduction to Project Development using Visual Basic
Introduction to Project Development using Visual Basic
 
Cucumber
CucumberCucumber
Cucumber
 
Vb 6.0 controls
Vb 6.0 controlsVb 6.0 controls
Vb 6.0 controls
 
Visusual basic
Visusual basicVisusual basic
Visusual basic
 
ADO Controls - Database Usage from Exploring MS Visual Basic 6.0 Book
ADO Controls - Database Usage from Exploring MS Visual Basic 6.0 BookADO Controls - Database Usage from Exploring MS Visual Basic 6.0 Book
ADO Controls - Database Usage from Exploring MS Visual Basic 6.0 Book
 
Vb unit t 1.1
Vb unit t 1.1Vb unit t 1.1
Vb unit t 1.1
 
Performance Testing for Mobile Apps & Sites using Apache JMeter
Performance Testing for Mobile Apps & Sites using Apache JMeterPerformance Testing for Mobile Apps & Sites using Apache JMeter
Performance Testing for Mobile Apps & Sites using Apache JMeter
 
Data base connectivity and flex grid in vb
Data base connectivity and flex grid in vbData base connectivity and flex grid in vb
Data base connectivity and flex grid in vb
 
Visual basic 6
Visual basic 6Visual basic 6
Visual basic 6
 
Creating a quiz using visual basic 6
Creating a quiz using visual basic 6Creating a quiz using visual basic 6
Creating a quiz using visual basic 6
 
Vb file
Vb fileVb file
Vb file
 
The Best Source Code VB
The Best Source Code VBThe Best Source Code VB
The Best Source Code VB
 
Selenium Basics Tutorial
Selenium Basics TutorialSelenium Basics Tutorial
Selenium Basics Tutorial
 
Presentation on visual basic 6 (vb6)
Presentation on visual basic 6 (vb6)Presentation on visual basic 6 (vb6)
Presentation on visual basic 6 (vb6)
 
Basic controls of Visual Basic 6.0
Basic controls of Visual Basic 6.0Basic controls of Visual Basic 6.0
Basic controls of Visual Basic 6.0
 
Visual basic ppt for tutorials computer
Visual basic ppt for tutorials computerVisual basic ppt for tutorials computer
Visual basic ppt for tutorials computer
 

Similar to Automating Visual Software Testing

Visual Testing: Quality and Speed -- by GIl Tayar, Applitools
Visual Testing: Quality and Speed -- by GIl Tayar, Applitools Visual Testing: Quality and Speed -- by GIl Tayar, Applitools
Visual Testing: Quality and Speed -- by GIl Tayar, Applitools Applitools
 
Colorful world-of-visual-automation-testing-latest
Colorful world-of-visual-automation-testing-latestColorful world-of-visual-automation-testing-latest
Colorful world-of-visual-automation-testing-latestOnur Baskirt
 
[webinar] Cutting-edge Functional UI Testing Techniques - w/ Adam Carmi
[webinar] Cutting-edge Functional UI Testing Techniques - w/ Adam Carmi[webinar] Cutting-edge Functional UI Testing Techniques - w/ Adam Carmi
[webinar] Cutting-edge Functional UI Testing Techniques - w/ Adam CarmiApplitools
 
Building Universal Servers (On-prem meets Azure PAAS)
Building Universal Servers (On-prem meets Azure PAAS)Building Universal Servers (On-prem meets Azure PAAS)
Building Universal Servers (On-prem meets Azure PAAS)adamcarmi
 
Automated Visual Testing That Doesn't Suck!
Automated Visual Testing That Doesn't Suck!Automated Visual Testing That Doesn't Suck!
Automated Visual Testing That Doesn't Suck!Applitools
 
Top 5 Features To Look for in a Codeless Automation Solution -- Presentation ...
Top 5 Features To Look for in a Codeless Automation Solution -- Presentation ...Top 5 Features To Look for in a Codeless Automation Solution -- Presentation ...
Top 5 Features To Look for in a Codeless Automation Solution -- Presentation ...Applitools
 
Web Automation Testing for developers?
Web Automation Testing for developers?Web Automation Testing for developers?
Web Automation Testing for developers?Victor Kushchenko
 
Triple Assurance: AI-Powered Test Automation in UI Design and Functionality
Triple Assurance: AI-Powered Test Automation in UI Design and FunctionalityTriple Assurance: AI-Powered Test Automation in UI Design and Functionality
Triple Assurance: AI-Powered Test Automation in UI Design and FunctionalityApplitools
 
"Software Quality in the Service of Innovation in the Insurance Industry"
"Software Quality in the Service of Innovation in the Insurance Industry""Software Quality in the Service of Innovation in the Insurance Industry"
"Software Quality in the Service of Innovation in the Insurance Industry"Applitools
 
Udvid din test portefølje med coded ui test og cloud load test
Udvid din test portefølje med coded ui test og cloud load testUdvid din test portefølje med coded ui test og cloud load test
Udvid din test portefølje med coded ui test og cloud load testPeter Lindberg
 
Selenium-based Visual Test Automation
Selenium-based Visual Test AutomationSelenium-based Visual Test Automation
Selenium-based Visual Test AutomationApplitools
 
Wso2 italia Open Break session #7 - Test di API e microservizi nelle architet...
Wso2 italia Open Break session #7 - Test di API e microservizi nelle architet...Wso2 italia Open Break session #7 - Test di API e microservizi nelle architet...
Wso2 italia Open Break session #7 - Test di API e microservizi nelle architet...Profesia Srl, Lynx Group
 
A Test Automation Platform Designed for the Future
A Test Automation Platform Designed for the FutureA Test Automation Platform Designed for the Future
A Test Automation Platform Designed for the FutureApplitools
 
DevOps for AI Apps
DevOps for AI AppsDevOps for AI Apps
DevOps for AI AppsRichin Jain
 
Best Practices in Mobile CI (webinar)
Best Practices in Mobile CI (webinar)Best Practices in Mobile CI (webinar)
Best Practices in Mobile CI (webinar)Sauce Labs
 
Mobile Automation with Appium
Mobile Automation with AppiumMobile Automation with Appium
Mobile Automation with AppiumManoj Kumar Kumar
 
Top 5 automation testing tools to gear up website development
Top 5 automation testing tools to gear up website developmentTop 5 automation testing tools to gear up website development
Top 5 automation testing tools to gear up website developmentBJIT Ltd
 
Microsoft DevOps Solution - DevOps
Microsoft DevOps Solution - DevOps  Microsoft DevOps Solution - DevOps
Microsoft DevOps Solution - DevOps Chetan Gordhan
 
Leveraging AI for Mobile App Testing on Real Devices | Applitools + Kobiton
Leveraging AI for Mobile App Testing on Real Devices | Applitools + KobitonLeveraging AI for Mobile App Testing on Real Devices | Applitools + Kobiton
Leveraging AI for Mobile App Testing on Real Devices | Applitools + KobitonApplitools
 

Similar to Automating Visual Software Testing (20)

Visual Testing: Quality and Speed -- by GIl Tayar, Applitools
Visual Testing: Quality and Speed -- by GIl Tayar, Applitools Visual Testing: Quality and Speed -- by GIl Tayar, Applitools
Visual Testing: Quality and Speed -- by GIl Tayar, Applitools
 
Colorful world-of-visual-automation-testing-latest
Colorful world-of-visual-automation-testing-latestColorful world-of-visual-automation-testing-latest
Colorful world-of-visual-automation-testing-latest
 
[webinar] Cutting-edge Functional UI Testing Techniques - w/ Adam Carmi
[webinar] Cutting-edge Functional UI Testing Techniques - w/ Adam Carmi[webinar] Cutting-edge Functional UI Testing Techniques - w/ Adam Carmi
[webinar] Cutting-edge Functional UI Testing Techniques - w/ Adam Carmi
 
Building Universal Servers (On-prem meets Azure PAAS)
Building Universal Servers (On-prem meets Azure PAAS)Building Universal Servers (On-prem meets Azure PAAS)
Building Universal Servers (On-prem meets Azure PAAS)
 
Automated Visual Testing That Doesn't Suck!
Automated Visual Testing That Doesn't Suck!Automated Visual Testing That Doesn't Suck!
Automated Visual Testing That Doesn't Suck!
 
Top 5 Features To Look for in a Codeless Automation Solution -- Presentation ...
Top 5 Features To Look for in a Codeless Automation Solution -- Presentation ...Top 5 Features To Look for in a Codeless Automation Solution -- Presentation ...
Top 5 Features To Look for in a Codeless Automation Solution -- Presentation ...
 
Web Automation Testing for developers?
Web Automation Testing for developers?Web Automation Testing for developers?
Web Automation Testing for developers?
 
Triple Assurance: AI-Powered Test Automation in UI Design and Functionality
Triple Assurance: AI-Powered Test Automation in UI Design and FunctionalityTriple Assurance: AI-Powered Test Automation in UI Design and Functionality
Triple Assurance: AI-Powered Test Automation in UI Design and Functionality
 
"Software Quality in the Service of Innovation in the Insurance Industry"
"Software Quality in the Service of Innovation in the Insurance Industry""Software Quality in the Service of Innovation in the Insurance Industry"
"Software Quality in the Service of Innovation in the Insurance Industry"
 
Udvid din test portefølje med coded ui test og cloud load test
Udvid din test portefølje med coded ui test og cloud load testUdvid din test portefølje med coded ui test og cloud load test
Udvid din test portefølje med coded ui test og cloud load test
 
Selenium-based Visual Test Automation
Selenium-based Visual Test AutomationSelenium-based Visual Test Automation
Selenium-based Visual Test Automation
 
DevOps in an Embedded World
DevOps in an Embedded WorldDevOps in an Embedded World
DevOps in an Embedded World
 
Wso2 italia Open Break session #7 - Test di API e microservizi nelle architet...
Wso2 italia Open Break session #7 - Test di API e microservizi nelle architet...Wso2 italia Open Break session #7 - Test di API e microservizi nelle architet...
Wso2 italia Open Break session #7 - Test di API e microservizi nelle architet...
 
A Test Automation Platform Designed for the Future
A Test Automation Platform Designed for the FutureA Test Automation Platform Designed for the Future
A Test Automation Platform Designed for the Future
 
DevOps for AI Apps
DevOps for AI AppsDevOps for AI Apps
DevOps for AI Apps
 
Best Practices in Mobile CI (webinar)
Best Practices in Mobile CI (webinar)Best Practices in Mobile CI (webinar)
Best Practices in Mobile CI (webinar)
 
Mobile Automation with Appium
Mobile Automation with AppiumMobile Automation with Appium
Mobile Automation with Appium
 
Top 5 automation testing tools to gear up website development
Top 5 automation testing tools to gear up website developmentTop 5 automation testing tools to gear up website development
Top 5 automation testing tools to gear up website development
 
Microsoft DevOps Solution - DevOps
Microsoft DevOps Solution - DevOps  Microsoft DevOps Solution - DevOps
Microsoft DevOps Solution - DevOps
 
Leveraging AI for Mobile App Testing on Real Devices | Applitools + Kobiton
Leveraging AI for Mobile App Testing on Real Devices | Applitools + KobitonLeveraging AI for Mobile App Testing on Real Devices | Applitools + Kobiton
Leveraging AI for Mobile App Testing on Real Devices | Applitools + Kobiton
 

Recently uploaded

(Genuine) Escort Service Lucknow | Starting ₹,5K To @25k with A/C 🧑🏽‍❤️‍🧑🏻 89...
(Genuine) Escort Service Lucknow | Starting ₹,5K To @25k with A/C 🧑🏽‍❤️‍🧑🏻 89...(Genuine) Escort Service Lucknow | Starting ₹,5K To @25k with A/C 🧑🏽‍❤️‍🧑🏻 89...
(Genuine) Escort Service Lucknow | Starting ₹,5K To @25k with A/C 🧑🏽‍❤️‍🧑🏻 89...gurkirankumar98700
 
W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...
W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...
W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...panagenda
 
5 Signs You Need a Fashion PLM Software.pdf
5 Signs You Need a Fashion PLM Software.pdf5 Signs You Need a Fashion PLM Software.pdf
5 Signs You Need a Fashion PLM Software.pdfWave PLM
 
Software Quality Assurance Interview Questions
Software Quality Assurance Interview QuestionsSoftware Quality Assurance Interview Questions
Software Quality Assurance Interview QuestionsArshad QA
 
Reassessing the Bedrock of Clinical Function Models: An Examination of Large ...
Reassessing the Bedrock of Clinical Function Models: An Examination of Large ...Reassessing the Bedrock of Clinical Function Models: An Examination of Large ...
Reassessing the Bedrock of Clinical Function Models: An Examination of Large ...harshavardhanraghave
 
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...ICS
 
Try MyIntelliAccount Cloud Accounting Software As A Service Solution Risk Fre...
Try MyIntelliAccount Cloud Accounting Software As A Service Solution Risk Fre...Try MyIntelliAccount Cloud Accounting Software As A Service Solution Risk Fre...
Try MyIntelliAccount Cloud Accounting Software As A Service Solution Risk Fre...MyIntelliSource, Inc.
 
Clustering techniques data mining book ....
Clustering techniques data mining book ....Clustering techniques data mining book ....
Clustering techniques data mining book ....ShaimaaMohamedGalal
 
How To Troubleshoot Collaboration Apps for the Modern Connected Worker
How To Troubleshoot Collaboration Apps for the Modern Connected WorkerHow To Troubleshoot Collaboration Apps for the Modern Connected Worker
How To Troubleshoot Collaboration Apps for the Modern Connected WorkerThousandEyes
 
SyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AI
SyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AISyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AI
SyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AIABDERRAOUF MEHENNI
 
Diamond Application Development Crafting Solutions with Precision
Diamond Application Development Crafting Solutions with PrecisionDiamond Application Development Crafting Solutions with Precision
Diamond Application Development Crafting Solutions with PrecisionSolGuruz
 
Advancing Engineering with AI through the Next Generation of Strategic Projec...
Advancing Engineering with AI through the Next Generation of Strategic Projec...Advancing Engineering with AI through the Next Generation of Strategic Projec...
Advancing Engineering with AI through the Next Generation of Strategic Projec...OnePlan Solutions
 
CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online ☂️
CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online  ☂️CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online  ☂️
CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online ☂️anilsa9823
 
Professional Resume Template for Software Developers
Professional Resume Template for Software DevelopersProfessional Resume Template for Software Developers
Professional Resume Template for Software DevelopersVinodh Ram
 
Hand gesture recognition PROJECT PPT.pptx
Hand gesture recognition PROJECT PPT.pptxHand gesture recognition PROJECT PPT.pptx
Hand gesture recognition PROJECT PPT.pptxbodapatigopi8531
 
Learn the Fundamentals of XCUITest Framework_ A Beginner's Guide.pdf
Learn the Fundamentals of XCUITest Framework_ A Beginner's Guide.pdfLearn the Fundamentals of XCUITest Framework_ A Beginner's Guide.pdf
Learn the Fundamentals of XCUITest Framework_ A Beginner's Guide.pdfkalichargn70th171
 
Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...
Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...
Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...Steffen Staab
 

Recently uploaded (20)

(Genuine) Escort Service Lucknow | Starting ₹,5K To @25k with A/C 🧑🏽‍❤️‍🧑🏻 89...
(Genuine) Escort Service Lucknow | Starting ₹,5K To @25k with A/C 🧑🏽‍❤️‍🧑🏻 89...(Genuine) Escort Service Lucknow | Starting ₹,5K To @25k with A/C 🧑🏽‍❤️‍🧑🏻 89...
(Genuine) Escort Service Lucknow | Starting ₹,5K To @25k with A/C 🧑🏽‍❤️‍🧑🏻 89...
 
W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...
W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...
W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...
 
Exploring iOS App Development: Simplifying the Process
Exploring iOS App Development: Simplifying the ProcessExploring iOS App Development: Simplifying the Process
Exploring iOS App Development: Simplifying the Process
 
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICECHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
 
5 Signs You Need a Fashion PLM Software.pdf
5 Signs You Need a Fashion PLM Software.pdf5 Signs You Need a Fashion PLM Software.pdf
5 Signs You Need a Fashion PLM Software.pdf
 
Software Quality Assurance Interview Questions
Software Quality Assurance Interview QuestionsSoftware Quality Assurance Interview Questions
Software Quality Assurance Interview Questions
 
Reassessing the Bedrock of Clinical Function Models: An Examination of Large ...
Reassessing the Bedrock of Clinical Function Models: An Examination of Large ...Reassessing the Bedrock of Clinical Function Models: An Examination of Large ...
Reassessing the Bedrock of Clinical Function Models: An Examination of Large ...
 
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...
 
Try MyIntelliAccount Cloud Accounting Software As A Service Solution Risk Fre...
Try MyIntelliAccount Cloud Accounting Software As A Service Solution Risk Fre...Try MyIntelliAccount Cloud Accounting Software As A Service Solution Risk Fre...
Try MyIntelliAccount Cloud Accounting Software As A Service Solution Risk Fre...
 
Clustering techniques data mining book ....
Clustering techniques data mining book ....Clustering techniques data mining book ....
Clustering techniques data mining book ....
 
How To Troubleshoot Collaboration Apps for the Modern Connected Worker
How To Troubleshoot Collaboration Apps for the Modern Connected WorkerHow To Troubleshoot Collaboration Apps for the Modern Connected Worker
How To Troubleshoot Collaboration Apps for the Modern Connected Worker
 
Vip Call Girls Noida ➡️ Delhi ➡️ 9999965857 No Advance 24HRS Live
Vip Call Girls Noida ➡️ Delhi ➡️ 9999965857 No Advance 24HRS LiveVip Call Girls Noida ➡️ Delhi ➡️ 9999965857 No Advance 24HRS Live
Vip Call Girls Noida ➡️ Delhi ➡️ 9999965857 No Advance 24HRS Live
 
SyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AI
SyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AISyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AI
SyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AI
 
Diamond Application Development Crafting Solutions with Precision
Diamond Application Development Crafting Solutions with PrecisionDiamond Application Development Crafting Solutions with Precision
Diamond Application Development Crafting Solutions with Precision
 
Advancing Engineering with AI through the Next Generation of Strategic Projec...
Advancing Engineering with AI through the Next Generation of Strategic Projec...Advancing Engineering with AI through the Next Generation of Strategic Projec...
Advancing Engineering with AI through the Next Generation of Strategic Projec...
 
CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online ☂️
CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online  ☂️CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online  ☂️
CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online ☂️
 
Professional Resume Template for Software Developers
Professional Resume Template for Software DevelopersProfessional Resume Template for Software Developers
Professional Resume Template for Software Developers
 
Hand gesture recognition PROJECT PPT.pptx
Hand gesture recognition PROJECT PPT.pptxHand gesture recognition PROJECT PPT.pptx
Hand gesture recognition PROJECT PPT.pptx
 
Learn the Fundamentals of XCUITest Framework_ A Beginner's Guide.pdf
Learn the Fundamentals of XCUITest Framework_ A Beginner's Guide.pdfLearn the Fundamentals of XCUITest Framework_ A Beginner's Guide.pdf
Learn the Fundamentals of XCUITest Framework_ A Beginner's Guide.pdf
 
Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...
Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...
Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...
 

Automating Visual Software Testing

  • 1. Automating visual software testing Adam Carmi Co-Founder and CTO Applitools INTRODUCTION TO AUTOMATED VISUAL TESTING
  • 2. Automating visual software testing YOU CAN AND SHOULD AUTOMATE YOUR VISUAL TESTS!
  • 4. Automating visual software testing AGENDA • Why automated visual testing? • Tools & Technology • Where does it fit? • Q & A
  • 5. Automating visual software testing WHAT IS VISUAL TESTING? A quality assurance activity aimed to verify that a Graphical User Interface appears correctly to users
  • 9. Automating visual software testing WHY AUTOMATE? The test matrix is too big to cover manually • Web browsers • Devices • Operating systems • Screen resolutions • Responsive design • L10n • 3rd Party upgrades
  • 10. Automating visual software testing WHY AUTOMATE? Mobile apps quality is critical • Harder to roll back changes • Can’t push daily • Updates take battery and data • Updates are not mandatory • Higher quality bar
  • 11. Automating visual software testing WHY AUTOMATE? Release cycles keep getting shorter
  • 12. Automating visual software testing AGENDA • Why automated visual testing? • Tools & Technology • Where does it fit? • Q & A
  • 13. Automating visual software testing HOW DOES IT WORK? Drive the AUT and take screenshots Compare screenshots with baseline images Report differences Update the baseline
  • 14. Automating visual software testing DEMOHELLO WORLD
  • 15. Automating visual software testing HOW DOES IT WORK? Drive the AUT and take screenshots Compare screenshots with baseline images Report differences Update the baseline
  • 21. Automating visual software testing AND MORE… • 1 pixel offsets in element positioning • Dynamic content • Moving elements • Images of different size • Performance
  • 23. Automating visual software testing IMAGE COMPARISON ENGINES
  • 24. Automating visual software testing IMAGE MAGICK • A command line tool for bitmap processing • Fuzzing is used to eliminate slight color differences • An error ratio is usually used to determine a match $ compare –metric AE –fuzz 5% img1.png img2.png diff.png 2246
  • 25. Automating visual software testing THE JAVASCRIPT ENGINES • Resemble.js Pixel by pixel + error ratio + anti-aliasing http://huddle.github.io/Resemble.js • Blink-Diff Pixel by pixel + error ratio + anti-aliasing + perceptual color distance computation https://github.com/yahoo/blink-diff • Looks-Same Pixel by pixel + perceptual color distance computation + ignore Caret https://github.com/gemini-testing/looks-same
  • 26. Automating visual software testing APPLITOOLS EYES • Handles anti-aliasing, pixel offsets, color similarity, and image scaling • Dynamic and moving content • Compare images of different sizes • No error ratio configuration required • Validates full UI pages • Fast! • Supports layout / structure matching
  • 27. Automating visual software testing DEMOVISUAL LAYOUT MATCHING
  • 28. Automating visual software testing HOW DOES IT WORK? Drive the AUT and take screenshots Compare screenshots with baseline images Report differences Update the baseline
  • 29. Automating visual software testing QUICK FEEDBACK TOOLS MOTIVATION • Get fast feedback on code changes • Run tests locally in the background CONS • Partial coverage • Chrome, IE? • Limited navigation • Stale browser versions SETUP • Render screenshots with a headless browser • PhatomJS, SlimerJS • Configuration file driven tests
  • 30. Automating visual software testing VISUAL COVERAGE TOOLS MOTIVATION • Verify that the app / site looks right in all real execution environments • Test as many UI states as possible • Execute many tests in parallel CONS • Requires test infrastructure • Implement and maintain test code SETUP • Render screenshots using real browsers on a variety of operating systems and devices in parallel • WebDriver, Grid • Code driven tests • WebDriver, DSL
  • 31. Automating visual software testing SAMPLE TOOLS CODE / SCRIPT ● Needle (Python WD) ● Gemini (JS DSL) ● Selenium Visual Diff (Java WD) ● VisualCeption (PHP CodeCeption) ● Shoov (JS WebDriverIO) ● Vizregress (.NET WD) ● Rspec Page Regression (Capibara) ● Applitools Eyes (Appium, XCUI, Espresso) ● PhantomCSS (JS CasperJS) ● Specter (JS DSL) ● FBSnapshotTestCase (XCTest) CONFIGURATION ● Viff ● CSS Visual Test ● Wraith ● Dpdxt ● Grunt PhotoBox ● Snap And Compare ● BackstopJS ● CSSCritic ● Kobold ● DiffCop ● SUCCSS
  • 32. Automating visual software testing HOW DOES IT WORK? Drive the AUT and take screenshots Compare screenshots with baseline images Report differences Update the baseline
  • 33. Automating visual software testing REPORT DIFFERENCES As files on the file system
  • 34. Automating visual software testing REPORT DIFFERENCES As a gallery
  • 35. Automating visual software testing REPORT DIFFERENCES In your CI build report
  • 36. Automating visual software testing HOW DOES IT WORK? Drive the AUT and take screenshots Compare screenshots with baseline images Report differences Update the baseline
  • 37. Automating visual software testing UPDATE THE BASELINE Gemini
  • 38. Automating visual software testing UI VERSION CONTROL • Keep CI tests green • Multiple app versions • Revert to previous baselines • A visual change log (documentation) 1 1 1 2 1 2 3 3 2 4 4
  • 40. Automating visual software testing DEMOAUTOMATED MAINTENANCE
  • 41. Automating visual software testing AGENDA • Why automated visual testing? • Tools & Technology • Where does it fit? • Q & A
  • 42. Automating visual software testing WHERE DOES IT FIT? CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION Before committing / merging code: • See how UI changes appear in other execution environments • Make sure there are no unintended changes
  • 43. Automating visual software testing WHERE DOES IT FIT? CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION Visual testing of frontend components by frontend developers
  • 44. Automating visual software testing WHERE DOES IT FIT? CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION Regression, Localization, Accessibility, Cross browser/device, responsive design testing and more… All team members can review and maintain A collaboration tool
  • 45. Automating visual software testing WHERE DOES IT FIT? CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION Validate your staging deployment using your production deployment as a baseline
  • 46. Automating visual software testing WHERE DOES IT FIT? CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION No missing resources in production No corruption due to 3rd party data, upgrades, hackers, etc.
  • 47. Automating visual software testing Adam Carmi (@carmiadam) Co-Founder and CTO at Applitools QUESTIONS?

Editor's Notes

  1. Components, Code review, Developers, Designers
  2. Components, Code review, Developers, Designers