SlideShare a Scribd company logo
1 of 117
Download to read offline
IMPLEMENTING A VISUAL 
CSS TESTING FRAMEWORK 
@jessicard 
hi everyone! thanks for having me here 
today i’m going to talk about implementing a visual css testing framework
using 
AUTOMATIC 
screenshot 
COMPARISON 
to catch style 
REGRESSIONS 
using automatic screenshot comparison to catch style regressions
@jessicard 
my name is jessicard on the internet, and jessica in real life
i work at a company called bugsnag here in san francisco 
bugsnag is an exception monitoring tool 
i’m a software engineer there, working primarily in ruby and javascript 
but our stack includes lots of languages, including node and go
(we’re hiring!) 
we are currently hiring 
so please get in touch if you’re interested in working on developer tools at a small company!
IMPLEMENTING A VISUAL 
CSS TESTING FRAMEWORK 
so, back to this whole “implementing a visual css testing framework” thing 
what am i even talking about?
well, at bugsnag, we decided we wanted a way to automatically take two screenshots 
of our website, at different times in the app
master feature 
for example, since we use git, lets say we had a feature branch that we just pushed a commit to, 
we’d want to take a screenshot of what our homepage looks like on that branch 
and a screenshot of how our homepage looks on production, or whats currently running on master
with those screenshots, we’d want to spot those differences
and produce some sort of diff image that highlighted the differences 
why would we want to do this, you might ask? well,
as we all know, writing, reading, and code reviewing CSS can be really intense
 
 
 
 
 
 
 
 
 
 
 
 
and even more intense to refactor 
at bugsnag, we decided to do a huge organizational and code-style refactor of our CSS 
and we wanted a way to test that our site looked the same, despite changing all of the code
unfortunately, as you can tell, that didn’t always work out for us 
we went through many iterations of refactoring, 
and we realized we needed a tool to help us test our pages automatically
does this 
EXIST? 
so, we went on the hunt for a way to test our CSS 
we wanted to know if there was a tool already built that did what we wanted 
without knowing exactly what it was that we wanted out of this framework
huxley 
we first stumbled upon one of facebook’s open source libraries, huxley
“Watches you browse, takes screenshots, 
tells you when they change” 
in huxley’s readme, it says 
“watches you browse, takes screenshots, tells you when they change” 
which sounds amazing!
but… i had noticed that it hadn’t been updated in over a year 
that wasn’t promising, but i decided to give it a shot anyway
 
after a good solid day of fiddling around with huxley, it ended up being a bit too buggy 
it would have random failures, randomly wouldn’t take screenshots 
and i started realizing it wasn’t exactly what i was looking for in a css testing tool
Huxley 
Phantom CSS 
GhostStory 
Cactus 
Needle 
CSSCritic 
fighting-layout-bugs 
sikuli 
Mogo 
Quixote 
i kept looking, and it turns out there are a lot frameworks that do some sort of css testing 
they work in all different ways - some take screenshots, some aren’t even visual 
i’d definitely recommend checking some of these out to see if they fit what you’re doing in your apps before building your own
WHAT DO I WANT? 
but i really started thinking about it 
what was i looking for in a CSS testing framework? 
what would fit the way bugsnag is built best? 
i decided i wanted a visual way to test my CSS, with screenshots 
rather than writing out tests describing the visuals
this is where i need a disclaimer 
at bugsnag, our web dashboard is written in rails 
this, mixed with the fact i wanted the tests to take screenshots, 
affected my decisions in what frameworks i wanted to use
we also use git for our source control at bugsnag, and we use it the “github” way
master 
what that means is, we have our master branch 
which is always deployable and production ready 
(or at least it should be)
master feature 
and whenever we want to create a feature, 
we branch off of master until it’s ready, and then we merge it back in
considering the tools we had at our disposal, and after taking a look at some of the screenshotting framework’s source codes, 
i realized that there wasn’t actually that much code to a lot of the screenshot libraries 
so I decided why not, i’m just gonna write one myself
PROCESS 
i came up with a process of how i thought i wanted my tests to work
1. VISIT 
number one, i wanted a way to somehow automatically visit pages of our site
so the test would, in an actual browser, hit each page of our site on a local server
2. SCREENSHOT 
once the test visited the page, i wanted the test to take a screenshot of that page
the important bit for the screenshot 
is that i’d want it to take a screenshot of the entire page
not just the current viewport of the browser
for example, if a change happened below the fold of our site, but we weren’t taking full page screenshots 
we wouldn’t be able to capture that diff
3. UPLOAD 
next, i’d need somewhere to store these screenshots, 
and i’d need a way to upload and download these screenshots from that storage area
upload 
master 
screenshot 
so, using git, every time i made a push to a branch, i’d upload a screenshot of the current state of each page 
including our master branch
download 
master 
screenshot 
upload 
feature 
screenshot 
and, if i had a screenshot already uploaded to our storage area from our master branch, 
i’d need a way to upload my current features branch to that storage area, 
and download my already uploaded master screenshot from there
4.DIFF 
i’d then need a way to make a diff of my screenshots
master feature 
i’d want to diff between a screenshot i took on master, downloaded from our storage area, 
and the newest screenshot i took, on my feature branch
so if i have the previous screenshot i took 
which would be the current commit on master 
vs my branch’s screenshot
and i’d need a way to mark the differences between these two screenshots visually
5.VIEW 
finally, after i have diffs for my screenshots, i’d need a way to view the diffs 
even though i’ll have a place to upload them, i need an accessible way for everyone on the project to view the diff screenshots depending on the commit
BUILDING 
our framework 
so now that we have a plan, we can start building our framework around these things that we need
TESTING 
with rspec 
we need a way to write some tests that will run automatically after each push 
so we decided to use rspec
rspec 
rspec is a testing tool for the ruby programming language 
we already had rspec for tests our rails app, so writing these tests with rspec made sense
we wanted to be able to write specs that looked like this 
where we would just be able to navigate to a local URL
and save a screenshot of that page 
we also wanted these tests to be separate of our main tests
we also wanted these tests to be separate of our main tests
we pulled out these visual specs into their own rspec tag 
that way, these specs wouldn’t run with our main specs when we were running the suite locally, unless we explicitly wanted them to 
this also made it so we could break out our specs on our CI
LOCAL 
speed 
we want our visual specs to be separate for a few reasons 
number one, our local build speed 
if our local tests were bogged down by waiting for visual specs, that would become a huge issue 
by having them broken out, we could iterate on our main app specs faster and be able to push more often
CI 
speed 
number two, is speed with our CI tests 
we wanted our main specs to still be fast on our CI, so that we can merge non-visual pull requests without waiting for our visual specs to finish
CI, or continuous integration, is a way for us to automatically run our test suite when we push things to github 
and after our specs run on our CI, we are able to see if our build is passing or not on github
i also learned that github just released a new feature where you can split up your builds 
so now, we’ll be able to split out our visual specs from our main specs here 
so we can quickly see which build is passing or failing or still running 
without the builds being combined
buildbox 
at bugsnag, we use buildbox for our CI
buildbox allows us to add steps to our tests 
that way, we can run our main specs first, apart from our visual specs 
when they’re separate, our visual specs don’t slow down our main specs 
and we can merge non-visual pull requests without waiting for our visual specs to finish
SCREENSHOT 
with selenium 
next, we needed a way to visit webpages and take screenshots with our rspec tests 
for that, we decided to use selenium
selenium 
selenium is a tool for automating browsers for testing purposes 
we would need to use, specifically, their web driver API 
this would allow us to drive a browser natively on a local or remote machine 
more specifically, this provides an API between us, and a browser
browserstack 
to use selenium, we’d need to use a service like sauce labs or browserstack 
we need to use a service like this because we need access to an actual browser 
since we are using a CI, it doesn’t have browsers just built in on the server 
we’d either have to set up our own virtual machines for these browsers or use one of these services. we ended up trying browserstack
before our visual tests, we’d need to start our proxy to browserstack and a forked rails server 
and then we’d make an instance of a selenium web driver 
and then of course, after all of our tests, we would terminate the services
we also had to allow webmock, 
web mock a library for stubbing and setting expectations on HTTP requests in Ruby, 
to run real web requests 
in order to use our local servers and upload our screenshots
to get our browserstack proxy running, we would just spawn a new browserstack process 
and terminate the process using its assigned pid
and to get our rails server running, we would spin up a new rails process at port 3000 
unless one was currently running 
and terminate it the same way as our browserstack process
to set up our selenium webdriver, we just have to pass it the capabilities we wanted 
and a url to hit, which was pointed at browserstack
setting up our selenium driver was easy, 
but when we were setting it up, 
we learned some interesting things about taking screenshots with different browsers
with our web driver, we wanted to hit web pages in a real browser
and be able to take screenshots of the full page, not just the current viewport
unfortunately, this feature only worked in firefox, which was not ideal
since internet explorer and chrome didn’t work, we couldn’t really transform this framework we’re making to be used for browser compatibility or anything like that 
although this wasn’t ideal, for our purposes right now as a refactoring tool, firefox worked fine
DYNAMIC 
DATA 
after writing tests for static pages such as our homepage, we quickly realized that we'd have an issue with the dynamic data on our dashboard
with dynamic data, you can get false positive diffs because data can change between the viewing times 
to combat this, we set up fixture data for our rspec tests, and manually adjusted any other data not covered by fixtures using Selenium's Javascript support 
so that we don’t get a false positive diff
DIFFING with imagemagick 
ok, we now have our tests taking screenshots 
we need to figure out a way to make a diff between two of our screenshots 
imagemagick worked perfectly for this
imagemagick 
despite having literally one of the worst sites i’ve ever seen, 
imagemagick did exactly what we needed 
imagemagick is a tool to convert, edit, and compose images
imagemagick has a command-line compare tool that, with various options enabled, 
allows us to shell out and produce diff screenshots based off of two other screenshots
for example, when we’d make a simple change like changing the header
imagemagick would spot those differences
and produce a screenshot like this
imagemagick has a lot of options you can pass to its compare tool 
and we take advantage of a few of these options in order to make it work
let’s go back to the options we use for a second and go over how they actually work with what we’re doing
imagemagick’s compare tool, from their website, “mathematically and visually annotates the differences between an image and its reconstruction” 
or, in my terms, takes two images and provides a visual diff
compare lets you provide a metric that outputs to standard error a measure of the differences between images according to the type given metric 
here we’re using PAE, where PAE stands for peak absolute 
we can use the peak absolute to find the size of the “fuzz” factor needed to make all pixels “similar”
1 
2 
so if we had screenshot1 and screenshot2, that were pretty different
it would end up producing a diff like this
and our peak absolute measurement would be outputted as needing a huge fuzz factor to make all pixels similar
the fuzz factor can be important in case we want to ignore pixels which only changed by a small amount 
we might want to ignore small changes in case of false positives 
for example, i’ve had false positives before because gradients rendered SLIGHTLY differently between two images
we don’t use this output right now, but it would be important if you wanted to make an assertion in your tests meaningful 
like, actually have a failure if a diff was produced by a certain amount 
we didn’t end up doing that because it doesn’t necessarily mean something is wrong if a diff is created
a few times we were running our specs, we noticed that diffs weren’t even being produced
we took a look at the screenshots, and realized that they were different heights or sizes for some reason, 
like if we made a change that accidentally removed the footer. 
imagemagick wouldn’t let us do a default compare on these images, so we had to use a subimage search
subimage searching is required to have compare search for the best match location of a small image within a larger image 
this option will produce two images (or two frames) 
the first is the "difference" image and the second will be the "match score" image
the "match-score" image is smaller image containing a pixel for every possible position of the top-left corner of the given sub-image 
the search will try to compare the sub-image at every possible location in the larger image 
this can make sub-image searching very slow 
as you could guess, the smaller the sub-image is, the faster this search is
that being said, this option doesn’t take effect unless you have two images that are different sizes 
this doesn’t happen often to us, so the amount it slows down our visual specs on CI is not meaningful 
especially since those specs aren’t tied to our main specs
another fun thing we ran into was sometimes our screenshots were COMPLETELY different
VS 
like, completely different
and imagemagick was NOT into it 
in fact, it just wouldn’t even give us a diff, because the images were so different 
we found an option called the “dissimilarity-threshold” 
this threshold determined how different two images could be in order to diff them 
it defaulted to .2, so we upped that to 1, and it seemed to do the trick
the only caveat to this, as you may have guessed, is doing diffs on completely different images can slow down your tests by a lot 
like our previous issue with sub-image searching, this doesn’t seem to happen much 
and since the tests are separate from our main specs, we aren’t too worried about it
the last 3 arguments aren’t exciting, these are just where our current screenshot is
where our master screenshot is
and where we want the diff to save to
STORING with aws 
now that we had our screenshots and diffs, 
we needed somewhere to throw the screenshots online 
and be able to grab them back out with our rails app 
so we decided to use aws
aws, or amazon web services, offers cloud storage and has a ruby api 
so we’ll store and retrieve our screenshots from one of their buckets
/#{commit-sha}/ 
#{area-of-site}/ 
#{page-name}/ 
#{image-type}.png 
we ended up using a naming pattern of commit-sha, area-of-site, page-name, 
and image-type
/a1a1a1a/ 
marketing/ 
index/ 
diff.png 
so, for an example, we could have a commit sha of a1a1a1a 
where we’re in the marketing part of our site 
on the index page 
uploading the diff for that page
diff.png 
current.png 
master.png 
the image-types could be current screenshot we took 
or the master screenshot we downloaded from S3 
or the diff we made from the two screenshots
VIEWING with the internet 
viewing the screenshots from an amazon bucket was far less than ideal 
so we decided to set up our own custom viewing page in our admin dashboard
we created a page that listed out our current branches with their last 3 commits
and when you would click through, it would show you, by area, all of your screenshots and diffs 
which was our end goal!
THE 
FUTURE 
however, our tool is not perfect right now 
but it’s certainly better than what we had 
which was nothing, but
MEANINGFUL 
TEST ASSERTIONS 
right now, all of our tests pass whether or not there's a diff 
they'll only fail if there's an issue executing the test 
it could be interesting to make our assertions mean something - like fail if there’s a diff 
we’d need to think more about that, because a diff doesn’t necessarily mean failure to us
0% DIFFS 
in the future, it’d be nice to account for 0% diffs
VS 
VS 
like, maybe we shouldn’t upload a diff image at all if there’s no diff between the images 
this could save us space on our aws bucket 
as well as speed up our tests because we’re trying to upload fewer screenshots 
and it would make our admin dashboard less noisy
PULL 
REQUESTS 
we also think it’d be nice to automatically link these to a github pull request
so, when a diff is created, maybe automatically attach it onto its relevant pull request 
this one is a little tricky, because maybe we don’t want to create that much noise every time we push 
but it’s an idea
DIFF BETWEEN 
CURRENT 
commit on 
BRANCH 
CURRENT 
commit on 
MASTER 
another thing is that we currently only diff between our current commit on a branch 
vs our current most recent commit on master
master feature 
so, when we push a new commit to our branch, it’ll diff versus the last thing that was pushed to master 
this way, we know what’s changed between the feature that i’m working on and what’s currently running in production
DIFF BETWEEN 
PREVIOUS 
commit on 
MASTER 
CURRENT 
commit on 
MASTER 
it would be nice if we could diff on master 
with the current most recent commit 
vs its previous commit
master feature 
so that way, in case you push a visual change directly to master, you can still see a diff of it
DIFF BETWEEN 
CURRENT 
commit on 
BRANCH 
PREVIOUS 
commit on 
BRANCH 
it would also be nice to see a diff between the previous commit on the current branch as well
master feature 
like, if you pushed a commit to your branch that changes some stuff visually, 
you might want to see a diff between those two, regardless of whats happening on master
i also would really love to get this hooked up to more browsers 
that would enable us to make this into a backwards compatibility tool 
as well as an automatic browser comparison 
to make sure things aren’t bonked in ie
thanks! @jessicard 
anyway, that’s all i have! 
feel free to find me after if you want to talk more about css testing frameworks or just want to say hi! 
thanks for having me!

More Related Content

What's hot

Tutorial: Develop Mobile Applications with AngularJS
Tutorial: Develop Mobile Applications with AngularJSTutorial: Develop Mobile Applications with AngularJS
Tutorial: Develop Mobile Applications with AngularJSPhilipp Burgmer
 
iOS Developers Conference-iOS Automation with Cucumber, Appium and Saucelabs
iOS Developers Conference-iOS Automation with Cucumber, Appium and SaucelabsiOS Developers Conference-iOS Automation with Cucumber, Appium and Saucelabs
iOS Developers Conference-iOS Automation with Cucumber, Appium and SaucelabsShashikant Jagtap
 
iOS Automation with Cucumber, Appium and Saucelabs
iOS Automation with Cucumber, Appium and SaucelabsiOS Automation with Cucumber, Appium and Saucelabs
iOS Automation with Cucumber, Appium and SaucelabsShashikant Jagtap
 
Experiences building apps with React Native @UtrechtJS May 2016
Experiences building apps with React Native @UtrechtJS May 2016Experiences building apps with React Native @UtrechtJS May 2016
Experiences building apps with React Native @UtrechtJS May 2016Adrian Philipp
 
What's This React Native Thing I Keep Hearing About?
What's This React Native Thing I Keep Hearing About?What's This React Native Thing I Keep Hearing About?
What's This React Native Thing I Keep Hearing About?Evan Stone
 
Headless BDD & Responsive Test Automation
Headless BDD & Responsive Test AutomationHeadless BDD & Responsive Test Automation
Headless BDD & Responsive Test AutomationShashikant Jagtap
 
Tooling for the productive front end developer
Tooling for the productive front end developerTooling for the productive front end developer
Tooling for the productive front end developerMaurice De Beijer [MVP]
 
Tek 2013 - Building Web Apps from a New Angle with AngularJS
Tek 2013 - Building Web Apps from a New Angle with AngularJSTek 2013 - Building Web Apps from a New Angle with AngularJS
Tek 2013 - Building Web Apps from a New Angle with AngularJSPablo Godel
 
Modern Web Application Development Workflow - EclipseCon France 2014
Modern Web Application Development Workflow - EclipseCon France 2014Modern Web Application Development Workflow - EclipseCon France 2014
Modern Web Application Development Workflow - EclipseCon France 2014Stéphane Bégaudeau
 
On Selecting JavaScript Frameworks (Women Who Code 10/15)
On Selecting JavaScript Frameworks (Women Who Code 10/15)On Selecting JavaScript Frameworks (Women Who Code 10/15)
On Selecting JavaScript Frameworks (Women Who Code 10/15)Zoe Landon
 
Building the Front End with AngularJS
Building the Front End with AngularJSBuilding the Front End with AngularJS
Building the Front End with AngularJSJohn Ennew
 
jQuery Conference Boston 2011 CouchApps
jQuery Conference Boston 2011 CouchAppsjQuery Conference Boston 2011 CouchApps
jQuery Conference Boston 2011 CouchAppsBradley Holt
 
125 고성능 web view-deview 2013 발표 자료_공유용
125 고성능 web view-deview 2013 발표 자료_공유용125 고성능 web view-deview 2013 발표 자료_공유용
125 고성능 web view-deview 2013 발표 자료_공유용NAVER D2
 
Testing Mobile JavaScript
Testing Mobile JavaScriptTesting Mobile JavaScript
Testing Mobile JavaScriptjeresig
 
Flu3nt highlights
Flu3nt highlightsFlu3nt highlights
Flu3nt highlightsdswork
 
Ionic adventures - Hybrid Mobile App Development rocks
Ionic adventures - Hybrid Mobile App Development rocksIonic adventures - Hybrid Mobile App Development rocks
Ionic adventures - Hybrid Mobile App Development rocksJuarez Filho
 
Real World Selenium Testing
Real World Selenium TestingReal World Selenium Testing
Real World Selenium TestingMary Jo Sminkey
 
Continuous Visual Integration - RailsConf 2016 - Mike Fotinakis - Percy.io
Continuous Visual Integration - RailsConf 2016 - Mike Fotinakis - Percy.ioContinuous Visual Integration - RailsConf 2016 - Mike Fotinakis - Percy.io
Continuous Visual Integration - RailsConf 2016 - Mike Fotinakis - Percy.ioMike Fotinakis
 

What's hot (20)

Tutorial: Develop Mobile Applications with AngularJS
Tutorial: Develop Mobile Applications with AngularJSTutorial: Develop Mobile Applications with AngularJS
Tutorial: Develop Mobile Applications with AngularJS
 
iOS Developers Conference-iOS Automation with Cucumber, Appium and Saucelabs
iOS Developers Conference-iOS Automation with Cucumber, Appium and SaucelabsiOS Developers Conference-iOS Automation with Cucumber, Appium and Saucelabs
iOS Developers Conference-iOS Automation with Cucumber, Appium and Saucelabs
 
iOS Automation with Cucumber, Appium and Saucelabs
iOS Automation with Cucumber, Appium and SaucelabsiOS Automation with Cucumber, Appium and Saucelabs
iOS Automation with Cucumber, Appium and Saucelabs
 
Experiences building apps with React Native @UtrechtJS May 2016
Experiences building apps with React Native @UtrechtJS May 2016Experiences building apps with React Native @UtrechtJS May 2016
Experiences building apps with React Native @UtrechtJS May 2016
 
What's This React Native Thing I Keep Hearing About?
What's This React Native Thing I Keep Hearing About?What's This React Native Thing I Keep Hearing About?
What's This React Native Thing I Keep Hearing About?
 
Headless BDD & Responsive Test Automation
Headless BDD & Responsive Test AutomationHeadless BDD & Responsive Test Automation
Headless BDD & Responsive Test Automation
 
Tooling for the productive front end developer
Tooling for the productive front end developerTooling for the productive front end developer
Tooling for the productive front end developer
 
Tek 2013 - Building Web Apps from a New Angle with AngularJS
Tek 2013 - Building Web Apps from a New Angle with AngularJSTek 2013 - Building Web Apps from a New Angle with AngularJS
Tek 2013 - Building Web Apps from a New Angle with AngularJS
 
Modern Web Application Development Workflow - EclipseCon France 2014
Modern Web Application Development Workflow - EclipseCon France 2014Modern Web Application Development Workflow - EclipseCon France 2014
Modern Web Application Development Workflow - EclipseCon France 2014
 
On Selecting JavaScript Frameworks (Women Who Code 10/15)
On Selecting JavaScript Frameworks (Women Who Code 10/15)On Selecting JavaScript Frameworks (Women Who Code 10/15)
On Selecting JavaScript Frameworks (Women Who Code 10/15)
 
Building the Front End with AngularJS
Building the Front End with AngularJSBuilding the Front End with AngularJS
Building the Front End with AngularJS
 
jQuery Conference Boston 2011 CouchApps
jQuery Conference Boston 2011 CouchAppsjQuery Conference Boston 2011 CouchApps
jQuery Conference Boston 2011 CouchApps
 
125 고성능 web view-deview 2013 발표 자료_공유용
125 고성능 web view-deview 2013 발표 자료_공유용125 고성능 web view-deview 2013 발표 자료_공유용
125 고성능 web view-deview 2013 발표 자료_공유용
 
Mobile native-hacks
Mobile native-hacksMobile native-hacks
Mobile native-hacks
 
React Native
React NativeReact Native
React Native
 
Testing Mobile JavaScript
Testing Mobile JavaScriptTesting Mobile JavaScript
Testing Mobile JavaScript
 
Flu3nt highlights
Flu3nt highlightsFlu3nt highlights
Flu3nt highlights
 
Ionic adventures - Hybrid Mobile App Development rocks
Ionic adventures - Hybrid Mobile App Development rocksIonic adventures - Hybrid Mobile App Development rocks
Ionic adventures - Hybrid Mobile App Development rocks
 
Real World Selenium Testing
Real World Selenium TestingReal World Selenium Testing
Real World Selenium Testing
 
Continuous Visual Integration - RailsConf 2016 - Mike Fotinakis - Percy.io
Continuous Visual Integration - RailsConf 2016 - Mike Fotinakis - Percy.ioContinuous Visual Integration - RailsConf 2016 - Mike Fotinakis - Percy.io
Continuous Visual Integration - RailsConf 2016 - Mike Fotinakis - Percy.io
 

Similar to VISUAL CSS TESTING

Visual regression testing
Visual regression testingVisual regression testing
Visual regression testingLiam McMurray
 
Colorful world-of-visual-automation-testing-latest
Colorful world-of-visual-automation-testing-latestColorful world-of-visual-automation-testing-latest
Colorful world-of-visual-automation-testing-latestOnur Baskirt
 
6 Methods to use page scroll animation.pdf
6 Methods to use page scroll animation.pdf6 Methods to use page scroll animation.pdf
6 Methods to use page scroll animation.pdfBe Problem Solver
 
E2E testing Single Page Apps and APIs with Cucumber.js and Puppeteer
E2E testing Single Page Apps and APIs with Cucumber.js and PuppeteerE2E testing Single Page Apps and APIs with Cucumber.js and Puppeteer
E2E testing Single Page Apps and APIs with Cucumber.js and PuppeteerPaul Jensen
 
Web driver selenium simplified
Web driver selenium simplifiedWeb driver selenium simplified
Web driver selenium simplifiedVikas Singh
 
Creating a Whatsapp Clone - Part I - Transcript.pdf
Creating a Whatsapp Clone - Part I - Transcript.pdfCreating a Whatsapp Clone - Part I - Transcript.pdf
Creating a Whatsapp Clone - Part I - Transcript.pdfShaiAlmog1
 
frontend-161011205424 (1).pdf
frontend-161011205424 (1).pdffrontend-161011205424 (1).pdf
frontend-161011205424 (1).pdfkajalkumari896241
 
Reworking our-workflows
Reworking our-workflowsReworking our-workflows
Reworking our-workflowsnolly00
 
Django for n00bs
Django for n00bsDjango for n00bs
Django for n00bsJen Zajac
 
End to end testing Single Page Apps & APIs with Cucumber.js and Puppeteer (Em...
End to end testing Single Page Apps & APIs with Cucumber.js and Puppeteer (Em...End to end testing Single Page Apps & APIs with Cucumber.js and Puppeteer (Em...
End to end testing Single Page Apps & APIs with Cucumber.js and Puppeteer (Em...Paul Jensen
 
Progressive Web Application by Citytech
Progressive Web Application by CitytechProgressive Web Application by Citytech
Progressive Web Application by CitytechRitwik Das
 
Visual Regression Testing: In search of an Ember solution
Visual Regression Testing: In search of an Ember solutionVisual Regression Testing: In search of an Ember solution
Visual Regression Testing: In search of an Ember solutionLisa Backer
 
European SharePoint Conference 2017 - SharePoint Framework, Angular & Azure F...
European SharePoint Conference 2017 - SharePoint Framework, Angular & Azure F...European SharePoint Conference 2017 - SharePoint Framework, Angular & Azure F...
European SharePoint Conference 2017 - SharePoint Framework, Angular & Azure F...Sébastien Levert
 
Adapting to a Responsive Design at Untangle the Web on 29th July 2013
Adapting to a Responsive Design at Untangle the Web on 29th July 2013Adapting to a Responsive Design at Untangle the Web on 29th July 2013
Adapting to a Responsive Design at Untangle the Web on 29th July 2013Matt Gibson
 
Bulletproof design systems using storybook
Bulletproof design systems using storybookBulletproof design systems using storybook
Bulletproof design systems using storybookChen Feldman
 
Server Driven UI in iOS
Server Driven UI in iOSServer Driven UI in iOS
Server Driven UI in iOSMohammad Azam
 

Similar to VISUAL CSS TESTING (20)

Visual regression testing
Visual regression testingVisual regression testing
Visual regression testing
 
Colorful world-of-visual-automation-testing-latest
Colorful world-of-visual-automation-testing-latestColorful world-of-visual-automation-testing-latest
Colorful world-of-visual-automation-testing-latest
 
6 Methods to use page scroll animation.pdf
6 Methods to use page scroll animation.pdf6 Methods to use page scroll animation.pdf
6 Methods to use page scroll animation.pdf
 
E2E testing Single Page Apps and APIs with Cucumber.js and Puppeteer
E2E testing Single Page Apps and APIs with Cucumber.js and PuppeteerE2E testing Single Page Apps and APIs with Cucumber.js and Puppeteer
E2E testing Single Page Apps and APIs with Cucumber.js and Puppeteer
 
Web driver selenium simplified
Web driver selenium simplifiedWeb driver selenium simplified
Web driver selenium simplified
 
Creating a Whatsapp Clone - Part I - Transcript.pdf
Creating a Whatsapp Clone - Part I - Transcript.pdfCreating a Whatsapp Clone - Part I - Transcript.pdf
Creating a Whatsapp Clone - Part I - Transcript.pdf
 
frontend-161011205424 (1).pdf
frontend-161011205424 (1).pdffrontend-161011205424 (1).pdf
frontend-161011205424 (1).pdf
 
Web Development
Web DevelopmentWeb Development
Web Development
 
Reworking our-workflows
Reworking our-workflowsReworking our-workflows
Reworking our-workflows
 
Django for n00bs
Django for n00bsDjango for n00bs
Django for n00bs
 
End to end testing Single Page Apps & APIs with Cucumber.js and Puppeteer (Em...
End to end testing Single Page Apps & APIs with Cucumber.js and Puppeteer (Em...End to end testing Single Page Apps & APIs with Cucumber.js and Puppeteer (Em...
End to end testing Single Page Apps & APIs with Cucumber.js and Puppeteer (Em...
 
frontend-161011205424.pptx
frontend-161011205424.pptxfrontend-161011205424.pptx
frontend-161011205424.pptx
 
Progressive Web Application by Citytech
Progressive Web Application by CitytechProgressive Web Application by Citytech
Progressive Web Application by Citytech
 
Visual Regression Testing: In search of an Ember solution
Visual Regression Testing: In search of an Ember solutionVisual Regression Testing: In search of an Ember solution
Visual Regression Testing: In search of an Ember solution
 
European SharePoint Conference 2017 - SharePoint Framework, Angular & Azure F...
European SharePoint Conference 2017 - SharePoint Framework, Angular & Azure F...European SharePoint Conference 2017 - SharePoint Framework, Angular & Azure F...
European SharePoint Conference 2017 - SharePoint Framework, Angular & Azure F...
 
Adapting to a Responsive Design at Untangle the Web on 29th July 2013
Adapting to a Responsive Design at Untangle the Web on 29th July 2013Adapting to a Responsive Design at Untangle the Web on 29th July 2013
Adapting to a Responsive Design at Untangle the Web on 29th July 2013
 
Gatsby vs. Next.js
Gatsby vs. Next.jsGatsby vs. Next.js
Gatsby vs. Next.js
 
Enhance Enhance
Enhance EnhanceEnhance Enhance
Enhance Enhance
 
Bulletproof design systems using storybook
Bulletproof design systems using storybookBulletproof design systems using storybook
Bulletproof design systems using storybook
 
Server Driven UI in iOS
Server Driven UI in iOSServer Driven UI in iOS
Server Driven UI in iOS
 

Recently uploaded

A brief look at visionOS - How to develop app on Apple's Vision Pro
A brief look at visionOS - How to develop app on Apple's Vision ProA brief look at visionOS - How to develop app on Apple's Vision Pro
A brief look at visionOS - How to develop app on Apple's Vision ProRay Yuan Liu
 
Computer Graphics Introduction, Open GL, Line and Circle drawing algorithm
Computer Graphics Introduction, Open GL, Line and Circle drawing algorithmComputer Graphics Introduction, Open GL, Line and Circle drawing algorithm
Computer Graphics Introduction, Open GL, Line and Circle drawing algorithmDeepika Walanjkar
 
ROBOETHICS-CCS345 ETHICS AND ARTIFICIAL INTELLIGENCE.ppt
ROBOETHICS-CCS345 ETHICS AND ARTIFICIAL INTELLIGENCE.pptROBOETHICS-CCS345 ETHICS AND ARTIFICIAL INTELLIGENCE.ppt
ROBOETHICS-CCS345 ETHICS AND ARTIFICIAL INTELLIGENCE.pptJohnWilliam111370
 
Input Output Management in Operating System
Input Output Management in Operating SystemInput Output Management in Operating System
Input Output Management in Operating SystemRashmi Bhat
 
Comprehensive energy systems.pdf Comprehensive energy systems.pdf
Comprehensive energy systems.pdf Comprehensive energy systems.pdfComprehensive energy systems.pdf Comprehensive energy systems.pdf
Comprehensive energy systems.pdf Comprehensive energy systems.pdfalene1
 
KCD Costa Rica 2024 - Nephio para parvulitos
KCD Costa Rica 2024 - Nephio para parvulitosKCD Costa Rica 2024 - Nephio para parvulitos
KCD Costa Rica 2024 - Nephio para parvulitosVictor Morales
 
2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.
2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.
2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.elesangwon
 
List of Accredited Concrete Batching Plant.pdf
List of Accredited Concrete Batching Plant.pdfList of Accredited Concrete Batching Plant.pdf
List of Accredited Concrete Batching Plant.pdfisabel213075
 
Mine Environment II Lab_MI10448MI__________.pptx
Mine Environment II Lab_MI10448MI__________.pptxMine Environment II Lab_MI10448MI__________.pptx
Mine Environment II Lab_MI10448MI__________.pptxRomil Mishra
 
SOFTWARE ESTIMATION COCOMO AND FP CALCULATION
SOFTWARE ESTIMATION COCOMO AND FP CALCULATIONSOFTWARE ESTIMATION COCOMO AND FP CALCULATION
SOFTWARE ESTIMATION COCOMO AND FP CALCULATIONSneha Padhiar
 
CS 3251 Programming in c all unit notes pdf
CS 3251 Programming in c all unit notes pdfCS 3251 Programming in c all unit notes pdf
CS 3251 Programming in c all unit notes pdfBalamuruganV28
 
Robotics-Asimov's Laws, Mechanical Subsystems, Robot Kinematics, Robot Dynami...
Robotics-Asimov's Laws, Mechanical Subsystems, Robot Kinematics, Robot Dynami...Robotics-Asimov's Laws, Mechanical Subsystems, Robot Kinematics, Robot Dynami...
Robotics-Asimov's Laws, Mechanical Subsystems, Robot Kinematics, Robot Dynami...Sumanth A
 
Main Memory Management in Operating System
Main Memory Management in Operating SystemMain Memory Management in Operating System
Main Memory Management in Operating SystemRashmi Bhat
 
Turn leadership mistakes into a better future.pptx
Turn leadership mistakes into a better future.pptxTurn leadership mistakes into a better future.pptx
Turn leadership mistakes into a better future.pptxStephen Sitton
 
Theory of Machine Notes / Lecture Material .pdf
Theory of Machine Notes / Lecture Material .pdfTheory of Machine Notes / Lecture Material .pdf
Theory of Machine Notes / Lecture Material .pdfShreyas Pandit
 
FUNCTIONAL AND NON FUNCTIONAL REQUIREMENT
FUNCTIONAL AND NON FUNCTIONAL REQUIREMENTFUNCTIONAL AND NON FUNCTIONAL REQUIREMENT
FUNCTIONAL AND NON FUNCTIONAL REQUIREMENTSneha Padhiar
 
Energy Awareness training ppt for manufacturing process.pptx
Energy Awareness training ppt for manufacturing process.pptxEnergy Awareness training ppt for manufacturing process.pptx
Energy Awareness training ppt for manufacturing process.pptxsiddharthjain2303
 
『澳洲文凭』买麦考瑞大学毕业证书成绩单办理澳洲Macquarie文凭学位证书
『澳洲文凭』买麦考瑞大学毕业证书成绩单办理澳洲Macquarie文凭学位证书『澳洲文凭』买麦考瑞大学毕业证书成绩单办理澳洲Macquarie文凭学位证书
『澳洲文凭』买麦考瑞大学毕业证书成绩单办理澳洲Macquarie文凭学位证书rnrncn29
 
Triangulation survey (Basic Mine Surveying)_MI10412MI.pptx
Triangulation survey (Basic Mine Surveying)_MI10412MI.pptxTriangulation survey (Basic Mine Surveying)_MI10412MI.pptx
Triangulation survey (Basic Mine Surveying)_MI10412MI.pptxRomil Mishra
 
Javier_Fernandez_CARS_workshop_presentation.pptx
Javier_Fernandez_CARS_workshop_presentation.pptxJavier_Fernandez_CARS_workshop_presentation.pptx
Javier_Fernandez_CARS_workshop_presentation.pptxJavier Fernández Muñoz
 

Recently uploaded (20)

A brief look at visionOS - How to develop app on Apple's Vision Pro
A brief look at visionOS - How to develop app on Apple's Vision ProA brief look at visionOS - How to develop app on Apple's Vision Pro
A brief look at visionOS - How to develop app on Apple's Vision Pro
 
Computer Graphics Introduction, Open GL, Line and Circle drawing algorithm
Computer Graphics Introduction, Open GL, Line and Circle drawing algorithmComputer Graphics Introduction, Open GL, Line and Circle drawing algorithm
Computer Graphics Introduction, Open GL, Line and Circle drawing algorithm
 
ROBOETHICS-CCS345 ETHICS AND ARTIFICIAL INTELLIGENCE.ppt
ROBOETHICS-CCS345 ETHICS AND ARTIFICIAL INTELLIGENCE.pptROBOETHICS-CCS345 ETHICS AND ARTIFICIAL INTELLIGENCE.ppt
ROBOETHICS-CCS345 ETHICS AND ARTIFICIAL INTELLIGENCE.ppt
 
Input Output Management in Operating System
Input Output Management in Operating SystemInput Output Management in Operating System
Input Output Management in Operating System
 
Comprehensive energy systems.pdf Comprehensive energy systems.pdf
Comprehensive energy systems.pdf Comprehensive energy systems.pdfComprehensive energy systems.pdf Comprehensive energy systems.pdf
Comprehensive energy systems.pdf Comprehensive energy systems.pdf
 
KCD Costa Rica 2024 - Nephio para parvulitos
KCD Costa Rica 2024 - Nephio para parvulitosKCD Costa Rica 2024 - Nephio para parvulitos
KCD Costa Rica 2024 - Nephio para parvulitos
 
2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.
2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.
2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.
 
List of Accredited Concrete Batching Plant.pdf
List of Accredited Concrete Batching Plant.pdfList of Accredited Concrete Batching Plant.pdf
List of Accredited Concrete Batching Plant.pdf
 
Mine Environment II Lab_MI10448MI__________.pptx
Mine Environment II Lab_MI10448MI__________.pptxMine Environment II Lab_MI10448MI__________.pptx
Mine Environment II Lab_MI10448MI__________.pptx
 
SOFTWARE ESTIMATION COCOMO AND FP CALCULATION
SOFTWARE ESTIMATION COCOMO AND FP CALCULATIONSOFTWARE ESTIMATION COCOMO AND FP CALCULATION
SOFTWARE ESTIMATION COCOMO AND FP CALCULATION
 
CS 3251 Programming in c all unit notes pdf
CS 3251 Programming in c all unit notes pdfCS 3251 Programming in c all unit notes pdf
CS 3251 Programming in c all unit notes pdf
 
Robotics-Asimov's Laws, Mechanical Subsystems, Robot Kinematics, Robot Dynami...
Robotics-Asimov's Laws, Mechanical Subsystems, Robot Kinematics, Robot Dynami...Robotics-Asimov's Laws, Mechanical Subsystems, Robot Kinematics, Robot Dynami...
Robotics-Asimov's Laws, Mechanical Subsystems, Robot Kinematics, Robot Dynami...
 
Main Memory Management in Operating System
Main Memory Management in Operating SystemMain Memory Management in Operating System
Main Memory Management in Operating System
 
Turn leadership mistakes into a better future.pptx
Turn leadership mistakes into a better future.pptxTurn leadership mistakes into a better future.pptx
Turn leadership mistakes into a better future.pptx
 
Theory of Machine Notes / Lecture Material .pdf
Theory of Machine Notes / Lecture Material .pdfTheory of Machine Notes / Lecture Material .pdf
Theory of Machine Notes / Lecture Material .pdf
 
FUNCTIONAL AND NON FUNCTIONAL REQUIREMENT
FUNCTIONAL AND NON FUNCTIONAL REQUIREMENTFUNCTIONAL AND NON FUNCTIONAL REQUIREMENT
FUNCTIONAL AND NON FUNCTIONAL REQUIREMENT
 
Energy Awareness training ppt for manufacturing process.pptx
Energy Awareness training ppt for manufacturing process.pptxEnergy Awareness training ppt for manufacturing process.pptx
Energy Awareness training ppt for manufacturing process.pptx
 
『澳洲文凭』买麦考瑞大学毕业证书成绩单办理澳洲Macquarie文凭学位证书
『澳洲文凭』买麦考瑞大学毕业证书成绩单办理澳洲Macquarie文凭学位证书『澳洲文凭』买麦考瑞大学毕业证书成绩单办理澳洲Macquarie文凭学位证书
『澳洲文凭』买麦考瑞大学毕业证书成绩单办理澳洲Macquarie文凭学位证书
 
Triangulation survey (Basic Mine Surveying)_MI10412MI.pptx
Triangulation survey (Basic Mine Surveying)_MI10412MI.pptxTriangulation survey (Basic Mine Surveying)_MI10412MI.pptx
Triangulation survey (Basic Mine Surveying)_MI10412MI.pptx
 
Javier_Fernandez_CARS_workshop_presentation.pptx
Javier_Fernandez_CARS_workshop_presentation.pptxJavier_Fernandez_CARS_workshop_presentation.pptx
Javier_Fernandez_CARS_workshop_presentation.pptx
 

VISUAL CSS TESTING

  • 1. IMPLEMENTING A VISUAL CSS TESTING FRAMEWORK @jessicard hi everyone! thanks for having me here today i’m going to talk about implementing a visual css testing framework
  • 2. using AUTOMATIC screenshot COMPARISON to catch style REGRESSIONS using automatic screenshot comparison to catch style regressions
  • 3. @jessicard my name is jessicard on the internet, and jessica in real life
  • 4. i work at a company called bugsnag here in san francisco bugsnag is an exception monitoring tool i’m a software engineer there, working primarily in ruby and javascript but our stack includes lots of languages, including node and go
  • 5. (we’re hiring!) we are currently hiring so please get in touch if you’re interested in working on developer tools at a small company!
  • 6. IMPLEMENTING A VISUAL CSS TESTING FRAMEWORK so, back to this whole “implementing a visual css testing framework” thing what am i even talking about?
  • 7. well, at bugsnag, we decided we wanted a way to automatically take two screenshots of our website, at different times in the app
  • 8. master feature for example, since we use git, lets say we had a feature branch that we just pushed a commit to, we’d want to take a screenshot of what our homepage looks like on that branch and a screenshot of how our homepage looks on production, or whats currently running on master
  • 9. with those screenshots, we’d want to spot those differences
  • 10. and produce some sort of diff image that highlighted the differences why would we want to do this, you might ask? well,
  • 11. as we all know, writing, reading, and code reviewing CSS can be really intense
  • 12.             and even more intense to refactor at bugsnag, we decided to do a huge organizational and code-style refactor of our CSS and we wanted a way to test that our site looked the same, despite changing all of the code
  • 13. unfortunately, as you can tell, that didn’t always work out for us we went through many iterations of refactoring, and we realized we needed a tool to help us test our pages automatically
  • 14. does this EXIST? so, we went on the hunt for a way to test our CSS we wanted to know if there was a tool already built that did what we wanted without knowing exactly what it was that we wanted out of this framework
  • 15. huxley we first stumbled upon one of facebook’s open source libraries, huxley
  • 16. “Watches you browse, takes screenshots, tells you when they change” in huxley’s readme, it says “watches you browse, takes screenshots, tells you when they change” which sounds amazing!
  • 17. but… i had noticed that it hadn’t been updated in over a year that wasn’t promising, but i decided to give it a shot anyway
  • 18.  after a good solid day of fiddling around with huxley, it ended up being a bit too buggy it would have random failures, randomly wouldn’t take screenshots and i started realizing it wasn’t exactly what i was looking for in a css testing tool
  • 19. Huxley Phantom CSS GhostStory Cactus Needle CSSCritic fighting-layout-bugs sikuli Mogo Quixote i kept looking, and it turns out there are a lot frameworks that do some sort of css testing they work in all different ways - some take screenshots, some aren’t even visual i’d definitely recommend checking some of these out to see if they fit what you’re doing in your apps before building your own
  • 20. WHAT DO I WANT? but i really started thinking about it what was i looking for in a CSS testing framework? what would fit the way bugsnag is built best? i decided i wanted a visual way to test my CSS, with screenshots rather than writing out tests describing the visuals
  • 21. this is where i need a disclaimer at bugsnag, our web dashboard is written in rails this, mixed with the fact i wanted the tests to take screenshots, affected my decisions in what frameworks i wanted to use
  • 22. we also use git for our source control at bugsnag, and we use it the “github” way
  • 23. master what that means is, we have our master branch which is always deployable and production ready (or at least it should be)
  • 24. master feature and whenever we want to create a feature, we branch off of master until it’s ready, and then we merge it back in
  • 25. considering the tools we had at our disposal, and after taking a look at some of the screenshotting framework’s source codes, i realized that there wasn’t actually that much code to a lot of the screenshot libraries so I decided why not, i’m just gonna write one myself
  • 26. PROCESS i came up with a process of how i thought i wanted my tests to work
  • 27. 1. VISIT number one, i wanted a way to somehow automatically visit pages of our site
  • 28. so the test would, in an actual browser, hit each page of our site on a local server
  • 29. 2. SCREENSHOT once the test visited the page, i wanted the test to take a screenshot of that page
  • 30. the important bit for the screenshot is that i’d want it to take a screenshot of the entire page
  • 31. not just the current viewport of the browser
  • 32. for example, if a change happened below the fold of our site, but we weren’t taking full page screenshots we wouldn’t be able to capture that diff
  • 33. 3. UPLOAD next, i’d need somewhere to store these screenshots, and i’d need a way to upload and download these screenshots from that storage area
  • 34. upload master screenshot so, using git, every time i made a push to a branch, i’d upload a screenshot of the current state of each page including our master branch
  • 35. download master screenshot upload feature screenshot and, if i had a screenshot already uploaded to our storage area from our master branch, i’d need a way to upload my current features branch to that storage area, and download my already uploaded master screenshot from there
  • 36. 4.DIFF i’d then need a way to make a diff of my screenshots
  • 37. master feature i’d want to diff between a screenshot i took on master, downloaded from our storage area, and the newest screenshot i took, on my feature branch
  • 38. so if i have the previous screenshot i took which would be the current commit on master vs my branch’s screenshot
  • 39. and i’d need a way to mark the differences between these two screenshots visually
  • 40. 5.VIEW finally, after i have diffs for my screenshots, i’d need a way to view the diffs even though i’ll have a place to upload them, i need an accessible way for everyone on the project to view the diff screenshots depending on the commit
  • 41. BUILDING our framework so now that we have a plan, we can start building our framework around these things that we need
  • 42. TESTING with rspec we need a way to write some tests that will run automatically after each push so we decided to use rspec
  • 43. rspec rspec is a testing tool for the ruby programming language we already had rspec for tests our rails app, so writing these tests with rspec made sense
  • 44. we wanted to be able to write specs that looked like this where we would just be able to navigate to a local URL
  • 45. and save a screenshot of that page we also wanted these tests to be separate of our main tests
  • 46. we also wanted these tests to be separate of our main tests
  • 47. we pulled out these visual specs into their own rspec tag that way, these specs wouldn’t run with our main specs when we were running the suite locally, unless we explicitly wanted them to this also made it so we could break out our specs on our CI
  • 48. LOCAL speed we want our visual specs to be separate for a few reasons number one, our local build speed if our local tests were bogged down by waiting for visual specs, that would become a huge issue by having them broken out, we could iterate on our main app specs faster and be able to push more often
  • 49. CI speed number two, is speed with our CI tests we wanted our main specs to still be fast on our CI, so that we can merge non-visual pull requests without waiting for our visual specs to finish
  • 50. CI, or continuous integration, is a way for us to automatically run our test suite when we push things to github and after our specs run on our CI, we are able to see if our build is passing or not on github
  • 51. i also learned that github just released a new feature where you can split up your builds so now, we’ll be able to split out our visual specs from our main specs here so we can quickly see which build is passing or failing or still running without the builds being combined
  • 52. buildbox at bugsnag, we use buildbox for our CI
  • 53. buildbox allows us to add steps to our tests that way, we can run our main specs first, apart from our visual specs when they’re separate, our visual specs don’t slow down our main specs and we can merge non-visual pull requests without waiting for our visual specs to finish
  • 54. SCREENSHOT with selenium next, we needed a way to visit webpages and take screenshots with our rspec tests for that, we decided to use selenium
  • 55. selenium selenium is a tool for automating browsers for testing purposes we would need to use, specifically, their web driver API this would allow us to drive a browser natively on a local or remote machine more specifically, this provides an API between us, and a browser
  • 56. browserstack to use selenium, we’d need to use a service like sauce labs or browserstack we need to use a service like this because we need access to an actual browser since we are using a CI, it doesn’t have browsers just built in on the server we’d either have to set up our own virtual machines for these browsers or use one of these services. we ended up trying browserstack
  • 57. before our visual tests, we’d need to start our proxy to browserstack and a forked rails server and then we’d make an instance of a selenium web driver and then of course, after all of our tests, we would terminate the services
  • 58. we also had to allow webmock, web mock a library for stubbing and setting expectations on HTTP requests in Ruby, to run real web requests in order to use our local servers and upload our screenshots
  • 59. to get our browserstack proxy running, we would just spawn a new browserstack process and terminate the process using its assigned pid
  • 60. and to get our rails server running, we would spin up a new rails process at port 3000 unless one was currently running and terminate it the same way as our browserstack process
  • 61. to set up our selenium webdriver, we just have to pass it the capabilities we wanted and a url to hit, which was pointed at browserstack
  • 62. setting up our selenium driver was easy, but when we were setting it up, we learned some interesting things about taking screenshots with different browsers
  • 63. with our web driver, we wanted to hit web pages in a real browser
  • 64. and be able to take screenshots of the full page, not just the current viewport
  • 65. unfortunately, this feature only worked in firefox, which was not ideal
  • 66. since internet explorer and chrome didn’t work, we couldn’t really transform this framework we’re making to be used for browser compatibility or anything like that although this wasn’t ideal, for our purposes right now as a refactoring tool, firefox worked fine
  • 67. DYNAMIC DATA after writing tests for static pages such as our homepage, we quickly realized that we'd have an issue with the dynamic data on our dashboard
  • 68. with dynamic data, you can get false positive diffs because data can change between the viewing times to combat this, we set up fixture data for our rspec tests, and manually adjusted any other data not covered by fixtures using Selenium's Javascript support so that we don’t get a false positive diff
  • 69. DIFFING with imagemagick ok, we now have our tests taking screenshots we need to figure out a way to make a diff between two of our screenshots imagemagick worked perfectly for this
  • 70. imagemagick despite having literally one of the worst sites i’ve ever seen, imagemagick did exactly what we needed imagemagick is a tool to convert, edit, and compose images
  • 71. imagemagick has a command-line compare tool that, with various options enabled, allows us to shell out and produce diff screenshots based off of two other screenshots
  • 72. for example, when we’d make a simple change like changing the header
  • 73. imagemagick would spot those differences
  • 74. and produce a screenshot like this
  • 75. imagemagick has a lot of options you can pass to its compare tool and we take advantage of a few of these options in order to make it work
  • 76. let’s go back to the options we use for a second and go over how they actually work with what we’re doing
  • 77. imagemagick’s compare tool, from their website, “mathematically and visually annotates the differences between an image and its reconstruction” or, in my terms, takes two images and provides a visual diff
  • 78. compare lets you provide a metric that outputs to standard error a measure of the differences between images according to the type given metric here we’re using PAE, where PAE stands for peak absolute we can use the peak absolute to find the size of the “fuzz” factor needed to make all pixels “similar”
  • 79. 1 2 so if we had screenshot1 and screenshot2, that were pretty different
  • 80. it would end up producing a diff like this
  • 81. and our peak absolute measurement would be outputted as needing a huge fuzz factor to make all pixels similar
  • 82. the fuzz factor can be important in case we want to ignore pixels which only changed by a small amount we might want to ignore small changes in case of false positives for example, i’ve had false positives before because gradients rendered SLIGHTLY differently between two images
  • 83. we don’t use this output right now, but it would be important if you wanted to make an assertion in your tests meaningful like, actually have a failure if a diff was produced by a certain amount we didn’t end up doing that because it doesn’t necessarily mean something is wrong if a diff is created
  • 84. a few times we were running our specs, we noticed that diffs weren’t even being produced
  • 85. we took a look at the screenshots, and realized that they were different heights or sizes for some reason, like if we made a change that accidentally removed the footer. imagemagick wouldn’t let us do a default compare on these images, so we had to use a subimage search
  • 86. subimage searching is required to have compare search for the best match location of a small image within a larger image this option will produce two images (or two frames) the first is the "difference" image and the second will be the "match score" image
  • 87. the "match-score" image is smaller image containing a pixel for every possible position of the top-left corner of the given sub-image the search will try to compare the sub-image at every possible location in the larger image this can make sub-image searching very slow as you could guess, the smaller the sub-image is, the faster this search is
  • 88. that being said, this option doesn’t take effect unless you have two images that are different sizes this doesn’t happen often to us, so the amount it slows down our visual specs on CI is not meaningful especially since those specs aren’t tied to our main specs
  • 89. another fun thing we ran into was sometimes our screenshots were COMPLETELY different
  • 90. VS like, completely different
  • 91. and imagemagick was NOT into it in fact, it just wouldn’t even give us a diff, because the images were so different we found an option called the “dissimilarity-threshold” this threshold determined how different two images could be in order to diff them it defaulted to .2, so we upped that to 1, and it seemed to do the trick
  • 92. the only caveat to this, as you may have guessed, is doing diffs on completely different images can slow down your tests by a lot like our previous issue with sub-image searching, this doesn’t seem to happen much and since the tests are separate from our main specs, we aren’t too worried about it
  • 93. the last 3 arguments aren’t exciting, these are just where our current screenshot is
  • 94. where our master screenshot is
  • 95. and where we want the diff to save to
  • 96. STORING with aws now that we had our screenshots and diffs, we needed somewhere to throw the screenshots online and be able to grab them back out with our rails app so we decided to use aws
  • 97. aws, or amazon web services, offers cloud storage and has a ruby api so we’ll store and retrieve our screenshots from one of their buckets
  • 98. /#{commit-sha}/ #{area-of-site}/ #{page-name}/ #{image-type}.png we ended up using a naming pattern of commit-sha, area-of-site, page-name, and image-type
  • 99. /a1a1a1a/ marketing/ index/ diff.png so, for an example, we could have a commit sha of a1a1a1a where we’re in the marketing part of our site on the index page uploading the diff for that page
  • 100. diff.png current.png master.png the image-types could be current screenshot we took or the master screenshot we downloaded from S3 or the diff we made from the two screenshots
  • 101. VIEWING with the internet viewing the screenshots from an amazon bucket was far less than ideal so we decided to set up our own custom viewing page in our admin dashboard
  • 102. we created a page that listed out our current branches with their last 3 commits
  • 103. and when you would click through, it would show you, by area, all of your screenshots and diffs which was our end goal!
  • 104. THE FUTURE however, our tool is not perfect right now but it’s certainly better than what we had which was nothing, but
  • 105. MEANINGFUL TEST ASSERTIONS right now, all of our tests pass whether or not there's a diff they'll only fail if there's an issue executing the test it could be interesting to make our assertions mean something - like fail if there’s a diff we’d need to think more about that, because a diff doesn’t necessarily mean failure to us
  • 106. 0% DIFFS in the future, it’d be nice to account for 0% diffs
  • 107. VS VS like, maybe we shouldn’t upload a diff image at all if there’s no diff between the images this could save us space on our aws bucket as well as speed up our tests because we’re trying to upload fewer screenshots and it would make our admin dashboard less noisy
  • 108. PULL REQUESTS we also think it’d be nice to automatically link these to a github pull request
  • 109. so, when a diff is created, maybe automatically attach it onto its relevant pull request this one is a little tricky, because maybe we don’t want to create that much noise every time we push but it’s an idea
  • 110. DIFF BETWEEN CURRENT commit on BRANCH CURRENT commit on MASTER another thing is that we currently only diff between our current commit on a branch vs our current most recent commit on master
  • 111. master feature so, when we push a new commit to our branch, it’ll diff versus the last thing that was pushed to master this way, we know what’s changed between the feature that i’m working on and what’s currently running in production
  • 112. DIFF BETWEEN PREVIOUS commit on MASTER CURRENT commit on MASTER it would be nice if we could diff on master with the current most recent commit vs its previous commit
  • 113. master feature so that way, in case you push a visual change directly to master, you can still see a diff of it
  • 114. DIFF BETWEEN CURRENT commit on BRANCH PREVIOUS commit on BRANCH it would also be nice to see a diff between the previous commit on the current branch as well
  • 115. master feature like, if you pushed a commit to your branch that changes some stuff visually, you might want to see a diff between those two, regardless of whats happening on master
  • 116. i also would really love to get this hooked up to more browsers that would enable us to make this into a backwards compatibility tool as well as an automatic browser comparison to make sure things aren’t bonked in ie
  • 117. thanks! @jessicard anyway, that’s all i have! feel free to find me after if you want to talk more about css testing frameworks or just want to say hi! thanks for having me!