Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Technical Tips: Visual Regression Testing and Environment Comparison with Backstop.JS


Published on

As a Front End Web Developer, experimenting with new tools to add to your workflow (and going down the rabbit hole with them!) is all part and parcel of refining your craft. Chris Eccles, Technical Manager at Building Blocks has been doing just this and has some invaluable insight into CSS Visual Regression using Backstop.JS.

CSS Visual Regression testing is the process of running automated visual test comparisons on pages or elements in your projects. Using Backstop.JS, Chris has discovered that this tool is intuitive, allowing quick configuration to allow you to get up and rolling quickly.

Backstop.JS serves your tests via a webpage which gives you the visual feedback needed for targeting bugs caused from CSS related issues. These comparisons can uncover bugs you’d otherwise not learn about until it’s too late. A very useful tool to have in your Front End arsenal, wouldn’t you agree?
Chris has been sharing his insights with the BB team and wanted to share with our blog readers also. So, sit back and enjoy the ride through the wonderful world of Backstop.JS.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Technical Tips: Visual Regression Testing and Environment Comparison with Backstop.JS

  1. 1. Backstop JS Visual Testing
  2. 2. 1 I would define visual testing as using a headless browser to capture and compare images of components of a website. This can be used to implement unit tests for how the CSS is rendering the HTML in a browser. Not intended to test how the HTML is generated or replace Unit tests in a back end service layer. Not intended to replace cross browser testing. Allows comparison between sites or comparing against a reference image captured form the same site before changes were made. What do you mean by Visual Testing ?
  3. 3. 1. Chrome Headless 2. Setup and Examples 3. What to test 4. Setup in your project Contents :
  4. 4. Chrome Headless 3 What has changed ? Why is this a big deal ?
  5. 5. 4  Headless Chrome is shipping in Chrome 59. It's a way to run the Chrome browser in a headless environment. Essentially, running Chrome without chrome! It brings all modern web platform features provided by Chromium and the Blink rendering engine to the command line. Why is that useful? A headless browser is a great tool for automated testing and server environments where you don't need a visible UI shell. For example, you may want to run some tests against a real web page, create a PDF of it, or just inspect how the browser renders an URL. Announcement blog : How to Get it: As long as you have a version of chrome installed greater than Chrome 59 – ( Chrome 61 on Windows) you already have it  Chrome Headless
  6. 6. 5  BackstopJS automates visual regression testing of your responsive web UI by comparing DOM screenshots over time. Backstop is a open source project to run visual tests using headless browsers to capture screenshots. Originally ran using either PhantomJS or SlimerJS headless browser libraries. These were implementations of WebKit (Chrome) or Gecko (Firefox). From release v3.0.19 this now supports Chrome headless. The latest release has a lot of stability improvements v3.0.27 Git Hub URL : Features : • Allows capture and comparison of screenshots form headless browser • Excellent comparison tool to identify and highlight differences • Allows setup of multiple breakpoints to test responsive sites • Uses simple CSS selectors to identify what to capture • Pre capture scripts can be enabled to be ran Backstop JS
  7. 7. 6 Chrome is based on WebKit but using PhantomJS as a headless browser caused issues making it unusable. There would be styling issues that would be thrown that would not be present on the site when viewed through chrome. As visual testing is about how the site renders this causes the tests to be unreliable. When I investigated implementing visual testing this was the stumbling block each time. Chrome != Webkit
  8. 8. Setup and Examples 7
  9. 9. 8 • backstop init : Set up a new BackstopJS instance -- specify URLs, cookies, screen sizes, DOM selectors, interactions etc. (see examples directory) • backstop test : BackstopJS creates a set of test screenshots and compares them with your reference screenshots. Any changes show up in a visual report. (Run this after making CSS changes as many times as needed.) • backstop approve : If the test you ran looks good, then go ahead and approve it. Approving changes will update your reference files with the results from your last test. Future tests are compared against your most recent approved test screenshots. The BackstopJS workflow
  10. 10. 9 backstop init
  11. 11. 10 backstop test (initial)
  12. 12. 11 backstop approve
  13. 13. 12 backstop test
  14. 14. 13 { "id": "backstop_default", "viewports": [ { "label": "phone", "width": 320, "height": 480 }, { "label": "tablet", "width": 1024, "height": 768 } ], "onBeforeScript": "chromy/onBefore.js", "onReadyScript": "chromy/onReady.js", "scenarios": [ ………… ], "paths": { "bitmaps_reference": "backstop_data/bitmaps_reference", "bitmaps_test": "backstop_data/bitmaps_test", "engine_scripts": "backstop_data/engine_scripts", "html_report": "backstop_data/html_report", "ci_report": "backstop_data/ci_report" }, "report": ["browser"], "engine": "chrome", "engineFlags": [], "asyncCaptureLimit": 5, "asyncCompareLimit": 50, "debug": false, "debugWindow": false } Default Configuration { "label": "BackstopJS Homepage", "cookiePath": "backstop_data/engine_scripts/cookies.json", "url": "", "referenceUrl": "", "readyEvent": "", "readySelector": "", "delay": 0, "hideSelectors": [], "removeSelectors": [], "hoverSelector": "", "clickSelector": "", "postInteractionWait": "", "selectors": [], "selectorExpansion": true, "misMatchThreshold" : 0.1, "requireSameDimensions": true }
  15. 15. 14 On the CIPD PM build we have incorporated it in to the HTML dev process. When components are created a new test scenario is added to the backstop tests. This captures the components from the locally setup demo site. This creates a reference image for that component that is then checked in to source control. Project Setup
  16. 16. What to Test 15
  17. 17. 16 Visual regression testing is the process of creating reference images for each component as they are created. This allows a comparison over time to monitor changes. Ran by the developer on their local development environment after changes made to ensure no regressions issues have occurred due to their changes. Why is that useful? • This allows the developer greater test coverage than they could do manually when ensuring their changes have not had regression impact on other components. • Allows greater detail comparison than possible reviewing the site manually. E.g. If the font size is increased by 1px then it will be caught which could easily be missed. • Allows developers without a full understanding of the project have a safety net about what the site should look like before their changes. Visual Regression Testing
  18. 18. 17 Using visual testing to compare different environments involves comparing one site against another . The reference images are not stored beyond the test. A reference image set is generated form one site and then another site is compared against them Why is that useful? • To ensure that the HTML created has been integrated within the website correctly and is rendering out in the same way. This highlights mistakes with the integration or conflicting integrations that are not present on the HTML Demo site. • Running regression tests against the Staging and Live instances of a site to see where the changes have occurred. • When setting up comparison between a demo and dev site powered by a CMS this can make the tests brittle so the tests pages need to be isolated from changes in content as possible. Environment Comparison
  19. 19. 18 Within a site their are hover states and interactive components. These need the mouse or user interactions to trigger them. How to test these in a headless browser? Backstop allows scripts to be ran before a screenshot is captured. Using the chromy API you can move the mouse and manipulate the page through script. This allows testing of : • Field entry • Hover states • Interactive items • Video loading – Playing • Ajax content loading Chromy Git Hub URL : Interacting with the site
  20. 20. 19 When capturing items it is much quicker to capture components on a page than the whole page to compare them • It is much quicker to compare the smaller images • It is more relevant the items that are being tested. This allows the components themselves to be bested in isolation and most of our development work is done using component based development. Smaller is better
  21. 21. Setup in your project 20
  22. 22. Process to include in you project 21 Install Backstop init within you project Create scenarios Run Baseline generation Keep running and monitoring
  23. 23. 22 Prerequisites • Node – (If you don’t already have it) • Chrome - (61 needed but latest is 62.0.3202.62 at the moment) • Update node to v8+ • Update npm to v5+ On Windows you will also need to have python installed > PhantomJS needs Python -- please make sure you have Python installed... see Installation • npm install -g backstopjs • Installs backstop globally This allows the backstop command to be ran from anywhere on the PC. Install Backstop - Globally
  24. 24. 23 Backstop can also be installed as a grunt dependency and called through grunt. var backstopjs = require('backstopjs'); grunt.registerTask('backstop', 'BackstopJS integration', function(cmd) { // cmd is either 'reference', 'test', or 'openReport' var done = this.async(); backstopjs(cmd).then(function() { done(true); }).catch(function() { done(false); }); }); This allows it to be integrated in to the existing grunt tasks. Including it in grunt also allows the dynamic creation of the scenarios using variables instead of them being held in a JSON file. Installing Backtop in Grunt
  25. 25. 24 There is a basic backstop.json file that is created when you run backstop init. This can be edited to add in your own scenarios. This is an example of the setup we are using. Key Points: • The Viewports are defined for use in all tests • resembleOutputOptions define the comparison colours and display style • The asyncCaptureLimit and asyncCompareLimit are used to control the number of threads and memory the tests use. • "engine": "chrome“ defines that the chrome headless browser will be used. Configuration Setup { "id": "pm", "viewports": [ { "label": "phone", "width": 320, "height": 1024 }, { "label": "tablet", "width": 768, "height": 1024 }, { "label": "desktop", "width": 1268, "height": 1024 } ], "onBeforeScript": "chromy/onBefore.js", "onReadyScript": "chromy/onReady.js", "scenarios": [ { …….. } ], "resembleOutputOptions": { "errorColor": { "red": 255, "green": 0, "blue": 255 }, "errorType": "movement", "transparency": 0.3, "ignoreAntialiasing" : true }, "asyncCaptureLimit": 25, "asyncCompareLimit": 400, "engineFlags": [], "engine": "chrome", "report": ["browser","CI"], "debug": false, "debugWindow": false }
  26. 26. 25 For scenarios setup there are two key parameters : • url • selectors These define the page the what will be captured on the page. The selectors used simple CSS selectors to identify elements on the page to capture. To capture the contents of the <div id=“header”> then the #header is used as a selector. If a referenceUrl is specified then this will be used instead of the url when generating the reference images. This is used when comparing sites. The hideSelectors parameter is used to define the elements on the page that should not be shown. This is useful to hide cookie notifications for example that could obscure capture of components. Scenario Setup { "label": "header", "url": "http://people-management.local/bb-prototype- website_example_grid.html", "selectors": [ "#header" ], "hideSelectors": [ "#monitor_mq", ".grid", ".visible-grid-btn" ], "readySelector": ".page-loaded“ }, { "label": "podcast", "url": "", "referenceUrl": "http://internal.people- website_components-podcast.html", "selectors": [ ".podcast-container" ], "hideSelectors": [ "#monitor_mq", "#SEStartButton", "#cookieMsgWrapper2", ".grid", ".visible-grid-btn" ] }
  27. 27. 26 Scenario Setup : "onReadyScript": "burgerMenuTest.js", Chromy Script : module.exports = function (chromy, scenario) { var clickSelector = scenario.clickSelector; var postInteractionWait = scenario.postInteractionWait; // selector [str] | ms [int] if (clickSelector) { chromy .wait(clickSelector) .click(clickSelector) } if (postInteractionWait) { chromy.wait(postInteractionWait); } }; Chromy
  28. 28. Backstop JS the future ….. 27 • Catch Regression • Allow comparison between DEMO HTML and Website • Quick to setup • Easily integrate in to existing HTML build process