More Related Content
What's hot
What's hot (20)
Similar to Visual Regression Testing: In search of an Ember solution
Similar to Visual Regression Testing: In search of an Ember solution (20)
Recently uploaded
Recently uploaded (20)
Visual Regression Testing: In search of an Ember solution
- 1. Visual Regression Testing In search of an Ember solution
- 2. Who am I ❖ Lisa Backer, Senior Software Engineer at DockYard ❖ Primary work is with Ember ❖ In web development for almost 20 years ❖ Running, dancing, volunteer waaay too much ❖ Bringer of Muppets to all code repositories Twitter: @eshtadc Github: eshtadc Discord: eshtadc Email: eshtadc@gmail.com
- 3. Testing Basics I love testing!!!
- 5. A Story of Passing Tests
- 6. A Story of Passing Tests
- 7. A Story of Passing Tests
- 8. A Story of Passing Tests
- 9. What do I want?
- 10. What do I want?
- 11. What do I want? ❖ Visual inspection to be done automatically with my tests ❖ To include responsive viewport testing ❖ To include cross browser testing ❖ To catch miscellaneous errors that I may not have had tests for in the first place
- 12. How do I get it?
- 13. Visual Regression Testing 1. Performs user-interface (UI) regression test by... 2. Capturing the screenshots of a particular state of UI and... 3. Compares them with baseline images of the same state of the UI ❖ manual process or automated (CI) ❖ master branch vs. pull request ❖ production environment vs. staging environment
- 14. You will need: ❖ Environment to run tests that is capable of taking screenshots ❖ Image diff tool to compare those screenshots to some baseline ❖ Ideally also a reporting interface accessible to humans (GUI) and CI Visual Regression Testing
- 15. So there are tons of headless browsers and image diff libraries, why not roll your own solution?
- 16. Then make sure you address: ❖ Animations/Dynamic data ❖ Image load times ❖ Approval workflow ❖ How to capture full page versus component ❖ Waiting for client-side rendered content ❖ Speed of screen capture and diff comparison ❖ How to integrate with your existing testing tools
- 17. So what's out there? ❖ Outdated and abandoned projects for every time a new tool came out and someone decided to roll their own ❖ Specialized projects for specific frameworks and tools ❖ Open source "loosely documented" solutions ❖ Paid services
- 18. What should I look for? ❖ What browsers are important to you? ❖ Selenium based approach allows cross-browser testing ❖ Headless approaches available for REAL browser SlimerJS (Firefox), Headless Chrome, Trifle (IE) ❖ Why write for browsers that your users will never open (Phantom - I'm looking at you)
- 19. What should I look for? ❖ Reliability and Performance ❖ Image capture and diff comparison is time consuming and performance intensive ❖ Slows down test suite mid-run ❖ Flexibility ❖ Configuration file + command line interface = ultimate flexibility?
- 20. Service Solution: Percy ❖ Solve the workflow and integration problems ❖ Image generation done on their servers so pain point is removed from your tests ❖ Support for multiple viewports and browsers (Chrome/Firefox) ❖ Comparison UI is slick ❖ Easy Ember integration
- 21. Service Solution: Percy ❖ Can sign up for a free trial at percy.io (only two weeks) ❖ Designed to integrate with tests run in CI generally by adding a few variables (token and project) to your existing setup ❖ Designed to integrate with PR workflow
- 22. Service Solution: Percy Creates PR Watching Builds application and runs tests using environment variables Sees Percy token/project and sends DOM snapshots to Percy Updates check in Github repo
- 23. Service Solution: Percy ❖ Easy ember integration with ember-percy addon ❖ Tell the addon when to snag a screenshot and it will be stored for comparison (optionally limited to an element on the page) ❖ Percy only grabs the DOM representation and then re-renders it on their servers for screen generation so the addon footprint is minimal DEMO TIME, Sorta.... https://github.com/eshtadc/super-rentals/tree/percy-regression
- 24. Installing Percy A demo of screenshots
- 25. Percy Account
- 26. Percy Account
- 27. Percy Account
- 30. Miscellaneous Add users Link project to repository for PR integration Review pricing plans Select browsers to use for tests
- 32. Test results
- 33. Why not use it?
- 34. Why not use it? ❖ Privacy - does not provide a hosted solution so your pre-release test images are transported across the Internet and stored on Percy's servers ❖ If these are not a concern to you - I completely recommend it. ❖ Super-easy integration ❖ Easy to train your team ❖ Easy to maintain ❖ Excellent response time from support
- 35. If you DO use it? ❖ Limit the repositories watched explicitly within GitHub ❖ Make sure you really need all those viewports ❖ You can limit tests to specific viewports within test helper ❖ Make sure you really need both browsers ❖ If you are testing in both Chrome/Firefox that is double the impressions ❖ Use tests more as a smokescreen and less for testing every possible permutation
- 36. Other Ember Solutions ❖ ember-visual-test ❖ Integrates well within tests ❖ Is entirely local ❖ Has some custom commands for CI usage
- 37. ember-visual-test ❖ Custom asynchronous test helper to "capture" a URL ❖ Makes a call out to the URL and then takes a screenshot ❖ Can pass custom query parameters, limit to a selector, and add delay ❖ This means you are waiting for an additional page load each time you call the test ❖ No way to interact with the page and/or take a screenshot mid- interaction
- 38. ember-cli-kimchi ❖ Integrates with ember-cli-deploy to provide uploading reference images to s3 storage ❖ Tightly coupled to PhantomJS ❖ Only had a single release and hasn't been updated since :( ❖ Also abandoned: ember-cli-visual-acceptance
- 39. Beyond Ember: BackstopJS ❖ Active open source project with almost 100 contributors ❖ Can work with headless Chrome, Phantom or Slimer (Firefox) ❖ Interactions with Puppeteer, ChromyJS, or CasperJS ❖ Includes a comparison GUI ❖ Considerations for source control, CI, Docker ❖ Allows for parallel test runs
- 40. BackstopJS ❖ Relies on an extensive configuration file that defines "scenarios" to test ❖ Handles waiting for client-rendered content out of the box ❖ Provides scripting capabilities to set up the screenshot you want to take (allowing captures mid-interaction) DEMO TIME https://github.com/eshtadc/super-rentals/tree/backstop
- 41. BackstopJS
- 42. BackstopJS: Ember? ❖ ember-cli-backstop has been abandoned but basically allows you to set up basic configuration and exposed backstop commands via ember-cli ❖ Backstop allows the configuration to be read from a file, but also to be passed into node as a parameter. We could: ❖ integrate with `ember test` ❖ dynamically build a configuration anytime a test helper is encountered ❖ run the test comparison at the end of the suite
- 43. ❖ Create reference images and check into source control ❖ When changes are made: ❖ Run a test to compare against reference images ❖ Fix any unwanted changes ❖ Approve the remaining changes --> overwrites references ❖ Pull request runs backstop in CI mode to ensure this process of reconciliation has taken place BackstopJS: CI?
- 44. A naive attempt BackstopJS: CI?
- 45. ❖ Must have ember application running while tests images are generated BackstopJS: CI?
- 46. ❖ We need Ember to still be running when our backstop command is executed ❖ Ideally we wouldn't trigger an additional Ember build ❖ Backstop allows a CI style reporting that returns 0 and non-zero exit codes as expected by CI via node ❖ Testem allows custom node processes to be run along with your configured browser BackstopJS: CI?
- 48. BackstopJS: CI? ❖ The location of the test URL changes from local running to CI ❖ Backstop configuration is a JSON file (which cannot read environment variables or do anything very cool) ❖ Maybe we can manipulate the options going into `ember test` to specify a known location for the host and port and use that in the backstop.json file and somehow make it work locally and in CI... https://thumbs.gfycat.com/FixedAthleticDevilfish-max-1mb.gif
- 49. Emberifying Backstop: Bivariate ❖ Open source project ❖ "An opinionated interface for writing, running, and saving BackstopJS tests" ❖ "Bivariate's goal is to allow for an approachable Visual Regression Testing suite that can be organized to accommodate small and large projects without overwhelming complexity." ❖ Small but actively under development
- 50. Bivariate ❖ Uses node's module system to break apart the large unwieldy JSON file used by Backstop into files written in JavaScript ❖ __config-baseURLs.js: Base URLs for all references and tests ❖ __config-common.js: Common config values that work across tests ❖ __config-viewports.js: Configure the screen resolutions to test ❖ Test files (either single tests prefaced with an "_" or test groups in a single file) ❖ Engine scripts: scripts that should be run before or when a test is ready
- 51. Bivariate Converting tests to Bivariate DEMO TIME https://github.com/eshtadc/super-rentals/tree/bivariate
- 52. Bivariate
- 53. Course correction ❖ Bivariate CI usage is pretty non-existent ❖ Possibly best left as a source of inspiration ❖ Keep coming back to the dynamic configuration accepted into BackstopJS
- 55. ❖ Basic users: Create a test helper that can accept or derive what is necessary to generate a scenario for given file location and element ❖ Advanced users: Allow for detailed scenario files ❖ At the end of the test suite begin the BackstopJS run with a dynamic configuration ❖ Bonus: Include an option in the GUI to approve changes to overwrite the reference files when not running within a CI ❖ Require passing regression tests for CI to complete (included) My Dream World
- 57. Resources ❖ https://github.com/mojoaxel/awesome-regression-testing ❖ Mike Fotinakis (Percy): https://www.youtube.com/watch?v=HHIFnRCiR30 ❖ https://visualregressiontesting.com/ ❖ https://percy.io/ ❖ https://github.com/garris/BackstopJS ❖ https://github.com/jparkerweb/bivariate ❖ https://github.com/eshtadc/super-rentals/
- 58. © 2018 DockYard, Inc. All rights reserved. Thank You!DockYard.com | @DockYard Twitter: @eshtadc Github: eshtadc Discord: eshtadc Email: eshtadc@gmail.com
Editor's Notes
- The obvious need and first request is that changing CSS can have unintended consequences.
- The obvious need and first request is that changing CSS can have unintended consequences.
- Started this exploration as part of DY days
- This comparison results in a third “diff” image to allow visually inspecting differences. Diffs can calculate based on a threshold of differences. Baseline images can be checked into source control or integrated into the CI process in a number of ways. This makes “visually” checking for changes as commonplace and automated as running any other automated tests prior to merging.
- If you are going to do that you'll need to address And of course you must have a mock data implementation in place regardless of visual regression test.
- One smaller list of only modern tools listed 30+ tools, 9 browser automation tools, and 14 paid services
- Important to note that we're not talking about comparing screenshots across different browsers to ensure pixel perfection, but rather ensuring that things within a specific browser don't suddenly break (accidentally break IE ever?)
- Image capture has been around forever and whether using ImageMagick or more modern libraries like resemble.js still involves a big performance hit.
- They have also solutions in place for the common issues of animation, ignoring specific css, etc.
- All of this is basically done for you. All you have to do is add a few variables, install an add-on, and take a snapshot whereever it works best for you in your existing tests (or make some new ones).
- ember-cli-kimchi: From Blue Apron
- They have also solutions in place for the common issues of animation, ignoring specific css, etc.
- Bivariate says it requires headless chrome, I need to investigate further into whether it will just use the configuration from backstop which allows more browsers.