Visual regression testing: easier than you think
•
1 like•521 views
Visual regression testing is seen as something very helpful but difficult to implement/maintain. This is no longer the case with storybook. The talk will give an overview of our approach to implementing visual regression testing for JIRA and mention some of the pitfalls to be aware of. There is also an option to delegate it if you don't want to own the service.
Recommended
More Related Content
Similar to Visual regression testing: easier than you think
Similar to Visual regression testing: easier than you think (20)
Recently uploaded
Recently uploaded (20)
Visual regression testing: easier than you think
- 1. ALEXEY SHPAKOV Visual regression testing easier than you think
- 3. Visual regression testing gives you confidence there are no unintended changes. WHY BOTHER?
- 4. Complicated setup Flakiness High maintenance Slow Y u no visual regression testing?
- 5. Simple setup Isolation Low maintenance Fast Y u ?
- 7. IT’S BEEN IN THE AIR voice.kadira.io/snapshot-testing-in-react-storybook-43b3b71cec4f
- 8. How do we approach it?
- 9. A separate build to collect urls Plan of actions Get stories urls Use urls for visual regression testing const getUrl = (kind, name) => "/iframe.html?selectedKind=" + encodeURIComponent(kind) + "&selectedStory=" + encodeURIComponent(name); const storyUrls = storybook.getStorybook().reduce( (agg, { kind, stories }) => { const urls = stories.map( ({ name }) => getUrl(kind, name) ); return [...agg, ...urls]; }, []
- 10. Generate test casesPlan of actions Get stories urls Use urls for visual regression testing storyUrls.forEach(url => { vrLibrary.suite(‘default', (suite) => { suite .setUrl(url) .setCaptureElements('html') .capture('default'); }); });
- 11. JIRA Frontend VR testing
- 12. images in 1000 seconds
- 13. PhantomJS /garris/BackstopJS Visual regression testing library Electron /segmentio/niffy Webdriver /gemini-testing/gemini
- 14. Visual regression testing is optional Visual regression testing is obligatory STORE SCREENSHOTS DO NOT STORE SCREENSHOTS
- 15. No interaction Don’t perform any webdriver interaction - create another story instead Useful tips Skip stories Some stories are not fit Stop animations Any inconsistency fails visual regression testing
- 16. Automated dependencies bumps Future To be continued…Automatically perform visual regression tests for every pull request
- 17. IF YOU DON’T WANT TO OWN IT percy.io screener.io
Editor's Notes
- Hi, my name is Alexey Shpakov and I work in JIRA Frontend Platform team. I am going to talk about visual regression testing.
- The idea behind it is simple: take a baseline screenshot of your component, perform changes, take another screenshot and compare it against the baseline.
- As any other type of testing, visual regression testing is there to increase confidence. Confidence that your change in one place has not broken anything completely unrelated in another. Confidence, that your component library did not introduce any breaking changes in the latest patch release. Confidence, that you have understood and addressed all the consequences of bumping a major version of that same library.
- If visual regression testing is so helpful, why so few people actually perform it? Often times visual regression testing is associated with end-to-end webdriver tests with all the corresponding drawbacks: 1. It is complicated to setup fresh test environment for every run 2. Webdriver tests are known for their flakiness and nobody believes flaky tests 3. The maintenance costs are high: you need to update your tests every time front end changes 4. Not to mention the slowness
- On the other end of the spectrum we have storybooks. They seem to be a complete opposite of end-to-end tests: 1. They are easy to setup - you already have a webserver that serves your stories, all you need to provide is render code 2. Isolation - you can simply stub all the data your components require, hence, no need for a server and AJAX calls or dependency on other components 3. Low maintenance - stories are just another consumer of component APIs, once you add static typing, there is no way you might ever forget to update them 4. Fast - they are pretty fast, as you don’t need to start the whole application to run them In general, storybooks allow you to render components in any state. It is helpful for development and for showcasing components' capabilities. If you don't use storybooks already, you definitely should.
- Storybooks allow us to render all components in their various states using a dedicated webserver... Why can't we use that for vr testing?
- Apparently, I am not the first to think about it. In fact, one of storybook developers suggested that option a year ago. Unfortunately, it has never landed in storybook. Let’s assume we have a visual regression library at hand. What do we need to make it work with storybook?
- The first naive idea that comes to mind is to write a script that will click through the stories on the left and take a screenshot on the right. That’s something. The obvious drawbacks is the webdriver flakiness. Just imagine clicking through a thousand stories - what could potentially go wrong? Another issue is this approach is difficult to run in parallel. As it happens, the iframe on the right has a well-defined url you can use. You can even generate a correct url for it given story kind and story name.
- The question is, how do we get those? We created a separate a separate build for that.
- Once we have all the stories and their kinds, we can generate test cases for our visual regression testing tool. The fact we generate our tests based on existing stories means we don’t need to maintain vr tests project separately.
- I’d like to share our journey to visual regression testing.
- At first, we looked at PhantomJS-based solution. It’s a headless browser and everybody uses it to test pages. There is thing about it, nobody tells you though. Which browser is it based on? It is an ancient version of QtWebkit. It took me a couple of hours and a number of shims just to make it render a page in our storybook… only to realize it does not support the css we use. Fortunately, BackstopJS we used supports Firefox-based engine as well. Switching to it helped a lot. Not headless anymore, but at least it works. The next question that popped up was to check the browser distribution among our customers. I haven’t spent much time with Niffy as the next concern was about poor IE users. Even though the majority of customers use Chrome, we should still care about IE. So, we’ve moved to the dreaded Webdriver-based solution. And it worked alright: 1. It allows to test in Chrome - most widely used browser. 2. It also allows to test in other browsers (IE) 3. It allows to offload testing to a third-party, e.g. Browsersrack or Saucelabs. 4. You can run them in parallel almost for free from within docker container.
- This is a very controversial matter. When you store screenshots, you save time for generating a baseline. It also allows you to have nice diffs in pull requests and ensures every change doesn’t go unnoticed. At the same time it bloats your codebase, changes dev loop and requires developers to regenerate the screenshots all the time. Not to mention merge conflicts. We’ve joined the dark side and decided to have visual regression testing as an optional tool.
- During our use of visual regression testing we aggregated a couple of best practices. 1. stop animation - an obvious one, once you have any animation, chances are you are never going to get a pass for your visual regression test 2. no interaction - interaction causes flakiness. If there is a number of actions you need to perform in order to put your component into a state for a screenshot, why don’t you create a story with that state to begin with? 3. add possibility to skip stories - some stories might simply be not fit for visual regression testing
- We are only in the beginning of our journey with visual regression testing. In future we intend to perform visual regression testing for all the pull requests created using CI. Also, we intend to perform automated dependencies bumps and having visual regression testing is crucial for that.
- If you want all this awesomeness but don’t want to maintain it, there are paid services available online that can help you.
- In conclusion, I suggest you give visual regression testing a spin. Who knows, maybe it will prevent a bug or two in production ;).