Download to read offline
Scaling up your Snapshot tests, without the friction
- 1. Arn old Noron h a arn old @ scre en sh otb ot.Io Scaling up your Snapshot Tests, without the friction
- 2. Quick refresher • Whatis snapshot testing? • Well known libraries: pointfreeco/swift-snapshot-testing • Libraries to convert SwiftUI previews to snapshots (e.g. doordash-oss/swiftui- preview-snapshots)
- 3. screenshot-test or snapshot-test? Androiddevs call it “screenshot” (usually) iOS devs call it “snapshot”. Blame me :( https://github.com/facebook/screenshot-tests-for-android (~ 2015) Flutter calls it “golden tests”, Web calls it “visual regression tests” ( ° °) ╯ □ ╯︵ ┻━┻
- 4. But the principlesare the same Every platform hits the same issues with scaling up
- 5. Why teams giveup on snapshot tests •Storage: where do you store the snapshots? •Flakiness: laptops might not match CI machines • OS version, Xcode version, ARM/Intel etc. •Friction: manually re-record snapshots •Education: Everyone needs to know how to re-record •Can’t demonstrate value
- 6. Storage •A single snapshotis typically ~50kB. •1000 snapshots is ~50MB •… across 100 UI changes is ~5GB •GitHub’s max repository size is 2GB •Git LFS can help, but adds DevOps overhead
- 7. Better tools todemonstrate value • Snapshot tests are the rare test that is understood by Managers, PMs, designers etc. • We need a better way to be able to share screenshots and notify of changes that is not in Git • Demonstrate value = more tests!
- 8. Let’s make snapshot testingbetter!
- 9. Let’s solve Storage •Don’t store images in Git/Git LFS • You could use Amazon S3, or a dedicated service like Screenshotbot • Might need in-house tooling if you choose S3
- 10. Let’s solve Flakiness •Always generate images in CI! • Easy to ensure CI machines are on the same OS version, CPU etc. • Typically generates 100% identical images
- 11. Let’s reduce theFriction • 99% of snapshot changes are intentional, not regressions • When a change happens, just “Accept” the change • Developers love the workflow, and you don’t need to teach anybody how to re-record screenshots!
- 12. Some companies buildthis in-house • And it can be done! • Google, Meta, Uber, AirBnb, etc. have built in-house systems like this. • But you don’t have to start from scratch …
- 13. Consider using Screenshotbot •Fully open-source, https://github.com/screenshotbot/screenshotbot- oss • Trusted by both small startups, and large enterprises such as ForeFlight (part of Boeing) and Sky UK among others.
- 14. Get notifications directlyon Pull Requests
- 15. Get better imagecomparison tools
- 16. And more… • Historyof screenshots • Bisect regressions • Compare releases • “mask” out animations/flakiness
- 17. I’d love tochat more! • If you don’t already do snapshot tests, try adding a snapshot test today to see how you like it! • If you already do snapshot tests, consider investing in your developer tooling to allow scaling up your snapshot testing
- 18. Questions? Arn old Noron h a arn old @ s cre en s h otb ot.io h ttp s: / / sc re e n sh otb ot. io
Editor's Notes
- #1 Hello I’m Arnold Noronha, I’m the founder of Screenshotbot. Today I’d like to talk to you about how you can scale up your snapshot testing.
- #2 But before that, show of hands: how many of you know what snapshot testing is? Okay, great.. How many of you use snapshot testing in your day-to-day work? Not bad, how many of you are working with more than 1000 snapshots in your code base? Excellent. Okay for those who don’t know what snapshot testing is, it’s a fast “unit test”: it sets up a view (either UIKit or SwiftUI), but unlike a unit test, you don’t simply write an assertion. Instead you render the view into a PNG file that is saved somewhere, typically in your Git repository. So now anytime your rendering changes, your tests will fail and ask you to re-record those snapshots. Now this isn’t new, (->) in fact there are some well known libraries, such as Pointfree’s swift-snapshot-testing that does this for you pretty well. (->) there are also libraries that will just convert all your SwiftUI previews into snapshot tests magically. So you get a lot of coverage for free.
- #3 First start with saying, “You’ll see some authors claiming that screenshot test is a type of snapshot test etc. but it’s not the way people actually use these terms in language” Blame me for this. I’m originally an Android dev, and my claim to fame is building the first snapshot testing library on Android. At that point, iOS already had snapshot tests, but I chose to call it screenshot tests on Android. That name has stuck ever since on Android. If you really want to know why I chose to call it “Screenshot tests”, it’s a pretty dumb reason, and has to do with laziness and trying to hit deadlines.
- #10 Remember,
- #16 History of individual screenshots