1. Snapshot testing
with native mobile frameworks

2. @alter_al
in/apesotskiy
medium.com/xcnotes
2

3. Contents
1. iOS Snapshot Testing
2. Android Screenshot Testing
3. Challenges
3

4. Snapshot testing
4

5. Tool Area Lang Notes
applitools
XCTest, Espresso,
Appium, Calabash
Any
Polyglot,
$$$
screenshot-tests-for-
android
Espresso Java, Kotlin
Support multiple devices,
lack of reports
shot Espresso Java, Kotlin
Cool reports,
support one device
ios-snapshot-test-case XCTest obj-C, Swift
Really flexible,
poor docs
swift-snapshot-testing XCTest obj-C, Swift
Multiformat,
lack of reports
Tool market
5

6. 6

7. Introduction
7

8. How it worked
TestClass
XCTestCase
8

9. How it works
iOSSnapshotTestCase FBSnapshotVerifyView( )FBSnapshotVerifyLayer( )
TestClass
XCTestCase
9

10. Snapshot
How it works
UIView
CALayer view.layer.sublayers
view.layer
UIImageView(XCUIScreenshot().image)
ViewController().view.subviews
ViewController().view
10
XCUITest

11. target "SampleUITests" do
use_frameworks!
pod 'iOSSnapshotTestCase'
end
✓ add an additional pod in Podfile:
set the environment variables in our test scheme:
Precondition
• FB_REFERENCE_IMAGE_DIR
• IMAGE_DIFF_DIR
11

12. Usage
12
20 48

13. Usageimport XCTest
import FBSnapshotTestCase
class SnapshotUITests: FBSnapshotTestCase {
override func setUp() {
super.setUp()
recordMode = true
XCUIApplication().launch()
}
func test_Snapshot() {
let score = XCUIApplication().staticTexts["score"]
let board = XCUIApplication().otherElements["board"]
let scoreImage = score.screenshot().image
let fullscreen =
XCUIApplication()screenshot().image.fill(element: board).removingStatusBar
FBSnapshotVerifyView(UIImageView(image: fullscreen), identifier: "fullscreen")
FBSnapshotVerifyView(UIImageView(image: scoreImage), identifier: "score")
}
}
XCTest
XCUITest
13

14. Usageimport XCTest
import FBSnapshotTestCase
class SnapshotUITests: FBSnapshotTestCase {
override func setUp() {
super.setUp()
recordMode = true
XCUIApplication().launch()
}
func test_Snapshot() {
let score = XCUIApplication().staticTexts["score"]
let board = XCUIApplication().otherElements["board"]
let scoreImage = score.screenshot().image
let fullscreen =
XCUIApplication()screenshot().image.fill(element: board).removingStatusBar
FBSnapshotVerifyView(UIImageView(image: fullscreen), identifier: "fullscreen")
FBSnapshotVerifyView(UIImageView(image: scoreImage), identifier: "score")
}
} 14

15. Usageimport XCTest
import FBSnapshotTestCase
class SnapshotUITests: FBSnapshotTestCase {
override func setUp() {
super.setUp()
recordMode = true
XCUIApplication().launch()
}
func test_Snapshot() {
let score = XCUIApplication().staticTexts["score"]
let board = XCUIApplication().otherElements["board"]
let scoreImage = score.screenshot().image
let fullscreen =
XCUIApplication()screenshot().image.fill(element: board).removingStatusBar
FBSnapshotVerifyView(UIImageView(image: fullscreen), identifier: "fullscreen")
FBSnapshotVerifyView(UIImageView(image: scoreImage), identifier: "score")
}
} 15

16. Usageimport XCTest
import FBSnapshotTestCase
class SnapshotUITests: FBSnapshotTestCase {
override func setUp() {
super.setUp()
recordMode = true
XCUIApplication().launch()
}
func test_Snapshot() {
let score = XCUIApplication().staticTexts["score"]
let board = XCUIApplication().otherElements["board"]
let scoreImage = score.screenshot().image
let fullscreen =
XCUIApplication()screenshot().image.fill(element: board).removingStatusBar
FBSnapshotVerifyView(UIImageView(image: fullscreen), identifier: "fullscreen")
FBSnapshotVerifyView(UIImageView(image: scoreImage), identifier: "score")
}
} 16

17. Usageimport XCTest
import FBSnapshotTestCase
class SnapshotUITests: FBSnapshotTestCase {
override func setUp() {
super.setUp()
recordMode = true
XCUIApplication().launch()
}
func test_Snapshot() {
let score = XCUIApplication().staticTexts["score"]
let board = XCUIApplication().otherElements["board"]
let scoreImage = score.screenshot().image
let fullscreen =
XCUIApplication()screenshot().image.fill(element: board).removingStatusBar
FBSnapshotVerifyView(UIImageView(image: fullscreen), identifier: "fullscreen")
FBSnapshotVerifyView(UIImageView(image: scoreImage), identifier: "score")
}
} 17

18. Usageimport XCTest
import FBSnapshotTestCase
class SnapshotUITests: FBSnapshotTestCase {
override func setUp() {
super.setUp()
recordMode = true
XCUIApplication().launch()
}
func test_Snapshot() {
let score = XCUIApplication().staticTexts["score"]
let board = XCUIApplication().otherElements["board"]
let scoreImage = score.screenshot().image
let fullscreen =
XCUIApplication()screenshot().image.fill(element: board).removingStatusBar
FBSnapshotVerifyView(UIImageView(image: fullscreen), identifier: "fullscreen")
FBSnapshotVerifyView(UIImageView(image: scoreImage), identifier: "score")
}
} 18

19. Usage
19
Fullscreen Snapshot
Score Snapshot
Original Screen

20. Usageimport XCTest
import FBSnapshotTestCase
@testable import Product_Module_Name
class SnapshotUITests: FBSnapshotTestCase {
override func setUp() {
super.setUp()
recordMode = true
}
func test_Snapshot() {
let game = NumberTileGameViewController(dimension: 4, threshold: 2048)
game.board?.reset()
game.board?.insertTile(at: (1, 1), value: 2)
game.view.subviews.last!.label.text = "12345678"
FBSnapshotVerifyView(game.view, identifier: "wholeView")
FBSnapshotVerifyView(game.board!, identifier: "boardView")
FBSnapshotVerifyLayer(game.board!.layer, identifier: "boardLayer")
FBSnapshotVerifyLayer(game.view.layer.sublayers.last!, identifier: "sublayer")
}
}
XCTest
20

21. Usageimport XCTest
import FBSnapshotTestCase
@testable import Product_Module_Name
class SnapshotUITests: FBSnapshotTestCase {
override func setUp() {
super.setUp()
recordMode = true
}
func test_Snapshot() {
let game = NumberTileGameViewController(dimension: 4, threshold: 2048)
game.board?.reset()
game.board?.insertTile(at: (1, 1), value: 2)
game.view.subviews.last!.label.text = "12345678"
FBSnapshotVerifyView(game.view, identifier: "wholeView")
FBSnapshotVerifyView(game.board!, identifier: "boardView")
FBSnapshotVerifyLayer(game.board!.layer, identifier: "boardLayer")
FBSnapshotVerifyLayer(game.view.layer.sublayers.last!, identifier: "sublayer")
}
}
XCTest
21

22. Usageimport XCTest
import FBSnapshotTestCase
@testable import Product_Module_Name
class SnapshotUITests: FBSnapshotTestCase {
override func setUp() {
super.setUp()
recordMode = true
}
func test_Snapshot() {
let game = NumberTileGameViewController(dimension: 4, threshold: 2048)
game.board?.reset()
game.board?.insertTile(at: (1, 1), value: 2)
game.view.subviews.last!.label.text = "12345678"
FBSnapshotVerifyView(game.view, identifier: "wholeView")
FBSnapshotVerifyView(game.board!, identifier: "boardView")
FBSnapshotVerifyLayer(game.board!.layer, identifier: "boardLayer")
FBSnapshotVerifyLayer(game.view.layer.sublayers.last!, identifier: "sublayer")
}
}
XCTest
22

23. Usageimport XCTest
import FBSnapshotTestCase
@testable import Product_Module_Name
class SnapshotUITests: FBSnapshotTestCase {
override func setUp() {
super.setUp()
recordMode = true
}
func test_Snapshot() {
let game = NumberTileGameViewController(dimension: 4, threshold: 2048)
game.board?.reset()
game.board?.insertTile(at: (1, 1), value: 2)
game.view.subviews.last!.label.text = "12345678"
FBSnapshotVerifyView(game.view, identifier: "wholeView")
FBSnapshotVerifyView(game.board!, identifier: "boardView")
FBSnapshotVerifyLayer(game.board!.layer, identifier: "boardLayer")
FBSnapshotVerifyLayer(game.view.layer.sublayers.last!, identifier: "sublayer")
}
}
XCTest
23

24. Usageimport XCTest
import FBSnapshotTestCase
@testable import Product_Module_Name
class SnapshotUITests: FBSnapshotTestCase {
override func setUp() {
super.setUp()
recordMode = true
}
func test_Snapshot() {
let game = NumberTileGameViewController(dimension: 4, threshold: 2048)
game.board?.reset()
game.board?.insertTile(at: (1, 1), value: 2)
game.view.subviews.last!.label.text = "12345678"
FBSnapshotVerifyView(game.view, identifier: "wholeView")
FBSnapshotVerifyView(game.board!, identifier: "boardView")
FBSnapshotVerifyLayer(game.board!.layer, identifier: "boardLayer")
FBSnapshotVerifyLayer(game.view.layer.sublayers.last!, identifier: "sublayer")
}
}
XCTest
24

25. Usage
25
Device & App
Whole View Snapshot
Board View Snapshot
Board Layer Snapshot
Score Layer
Snapshot

26. Snapshot name
26
Identifier argument
Custom
File name options
.screenScale test_name@3x.png
.device test_name_iPhone.png
.OS test_name_12_2.png
.screenSize test_name_320x728.png

27. Snapshot name
import XCTest
import FBSnapshotTestCase
class SnapshotUITests: FBSnapshotTestCase {
override func setUp() {
super.setUp()
recordMode = true
fileNameOptions = [
FBSnapshotTestCaseFileNameIncludeOption.OS,
FBSnapshotTestCaseFileNameIncludeOption.screenScale
]
}
func test_Snapshot() {
let os = UIDevice.current.systemVersion
let scale = UIScreen.main.scale
FBSnapshotVerifyView(view, identifier: "(os)_(scale)")
}
} 27

28. import XCTest
import FBSnapshotTestCase
class SnapshotUITests: FBSnapshotTestCase {
override func setUp() {
super.setUp()
recordMode = true
fileNameOptions = [
FBSnapshotTestCaseFileNameIncludeOption.OS,
FBSnapshotTestCaseFileNameIncludeOption.screenScale
]
}
func test_Snapshot() {
let os = UIDevice.current.systemVersion
let scale = UIScreen.main.scale
FBSnapshotVerifyView(view, identifier: "(os)_(scale)")
}
} 28
Snapshot name

29. import XCTest
import FBSnapshotTestCase
class SnapshotUITests: FBSnapshotTestCase {
override func setUp() {
super.setUp()
recordMode = true
fileNameOptions = [
FBSnapshotTestCaseFileNameIncludeOption.OS,
FBSnapshotTestCaseFileNameIncludeOption.screenScale
]
}
func test_Snapshot() {
let os = UIDevice.current.systemVersion
let scale = UIScreen.main.scale
FBSnapshotVerifyView(view, identifier: "(os)_(scale)")
}
} 29
Snapshot name

30. Tolerance
overall: 0.05
How many pixels may not match
perPixel: 0.05
How each pixel may not match
$ $$ $
ref actual ref actual#ff0000 #ff1a1a
30
pass passfail fail

31. Tolerance
import XCTest
import FBSnapshotTestCase
class SnapshotUITests: FBSnapshotTestCase {
func test_overallTolerance() {
FBSnapshotVerifyView(view, overallTolerance: 0.01)
}
func test_perPixelTolerance() {
FBSnapshotVerifyView(view, perPixelTolerance: 0.1)
}
}
31

32. Record
XCode
xcodebuild
fastlane
32

33. Record
lane :record do
scan(
devices: ['iPhone 8 Plus', 'iPhone 7'],
only_testing: ['snapshotTests/Sample', 'snapshotUITests/Sample'],
xcargs: 'RECORD_MODE=true',
scheme: '<test scheme>'
)
end
Fastfile
$ fastlane recordTerminal
33

34. Report
ref fail diff
34

35. 35

36. Introduction
36

37. How it worked
TestClass
AndroidJUnitRunner
37

38. How it works
ScreenshotRunner
TestClass
AndroidJUnitRunner
Screenshot
snap( ) snapActivity( )
38

39. Screenshot
How it works
Activity
View
ActivityTestRule(MainActivity::class.java).activity
39
measure( )
layout( )
draw( )LayoutInflater.from(targetContext)
.inflate(viewId)
activityTestRule.activity.findView(viewId)
View(targetContext)
record( )

40. Precondition
$ pip install mock
$ pip install Pillow
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
Terminal
AndroidManifest.xml
40

41. Precondition
buildscript {
// ...
dependencies {
// ...
classpath 'com.facebook.testing.screenshot:plugin:0.8.0'
}
}
apply plugin: 'com.facebook.testing.screenshot'
android {
// ...
defaultConfig {
// ...
testInstrumentationRunner "com.my.package.ScreenshotTestRunner"
}
}
class ScreenshotTestRunner: AndroidJUnitRunner() {
override fun onCreate(arguments: Bundle) {
ScreenshotRunner.onCreate(this, arguments)
super.onCreate(arguments)
}
override fun finish(resultCode: Int, results: Bundle) {
ScreenshotRunner.onDestroy()
super.finish(resultCode, results)
}
}
root
build.gradle
app
build.gradle
custom
test runner
41

42. Usage
42
20 48

43. 43
Usageclass ScreenshotTest {
@get:Rule
var activityTestRule = ActivityTestRule(MainActivity::class.java, false, false)
@Test
fun test_Snapshot_With_Activity() {
val activity = activityTestRule.launchActivity(null)
val score = activity.findViewById<View>(R.id.score)
Screenshot.snapActivity(activity).setName("whole_activity").record()
Screenshot.snap(score).setName("score").record()
}
@Test
fun test_Snapshot() {
val context = InstrumentationRegistry.getInstrumentation().targetContext
val view = MainView(context)
view.game.grid.clearGrid()
view.game.grid.insertTile(Tile(Cell(1, 1), 2))
view.game.score = 12345678
ViewHelpers.setupView(view)
.setExactWidthDp(300).setExactHeightDp(500).layout().draw()
Screenshot.snap(view).record()
}
}

44. 44
Usageclass ScreenshotTest {
@get:Rule
var activityTestRule = ActivityTestRule(MainActivity::class.java, false, false)
@Test
fun test_Snapshot_With_Activity() {
val activity = activityTestRule.launchActivity(null)
val score = activity.findViewById<View>(R.id.score)
Screenshot.snapActivity(activity).setName("whole_activity").record()
Screenshot.snap(score).setName("score").record()
}
@Test
fun test_Snapshot() {
val context = InstrumentationRegistry.getInstrumentation().targetContext
val view = MainView(context)
view.game.grid.clearGrid()
view.game.grid.insertTile(Tile(Cell(1, 1), 2))
view.game.score = 12345678
ViewHelpers.setupView(view)
.setExactWidthDp(300).setExactHeightDp(500).layout().draw()
Screenshot.snap(view).record()
}
}

45. 45
Usageclass ScreenshotTest {
@get:Rule
var activityTestRule = ActivityTestRule(MainActivity::class.java, false, false)
@Test
fun test_Snapshot_With_Activity() {
val activity = activityTestRule.launchActivity(null)
val score = activity.findViewById<View>(R.id.score)
Screenshot.snapActivity(activity).setName("whole_activity").record()
Screenshot.snap(score).setName("score").record()
}
@Test
fun test_Snapshot() {
val context = InstrumentationRegistry.getInstrumentation().targetContext
val view = MainView(context)
view.game.grid.clearGrid()
view.game.grid.insertTile(Tile(Cell(1, 1), 2))
view.game.score = 12345678
ViewHelpers.setupView(view)
.setExactWidthDp(300).setExactHeightDp(500).layout().draw()
Screenshot.snap(view).record()
}
}

46. 46
Usageclass ScreenshotTest {
@get:Rule
var activityTestRule = ActivityTestRule(MainActivity::class.java, false, false)
@Test
fun test_Snapshot_With_Activity() {
val activity = activityTestRule.launchActivity(null)
val score = activity.findViewById<View>(R.id.score)
Screenshot.snapActivity(activity).setName("whole_activity").record()
Screenshot.snap(score).setName("score").record()
}
@Test
fun test_Snapshot() {
val context = InstrumentationRegistry.getInstrumentation().targetContext
val view = MainView(context)
view.game.grid.clearGrid()
view.game.grid.insertTile(Tile(Cell(1, 1), 2))
view.game.score = 12345678
ViewHelpers.setupView(view)
.setExactWidthDp(300).setExactHeightDp(500).layout().draw()
Screenshot.snap(view).record()
}
}

47. 47
Usageclass ScreenshotTest {
@get:Rule
var activityTestRule = ActivityTestRule(MainActivity::class.java, false, false)
@Test
fun test_Snapshot_With_Activity() {
val activity = activityTestRule.launchActivity(null)
val score = activity.findViewById<View>(R.id.score)
Screenshot.snapActivity(activity).setName("whole_activity").record()
Screenshot.snap(score).setName("score").record()
}
@Test
fun test_Snapshot() {
val context = InstrumentationRegistry.getInstrumentation().targetContext
val view = MainView(context)
view.game.grid.clearGrid()
view.game.grid.insertTile(Tile(Cell(1, 1), 2))
view.game.score = 12345678
ViewHelpers.setupView(view)
.setExactWidthDp(300).setExactHeightDp(500).layout().draw()
Screenshot.snap(view).record()
}
}

48. 48
Usageclass ScreenshotTest {
@get:Rule
var activityTestRule = ActivityTestRule(MainActivity::class.java, false, false)
@Test
fun test_Snapshot_With_Activity() {
val activity = activityTestRule.launchActivity(null)
val score = activity.findViewById<View>(R.id.score)
Screenshot.snapActivity(activity).setName("whole_activity").record()
Screenshot.snap(score).setName("score").record()
}
@Test
fun test_Snapshot() {
val context = InstrumentationRegistry.getInstrumentation().targetContext
val view = MainView(context)
view.game.grid.clearGrid()
view.game.grid.insertTile(Tile(Cell(1, 1), 2))
view.game.score = 12345678
ViewHelpers.setupView(view)
.setExactWidthDp(300).setExactHeightDp(500).layout().draw()
Screenshot.snap(view).record()
}
}

49. 49
Usageclass ScreenshotTest {
@get:Rule
var activityTestRule = ActivityTestRule(MainActivity::class.java, false, false)
@Test
fun test_Snapshot_With_Activity() {
val activity = activityTestRule.launchActivity(null)
val score = activity.findViewById<View>(R.id.score)
Screenshot.snapActivity(activity).setName("whole_activity").record()
Screenshot.snap(score).setName("score").record()
}
@Test
fun test_Snapshot() {
val context = InstrumentationRegistry.getInstrumentation().targetContext
val view = MainView(context)
view.game.grid.clearGrid()
view.game.grid.insertTile(Tile(Cell(1, 1), 2))
view.game.score = 12345678
ViewHelpers.setupView(view)
.setExactWidthDp(300).setExactHeightDp(500).layout().draw()
Screenshot.snap(view).record()
}
}

50. 50
Usageclass ScreenshotTest {
@get:Rule
var activityTestRule = ActivityTestRule(MainActivity::class.java, false, false)
@Test
fun test_Snapshot_With_Activity() {
val activity = activityTestRule.launchActivity(null)
val score = activity.findViewById<View>(R.id.score)
Screenshot.snapActivity(activity).setName("whole_activity").record()
Screenshot.snap(score).setName("score").record()
}
@Test
fun test_Snapshot() {
val context = InstrumentationRegistry.getInstrumentation().targetContext
val view = MainView(context)
view.game.grid.clearGrid()
view.game.grid.insertTile(Tile(Cell(1, 1), 2))
view.game.score = 12345678
ViewHelpers.setupView(view)
.setExactWidthDp(300).setExactHeightDp(500).layout().draw()
Screenshot.snap(view).record()
}
}

51. Usage
51
Device & App Activity Snapshot
Score Snapshot
View Snapshot

52. Usage
buildscript {
// ...
}
screenshots {
multipleDevices true
}
$ ./gradlew recordDebugAndroidTestScreenshotTest
$ ./gradlew verifyDebugAndroidTestScreenshotTest
Terminal
app
build.gradle
52

53. Flexibility & Tolerance
✓Equal
✓Unique
53

54. Record
54

55. android {
// ...
defaultConfig {
if (project.gradle.startParameter.taskNames.toString().contains("record")) {
Map<String, String> map = new HashMap<String, String>()
map.put("package", "my.com.samplesnapshot.screenshots")
setTestInstrumentationRunnerArguments map
}
// ...
}
}
app
build.gradle
Terminal $ ./gradlew recordDebugAndroidTestScreenshotTest
Record
55

56. Report
56

57. Report
57

58. extension UIImage {
var removingStatusBar: UIImage? {
// some stuff
}
func fill(el: XCUIElement) -> UIImage {
// some stuff
}
}
Challenges
iOS
XCTest/XCUITest Confusing grey diff images
XCUITest Excess details on snapshots
XCUITest Animation
Common
Repo becomes fat
Snapshot collecting
Lack of reports
58
Android
What is the difference?
Does multipleDevices really work?
iOS
Android
fastlane
perPixelTolerance: 0.05$ export ANDROID_SERIAL=${udid}
usesDrawViewHierarchyInRect
works only in XCTest

59. Competitors Killer features Limitations
swift-snapshot-
testing
recordMode tolerance
assertSnaphot
as .recursiveDescription
lack of reports
custom assertions rootViewController
Competitors
shot report
support only one
device
59

60. Challenges
60
github.com/uber/
ios-snapshot-test-case
github.com/facebook/
screenshot-tests-for-android
github.com/pointfreeco/
swift-snapshot-testing
github.com/karumi/
shot

61. Resources
https://github.com/alter-al/sample_of_ios_snapshot_testing_2048
https://medium.com/xcnotes/snapshot-testing-in-xcuitest-d18ca9bdeae
https://github.com/alter-al/sample_of_android_screenshot_testing_2048
https://medium.com/xcnotes/snapshot-testing-in-espresso-e159cba817d6
61

62. Q&A
@alter_al
in/apesotskiy
medium.com/xcnotes