Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

QA Fest 2019. Алексей Альтер-Песоцкий. Snapshot testing with native mobile frameworks

20 views

Published on

Я расскажу про мобильное тестирование, посредством поскриншотового сравнения, с использованием нативных фреймворков (Espresso & XCTest) и сторонних библиотек. Открою секрет, что поскриншотовое сравнение доступно как в Unit, так и в UI тестах, рассмотрю, какие элементы и в каком виде доступны для верификации и поделюсь радостями и болью жизни со снэпшотами.

Published in: Education
  • Be the first to comment

  • Be the first to like this

QA Fest 2019. Алексей Альтер-Песоцкий. Snapshot testing with native mobile frameworks

  1. 1. Snapshot testing with native mobile frameworks
  2. 2. @alter_al in/apesotskiy medium.com/xcnotes 2
  3. 3. Contents 1. iOS Snapshot Testing 2. Android Screenshot Testing 3. Challenges 3
  4. 4. Snapshot testing 4
  5. 5. Tool Area Lang Notes applitools XCTest, Espresso, Appium, Calabash Any Polyglot, $$$ screenshot-tests-for- android Espresso Java, Kotlin Support multiple devices, lack of reports shot Espresso Java, Kotlin Cool reports, support one device ios-snapshot-test-case XCTest obj-C, Swift Really flexible, poor docs swift-snapshot-testing XCTest obj-C, Swift Multiformat, lack of reports Tool market 5
  6. 6. 6
  7. 7. Introduction 7
  8. 8. How it worked TestClass XCTestCase 8
  9. 9. How it works iOSSnapshotTestCase FBSnapshotVerifyView( )FBSnapshotVerifyLayer( ) TestClass XCTestCase 9
  10. 10. Snapshot How it works UIView CALayer view.layer.sublayers view.layer UIImageView(XCUIScreenshot().image) ViewController().view.subviews ViewController().view 10 XCUITest
  11. 11. target "SampleUITests" do use_frameworks! pod 'iOSSnapshotTestCase' end ✓ add an additional pod in Podfile: set the environment variables in our test scheme: Precondition • FB_REFERENCE_IMAGE_DIR • IMAGE_DIFF_DIR 11
  12. 12. Usage 12 20 48
  13. 13. Usageimport XCTest import FBSnapshotTestCase class SnapshotUITests: FBSnapshotTestCase { override func setUp() { super.setUp() recordMode = true XCUIApplication().launch() } func test_Snapshot() { let score = XCUIApplication().staticTexts["score"] let board = XCUIApplication().otherElements["board"] let scoreImage = score.screenshot().image let fullscreen = XCUIApplication()screenshot().image.fill(element: board).removingStatusBar FBSnapshotVerifyView(UIImageView(image: fullscreen), identifier: "fullscreen") FBSnapshotVerifyView(UIImageView(image: scoreImage), identifier: "score") } } XCTest XCUITest 13
  14. 14. Usageimport XCTest import FBSnapshotTestCase class SnapshotUITests: FBSnapshotTestCase { override func setUp() { super.setUp() recordMode = true XCUIApplication().launch() } func test_Snapshot() { let score = XCUIApplication().staticTexts["score"] let board = XCUIApplication().otherElements["board"] let scoreImage = score.screenshot().image let fullscreen = XCUIApplication()screenshot().image.fill(element: board).removingStatusBar FBSnapshotVerifyView(UIImageView(image: fullscreen), identifier: "fullscreen") FBSnapshotVerifyView(UIImageView(image: scoreImage), identifier: "score") } } 14
  15. 15. Usageimport XCTest import FBSnapshotTestCase class SnapshotUITests: FBSnapshotTestCase { override func setUp() { super.setUp() recordMode = true XCUIApplication().launch() } func test_Snapshot() { let score = XCUIApplication().staticTexts["score"] let board = XCUIApplication().otherElements["board"] let scoreImage = score.screenshot().image let fullscreen = XCUIApplication()screenshot().image.fill(element: board).removingStatusBar FBSnapshotVerifyView(UIImageView(image: fullscreen), identifier: "fullscreen") FBSnapshotVerifyView(UIImageView(image: scoreImage), identifier: "score") } } 15
  16. 16. Usageimport XCTest import FBSnapshotTestCase class SnapshotUITests: FBSnapshotTestCase { override func setUp() { super.setUp() recordMode = true XCUIApplication().launch() } func test_Snapshot() { let score = XCUIApplication().staticTexts["score"] let board = XCUIApplication().otherElements["board"] let scoreImage = score.screenshot().image let fullscreen = XCUIApplication()screenshot().image.fill(element: board).removingStatusBar FBSnapshotVerifyView(UIImageView(image: fullscreen), identifier: "fullscreen") FBSnapshotVerifyView(UIImageView(image: scoreImage), identifier: "score") } } 16
  17. 17. Usageimport XCTest import FBSnapshotTestCase class SnapshotUITests: FBSnapshotTestCase { override func setUp() { super.setUp() recordMode = true XCUIApplication().launch() } func test_Snapshot() { let score = XCUIApplication().staticTexts["score"] let board = XCUIApplication().otherElements["board"] let scoreImage = score.screenshot().image let fullscreen = XCUIApplication()screenshot().image.fill(element: board).removingStatusBar FBSnapshotVerifyView(UIImageView(image: fullscreen), identifier: "fullscreen") FBSnapshotVerifyView(UIImageView(image: scoreImage), identifier: "score") } } 17
  18. 18. Usageimport XCTest import FBSnapshotTestCase class SnapshotUITests: FBSnapshotTestCase { override func setUp() { super.setUp() recordMode = true XCUIApplication().launch() } func test_Snapshot() { let score = XCUIApplication().staticTexts["score"] let board = XCUIApplication().otherElements["board"] let scoreImage = score.screenshot().image let fullscreen = XCUIApplication()screenshot().image.fill(element: board).removingStatusBar FBSnapshotVerifyView(UIImageView(image: fullscreen), identifier: "fullscreen") FBSnapshotVerifyView(UIImageView(image: scoreImage), identifier: "score") } } 18
  19. 19. Usage 19 Fullscreen Snapshot Score Snapshot Original Screen
  20. 20. Usageimport XCTest import FBSnapshotTestCase @testable import Product_Module_Name class SnapshotUITests: FBSnapshotTestCase { override func setUp() { super.setUp() recordMode = true } func test_Snapshot() { let game = NumberTileGameViewController(dimension: 4, threshold: 2048) game.board?.reset() game.board?.insertTile(at: (1, 1), value: 2) game.view.subviews.last!.label.text = "12345678" FBSnapshotVerifyView(game.view, identifier: "wholeView") FBSnapshotVerifyView(game.board!, identifier: "boardView") FBSnapshotVerifyLayer(game.board!.layer, identifier: "boardLayer") FBSnapshotVerifyLayer(game.view.layer.sublayers.last!, identifier: "sublayer") } } XCTest 20
  21. 21. Usageimport XCTest import FBSnapshotTestCase @testable import Product_Module_Name class SnapshotUITests: FBSnapshotTestCase { override func setUp() { super.setUp() recordMode = true } func test_Snapshot() { let game = NumberTileGameViewController(dimension: 4, threshold: 2048) game.board?.reset() game.board?.insertTile(at: (1, 1), value: 2) game.view.subviews.last!.label.text = "12345678" FBSnapshotVerifyView(game.view, identifier: "wholeView") FBSnapshotVerifyView(game.board!, identifier: "boardView") FBSnapshotVerifyLayer(game.board!.layer, identifier: "boardLayer") FBSnapshotVerifyLayer(game.view.layer.sublayers.last!, identifier: "sublayer") } } XCTest 21
  22. 22. Usageimport XCTest import FBSnapshotTestCase @testable import Product_Module_Name class SnapshotUITests: FBSnapshotTestCase { override func setUp() { super.setUp() recordMode = true } func test_Snapshot() { let game = NumberTileGameViewController(dimension: 4, threshold: 2048) game.board?.reset() game.board?.insertTile(at: (1, 1), value: 2) game.view.subviews.last!.label.text = "12345678" FBSnapshotVerifyView(game.view, identifier: "wholeView") FBSnapshotVerifyView(game.board!, identifier: "boardView") FBSnapshotVerifyLayer(game.board!.layer, identifier: "boardLayer") FBSnapshotVerifyLayer(game.view.layer.sublayers.last!, identifier: "sublayer") } } XCTest 22
  23. 23. Usageimport XCTest import FBSnapshotTestCase @testable import Product_Module_Name class SnapshotUITests: FBSnapshotTestCase { override func setUp() { super.setUp() recordMode = true } func test_Snapshot() { let game = NumberTileGameViewController(dimension: 4, threshold: 2048) game.board?.reset() game.board?.insertTile(at: (1, 1), value: 2) game.view.subviews.last!.label.text = "12345678" FBSnapshotVerifyView(game.view, identifier: "wholeView") FBSnapshotVerifyView(game.board!, identifier: "boardView") FBSnapshotVerifyLayer(game.board!.layer, identifier: "boardLayer") FBSnapshotVerifyLayer(game.view.layer.sublayers.last!, identifier: "sublayer") } } XCTest 23
  24. 24. Usageimport XCTest import FBSnapshotTestCase @testable import Product_Module_Name class SnapshotUITests: FBSnapshotTestCase { override func setUp() { super.setUp() recordMode = true } func test_Snapshot() { let game = NumberTileGameViewController(dimension: 4, threshold: 2048) game.board?.reset() game.board?.insertTile(at: (1, 1), value: 2) game.view.subviews.last!.label.text = "12345678" FBSnapshotVerifyView(game.view, identifier: "wholeView") FBSnapshotVerifyView(game.board!, identifier: "boardView") FBSnapshotVerifyLayer(game.board!.layer, identifier: "boardLayer") FBSnapshotVerifyLayer(game.view.layer.sublayers.last!, identifier: "sublayer") } } XCTest 24
  25. 25. Usage 25 Device & App Whole View Snapshot Board View Snapshot Board Layer Snapshot Score Layer Snapshot
  26. 26. Snapshot name 26 Identifier argument Custom File name options .screenScale test_name@3x.png .device test_name_iPhone.png .OS test_name_12_2.png .screenSize test_name_320x728.png
  27. 27. Snapshot name import XCTest import FBSnapshotTestCase class SnapshotUITests: FBSnapshotTestCase { override func setUp() { super.setUp() recordMode = true fileNameOptions = [ FBSnapshotTestCaseFileNameIncludeOption.OS, FBSnapshotTestCaseFileNameIncludeOption.screenScale ] } func test_Snapshot() { let os = UIDevice.current.systemVersion let scale = UIScreen.main.scale FBSnapshotVerifyView(view, identifier: "(os)_(scale)") } } 27
  28. 28. import XCTest import FBSnapshotTestCase class SnapshotUITests: FBSnapshotTestCase { override func setUp() { super.setUp() recordMode = true fileNameOptions = [ FBSnapshotTestCaseFileNameIncludeOption.OS, FBSnapshotTestCaseFileNameIncludeOption.screenScale ] } func test_Snapshot() { let os = UIDevice.current.systemVersion let scale = UIScreen.main.scale FBSnapshotVerifyView(view, identifier: "(os)_(scale)") } } 28 Snapshot name
  29. 29. import XCTest import FBSnapshotTestCase class SnapshotUITests: FBSnapshotTestCase { override func setUp() { super.setUp() recordMode = true fileNameOptions = [ FBSnapshotTestCaseFileNameIncludeOption.OS, FBSnapshotTestCaseFileNameIncludeOption.screenScale ] } func test_Snapshot() { let os = UIDevice.current.systemVersion let scale = UIScreen.main.scale FBSnapshotVerifyView(view, identifier: "(os)_(scale)") } } 29 Snapshot name
  30. 30. Tolerance overall: 0.05 How many pixels may not match perPixel: 0.05 How each pixel may not match $ $$ $ ref actual ref actual#ff0000 #ff1a1a 30 pass passfail fail
  31. 31. Tolerance import XCTest import FBSnapshotTestCase class SnapshotUITests: FBSnapshotTestCase { func test_overallTolerance() { FBSnapshotVerifyView(view, overallTolerance: 0.01) } func test_perPixelTolerance() { FBSnapshotVerifyView(view, perPixelTolerance: 0.1) } } 31
  32. 32. Record XCode xcodebuild fastlane 32
  33. 33. Record lane :record do scan( devices: ['iPhone 8 Plus', 'iPhone 7'], only_testing: ['snapshotTests/Sample', 'snapshotUITests/Sample'], xcargs: 'RECORD_MODE=true', scheme: '<test scheme>' ) end Fastfile $ fastlane recordTerminal 33
  34. 34. Report ref fail diff 34
  35. 35. 35
  36. 36. Introduction 36
  37. 37. How it worked TestClass AndroidJUnitRunner 37
  38. 38. How it works ScreenshotRunner TestClass AndroidJUnitRunner Screenshot snap( ) snapActivity( ) 38
  39. 39. Screenshot How it works Activity View ActivityTestRule(MainActivity::class.java).activity 39 measure( ) layout( ) draw( )LayoutInflater.from(targetContext) .inflate(viewId) activityTestRule.activity.findView(viewId) View(targetContext) record( )
  40. 40. Precondition $ pip install mock $ pip install Pillow <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" /> Terminal AndroidManifest.xml 40
  41. 41. Precondition buildscript { // ... dependencies { // ... classpath 'com.facebook.testing.screenshot:plugin:0.8.0' } } apply plugin: 'com.facebook.testing.screenshot' android { // ... defaultConfig { // ... testInstrumentationRunner "com.my.package.ScreenshotTestRunner" } } class ScreenshotTestRunner: AndroidJUnitRunner() { override fun onCreate(arguments: Bundle) { ScreenshotRunner.onCreate(this, arguments) super.onCreate(arguments) } override fun finish(resultCode: Int, results: Bundle) { ScreenshotRunner.onDestroy() super.finish(resultCode, results) } } root build.gradle app build.gradle custom test runner 41
  42. 42. Usage 42 20 48
  43. 43. 43 Usageclass ScreenshotTest { @get:Rule var activityTestRule = ActivityTestRule(MainActivity::class.java, false, false) @Test fun test_Snapshot_With_Activity() { val activity = activityTestRule.launchActivity(null) val score = activity.findViewById<View>(R.id.score) Screenshot.snapActivity(activity).setName("whole_activity").record() Screenshot.snap(score).setName("score").record() } @Test fun test_Snapshot() { val context = InstrumentationRegistry.getInstrumentation().targetContext val view = MainView(context) view.game.grid.clearGrid() view.game.grid.insertTile(Tile(Cell(1, 1), 2)) view.game.score = 12345678 ViewHelpers.setupView(view) .setExactWidthDp(300).setExactHeightDp(500).layout().draw() Screenshot.snap(view).record() } }
  44. 44. 44 Usageclass ScreenshotTest { @get:Rule var activityTestRule = ActivityTestRule(MainActivity::class.java, false, false) @Test fun test_Snapshot_With_Activity() { val activity = activityTestRule.launchActivity(null) val score = activity.findViewById<View>(R.id.score) Screenshot.snapActivity(activity).setName("whole_activity").record() Screenshot.snap(score).setName("score").record() } @Test fun test_Snapshot() { val context = InstrumentationRegistry.getInstrumentation().targetContext val view = MainView(context) view.game.grid.clearGrid() view.game.grid.insertTile(Tile(Cell(1, 1), 2)) view.game.score = 12345678 ViewHelpers.setupView(view) .setExactWidthDp(300).setExactHeightDp(500).layout().draw() Screenshot.snap(view).record() } }
  45. 45. 45 Usageclass ScreenshotTest { @get:Rule var activityTestRule = ActivityTestRule(MainActivity::class.java, false, false) @Test fun test_Snapshot_With_Activity() { val activity = activityTestRule.launchActivity(null) val score = activity.findViewById<View>(R.id.score) Screenshot.snapActivity(activity).setName("whole_activity").record() Screenshot.snap(score).setName("score").record() } @Test fun test_Snapshot() { val context = InstrumentationRegistry.getInstrumentation().targetContext val view = MainView(context) view.game.grid.clearGrid() view.game.grid.insertTile(Tile(Cell(1, 1), 2)) view.game.score = 12345678 ViewHelpers.setupView(view) .setExactWidthDp(300).setExactHeightDp(500).layout().draw() Screenshot.snap(view).record() } }
  46. 46. 46 Usageclass ScreenshotTest { @get:Rule var activityTestRule = ActivityTestRule(MainActivity::class.java, false, false) @Test fun test_Snapshot_With_Activity() { val activity = activityTestRule.launchActivity(null) val score = activity.findViewById<View>(R.id.score) Screenshot.snapActivity(activity).setName("whole_activity").record() Screenshot.snap(score).setName("score").record() } @Test fun test_Snapshot() { val context = InstrumentationRegistry.getInstrumentation().targetContext val view = MainView(context) view.game.grid.clearGrid() view.game.grid.insertTile(Tile(Cell(1, 1), 2)) view.game.score = 12345678 ViewHelpers.setupView(view) .setExactWidthDp(300).setExactHeightDp(500).layout().draw() Screenshot.snap(view).record() } }
  47. 47. 47 Usageclass ScreenshotTest { @get:Rule var activityTestRule = ActivityTestRule(MainActivity::class.java, false, false) @Test fun test_Snapshot_With_Activity() { val activity = activityTestRule.launchActivity(null) val score = activity.findViewById<View>(R.id.score) Screenshot.snapActivity(activity).setName("whole_activity").record() Screenshot.snap(score).setName("score").record() } @Test fun test_Snapshot() { val context = InstrumentationRegistry.getInstrumentation().targetContext val view = MainView(context) view.game.grid.clearGrid() view.game.grid.insertTile(Tile(Cell(1, 1), 2)) view.game.score = 12345678 ViewHelpers.setupView(view) .setExactWidthDp(300).setExactHeightDp(500).layout().draw() Screenshot.snap(view).record() } }
  48. 48. 48 Usageclass ScreenshotTest { @get:Rule var activityTestRule = ActivityTestRule(MainActivity::class.java, false, false) @Test fun test_Snapshot_With_Activity() { val activity = activityTestRule.launchActivity(null) val score = activity.findViewById<View>(R.id.score) Screenshot.snapActivity(activity).setName("whole_activity").record() Screenshot.snap(score).setName("score").record() } @Test fun test_Snapshot() { val context = InstrumentationRegistry.getInstrumentation().targetContext val view = MainView(context) view.game.grid.clearGrid() view.game.grid.insertTile(Tile(Cell(1, 1), 2)) view.game.score = 12345678 ViewHelpers.setupView(view) .setExactWidthDp(300).setExactHeightDp(500).layout().draw() Screenshot.snap(view).record() } }
  49. 49. 49 Usageclass ScreenshotTest { @get:Rule var activityTestRule = ActivityTestRule(MainActivity::class.java, false, false) @Test fun test_Snapshot_With_Activity() { val activity = activityTestRule.launchActivity(null) val score = activity.findViewById<View>(R.id.score) Screenshot.snapActivity(activity).setName("whole_activity").record() Screenshot.snap(score).setName("score").record() } @Test fun test_Snapshot() { val context = InstrumentationRegistry.getInstrumentation().targetContext val view = MainView(context) view.game.grid.clearGrid() view.game.grid.insertTile(Tile(Cell(1, 1), 2)) view.game.score = 12345678 ViewHelpers.setupView(view) .setExactWidthDp(300).setExactHeightDp(500).layout().draw() Screenshot.snap(view).record() } }
  50. 50. 50 Usageclass ScreenshotTest { @get:Rule var activityTestRule = ActivityTestRule(MainActivity::class.java, false, false) @Test fun test_Snapshot_With_Activity() { val activity = activityTestRule.launchActivity(null) val score = activity.findViewById<View>(R.id.score) Screenshot.snapActivity(activity).setName("whole_activity").record() Screenshot.snap(score).setName("score").record() } @Test fun test_Snapshot() { val context = InstrumentationRegistry.getInstrumentation().targetContext val view = MainView(context) view.game.grid.clearGrid() view.game.grid.insertTile(Tile(Cell(1, 1), 2)) view.game.score = 12345678 ViewHelpers.setupView(view) .setExactWidthDp(300).setExactHeightDp(500).layout().draw() Screenshot.snap(view).record() } }
  51. 51. Usage 51 Device & App Activity Snapshot Score Snapshot View Snapshot
  52. 52. Usage buildscript { // ... } screenshots { multipleDevices true } $ ./gradlew recordDebugAndroidTestScreenshotTest $ ./gradlew verifyDebugAndroidTestScreenshotTest Terminal app build.gradle 52
  53. 53. Flexibility & Tolerance ✓Equal ✓Unique 53
  54. 54. Record 54
  55. 55. android { // ... defaultConfig { if (project.gradle.startParameter.taskNames.toString().contains("record")) { Map<String, String> map = new HashMap<String, String>() map.put("package", "my.com.samplesnapshot.screenshots") setTestInstrumentationRunnerArguments map } // ... } } app build.gradle Terminal $ ./gradlew recordDebugAndroidTestScreenshotTest Record 55
  56. 56. Report 56
  57. 57. Report 57
  58. 58. extension UIImage { var removingStatusBar: UIImage? { // some stuff } func fill(el: XCUIElement) -> UIImage { // some stuff } } Challenges iOS XCTest/XCUITest Confusing grey diff images XCUITest Excess details on snapshots XCUITest Animation Common Repo becomes fat Snapshot collecting Lack of reports 58 Android What is the difference? Does multipleDevices really work? iOS Android fastlane perPixelTolerance: 0.05$ export ANDROID_SERIAL=${udid} usesDrawViewHierarchyInRect works only in XCTest
  59. 59. Competitors Killer features Limitations swift-snapshot- testing recordMode tolerance assertSnaphot as .recursiveDescription lack of reports custom assertions rootViewController Competitors shot report support only one device 59
  60. 60. Challenges 60 github.com/uber/ ios-snapshot-test-case github.com/facebook/ screenshot-tests-for-android github.com/pointfreeco/ swift-snapshot-testing github.com/karumi/ shot
  61. 61. Resources https://github.com/alter-al/sample_of_ios_snapshot_testing_2048 https://medium.com/xcnotes/snapshot-testing-in-xcuitest-d18ca9bdeae https://github.com/alter-al/sample_of_android_screenshot_testing_2048 https://medium.com/xcnotes/snapshot-testing-in-espresso-e159cba817d6 61
  62. 62. Q&A @alter_al in/apesotskiy medium.com/xcnotes

×