Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

CookpadTechConf2018-(Mobile)TestAutomation

911 views

Published on

Show a concept to design test automation for your team.
https://techconf.cookpad.com/2018/

Published in: Software
  • Be the first to comment

CookpadTechConf2018-(Mobile)TestAutomation

  1. 1. Kazuaki Matsuo International Business Development Product/Engineering Group What/How To Design Test Automation for Mobile Cookpad Inc.Feb 10th, 2018
  2. 2. Who I am 2 Kazuaki Matsuo • International Business Development Product/Engineering Group • Software Engineer in Quality • Social Media • Twitter: @Kazu_cocoa, GitHub: KazuCocoa • OSS: Appium Project
  3. 3. Why do you evolve test automation? 3
  4. 4. Okay, let’s go 4 Motivations • Build fast feedback cycle • Make test deterministic • Design and build product lines • Build “Culture of automation” to be autonomous organisation
  5. 5. By the way, 5 Can you work with the same context? A: What about this integration test? B: Well… we should use mock to make them stable. A: I won’t use mocking since it hides real world. B: I see, but without mock, the test may be flaky. A: This is integration and my understanding of the type is less mocking and close to in the wild, though…
  6. 6. In this session, you can learn 6 Concepts and Ways you go • Concepts for automated test design / plan • How adapt them to real projects/products • Cookpad International applications • Cookpad Japan applications
  7. 7. In this session, you can learn 7 Concepts and Ways you go • Concepts for automated test design / plan • How adapt them to real projects/products • Cookpad International applications • Cookpad Japan applications Won’t talk about business/humans/ techniques/tools
  8. 8. Scope Phase Level sIze Type
  9. 9. Scope 9 What is your test target?
  10. 10. Entire system? 10
  11. 11. Entire system? 11
  12. 12. Entire system? 12
  13. 13. Entire system? 13
  14. 14. Focus on the mobile 14
  15. 15. Focus on the mobile 15
  16. 16. The app has some UI flows 16
  17. 17. The app has some UI flows 17
  18. 18. Focus on UI flow? 18
  19. 19. Focus on UI flow? 19
  20. 20. Focus on UI flow? 20
  21. 21. Focus on UI flow? 21
  22. 22. More internal process? 22
  23. 23. More internal process? 23
  24. 24. More internal process? 24
  25. 25. Thus, 25
  26. 26. Thus, 26
  27. 27. Thus, 27
  28. 28. Scope Phase Level sIze Type
  29. 29. 29 Phase What phase do you focus?
  30. 30. What phase would you like to automate? 30
  31. 31. What phase would you like to automate? 31 Timeline for release
  32. 32. What phase would you like to automate? 32 release
  33. 33. What phase would you like to automate? 33 unit test component test coding low-level design integration test high-level design system test acceptance test specification analyse requirement
  34. 34. What phase would you like to automate? 34 unit test component test coding low-level design integration test high-level design system test acceptance test specification analyse requirement
  35. 35. Phase for development or production 35
  36. 36. Phase for development or production 36 release
  37. 37. What phase would you automate? 37 In Development
  38. 38. What phase would you automate? 38 In Production
  39. 39. What phase would you automate? 39 In Development In Production
  40. 40. Scope Phase Level sIze Type
  41. 41. 41 Level How deep do you dive to automation world?
  42. 42. What activity is the target? 42 Plan/Design Implement Run Analyse The Results
  43. 43. How deep would you like to drill down? 43 https://blog.appdiff.com/test-autonomy-levels-7de7967d030e 0. Manual Testing 1. Scripted Testing 2. Exploratory Bots 3. Human-Directed 4. Generative 5. Fully Autonomous
  44. 44. Scope Phase Level sIze Type
  45. 45. 45 Size What size to you test?
  46. 46. Test Pyramid 46 UI Tests Integration Tests Unit Tests
  47. 47. 47 UI Tests Unit Tests manual testing Integration Tests Test Pyramid
  48. 48. Test Size 48http://techlife.cookpad.com/entry/2016/08/13/test-size-for-mobile https://testing.googleblog.com/2010/12/test-sizes.html feature Small Medium Large Enormous network no localhost localhost/yes yes system access no partial/yes yes yes OS(APIs) no yes yes yes View no partial yes yes external system no no no yes time per a test < 100ms < 2s < 120s < 500s
  49. 49. Scope Phase Level sIze Type
  50. 50. 50 Type What type do you test?
  51. 51. Agile Testing Quadrants 51 Business Facing Technology Facing GuideDevelopment CritiquetheProduct http://www.informit.com/articles/article.aspx?p=2253544&ranMID=24808
  52. 52. Agile Testing Quadrants 52 Unit Tests Component Tests (code level) Testing Connectivity Business Facing Technology Facing GuideDevelopment CritiquetheProduct http://www.informit.com/articles/article.aspx?p=2253544&ranMID=24808
  53. 53. Agile Testing Quadrants 53 Example A/B Tests Story Tests (Written First) UX (User Experience) Testing Prototypes (paper/wireframes) Simulations Unit Tests Component Tests (code level) Testing Connectivity Business Facing Technology Facing GuideDevelopment CritiquetheProduct http://www.informit.com/articles/article.aspx?p=2253544&ranMID=24808
  54. 54. Agile Testing Quadrants 54 Example A/B Tests Story Tests (Written First) UX (User Experience) Testing Prototypes (paper/wireframes) Simulations Unit Tests Component Tests (code level) Testing Connectivity Exploratory Testing Workflows System Integration (business oriented) Usability Testing User Acceptance Testing Business Facing Technology Facing GuideDevelopment CritiquetheProduct http://www.informit.com/articles/article.aspx?p=2253544&ranMID=24808
  55. 55. Agile Testing Quadrants 55 Example A/B Tests Story Tests (Written First) UX (User Experience) Testing Prototypes (paper/wireframes) Simulations Unit Tests Component Tests (code level) Testing Connectivity Exploratory Testing Workflows System Integration (business oriented) Usability Testing User Acceptance Testing Performance Testing Load Testing Security Testing Quality Attributes (…ilities) Business Facing Technology Facing GuideDevelopment CritiquetheProduct http://www.informit.com/articles/article.aspx?p=2253544&ranMID=24808
  56. 56. Agile Testing Quadrants 56 Example A/B Tests Story Tests (Written First) UX (User Experience) Testing Prototypes (paper/wireframes) Simulations Unit Tests Component Tests (code level) Testing Connectivity Exploratory Testing Workflows System Integration (business oriented) Usability Testing User Acceptance Testing Performance Testing Load Testing Security Testing Quality Attributes (…ilities) Outside-in Barrier-free Collaborative Business Facing Technology Facing GuideDevelopment CritiquetheProduct http://www.informit.com/articles/article.aspx?p=2253544&ranMID=24808
  57. 57. Scope Phase Level sIze Type
  58. 58. Adapt the design to real products 58
  59. 59. Cookpad Android for Global 59
  60. 60. Before stepping into concrete design… 60 Define your ideal product line • What is your team’s ideal product line? • To achieve the line, what kind of activities would you automate? • Planning/Designing test automation as well
  61. 61. Apply SPLIT 61 Scope Phase Level sIze Type
  62. 62. Apply SPLIT 62 Scope Phase Level sIze Type Android provides concept for “Test Size” officially
  63. 63. Split entire system to some parts 63
  64. 64. Scope 64
  65. 65. Scope 1 / 4 65
  66. 66. Scope 2 / 4 66
  67. 67. Scope 3 / 4 67
  68. 68. Scope 4 / 4 68
  69. 69. Phase 69 In Development In Production
  70. 70. Phase 70 In Development
  71. 71. Size 71 UI Tests Integration Tests Unit Tests
  72. 72. Size 72 UI Tests Integration Tests Unit Tests
  73. 73. Size 73 UI Tests Integration Tests Unit Tests New New
  74. 74. Size 74
  75. 75. Size 75 L: UI Component based M: Integration Tests S: Unit Tests E: User Scenario based
  76. 76. Size 76 L: UI Component based M: Integration Tests S: Unit Tests E: User Scenario based
  77. 77. Size 77 L: UI Component based M: Integration Tests S: Unit Tests E: User Scenario based
  78. 78. Size 78 L: UI Component based M: Integration Tests S: Unit Tests E: User Scenario based
  79. 79. Size 79 L: UI Component based M: Integration Tests S: Unit Tests E: User Scenario based
  80. 80. Type 80 Example A/B Tests Story Tests (Written First) UX (User Experience) Testing Prototypes (paper/wireframes) Simulations Unit Tests Component Tests (code level) Testing Connectivity Exploratory Testing Workflows System Integration (business oriented) Usability Testing User Acceptance Testing Performance Testing Load Testing Security Testing Quality Attributes (…ilities) Outside-in Barrier-free Collaborative Business Facing Technology Facing GuideDevelopment CritiquetheProduct http://www.informit.com/articles/article.aspx?p=2253544&ranMID=24808
  81. 81. Type 81 Example Story Tests (Written First) Unit Tests Component Tests (code level) Testing Connectivity User Acceptance Testing Business Facing Technology Facing GuideDevelopment CritiquetheProduct http://www.informit.com/articles/article.aspx?p=2253544&ranMID=24808
  82. 82. Convince other members with SPLIT 82
  83. 83. In the real world… 83
  84. 84. Current status ( Jan, 2018 ~ Feb, 2018) 84 L: UI Component based M: Integration Tests S: Unit Tests E: User Scenario based
  85. 85. Scope Phase Level sIze Type
  86. 86. Cookpad Android/iOS for Japan 86
  87. 87. History 87 Cookpad Android / iOS run long history following with Android / iOS platforms evolution
  88. 88. History 88 Cookpad Android / iOS run long history following with Android / iOS platforms evolution https://academy.realm.io/posts/tasting-tests-at-cookpad-try-swift-2017/
  89. 89. Current status ( Sep?, 2014~ Feb, 2018) 89http://techlife.cookpad.com/entry/2016/08/13/test-size-for-mobile L: UI Component based M: Integration Tests S: Unit Tests E: User Scenario based
  90. 90. 90 L: UI Component based M: Integration Tests S: Unit Tests E: User Scenario based http://techlife.cookpad.com/entry/2016/08/13/test-size-for-mobile some of manual test for regressions Current status ( Sep?, 2014~ Feb, 2018)
  91. 91. 91 Cookpad iOS for Global
  92. 92. Any plan or design? 92 No particular entire plan / design • but, one interesting automaton
  93. 93. Type 93 Example A/B Tests Story Tests (Written First) UX (User Experience) Testing Prototypes (paper/wireframes) Simulations Unit Tests Component Tests (code level) Testing Connectivity Exploratory Testing Workflows System Integration (business oriented) Usability Testing User Acceptance Testing Performance Testing Load Testing Security Testing Quality Attributes (…ilities) Outside-in Barrier-free Collaborative Business Facing Technology Facing GuideDevelopment CritiquetheProduct http://www.informit.com/articles/article.aspx?p=2253544&ranMID=24808
  94. 94. Type for Android 94 Example Story Tests (Written First) Unit Tests Component Tests (code level) Testing Connectivity User Acceptance Testing Business Facing Technology Facing GuideDevelopment CritiquetheProduct http://www.informit.com/articles/article.aspx?p=2253544&ranMID=24808
  95. 95. Automate performance testing 95 Performance Testing Business Facing Technology Facing GuideDevelopment CritiquetheProduct http://www.informit.com/articles/article.aspx?p=2253544&ranMID=24808
  96. 96. Challenge to integrate performance testing in CI 96 before after Measure performance for particular scenarios
  97. 97. Challenge to integrate performance testing in CI 97 before after Measure performance for particular scenarios
  98. 98. Challenge to integrate performance testing in CI 98 before after Measure performance for particular scenarios
  99. 99. resources/tools 99
  100. 100. Tools 100 • Common • Appium, appium_lib • fastlane, RSpec, Turnip, turnip_formatter • KazuCocoa/http_proxy, tomakehurst/wiremock, Kobold • Android • JUnit 4, AssertJ, Truth, Espresso, FindBugs(SpotBugs), Composer • Robolectric, Powermock, Mockito • KazuCocoa/droid-monitor, KazuCocoa/droid_adbs, KazuCocoa/DroidTestHelper • iOS • XCTest(XCUITest), EarlGrey, wix/AppleSimulatorUtils • OCMock, OHHTTPStubs
  101. 101. Scope Phase Level sIze Type
  102. 102. kazuaki-matsuo@cookpad.com @Kazu_cocoa Thank you 102

×