Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Супер быстрая автоматизация тестирования на iOS

446 views

Published on

Доклад Виктора Короневича на конференции SQA Days-21
www.sqadays.com

Published in: Education
  • Be the first to comment

  • Be the first to like this

Супер быстрая автоматизация тестирования на iOS

  1. 1. iOS Parallel Automation: Run faster than fast Viktar Karanevich · 26 May
  2. 2. I. History of iOS Automation Frameworks (10 min) II. Path to the current approach at Badoo (10 min) III. Summary (5 min) IV. Movie: Test Run (3 min)
  3. 3. self.introduce => { name: "Viktar Karanevich", title: ["SSATE", "SSDET", "SQAAE"], languages: ["Java", "Ruby"], experience: "iOS Automation 7+ years" }
  4. 4. BIGGER THAN DATING 348M+ registered users 50M+ downloads on iOS 800+ automated tests on iOS
  5. 5. Steve Jobs Steve Wozniak I. History
  6. 6. Jun 2010 UI Automation Apple
  7. 7. Jun 2010 UI Automation Sep 2014 UI Automation xcrun simctl Apple "xcrun simctl" is introduced in xCode 6 It is command line utility to control the Simulator (create, delete, erase, install, uninstall, ...).
  8. 8. Jun 2010 UI Automation Sep 2015 UI Automation xcrun simctl UI Testing Apple UI Testing is introduced as replacement for UIAutomation in xCode 7 Sep 2014 UI Automation xcrun simctl
  9. 9. Jun 2010 UI Automation Sep 2016 UI Testing xcrun simctl Apple Sep 2015 UI Automation xcrun simctl UI Testing Sep 2014 UI Automation xcrun simctl UI Automation removed in xCode 8
  10. 10. Jun 2010 UI Automation Sep 2014 xcrun simctl Sep 2016 UI Testing Sep 2016 UI Testing Apple Calabash Nov 2009 UI Spec
  11. 11. Calabash Nov 2009 UI Spec Feb 2011 Frank
  12. 12. Calabash Nov 2009 UI Spec Feb 2011 Frank Jan 2012 Calabash Step Definitions Ruby Client Library Features Developer host Calabash server Simulator Your app UIQuery Touch Actions
  13. 13. Calabash Apr 2013 Calabash + UIAutomation Link: https://github.com/calabash/run_loop Step Definitions Ruby Client Library Features Developer host Calabash server Simulator Your app UIQuery Run loop: UIAutomation
  14. 14. Calabash Sep 2016 Calabash + DeviceAgent Link: https://github.com/calabash/run_loop/ tree/develop/lib/run_loop/device_agent Developer host Simulator Step Definitions Ruby Client Library XCTest Runner Features Calabash server Your app UIQuery Device Agent
  15. 15. Calabash Nov 2009 UI Spec Feb 2011 Frank Jan 2012 Calabash iOSAutomation Frameworks Apr 2013 Calabash + UIAutomation Sep 2016 Calabash + DeviceAgent Jun 2010 UI Automation Sep 2016 UI Testing xcrun simctl Sep 2015 UI Automation xcrun simctl UI Testing Sep 2014 UI Automation xcrun simctl Apple 3 years 1 year
  16. 16. Calabash Nov 2009 UI Spec Feb 2011 Frank Jan 2012 Calabash iOSAutomation Frameworks Apr 2013 Calabash + UIAutomation Sep 2016 Calabash + DeviceAgent Jun 2010 UI Automation Sep 2016 UI Testing xcrun simctl Sep 2015 UI Automation xcrun simctl UI Testing Sep 2014 UI Automation xcrun simctl Apple 3 years 1 year
  17. 17. II. iOS Automation at Badoo Approach ~80 flaky tests 80 data fail tests
  18. 18. II. Before Parallel Run Test Suite Run Nightly Cucumber Report Nightmare! 15h total time 1 simulator 80 data ~500 tests
  19. 19. II. Before Parallel Run Test Suite Run Nightly Cucumber Report Nightmare! 80 data fail tests 1simulator Key Issue: How to speed up Test Run? 15h total time ~500 tests 1 simulator
  20. 20. Let's Parallel!
  21. 21. Let's Parallel! Blocker Issue: - Calabash does not support Parallel run on 1 desktop
  22. 22. Let's Parallel with multi-desktops!
  23. 23. https://techblog.badoo.com Tim's talk at Badoo: "Parallel Calabash Testing on iOS"
  24. 24. II. Test Result: multi-desktops ~4 hours regression suite run time 5 Desktops
  25. 25. II. Test Result: multi-desktops Mac PRO (24 cores) ~2 hours regression suite run time 10 Desktops
  26. 26. X Desktops: host limit reached
  27. 27. II. Clustering Hosts
  28. 28. II. Clustering Hosts - Parallel run on multi (virtual) hosts - Mac Pro and hosts: test suite - ~1h 30 min
  29. 29. II. Clustering Hosts Issues: - multi-desktops start up is Flaky - simulators can get stuck
  30. 30. II. Clustering Hosts Is it possible to automate multi-simulators on one desktop?
  31. 31. II. Facebook: Web Driver Agent
  32. 32. Developer host Simulator Selenium Driver WDA Runner Your app FB UIA WDA Key Change: Reimplemented UIAutomation dependency Link: https://github.com/facebook/WebDriverAgent
  33. 33. Profit: - 7 simulators on single desktop - More stable! more efficient!
  34. 34. II. Full Migration to FB UIA WDA Issues: - all tests have calabash code
  35. 35. II. Full Migration to FB UIA WDA Issues: - all tests have calabash code - custom app backdoors
  36. 36. II. Full Migration to FB UIA WDA Issues: - all tests have calabash code - custom app backdoors - workarounds and other
  37. 37. II. Full Migration to FB UIA WDA Full Migration Costs: ~x hours Is it possible to do Fast Migration?
  38. 38. II. Calabash + Facebook Fast migration: Facebash! Step Definitions Ruby Client Library Features Developer host Calabash server Simulator Your app UIQuery Run loop: UIAutomation
  39. 39. II. Calabash + Facebook Fast migration: Facebash! Step Definitions Ruby Client Library Features Developer host Calabash server Simulator Your app UIQuery Run loop: UIAutomation Developer host Simulator Step Definitions Ruby Client Library WDA Runner Features Calabash server Your app UIQuery FB UIA WDA
  40. 40. II. iOS 10 Update: from UIAutomation WDAto XCTest WDA Link: https://github.com/facebook/WebDriverAgent Developer host Simulator Selenium Driver Your app FB UIA WDA WDA Runner
  41. 41. II. iOS 10 Update: from UIAutomation WDAto XCTest WDA Link: https://github.com/facebook/WebDriverAgent Developer host Simulator Selenium Driver Your app FB UIA WDA Developer host Simulator Your app FB XCTest WDA WDA Runner Selenium Driver FB simctl
  42. 42. II. iOS Hybrid 2.0 iOS 10 update Developer host Simulator Step Definitions Ruby Client Library WDA Runner Features Calabash server Your app UIQuery FB UIA WDA
  43. 43. II. iOS Hybrid 2.0 iOS 10 update Developer host Simulator Step Definitions Ruby Client Library WDA Runner Features Calabash server Your app UIQuery FB UIA WDA Developer host Simulator Step Definitions Ruby Client Library FB simctl Features Calabash server Your app UIQuery FB XCTest WDA
  44. 44. III. Summary: multi-desktops vs multi-simulators Test Suite: 500 tests Nightly Cucumber Report Happy morning! 80 data Mac PRO Test Suite Run Time 0 min 225 min 450 min 675 min 900 min 1 sim 10 sims 16 sims 75 min 120 min 900 min Multi Desktops Multi Simulators
  45. 45. Developer host Simulator Step Definitions Ruby Client Library FB simctl Features Calabash server Your app UIQuery FB XCTest WDA "Face"book + Cala"bash" = Facebash III. Summary: current approach
  46. 46. - easy to set up Parallel Run on 1 Desktop - keep using of backdoors - full access to internal AUT state - cross iOS app automation - iOS simulator control with FBSimCtl - FB WDA supports Selenium protocol - FB WDA independent of Calabash/Appium Facebook Calabash Calabash Facebook Facebook Facebook Facebook III. Summary: PROs
  47. 47. - support framework parts: calabash server, FB WDA, fbsimctl, ... - "poor" documentation - using of mixed locator strategies III. Summary: PROs and CONs
  48. 48. - Parallel Run in Headless Mode - Simulator Start/Shutdown/Reset optimization - Cold/Hot start iOS app with relogin - record video per the each test in Parallel Run Optimization
  49. 49. Thank you! v.karanevich@corp.badoo.com

×