Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How we use Appium as our product's core library

482 views

Published on

Presentation held in AppiumConf 2019 ( https://appiumconf.com/ )

Published in: Technology
  • My car battery was completely dead before I used your methods! I reconditioned my dead car battery a few weeks ago with your program and it's been working perfectly since then! My car battery was completely dead before I used your methods. This just saved me hundreds of dollars on a new battery! ▲▲▲ https://bit.ly/2t1uc6e
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

How we use Appium as our product's core library

  1. 1. How we use Appium as our product’s core library Nozomi Ito @ito_nozomi TRIDENT (Japan)
  2. 2. About Me p Founder/CEO/Developer of TRIDENT Inc. (Japan) p Love Selenium and Appium p Organizer of selenium.jp and SeConf Tokyo
  3. 3. About Me and India p My first visit to India n When I was an university student (2007) n My first international trip
  4. 4. Agenda p Background of this talk p Provide more user-friendly Appium wrapper feature p Make Appium more reliable and stable
  5. 5. Agenda p Background of this talk p Provide more user-friendly Appium wrapper feature p Make Appium more reliable and stable
  6. 6. Background - My company’s product - p Magic Pod ( magic-pod.com/en ) n Test automation cloud service n Codeless / human-friendly script p Backend is Appium n iOS: XCUITest Driver n Android: UiAutomator2 Driver n Client library: node.js
  7. 7. Background - Difficulty Magic Pod faces - 1. Some users are not familiar with Appium n Because of codeless testing tool n Need to provide user-friendly commands 2. Requirements for quality/stability of Appium is severe n Appium not for test code, but for product code
  8. 8. Agenda p Background of this talk p Provide more user-friendly Appium wrapper feature p Make Appium more reliable and stable
  9. 9. User-friendly feature p Commands of Magic Pod are basically wrapper of Appium commands p Some Magic Pod commands are aggressive/tricky p I introduce unique ones and their implementations n Scroll/Wait/Picker/Source Tree/AI locator
  10. 10. Scroll/Swipe to lazy-loaded element p Appium has scroll-to-specific-element feature: n XCUITest Driver: driver.execute(“mobile: scroll”, {“name”: “…”}) n UiAutomator2 Driver: driver.elementByAndroidUIAutomator(( ‘new UiScrollable(new UiSelector()).scrollIntoView(text(”…"))’) p Problem: n XCUITest Driver: cannot scroll to a lazy-loaded element n UiAutomator2 Driver: not work for all elements type
  11. 11. Scroll/Swipe to lazy-loaded element p Solution: helper.swipeTo(“accessibility id”, ”Next”, “top”, 20) p Covers every cases by this single method
  12. 12. Scroll/Swipe to lazy-loaded element Code & Demo
  13. 13. Repository p Most of code for my presentation is open source p Magic-Pod/wd-appium-helper
  14. 14. p Problem: n Magic Pod users are not usually Appium expert n Writing wait logic is sometimes difficult p Wait commands Magic Pod supports n Implicit wait for findElement/Assert n Various explicit wait commands n Plus we have an aggressive catch-all wait Catch-all wait
  15. 15. p Solution: waitWholeRendered(60) p Wait until the screen capture result is the same p The same behavior as human’s wait p Ignores header part (clock, Wi-Fi, battery, etc) Catch-all wait
  16. 16. Catch-all wait Code & Demo
  17. 17. Catch-all wait p Limitation: n AUT must have proper load animation/indicator
  18. 18. p Appium has Picker manipulation method n UiAutomator2 Driver: driver.sendKeys(element, “…”) n XCUITest Driver: driver.execute(“mobile: selectPickerWheelValue”, {“order”: “next”, “offset”: 0.15, element: …}) p iOS Problem: n What we want is to set the specific picker value n Picker move direction depends on the current value iOS picker wheel
  19. 19. p Solution: inputToPicker(element,” 1998”, “auto”, 20) p Set the value by continuous small swipes p Auto swipe direction calculation iOS picker wheel
  20. 20. p Direction n top/bottom: The case you know the swipe direction n top_then_bottom/bottom_then_top: direction + fallback n auto: infer the direction based on number value iOS picker wheel
  21. 21. iOS picker wheel Code & Demo
  22. 22. p Improvement idea: n Infer more data: Sunday ~ Saturday, Jun ~ Dec, country name in dictionary order, etc iOS picker wheel
  23. 23. Faster source XML for iOS p Source XML is very important for the auto-locator generation logic p Appium has source-XML-retrieval feature driver.source() p Problem: n Big tree retrieval takes too much time n Big tree retrieval result can be empty tree due to time out
  24. 24. Faster source XML for iOS p driver.execute( “mobile: source”, {format: “description”}) n iOS only n YAML-like Apple’s original format n Difficult to parse: separators in the value are sometimes not escaped n Format may be changed in the future n size/position is 1 ~ 2 pixel different from standard tree n no “visible” information p We can generate XML tree based on this source tree
  25. 25. Faster source XML for iOS p Performance comparison for Instagram app n Normal source tree: 1,274 msec n Description source tree: 189 msec p Performance comparison for a heavy app n Normal source tree: 166,580 msec n Description source tree: 5,896 msec
  26. 26. Faster source XML for iOS Demo & Code
  27. 27. Use of AI locator p What is the AI locator? n appium-classifier-plugin (testdotai/appium-classifier-plugin ) n Created by test.ai team p Identify element by deep-learning image recognition driver.elementByCustom("ai:cart") http://nkdkccmbr.hateblo.jp/entry/2016/10/06/222245 Neural network
  28. 28. Use of AI locator p Challenges of AI locator n AI recognition result is not always stable n Need to know which AI locator can be used for an image preliminarily
  29. 29. Use of AI locator p AI locator is important part of Magic Pod p Currently our own logic + model + training data set p Now switching to the appium-classifier-plugin
  30. 30. Use of AI locator - UI map generation - { “menu icon”: “accessibility id=btn1”, “search icon”: “accessibility id=btn2”, “add icon”: “accessibility id=btn3”, ….. }Neural network + OCR
  31. 31. Use of AI locator - UI map generation - p Recognition Example
  32. 32. Use of AI locator - find element by AI locator - p Locator candidates in UI map contain AI locator p Access to AI locator is easier
  33. 33. Use of AI locator - Solution for challenges - p AI recognition result is not always stable n → Use for UI map p Need to know which AI locator can be used for an image n → Suggestion by the locator generator
  34. 34. p Appium has hideKeyboard feature driver.hideKeyboard() p Problem for iOS: n Last key of the Japanese keyboard is not hide key! n Kana-Kanji conversion Hide keyboard for Japanese
  35. 35. p Solution: p Use bottom-right-most button as hideKey instead of last button in the element tree p Tap last button twice if Kana-Kanji conversion happens p Limitation: n Slower than standard hideKeyboard Hide keyboard for Japanese
  36. 36. Agenda p Background of this talk p Provide more user-friendly Appium wrapper feature p Make Appium more reliable and stable
  37. 37. Make Appium more reliable and stable 1. How to prevent the regression of Appium (and Xcode, Apple framework etc) n Unfortunately sometimes we’ve encountered 2. How to resolve issues derived from Appium (and Xcode, Apple framework etc) n Many trouble reports from users with various apps, devices
  38. 38. 1. Prevent regression - Prepare test set for Appium - p The test set for the important features for Magic Pod n Headless, ReduceMotion, Session-less source, Parallel test p Android/iOS, real device/emulator/simulator, app/bundleID/package&activity p Magic-Pod/AppiumRegressionCheck
  39. 39. 1. Prevent regression - Prepare test set for Appium - p Easily identify the cause of the bug n Whether due to our own code or Appium p The product has become much more stable
  40. 40. 2. Resolve issues - Report issues, submit PRs - p Issues n appium/appium#9934 p Pull Requests n appium/appium-xcuitest-driver#537 n appium/appium-adb#314 n appium/appium-adb#315 n mykola-mokhnach/appium-adb#1
  41. 41. 2. Resolve issues - Collaborate with a committer - p Problems I faced during issue/PR submission: n Sometimes deep knowledge about Appium is required p Then I asked help to @Kazu_cocoa
  42. 42. 2. Resolve issues - Collaborate with a committer - p @Kazu_cocoa (at HeadSpin) has helped Magic Pod by providing consultation about Appium-related issues and fixing them since early 2018 p Share the reproducible environment with him p Many PRs based on my bug reports n Issues of Magic-Pod/AppiumRegressionCheck
  43. 43. 2. Resolve issues - Easy bug report feature - p Problem: n Exchanging Appium log with users are very troublesome p Solution n Inquiry button on the test result page n Script/Appium Log are sent to support team just by one click n Troubleshooting became much easier
  44. 44. 2. Resolve issues - Easy bug report feature - p Problem: Half of inquiries are actually not bug n No problem for Magic Pod, since it is the commercial tool p The support team can categorize reports and can report only real Appium issue to Appium team.
  45. 45. Future plan & Conclusion p Selective use of Espresso Driver and UiAutomator2 Driver p Switch to WebdriverIO p Switch to the AI model of appium-classifier-plugin p Continue to contribute to Appium in some way!

×