Droid con slides 2013 mobileci-v1.0


Published on

Slides for Virtual Device Lab

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Need to scale to large # of mobile apps to support
  • Device access, app installation, tedious testingacross large configuration matrix – turnaround time isdays to weeks
  • Talk Points: For all these reasons, we built our own solution, mobile continuous integration with VDL.- Briefly talk about overall mobile CI to show the big picture first.
  • Talk Points: benefits - Saving in QE effort
  • Droid con slides 2013 mobileci-v1.0

    1. 1. Speed to Market in Mobile Development Finding the right solution with continuous integration on real devices
    2. 2. Agenda  Mobile trends & developers’ challenges  Speed & quality through continuous integration - Intuit Virtual Device Lab - Intuit mobile test automation  Takeaways
    3. 3. Aggressive mobile momentum Will surpass desktop by 2014 Proliferation of platforms, OS versions & device types It should just work! “2013 Internet Trends,” KPCB
    4. 4. • Developer challenges Immense configuration matrix – – – • Application delivery through the markets – • Network, different network bandwidth/latency, no-network HTML5 offline app Media challenges – – • App signing, provision (app & device), submission Offline experience – – • OS platforms (Android, iOS, Windows, phone vs. tablet, …) Inputs (touch/gesture, real buttons, voice, camera, GPS receiver) Outputs (portrait/landscape, screen resolution, language) Some codecs don’t work on a device Streaming, sound Security & privacy
    5. 5. Intuit mobile app portfolio today 22 Apple 15 Google 11 Amazon 3 SMS 491 mobile apps for financial institutions 0 5 10 15 20 490
    6. 6. Ramping up Intuit’s innovation engine Pain: Takes days per build to test combinations of device/OS configurations physically & manually, one by one … QA can’t keep up with the changes Goal: Reduce development iteration cycle from days to minutes on an ever-growing number of devices Strategy: Speed & quality through mobile continuous integration Platforms Devices X • Apple • Google • Amazon OS versions X • • • • • • iPhone 4 iPhone 4S iPhone 5 iPad 2 iPad 3 iPad Mini Builds X • • • • • iOS 5.0 iOS 5.1 iOS 6.0 iOS 6.1.x iOS 7.0 Daily builds
    7. 7. What is CI? • • • • Continuous integration (CI) Automate build, test & deployment process Integrate & build early, often, daily, for every check-in Best practice of Agile development Jenkins is widely adopted, flexible & extensible Benefits of CI in a mobile world • Instant feedback on quality, functionality & system impact • Less complex, no integration hell • Detect and address device configuration issues early • Involve stakeholders early • Ultimately achieve shippable release any time
    8. 8. Speed and quality Developers/ IDE Check in Test results through mobile CI Process starts here SCM: source repository Submit the app to app store if passed Analyze test results Key enablers: Check-in triggers automated CI • Mobile device lab • Test automation Automatic test run Run tests on real devices in parallel Jenkins Trigger testing in mobile device lab Real devices Test processor Install the test app to real devices in parallel
    9. 9. rd-party 3 vendor solutions didn’t work for us • Couldn’t handle E2E automation of testing pre-release apps – Dependency on internal resources & infrastructure • Vendor lock-in: customized CI & test automation solutions – No flexibility to use our existing tools & frameworks • Disadvantages of vendors’ shared devices – Security concern on testing on shared devices – No access to corporate network – Cost-prohibitive to reserve devices • Private cloud (on-site setup) with local devices not practical – With sensitive customer data, we need private cloud – Extensive use of devices for daily testing of hundreds of mobile apps is expensive
    10. 10. Why test on real devices? Most mobile features can be tested on simulators, but simulators have limitations: • Device capabilities (GPS, camera, accelerometer, gyros), form factors & pixel density • Real-world experience over the real connection (3G, 4G, crappy Wi-Fi, etc.) • True performance, security tests • Testing on non-default Web browsers (e.g., Opera Mini) • Not all simulator/emulators have good quality (don’t reflect real rendering/behavior, bugs) • Testing SMS & app integration
    11. 11. Intuit Virtual Device Lab
    12. 12. Intuit Virtual Device Lab (VDL) Gives access to real mobile devices using a browser 3-click access User machine Always on Intranet User machine User machine VDL Manufacturers/ devices: iPhone iPad Galaxy Captivate Droid Milestone Moto Optimus Globally available Integrated with dev infrastructure Nexus One Desire Incredible
    13. 13. VDL architecture – Android Internet/intr anet Device interaction app Windows PC Tomcat USB connection Image transfer Java image transferring application Browser & Java applet Developer’s laptop VDL
    14. 14. VDL architecture – iOS Internet/intr anet Mac with Xcode installed VNC server Tomcat USB connection Port forwarding Java VNC client Browser & Java applet Developer’s laptop VDL
    15. 15. Open source tools & technologies used in VDL Key capabilities Android Installing mobile applications into the adb install <APK file> real device iOS Fruitstrap https://github.com/ghughes/fruitstrap Transferring device screen images, controlling device Screencast tool Veency VNC app http://code.google.com/p/androidscre encast Fetching device local files adb pull <filename> sftp command (jailbroken devices only)
    16. 16. VDL key features & functionality today Control the device through standard browser/RESTful APIs Change orientation to landscape or portrait Copy local file to the device from your PC Get file from the device onto your local PC Execute an ADB command on the device See current running logs in your browser Download history logs to your local PC
    17. 17. Continuous integration – how it works Trigger file launches test Monito r for new builds Resource manager Test execution engine Product builds Results Select a device VDL device inventory
    18. 18. Mobile continuous integration Leveraging the Virtual Device Lab for test automation on multiple, real devices simultaneously. 1 Select devices & tests 2 Tests run automatically across selected devices with each build Submit, build, test … in minutes vs. hours 3 See the results!
    19. 19. Next step: over-the-air replaces USB connection VDL Fast onboarding App installation Scalable Test execution Results retrieval Internet/intr anet Any device
    20. 20. Mobile Test Automation
    21. 21. Mobile test automation tools iOS Instrumentation-based Android MonkeyTalk Frank Calabash (LessPainful) TestDroid (BitBar) UISpec Robotium UIAutomation iOS SDK Soasta iOS Driver Appium Non-instrumentation MOET Sikuli Cross-platform solution Android SDK MonkeyRunner DA/PerfectoMob ile eggPlant • Assessed as of 2012
    22. 22. Mobile test automation trends • Abstraction with domain-specific language (DSL) – – Common test script over different devices (e.g., Cucumber, MonkeyScript, MOET) Support behavior-driven development (BDD) • Client/server-based approach (remotely driven tests via HTTP) • Instrumentation & non-instrumentation solutions complement each other • Mobile cloud testing environment – From hardware (cradle) to software approach to drive tests on real devices • Automated OTA app installation & tests
    23. 23. Intuit automation framework of choice/recommendation • Native apps: – Cucumber + Calabash for BDD on iOS & Android • Mobile Web: – Selenium/WebDriver
    24. 24. What is Cucumber? It lets stakeholders (customers, QE or developers) describe how software should behave in plain text. • BDD: Written in business domain-specific language (DSL) – Good abstraction over different device types • One format for multiple purposes – UI feature specification – Automated tests • Cucumber itself is NOT a test automation tool – Add-on to any test automation framework – Examples: Calabash, Sikuli & Frank mobile test frameworks
    25. 25. What is Calabash? • Open source cross-platform mobile test framework – iOS + Android, simulator + real device, native + hybrid • Supports Cucumber for BDD • Application needs instrumentation that embeds a Calabash HTTP server – Tests run from a remote client (in a Jenkins slave) – Tests can be driven via Wi-Fi (no USB wire) • Ruby clients in API level (Java clients in experiment) • Works great in touch simulation & real device testing
    26. 26. Intuit shared test library – rapid test development Common test library High-level test domain-specific language (Cucumber steps) Mobile front-end Product tests Teams focus on creating Native/hybrid test framework Use shared test lib Tests Tests Tests Tests Globalization test framework (I18N/L10N) Security test framework Mobile back-end Performance/load test framework Internal open source repository Contribute … and then contribute to common test library
    27. 27. Stunning results: reduced iteration cycle from hours to minutes From: manual 10 5 Code build Notify email Consolidate report * 50*5 = 250 Upload report 30 Get devices 10 = 6 hours 20 15 30 15 To: automated CI Code build Trigger Deploy on devices Execute tests in parallel Consolidated report in archive 0 5 10 0 Tested on 5 devices Release quality increased = 15 mins TIME SAVED: 95%
    28. 28. VDL cost saving over vendors’ private clouds (today) Assumptions: 50 native apps & Annual cost ($) daily check-in 3rd-party vendor 1 local device in cradle 3rd-party vendor 2 local device w/o cradle (Software approach) Savings Savings Intuit Virtual Device Lab 25 150 Private cloud setup behind corp firewall # of devices Note: As vendor pricing changes, so do the savings
    29. 29. Key takeaways Building custom solution vs. rd-party 3 vendor solution  Understand your technical/platform/global spread  Many vendor solutions provide a larger set of devices/OS versions in the global market  Quite a few offer mobile test services – might be tightly coupled with their device access solution & automation framework  Connecting to your corporate network is a challenge – some provide private cloud (on-site setup within your network) but it’s cost-prohibitive Invest in automation and CI  Automate tests with robust automation framework  Enable rapid test development through shared test library  Adopt CI in mobile development for speed & quality
    30. 30. Thank you!!