Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Velocity2013 mobile ci_intuit


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Velocity2013 mobile ci_intuit

  1. 1. Speed to Market inMobile DevelopmentFinding the right solution with continuousintegration on real devicesTina Su, Director of Development, Intuit
  2. 2. About meNearly 20 years of industry experience in R&D,quality and engineering management, with a strongpassion for automation.§  Engineering Director at Intuit, Developer Productivity§  Sr. Engineering Director at Yahoo, Mobile Platforms§  Sr. Engineering Manager at Sun, Java Standard andMobile Editions§  Compiler Developer at MIPS and TandemTina Su, Intuit
  3. 3. Agenda§  Mobile trends & developers’ challenges§  Speed & quality through continuous integration- Intuit Virtual Device Lab- Intuit mobile test automation§  Takeaways
  4. 4. Aggressive mobile momentum“2013 Internet Trends,” KPCBWill surpass desktopby 2014It should just work!Proliferation of platforms,OS versions & device types
  5. 5. Developer challenges§  Immense configuration matrix-  OS platforms (Android, iOS, Windows, phone vs. tablet, …)-  Inputs (touch/gesture, real buttons, voice, camera, GPS receiver)-  Outputs (portrait/landscape, screen resolution, language)§  Application delivery through the markets-  App signing, provision (app & device), submission§  Offline experience-  Network, different network bandwidth/latency, no-network-  HTML5 offline app§  Media challenges-  Some codecs don’t work on a device-  Streaming, sound§  Security & privacy
  6. 6. Intuit mobile app portfolio todayApple Google Amazon SMS 0 5 10 15 20 490 22 15 11 3 491 mobile apps for financial institutions
  7. 7. Ramping up Intuit’s innovation enginePain: Takes days per build to testcombinations of device/OS configurationsphysically & manually, one by one …QA can’t keep up with the changesGoal: Reduce development iteration cyclefrom days to minutes on an ever-growingnumber of devicesStrategy: Speed & quality through mobilecontinuous integration•  iPhone 4•  iPhone 4S•  iPhone 5•  iPad 2•  iPad 3•  iPad MiniDevices OS versions•  iOS 5.0•  iOS 5.1•  iOS 6.0•  iOS 6.1.x•  iOS 7.0BuildsX XDailybuilds•  Apple•  Google•  AmazonPlatformsX
  8. 8. Continuous integration (CI)What is CI?§  Automate build, test & deployment process§  Integrate & build early, often, daily, for every check-in§  Best practice of Agile development§  Jenkins is widely adopted, flexible & extensibleBenefits of CI in a mobile world§  Instant feedback on quality, functionality & system impact§  Less complex, no integration hell§  Detect and address device configuration issues early§  Involve stakeholders early§  Ultimately achieve shippable release any time
  9. 9. Speed and qualitythrough mobile CIKey enablers:•  Mobile device lab•  Test automationTest processorRealdevicesJenkinsSCM:sourcerepositoryDevelopers/IDEAutomatictest runTest resultsSubmit the app toapp store if passedCheck inCheck-intriggersautomated CIInstall the test app toreal devices in parallelRun tests on realdevices in parallelAnalyzetest resultsTrigger testing inmobile device labProcess starts here
  10. 10. 3rd-party vendor solutions didn’t work for us§  Couldn’t handle E2E automation of testing pre-release apps-  Dependency on internal resources & infrastructure§  Vendor lock-in: customized CI & test automation solutions-  No flexibility to use our existing tools & frameworks§  Disadvantages of vendors’ shared devices-  Security concern on testing on shared devices-  No access to corporate network-  Cost-prohibitive to reserve devices§  Private cloud (on-site setup) with local devices not practical-  With sensitive customer data, we need private cloud-  Extensive use of devices for daily testing of hundreds of mobile apps is expensive
  11. 11. Why test on real devices?Most mobile features can be tested on simulators, butsimulators have limitations:§  Device capabilities (GPS, camera, accelerometer, gyros),form factors & pixel density§  Real-world experience over the real connection(3G, 4G, crappy Wi-Fi, etc.)§  True performance, security tests§  Testing on non-default Web browsers (e.g., Opera Mini)§  Not all simulator/emulators have good quality(don’t reflect real rendering/behavior, bugs)§  Testing SMS & app integration
  12. 12. Intuit Virtual Device Lab
  13. 13. Intuit Virtual Device Lab (VDL)Gives access to real mobile devices using a browserIntranetVDLUser machineUser machineUser machine3-click accessAlways onIntegrated withdev infrastructureGlobally availableiPhoneiPadGalaxyCaptivateOptimus Nexus OneDesireIncredibleManufacturers/devices: DroidMilestoneMoto
  14. 14. VDL architecture – AndroidWindows PCTomcatJava imagetransferringapplicationBrowser Java appletInternet/intranetDeviceinteraction appDeveloper’s laptopVDLImage transferUSB connection
  15. 15. VDL architecture – iOSMac with XcodeinstalledTomcatJava VNCclientPort forwardingBrowser Java appletInternet/intranetVNC serverUSB connectionDeveloper’s laptopVDL
  16. 16. Open source tools technologies used in VDLKey capabilities Android iOSInstalling mobile applications intothe real deviceadb install APK file Fruitstrap device screen images,controlling deviceScreencast tool VNC appFetching device local files adb pull filename sftp command(jailbroken devices only)
  17. 17. VDL key features functionality todayControl the device through standard browser/RESTful APIsChange orientation to landscape or portraitCopy local file to the device from your PCGet file from the device onto your local PCExecute an ADB command on the deviceSee current running logs in your browserDownload history logs to your local PC
  18. 18. Continuous integration – how it worksProduct buildsTrigger file launches testVDL deviceinventoryResourcemanagerSelect a deviceTest executionengineResultsMonitorfor newbuilds
  19. 19. Mobile continuous integrationLeveraging the Virtual Device Lab for test automation on multiple, real devices simultaneously.Submit, build, test … in minutes vs. hoursSelect devices tests1 Tests run automatically across selecteddevices with each build2 See the results!3
  20. 20. Next step: over-the-air replaces USB connectionFast onboardingScalableAny deviceInternet/intranetAppinstallationTestexecutionResultsretrievalVDL
  21. 21. Mobile Test Automation
  22. 22. Mobile test automation toolsInstrumentation-basedNon-instrumentationAndroidiOSMonkeyRunnerMOETSikuliAndroid SDKRobotiumTestDroid (BitBar)Calabash(LessPainful)SoastaiOS SDKUIAutomationUISpecFrankMonkeyTalkDA/PerfectoMobileeggPlantCross-platform solutioniOS Driver Appium•  Assessed as of 2012
  23. 23. Mobile test automation trends§  Abstraction with domain-specific language (DSL)-  Common test script over different devices (e.g., Cucumber, MonkeyScript, MOET)-  Support behavior-driven development (BDD)§  Client/server-based approach (remotely driven tests via HTTP)§  Instrumentation non-instrumentation solutions complementeach other§  Mobile cloud testing environment-  From hardware (cradle) to software approach to drive tests onreal devices§  Automated OTA app installation tests
  24. 24. Intuit automation framework of choice/recommendation§  Native apps:-  Cucumber + Calabash for BDD on iOS Android§  Mobile Web:-  Selenium/WebDriver
  25. 25. What is Cucumber?It lets stakeholders (customers, QE or developers) describe howsoftware should behave in plain text.§  BDD: Written in business domain-specific language (DSL)-  Good abstraction over different device types§  One format for multiple purposes-  UI feature specification-  Automated tests§  Cucumber itself is NOT a test automation tool-  Add-on to any test automation framework-  Examples: Calabash, Sikuli Frank mobile test frameworks
  26. 26. What is Calabash?§  Open source cross-platform mobile test framework-  iOS + Android, simulator + real device, native + hybrid§  Supports Cucumber for BDD§  Application needs instrumentation that embeds a Calabash HTTP server-  Tests run from a remote client (in a Jenkins slave)-  Tests can be driven via Wi-Fi (no USB wire)§  Ruby clients in API level (Java clients in experiment)§  Works better in touch simulation real device testing§  Mobile cloud test execution available by LessPainful
  27. 27. Intuit shared test library – rapid test developmentInternal open source repositoryTeams focus on creatingUse shared test lib… and then contribute tocommon test libraryTestsContributeTestsCommon test library Product testsTestsTestsGlobalization test framework(I18N/L10N)Performance/load test frameworkSecurity test frameworkNative/hybrid test frameworkHigh-level test domain-specific language(Cucumber steps)Mobileback-endMobilefront-end
  28. 28. CodebuildDeployTestResultsConnectdeviceNotifyemailGetdevicesConsolidatereportUploadreportFrom: manual50*5= 25015 30510201530 10= 6 hoursCodebuildTriggerExecute testsin parallelConsolidatedreport in archiveDeploy ondevices0 5 10 0= 15 minsTo: automated CITIME SAVED: 95%Release quality increased*Stunning results: reduced iteration cycle from hours to minutesTested on 5 devices
  29. 29. VDL cost saving over vendors’ private clouds (today)Annualcost ($)# of devicesIntuit Virtual Device Lab1503rd-party vendor 1local device in cradleAssumptions: 50 nativeapps daily check-inNote: As vendor pricingchanges, so do the savingsSavings253rd-party vendor 2local device w/o cradle(Software approach)SavingsPrivate cloud setup behind corp firewall
  30. 30. Key takeawaysBuilding custom solution vs. 3rd-party vendor solution§  Understand your technical/platform/global spread§  Many vendor solutions provide a larger set of devices/OS versionsin the global market§  Quite a few offer mobile test services – might be tightly coupled with theirdevice access solution automation framework§  Connecting to your corporate network is a challenge – some provideprivate cloud (on-site setup within your network) but it’s cost-prohibitiveInvest in automation and CI§  Automate tests with robust automation framework§  Enable rapid test development through shared test library§  Adopt CI in mobile development for speed quality
  31. 31. More mobile innovations at Intuit§  Mobile crash reporter§  Mobile components design patterns§  Mobile analytics SDK gatewayVisit booth #601 – Accelerate Development§  Demos conversations§  Contest§  Mobile device charging station§  Foosball§  We’re hiring !Win a weekend with a TESLA!
  32. 32. Thank you!!