Your SlideShare is downloading. ×
  • Like
Velocity2013 mobile ci_intuit
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Velocity2013 mobile ci_intuit



Published in Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads


Total Views
On SlideShare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. Speed to Market inMobile DevelopmentFinding the right solution with continuousintegration on real devicesTina Su, Director of Development, Intuit
  • 2. About meNearly 20 years of industry experience in R&D,quality and engineering management, with a strongpassion for automation.§  Engineering Director at Intuit, Developer Productivity§  Sr. Engineering Director at Yahoo, Mobile Platforms§  Sr. Engineering Manager at Sun, Java Standard andMobile Editions§  Compiler Developer at MIPS and TandemTina Su, Intuit
  • 3. Agenda§  Mobile trends & developers’ challenges§  Speed & quality through continuous integration- Intuit Virtual Device Lab- Intuit mobile test automation§  Takeaways
  • 4. Aggressive mobile momentum“2013 Internet Trends,” KPCBWill surpass desktopby 2014It should just work!Proliferation of platforms,OS versions & device types
  • 5. Developer challenges§  Immense configuration matrix-  OS platforms (Android, iOS, Windows, phone vs. tablet, …)-  Inputs (touch/gesture, real buttons, voice, camera, GPS receiver)-  Outputs (portrait/landscape, screen resolution, language)§  Application delivery through the markets-  App signing, provision (app & device), submission§  Offline experience-  Network, different network bandwidth/latency, no-network-  HTML5 offline app§  Media challenges-  Some codecs don’t work on a device-  Streaming, sound§  Security & privacy
  • 6. Intuit mobile app portfolio todayApple Google Amazon SMS 0 5 10 15 20 490 22 15 11 3 491 mobile apps for financial institutions
  • 7. Ramping up Intuit’s innovation enginePain: Takes days per build to testcombinations of device/OS configurationsphysically & manually, one by one …QA can’t keep up with the changesGoal: Reduce development iteration cyclefrom days to minutes on an ever-growingnumber of devicesStrategy: Speed & quality through mobilecontinuous integration•  iPhone 4•  iPhone 4S•  iPhone 5•  iPad 2•  iPad 3•  iPad MiniDevices OS versions•  iOS 5.0•  iOS 5.1•  iOS 6.0•  iOS 6.1.x•  iOS 7.0BuildsX XDailybuilds•  Apple•  Google•  AmazonPlatformsX
  • 8. Continuous integration (CI)What is CI?§  Automate build, test & deployment process§  Integrate & build early, often, daily, for every check-in§  Best practice of Agile development§  Jenkins is widely adopted, flexible & extensibleBenefits of CI in a mobile world§  Instant feedback on quality, functionality & system impact§  Less complex, no integration hell§  Detect and address device configuration issues early§  Involve stakeholders early§  Ultimately achieve shippable release any time
  • 9. Speed and qualitythrough mobile CIKey enablers:•  Mobile device lab•  Test automationTest processorRealdevicesJenkinsSCM:sourcerepositoryDevelopers/IDEAutomatictest runTest resultsSubmit the app toapp store if passedCheck inCheck-intriggersautomated CIInstall the test app toreal devices in parallelRun tests on realdevices in parallelAnalyzetest resultsTrigger testing inmobile device labProcess starts here
  • 10. 3rd-party vendor solutions didn’t work for us§  Couldn’t handle E2E automation of testing pre-release apps-  Dependency on internal resources & infrastructure§  Vendor lock-in: customized CI & test automation solutions-  No flexibility to use our existing tools & frameworks§  Disadvantages of vendors’ shared devices-  Security concern on testing on shared devices-  No access to corporate network-  Cost-prohibitive to reserve devices§  Private cloud (on-site setup) with local devices not practical-  With sensitive customer data, we need private cloud-  Extensive use of devices for daily testing of hundreds of mobile apps is expensive
  • 11. Why test on real devices?Most mobile features can be tested on simulators, butsimulators have limitations:§  Device capabilities (GPS, camera, accelerometer, gyros),form factors & pixel density§  Real-world experience over the real connection(3G, 4G, crappy Wi-Fi, etc.)§  True performance, security tests§  Testing on non-default Web browsers (e.g., Opera Mini)§  Not all simulator/emulators have good quality(don’t reflect real rendering/behavior, bugs)§  Testing SMS & app integration
  • 12. Intuit Virtual Device Lab
  • 13. Intuit Virtual Device Lab (VDL)Gives access to real mobile devices using a browserIntranetVDLUser machineUser machineUser machine3-click accessAlways onIntegrated withdev infrastructureGlobally availableiPhoneiPadGalaxyCaptivateOptimus Nexus OneDesireIncredibleManufacturers/devices: DroidMilestoneMoto
  • 14. VDL architecture – AndroidWindows PCTomcatJava imagetransferringapplicationBrowser &Java appletInternet/intranetDeviceinteraction appDeveloper’s laptopVDLImage transferUSB connection
  • 15. VDL architecture – iOSMac with XcodeinstalledTomcatJava VNCclientPort forwardingBrowser &Java appletInternet/intranetVNC serverUSB connectionDeveloper’s laptopVDL
  • 16. Open source tools & technologies used in VDLKey capabilities Android iOSInstalling mobile applications intothe real deviceadb install <APK file> Fruitstrap device screen images,controlling deviceScreencast tool VNC appFetching device local files adb pull <filename> sftp command(jailbroken devices only)
  • 17. VDL key features & functionality todayControl the device through standard browser/RESTful APIsChange orientation to landscape or portraitCopy local file to the device from your PCGet file from the device onto your local PCExecute an ADB command on the deviceSee current running logs in your browserDownload history logs to your local PC
  • 18. Continuous integration – how it worksProduct buildsTrigger file launches testVDL deviceinventoryResourcemanagerSelect a deviceTest executionengineResultsMonitorfor newbuilds
  • 19. Mobile continuous integrationLeveraging the Virtual Device Lab for test automation on multiple, real devices simultaneously.Submit, build, test … in minutes vs. hoursSelect devices & tests1 Tests run automatically across selecteddevices with each build2 See the results!3
  • 20. Next step: over-the-air replaces USB connectionFast onboardingScalableAny deviceInternet/intranetAppinstallationTestexecutionResultsretrievalVDL
  • 21. Mobile Test Automation
  • 22. Mobile test automation toolsInstrumentation-basedNon-instrumentationAndroidiOSMonkeyRunnerMOETSikuliAndroid SDKRobotiumTestDroid (BitBar)Calabash(LessPainful)SoastaiOS SDKUIAutomationUISpecFrankMonkeyTalkDA/PerfectoMobileeggPlantCross-platform solutioniOS Driver Appium•  Assessed as of 2012
  • 23. Mobile test automation trends§  Abstraction with domain-specific language (DSL)-  Common test script over different devices (e.g., Cucumber, MonkeyScript, MOET)-  Support behavior-driven development (BDD)§  Client/server-based approach (remotely driven tests via HTTP)§  Instrumentation & non-instrumentation solutions complementeach other§  Mobile cloud testing environment-  From hardware (cradle) to software approach to drive tests onreal devices§  Automated OTA app installation & tests
  • 24. Intuit automation framework of choice/recommendation§  Native apps:-  Cucumber + Calabash for BDD on iOS & Android§  Mobile Web:-  Selenium/WebDriver
  • 25. What is Cucumber?It lets stakeholders (customers, QE or developers) describe howsoftware should behave in plain text.§  BDD: Written in business domain-specific language (DSL)-  Good abstraction over different device types§  One format for multiple purposes-  UI feature specification-  Automated tests§  Cucumber itself is NOT a test automation tool-  Add-on to any test automation framework-  Examples: Calabash, Sikuli & Frank mobile test frameworks
  • 26. What is Calabash?§  Open source cross-platform mobile test framework-  iOS + Android, simulator + real device, native + hybrid§  Supports Cucumber for BDD§  Application needs instrumentation that embeds a Calabash HTTP server-  Tests run from a remote client (in a Jenkins slave)-  Tests can be driven via Wi-Fi (no USB wire)§  Ruby clients in API level (Java clients in experiment)§  Works better in touch simulation & real device testing§  Mobile cloud test execution available by LessPainful
  • 27. Intuit shared test library – rapid test developmentInternal open source repositoryTeams focus on creatingUse shared test lib… and then contribute tocommon test libraryTestsContributeTestsCommon test library Product testsTestsTestsGlobalization test framework(I18N/L10N)Performance/load test frameworkSecurity test frameworkNative/hybrid test frameworkHigh-level test domain-specific language(Cucumber steps)Mobileback-endMobilefront-end
  • 28. CodebuildDeployTestResultsConnectdeviceNotifyemailGetdevicesConsolidatereportUploadreportFrom: manual50*5= 25015 30510201530 10= 6 hoursCodebuildTriggerExecute testsin parallelConsolidatedreport in archiveDeploy ondevices0 5 10 0= 15 minsTo: automated CITIME SAVED: 95%Release quality increased*Stunning results: reduced iteration cycle from hours to minutesTested on 5 devices
  • 29. VDL cost saving over vendors’ private clouds (today)Annualcost ($)# of devicesIntuit Virtual Device Lab1503rd-party vendor 1local device in cradleAssumptions: 50 nativeapps & daily check-inNote: As vendor pricingchanges, so do the savingsSavings253rd-party vendor 2local device w/o cradle(Software approach)SavingsPrivate cloud setup behind corp firewall
  • 30. Key takeawaysBuilding custom solution vs. 3rd-party vendor solution§  Understand your technical/platform/global spread§  Many vendor solutions provide a larger set of devices/OS versionsin the global market§  Quite a few offer mobile test services – might be tightly coupled with theirdevice access solution & automation framework§  Connecting to your corporate network is a challenge – some provideprivate cloud (on-site setup within your network) but it’s cost-prohibitiveInvest in automation and CI§  Automate tests with robust automation framework§  Enable rapid test development through shared test library§  Adopt CI in mobile development for speed & quality
  • 31. More mobile innovations at Intuit§  Mobile crash reporter§  Mobile components & design patterns§  Mobile analytics SDK & gatewayVisit booth #601 – Accelerate Development§  Demos & conversations§  Contest§  Mobile device charging station§  Foosball§  We’re hiring !Win a weekend with a TESLA!
  • 32. Thank you!!