Velocity Conference: Increasing Speed To Market In Mobile Development Through Continuous Integration


Published on

In the past few years, Intuit has put an intense focus on ramping up its innovation engine. A big part of this effort is speeding up time-to-market for its hundreds of native mobile apps, while maintaining the highest quality standards. The apps need to work across an ever-increasing number of devices and versions of the operating system.

Once our engineers had access to all possible combinations of devices and operating systems, they needed to be able to kick off automated testing whenever they checked in code. We enable mobile continuous integration by integrating our virtual device lab with our test execution platform and Jenkins system. A notification system detects the arrival of a new build of our mobile products, which kicks off execution of automated tests based on a pre-configured test matrix.

The result is a 95% reduction in the time it takes to test our mobile apps.

Published in: Technology
1 Comment
  • Hello my dear
    My name is miss Mercy i saw your profile today and become interested in you i will like you to send to me mail to my email address so that i will send my picture to you to know me more
    My EMAIL IS ( please i am waiting for your reply.
    yours in love,
    Mercy Kones
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Velocity Conference: Increasing Speed To Market In Mobile Development Through Continuous Integration

  1. 1. Speed to Market inMobile DevelopmentFinding the right solution with continuousintegration on real devicesTina Su, Director of Development, Intuit
  2. 2. About meNearly 20 years of industry experience in R&D,quality and engineering management, with a strongpassion for automation. Engineering Director at Intuit, Developer Productivity Sr. Engineering Director at Yahoo, Mobile Platforms Sr. Engineering Manager at Sun, Java Standard andMobile Editions Compiler Developer at MIPS and TandemTina Su, Intuit
  3. 3. Agenda Mobile trends & developers’ challenges Speed & quality through continuous integration- Intuit Virtual Device Lab- Intuit mobile test automation Takeaways
  4. 4. Aggressive mobile momentum“2013 Internet Trends,” KPCBWill surpass desktopby 2014It should just work!Proliferation of platforms,OS versions & device types
  5. 5. Developer challenges Immense configuration matrix- OS platforms (Android, iOS, Windows, phone vs. tablet, …)- Inputs (touch/gesture, real buttons, voice, camera, GPS receiver)- Outputs (portrait/landscape, screen resolution, language) Application delivery through the markets- App signing, provision (app & device), submission Offline experience- Network, different network bandwidth/latency, no-network- HTML5 offline app Media challenges- Some codecs don’t work on a device- Streaming, sound Security & privacy
  6. 6. Intuit mobile app portfolio todayAppleGoogleAmazonSMS0 5 10 15 20 4902215113491 mobile apps for financial institutions
  7. 7. Ramping up Intuit’s innovation enginePain: Takes days per build to testcombinations of device/OS configurationsphysically & manually, one by one …QA can’t keep up with the changesGoal: Reduce development iteration cyclefrom days to minutes on an ever-growingnumber of devicesStrategy: Speed & quality through mobilecontinuous integration• iPhone 4• iPhone 4S• iPhone 5• iPad 2• iPad 3• iPad MiniDevicesOS versions• iOS 5.0• iOS 5.1• iOS 6.0• iOS 6.1.x• iOS 7.0BuildsX XDailybuilds• Apple• Google• AmazonPlatformsX
  8. 8. Continuous integration (CI)What is CI? Automate build, test & deployment process Integrate & build early, often, daily, for every check-in Best practice of Agile development Jenkins is widely adopted, flexible & extensibleBenefits of CI in a mobile world Instant feedback on quality, functionality & system impact Less complex, no integration hell Detect and address device configuration issues early Involve stakeholders early Ultimately achieve shippable release any time
  9. 9. Speed and qualitythrough mobile CIKey enablers:• Mobile device lab• Test automationTest processorRealdevicesJenkinsSCM:sourcerepositoryDevelopers/IDEAutomatictest runTestresultsSubmit the app toapp store ifpassedCheck inCheck-intriggersautomated CIInstall the test app toreal devices inparallelRun tests on realdevices in parallelAnalyzetestresultsTrigger testing inmobile device labProcess starts here
  10. 10. 3rd-party vendor solutions didn’t work for us Couldn’t handle E2E automation of testing pre-release apps- Dependency on internal resources & infrastructure Vendor lock-in: customized CI & test automation solutions- No flexibility to use our existing tools & frameworks Disadvantages of vendors’ shared devices- Security concern on testing on shared devices- No access to corporate network- Cost-prohibitive to reserve devices Private cloud (on-site setup) with local devices not practical- With sensitive customer data, we need private cloud- Extensive use of devices for daily testing of hundreds of mobile apps is expensive
  11. 11. Why test on real devices?Most mobile features can be tested on simulators, but simulatorshave limitations: Device capabilities (GPS, camera, accelerometer, gyros), formfactors & pixel density Real-world experience over the real connection(3G, 4G, crappy Wi-Fi, etc.) True performance, security tests Testing on non-default Web browsers (e.g., Opera Mini) Not all simulator/emulators have good quality(don’t reflect real rendering/behavior, bugs) Testing SMS & app integration
  12. 12. Intuit Virtual Device Lab
  13. 13. Intuit Virtual Device Lab (VDL)Gives access to real mobile devices using a browserIntranetVDLUser machineUser machineUser machine3-click accessAlways onIntegrated withdev infrastructureGlobally availableiPhoneiPadGalaxyCaptivateOptimus Nexus OneDesireIncredibleManufacturers/devices:DroidMilestoneMoto
  14. 14. VDL architecture – AndroidWindows PCTomcatJava imagetransferringapplicationBrowser &Java appletInternet/intranetDeviceinteractionappDeveloper’s laptopVDLImage transferUSB connection
  15. 15. VDL architecture – iOSMac with XcodeinstalledTomcatJava VNCclientPortforwardingBrowser &Java appletInternet/intranetVNC serverUSB connectionDeveloper’s laptopVDL
  16. 16. Open source tools & technologies used in VDLKey capabilities Android iOSInstalling mobile applications intothe real deviceadb install <APK file> Fruitstrap device screen images,controlling deviceScreencast tool VNC appFetching device local files adb pull <filename> sftp command(jailbroken devices only)
  17. 17. VDL key features & functionality todayControl the device through standardbrowser/RESTful APIsChange orientation to landscape or portraitCopy local file to the device from your PCGet file from the device onto your local PCExecute an ADB command on the deviceSee current running logs in your browserDownload history logs to your local PC
  18. 18. Continuous integration – how it worksProductbuildsTrigger file launches testVDL deviceinventoryResourcemanagerSelect a deviceTest executionengineResultsMonitor fornewbuilds
  19. 19. Mobile continuous integrationLeveraging the Virtual Device Lab for test automation on multiple, real devices simultaneously.Submit, build, test … in minutes vs. hoursSelect devices & tests1 Tests run automatically across selecteddevices with each build2 See the results!3
  20. 20. Next step: over-the-air replaces USB connectionFast onboardingScalableAny deviceInternet/intranetAppinstallationTestexecutionResultsretrievalVDL
  21. 21. Mobile Test Automation
  22. 22. Mobile test automation toolsInstrumentation-basedNon-instrumentationAndroidiOSMonkeyRunnerMOETSikuliAndroid SDKRobotiumTestDroid (BitBar)Calabash(LessPainful)SoastaiOS SDKUIAutomationUISpecFrankMonkeyTalkDA/PerfectoMobileeggPlantCross-platform solutioniOS Driver Appium• Assessed as of 2012
  23. 23. Mobile test automation trends Abstraction with domain-specific language (DSL)- Common test script over different devices (e.g., Cucumber, MonkeyScript, MOET)- Support behavior-driven development (BDD) Client/server-based approach (remotely driven tests via HTTP) Instrumentation & non-instrumentation solutions complementeach other Mobile cloud testing environment- From hardware (cradle) to software approach to drive tests onreal devices Automated OTA app installation & tests
  24. 24. Intuit automation framework of choice/recommendation Native apps:- Cucumber + Calabash for BDD on iOS & Android Mobile Web:- Selenium/WebDriver
  25. 25. What is Cucumber?It lets stakeholders (customers, QE or developers) describe howsoftware should behave in plain text. BDD: Written in business domain-specific language (DSL)- Good abstraction over different device types One format for multiple purposes- UI feature specification- Automated tests Cucumber itself is NOT a test automation tool- Add-on to any test automation framework- Examples: Calabash, Sikuli & Frank mobile test frameworks
  26. 26. What is Calabash? Open source cross-platform mobile test framework- iOS + Android, simulator + real device, native + hybrid Supports Cucumber for BDD Application needs instrumentation that embeds a Calabash HTTP server- Tests run from a remote client (in a Jenkins slave)- Tests can be driven via Wi-Fi (no USB wire) Ruby clients in API level (Java clients in experiment) Works great in touch simulation & real device testing
  27. 27. Intuit shared test library – rapid test developmentInternal open source repositoryTeams focus oncreatingUse shared test lib… and then contribute tocommon test libraryTestsContributeTestsCommon test library Product testsTestsTestsGlobalization test framework(I18N/L10N)Performance/load test frameworkSecurity test frameworkNative/hybrid test frameworkHigh-level test domain-specific language(Cucumber steps)Mobileback-endMobilefront-end
  28. 28. CodebuildDeployTestResultsConnectdeviceNotifyemailGetdevicesConsolidatereportUploadreportFrom: manual50*5= 25015 30510201530 10= 6 hoursCodebuildTriggerExecute testsin parallelConsolidatedreport in archiveDeploy ondevices0 5 10 0= 15 minsTo: automated CITIME SAVED: 95%Release quality increased*Stunning results: reduced iteration cycle from hours to minutesTested on 5 devices
  29. 29. VDL cost saving over vendors’ private clouds (today)Annualcost ($)# of devicesIntuit Virtual Device Lab1503rd-party vendor 1local device in cradleAssumptions: 50 nativeapps & daily check-inNote: As vendor pricingchanges, so do the savingsSavings253rd-party vendor 2local device w/o cradle(Software approach)SavingsPrivate cloud setup behind corp firewall
  30. 30. Key takeawaysBuilding custom solution vs. 3rd-party vendor solution Understand your technical/platform/global spread Many vendor solutions provide a larger set of devices/OS versionsin the global market Quite a few offer mobile test services – might be tightly coupled with theirdevice access solution & automation framework Connecting to your corporate network is a challenge – some provideprivate cloud (on-site setup within your network) but it’s cost-prohibitiveInvest in automation and CI Automate tests with robust automation framework Enable rapid test development through shared test library Adopt CI in mobile development for speed & quality
  31. 31. More mobile innovations at Intuit Mobile crash reporter Mobile components & design patterns Mobile analytics SDK & gatewayVisit booth #601 – Accelerate Development Demos & conversations Contest Mobile device charging station Foosball We’re hiring !Win a weekend with a TESLA!
  32. 32. Thank you!!