Tina Su discusses how Intuit implemented continuous integration and mobile test automation to speed up their mobile development cycle. They created an Intuit Virtual Device Lab (VDL) that gives developers browser-based access to real mobile devices. This allows automated tests to be run simultaneously across many device configurations. Intuit also developed a shared test library and uses Cucumber and Calabash for behavior-driven testing on Android and iOS. Continuous integration with the VDL and test automation reduced Intuit's iteration cycle from days to minutes, improving release quality and developer productivity.

1. Speed to Market in
Mobile Development
Finding the right solution with continuous
integration on real devices
Tina Su, Director of Development, Intuit

2. About me
Nearly 20 years of industry experience in R&D,
quality and engineering management, with a strong
passion for automation.
§ Engineering Director at Intuit, Developer Productivity
§ Sr. Engineering Director at Yahoo, Mobile Platforms
§ Sr. Engineering Manager at Sun, Java Standard and
Mobile Editions
§ Compiler Developer at MIPS and Tandem
Tina Su, Intuit

3. Agenda
§ Mobile trends & developers’ challenges
§ Speed & quality through continuous integration
- Intuit Virtual Device Lab
- Intuit mobile test automation
§ Takeaways

4. Aggressive mobile momentum
“2013 Internet Trends,” KPCB
Will surpass desktop
by 2014
It should just work!
Proliferation of platforms,
OS versions & device types

5. Developer challenges
§ Immense configuration matrix
- OS platforms (Android, iOS, Windows, phone vs. tablet, …)
- Inputs (touch/gesture, real buttons, voice, camera, GPS receiver)
- Outputs (portrait/landscape, screen resolution, language)
§ Application delivery through the markets
- App signing, provision (app & device), submission
§ Offline experience
- Network, different network bandwidth/latency, no-network
- HTML5 offline app
§ Media challenges
- Some codecs don’t work on a device
- Streaming, sound
§ Security & privacy

6. Intuit mobile app portfolio today
Apple
Google
Amazon
SMS
0
5
10
15
20
490
22
15
11
3
491 mobile apps for financial institutions

7. Ramping up Intuit’s innovation engine
Pain: Takes days per build to test
combinations of device/OS configurations
physically & manually, one by one …
QA can’t keep up with the changes
Goal: Reduce development iteration cycle
from days to minutes on an ever-growing
number of devices
Strategy: Speed & quality through mobile
continuous integration
• iPhone 4
• iPhone 4S
• iPhone 5
• iPad 2
• iPad 3
• iPad Mini
Devices OS versions
• iOS 5.0
• iOS 5.1
• iOS 6.0
• iOS 6.1.x
• iOS 7.0
Builds
X X
Daily
builds
• Apple
• Google
• Amazon
Platforms
X

8. Continuous integration (CI)
What is CI?
§ Automate build, test & deployment process
§ Integrate & build early, often, daily, for every check-in
§ Best practice of Agile development
§ Jenkins is widely adopted, flexible & extensible
Benefits of CI in a mobile world
§ Instant feedback on quality, functionality & system impact
§ Less complex, no integration hell
§ Detect and address device configuration issues early
§ Involve stakeholders early
§ Ultimately achieve shippable release any time

9. Speed and quality
through mobile CI
Key enablers:
• Mobile device lab
• Test automation
Test processorReal
devices
Jenkins
SCM:
source
repository
Developers/
IDE
Automatic
test run
Test results
Submit the app to
app store if passed
Check in
Check-in
triggers
automated CI
Install the test app to
real devices in parallel
Run tests on real
devices in parallel
Analyze
test results
Trigger testing in
mobile device lab
Process starts here

10. 3rd-party vendor solutions didn’t work for us
§ Couldn’t handle E2E automation of testing pre-release apps
- Dependency on internal resources & infrastructure
§ Vendor lock-in: customized CI & test automation solutions
- No flexibility to use our existing tools & frameworks
§ Disadvantages of vendors’ shared devices
- Security concern on testing on shared devices
- No access to corporate network
- Cost-prohibitive to reserve devices
§ Private cloud (on-site setup) with local devices not practical
- With sensitive customer data, we need private cloud
- Extensive use of devices for daily testing of hundreds of mobile apps is expensive

11. Why test on real devices?
Most mobile features can be tested on simulators, but
simulators have limitations:
§ Device capabilities (GPS, camera, accelerometer, gyros),
form factors & pixel density
§ Real-world experience over the real connection
(3G, 4G, crappy Wi-Fi, etc.)
§ True performance, security tests
§ Testing on non-default Web browsers (e.g., Opera Mini)
§ Not all simulator/emulators have good quality
(don’t reflect real rendering/behavior, bugs)
§ Testing SMS & app integration

12. Intuit Virtual Device Lab

13. Intuit Virtual Device Lab (VDL)
Gives access to real mobile devices using a browser
Intranet
VDL
User machine
User machine
User machine
3-click access
Always on
Integrated with
dev infrastructure
Globally available
iPhone
iPad
Galaxy
Captivate
Optimus Nexus One
Desire
Incredible
Manufacturers/
devices:
Droid
Milestone
Moto

14. VDL architecture – Android
Windows PC
Tomcat
Java image
transferring
application
Browser
Java applet
Internet/
intranet
Device
interaction app
Developer’s laptop
VDL
Image transfer
USB connection

15. VDL architecture – iOS
Mac with Xcode
installed
Tomcat
Java VNC
client
Port forwarding
Browser
Java applet
Internet/
intranet
VNC server
USB connection
Developer’s laptop
VDL

16. Open source tools technologies used in VDL
Key capabilities Android iOS
Installing mobile applications into
the real device
adb install APK file Fruitstrap
https://github.com/ghughes/
fruitstrap
Transferring device screen images,
controlling device
Screencast tool
http://code.google.com/p/
androidscreencast
Veency VNC app
Fetching device local files adb pull filename sftp command
(jailbroken devices only)

17. VDL key features functionality today
Control the device through standard browser/
RESTful APIs
Change orientation to landscape or portrait
Copy local file to the device from your PC
Get file from the device onto your local PC
Execute an ADB command on the device
See current running logs in your browser
Download history logs to your local PC

18. Continuous integration – how it works
Product builds
Trigger file launches test
VDL device
inventoryResource
manager
Select a device
Test execution
engine
Results
Monitor
for new
builds

19. Mobile continuous integration
Leveraging the Virtual Device Lab for test automation on multiple, real devices simultaneously.
Submit, build, test … in minutes vs. hours
Select devices tests1 Tests run automatically across selected
devices with each build
2 See the results!3

20. Next step: over-the-air replaces USB connection
Fast onboarding
Scalable
Any device
Internet/
intranet
App
installation
Test
execution
Results
retrieval
VDL

21. Mobile Test Automation

22. Mobile test automation tools
Instrumentation-based
Non-instrumentation
AndroidiOS
MonkeyRunner
MOET
Sikuli
Android SDK
Robotium
TestDroid (BitBar)
Calabash
(LessPainful)
Soasta
iOS SDK
UIAutomation
UISpec
Frank
MonkeyTalk
DA/
PerfectoMobile
eggPlant
Cross-platform solution
iOS Driver Appium
• Assessed as of 2012

23. Mobile test automation trends
§ Abstraction with domain-specific language (DSL)
- Common test script over different devices (e.g., Cucumber, MonkeyScript, MOET)
- Support behavior-driven development (BDD)
§ Client/server-based approach (remotely driven tests via HTTP)
§ Instrumentation non-instrumentation solutions complement
each other
§ Mobile cloud testing environment
- From hardware (cradle) to software approach to drive tests on
real devices
§ Automated OTA app installation tests

24. Intuit automation framework of choice/recommendation
§ Native apps:
- Cucumber + Calabash for BDD on iOS Android
§ Mobile Web:
- Selenium/WebDriver

25. What is Cucumber?
It lets stakeholders (customers, QE or developers) describe how
software should behave in plain text.
§ BDD: Written in business domain-specific language (DSL)
- Good abstraction over different device types
§ One format for multiple purposes
- UI feature specification
- Automated tests
§ Cucumber itself is NOT a test automation tool
- Add-on to any test automation framework
- Examples: Calabash, Sikuli Frank mobile test frameworks

26. What is Calabash?
§ Open source cross-platform mobile test framework
- iOS + Android, simulator + real device, native + hybrid
§ Supports Cucumber for BDD
§ Application needs instrumentation that embeds a Calabash HTTP server
- Tests run from a remote client (in a Jenkins slave)
- Tests can be driven via Wi-Fi (no USB wire)
§ Ruby clients in API level (Java clients in experiment)
§ Works better in touch simulation real device testing
§ Mobile cloud test execution available by LessPainful

27. Intuit shared test library – rapid test development
Internal open source repository
Teams focus on creating
Use shared test lib
… and then contribute to
common test library
Tests
Contribute
Tests
Common test library Product tests
TestsTests
Globalization test framework
(I18N/L10N)
Performance/load test framework
Security test framework
Native/hybrid test framework
High-level test domain-specific language
(Cucumber steps)
Mobile
back-end
Mobile
front-end

28. Code
build
Deploy
Test
Results
Connect
device
Notify
email
Get
devices
Consolidate
report
Upload
report
From: manual
50*5
= 250
15 30
5
10
20
15
30 10
= 6 hours
Code
build
Trigger
Execute tests
in parallel
Consolidated
report in archive
Deploy on
devices
0 5 10 0
= 15 mins
To: automated CI
TIME SAVED: 95%Release quality increased
*
Stunning results: reduced iteration cycle from hours to minutes
Tested on 5 devices

29. VDL cost saving over vendors’ private clouds (today)
Annual
cost ($)
# of devices
Intuit Virtual Device Lab
150
3rd-party vendor 1
local device in cradle
Assumptions: 50 native
apps daily check-in
Note: As vendor pricing
changes, so do the savings
Savings
25
3rd-party vendor 2
local device w/o cradle
(Software approach)
Savings
Private cloud setup behind corp firewall

30. Key takeaways
Building custom solution vs. 3rd-party vendor solution
§ Understand your technical/platform/global spread
§ Many vendor solutions provide a larger set of devices/OS versions
in the global market
§ Quite a few offer mobile test services – might be tightly coupled with their
device access solution automation framework
§ Connecting to your corporate network is a challenge – some provide
private cloud (on-site setup within your network) but it’s cost-prohibitive
Invest in automation and CI
§ Automate tests with robust automation framework
§ Enable rapid test development through shared test library
§ Adopt CI in mobile development for speed quality

31. More mobile innovations at Intuit
§ Mobile crash reporter
§ Mobile components design patterns
§ Mobile analytics SDK gateway
Visit booth #601 – Accelerate Development
§ Demos conversations
§ Contest
§ Mobile device charging station
§ Foosball
§ We’re hiring !
Win a weekend with a TESLA!

32. Thank you!!
Tina_Su@intuit.com