Slides for Virtual Device Lab

1. Speed to Market in Mobile
Development
Finding the right solution with continuous
integration on real devices

2. Agenda
 Mobile trends & developers’ challenges
 Speed & quality through continuous integration
- Intuit Virtual Device Lab
- Intuit mobile test automation
 Takeaways

3. Aggressive mobile momentum
Will surpass desktop
by 2014
Proliferation of platforms, OS
versions & device types
It should just work!
“2013 Internet Trends,” KPCB

4. •
Developer challenges
Immense configuration matrix
–
–
–
•
Application delivery through the markets
–
•
Network, different network bandwidth/latency, no-network
HTML5 offline app
Media challenges
–
–
•
App signing, provision (app & device), submission
Offline experience
–
–
•
OS platforms (Android, iOS, Windows, phone vs. tablet, …)
Inputs (touch/gesture, real buttons, voice, camera, GPS receiver)
Outputs (portrait/landscape, screen resolution, language)
Some codecs don’t work on a device
Streaming, sound
Security & privacy

5. Intuit mobile app portfolio today
22
Apple
15
Google
11
Amazon
3
SMS
491 mobile apps for financial institutions
0
5
10
15
20
490

6. Ramping up Intuit’s innovation engine
Pain: Takes days per build to test
combinations of device/OS configurations
physically & manually, one by one …
QA can’t keep up with the changes
Goal: Reduce development iteration cycle
from days to minutes on an ever-growing
number of devices
Strategy: Speed & quality through mobile
continuous integration
Platforms
Devices
X
• Apple
• Google
• Amazon
OS
versions
X
•
•
•
•
•
•
iPhone 4
iPhone 4S
iPhone 5
iPad 2
iPad 3
iPad Mini
Builds
X
•
•
•
•
•
iOS 5.0
iOS 5.1
iOS 6.0
iOS 6.1.x
iOS 7.0
Daily
builds

7. What is CI?
•
•
•
•
Continuous integration (CI)
Automate build, test & deployment process
Integrate & build early, often, daily, for every check-in
Best practice of Agile development
Jenkins is widely adopted, flexible & extensible
Benefits of CI in a mobile world
• Instant feedback on quality, functionality & system
impact
• Less complex, no integration hell
• Detect and address device configuration issues early
• Involve stakeholders early
• Ultimately achieve shippable release any time

8. Speed and quality
Developers/
IDE
Check in
Test results
through mobile CI
Process starts here
SCM: source
repository
Submit the app to
app store if passed
Analyze
test results
Key enablers:
Check-in
triggers
automated CI
• Mobile device lab
• Test automation
Automatic
test run
Run tests on real
devices in parallel
Jenkins
Trigger testing in
mobile device lab
Real
devices
Test processor
Install the test app to
real devices in parallel

9. rd-party
3
vendor solutions didn’t work for us
• Couldn’t handle E2E automation of testing pre-release apps
– Dependency on internal resources & infrastructure
• Vendor lock-in: customized CI & test automation solutions
– No flexibility to use our existing tools & frameworks
• Disadvantages of vendors’ shared devices
– Security concern on testing on shared devices
– No access to corporate network
– Cost-prohibitive to reserve devices
• Private cloud (on-site setup) with local devices not practical
– With sensitive customer data, we need private cloud
– Extensive use of devices for daily testing of hundreds of mobile apps is expensive

10. Why test on real devices?
Most mobile features can be tested on simulators,
but simulators have limitations:
• Device capabilities (GPS, camera, accelerometer,
gyros), form factors & pixel density
• Real-world experience over the real connection
(3G, 4G, crappy Wi-Fi, etc.)
• True performance, security tests
• Testing on non-default Web browsers (e.g.,
Opera Mini)
• Not all simulator/emulators have good quality
(don’t reflect real rendering/behavior, bugs)
• Testing SMS & app integration

11. Intuit Virtual Device Lab

12. Intuit Virtual Device Lab (VDL)
Gives access to real mobile devices using a browser
3-click access
User machine
Always on
Intranet
User machine
User machine
VDL
Manufacturers/
devices:
iPhone
iPad
Galaxy
Captivate
Droid
Milestone
Moto
Optimus
Globally available
Integrated with
dev infrastructure
Nexus
One
Desire
Incredible

13. VDL architecture – Android
Internet/intr
anet
Device
interaction app
Windows PC
Tomcat
USB connection
Image
transfer
Java image
transferring
application
Browser &
Java applet
Developer’s laptop
VDL

14. VDL architecture – iOS
Internet/intr
anet
Mac with Xcode
installed
VNC server
Tomcat
USB connection
Port
forwarding
Java VNC
client
Browser &
Java applet
Developer’s laptop
VDL

15. Open source tools & technologies used in VDL
Key capabilities
Android
Installing mobile applications into the adb install <APK file>
real device
iOS
Fruitstrap
https://github.com/ghughes/fruitstrap
Transferring device screen images,
controlling device
Screencast tool
Veency VNC app
http://code.google.com/p/androidscre
encast
Fetching device local files
adb pull <filename>
sftp command
(jailbroken devices only)

16. VDL key features & functionality today
Control the device through standard browser/RESTful
APIs
Change orientation to landscape or portrait
Copy local file to the device from your PC
Get file from the device onto your local PC
Execute an ADB command on the device
See current running logs in your browser
Download history logs to your local PC

17. Continuous integration – how it works
Trigger file launches test
Monito
r for
new
builds
Resource
manager
Test execution
engine
Product builds
Results
Select a device
VDL device
inventory

18. Mobile continuous integration
Leveraging the Virtual Device Lab for test automation on multiple, real devices simultaneously.
1 Select devices & tests
2 Tests run automatically across selected devices
with each build
Submit, build, test … in minutes vs. hours
3 See the results!

19. Next step: over-the-air replaces USB connection
VDL
Fast onboarding
App
installation
Scalable
Test
execution
Results
retrieval
Internet/intr
anet
Any device

20. Mobile Test Automation

21. Mobile test automation tools
iOS
Instrumentation-based
Android
MonkeyTalk
Frank
Calabash
(LessPainful)
TestDroid (BitBar)
UISpec
Robotium
UIAutomation
iOS SDK
Soasta
iOS Driver
Appium
Non-instrumentation
MOET
Sikuli
Cross-platform solution
Android SDK
MonkeyRunner
DA/PerfectoMob
ile
eggPlant
• Assessed as of 2012

22. Mobile test automation trends
• Abstraction with domain-specific language (DSL)
–
–
Common test script over different devices (e.g., Cucumber, MonkeyScript, MOET)
Support behavior-driven development (BDD)
• Client/server-based approach (remotely driven tests via HTTP)
• Instrumentation & non-instrumentation solutions complement
each other
• Mobile cloud testing environment
–
From hardware (cradle) to software approach to drive tests on
real devices
• Automated OTA app installation & tests

23. Intuit automation framework of
choice/recommendation
• Native apps:
– Cucumber + Calabash for BDD on iOS &
Android
• Mobile Web:
– Selenium/WebDriver

24. What is Cucumber?
It lets stakeholders (customers, QE or developers)
describe how software should behave in plain
text.
•
BDD: Written in business domain-specific language (DSL)
– Good abstraction over different device types
•
One format for multiple purposes
– UI feature specification
– Automated tests
•
Cucumber itself is NOT a test automation tool
– Add-on to any test automation framework
– Examples: Calabash, Sikuli & Frank mobile test
frameworks

25. What is Calabash?
• Open source cross-platform mobile test framework
– iOS + Android, simulator + real device, native + hybrid
• Supports Cucumber for BDD
• Application needs instrumentation that embeds a Calabash HTTP server
– Tests run from a remote client (in a Jenkins slave)
– Tests can be driven via Wi-Fi (no USB wire)
• Ruby clients in API level (Java clients in experiment)
• Works great in touch simulation & real device testing

26. Intuit shared test library – rapid test development
Common test library
High-level test domain-specific language
(Cucumber steps)
Mobile
front-end
Product tests
Teams focus on
creating
Native/hybrid test framework
Use shared test lib
Tests
Tests
Tests
Tests
Globalization test framework
(I18N/L10N)
Security test framework
Mobile
back-end
Performance/load test framework
Internal open source repository
Contribute
… and then contribute to
common test library

27. Stunning results: reduced iteration cycle from hours to minutes
From: manual
10
5
Code
build
Notify
email
Consolidate
report
* 50*5
= 250
Upload
report
30
Get
devices
10
= 6 hours
20
15
30
15
To: automated CI
Code
build
Trigger
Deploy on
devices
Execute tests in
parallel
Consolidated
report in archive
0
5
10
0
Tested on 5 devices
Release quality increased
= 15 mins
TIME SAVED: 95%

28. VDL cost saving over vendors’ private clouds (today)
Assumptions: 50 native apps &
Annual
cost ($)
daily check-in
3rd-party vendor 1
local device in cradle
3rd-party vendor 2
local device w/o cradle
(Software approach)
Savings
Savings
Intuit Virtual Device Lab
25
150
Private cloud setup behind corp firewall
# of devices
Note: As vendor pricing changes,
so do the savings

29. Key takeaways
Building custom solution vs.
rd-party
3
vendor solution
 Understand your technical/platform/global spread
 Many vendor solutions provide a larger set of devices/OS versions
in the global market
 Quite a few offer mobile test services – might be tightly coupled with their
device access solution & automation framework
 Connecting to your corporate network is a challenge – some provide private
cloud (on-site setup within your network) but it’s cost-prohibitive
Invest in automation and CI
 Automate tests with robust automation framework
 Enable rapid test development through shared test library
 Adopt CI in mobile development for speed & quality

30. Thank you!!