Your SlideShare is downloading. ×
0
Droid con slides 2013  mobileci-v1.0
Droid con slides 2013  mobileci-v1.0
Droid con slides 2013  mobileci-v1.0
Droid con slides 2013  mobileci-v1.0
Droid con slides 2013  mobileci-v1.0
Droid con slides 2013  mobileci-v1.0
Droid con slides 2013  mobileci-v1.0
Droid con slides 2013  mobileci-v1.0
Droid con slides 2013  mobileci-v1.0
Droid con slides 2013  mobileci-v1.0
Droid con slides 2013  mobileci-v1.0
Droid con slides 2013  mobileci-v1.0
Droid con slides 2013  mobileci-v1.0
Droid con slides 2013  mobileci-v1.0
Droid con slides 2013  mobileci-v1.0
Droid con slides 2013  mobileci-v1.0
Droid con slides 2013  mobileci-v1.0
Droid con slides 2013  mobileci-v1.0
Droid con slides 2013  mobileci-v1.0
Droid con slides 2013  mobileci-v1.0
Droid con slides 2013  mobileci-v1.0
Droid con slides 2013  mobileci-v1.0
Droid con slides 2013  mobileci-v1.0
Droid con slides 2013  mobileci-v1.0
Droid con slides 2013  mobileci-v1.0
Droid con slides 2013  mobileci-v1.0
Droid con slides 2013  mobileci-v1.0
Droid con slides 2013  mobileci-v1.0
Droid con slides 2013  mobileci-v1.0
Droid con slides 2013  mobileci-v1.0
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Droid con slides 2013 mobileci-v1.0

487

Published on

Slides for Virtual Device Lab

Slides for Virtual Device Lab

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
487
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
14
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Need to scale to large # of mobile apps to support
  • Device access, app installation, tedious testingacross large configuration matrix – turnaround time isdays to weeks
  • Talk Points: For all these reasons, we built our own solution, mobile continuous integration with VDL.- Briefly talk about overall mobile CI to show the big picture first.
  • Talk Points: benefits - Saving in QE effort
  • Transcript

    • 1. Speed to Market in Mobile Development Finding the right solution with continuous integration on real devices
    • 2. Agenda  Mobile trends & developers’ challenges  Speed & quality through continuous integration - Intuit Virtual Device Lab - Intuit mobile test automation  Takeaways
    • 3. Aggressive mobile momentum Will surpass desktop by 2014 Proliferation of platforms, OS versions & device types It should just work! “2013 Internet Trends,” KPCB
    • 4. • Developer challenges Immense configuration matrix – – – • Application delivery through the markets – • Network, different network bandwidth/latency, no-network HTML5 offline app Media challenges – – • App signing, provision (app & device), submission Offline experience – – • OS platforms (Android, iOS, Windows, phone vs. tablet, …) Inputs (touch/gesture, real buttons, voice, camera, GPS receiver) Outputs (portrait/landscape, screen resolution, language) Some codecs don’t work on a device Streaming, sound Security & privacy
    • 5. Intuit mobile app portfolio today 22 Apple 15 Google 11 Amazon 3 SMS 491 mobile apps for financial institutions 0 5 10 15 20 490
    • 6. Ramping up Intuit’s innovation engine Pain: Takes days per build to test combinations of device/OS configurations physically & manually, one by one … QA can’t keep up with the changes Goal: Reduce development iteration cycle from days to minutes on an ever-growing number of devices Strategy: Speed & quality through mobile continuous integration Platforms Devices X • Apple • Google • Amazon OS versions X • • • • • • iPhone 4 iPhone 4S iPhone 5 iPad 2 iPad 3 iPad Mini Builds X • • • • • iOS 5.0 iOS 5.1 iOS 6.0 iOS 6.1.x iOS 7.0 Daily builds
    • 7. What is CI? • • • • Continuous integration (CI) Automate build, test & deployment process Integrate & build early, often, daily, for every check-in Best practice of Agile development Jenkins is widely adopted, flexible & extensible Benefits of CI in a mobile world • Instant feedback on quality, functionality & system impact • Less complex, no integration hell • Detect and address device configuration issues early • Involve stakeholders early • Ultimately achieve shippable release any time
    • 8. Speed and quality Developers/ IDE Check in Test results through mobile CI Process starts here SCM: source repository Submit the app to app store if passed Analyze test results Key enablers: Check-in triggers automated CI • Mobile device lab • Test automation Automatic test run Run tests on real devices in parallel Jenkins Trigger testing in mobile device lab Real devices Test processor Install the test app to real devices in parallel
    • 9. rd-party 3 vendor solutions didn’t work for us • Couldn’t handle E2E automation of testing pre-release apps – Dependency on internal resources & infrastructure • Vendor lock-in: customized CI & test automation solutions – No flexibility to use our existing tools & frameworks • Disadvantages of vendors’ shared devices – Security concern on testing on shared devices – No access to corporate network – Cost-prohibitive to reserve devices • Private cloud (on-site setup) with local devices not practical – With sensitive customer data, we need private cloud – Extensive use of devices for daily testing of hundreds of mobile apps is expensive
    • 10. Why test on real devices? Most mobile features can be tested on simulators, but simulators have limitations: • Device capabilities (GPS, camera, accelerometer, gyros), form factors & pixel density • Real-world experience over the real connection (3G, 4G, crappy Wi-Fi, etc.) • True performance, security tests • Testing on non-default Web browsers (e.g., Opera Mini) • Not all simulator/emulators have good quality (don’t reflect real rendering/behavior, bugs) • Testing SMS & app integration
    • 11. Intuit Virtual Device Lab
    • 12. Intuit Virtual Device Lab (VDL) Gives access to real mobile devices using a browser 3-click access User machine Always on Intranet User machine User machine VDL Manufacturers/ devices: iPhone iPad Galaxy Captivate Droid Milestone Moto Optimus Globally available Integrated with dev infrastructure Nexus One Desire Incredible
    • 13. VDL architecture – Android Internet/intr anet Device interaction app Windows PC Tomcat USB connection Image transfer Java image transferring application Browser & Java applet Developer’s laptop VDL
    • 14. VDL architecture – iOS Internet/intr anet Mac with Xcode installed VNC server Tomcat USB connection Port forwarding Java VNC client Browser & Java applet Developer’s laptop VDL
    • 15. Open source tools & technologies used in VDL Key capabilities Android Installing mobile applications into the adb install <APK file> real device iOS Fruitstrap https://github.com/ghughes/fruitstrap Transferring device screen images, controlling device Screencast tool Veency VNC app http://code.google.com/p/androidscre encast Fetching device local files adb pull <filename> sftp command (jailbroken devices only)
    • 16. VDL key features & functionality today Control the device through standard browser/RESTful APIs Change orientation to landscape or portrait Copy local file to the device from your PC Get file from the device onto your local PC Execute an ADB command on the device See current running logs in your browser Download history logs to your local PC
    • 17. Continuous integration – how it works Trigger file launches test Monito r for new builds Resource manager Test execution engine Product builds Results Select a device VDL device inventory
    • 18. Mobile continuous integration Leveraging the Virtual Device Lab for test automation on multiple, real devices simultaneously. 1 Select devices & tests 2 Tests run automatically across selected devices with each build Submit, build, test … in minutes vs. hours 3 See the results!
    • 19. Next step: over-the-air replaces USB connection VDL Fast onboarding App installation Scalable Test execution Results retrieval Internet/intr anet Any device
    • 20. Mobile Test Automation
    • 21. Mobile test automation tools iOS Instrumentation-based Android MonkeyTalk Frank Calabash (LessPainful) TestDroid (BitBar) UISpec Robotium UIAutomation iOS SDK Soasta iOS Driver Appium Non-instrumentation MOET Sikuli Cross-platform solution Android SDK MonkeyRunner DA/PerfectoMob ile eggPlant • Assessed as of 2012
    • 22. Mobile test automation trends • Abstraction with domain-specific language (DSL) – – Common test script over different devices (e.g., Cucumber, MonkeyScript, MOET) Support behavior-driven development (BDD) • Client/server-based approach (remotely driven tests via HTTP) • Instrumentation & non-instrumentation solutions complement each other • Mobile cloud testing environment – From hardware (cradle) to software approach to drive tests on real devices • Automated OTA app installation & tests
    • 23. Intuit automation framework of choice/recommendation • Native apps: – Cucumber + Calabash for BDD on iOS & Android • Mobile Web: – Selenium/WebDriver
    • 24. What is Cucumber? It lets stakeholders (customers, QE or developers) describe how software should behave in plain text. • BDD: Written in business domain-specific language (DSL) – Good abstraction over different device types • One format for multiple purposes – UI feature specification – Automated tests • Cucumber itself is NOT a test automation tool – Add-on to any test automation framework – Examples: Calabash, Sikuli & Frank mobile test frameworks
    • 25. What is Calabash? • Open source cross-platform mobile test framework – iOS + Android, simulator + real device, native + hybrid • Supports Cucumber for BDD • Application needs instrumentation that embeds a Calabash HTTP server – Tests run from a remote client (in a Jenkins slave) – Tests can be driven via Wi-Fi (no USB wire) • Ruby clients in API level (Java clients in experiment) • Works great in touch simulation & real device testing
    • 26. Intuit shared test library – rapid test development Common test library High-level test domain-specific language (Cucumber steps) Mobile front-end Product tests Teams focus on creating Native/hybrid test framework Use shared test lib Tests Tests Tests Tests Globalization test framework (I18N/L10N) Security test framework Mobile back-end Performance/load test framework Internal open source repository Contribute … and then contribute to common test library
    • 27. Stunning results: reduced iteration cycle from hours to minutes From: manual 10 5 Code build Notify email Consolidate report * 50*5 = 250 Upload report 30 Get devices 10 = 6 hours 20 15 30 15 To: automated CI Code build Trigger Deploy on devices Execute tests in parallel Consolidated report in archive 0 5 10 0 Tested on 5 devices Release quality increased = 15 mins TIME SAVED: 95%
    • 28. VDL cost saving over vendors’ private clouds (today) Assumptions: 50 native apps & Annual cost ($) daily check-in 3rd-party vendor 1 local device in cradle 3rd-party vendor 2 local device w/o cradle (Software approach) Savings Savings Intuit Virtual Device Lab 25 150 Private cloud setup behind corp firewall # of devices Note: As vendor pricing changes, so do the savings
    • 29. Key takeaways Building custom solution vs. rd-party 3 vendor solution  Understand your technical/platform/global spread  Many vendor solutions provide a larger set of devices/OS versions in the global market  Quite a few offer mobile test services – might be tightly coupled with their device access solution & automation framework  Connecting to your corporate network is a challenge – some provide private cloud (on-site setup within your network) but it’s cost-prohibitive Invest in automation and CI  Automate tests with robust automation framework  Enable rapid test development through shared test library  Adopt CI in mobile development for speed & quality
    • 30. Thank you!!

    ×