Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Best Practices for DevOps in Mobile App Testing

3,247 views

Published on

Watch a live presentation at http://offer.bitbar.com/best-practices-for-devops-in-mobile-app-testing

In essence, the core of DevOps methodology aims to speed up the app development delivery and process by getting devs and operation specialists to collaborate throughout the end-to-end app development and deployment process.

Stay tuned and join our upcoming webinars at http://bitbar.com/testing/webinars/

Published in: Technology

Best Practices for DevOps in Mobile App Testing

  1. 1. 3 March 2016 – 12pm GMT Ville-Veikko Helppi Head of Demand Generation ville-veikko.helppi@bitbar.com Best Practices for DevOps in Mobile App Testing W E B I N A R
  2. 2. Agenda • Deconstructing ‘Mobile DevOps’ • Mobile Product Development Life-Cycle & Used Tools • The DevOps Approach in Mobile App Development & Testing • Best Practices for DevOps in Mobile App Testing • Q&A http://testdroid.com
  3. 3. DECONSTRUCTING “MOBILE DEVOPS” http://testdroid.com
  4. 4. Mobile DevOps DevOps = Practices that emphasize the collaboration and communication of both software developers and other information-technology (IT) professionals while automating the process of software delivery and infrastructure changes. --Wikipedia • Development focus on building the actual product and using relevant tools • QA doing continuous testing on robustness, functionality, performance, and preferably use test automation • Operations maintain the continuous build/integration/deployment/ delivery environments, and do releases
  5. 5. 3 Things in Mobile DevOps • 3 DevOps Principles 1. Develop & Test Against Real Environment 2. Deploy Frequently 3. Continuously Validate Quality Characteristics • 3 Business-Critical DevOps Categories 1. Continuous Integration & Continuous Delivery 2. App and Infrastructure (e.g. Back-End) Testing & Monitoring 3. Entire Dev&Test Process and Mobile App Delivery http://testdroid.com
  6. 6. DevOps Strategy & Typical Goals • Maximize the Delivered Value • Customers, Internally between BUs and Teams • Maximize the Efficiency and Productivity • Improved Process, The ‘Right’ Tools & Methods, Faster T2M • Maximize the Quality and Robustness of Apps • Better Quality with Equal Effort, Investments in Quality • Maximize the Support, Service & The Delivery • Alignment and Collaboration between Teams The highest priority is to satisfy the customer through early and continuous delivery of valuable software. Agile Manifesto – Principle #1 “ “http://testdroid.com
  7. 7. How DevOps Can Make A Difference • Based on Testdroid Customer Survey 2015 • Things That Go Wrong with App Testing 1. Too Long Dev-Test-Feedback Cycles 2. Bug Catching/Reporting Too Slow to Developers 3. Non-Functional or Not Well Utilized Test Automation 4. Use of Emulators - Setting Up & Maintaining Real Devices 5. Access to Relevant Mobile Platform / OS version / Devices 6. Use of Manual Testing – The Weakest Link in Agile and Devops Process 7. No Geographically Relevant Testbed – Devices, Networks, Back-End Integration http://testdroid.com
  8. 8. Where DevOps Make A Difference • Devices / Hardware - Handling of new devices and other new HW used in the infrastructure • Infrastructure - Ready-to-Go infrastructure means no delays for development, testing and result-driven approach speeds things up & saves time and money • Support - SW devs, SW devs in test (SDET), QA engineers – all need different type of support when using the infrastructure and DevOps approach can help a lot • Operations - Maintaining and Upgrading infrastructure, devices, networks, additional SW • Tools Integration - Support for multiple test tools gives the users/developers freedom of choice. No vendor, tech, tools lock-ins! http://testdroid.com
  9. 9. MOBILE DEVOPS & DEVELOPMENT - LIFECYCLE & TOOLS http://testdroid.com
  10. 10. DevOps Challenges * • Building the DevOps culture across organization • Overcoming the Dev vs. Ops mentality • Moving from legacy (and sometimes on-premise setup) to cloud based tools and environment • Broader scope than before: SW and HW for mobile dev • Learning new skills – DevOps practice requires change and collaboration across teams • Mobile Challenges: • Fragmentation, Tools and Platforms, Not Enough Real Devices • Less Worrying Challenges on Mobile: • Cross-platform, Using Cloud Efficiently * = Based on Testdroid Customer Survey 2015http://testdroid.com
  11. 11. DevOps ‘Toolchain’ • Code – Code Development and Code Reviews, Continuous Integration tools • Build – Version control tools, SCM, code merging, Build status • Test – Test and results determine performance • Package – Artifact repository, Application pre-deployment staging • Release – Change management, Release approvals, release automation • Configure – Infrastructure configuration and management, Infrastructure as Code tools • Monitor – App perf. monitoring, UX
  12. 12. DevOps ‘Toolchain’ Code & Build Continuous Integration, Continuous Development Package, Release, Configure & Monitoring Continuous Delivery, Continuous Deployment/Release GOAL: End-to-End Transparency, Traceability, Seamless Workflow, Continuous Everything! 2 Valid Options for Mobile Testing: Cloud or On- Premise http://testdroid.com
  13. 13. DevOps – Testing in the ‘Cloud’ 1 2 3
  14. 14. DevOps – Testing On-Premise CONTINUOUS INTEGRATION http://testdroid.com
  15. 15. Benefits of Test Automation • App Compatibility & Global coverage • Reusability • Speed & Efficiency • Test efficiency • Test effectiveness • Time-to-market http://testdroid.com
  16. 16. DevOps Must Rely on Automation • DevOps is more than just a tool or a process change; it inherently requires an organizational culture shift • Agile methodologies back up this thinking • Increase focus on test automation and CI methods • Best practices and continuous iteration Time Resources Infrastructure Training Tools http://testdroid.com
  17. 17. THE DEVOPS APPROACH IN MOBILE APP TESTING http://testdroid.com
  18. 18. Testing Philosophy http://testdroid.com
  19. 19. Manual Automation Testing Philosophy http://testdroid.com
  20. 20. Smaller coverage, More money burnt & time wasted, Error-prone Manual Automation Large coverage, quickly completed, Less money & time wasted, Exact results. Testing Philosophy
  21. 21. Top Requirement for Testing • Real Devices and Real Browsers, as emulators cannot help you to test... • User Experience • Usability • Hardware • Software • Infrastructure 0 % = the percentage of your app users that use emulator to run your app! http://testdroid.com
  22. 22. Devices & Daily Tasks for DevOps • In order to maintain mobile device farm, environment and all aspects of SW & HW the following daily check-ups must be done: • Are all devices connected and operating? • Are all devices WiFi connected? • Do all devices have an active USB connection? • Are battery levels adequate (at least 50%) for every device? • Are there any pop-ups, system notification or other system dialogs open on the screen? • Are screens unlocked? http://testdroid.com
  23. 23. Troubleshooting Ain’t The Same http://testdroid.com
  24. 24. Automation vs Manual http://testdroid.com
  25. 25. Test Breakdown FAILED FAILED http://testdroid.com
  26. 26. Mobile Test Automation Automatic test exercisersRecord and PlaybackHand written test scripts Benefits: Accurate, specific to your testing needs, plenty of options with frameworks, tools Fast to create, accurate, not as sensitive to human-errors as hand- written tests, tools avail’ty Fastest & extremely automated, excellent for smoke testing/quick testing, availability Tradeoffs: Takes a lot of time, ties resources to write test cases/scripts, error- prone (humans) Compelling Recorder + Playback tools available for only few test automation frameworks Not accurate as real test cases http://testdroid.com
  27. 27. Why Should DevOps Be Interested in … Robotium uiautomator Espresso Appium Calabash Android Yes Yes Yes Yes Yes iOS No No No Yes Yes Mobile web Yes (Android) Limited to x.y clicks No Yes (Android & iOS) Yes (Android) Scripting Language Java Java Java Almost any Ruby Test creation tools Testdroid Recorder UI Automator viewer Hierarchy Viewer Appium.app CLI Supported API levels All 16 => 8, 10, 15- All All Community Contributors Google Google Active Pretty quiet http://testdroid.com
  28. 28. BEST PRACTICES FOR DEVOPS IN MOBILE APP TESTING http://testdroid.com
  29. 29. Automate (Everything) • When it comes to mobile app testing, the more you can automate repetitive things, the better • Not always possible but leave the last ‘exploratory testing’ for the final phase • Smoke, Unit, All sorts of Performance, Regression testing should be automated • Run your test patterns against EVERY build • Start small but make sure you can scale-up http://testdroid.com
  30. 30. Ensure Workflow is Seamless • Only use tools, frameworks and software that is fully compatible across used toolchain/workflow • Ensure all processes, results, tools are entirely transparent and available for use across teams • This has a significant impact on productivity and efficiency • Open source is always highly recommended http://testdroid.com
  31. 31. Rely Only On Real & Quality HW • Mobile device farms require lots of hardware – this is the correct place to invest in quality HW • Make sure the mobile device coverage is adequate – globally • Execute the upgrade and support policies relentlessly http://testdroid.com
  32. 32. “The Right” Testing Framework • Variety of choices when it comes mobile test automation frameworks, ask the right questions: • App, Game, Web? Native, Hybrid? • Mobile Back-End Integrated? • Who needs to write, read and interpret tests? • Does test run results provide you what you organization needs / is looking for? http://testdroid.com
  33. 33. THANK YOU! More information about frameworks, devices and testing at www.testdroid.com http://testdroid.com

×