Best Practices for DevOps in Mobile App Testing

1,849 views

Published on

Watch a live presentation at http://offer.bitbar.com/best-practices-for-devops-in-mobile-app-testing

In essence, the core of DevOps methodology aims to speed up the app development delivery and process by getting devs and operation specialists to collaborate throughout the end-to-end app development and deployment process.

Stay tuned and join our upcoming webinars at http://bitbar.com/testing/webinars/

Published in: Technology

Best Practices for DevOps in Mobile App Testing

  1. 1. 3 March 2016 – 12pm GMT Ville-Veikko Helppi Head of Demand Generation ville-veikko.helppi@bitbar.com Best Practices for DevOps in Mobile App Testing W E B I N A R
  2. 2. Agenda • Deconstructing ‘Mobile DevOps’ • Mobile Product Development Life-Cycle & Used Tools • The DevOps Approach in Mobile App Development & Testing • Best Practices for DevOps in Mobile App Testing • Q&A http://testdroid.com
  3. 3. DECONSTRUCTING “MOBILE DEVOPS” http://testdroid.com
  4. 4. Mobile DevOps DevOps = Practices that emphasize the collaboration and communication of both software developers and other information-technology (IT) professionals while automating the process of software delivery and infrastructure changes. --Wikipedia • Development focus on building the actual product and using relevant tools • QA doing continuous testing on robustness, functionality, performance, and preferably use test automation • Operations maintain the continuous build/integration/deployment/ delivery environments, and do releases
  5. 5. 3 Things in Mobile DevOps • 3 DevOps Principles 1. Develop & Test Against Real Environment 2. Deploy Frequently 3. Continuously Validate Quality Characteristics • 3 Business-Critical DevOps Categories 1. Continuous Integration & Continuous Delivery 2. App and Infrastructure (e.g. Back-End) Testing & Monitoring 3. Entire Dev&Test Process and Mobile App Delivery http://testdroid.com
  6. 6. DevOps Strategy & Typical Goals • Maximize the Delivered Value • Customers, Internally between BUs and Teams • Maximize the Efficiency and Productivity • Improved Process, The ‘Right’ Tools & Methods, Faster T2M • Maximize the Quality and Robustness of Apps • Better Quality with Equal Effort, Investments in Quality • Maximize the Support, Service & The Delivery • Alignment and Collaboration between Teams The highest priority is to satisfy the customer through early and continuous delivery of valuable software. Agile Manifesto – Principle #1 “ “http://testdroid.com
  7. 7. How DevOps Can Make A Difference • Based on Testdroid Customer Survey 2015 • Things That Go Wrong with App Testing 1. Too Long Dev-Test-Feedback Cycles 2. Bug Catching/Reporting Too Slow to Developers 3. Non-Functional or Not Well Utilized Test Automation 4. Use of Emulators - Setting Up & Maintaining Real Devices 5. Access to Relevant Mobile Platform / OS version / Devices 6. Use of Manual Testing – The Weakest Link in Agile and Devops Process 7. No Geographically Relevant Testbed – Devices, Networks, Back-End Integration http://testdroid.com
  8. 8. Where DevOps Make A Difference • Devices / Hardware - Handling of new devices and other new HW used in the infrastructure • Infrastructure - Ready-to-Go infrastructure means no delays for development, testing and result-driven approach speeds things up & saves time and money • Support - SW devs, SW devs in test (SDET), QA engineers – all need different type of support when using the infrastructure and DevOps approach can help a lot • Operations - Maintaining and Upgrading infrastructure, devices, networks, additional SW • Tools Integration - Support for multiple test tools gives the users/developers freedom of choice. No vendor, tech, tools lock-ins! http://testdroid.com
  9. 9. MOBILE DEVOPS & DEVELOPMENT - LIFECYCLE & TOOLS http://testdroid.com
  10. 10. DevOps Challenges * • Building the DevOps culture across organization • Overcoming the Dev vs. Ops mentality • Moving from legacy (and sometimes on-premise setup) to cloud based tools and environment • Broader scope than before: SW and HW for mobile dev • Learning new skills – DevOps practice requires change and collaboration across teams • Mobile Challenges: • Fragmentation, Tools and Platforms, Not Enough Real Devices • Less Worrying Challenges on Mobile: • Cross-platform, Using Cloud Efficiently * = Based on Testdroid Customer Survey 2015http://testdroid.com
  11. 11. DevOps ‘Toolchain’ • Code – Code Development and Code Reviews, Continuous Integration tools • Build – Version control tools, SCM, code merging, Build status • Test – Test and results determine performance • Package – Artifact repository, Application pre-deployment staging • Release – Change management, Release approvals, release automation • Configure – Infrastructure configuration and management, Infrastructure as Code tools • Monitor – App perf. monitoring, UX
  12. 12. DevOps ‘Toolchain’ Code & Build Continuous Integration, Continuous Development Package, Release, Configure & Monitoring Continuous Delivery, Continuous Deployment/Release GOAL: End-to-End Transparency, Traceability, Seamless Workflow, Continuous Everything! 2 Valid Options for Mobile Testing: Cloud or On- Premise http://testdroid.com
  13. 13. DevOps – Testing in the ‘Cloud’ 1 2 3
  14. 14. DevOps – Testing On-Premise CONTINUOUS INTEGRATION http://testdroid.com
  15. 15. Benefits of Test Automation • App Compatibility & Global coverage • Reusability • Speed & Efficiency • Test efficiency • Test effectiveness • Time-to-market http://testdroid.com
  16. 16. DevOps Must Rely on Automation • DevOps is more than just a tool or a process change; it inherently requires an organizational culture shift • Agile methodologies back up this thinking • Increase focus on test automation and CI methods • Best practices and continuous iteration Time Resources Infrastructure Training Tools http://testdroid.com
  17. 17. THE DEVOPS APPROACH IN MOBILE APP TESTING http://testdroid.com
  18. 18. Testing Philosophy http://testdroid.com
  19. 19. Manual Automation Testing Philosophy http://testdroid.com
  20. 20. Smaller coverage, More money burnt & time wasted, Error-prone Manual Automation Large coverage, quickly completed, Less money & time wasted, Exact results. Testing Philosophy
  21. 21. Top Requirement for Testing • Real Devices and Real Browsers, as emulators cannot help you to test... • User Experience • Usability • Hardware • Software • Infrastructure 0 % = the percentage of your app users that use emulator to run your app! http://testdroid.com
  22. 22. Devices & Daily Tasks for DevOps • In order to maintain mobile device farm, environment and all aspects of SW & HW the following daily check-ups must be done: • Are all devices connected and operating? • Are all devices WiFi connected? • Do all devices have an active USB connection? • Are battery levels adequate (at least 50%) for every device? • Are there any pop-ups, system notification or other system dialogs open on the screen? • Are screens unlocked? http://testdroid.com
  23. 23. Troubleshooting Ain’t The Same http://testdroid.com
  24. 24. Automation vs Manual http://testdroid.com
  25. 25. Test Breakdown FAILED FAILED http://testdroid.com
  26. 26. Mobile Test Automation Automatic test exercisersRecord and PlaybackHand written test scripts Benefits: Accurate, specific to your testing needs, plenty of options with frameworks, tools Fast to create, accurate, not as sensitive to human-errors as hand- written tests, tools avail’ty Fastest & extremely automated, excellent for smoke testing/quick testing, availability Tradeoffs: Takes a lot of time, ties resources to write test cases/scripts, error- prone (humans) Compelling Recorder + Playback tools available for only few test automation frameworks Not accurate as real test cases http://testdroid.com
  27. 27. Why Should DevOps Be Interested in … Robotium uiautomator Espresso Appium Calabash Android Yes Yes Yes Yes Yes iOS No No No Yes Yes Mobile web Yes (Android) Limited to x.y clicks No Yes (Android & iOS) Yes (Android) Scripting Language Java Java Java Almost any Ruby Test creation tools Testdroid Recorder UI Automator viewer Hierarchy Viewer Appium.app CLI Supported API levels All 16 => 8, 10, 15- All All Community Contributors Google Google Active Pretty quiet http://testdroid.com
  28. 28. BEST PRACTICES FOR DEVOPS IN MOBILE APP TESTING http://testdroid.com
  29. 29. Automate (Everything) • When it comes to mobile app testing, the more you can automate repetitive things, the better • Not always possible but leave the last ‘exploratory testing’ for the final phase • Smoke, Unit, All sorts of Performance, Regression testing should be automated • Run your test patterns against EVERY build • Start small but make sure you can scale-up http://testdroid.com
  30. 30. Ensure Workflow is Seamless • Only use tools, frameworks and software that is fully compatible across used toolchain/workflow • Ensure all processes, results, tools are entirely transparent and available for use across teams • This has a significant impact on productivity and efficiency • Open source is always highly recommended http://testdroid.com
  31. 31. Rely Only On Real & Quality HW • Mobile device farms require lots of hardware – this is the correct place to invest in quality HW • Make sure the mobile device coverage is adequate – globally • Execute the upgrade and support policies relentlessly http://testdroid.com
  32. 32. “The Right” Testing Framework • Variety of choices when it comes mobile test automation frameworks, ask the right questions: • App, Game, Web? Native, Hybrid? • Mobile Back-End Integrated? • Who needs to write, read and interpret tests? • Does test run results provide you what you organization needs / is looking for? http://testdroid.com
  33. 33. THANK YOU! More information about frameworks, devices and testing at www.testdroid.com http://testdroid.com

×