Mobile App Testing Best Practices
•Download as PPTX, PDF•
1 like•1,123 views
This document discusses building an ideal mobile app testing strategy. It recommends including various types of testing at different stages, such as unit testing, exploratory testing, build acceptance testing, and regression testing. It emphasizes testing apps on real devices rather than emulators to uncover issues related to hardware diversity, screen sizes, customizations, and memory/CPU. The document advocates enabling real device testing earlier in the process to lower bug fixing costs. It also recommends automating tests to speed up release cycles and integrating testing with continuous integration processes. The ideal strategy is described as using a cloud-based solution like Keynote Mobile Testing to provide on-demand access to many real devices for various types of automated and manual testing by development teams.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Mobile App Testing Best Practices
Similar to Mobile App Testing Best Practices (20)
More from Keynote Mobile Testing
More from Keynote Mobile Testing (19)
Recently uploaded
Recently uploaded (20)
Mobile App Testing Best Practices
- 1. Rachel Obstler VP Product Chris Karnacki Solutions Director Building a Complete, Multi-Stage Test Strategy with Keynote Mobile Testing Mobile App Testing Best Practices
- 2. 227BILLION apps downloaded in 2015 16%are willing to give a poor quality app more than one attempt 24,000 distinct Android devices in market
- 3. Today’s users expect apps to just work on their device 1-2 releases per month Access to Devices Faster Release Cycles -> Agile Processes Software Diversity Hardware Diversity Mobile and Agile Change the Test Process © 2016 Keynote by Dynatrace 3 Automation Continuous Integration Access to Real Devices Collaboration
- 4. 300+ shared devices Thousands of Private devices Enabling your agile process with automated and interactive access to hundreds of real devices, over the internet Keynote Mobile Testing © 2016 Keynote by Dynatrace 4 Interactive access to real devices for exploratory or negative testing Automation to speed the release cycle Multiple scripting options to meet the needs of your agile team Integrates with your CI process to automatically kick off tests Supports all stages of the agile test process
- 5. Building the Ideal Test Strategy © 2016 Keynote by Dynatrace 5 Unit testing Exploratory or Negative Testing Build Acceptance Testing Regression Testing
- 6. Building the Ideal Test Strategy Different screen resolution / screen size Unreadable text, blurred images, misalignment of screen elements, and items that fall off the screen. Android Customizations On-screen and physical controls that function differently across devices and emulators. Memory / CPU Low or insufficient memory or processing power. Impacts of other services running on the device. + Real Devices © 2016 Keynote by Dynatrace 6 Unit testing Exploratory or Negative Testing Build Acceptance Testing Regression Testing
- 7. Building the Ideal Test Strategy Development Unit Tests on Real Devices Cost of fixing bugs © 2016 Keynote by Dynatrace 7 Unit testing Exploratory or Negative Testing Build Acceptance Testing Regression Testing Development Testing Production
- 8. Building the Ideal Test Strategy Goal: Continuous Integration Automated tests run for each build On a small number of target devices © 2016 Keynote by Dynatrace 8 Unit testing Exploratory or Negative Testing Build Acceptance Testing Regression Testing
- 9. Building the Ideal Test Strategy © 2016 Keynote by Dynatrace 9 Unit testing Exploratory or Negative Testing Build Acceptance Testing Regression Testing Scripting Results
- 10. A. 0-25% B. 26-50% C. 51-75% D. 76-100% E. Not sure Poll: Of your organizations mobile development projects, what % of regression testing is automated? © 2016 Keynote by Dynatrace 10
- 11. Base: 155 professionals with knowledge of their firm’s Agile practices. Source: Forrester's Q2 2015 Global Agile Software Application Development Online Survey “Using your best estimate, how much do you automate the following?” How the Industry Prioritizes Automation © 2016 Keynote by Dynatrace 11 % Automation (mean across responses) Unit tests 53% Functional tests (GUI) 45% Functional tests (Beyond GUI, e.g., APIs) 42% We implement continuous testing – tests are kicked off at each code check in and integration (CI) 42% We automate nonfunctional (performance) 38% We automate nonfunctional (integration) 35% We automate test environment provisioning 30%
- 12. Building the Ideal Test Strategy © 2016 Keynote by Dynatrace 12 …but where possible, use real devices in the cloud Unit testing Exploratory or Negative Testing Build Acceptance Testing Regression Testing Test on real devices…
- 13. Building the Ideal Test Strategy Unit testing Exploratory or Negative Testing Build Acceptance Testing Regression Testing © 2016 Keynote by Dynatrace 13
- 14. Building the Ideal Test Strategy with Keynote Mobile Testing Unit testing Exploratory or Negative Testing Build Acceptance Testing Regression Testing Enable real device testing earlier in the process, lowering the cost of finding issues Speed time to market with automation and integration with your CI process Flexibly meet the needs of your entire agile team with various integrations and scripting options Reduce the asset management headache of managing mobile devices Enable remote teams with access to real devices, wherever they are © 2016 Keynote by Dynatrace 14
- 15. Schedule a demo of Keynote Mobile Testing Enterprise www.keynote.com GET STARTED TODAY!
Editor's Notes
- Mobile adoption is growing at incredible pace. But, mobile users of apps are less tolerant of errors and crashes in mobile apps than they are in the non-mobile application world. The first few times an app crashes, users just delete the app and move on. In app stores that have hundreds of other alternative apps, they just have too many choices. Quality is paramount. But it’s hard! The number and variety of mobile devices live in the world only makes updating apps more challenging. The pace of updates has gone from months to weeks, to days! Not to mention the complexity brought by the fragmentation of android devices
- I’ve heard people say that in agile there is no testing. Or at least no testers. This may make sense for the rare companies that are #1 b to c, and #2, don’t directly generate revenue from transactions. But for most companies this is not the case. You can’t imagine a bank saying “oops, sorry that transaction didn’t go through – it’s a beta”. Likewise a retail application losing a mobile sale. Agile and mobile do not reduce the need for testing, or remove the need for testers – it’s really the other way around. The additional platforms you need to test on, the frequency of releases, these things require even more testing. But to meet the demands of the market, the more rapid release schedules, and the need to support many different platforms and devices, agile and mobile do necessitate a change to the nature of testing. Because if you continue to test the way you always did with desktop apps and waterfall-like processes, you just won’t be able to get your apps out the door with speed and quality. So how does testing need to change to meet the needs of agile processes and mobile product releases? Well first, Automation is crucial. There’s no way to get all the testing done that you need without it, firstly at the build acceptance stage, and then for regressions. Testing can’t wait until the end of a development cycle. Agile practices means you are testing each small feature as you build it, constantly running build acceptance tests so you never break the build, and utilizing continuous integration techniques because you need to know within minutes if a build is broken – not days or weeks, or even hours later. Testing on real devices, and having a cloud. Agile also means developers are taking some responsibility for testing – namely unit testing. So testing is performed by the whole team, albeit at different stages. Test also often drives the outcome – many agile teams practice test driven development where the developer defines the tests first, and then develops the capability to pass them. And a space where your developers and testers can collaborate.
- Keynote mobile testing does just that. It solves the challenge of testing mobile apps and websites. The system solves the problems of device access and time to market with automation. It allows your testing and development teams access to real devices, using the tools that they prefer. It makes testing mobile apps straightforward! These are real working handsets that if you were to watch them live you would see the screen being navigate programmatically by our users from around the world. Direct connections to every device input and output allowing for full device functionality; as if the device were in your hand. Such features include: Automated testing brings a more efficient way to perform the same repeated manual tests over shorter period of time allowing you to increase the volume of testing you have to perform thereby increasing the quality of your products and improving ROI of your testing tools. With Keynote’s DeviceAnywhere Enterprise platform you are able to easily record test scripts at the object level, just by interacting with the device; create test cases with advanced logic and data-driven iterations; create test sequences with hundreds of test cases that run across multiple devices; run ad hoc testing anytime or schedule it at your convenience; view success rates, detailed errors, and screen-by-screen results in a web-based reporting portal; email and share test results with colleagues to provide continuous debugging functionality. Of course, we know that you don't use Keynote in isolation, so it's critical that we integrate with and leverage your existing investments in development and automated testing tools from major partners like SAP, IBM and HP. You can also utilize our Java API to program and execute scripts from existing test tools or integrate with your own build process to support continuous testing, such as Jenkins. This breadth of integration capability is also unique to Keynote. With Keynote you get a common scripting environment for both quality assurance and performance monitoring, so you can move an application from functional testing to monitoring and reuse the same scripts across different parts of your organization. Again, common scripting across QA and operations is available only from Keynote. Automated testing throughout your iterative development process is critical to ensuring the highest performance, with instant alerts whenever there is a problem. Once the application is released, Keynote delivers mobile app monitoring so you can monitor any use case for optimal performance regardless of platform or connectivity model.
- So really how is it done. There are lots of different needs of a mobile testing system – from interactive testing, automation, continuous integration, support for different tools in the development ecosystem, etc. So then let’s get down to specifics, and real world examples. In the next series of slides, we’ll talk about different testing stages, and the specific strategies to employ for each of them. And we’ll show you exactly how it’s done using Keynote Mobile Testing. There are of course other stages and types of testing not mentioned here. But these are certainly if not the most common, very close to it, and testing that is repeated often.
- So let’s start with unit testing. What are some of the key components of unit testing, that dictate how you carry it out? In agile, unit testing should be completed by the developer. In fact, in many agile organizations, the developer will hand off not just the feature but the unit tests and results that the testing generated. Agile processes like test driven development even promote writing the test before developing the functionality. And why does the developer do the testing? Well, one of the things that slows down development processes is inefficient handoffs. Imagine that a developer codes a task, and then passes it to QA. Who tests it, finds it doesn’t work, and then passes it back. That’s an unnecessary cycle that just wasted a bunch of time. QA / testers are trying to find unforeseen impacts of changes (regressions) and acting like a user to test a user story end to end. Or ensures that the user story is what was intended and there are no misunderstandings. As a developer, ideally you can create some unit tests easily in a language you prefer, and then run them on real devices. Why is it important to run on real devices? Well, it’s not ultimately a requirement to run on real devices at the unit testing stage. But if it’s easy enough, it’s always better to run on the actual platform the users will be using. Maybe not multiple devices – that’s something that can be done by QA in a regression cycle, or at the user story level. But the fact is that emulators operate differently than devices. IN our development environment, we not only specify that the developers test their own features, but that if they are handing something off to QA it gets demonstrated in the environment where they will be testing it, which should be as close to production as possible. Same goes for mobile apps – if you can test on the actual platform it will work on (e.g. an iphone 6S), you are increasing quality of the release, and catching any issues earlier (or making it less likely that you’ll encounter unexpected platform-based issues later in the process.
- And that’s because it’s cheaper to fix issues earlier; when they are still top of mind, and less people have had their hands on it. Now I’m going to hand it over to Chris, who will show us some unit testing in action, and how easy it is to run appium unit tests on real devices. Show utilizing Appium on shared, private or local devices Easy to run Also show SAP – easy to validate
- The key elements of build acceptance testing are very few – basically the goals of agile is to never break the build. Of course, it’s never that easy. That’s why build acceptance testing is so crucial. If it does break, you need to know ASAP. Gone are the days where development would hand over a build to QA and find out days later that the build is no good. That’s another wasted handoff, wasted time, and also at that point most of the developers have lost context of what they had worked on, there are too many changes, and it’s hard to find the problem. Agile changed this. But – with this change comes challenges – rapid change means testing must become much more efficient. So – this is why continuous integration and automation are so important. Every time a new build is generated (imaging daily, or even more often), a set of full automated tests should automatically run against the build to quickly assess whether it is good. Ideally these tests should run within minutes so you get quick feedback, and also don’t overload your testers with a bunch of repetitive testing. Back now to Chris, who will show us an example of a build acceptance test running on multiple real devices, automatically kicked off for a new build, from Jenkins.
- So that leads right into regression testing. There is no way in a 2-week cycle, or even every month or two month releases to do a full regression unless you are automating at least some of it. There are too many devices to test across. So similar to your unit testing and build acceptance testing, you need an ability to script and run on real devices. But for a regression test, you need more. You need ways to manage your test cases. Reuse pieces of them. Collate all of your results. Regression tests can be numerous, and a full regression can take hours. You need a stable system to run long series of tests within. The other thing is to understand the profile and preferences of your automation team. Who is going to be writing this automation, and what skills do they have, or tools do they work within? HP has had a large portion of the wired / web testing market for many years. Are your employees used to using QTP? Are they programmers, and have they for instance been using Appium? Do they want a GUI within which to write tests? Are they new, so a recorder is a helpful place to start, to make it very easy? If your org is like a lot of orgs out there, your answer may be all of the above. Or what about other tools in the ecosystem. Do you have a requirements management process? In any case, find a tool that can support your users and processes – you can always train them and get them up to speed on new stuff, but the extent to which they are using something where they are comfortable always assists the adoption process. Keynote Mobile Testing supports
- So whether you are agile or not, automation is important for both improving timelines and efficiency, and also for ensuring quality, through offloading rote actions that manual testers can find boring and may experience fatigue, and also offloading testers so they can do more high value manual testing. I can talk about how it’s always a struggle to spend time in a given release cycle to automate – really need to set aside resources for this purpose.
- We’ve talked so far about 3 stages of testing – certainly have not covered every stage there is, but in all of these stages, a common element is automation is prevalent. But also not everything can be automated. Not all regression tests easily lend themselves to automation. Automation is always an equation – if it takes more time to automate than you save not having to do the manual testing, it probably doesn’t make sense! So Manual testing still remains an important piece of the puzzle.
- So that brings us to the last stage of testing we are going to cover – exploratory or negative testing. Exploratory testing important for mobile apps. User experience as we mentioned earlier important. People will delete your app! I mentioned manual testing plays a big role – and here is why. Act like a user – test on real devices. You don’t have to test on every device – but make it a point to know what devices your users are using. It’s often straightforward to get this type of data from marketing applications like adobe. If it’s the first launch or you don’t have access to this info, use info on the most popular devices – or you can always ask Keynote. We often work with our customers to come up with a device testing strategy that hits the major OS’s and the major manufacturers, so you get a good cross section of test data. And secondly, use devices in the cloud. There are manual testing offers out there (keynote offers one) through which you can really do just about everything, including orientation testing, turning devices on and off, And you don’t need to manage devices, buy devices, search for devices, manage plans, etc. Most companies at this point enable BYOD, so there is not a function that manages devices. You don’t want to have to have one within your testing or dev teams – it’s inconvenient! The oher reason why devices in the cloud are important is that it automatically saves a record of your activity. Let’s say you find a bug. Chris is now going to show us how easy it is to do negative or exploratory testing, without having to have any devices at your desk. (include SAP example).
- Testing is varied. Different stages need different things, are performed by different people, etc. but has common elements. Real devices All people using a consistent set of real devices Managed by someone else So you can focus Utilizing Keynote Mobile Testing for your mobile development practice allows you to: