Exploratory Mobile Testing Webinar_XBOSoft_jean_annharrison
Upcoming SlideShare
Loading in...5

Exploratory Mobile Testing Webinar_XBOSoft_jean_annharrison



To Automate or not to Automate your Mobile Testing. ...

To Automate or not to Automate your Mobile Testing.

In mobile testing just poking at the GUI will leave bugs hiding. So different tests and a variety of testers are needed. Context is also important; there is no one test set or test approach that will work all the time.

JeanAnn Harrison has years of experience with mobile testing and is a well-known figure in the QA and software testing community. She regularly speaks at conferences and publishes in software testing magazines.

In these slides JeanAnn discusses mobile testing strategies that deliver the right results.

You will learn:

- Types of Mobile Testing
- When and when not to automate your mobile testing
- Mobile exploratory testing strategies and guidelines
- Lesson learned



Total Views
Views on SlideShare
Embed Views



1 Embed 10

https://twitter.com 10



Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

Exploratory Mobile Testing Webinar_XBOSoft_jean_annharrison Exploratory Mobile Testing Webinar_XBOSoft_jean_annharrison Presentation Transcript

  • Project Realms & XBOSoftExploratory Mobile TestingWebinar
  • JeanAnn Harrison & Jan Princen
  • XBOSoft info• Founded in 2006• Dedicated to software quality• Software QA consulting• Software testing services• Offices in San Francisco and Amsterdam
  • Housekeeping• Everyone except the speakers is muted• Questions via the gotowebinar control on theright side of your screen• Questions can be asked throughout thewebinar, we’ll try to fit them in whenappropriate• General Q and A at the end of the webinar• You will receive info on recording after thewebinar
  • AGENDA• Introduction• Note taking & use of slideinformation• Types of devices & Apps• Quick overview of definitionsfor Mobile Testing• Planning out Testing• Automation ConceptsJean Ann Harrison Copyright 2013• Exploratory Testing Definition• Good Exploratory TestingSkills/Mobile ET Skills• Keys to Mobile ET• Designing your Heuristics• Organizing Tests/Stories• Control System Considerations• Hardware/Firmware test ideas• Common tools for ET• Summation & ConclusionNot covered – User Experience, Usability,Trainability, Web Application & Security Tests
  • MOBILE/EMBEDDEDSOFTWARE & DEVICES• Mobile phones, tablets• Proprietary devices: mobile heart monitors, law enforcementticket generators, restaurant personnel taking customer orderson devices, GPS devices, PDAs• Embedded Software examples: automobile’s computer system,air traffic control systems, airplane navigation system, druginfusion pumps in hospital rooms, elevators, cameras, robotsJean Ann Harrison Copyright 2013
  • TYPES OF MOBILEAPPS• Native Applications• Local to device• Hybrid Applications• Local to device butinteracts w/internet• Web Applications• Not local to device.All interactions oninternetJean Ann Harrison Copyright 2013
  • DEFINING SKILL SETFOR THEMOBILE/EMBEDDEDTESTER• Some exposure or knowledge about products from the domain in which youare testing: aerospace, medical, automobile manufacturing, airplanes,factory systems, robotics, regulated environments, etc.• Some knowledge of: hard sciences: math, physics, electronics, engineering,etc for logical thought process.• Software sciences: psychology, philosophy, sociology, human factors (humanmachine interface) for creative & conceptual thought process• These testers use skills from many knowledge domains, patterns of errors,and basic testing skills to create tests. They are using their intuition, criticalthinking, and their mental models.Chapter 1 – Software Test Attacks to Break Mobile & Embedded DevicesJean Ann Harrison Copyright 2013
  • MOBILE TESTINGDEFINITIONS• Mobile Application Testing is testing the application on a mobiledevice.• Mobile Device Testing is testing hardware and operating system. Doesthe Operating System install? Does the device power on? Do the LEDlights work as expected? Does the battery charge when the ACAdapter is plugged into the phone?• Mobile Phone Testing is doing any testing on a mobile phone.• Mobile System Testing – incorporates testing more than oneapplication and can combine hardware, software, firmware, along withother applications.• Mobile Testing – all of the above.Be clear when using this terminology. If you are only testing apps onmobile phones, then state “mobile apps testing” and mobile testingwhich technically incorporates mobile website testing, mobile hybridapps, mobile hardware.Jean Ann Harrison Copyright 2013
  • TO AUTOMATE OR NOT TO AUTOMATE?CONCEPT: Planning – What do you want to Test?Jean Ann Harrison Copyright 2013Plan out types of tests – what do you want to test so you know thenwhat you can even consider to automate?Types of tests include: functional, regression, usability, performance,stress, load tests, system integration tests, trainability andconfiguration tests.
  • DEFINE YOUR TESTS• Performance testing for mobile software is not the same asfor web or client/server apps• How does the app behave on a device vs talking to aserver?• How does the app perform interacting with otherapplications on device? Does it play nice? <note iPhone 4error shutdown>• Stress & load: use the environment conditions to stressand put load on device.Jean Ann Harrison Copyright 2013
  • Jean Ann Harrison Copyright 2013
  • CONCEPT: USABILITY TESTING &CONFIGURATION TEST COMPARISONS• Usability Testing & configuration test comparisons:• Critical testing for mobile applications and to automate isdifficult. You can test font sizes but then do these sizescompare across configurations? Does one automatedscript fit all? iPhone 5 vs Galaxy 4 or how about an iPad vsa Kindle?• Ease of use on devices are difficult to automate – questionis do you want to spend the time trying to write scriptswhen manually you not only can test faster, you cancombine your tests and test once within a project.Jean Ann Harrison Copyright 2013
  • CONCEPT: TESTCOVERAGE• When applying one script on an iPhone vs KindleFire, your script may not contain all recognizableobjects on one configuration vs the other. Youmay need more than one script to cover variousconfigurations. Is it worth writing these scriptsup? Planning out is key.Jean Ann Harrison Copyright 2013
  • CONCEPT: SPEED OFTEST• Think about how fast you could test for examplethe displayable area for the application on aphone and then a tablet. How quickly could theeye scan the displayable area? How would youautomate this?Jean Ann Harrison Copyright 2013
  • CONCEPT:MAINTAINABILITY• Consider how easily to maintain your scripts. Will you beforced to rewrite your scripts after each new release? Whatcan you easily maintain? Mobile applications often changedramatically from release to release. Work with Design teamfrom release to release to see what changes will come up.Jean Ann Harrison Copyright 2013
  • CONCEPT: REGRESSION ANDFUNCTIONAL TESTING• Scripts can be done to record how someonewould use the application but if there are differentpaths, many options, do you need to record thesescripts? Sometimes a “once through” is enoughtesting to make sure the functionality works.Jean Ann Harrison Copyright 2013
  • CONCEPT: TESTING BEYONDTHE GUI• Charging the device, notification testing, networkcommunication are all tests which can becombined with your performance testing andfunctional testing. But you need to OBSERVEwhat is happening when it’s happening. Is itnecessary to have the test run automatically?Jean Ann Harrison Copyright 2013
  • CONCEPT: ONE SIZE DOES NOTFIT ALL• Do not expect one tool to do it all. Operatingsystem testing will not be included in a touch andrecord tool. You may have to create small toolsyourself to get the job done but again, is it worthyour time to do so?Jean Ann Harrison Copyright 2013
  • EXPLORATORY TESTING -DEFINITION• Quoting James Bach: “The plainest definition of exploratorytesting is test design and test execution at the same time. This isthe opposite of scripted testing (predefined test procedures,whether manual or automated). Exploratory tests, unlikescripted tests, are not defined in advance and carried outprecisely according to plan.”http://www.satisfice.com/articles/what_is_et.shtmlJean Ann Harrison Copyright 2013
  • WHAT YOU NEED TO BEA GOOD EXPLORATORYTESTER• Awareness• Discipline• FocusJean Ann Harrison Copyright 2013
  • KEYS TO MOBILEEXPLORATORY TESTING• Look beyond the GUI• Timing• Domain knowledge• Knowledge of System(hardware/firmware/software)• Characteristic of bugs behavior, formulatepatterns• Use life experiences for inspirationJean Ann Harrison Copyright 2013
  • TESTING BEYOND THE GUI = SYSTEMINTEGRATION TESTINGJean Ann Harrison Copyright 2013SIT for Mobile/Embedded contains elements:Software + Hardware + Software Variants +Hardware Variants + Timing + OperatingSystem Drivers + Network Protocols
  • WHAT DO WETEST?Jean Ann Harrison Copyright 2013
  • LEARNING THE ARCHITECTUREWork closely withdevelopment to learn moreabout how the mobileapplication under test works.Jean Ann Harrison Copyright 2013
  • WHERE DO BUGS LURK?Desktop, Web, Mobile/Embedded Considerations:• Requirements & Design• Logic & Math• Control Flow• Data• Initialization & Mode changes• Interfaces• Security• Gaming functions• etc…Originally written by Jon D Hagar Copyright 2013 Jean Ann Harrison Copyright 2013Mobile /Embedded Considerations:• Software and hardwaredevelopment cycles done inparallel, where aspects of thehardware may be unknown to thesoftware development effort• Hardware problems which areoften fixed with software late inthe project• Small amounts of dense complexfunctions often in the controltheory or safety/hazard domains• Very tight real-time performanceissues (often in mili or microsecond ranges)
  • ”WE BECOME MORE EXPLORATORY WHEN WECANT TELL WHAT TESTS SHOULD BE RUN, INADVANCE OF THE TEST CYCLE, OR WHEN WEHAVENT YET HAD THE OPPORTUNITY TOCREATE THOSE TESTS.” JAMES BACH• Create your description or theory. “What happens when…?”• … battery runs low?• … battery is charging from a dead battery?• … when the device gets too hot?• … when the battery is replaced due to defective battery?• … device memory is full?• … I receive a notification while using the phone or another application?• … when there is a time/date change?• … device searches are too slow?Jean Ann Harrison Copyright 2013
  • ORGANIZE YOURTHOUGHTS/TESTSJean Ann Harrison Copyright 2013BatterylevelTime/Temp ApplicationInstallingApplicationInstalled/not inuse/startedApp in use (note –various conditionsapply here)0%-5% Time each levelincrement ofchargeTiming ofdownload?Wouldnotificationsfactor?Notifications?Memory usageCPU speed6%-15% Check the tempat eachincrementCompletedinstallation?Note CPU speed,memory usageSearches withapplication DB, onlineDB?16%-30% Interaction with datatransfer/networkcommunication31%-50%
  • CONTROL SYSTEM CONSIDERATIONSDEVELOPING TESTS:HARDWARE TO SOFTWARE SIGNALINTERFACE• Know the input and output connections to the software• Consider differences in variations between devices like calibrations, physical noise,electron, light, cold, water, dust device, response time, wear & tear on the hardware.• Identify input & output devices with ranges and resolutions of values (how is thesoftware installed onto the device?)• Define the full range of environmental input disturbances (unexpected inputs)• Define possible environmental output disturbances (unexpected outputs)• Determine what is or is not possible in the test lab• Conduct a risk analysis of what is acceptable and what is not• Understand device limitations: CPU processing capabilities, memory, time.• The Aerospace Industry use the term, “Test like you fly, fly like you test” to combine allfacets of how a device’s software should be tested.Originally written by Jon D Hagar Copyright 2013 Jean Ann Harrison Copyright 2013
  • • Hardware to Software signal type test• Network/cellular communication signal strength tests• Installation of software on a mobile device• Device limitation tests: CPU processing capabilities, storage, memoryJean Ann Harrison Copyright 2013
  • HARDWARE / SOFTWARE• Evaluate Software with Hardware Variants and System Operations• Finding Bugs in Hardware–Software Communications (includeFirmware)• Stressing Software Error RecoveryJean Ann Harrison Copyright 2013
  • SPECIALIZED MOBILE / EMBEDDEDSOFTWARE ATTACKS• Network Protocol Communications• Time-Based• Security & FraudJean Ann Harrison Copyright 2013
  • COMMON TOOLS FORORGANIZATION• Sticky notes per conditions, per variables, per test case idea• Keep a log of what you are doing, record conditions• Utilize the device’s log or find an open source logging tool• Use tables to record all variables/conditions per test• Document situation of the test, list out your test conditionsJean Ann Harrison Copyright 2013
  • SUMMARIZATION OFTECHNIQUES• Combining tests can help shorten project schedulesbut what also happens is you can witness results.Automation results do not give you all you need towitness as it happens.• Automate positive end to end use for regression onlybetween builds.• Using tools which measure memory usage, to getstatistics while conducting regression tests.• Documenting includes use of video,writing/development of tests in story formats, useraction formatsJean Ann Harrison Copyright 2013
  • CONCLUSION• Remember: In mobile and embedded just poking at the GUI will leave bugs“hiding”• Different tests and attacks by a variety of testers (developer and titledtesters) is commonly needed• Context matters: there is no one set of tests, approaches, testers, orattacks which will work all the time• Testing is hard and takes the whole teamJean Ann Harrison Copyright 2013
  • JEAN ANN HARRISON’SCONTACTINFORMATION• EMAIL: jaharrison@projectrealms.com• Twitter: https://twitter.com/JA_HarrisonOr @JA_Harrison• Project Realms Inc website:www.projectrealms.comJean Ann Harrison Copyright 2013
  • Q & AWant to keep updated on upcoming webinars?Follow us @xbosoftNeed any help with mobile testing?Contact us: services@xbosoft.com
  • Thanks!
  • REFERENCES• Software Test Attacks to Break Mobile & Embedded Devices by Jon D Hagar to bepublished and released fall 2013.• 2013 STPCon Presentation slides from “Testing Beyond the GUI” by Jon D Hagar &Jean Ann Harrison• http://www.ministryoftesting.com/2012/06/getting-started-with-mobile-testing-a-mindmap/ Getting started with Mobile Testing Mindmap• http://karennicolejohnson.com/ Karen Johnson’s website• http://www.satisfice.com/articles/what_is_et.shtml by James Bach, Satisfice, Inc• http://www.softwaretestpro.com/Item/5567/ A Different Take on Mobile Testing –Test Beyond the GUI by Jean Ann HarrisonJean Ann Harrison Copyright 2013
  • IMAGESREFERENCES• Slide 3 - Embedded Systems examples• http://www.jijesoft.com/en/?option=com_content&view=article&id=18&Itemid=45• http://www.mseedsystems.com/products/view/268/aldelo-for-restaurants-pos-wireless-edition• http://blog.laptopmag.com/wpress/wp-content/uploads/2013/02/Pandora-Limit-Free-Listening.jpg• slide 4 - types of Apps• http://myeventapps.com/whats-the-difference-native-vs-web-apps/2012/794/• slide 5 - defining skills• http://cdn.cutestpaw.com/wp-content/uploads/2012/11/l-The-Thinker.jpgJean Ann Harrison Copyright 2013
  • IMAGESREFERENCES• slide 7 to automate or not to automate• http://www.eoi.es/blogs/veronicarecanati/2013/02/10/project-management-why-do-projects-fail/• Slide 9 - mindmap used with permission from originators Karen Johnson & James Bach• www.ministryoftesting.com• slide 18 - What you need to be a good Exploratory• Testerhttp://www.wallcoo.net/paint/kagaya_celestial_exploring/Kagaya_art_Celestial_Exploring_INSPIRATION1.html• Slide 19 - puzzling together pieces• http://www.aspire2develop.co.uk/images/People%20development%20page%20&%20HOME%20page%204%20piece%20jigsaw%20iStock_000002077956Small.jpg• slide 20 - Testing beyond the GUI• http://courses.cs.vt.edu/csonline/OS/Lessons/Introduction/onion-skin-diagram.gifJean Ann Harrison Copyright 2013