Are You Missing Critical Mobile Tests?


Published on

Expert tester and speaker JeanAnn Harrison reveals critical mobile tests that can be used on many mobile devices, even proprietary. Exercises and ideas for testing that you can use and build on. Sponsored by XBOSoft.

Published in: Technology, Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Clareice: Master’s Degree PMP certification from the Project Management Institute and is a Certified Professional Contracts Manager through the National Contract Management Association (NCMA). She has presented at the National Contract Management Association World Congress.Clyneice:She has participated as an examiner for state quality awards for Georgia and Virginia. She is currently an instructor for the International Institute for Software Testing and has presented technical papers at the Software Engineering Institute: SEPG Conference, American Society for Quality: Quality Manager's conference, Quality Assurance Institute International Testing Conference, International Conference on Software Process Improvement and Software Test and Performance Testing Conferences.
  • We can change the wording but the concept I think is important – chapter 1
  • Reference “Integration Attack”.Attacking “internals” where bugs may exist but not be seen in the GUI – give examples of system attacks
  • BASED ON NOTES BELOW: LET’S ASK THE CLASS IF THEY CAN SHARE NON-GUI BUGS THEY HAVE FOUND AND START A DISCUSSION AS A WAY TO SUMMARIZE. IF NONE-What can if “exercise can we do” to engage them??? Some ideas - Share a bug (can you tell us a bug you found and why or better a bug that escaped you and why” - What are the types of testing do you do now exercise? GUI testing, Requirements checks, exploration, attacks, what? - How many test techniques and/or attacks can you name (a white board organized brain storming exercise)
  • Are You Missing Critical Mobile Tests?

    1. 1. JeanAnn Harrison & XBOSoft Are You Missing Critical Mobile Tests?
    2. 2. XBOSoft info • Founded in 2006 • Dedicated to software quality • Software QA consulting • Software testing services • Offices in San Francisco, Beijing, Oslo and Amsterdam
    3. 3. On the XBO side: Amy Jan •
    4. 4. Housekeeping • Everyone except the speaker is muted • Questions via the gotowebinar control on the right side of your screen • Questions can be asked throughout the webinar, we’ll try to fit them in when appropriate • General Q & A at the end of the webinar • You will receive info on recording after the webinar
    5. 5. Reminders Make sure your drained device is available, as well as another device to send email to the drained device and a charger. When you see the heart “pumping iron” slides JeanAnn will walk us through a test on your device. Please type questions in the Chat box. We are looking for 1-2 volunteers to take the microphone during their device testing to let everyone know what is happening.
    6. 6. About the Speaker JeanAnn Harrison regularly speaks at industry conferences and is published in software testing magazines. She has years of experience with mobile testing and is a well-known figure in the QA and software testing community. Besides software testing , JeanAnn really likes baseball & ice skating.
    7. 7. WEBINAR: EXERCISES IN EXPLORATORY TESTING JEAN ANN HARRISON Software Test Consultant for Project Realms. FOR MOBILE AND EMBEDDED TESTERS Jean Ann Harrison Copyright 2013
    8. 8. AGENDA • Introduction • Note taking & use of slide information • Quick overview of Mobile/Embedded Software • Types of Mobile Apps • Exploratory Testing Definition • Defining Skill Set for the Mobile Tester • Mindmap getting started with Mobile Testing • Good Exploratory Testing Skills/Mobile ET Skills Jean Ann Harrison Copyright 2013 • Keys to Mobile Exploratory Testing • Testing Beyond the GUI • Learning the Architecture • Organizing Tests/Stories • Common tools for ET • Some Test Case Ideas • Exercises & Test Case Examples • Summation & Conclusion Not covered – User Experience, Usability, Trainability, Web Application & Security Tests
    9. 9. MOBILE/EMBEDDED SOFTWARE & DEVICES • Mobile phones, tablets • Proprietary devices: mobile heart monitors, law enforcement ticket generators, restaurant personnel taking customer orders on devices, GPS devices, PDAs • Embedded Software examples: automobile’s computer system, air traffic control systems, airplane navigation system, drug infusion pumps in hospital rooms, elevators, cameras, robots Jean Ann Harrison Copyright 2013
    10. 10. TYPES OF MOBILE APPS • Native Applications • Local to device • Hybrid Applications • Local to device but interacts w/internet • Web Applications • Not local to device. All interactions on internet Jean Ann Harrison Copyright 2013
    11. 11. EXPLORATORY TESTING - DEFINITION • Quoting James Bach: “The plainest definition of exploratory testing is test design and test execution at the same time. This is the opposite of scripted testing (predefined test procedures, whether manual or automated). Exploratory tests, unlike scripted tests, are not defined in advance and carried out precisely according to plan.” Jean Ann Harrison Copyright 2013
    12. 12. DEFINING SKILL SET FOR THE MOBILE/EMBEDDED TESTER • Some exposure or knowledge about products from the domain in which you are testing: aerospace, medical, automobile manufacturing, airplanes, factory systems, robotics, regulated environments, etc. • Some knowledge of: hard sciences: math, physics, electronics, engineering, etc for logical thought process. • Software sciences: psychology, philosophy, sociology, human factors (human machine interface) for creative & conceptual thought process • These testers use skills from many knowledge domains, patterns of errors, and basic testing skills to create tests. They are using their intuition, critical thinking, and their mental models. Chapter 1 – Software Test Attacks to Break Mobile & Embedded Devices Jean Ann Harrison Copyright 2013
    13. 13. Jean Ann Harrison Copyright 2013
    14. 14. WHAT YOU NEED TO BE A GOOD EXPLORATORY TESTER • Awareness • Discipline • Focus Jean Ann Harrison Copyright 2013
    15. 15. KEYS TO MOBILE EXPLORATORY TESTING • Look beyond the GUI • Timing • Domain knowledge • Knowledge of System (hardware/firmware/software) • Characteristic of bugs behavior, formulate patterns • Use life experiences for inspiration Jean Ann Harrison Copyright 2013
    16. 16. TESTING BEYOND THE GUI = SYSTEM INTEGRATION TESTING Jean Ann Harrison Copyright 2013 SIT for Mobile/Embedded contains elements: Software + Hardware + Software Variants + Hardware Variants + Timing + Operating System Drivers + Network Protocols
    17. 17. WHAT DO WE TEST? Jean Ann Harrison Copyright 2013
    18. 18. LEARNING THE ARCHITECTURE Work closely with development to learn more about how the mobile application under test works. Jean Ann Harrison Copyright 2013
    19. 19. EXERCISE FOR GROUP • In the question box, enter 1 test idea of what you want to explore. The point of this exercise is you do NOT know what the application is, so you can’t tell me what kind of functional test to perform. • I’m testing YOU to see what kinds of non-functional tests you think might be good to explore. Jean Ann Harrison Copyright 2013
    20. 20. COMMON TOOLS FOR ORGANIZATION • Sticky notes per conditions, per variables, per test case idea • Keep a log of what you are doing, record conditions • Utilize the device’s log or find an open source logging tool • Use tables to record all variables/conditions per test • Document situation of the test, list out your test conditions Jean Ann Harrison Copyright 2013
    21. 21. ”WE BECOME MORE EXPLORATORY WHEN WE CAN'T TELL WHAT TESTS SHOULD BE RUN, IN ADVANCE OF THE TEST CYCLE, OR WHEN WE HAVEN'T YET HAD THE OPPORTUNITY TO CREATE THOSE TESTS.” JAMES BACH • Create your description or theory. “What happens when…?” • … battery runs low? • … battery is charging from a dead battery? • … when the device gets too hot? • … when the battery is replaced due to defective battery? • … device memory is full? • … I receive a notification while using the phone or another application? • … when there is a time/date change? • … device searches are too slow? Jean Ann Harrison Copyright 2013
    23. 23. TEST CASE STEPS 1. Pre-select any application to download onto device 2. Drain your device’s battery charge completely 3. Notate time 4. Plug in charging device 5. Immediately power up device 6. Record times for first condition where app is not installed, device powered up Jean Ann Harrison Copyright 2013 7. Drain your device’s battery charge completely 8. Plug in charging device 9. Immediately power up device 10. Notate time 11. Start to download pre-selected app 12. Record times for each battery level while downloading
    24. 24. EXERCISE: EXPLORATION FOR MORE TESTS Jean Ann Harrison Copyright 2013 • What other kinds of tests can you apply based on the condition of starting to charge from a dead battery? • Record your tester stories
    25. 25. ORGANIZE YOUR THOUGHTS/TESTS Jean Ann Harrison Copyright 2013 Battery level Time/Temp Application Installing Application Installed/not in use/started App in use (note – various conditions apply here) 0%-5% Time each level increment of charge Timing of download? Would notifications factor? Notifications? Memory usage CPU speed 6%-15% Check the temp at each increment Completed installation? Note CPU speed, memory usage Searches with application DB, online DB? 16%-30% Interaction with data transfer/network communication 31%-50%
    26. 26. CONTROL SYSTEM CONSIDERATIONS DEVELOPING TESTS: HARDWARE TO SOFTWARE SIGNAL INTERFACE • Know the input and output connections to the software • Consider differences in variations between devices like calibrations, physical noise, electron, light, cold, water, dust device, response time, wear & tear on the hardware. • Identify input & output devices with ranges and resolutions of values (how is the software installed onto the device?) • Define the full range of environmental input disturbances (unexpected inputs) • Define possible environmental output disturbances (unexpected outputs) • Determine what is or is not possible in the test lab • Conduct a risk analysis of what is acceptable and what is not • Understand device limitations: CPU processing capabilities, memory, time. • The Aerospace Industry use the term, “Test like you fly, fly like you test” to combine all facets of how a device’s software should be tested. Originally written by Jon D Hagar Copyright 2013 Jean Ann Harrison Copyright 2013
    27. 27. Jean Ann Harrison Copyright 2013 • Download either Twitter or Facebook onto device • Start either downloaded app • From another device, send an email to the device’s email account • Immediately send a tweet or post a status • Continue to do engage Twitter or Facebook app for at least 1 minute • Record email notification and time when sent and received. • What other observations occurred? EXERCISE: NOTIFICATION TEST
    28. 28. EXERCISE: EXPLORING MORE FOR THE INTERRUPT TEST • What other kinds of interrupts could be tested? • What other tests would you consider if something goes wrong? Jean Ann Harrison Copyright 2013
    29. 29. TEST CASE EXAMPLE: INTERRUPTS ON MOBILE PHONES • Go to your App store and choose an application to download • While the downloading is occurring, call the mobile phone • Record observations with the download • You may need to rely on observing a log file while implementing these tests • If it fails, what kind of error recovery occurs? Jean Ann Harrison Copyright 2013
    30. 30. LET’S TALK DATA AND STORAGE • Mobile devices contain limited amount of storage space. • Testing for mobile apps storing data locally on a mobile device means testing the boundaries of the app’s design. How will developers handle the limitations? • Work with your developers to understand those boundaries. • If your app doesn’t have a boundary set, then the app can fill up your device’s storage capabilities. • Testing the limits might mean testing how the software polices that boundary by having a specific limit before automatically downloading only so much information. • Per usage, email programs only download so much data and most recent data to the device. Jean Ann Harrison Copyright 2013
    31. 31. TEST CASE EXAMPLE: TESTING STORAGE LIMITS • Download several large applications (ex: Angry Birds & Facebook are large apps) • Go to your Settings and then to “Manage applications”. • Review the amount of total storage free. • Keep downloading apps until you reach below 5KB • Then start an application where you would save data on the device. • Check your free storage amount. • Keep saving data until you reach a point where you receive an error message or a “can’t save” message. • Check the remaining free storage in the Settings again to see what the threshold is for free space is maxed out. • Delete the last data which was just added. • Send data though the internet, but use the app under test (using the application where you first started saving data) and not just email. • What other tests can you do where storage is maxed out? Jean Ann Harrison Copyright 2013
    32. 32. SUMMARIZATION OF TECHNIQUES • Combining tests can help shorten project schedules but what also happens is you can witness results. Automation results do not give you all you need to witness as it happens. • Ask yourself “what if I … “ and consider tests not related to functional behavior or conditions. • Using tools, especially log files, can reveal memory usage, getting statistics while conducting tests. • Documenting includes use of video, writing/development of tests in story formats, user action formats Jean Ann Harrison Copyright 2013
    33. 33. CONCLUSION • Remember: In mobile and embedded just poking at the GUI will leave bugs “hiding” • Different tests by a variety of testers (developer and titled testers) is commonly needed • Context matters: there is no one set of tests, approaches, testers, or attacks which will work all the time • Limited resources on device can make testing challenging and requires an entire team effort Jean Ann Harrison Copyright 2013
    34. 34. JEAN ANN HARRISON’S CONTACT INFORMATION • EMAIL: • Twitter: Or @JA_Harrison • Project Realms Inc website: • 8/10/2013, I will be facilitating a Weekend Testing Session on Risk Based Testing for Mobile Devices… details can be found through my Twitter updates. Jean Ann Harrison Copyright 2013
    35. 35. REFEREN CES • Software Test Attacks to Break Mobile & Embedded Devices by Jon D Hagar to be published and released fall 2013. • 2013 STPCon Presentation slides from “Testing Beyond the GUI” by Jon D Hagar & Jean Ann Harrison • mindmap/ Getting started with Mobile Testing Mindmap • Karen Johnson’s website • by James Bach, Satisfice, Inc • A Different Take on Mobile Testing – Test Beyond the GUI by Jean Ann Harrison Jean Ann Harrison Copyright 2013
    36. 36. IMAGES REFERENCES • Slide 3 - Embedded Systems examples • • • • slide 4 - types of Apps • • Slide 5 – Exploratory Testing Definition • • slide 6 - Defining skill set • Jean Ann Harrison Copyright 2013
    37. 37. IMAGES REFERENCES • Slide 7 - mindmap used with permission from originators Karen Johnson & James Bach • • slide 8 - What you need to be a good Exploratory • Tester lestial_Exploring_INSPIRATION1.html • Slide 9 - puzzling together pieces • 0&%20HOME%20page%204%20piece%20jigsaw%20iStock_000002077956Smal l.jpg • slide 10 - Testing beyond the GUI • diagram.gif Jean Ann Harrison Copyright 2013
    38. 38. • Slide 12 – Leaning the architecture • gO3DIydtaQA/UBoUj4afwsI/AAAAAAAAAvk/cwPAtXkcgJ4/s1600/12152009_muppets7.jpg • Slides 13, 18, 21 22 • • Slide 14 – Common organizational tools • carpenter-workplace-with-desk-design-my-room-interior-design-tool-936x624.jpg • Slide 15 – James Bach quote of Exploratory Testing • Jean Ann Harrison Copyright 2013 IMAGES REFERENCES
    39. 39. • Slide 24 – Let’s Talk Data Storage • https://encrypted- wPTm_05O4Yr7uhEP--PUZhy-bX • Slide 27 – Conclusion • Apps-300x188.jpg Jean Ann Harrison Copyright 2013 IMAGES REFERENCES
    40. 40. Follow us: @XBOSoft Questions at: