Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Mobile Software Testing Challenges

1,308 views

Published on

While mobile applications are becoming so extraordinarily adopted, it is still unclear if they deserve any specific testing approach for their verification and validation. This presentation is touching on this subject and prepared for SWE-550 Software Quality Assurance & Realibility course at Bogazici University

Published in: Software
  • Test your mobile web and native mobile applications on multiple mobile devices that increasingly command adoption & usage for enterprise and everyday computing. https://goo.gl/caU3Xp
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Mobile Software Testing Challenges

  1. 1. Mobile Software Testing Challenges Busra Deniz SWE 550, Fall 2015 2014719135
  2. 2. Mobile World Statistics Mobile phone users worldwide (in billions)
  3. 3. Mobile World Statistics Number of mobile app downloads worldwide (in millions)
  4. 4. Mobile World Statistics Worldwide mobile app revenues (in billion U.S. dollars)
  5. 5. Mobile in critical domains Payment Health m-government
  6. 6. 3 Questions ❖ RQ1 : Are mobile applications so different from traditional ones, so to require different and specialized new testing techniques?
  7. 7. 3 Questions ❖ RQ1 : Are mobile applications so different from traditional ones, so to require different and specialized new testing techniques? ❖ RQ2 : What are the new challenges on testing mobile applications ?
  8. 8. 3 Questions ❖ RQ1 : Are mobile applications so different from traditional ones, so to require different and specialized new testing techniques? ❖ RQ2 : What are the new challenges on testing mobile applications ? ❖ RQ3 : Which is the role automation may play in testing mobile applications ?
  9. 9. What is a mobile application ? ❖ A mobile application is vaguely defined as an application running on mobile devices and/or taking in input contextual information. ❖ Mobility and context-awareness ➢ In mobile computing, an application is considered to be mobile if it runs on an electronic device that may move (mp3 readers, digital camera, mobile phones) ■ limites resources, security, performance, reliability and finite energy source ➢ In context-aware computing, an application is aware of computing environment in which it runs, and adapts/reacts according to its computing, user, physical or time context ■ location, time, nearby objects, orientation, social contexts. ❖ App4Mobile & MobileApps Applications Apps4Mobile MobileApps C ontext-aw are applications M obile A pplications
  10. 10. Mobile Application Peculiarities & Implications Type of Mobile Applications Peculiarity Implications on Testing Mobile Connectivity Reliability, performance, security and functional testing through different networks Limited Resources Performance and functional monitoring Autonomy Monitoring for energy consumption User Interface GUI Testing Context Awareness Context dependent functional and extra functional testing Adaptation Adaptation correctness New programming language New white-box and black-box testing,byte code analysis New O.S.s Compatibility and O.S testing Diversity of phones and OS Diversity coverage Touch Screen Usability, and response to screen touch MobileApplications MobileApps Apps4Mobile
  11. 11. Mobile Connectivity ❖ Peculiarities : ➢ Connectivity is one of the most peculiar and critical characteristic of mobile ➢ Different types of networks ■ Wi-Fi , 3G , LTE , Restrictive networks ➢ Connection change between different type of these networks ➢ Speed, security and reliability ➢ Interoperability : bluetooth
  12. 12. Mobile Connectivity ❖ Peculiarities : ➢ Connectivity is one of the most peculiar and critical characteristic of mobile ➢ Different types of networks ■ Wi-Fi , 3G , LTE , Restrictive networks ➢ Connection change between different type of these networks ➢ Speed, security and reliability ➢ Interoperability : bluetooth ❖ Implications on Testing : ➢ Reliability, performance, security and correct functioning strongly relies on connectivity ➢ Functional and extra functional testing has to be performed in ■ Different connection type ■ Different connectivity scenarios ■ Connectivity changes scenarios with different connection types ➢ Not ignorable rate about connectivity issues
  13. 13. Mobile Connectivity - Real World Example
  14. 14. Limited Resources ❖ Peculiarities : ➢ Mobile device resources are far away those on laptop or desktop computers
  15. 15. Limited Resources ❖ Peculiarities : ➢ Mobile device resources are far away those on laptop or desktop computers ❖ Implications on Testing : ➢ Resource usage has to be continuously monitored and avoid performance degradation ➢ Actions have to be taken in case of resource shortage
  16. 16. Limited Resources- Real World Example
  17. 17. Autonomy ❖ Peculiarities : ➢ Different mobile application may require very different energy consumption ➢ Reduced autonomy can be an issue ➢ Autonomy improvements can be a business advantage Model Standby time Internet use iPhone 6s Plus Up to 16 days Up to 12 hours on 3G,LTE, Wi-Fi
  18. 18. Autonomy ❖ Peculiarities : ➢ Different mobile application may require very different energy consumption ➢ Reduced autonomy can be an issue ➢ Autonomy improvements can be a business advantage ❖ Implications on testing : ➢ Energy consumption of mobile applications have to be evaluated through testing or continuous monitoring Model Standby time Internet use iPhone 6s Plus Up to 16 days Up to 12 hours on 3G,LTE, Wi-Fi
  19. 19. Autonomy - Real World Example
  20. 20. User Interface ❖ Peculiarities : ➢ Developers have to follow some strictly defined user interface guidelines for producing mobile applications ➢ Different screen resolutions and dimensions ➢ Changing user interface guidelines
  21. 21. User Interface ❖ Peculiarities : ➢ Developers have to follow some strictly defined user interface guidelines for producing mobile applications ➢ Different screen resolutions and dimensions ➢ Changing user interface guidelines ❖ Implications on testing : ➢ GUI testing ➢ Need to test same application on different mobile devices
  22. 22. User Interface - Real World Example
  23. 23. Context-Awareness ❖ Peculiarities : ➢ MobileApps rely on sensed data provided by context providers ➢ Deal with huge amount of inputs that vary depending on the environment and/or user actions ➢ Brightness, temperature, altitude, noise level, type of connectivity, bandwidth, neighboring devices, location, etc...
  24. 24. Context-Awareness ❖ Peculiarities : ➢ MobileApps rely on sensed data provided by context providers ➢ Deal with huge amount of inputs that vary depending on the environment and/or user actions ➢ Brightness, temperature, altitude, noise level, type of connectivity, bandwidth, neighboring devices, location, etc... ❖ Implications on testing : ➢ Testing whether application is going to correctly work on any environment and under any input is a challenge! ➢ May lead to combinatorial explosion ➢ Context specific test selection techniques and coverage have to be produced ➢ Not ignorable rate associated to contextual input
  25. 25. Context-Awareness - Real World Example ● Bluetooth problem reported by Customer side ● Problem is reproduced with only car bluetooth kit ● Working in a car to reproduce issue and fix it !
  26. 26. New Programming Language ❖ Peculiarities : ➢ Programming languages of mobile applications have been designed for supporting mobility, managing resource consumption and handling new GUIs ➢ iOS , uses UIKit framework and dynamic language ➢ Android replace java.awt and java.swing APIs with new libraries, uses Dalvik opcode compiler optimized for mobile devices ➢ New language : Swift
  27. 27. New Programming Language ❖ Peculiarities : ➢ Programming languages of mobile applications have been designed for supporting mobility, managing resource consumption and handling new GUIs ➢ iOS , uses UIKit framework and dynamic language ➢ Android replace java.awt and java.swing APIs with new libraries, uses Dalvik opcode compiler optimized for mobile devices ➢ New language : Swift ❖ Implications on Testing : ➢ Traditional structural test techniques need to be revised in order to be applied to new mobile languages ➢ Bytecode analysis tools required to be updated
  28. 28. New(and rapidly evolving) Mobile OS ❖ Peculiarities : ➢ Mobile applications run on top of new operation systems that are still only partially reliable ➢ Many are examples of applications failing due to operation systems problems ➢ New versions of mobile operating systems are frequently released ➢ New versions are not backward compatible with previous version
  29. 29. New(and rapidly evolving) Mobile OS ❖ Peculiarities : ➢ Mobile applications run on top of new operation systems that are still only partially reliable ➢ Many are examples of applications failing due to operation systems problems ➢ New versions of mobile operating systems are frequently released ➢ New versions are not backward compatible with previous version ❖ Implications on testing : ➢ Failures can not be ascribable to mobile applications ➢ Mobile applications have to tested on all versions of mobile operating system
  30. 30. New(and rapidly evolving) Mobile OS
  31. 31. New Mobile OS - Real World Example ● App Transport Security came within iOS 9 ● Enforce secure network connections ● Minimum TLS version updated
  32. 32. Diversity ❖ Peculiarities : ➢ Hundreds of different mobile devices on market ■ produced by different vendors ■ different software features and hardware components ➢ Specially, sensors are typically calibrated differently so that two different phones running the same application, in the same environment, may compute different outputs ➢ Supported device list : roughly 1228 https://support.google. com/googleplay/answer/1727131?hl=en
  33. 33. Diversity ❖ Peculiarities : ➢ Hundreds of different mobile devices on market ■ produced by different vendors ■ different software features and hardware components ➢ Specially, sensors are typically calibrated differently so that two different phones running the same application, in the same environment, may compute different outputs ➢ Supported device list : roughly 1228 https://support.google. com/googleplay/answer/1727131?hl=en ❖ Implications on Testing : ➢ Testing techniques for maximizing the diversity coverage while minimizing the devices under test have to be devised
  34. 34. Diversity - Real World Example
  35. 35. Diversity - Real World Example
  36. 36. Diversity - Real World Example Default Video resolution : 360 X 480 Exynos hardware which uses in Samsung devices causes problem while doing hardware video decoding (V8 codecs) Wrong resolution : 368 X 480
  37. 37. Diversity - Real World Example -2 ● Highly critical no media issue reported by customer in Japan ● Only reproducible on Sony Ericsson Xperia SO-02E ! ● Device is only available in Japan ! ● Device rejected due to Turkey’s import policy !!
  38. 38. Touch Screens ❖ Peculiarities : ➢ Touch screens are main means for inputting user data into a mobile application ➢ The system response time to a touch strongly depends on the device resource utilization ➢ May become slow in certain circumstances like busy processor ➢ Lots of reported issue related with lack of responsiveness of touch screens
  39. 39. Touch Screens ❖ Peculiarities : ➢ Touch screens are main means for inputting user data into a mobile application ➢ The system response time to a touch strongly depends on the device resource utilization ➢ May become slow in certain circumstances like busy processor ➢ Lots of reported issue related with lack of responsiveness of touch screens ❖ Implications on Testing : ➢ Testing techniques have to be introduced to test the touch screen functioning under different circumstances and different mobile devices ➢ Resource usage, processor load, memory load, etc..
  40. 40. Touch Screens - Real World Example
  41. 41. Challenges on Testing Mobile Applications ❖ Testing process ➢ Test selection ➢ Test execution ❖ Testing artifacts ➢ Structural ➢ Functional ❖ Testing levels ➢ unit test ➢ integration test ❖ Type of test ➢ Performance & Reliability testing ➢ Memory & Energy testing ➢ Security testing ➢ GUI testing ➢ Product line testing
  42. 42. Testing Process ❖ Test Selection ➢ Challenge : Unpredictability and high variability of inputs ➢ Potentials & automation : new testing criteria are required to provide guidelines,rules, strategies by which mobile test cases are selected so to maximize coverage in case of unpredictable and variable contextual input ➢ SOTA : Contexts that have a higher probability to generate faults are selected but coverage criteria for context-aware inputs seem to be still missing. ❖ Test Execution ➢ Challenge : Contextual information leads to challenge on how to execute test cases including rich contextual inputs ➢ Potentials & automation : New capture-and-replay automated techniques can be realized for executing contextual input selected during the test selection phase ➢ SOTA : Android Robotium is an open source tool that enables the automated and black box test execution of mobile application
  43. 43. Testing Artifacts ❖ Structural (code base) ➢ Challenge : Mobile app languages add some specific constructs for managing mobility, sensing and energy consumption. ➢ Potentials & automation : new coverage criteria shall be though as a way to consider at best the new mobility , sensing and energy constructs. New bytecode analysis tool ➢ SOTA : The tool JaBUTi/ME is proposed ❖ Functional ➢ Challenge : Mobile application functional testing requires to specify both app and environment ➢ Potentials & automation : State-based approaches to model different execution modes (e.g low battery, meeting, and flying mode ) ➢ SOTA : Android Robotium, MonkeyRunner
  44. 44. Testing Levels ❖ Unit Testing ➢ Challenge : no specific challenge seems to apply to unit testing of mobile apps ➢ Potentials & automation : there is the need and potential to realize automated testing tools supporting and enabling the specification of contextual values and context-based coverage ➢ SOTA : JUnit / XCTest ❖ Integration Testing ➢ Challenge : Inter- and intra-application information flow must be followed ➢ Potentials & automation : Existing integration testing approaches have the potentials to be adapted for mobile applications ➢ SOTA : Mature research on integration testing of mobile applications is still missing today
  45. 45. Type of Testing ❖ Performance & Reliability testing ➢ Challenge : depends on mobile resources, on the device operational mode, connectivity quality and variability ➢ Potentials & automation : new techniques have to explicitly consider characteristics related to changing context and different devices. Runtime analysis techniques can also adopted to monitor the resources and connectivity state and prevent performance degradation ➢ SOTA : Berardinelli performance analysis tool ❖ Memory & Energy testing ➢ Challenge : memory leaks, active processes may reduce device battery ➢ Potentials & automation : Metrics for estimating energy and memory consumption could be devised so to automatically predict memory leak and energy outage ➢ SOTA : iOS Instruments tool
  46. 46. Type of Testing ❖ Security testing ➢ Challenge : Security is particularly relevant due to the mobility of the device into networks with different security levels. A trojan might infact access to personal date, private networks and private contextual information ➢ Potentials & automation : traditional security testing approaches shall be revised so to keep in consideration contextual factors ➢ SOTA : The authors analyze the threats Android applications pose to the security and privacy of enterprise and propose several approaches for defending enterprises against security risks
  47. 47. Conclusion ❖ The need of automation in testing mobile applications is exacerbated by two orthogonal aspects: ➢ Cost of testing : Automation is certainly among the most important means for keeping the cost of testing low, while guaranteeing and adequate degree of dependability. ➢ Testing through the layers : current bugs are due to interoperability problems that exist today among the application, application framework, operating system, and hardware (sensoring) layers. The need of testing automation towards all the different layers, and able to clearly separate application-level failures from application framework or OS failures.
  48. 48. References ❖ Software Testing of Mobile Applications : Challenges and Future Research Directions ➢ http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=6228987 ❖ Mobile Application Testing - Challenges and solution approaches through automation ➢ http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=6496451 ❖ statistica.com ➢ http://www.statista.com/topics/1002/mobile-app-usage/ ❖ developer.android.com ❖ developer.apple.com

×