Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Mobile Testing Challenges


Published on

Mobile software gets more and more capable and complex in recent years and quality becomes more distinctive to create successful apps. We'll see the answers of these questions in this presentation; Is mobile software so different from traditional ones, so to require different and specialised new testing techniques, what are the new challenges on testing mobile applications and what are the current solutions for them? We'll walk through some essays and real world examples to understand the challenge of mobile testing.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Mobile Testing Challenges

  1. 1. Mobile Software Testing Challenges Busra Deniz Akin @busradeniz
  2. 2. About Me Busra Deniz Akin busradeniz busradeniz busradeniz Mobile Software Consultant Scrum Master and Mobile Software Developer 
 @Netas,@Dotto Co-Organizer @GDGIstanbul and @WTMIstanbul Master Student @UniBogazici
  3. 3. What we’ll talk about Pecularities Implications Connectivity Diversity Autonomy New OS releases Context Awareness Energy Consumptions Reliability Perform ance Compatibility Usability
  4. 4. Are mobile applications so different from traditional ones? Let’s start What are the new challenges on testing mobile applications? What are solutions in industry?
  5. 5. Mobile World Statistics 2019 5.01 2013 4.01 2014 4.23 2015 4.43 2016 4.61 2017 4.77 2018 4.93 Mobile phone users worldwide (in billions)
  6. 6. Mobile World Statistics 2015 179,628 2016 224,801 2017 268,692 2009 2,516 2010 4,507 2011 21,646 2012 63,985 2013 102,062 2014 138,809 Number of mobile app downloads worldwide (in millions)
  7. 7. Mobile World Statistics 2017 76,52 2011 8,32 2012 18,56 2013 26,68 2014 34,99 2015 45,37 2016 58,21 Worldwide mobile app revenues (in billion U.S. dollars)
  8. 8. HealthPayments Mobile in Critical Domains Banking
  9. 9. Mobile Application Applications Apps For Mobile MobileApps
  10. 10. Connectivity
  11. 11. Connectivity • One of the most critical characteristic of mobile
 • Different types of networks • 3G, WiFi , LTE , Restrictive networks 
 • Connection change between different type of these network
 • Speed, security and reliability Peculiarities
  12. 12. Connectivity Real World Example
  13. 13. Connectivity • Extra functional testing has to performed in • Different connection type • Different connectivity scenarios • Connectivity changes scenarios with different connection types
 • Reliability, performance, security and correct functioning strongly relies on connectivity • Not ignorable rate about connectivity issues 
 Implications on Testing
  14. 14. Autonomy
  15. 15. Autonomy • Different mobile application may require very different energy consumption 
 • Energy consumption can be a big problem 
 • Energy consumption improvements can be business advantage 12 hours up to 16 days iPhone 6s Plus Peculiarities
  16. 16. Autonomy Real World Example
  17. 17. Autonomy • Energy consumption have to be evaluated through testing and continuous monitoring
 Implications on Testing 68% Phone y 33% Tablet x 10% Phone t 33% Tablet z
  18. 18. Limited Resources
  19. 19. Limited Resources • Mobile resources are away from those on laptops and computers • CPU • Memory VS MobileComputer Peculiarities
  20. 20. Limited Resources Real World Example
  21. 21. Limited Resources • Resource usage has to be continuously monitored • Avoid performance degradation • CPU Usage, Memory Leaks Implications on Testing Phone XPhone Y v1.0 v1.1 v1.2 v1.3 v1.0 v1.1 v1.2 v1.3
  22. 22. Context Awareness
  23. 23. Context Awareness • Mobile applications rely on sensed data provided by context providers
 • Deal with huge amount of inputs : • depending on the environment • depending on user actions
 • Neighbouring devices, health data, location, light, pressure, temperature, altitude, etc… Peculiarities
  24. 24. Context Awareness Real World Example • Bluetooth problem reported by customer side 
 • Problem is reproduced with only car bluetooth kit 
 • Solution : working in a car to reproduce issue and fix it !!!
  25. 25. Context Awareness Implications on Testing • Testing whether application is going to correctly work on any environment and under any input is a challenge! 
 • May lead to combinatorial explosion !
 • Context specific test selection techniques and coverage have to be produced
 • Not ignorable rate associated to contextual input
  26. 26. User Interface
  27. 27. User Interface • Need to follow some user interface guidelines for producing mobile apps
 • Different screen resolutions and dimensions 
 • Changing user interface guidelines Peculiarities
  28. 28. User Interface Real World Example
  29. 29. User Interface • UI Tests • Need to test same application on different mobile devices Implications on Testing User
  30. 30. New Programming 
  31. 31. New Programming Languages • Programming languages of mobile applications have been designed for supporting mobility 
 • Managing resource consumption
 • Handling UIs
 • New compilers : Dalvik , ART 
 • New languages : Swift, Kotlin Peculiarities
  32. 32. New Programming Languages • All support frameworks required to be updated • Bytecode analysis tool • Test frameworks • Package managers, all supportive libraries …
 • Learning curve Implications on Testing
  33. 33. Rapidly evolving 
  34. 34. New Mobile OS • New versions of mobile operating systems are frequently released 
 • New versions are not always backward compatible with previous version 
 • Updating OS is not controllable
 • Many of mobile apps failing due to operation systems problems
  35. 35. Challenges New Mobile OS
  36. 36. New Mobile OS Real World Example • App Transport Security came within iOS 9
 • Enforce security network connections minimum TLS version updated
  37. 37. New Mobile OS • Failures can not be ascribable to mobile applications
 • Mobile applications have to tested on all versions of mobile operating system 
 Implications on Testing
  38. 38. Diversity
  39. 39. Diversity • Hundreds of different mobile devices on market • produced by different vendors • different software features and hardware components
 • Specially, sensors are typically calibrated differently so that two different phones running the same application, in the same environment, may compute different outputs
 • Supported device list : roughly 13000 
  40. 40. Diversity Real World Example Default Video resolution : 360 X 480 
 Exynos hardware which uses in Samsung devices causes problem while doing hardware video decoding (V8 codecs)
 Wrong resolution : 368 X 480
  41. 41. Diversity Real World Example • Highly critical no media issue reported by 
 customer in Japan
 • Only reproducible on Sony Ericsson Xperia
 SO-02E !
 • Device is only available in Japan ! 
 • Device rejected due to Turkey’s import policy !!
  42. 42. Diversity • Testing techniques for maximising the diversity coverage while minimizing the devices under test have to be devised 
 Implications on Testing
  43. 43. Challenges on Testing Mobile Applications - Unit Tests 
 - Integration Tests Testing Levels - Performance and Reliability Testing - Memory & Energy Testing - Security Testing - UI Testing Type of test - Test selection - Test execution Testing Process
  44. 44. • Test Selection • Unpredictability and high variability of inputs • New testing criteria are required to provide guidelines, rules, strategies by which mobile test cases are selected so to maximise coverage in case of unpredictable and variable contextual input
 • Test Execution • Contextual information leads to challenge on how to 
 execute test cases including rich contextual inputs • New capture-and-replay automated techniques can 
 be realised for executing contextual input selected during 
 the test selection phase Testing Process
  45. 45. • Unit Tests • No specific challenge seems to apply to unit testing of mobile apps • There is the need and potential to realise automated testing tools supporting and enabling the specification of contextual values and context-based coverage 
 • Integration Tests • Inter- and intra-application information flow must 
 be followed Testing Levels
  46. 46. • Performance and Reliability Testing • Depends on mobile resources, on the device operational mode, connectivity quality and variability
 • New techniques have to explicitly consider characteristics related to changing context and different devices. 
 • Runtime analysis techniques can also adopted to monitor the resources and connectivity state and prevent 
 performance degradation Type of Testing
  47. 47. • Memory and Energy Testing • Memory leaks, active processes may reduce device battery
 • Metrics for estimating energy and memory consumption could be devised so to automatically predict memory leak and energy outage
 • UI Testing • Touch screens are main means for inputting user data 
 into a mobile application 
 • Lots of reported issue related with lack of 
 responsiveness of touch screens Type of Testing
  48. 48. Take Aways • Mobile is so different from traditional ones
 • Automation is certainly among the most important 
 means for keeping the cost of testing low, while 
 guaranteeing and adequate degree of dependability 
 • There are some solutions for mobile specifics 
 • New testing techniques, guidelines, rules, strategies 
  49. 49. Thanks for coming! #dfist #gdg #gdgistanbul @busradeniz