Mobile App Testing by Mark Wilson

1. © Planit Software Testing Mobile App Testing & Other Considerations Presenter: Mark Wilson

2. © Planit Software Testing Our knowledge is your power Software Testing Consulting, Advisory, Delivery, Managed Services, Training

3. © Planit Software Testing Why is Mobile Computing Important? People worldwide now spend more time on their smartphones than on traditional PCs. Smartphone sales surpassed PC sales in 2011.

4. © Planit Software Testing 64% 15 - 65 The Mobile Landscape In New Zealand Have a smartphone now

5. © Planit Software Testing 90% 15 - 65 The Mobile Landscape In New Zealand Will have a smartphone by 2018

6. © Planit Software Testing 44% 2013 The Mobile Landscape In New Zealand Have a tablet at home now

7. © Planit Software Testing 78% 2018 The Mobile Landscape In New Zealand Will have a tablet by 2018

8. © Planit Software Testing Apps vs Web

9. © Planit Software Testing The Future Of Mobile Apps

10. © Planit Software Testing What Is Mobile Computing Used For? • Apps continue to grow • Gaming and Social Media are over 60% of apps usage

11. © Planit Software Testing How Successful Are most Mobile Apps? Apps on our smart phones • Average 25-26 apps • 1 in 4 apps downloaded are never used • On average 22% of Apple users who download an app only use it once. • On average only 34% of Apple users who download an app will use it more than 11 times. • Over 1 million applications are available from the app store alone

12. © Planit Software Testing What Spurs Mobile App Growth? Mobile app usage worldwide. What will be the next big thing?

13. © Planit Software Testing Do Mobile Apps Have Problems? Have you had problems with a mobile app within the last 6 months? NO 44% YES 56%

14. © Planit Software Testing Types Of Problems Encountered? •The mobile app crashed/froze/ displayed an error •The mobile app was slow to launch •The mobile app would not launch •The mobile app didn’t function as expected •None of these

15. © Planit Software Testing What The Customer Expects •Things to work first time •They expect things to work fast •They expect things to be intuitive •They expect instructions to be clear and self explanatory •Your app generally has to be usable by people with differing abilities •Most apps don’t come with a 1-2 day training course! It’s a Web App not SAP

16. © Planit Software Testing So Why Do Mobile Apps Fail?? Customers Can’t Make It Work!! 1. They have problems installing 2. They don’t run properly 3. They freeze or lock

17. © Planit Software Testing The App Won’t Install, Won’t Work Or Freezes! Why is this? 3 Answers: • They have not been functionally tested properly! • They have not been useability tested on that platform combination. • They have bugs which were not found in testing. • This application documentation clearly states that this device in this configuration is not supported. People Love to Complain

18. © Planit Software Testing Scope Targeted Platforms What platforms will you target and test your application on? • Operating Systems? • Browsers ? • Hardware?

19. © Planit Software Testing Scope Targeted Platforms? Each one has multiple versions of the operating system

20. © Planit Software Testing Targeted Platforms?

21. © Planit Software Testing Targeted Platforms? What Devices? Phones – Iphone 3-6, Galaxy S3-S5, Nexus 4-7 Blackberry ??, HTC ?, Nokia windows phone, Huawei, Redme + hundreds of clones etc. Tablets – IPad 1-4, Galaxy Note 1-4, Windows Surface ??. Le-Novo, Sony, Acer, Asus, Amazon Kindle, HP, IPOD Touch + Hundreds of clones. Different screen sizes Different resolutions Different memory Different processing power Different operating systems Different quality antennas

22. © Planit Software Testing 153 Targeted Platforms?

23. © Planit Software Testing Lessons Learnt – Devices Manage your platform scope • How many devices, operating systems, versions? o iOS? o Android? o Windows Phone? o Blackberry, Symbian, . . .? o Smart Phones, Tablets, . . . ? • Your systems future? o Device & OS fragmentation Requirements

24. © Planit Software Testing Problems with Look Feel and Function 1. The application doesn’t do what the customer thinks it should 2. The application is just too complicated or confusing 3. It doesn’t look very good or is isn’t nice to use (Subjective but unfortunately customers are)

25. © Planit Software Testing The Application Is Not Intuitive • Make your app as clear and easy to use as possible. • Keep a common look and feel throughout the application. • Get your kids to trial it, then get you mum and gran to have a play. • There are guides around usability – look at them. Government have usability and accessibility guidelines which are being enforced more rigorously. • If you are targeting a bigger audience then look at the issues around accessibility. http://blindfoundation.org.nz/learn/information-for- businesses/accessible-websites is a good place to start.

26. © Planit Software Testing Lessons Learnt – Look/Feel/Function Have a clear vision for the application • Put together clearly defined requirements (‘Just like our website’ is not good enough!!) • Target your market demographic • Look at what is successful in that market • Look and feel can be as important as function • But look and feel won’t keep your customers happy if the application doesn’t do what it’s supposed to Requirements Requirements

27. © Planit Software Testing Be Prepared To Speed Up First to market is a big advantage!

28. © Planit Software Testing Focus on time – everything moves quicker on mobile projects • The market is moving incredibly fast • The technology is moving very rapidly • The fashions and trends are moving fast The Application Takes Too Long To Get To Market http://williesun.com/tag/ramble- away/?doing_wp_cron=1402289455.7205860614776611328125 Remember you are better off being second to market with a product that works than being first with one that doesn’t

29. © Planit Software Testing A Newer, Quicker Way Of Working! • Manage your stakeholders • Get marketing and the business involved early • Gear up for more releases

30. © Planit Software Testing Focus on time Look at continuous integration Automated functional regression testing Automated regression performance testing Automated penetration testing/vulnerability scanning Service virtualisation to allow parallel development Farm out the bulk of your device usability testing Look To Reduce Cycle Times – Not Quality!

31. © Planit Software Testing Prepare For Regular Release Cycles • Plan for your ongoing release cycles o Business will see new opportunities o Expect Public APIs to change (e.g. Google Maps, TradeMe) o Technologies will change o Taste and fashion change • How will you deploy future releases? o Plan for availability of infrastructure

32. © Planit Software Testing Lessons Learnt – Be Prepared To Speed Up • Be prepared for small projects o With frequent follow-ons • Build your team early o Get the right mobile expertise in at the start • Connect early to the business stakeholders • Get your scope nailed early o Especially what devices and OSs you will support • Start small o Plan for measured rollouts • Plan your device management and your security early Requirements Requirements Requirements!

33. © Planit Software Testing Test Improvement Consider: • Testing smarter • Device emulation tools • Test automation

34. © Planit Software Testing Test Smarter Consider • Do you need to run full functional regression tests against every device combination? • Could you use Bluetooth keyboards to reduce fatigue? • Could you use an adaptor and a full size screen?

35. © Planit Software Testing Device Emulation Pros: • Makes many device combinations available quickly and affordably • Testers use a proper desktop to drive the application • Defect creation is faster using cut and paste, screen shots etc. Cons: • It’s a step away from production like. You have introduced another layer that can create and/or mask problems.

36. © Planit Software Testing Test Automation Consider • You may need test automation to enable you to perform regression testing on each release in a timely manner • Do an ROI calculation before you invest in this area • If you are going to invest in this area do it early

37. © Planit Software Testing Lessons Learnt – Test Improvement Emulation can speed up testing but is not a substitute for testing on the actual device. Look in to test automation early. Devices: • Track carefully – they can go walkies . . . • Testing infrastructure – rent or buy, Planit Mobile Test Lab • Bluetooth keyboards can save the thumb • Screen adaptors can save eyes • Invest in antibacterial wipes

38. © Planit Software Testing Users Expectation Mobile Apps vs Mobile Web Performance is important in the mobile world

42. © Planit Software Testing The Apps Are Too Slow! • Performance is a really big deal. • How do you test mobile performance? • What is good performance? • What is the total impact on the performance of your system?

43. © Planit Software Testing The Apps Are Too Slow! We are not fully in control • The users location determines the strength of signal they have at the time • The network provider and plan they use affects performance • The base performance of their device can vary • There could be other applications running on their device at the same time As a general rule – we cannot control these!

44. © Planit Software Testing The Apps Are Too Slow! Mobile Networks System Under Test F 5 F 5 APP APP APP APP Legacy 1 Address FWK Account SystemInternet

45. © Planit Software Testing The Apps Are Too Slow! System Under Test F 5 F 5 APP APP APP APP Legacy 1 Address FWK Account System x 220 Staff x 10 Manager x 20 Accounts Mobile Networks Internet

46. © Planit Software Testing The Apps Are Too Slow! To get production like response times you need: • A production like environment to test against • To create a production like load on all components • A production like test data selection If it isn’t production like then you introduce the risk of missing a problem or creating a problem! Also, the response times you see during testing most likely wont be representative of production.

47. © Planit Software Testing The Apps Are Too Slow! You can’t control what people have running on their devices, where they use it or who their provider is! So keep it mean and lean!

48. © Planit Software Testing The Apps Are Too Slow! What is good performance? 1 second response time? 2 second response time? Whatever enables your customers and staff to work efficiently without getting frustrated.

49. © Planit Software Testing Lessons Learnt – Performance • Think about performance testing early • Consider the whole system not just the mobile component • As production like as possible: o Environment, hardware, configuration, application versions o Data populated to production like levels, test cases selected in a production like way o Production like transaction mix and durations o Production like load on all system components. • Look to test at lower levels earlier e.g. web services. • Look to introduce regression performance testing if appropriate. • Be aware of the performance of any third party services such as payment gateways and try to ensure you can see the performance costs in your logs

50. © Planit Software Testing Security Issues? 90% of top mobile banking apps have security issues (IOActive, Jan 2014) 90% of mobile apps have at least 1 vulnerability (ZDNet, Nov 2013)

51. © Planit Software Testing Security Security Security The top 10 security issues faced by Mobile devices: 1. Lack of passwords 2. Lack of 2 factor authentication 3. Wireless transmissions not being encoded 4. Mobile devices are susceptible to malware 5. Mobile devices often don’t have security software 6. OS is out of date and doesn’t contain latest security patches 7. App software is often out of date 8. Unsecure ports on devices can be open 9. Mobile devices can be modified, jail broken, or rooted which can allow unauthorised access 10. Unsecure Wi-Fi can allow access to the device. Consider BYOD

52. © Planit Software Testing Lessons Learnt - Security • Don’t forget mobile security • Address BYOD o Strategy to manage apps on devices o Security policies o User agreements www. webhostinggeeks.com

53. © Planit Software Testing Mobile Projects Growing in NZ • Telcos • Banks • Insurance • Media • Government • Entertainment

54. © Planit Software Testing How Our Projects Are Changing Business changes lead to new projects • Mobile projects are increasing o In quantity o In business importance o And in speed – “First-mover advantage”

55. © Planit Software Testing Summary • Growing business pressures on schedule • The user / customer experience is critical • Mobile projects are increasing o In quantity, in business importance, and in speed • Lessons Learnt o How many devices, versions? o Define requirements o Involve the business early o Involve testing early o Device management

56. © Planit Software Testing Mark Wilson Principal Consultant mwilson@planittesting.co.nz +64 27 367 1939 Lessons Learnt In Managing And Delivering Mobile Technology Projects Contact Planit www. planittesting.co.nz +64 4 815 8199