Mobile testing offers numerous challenges and the potential for massive scope creep with so many device combinations and tool choices available. We can’t afford to test everything so we make decisions on scope. We have learn to use risk as part of the the decision making process: business risk, and technical risk. In this talk, Alan will describe his experiences of hands-on testing of mobile native applications and mobile web applications, and the application of technical and risk-based thought processes in deciding what, and how, to test. Expand your knowledge of mobile technology and improve your mobile testing process. Learn to apply important techniques like; viewing traffic via proxies and Wireshark, and using local wi-fi hotspots. As well as multiple ways to screen capture for defect reporting, how to use emulators and browser developer tools, and how to assess the risk that each new tool adds to your process. Presented by Alan Richardson at the EuroSTAR Mobile Testing Deep Dive 2015 in Maastricht http://www.eviltester.com

1. Technical Mobile Tes.ng
Risk, Issues and Experiences
Alan Richardson
- @eviltester
EvilTester.com

2. What we learned when we tested mobile
• Decisions we made
• Pain we went through
• What we did about it

3. What I like about tes@ng mobile
@eviltester

4. What is hard about mobile tes@ng
• Ergonomics
• Observing
• Repor.ng
• Manipula.on

5. Phone Experience
@eviltester

6. Find workarounds
@eviltester

7. Pay aEen@on to pain to improve process
• Gain some experience
• Pay aBen.on to pain
• Resolve pain to improve process
• Status Quo != Dishwasher

8. Workarounds add risk
• Adopt workaround, and mi.gate the risk
• Communicate the risk
• Accept risks through knowledge not fear (where
possible)
@eviltester

9. Some risks must be accepted
Everyone should
have the
opportunity to
aspire to Rock
Godhood
@eviltester

10. Mobile Technical and Risk Based Tes@ng
• Our test approaches are oNen built around fear
mi.ga.on
– Lots and lots of devices
– Because we don’t build for device compa.bility
– We caveat for a subset of devices

11. Na@ve and Web
• Which is worse for compa.bility?

12. Test Like Normal Users?
• Default Browser? Dolphin, Chrome
• LastPass Password Manager
• Auto-Comple.on Off
• Alterna.ve Keyboards – hacker’s Keyboard

13. Automa@ng?
• Automa.ng mobile is hard
• Strategic choice

14. Automa@ng Tac@cally
• Screenshots for different screen sizes
• Server based responses from mobile device headers
• Not automated on device
• Target Technical Risk Tac.cally

15. The challenge of observability
• I hate tes.ng blind
• Web Tes.ng more observable?
• Proxies and Sniffers

16. Proxy Tools found Preload caching issue
• JavaScript library for pre-load caching
• Observing traffic through proxy saw mul.ple loading

17. Wifi Hoops and Roundabouts
• Environmental Security
• Tes.ng on wi-fi
• Laptop Hotspots
• Wireshark for na.ve via wi-fi hotspot

18. Proxy Benefits
• Isolate device from server
• Feed in new JS and CSS prior to release
• Trigger error situa.ons on device without
messing with server
• Mul.ple scenarios configured by data fed
to device by proxy

19. Ergonomics
paulbishopbooks.com/2013_01_27_archive.html
My hands have not
evolved to use a
mobile phone

20. Cheap Ergonomic Solu@ons
• Tablet stands
• Stylus
• Keyboards
• Risk?
– Fear mi.ga.on, use phone too
– Even though this is built in OS func.onality

21. Phones make themselves hard to test
• Sim card needed to
connect to wi-fi
• Enter the ‘porta £1 sim
card collec.on’ ™

22. Repor@ng
• Informa.on is trapped on the mobile device
• Get info off mobile and on to desktop

23. Olden days inefficiency
Use phone to take
photo of screenshot
of other phone for
uploading to desktop
for bug repor.ng

24. Screenshots from phone to desktop
• AirPlay
• VNC
• Evernote
• Dropbox
• Email
• Each tool adds risk?

25. Emulators and Simulators
• When to use Emulators?
• Risk of Browser Dev Tools for mobile web?

26. Non-Technical Risks
• Phones go walking
• Start with secure library system
• End up with cardboard box filled with devices

27. Lessons Learned
• Don’t bring in your own kit
• Never lend your kit
• Buy 2nd hand/refurbished kit

28. The right kit is important
• Do users charge their phones when using them?
• Random chars on old android

29. List of how to test on each device
• Each device is different
• List the ‘shortcuts’ and lessons learned
– How to take a screenshot
– What tools to use

30. Future is geTng beEer
• AirPlay
• Chromecast
• Miracast
• Remote debug
• Bigger phones

31. Risk Management
• Iden.fy
• Mi.gate
• Accept
• Communicate

32. Final Notes
• Tes.ng on device is a risk
• Iden.fy Technical Risk
– Target to reduce plakorm tes.ng scope
• If we were really serious…

33. Alan Richardson
@EvilTester
www.JavaForTesters.com
www.SeleniumSimplified.com
www.EvilTester.com
www.CompendiumDev.co.uk