Slides from Paris QA meetup #1 in 10/2014 Quick overview of the difficulties of conducting manual tests on mobile and how to solve them.

1. QA tools / Mobile testing

2. Basic Desktop tools
 Bug tracker (jira, mantis ..)
 Bug tracker hooks (browser extension, screen capture ..)
 Communication tools (emails, chat, wiki ..)
 Test plans (excel, testrail, google doc ..)
 Video capture (quicktime ..)
 Image editing (paint ..)
 Network monitoring (shark)
 Others
 Dbgview
 Process explorer
 Text editors
 ..
04/04/2013 2

3. Advanced needs
We have more advanced needs, such as:
 Kill / relaunch application easily
 « Autofill » strings that we type hundred of times daily
 Set a .ini file with some specific attributes to provide meaningfull logs to devs
 Easily find users profiles on local machine
 Clean local settings to simulate a « fresh install »
 Get some informations on any given test account
 Premium status and history
 A/B testing test cell
 Data transactions
 ..
 Act on the above account’s details
 Being able to simulate a premium purchase, just as a user would do
04/04/2013 3

4. Problems
 Developpers are busy working on actual user facing problems / features
 Developpers are busy developping
 QA is busy QAing
 It’s never the right moment
 It’s not so important after all, is it ?
 No one cares but you
04/04/2013 4

5. How we got it done
 Use our talented QA people to do a windows tool for all client side needs
 Accessible languages: C#, python, nodejs
 Fullfills the following needs
 .ini
 Kill / relaunch
 Profiles
 Autofill strings
 User support needed a server side tool too
 When problem arised, everyone quickly understood how critical it was to
have a real server side tools
04/04/2013 5

6. It’s all great, but now that I have to test on an iPhone / Android ..
 There are many devices to test (Android screen sizes ..)
 It’s difficult to get the build on the device
 How do I take a screenshot and add it to a bug ?
 How do I make a video of a bug ?
 Tests take 5x the time because of touch & tiny keyboard
 Someone sent me a link on my desktop, I need it on the test device. How
to send it quickly ?
 I don’t have access to the file system, and thus cannot change some
configuration files or reset local settings
 How to test play/apple store payments ?
 How do I simulate a poor connexion ?
04/04/2013 6

7. There are ways ..
 Use VMs (genymotion) or simulators
 Some bugs will be VM only and some will be real device only ..
 Mutualize devices with other teams (extQA)
 Hockeyapp, testflight to distribute the application
 Beware of the auto update integration in the app. This can cause bugs that no user will ever have
 Dropbox to sync screenshots to desktop in a breeze
 If working with shared devices, can bring a lot of screenshot / photos from other teams
 Use the same email account on all devices, and send links to this address
 Video
 Record a video with another device
 Or use tools such as reflexion (sends iphone screen to a desktop client)
 Code in your app some test specific behaviors when using test accounts
 Beware not shipping a test enabled app !
 Beware not missing a bug because of test mode !
 Apple has some built in dev tools to simulate poor connexion
04/04/2013 7

8. But not for everything
 Apple and Android both feature test payments, but with limitations
 On android you need an actual google account with a valid CC saved. This
prevents externalQA from testing purchases
 On iOS you might make it work if using the correct app ID. But test purchases
won’t work on the live build, thus you cannot reproduce a live payment bug
 iOS is a closed system
 Testers are in full black box mod. But not able to replicate real users
conditions
 You may implement a debug menu in your app, but can be tedious to dev,
maintain and use
04/04/2013 8

9. Conclusion
 We are yet to find better solutions to ease the tests on mobile
 We are very curious as to how it goes on your side, are you facing the
same problems ? Please react!
04/04/2013 9