Outside the box: User-testing context-aware mobile technology in the field – a case study


Published on

This was a talk given at the second meeting of Brum Interaction Design Assocation. It discusses the method (but not so much the findings) of user-testing a GPS Trail application on a mobile device. Some videos and audio may not work, apologies.

Details also at: http://brumixda.wordpress.com/2009/09/04/outside-the-box-user-testing-context-aware-mobile-technology/

Published in: Technology, Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Outside the box: User-testing context-aware mobile technology in the field – a case study

  1. 1. Outside the box User -testing context-aware mobile technology in the field – a case study <ul><li>Jerome Turner </li></ul><ul><li>The product </li></ul><ul><li>Planning </li></ul><ul><li>The user-tests </li></ul><ul><li>Analysing results </li></ul><ul><li>Lessons learned: testing mobile apps in the field </li></ul><ul><li>(oh yes, the results) </li></ul>
  2. 2. The product
  3. 3. Content screens
  4. 4. Planning <ul><li>Main problem – how do we record and observe mobiles? </li></ul><ul><li>One solution, fixed in lab conditions: </li></ul>Validconcept.com
  5. 5. <ul><li>Another solution, Liliput. In the field but intrusive: </li></ul>Mobile observation http://portal.acm.org/citation.cfm?id=1240907
  6. 6. <ul><li>User-lab’s kit to date, far too cumbersome </li></ul><ul><li>What are we interested in recording and at what cost? </li></ul><ul><li>Screen capture won’t record hand/stylus movement. </li></ul><ul><li>Is visual recording more useful than notetaking? </li></ul><ul><li>Do we record all screen / mobile activity if it will greatly affect the user’s experience / usage? </li></ul><ul><li>Compromise: use one discrete hat cam to record as much video/audio as possible. </li></ul><ul><li>NB Interested in broad experience, not just around device </li></ul>
  7. 7. Hat cam
  8. 8. <ul><li>Main difference, no live observer. </li></ul><ul><li>Tests should still be structured </li></ul><ul><ul><li>Recruit participants, for timed sessions </li></ul></ul><ul><ul><li>Use script and task to keep schedule and ensure equal treatment throughout day </li></ul></ul><ul><ul><li>Use demographic form, tweaked to suit the tests </li></ul></ul><ul><ul><li>Use a ‘log sheet’ to take notes whilst facilitating </li></ul></ul><ul><ul><li>End of session, use </li></ul></ul><ul><ul><ul><li>randomised word list to tick </li></ul></ul></ul><ul><ul><ul><li>questionnaire, responses to the product </li></ul></ul></ul><ul><ul><ul><li>understanding of some technical terms </li></ul></ul></ul><ul><ul><ul><li>incentive sign-off sheet </li></ul></ul></ul>Planning – from lab to the streets
  9. 9. <ul><li>Usually, would test product first, establish expected issues during sessions, and thus make log list. </li></ul><ul><li>Here, facilitator made log list based only on experience of building product / expectations of technology i.e. many tech errors included </li></ul><ul><li>Also included boxes for other observational logging… </li></ul><ul><li>… and space for comments and notes. </li></ul>Planning – facilitator’s log sheet
  10. 10. <ul><li>Through external recruiters. </li></ul><ul><li>Requirements sent without wanting to give too much away </li></ul><ul><li>They should be between 18 and 70 years of age. </li></ul><ul><li>They should have some experience of using a PDA, iPhone, Blackberry-style phone, smartphone or other similar digital mobile device. </li></ul><ul><li>They will need to be on their feet with me for potentially a full hour, walking around the centre of Birmingham (at a leisurely pace). </li></ul><ul><li>They will need to be comfortable with wearing a baseball style cap with a camera fitted that records their ‘point of view’ and voice. It doesn’t record their face. (if we really have trouble finding people for this, this requirement can be removed for a couple of people). </li></ul><ul><li>We need 8 people for Thursday and 8 for Friday. Sessions on each day to start at 9.00, 10.00, 11.00, 12.00, 13.30, 14.30, 15.30, 16.30 </li></ul><ul><li>They will receive £30 in cash on the day. </li></ul><ul><li>It would be helpful if they weren’t people booked for us before, so they don’t know me. </li></ul><ul><li>They should dress expecting to be outside for the full hour, not under cover. E.g. Bring a brolly! </li></ul>Recruiting
  11. 11. <ul><li>Meeting users – make sure you have each others’ mobile numbers and have told them what you’ll be wearing (something distinctive) </li></ul><ul><li>The street: it’s harder to have everything to hand, prepare yourself 1. clipboard and pen 2. device 3. recording device/s </li></ul><ul><li>The sessions should feel more relaxed than lab, but still remember script and schedule </li></ul><ul><li>Additional safety announcements </li></ul>The tests – meeting users
  12. 12. <ul><li>Users asked to ‘think aloud’ (en.wikipedia.org/wiki/Think_aloud_protocol) main recording method, for camera / mic / clipboard </li></ul><ul><li>Sessions were relaxed, users didn’t display signs of being under pressure, partly due to ‘open’ nature of main task? </li></ul><ul><li>Users allowed to roam, even slightly off course e.g. Bullring </li></ul>The tests – the session
  13. 13. <ul><li>As in lab tests, facilitator only intervened when help required </li></ul><ul><li>Facilitator very aware log sheet was main observing tool, no other observer </li></ul><ul><li>Facilitator was also relaxed, an effect of doing ‘own research’ rather than for a client? i.e. knowing expectations </li></ul><ul><li>Facilitator logged notes, also watched time, to bring users back to start point after 45 minutes </li></ul><ul><li>After 7 sessions, very tiring </li></ul>The tests – facilitator’s role
  14. 14. <ul><li>It broke. After first session, found it had crashed. </li></ul><ul><li>May have been because facilitator held receiver whilst wireless camera roamed slightly. </li></ul><ul><li>Luckily, had fitted each participant with audio recorder and tie clip mic. </li></ul><ul><li>Mic alone, not intrusive, especially as one user walked off wearing it! </li></ul><ul><li>NB what is the effect of not having to wear hat cam, when they were expecting to? </li></ul>The tests – the problem with hat cam
  15. 15. <ul><li>Some collation very easy i.e. questionnaires </li></ul><ul><li>Collation of log sheet issues, plus other common issues found from notes (see next) </li></ul><ul><li>No video at all, camera crashed both days </li></ul><ul><li>Day 2 of audio recordings lost, Day 1 recordings yet to be played back </li></ul><ul><li>Some quotes pulled out </li></ul><ul><li>All notes could have been typed up to create a word cloud, but danger of words taken out of context of sentences </li></ul>Analysing results
  16. 16. <ul><li>Keep log sheet simple, issues to tick off, then a large empty area. Maybe use error, confusion boxes, but these less used on 2 nd day </li></ul><ul><li>What are legal issues of mobile testing in streets? E.g. user gets run over, filming in public. Add disclaimer to participation agreement and script </li></ul><ul><li>Meeting strangers, wear something distinctive </li></ul><ul><li>In analysis, paper notes easier to refer to than time-based recordings </li></ul><ul><li>Still unsure of what could be learned from a working hat cam alone How much can we see? </li></ul><ul><li>Space out test days or alternate facilitators to avoid tiring out. Maybe 4 sessions a day instead of 7. </li></ul>Mobile testing – lessons learned
  17. 17. <ul><li>What is effect of asking users to do something, and then telling them they don't need to? e.g. hat cam </li></ul><ul><li>Get hat cam working. How can it be most useful? </li></ul><ul><li>What is effect of wearing our old style kit on the test? </li></ul><ul><li>Could shorter ‘café’ sessions be run in the field on mobile devices? </li></ul>Further research
  18. 18. Finally, some of the results
  19. 23. Words picked from word list, limited to top 30 words (wordle.net)
  20. 24. Words rated as ‘top 3’ from word list, no limit (wordle.net)
  21. 25. Logged issues. Did I get tired / bored?
  22. 26. Logged issues – totals
  23. 27. Logged issues totals <ul><li>Would want to select where I'm going to. </li></ul><ul><li>[walking to tree of life marker] &quot;I'm intrigued now. </li></ul><ul><li>Very interesting, learned a few things on the way’ </li></ul><ul><li>[in Bullring] 'Don't think it could give you shop info because of 3 floors [and GPS] and there are too many shops </li></ul><ul><li>Thank you, I really enjoyed that </li></ul><ul><li>It would be good to have a button for each saying 'Take me there', doing so on the map, or giving me text directions. </li></ul><ul><li>The map is so accurate! </li></ul><ul><li>Its got all the numbers for the hotels. </li></ul><ul><li>Do like the history information.       </li></ul><ul><li>[tree of life] 'was totally unaware of that […] as a citizen, that's more interesting to me than the shops </li></ul><ul><li>I didn't even realise there were that many places in the Bullring     </li></ul><ul><li>If you were a stranger and you could download this, not only would it guide you there but give you other information. </li></ul><ul><li>It's told me its free to go into the museum. </li></ul><ul><li>I'm a schoolteacher so this would be great, you could plan a school trip around it.                  </li></ul><ul><li>You've got your very own tour guide with you! </li></ul><ul><li>Architecture tours for architecture students?                 </li></ul><ul><li>As a tourist, would pay £10 or £15 for it   </li></ul>