Your SlideShare is downloading. ×
Outside the box: User-testing context-aware mobile technology in the field – a case study
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Outside the box: User-testing context-aware mobile technology in the field – a case study

870
views

Published on

This was a talk given at the second meeting of Brum Interaction Design Assocation. It discusses the method (but not so much the findings) of user-testing a GPS Trail application on a mobile device. …

This was a talk given at the second meeting of Brum Interaction Design Assocation. It discusses the method (but not so much the findings) of user-testing a GPS Trail application on a mobile device. Some videos and audio may not work, apologies.

Details also at: http://brumixda.wordpress.com/2009/09/04/outside-the-box-user-testing-context-aware-mobile-technology/

Published in: Technology, Business

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
870
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
25
Comments
0
Likes
2
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Outside the box User -testing context-aware mobile technology in the field – a case study
    • Jerome Turner
    • The product
    • Planning
    • The user-tests
    • Analysing results
    • Lessons learned: testing mobile apps in the field
    • (oh yes, the results)
  • 2. The product
  • 3. Content screens
  • 4. Planning
    • Main problem – how do we record and observe mobiles?
    • One solution, fixed in lab conditions:
    Validconcept.com
  • 5.
    • Another solution, Liliput. In the field but intrusive:
    Mobile observation http://portal.acm.org/citation.cfm?id=1240907
  • 6.
    • User-lab’s kit to date, far too cumbersome
    • What are we interested in recording and at what cost?
    • Screen capture won’t record hand/stylus movement.
    • Is visual recording more useful than notetaking?
    • Do we record all screen / mobile activity if it will greatly affect the user’s experience / usage?
    • Compromise: use one discrete hat cam to record as much video/audio as possible.
    • NB Interested in broad experience, not just around device
  • 7. Hat cam
  • 8.
    • Main difference, no live observer.
    • Tests should still be structured
      • Recruit participants, for timed sessions
      • Use script and task to keep schedule and ensure equal treatment throughout day
      • Use demographic form, tweaked to suit the tests
      • Use a ‘log sheet’ to take notes whilst facilitating
      • End of session, use
        • randomised word list to tick
        • questionnaire, responses to the product
        • understanding of some technical terms
        • incentive sign-off sheet
    Planning – from lab to the streets
  • 9.
    • Usually, would test product first, establish expected issues during sessions, and thus make log list.
    • Here, facilitator made log list based only on experience of building product / expectations of technology i.e. many tech errors included
    • Also included boxes for other observational logging…
    • … and space for comments and notes.
    Planning – facilitator’s log sheet
  • 10.
    • Through external recruiters.
    • Requirements sent without wanting to give too much away
    • They should be between 18 and 70 years of age.
    • They should have some experience of using a PDA, iPhone, Blackberry-style phone, smartphone or other similar digital mobile device.
    • They will need to be on their feet with me for potentially a full hour, walking around the centre of Birmingham (at a leisurely pace).
    • They will need to be comfortable with wearing a baseball style cap with a camera fitted that records their ‘point of view’ and voice. It doesn’t record their face. (if we really have trouble finding people for this, this requirement can be removed for a couple of people).
    • We need 8 people for Thursday and 8 for Friday. Sessions on each day to start at 9.00, 10.00, 11.00, 12.00, 13.30, 14.30, 15.30, 16.30
    • They will receive £30 in cash on the day.
    • It would be helpful if they weren’t people booked for us before, so they don’t know me.
    • They should dress expecting to be outside for the full hour, not under cover. E.g. Bring a brolly!
    Recruiting
  • 11.
    • Meeting users – make sure you have each others’ mobile numbers and have told them what you’ll be wearing (something distinctive)
    • The street: it’s harder to have everything to hand, prepare yourself 1. clipboard and pen 2. device 3. recording device/s
    • The sessions should feel more relaxed than lab, but still remember script and schedule
    • Additional safety announcements
    The tests – meeting users
  • 12.
    • Users asked to ‘think aloud’ (en.wikipedia.org/wiki/Think_aloud_protocol) main recording method, for camera / mic / clipboard
    • Sessions were relaxed, users didn’t display signs of being under pressure, partly due to ‘open’ nature of main task?
    • Users allowed to roam, even slightly off course e.g. Bullring
    The tests – the session
  • 13.
    • As in lab tests, facilitator only intervened when help required
    • Facilitator very aware log sheet was main observing tool, no other observer
    • Facilitator was also relaxed, an effect of doing ‘own research’ rather than for a client? i.e. knowing expectations
    • Facilitator logged notes, also watched time, to bring users back to start point after 45 minutes
    • After 7 sessions, very tiring
    The tests – facilitator’s role
  • 14.
    • It broke. After first session, found it had crashed.
    • May have been because facilitator held receiver whilst wireless camera roamed slightly.
    • Luckily, had fitted each participant with audio recorder and tie clip mic.
    • Mic alone, not intrusive, especially as one user walked off wearing it!
    • NB what is the effect of not having to wear hat cam, when they were expecting to?
    The tests – the problem with hat cam
  • 15.
    • Some collation very easy i.e. questionnaires
    • Collation of log sheet issues, plus other common issues found from notes (see next)
    • No video at all, camera crashed both days
    • Day 2 of audio recordings lost, Day 1 recordings yet to be played back
    • Some quotes pulled out
    • All notes could have been typed up to create a word cloud, but danger of words taken out of context of sentences
    Analysing results
  • 16.
    • Keep log sheet simple, issues to tick off, then a large empty area. Maybe use error, confusion boxes, but these less used on 2 nd day
    • What are legal issues of mobile testing in streets? E.g. user gets run over, filming in public. Add disclaimer to participation agreement and script
    • Meeting strangers, wear something distinctive
    • In analysis, paper notes easier to refer to than time-based recordings
    • Still unsure of what could be learned from a working hat cam alone How much can we see?
    • Space out test days or alternate facilitators to avoid tiring out. Maybe 4 sessions a day instead of 7.
    Mobile testing – lessons learned
  • 17.
    • What is effect of asking users to do something, and then telling them they don't need to? e.g. hat cam
    • Get hat cam working. How can it be most useful?
    • What is effect of wearing our old style kit on the test?
    • Could shorter ‘café’ sessions be run in the field on mobile devices?
    Further research
  • 18. Finally, some of the results
  • 19.  
  • 20.  
  • 21.  
  • 22.  
  • 23. Words picked from word list, limited to top 30 words (wordle.net)
  • 24. Words rated as ‘top 3’ from word list, no limit (wordle.net)
  • 25. Logged issues. Did I get tired / bored?
  • 26. Logged issues – totals
  • 27. Logged issues totals
    • Would want to select where I'm going to.
    • [walking to tree of life marker] "I'm intrigued now.
    • Very interesting, learned a few things on the way’
    • [in Bullring] 'Don't think it could give you shop info because of 3 floors [and GPS] and there are too many shops
    • Thank you, I really enjoyed that
    • It would be good to have a button for each saying 'Take me there', doing so on the map, or giving me text directions.
    • The map is so accurate!
    • Its got all the numbers for the hotels.
    • Do like the history information.      
    • [tree of life] 'was totally unaware of that […] as a citizen, that's more interesting to me than the shops
    • I didn't even realise there were that many places in the Bullring    
    • If you were a stranger and you could download this, not only would it guide you there but give you other information.
    • It's told me its free to go into the museum.
    • I'm a schoolteacher so this would be great, you could plan a school trip around it.                 
    • You've got your very own tour guide with you!
    • Architecture tours for architecture students?                
    • As a tourist, would pay £10 or £15 for it  

×