Multi-national user testing: Tools, tricks and pitfalls
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Multi-national user testing: Tools, tricks and pitfalls

on

  • 618 views

Ilana Kaplan presented on multi-national user testing at UXNZ in Wellington, November 2013.

Ilana Kaplan presented on multi-national user testing at UXNZ in Wellington, November 2013.

Statistics

Views

Total Views
618
Views on SlideShare
574
Embed Views
44

Actions

Likes
1
Downloads
6
Comments
0

3 Embeds 44

http://uxnewzealand.co.nz 29
https://twitter.com 9
http://librosweb.es 6

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Multi-national user testing: Tools, tricks and pitfalls Presentation Transcript

  • 1. Multi-national user testing: Tools, tricks and pitfalls Ilana Kaplan UX NZ 2013
  • 2. Multi country testing becoming more common place  Understanding international customers is more critical than ever as emerging market economies continue their rapid expansion  On the flip side, companies that don’t understand their international customers are at a strategic disadvantage and at greater risk of making missteps that could undermine their success in diverse markets http://uxmag.com/articles/6-key-questions-to-guide-international-ux-research
  • 3. Multi country testing becoming more common place  International UX research is important even in countries or regions with customers who speak the same language, as regional differences in nomenclature , cultural norms, and customer needs can dramatically impact the success of a product, and…. http://uxmag.com/articles/6-key-questions-to-guide-international-ux-research
  • 4. Multi country testing becoming more common place  International UX research is important even in countries or regions with customers who speak the same language, as regional differences in nomenclature , cultural norms, and customer needs can dramatically impact the success of a product, and….  Legislation (medical devices)  One size fits all – or does it?  Soft launch in one county before we launch in others  Benchmarking against international competitors
  • 5. Testing options Moderated In person Remote Unmoderated
  • 6. Quant First click Card sorting Surveys Task based Diaries Data o o o o o Qual Unmoderated Testing options Less More Cost
  • 7. Testing options Moderated In person Remote o Traditional user testing o Eye tracking Unmoderated
  • 8. Testing options Moderated In person Remote o Observer shares monitor and participant uses system Unmoderated
  • 9. Testing options Moderated Qual Remote Quant Data In person Unmoderated Less More Cost
  • 10. Medical device Coordinated out of USA Face to face moderated usability testing • Observer travelled to each country • Testing coordinated by local UX partner
  • 11. Global consumer site Coordinated out of Australia Phase 2 Unmoderated Treejack testing in USA, Australia and Germany Phase 1 Face to face moderated research on visual concepts • • • Study designed in Australia Skype briefing and debriefing Testing conducted by local partners in each country Phase 3 Unmoderated ChalkMark testing in USA, Australia and Germany • All recruit and analysis out of Australia, with help from German speaking staff Phase 4 Face to face moderated usability testing in Australia and USA • • • Study designed in Australia Skype briefing and debriefing Testing conducted by local partners in each country
  • 12. International website NZ company operating internationally 1. User Survey on site in USA, Australia and NZ (around 1000 participants) 2. UX partners in Canada, the UK, and Hong Kong identified best practice examples in their country 3. Multi-national usability testing in Australia and NZ (34 particpants) 4. Eye-tracking in NZ 5. Expert reviews
  • 13. Smart Blood Pressure Monitor Study 2013 Qualitative insights into the purchase of a smart blood pressure monitor, the out-of-the-box experience and using the device over 10 days
  • 14. Digital health and wellness • More and more, patients are becoming consumers, spending more time monitoring their own health and well being and taking increased responsibility for themselves • There is a whole new genre of Apps and devices that support and encourage this movement • Electronic blood pressure monitors fit into this • UX alliance partners set out to understand some of the short term factors which could impinge on the usefulness of these device
  • 15. 20 participants from 17 countries Brazil Canada Japan New Zealand Italy India Poland Denmark Switzerland Australia USA France Finland Russia Ireland United Kingdom Germany
  • 16. The study – 3 stages Purchasing the device UX professionals purchase the device and record the experience Out-of-the-box experience Observe and record the participant’s outof-the-box experience Using the device Participants measure their blood pressure and complete a survey daily, for 10 days
  • 17. Smart Blood Pressure Monitor Study 2013: „Out-of-the-Box‟ Experience Unpacking device “It looks like a apple product. White package… it looks like an expensive product.” “There was no Russian version of the manual [or] list of languages on the cover of the manual so [the] participant had to look through the entire brochure.” “It was hard to insert the batteries and there was no explanation about this in the manual.” “I spent something like 10 minutes trying to force the box open. I missed a transparent sticker that prevented the box to slide.” Purchase accessory App Assembling • • • “It worked fine from the very first time.” “It was not clear which application you have to use with the device, and how to use it.” “Reading on the package I realised that it wasn't iPhone 5 compatible. I didn't have [an] adapter though.” Insights • It was not clear if an adapter was needed • Users struggled to read instructions in a foreign language • Most users had difficulties assembling the device First attempt “Not quite as slick as it could be.” “The participant received a high reading. No information was given to prompt him to consider the cause.” Many users found that batteries were needed to charge the device after attempting to charge via connection to a computer Different apps were available for download which confused users Most users were not successful reading their blood pressure the first time
  • 18. Multi-national user testing 21
  • 19. Testing options – how to choose? Moderated      Physical artifact Need rich qualitative feedback Need to have the human connection Ensure high-level of consistency Uncertain of quality or environment Unmoderated      Web/software UI More quantitative based Large, distributed sample or low incidence Low(er) budget High penetration of Internet access
  • 20. Cultural differences In multi country moderated testing Planning Running sessions
  • 21. Planning Check out the local calendar – make sure you find out what really happens In Australia and many other countries recruiting participants is difficult from mid December to January because of Christmas and New Year holidays. In France, avoid running test sessions in the first half of May, around July 14th (Bastille Day) In Brazil, don’t even think of recruiting during Carnival (February or March, it varies). In Canada, hockey is a religion! It’s difficult to recruit evenings during the hockey season. In Japan, at the start of May there is a week-long public holiday called “Golden Week”; in August, there is another week-long public holiday called “Bon Festival”. Most companies shut down so difficult to recruit and test.
  • 22. Planning Pick the time for sessions carefully In Japan, evening and weekend sessions are common because office workers are usually unable to attend daytime sessions during weekdays. In the US, it is difficult to schedule usability studies on the weekend. Most US participants place a high premium on their weekend time and reserve it for family, friends, or weekend errands and chores. In Moscow, many people have flexible working schedules; some of them work late in the evening, so session times can be varied. Free time is valuable (especially in large cities), so it’s difficult to convince people to participate on weekends. Brazilians often run late – traffic and other occurrences that prevent them from being in control contribute to laid-back attitude to punctuality.
  • 23. Planning Consider local demographics What does elderly mean? Which languages do you consider? Participants born abroad?
  • 24. Planning Plan ahead for recruitment. Get help from someone local How many? • • • • Italy over recruits by 20-30% UK 1 standby for each 6 participants Poland may one extra China, no shows rare Incentives?
  • 25. Planning Allow enough time to plan Protocol • • Develop detailed testing docs for recruitment, running, d ata collection and reporting Translation • • • • Technolog y Recordings Sharing files/streaming Cameras Test equipment Travel • • • Jetlag Where’s the lab? Time for a dry run
  • 26. Running sessions Does it matter who moderates? English isn’t always English. In Australia, it’s not unusual to use a local facilitator, even when the client is English speaking and has travelled to Australia to observe the sessions. In South Africa, it is most important to have the right cultural fit between moderators and participants. As such, moderators are not typically experienced usability professionals. What’s polite in one culture can be offensive in another
  • 27. Running sessions ‘Thinking aloud’ is very difficult in some cultures and too easy in others • The Brazilians and Irish love to talk, so thinking aloud is no problem! • Chinese users are often not willing to talk while doing the tasks. • • Indians typically think aloud. • In the UK, some participants are direct, while others may be more circumspect. In some cases, participants have struggled with tasks and yet are still politely positive about it. • In Japan, there is a cultural bias for men not to say “I don’t know” or “I don’t understand,” • For Germans and the British, small talk at the front of sessions is very important to get participants warmed up. Italians like to express their opinion on everything - they are experts in all matters and when you ask to think aloud during a task, you always get opinion on the layout!
  • 28. In conclusion • • There are valuable UX research options across a range of budgets • • Allow enough time to plan, and Local culture, beyond language influences design, but also how you do your research Consider working with local partners
  • 29. Ilana Kaplan UX NZ 2013 ilana.kaplan@optimalexperience.com