Overseas Usability Testing


Published on


Are you planning to conduct usability tests in other countries? It is more challenging (and rewarding) than you might think.

Here is a case study of usability tests conducted in three European markets that highlights what worked — and what did not. This unique case study explores what happens when you drive a test with the following statistics:

12 days
10,723 miles in travel
3 testing locations
3 third-party agencies
1 prototype
3 languages
24 test participants

Sound like fun? Read on.

Published in: Design, Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • stop and make the crowd smile for photo for the blog

    hand camera to julee
  • managed all aspects of testing
    an e-commerce solution prototype in three carefully
    chosen markets in Europe: London, Paris, and Milan

  • this slide will give you a sense of the logistics for such test

  • Words to describe my trip: sleep-deprived, always-moving
  • tasked to test your site or service internationally?
    are there any usability professionals here today?
  • given the responsibility of creating success in a foreign market?
    any biz people here?

  • Are you at all curious about usability testing in other countries?

  • so, why are we talking about overseas testing as being any different from domestic testing?
  • Do you know what a Value Added Tax is?
    Test in Europe and you’ll learn all about it.
  • rates are a bit better these days but who knows where they might go ...
  • The old stereotype of cheap rail in Europe is gone.
  • most testing centers are equipped to handle the native language only
  • I have just shared with you four major ways testing overseas is $
    So, if you are going to do it, you should get your money’s worth
  • the following is a comedy of errors

    it is my hope to entertain and enlighten so you not make the
    similar mistakes
  • this we did right.

    We wrote our test plan, scripts and success criteria all before we
    hired an external agency for testing
  • we conducted scenario based tasks.
    we selected the right number
    but poor execution made some tasks rushed or incomplete results
  • the agency that was hired was well versed in bench mark testing.
    that is, testing of live sites to get both objective and subjective feedback

    this competed with time to complete what we needed to evaluate our ecommerce solution
  • contact was made with the facilitators within just a few weeks of testing

    perhaps if we had sooner, we could have uncovered their
    lack of experience in formative testing
  • Looking back, I am not sure they were listening to our requirements much anyway
  • our agency was highly recommended to us and therefore additional bids were not sought

    further, we did not speak the same language. we said task based testing, they heard benchmark testing

    communicate until they become one of you
  • we created some maintenance issues for ourselves
    but we wanted to compare one solution across three different markets
    to see what worked and what didn’t
  • parts of our model worked and some parts did not

    this inhibited natural learning curve confused users

  • We had some help with the translations
    but some words were just not quite right
    and sometimes this can be pretty funny
  • an actual sign in a paris hotel elevator

    we all know what this supposed to say

    let’s further illustrate the point so that
    hopefully you will never create something like this
  • at a Bed & Breakfast in France

    i love the last line. reminds me of Gusteau from Ratatouille

    this one just sounds French

  • Switzerland: in a Swiss mountain inn:
  • also from Switzerland
  • for the people who flew here,
    how would you like to see such a sign
    when checking your bags?

    Denmark airline ticket office
  • In an Acapulco hotel a sign read
  • this also extends to multimillion dollar marketing campaigns

    remember this slogan? (read aloud)
  • ok, last one

    cigarette ad in Japan

    so regarding translations, have a native speaker
    with your industry experience complete your translations
  • seriously, my french is so bad,
    my first meal in paris was so bad because i thought
    i knew what i was ordering. it was a plate of legs and eyeballs
    it was like i had dinner at a bait shop
  • With all the languages and different facilitators
    we tried to normalize the presentation of the scenarios by
    handing users a card with instructions.
  • Despite asking for our prototype days in advance to ‘prepare’
    it was obvious that the facilitators did not explore the model themselves
    like they were almost seeing it for the first time at the pilot session
    (get to the next slide)
  • for instance,

    search page was at the start of the prototype and
    all they needed was to click the button or push enter

    we told them we were not evaluating the page. it is a common page in the industry

    but instead many pointless questions were asked instead
  • If this happens to you,

    Be firm and get them to listen if they are not conducting properly

  • soon, we started providing feedback during the session
    instead of waiting for session breaks

    “what are you doing? i thought we told you to stop asking that question ...”
  • let’s review how seemingly
    small issues can build up and cascade
    into larger problems
  • it wasn’t the looking for an agency so much
    as it was the executive approval process
    appropriately so as it was a lot of money
    then a long legal review of the contract with our corp counsel
  • leaving too little time to communicate with our highly
    recommended agency

  • each of these elements chipped away at our results
    again, these tests were highly successful
    but we can always improve, right?
  • You will most likely be tantalized with a ‘comprehensive (cough) expensive
    test reports
    probably filled with recommendations that may well be off target anyway
    take your own notes, the sessions are recorded if you want to focus

  • Testing in three markets nearly at once helped underscore the attitudinal and behavioral differences that exist in each country or location. It also helped reduce travel costs.
  • without specifics
    We learned

    Large market worked well for us
    we could launch right away
  • Medium market a chance to succeed but more refinement and testing are needed to bear out the final solution
  • not successful in Italy and it is recommend
    traditional shopping model be used
  • testing three cities at once limited the flights over the atlantic
    and saved us a lot of money
  • we had several stakeholders present for the tests
    this was a good thing.
    besides who doesn’t want a trip to europe?
  • we brought our prototype on CD as we did not have it on a server
    Due to the differences in time zones and developer on PTO, it was difficult to make substantive changes to the model.
    i was up late a few nights making changes in NOTEPAD. not fun
    (get to next slide, let me illustrate this point)
  • let me illustrate just how different the time was ...
    i know this seems obvious but it really turned out
    to be important and just another hurdle
  • in London we are wrapping up for the day
  • that says dinner
    but in reality it was a pub
  • manage any stateside resources well.
    they might just save your neck
  • test streaming was one of the reasons why we chose the agencies we did.
    This was a huge problem when stateside execs wanted to see the expensive tests they were paying for
  • tremendously good idea. with so much testing going on,
    it can all run together and many details can be lost.
    consensus building was also very important.
  • we left very little time to travel between the testing cities.
    (tell the story of the suitcase wheel)
  • something else we did that was interesting
    we had Internet Explorer and Fire Fox open to a blank page.

    Several people were nearly unable to begin a browsing session without the familiar search engine. In fact, several people were simply unable to type any URL at all,lacking all sense of correct URL syntax.
  • let’s look at 4 different ways you can test overseas

    i will start with most and end with least desirable

    it is also ranked most expensive to least expensive

  • Travel overseas and conduct a formal usability study
    like we did

  • if you are REALLY BRAVE, farm it out

  • Have staff from local branches overseas conduct Ad-Hoc or Rogue usability tests, even if they are not trained in usability testing.

  • Overseas Usability Testing

    1. 1. Overseas usability testing dy e stu A Ca s DYE R E B Y JO @joeplanet joedyer@gmail.com
    2. 2. A long time ago in a country far, far away... I  had  the  remarkable  experience  of  managing  the   usability  tes;ng  of  a  prototype  in  Europe  across   three  markets.
    3. 3. by the numbers ★ 12  days ★ 10,723  miles ★ 3  tes;ng  loca;ons ★ 1  prototype ★ 3  languages ★ 24  test  par;cipants
    4. 4. by the numbers ★ 12  days ★ 10,723  miles LONDON ★ 3  tes;ng  loca;ons PARIS MILAN ★ 1  prototype ★ 3  languages ★ 24  test  par;cipants
    5. 5. Who is this for? usability experts
    6. 6. Who is this for? Business or product managers
    7. 7. Who is this for? the morbidly curious
    8. 8. Who is this for? wait, what are we talking about? the morbidly curious
    9. 9. is testing Different overseas?
    10. 10. How testing is different overseas INTERACTIVE AGENCY Tes;ng  abroad  almost  always  involves  working   with  a  tes;ng  agency.  Think  expensive  contracts.
    11. 11. How testing is different overseas Less  than  favorable  exchange  rates  in  the  last  several  years
    12. 12. How testing is different overseas London milan Have  you  seen  the  cost  of  interna;onal  travel  these  days?
    13. 13. How testing is different overseas He says, your breath He says, your breath smells like garlic! smells like garlic! votr e ha sen lein t l'a e il! Transla;on  services  are  likely  to  be  an  addi;onal  cost.
    14. 14. 4
    15. 15. what worked well and what did not? preparation facilitation execution
    16. 16. Plan early & Share early Test  objec;ves  and   scenarios  were  wriRen   internally  before  shared   with  tes;ng  consultants.
    17. 17. appropriate # of tasks Number  of  test  objec;ves  in  the  usability   test  plan  were  too  numerous  for  the  test   as  it  was  executed.  
    18. 18. appropriate # of tasks Facilitators  wasted  too  much  ;me  on   unrequested  benchmark  tes;ng  ques;ons.
    19. 19. Engage early & often agency client Maybe two conference calls and a dozen or so emails were not quite enough.
    20. 20. Engage early & often ZZZZ agency client Maybe two conference calls and a dozen or so emails were not quite enough.
    21. 21. Test preparation Do  all  that  you  can  to  assess  the  skills  and  abili;es   of  any  agency  or  contractor  you  intend  to  hire. Get  mul;ple  bids  if  you  can. Over  communicate  all  tes;ng  objec;ves  un;l  they   are  capable  of  being  your  proxy.
    22. 22. the prototype We  created  a  single  prototype,  in  three  languages.
    23. 23. go high-fidelity Limited  sta;c  HTML  model  made  it  nearly  impossible  for  test   par;cipants  to  explore  the  possibili;es  of  the  prototype.
    24. 24. lost in translation ected to Visito rs are exp between the office c omplain at 1 am daily of 9 and 1 the hours Some  translated  words  were  too  literal  and  were  not   reviewed  for  context  within  the  prototype.
    25. 25. Please leave your values at the front desk
    26. 26. The genuine antics in your room come from our family castle. Long life to it.
    27. 27. special today: no ice cream
    28. 28. special today: no ice cream es l eave ou r win thing to yo u no h ope for
    29. 29. We take your b ags and send them in all directions
    30. 30. We take your b ags and send them in all directions The manag er has pers onally passed all the serve d here water
    31. 31. Come alive with the Pepsi Generation
    32. 32. Pepsi will bring your ancestors back from the dead
    33. 33. Salem - Feeling Free
    34. 34. When smoking Salem, you feel so refreshed that your mind seems to be free and empty
    35. 35. selecting test participants The  English  call  screener   was  easy  to  review  and   approve. The  French  screener? The  Italian  screener?
    36. 36. printed test scenarios Scenarios  were  given  to   par;cipants  wriRen  on  a   card  for  consistency.
    37. 37. and even more issues ... Facilitators  were  not   familiar  with  the   prototype  or  test   objec;ves  prior  to   tes;ng.
    38. 38. keeping it on the fairway
    39. 39. keeping it on the fairway How am i doing?
    40. 40. keeping it on the fairway do you really How am i need me to tell doing? you again?
    41. 41. test facilitation Instant  Messenger  can  be  your  friend.    
    42. 42. the domino effect
    43. 43. the domino effect Long  process  of   contrac;ng,  legal   review,  etc
    44. 44. the domino effect Test  plan   covered  in  two   conference  calls   &  email Long  process  of   contrac;ng,  legal   review,  etc
    45. 45. the domino effect Benchmark   tes;ng   encroached  on   scenarios Test  plan   covered  in  two   conference  calls   &  email Long  process  of   contrac;ng,  legal   review,  etc
    46. 46. the domino effect Facilitator’s  lack   of  familiarity   with  prototype Benchmark   tes;ng   encroached  on   scenarios Test  plan   covered  in  two   conference  calls   &  email Long  process  of   contrac;ng,  legal   review,  etc
    47. 47. beware expensive reports rt rt r epo r epo ft al dra fin
    48. 48. so, why three cities at nearly the same time?
    49. 49. underscore user differences MARKET COMPARISON LARGE medium SMALL
    50. 50. underscore user differences MARKET COMPARISON LARGE medium SMALL i 1. Very internet savvy 2. High rate of adoption 3. Not likely to need offline help 4. market is ready for the solution
    51. 51. underscore user differences MARKET COMPARISON LARGE medium SMALL i 1. more market potential 2. not likely to need offline help 3. solution has a chance to succeed with some modifications
    52. 52. underscore user differences MARKET COMPARISON LARGE medium SMALL i 1. catastrophic usability findings 2. Lower rate of adoption 3. most likely to need offline help 4. solution doesn't work in this locale
    53. 53. reduced travel costs
    54. 54. reduced travel costs
    55. 55. so biz managers could see market reactions Key  business  managers  were  present  during  the  tests,   directly  observing  poten;al  customers  interac;ng  with  the   new  e-­‐commerce  solu;on.
    56. 56. the time machine
    57. 57. the time machine just getting to work
    58. 58. the time machine just getting to work last test of the day
    59. 59. the time machine just getting to work last test of the day where to eat dinner?
    60. 60. this is no time for a vacation All resources working on the prototypes should not be on vacation or tied up with other commitments during the time of testing. Consider moving to an alternate schedule.
    61. 61. technical difficulties Sorry. The streaming video has timed out. ! See your System Administrator for help. 0:00 / 4:59
    62. 62. daily conference calls Daily  conference  calls  were  setup  to  discuss   test  findings  and  build  consensus  on  the   severity  of  each  finding.
    63. 63. allow for travel time
    64. 64. it's just a browser, right?
    65. 65. 4 ways to test overseas
    66. 66. 4 ways to test overseas plan it all and hire help
    67. 67. 4 ways to test overseas keynote (formerly vividence) study
    68. 68. 44 waysto test overseas ways to test abroad outsource it completely
    69. 69. 44 waysto test overseas ways to test abroad rogue or ad hoc testing
    70. 70. and so, in conclusion ... OVERSEAS  TESTING  PRE-­‐FLIGHT  CHECKLIST
    71. 71. questions?
    72. 72. @joeplanet newtrality.com joedyer@gmail.com
    73. 73. resources iStockphoto Flickr UCD  Stencil  by  Zazelenchuk  &  Boling Glyphish  Icons My  Wacom  tablet  and  two  hands Created  while  listening  to  Master  of  Puppets by  Metallica  and    Are  You  Experienced  by  Jimi  Hendrix