Overseas Usability Testing
•
1 like•679 views
NOTE: SOME OF THE SLIDES IN THIS ONLINE SLIDESHOW ARE DEFECTIVE. THE DOWNLOAD VERSION IS FINE. Are you planning to conduct usability tests in other countries? It is more challenging (and rewarding) than you might think. Here is a case study of usability tests conducted in three European markets that highlights what worked — and what did not. This unique case study explores what happens when you drive a test with the following statistics: 12 days 10,723 miles in travel 3 testing locations 3 third-party agencies 1 prototype 3 languages 24 test participants Sound like fun? Read on.
Recommended
More Related Content
Recently uploaded
Recently uploaded (20)
Featured
Featured (20)
Overseas Usability Testing
- 1. Overseas usability testing dy e stu A Ca s DYE R E B Y JO @joeplanet joedyer@gmail.com
- 2. A long time ago in a country far, far away... I had the remarkable experience of managing the usability tes;ng of a prototype in Europe across three markets.
- 3. by the numbers ★ 12 days ★ 10,723 miles ★ 3 tes;ng loca;ons ★ 1 prototype ★ 3 languages ★ 24 test par;cipants
- 4. by the numbers ★ 12 days ★ 10,723 miles LONDON ★ 3 tes;ng loca;ons PARIS MILAN ★ 1 prototype ★ 3 languages ★ 24 test par;cipants
- 5. Who is this for? usability experts
- 6. Who is this for? Business or product managers
- 7. Who is this for? the morbidly curious
- 8. Who is this for? wait, what are we talking about? the morbidly curious
- 9. is testing Different overseas?
- 10. How testing is different overseas INTERACTIVE AGENCY Tes;ng abroad almost always involves working with a tes;ng agency. Think expensive contracts.
- 11. How testing is different overseas Less than favorable exchange rates in the last several years
- 12. How testing is different overseas London milan Have you seen the cost of interna;onal travel these days?
- 13. How testing is different overseas He says, your breath He says, your breath smells like garlic! smells like garlic! votr e ha sen lein t l'a e il! Transla;on services are likely to be an addi;onal cost.
- 14. 4
- 15. what worked well and what did not? preparation facilitation execution
- 16. Plan early & Share early Test objec;ves and scenarios were wriRen internally before shared with tes;ng consultants.
- 17. appropriate # of tasks Number of test objec;ves in the usability test plan were too numerous for the test as it was executed.
- 18. appropriate # of tasks Facilitators wasted too much ;me on unrequested benchmark tes;ng ques;ons.
- 19. Engage early & often agency client Maybe two conference calls and a dozen or so emails were not quite enough.
- 20. Engage early & often ZZZZ agency client Maybe two conference calls and a dozen or so emails were not quite enough.
- 21. Test preparation Do all that you can to assess the skills and abili;es of any agency or contractor you intend to hire. Get mul;ple bids if you can. Over communicate all tes;ng objec;ves un;l they are capable of being your proxy.
- 22. the prototype We created a single prototype, in three languages.
- 23. go high-fidelity Limited sta;c HTML model made it nearly impossible for test par;cipants to explore the possibili;es of the prototype.
- 24. lost in translation ected to Visito rs are exp between the office c omplain at 1 am daily of 9 and 1 the hours Some translated words were too literal and were not reviewed for context within the prototype.
- 25. Please leave your values at the front desk
- 26. The genuine antics in your room come from our family castle. Long life to it.
- 27. special today: no ice cream
- 28. special today: no ice cream es l eave ou r win thing to yo u no h ope for
- 29. We take your b ags and send them in all directions
- 30. We take your b ags and send them in all directions The manag er has pers onally passed all the serve d here water
- 31. Come alive with the Pepsi Generation
- 32. Pepsi will bring your ancestors back from the dead
- 33. Salem - Feeling Free
- 34. When smoking Salem, you feel so refreshed that your mind seems to be free and empty
- 35. selecting test participants The English call screener was easy to review and approve. The French screener? The Italian screener?
- 36. printed test scenarios Scenarios were given to par;cipants wriRen on a card for consistency.
- 37. and even more issues ... Facilitators were not familiar with the prototype or test objec;ves prior to tes;ng.
- 38. keeping it on the fairway
- 39. keeping it on the fairway How am i doing?
- 40. keeping it on the fairway do you really How am i need me to tell doing? you again?
- 41. test facilitation Instant Messenger can be your friend.
- 43. the domino effect Long process of contrac;ng, legal review, etc
- 44. the domino effect Test plan covered in two conference calls & email Long process of contrac;ng, legal review, etc
- 45. the domino effect Benchmark tes;ng encroached on scenarios Test plan covered in two conference calls & email Long process of contrac;ng, legal review, etc
- 46. the domino effect Facilitator’s lack of familiarity with prototype Benchmark tes;ng encroached on scenarios Test plan covered in two conference calls & email Long process of contrac;ng, legal review, etc
- 47. beware expensive reports rt rt r epo r epo ft al dra fin
- 48. so, why three cities at nearly the same time?
- 49. underscore user differences MARKET COMPARISON LARGE medium SMALL
- 50. underscore user differences MARKET COMPARISON LARGE medium SMALL i 1. Very internet savvy 2. High rate of adoption 3. Not likely to need offline help 4. market is ready for the solution
- 51. underscore user differences MARKET COMPARISON LARGE medium SMALL i 1. more market potential 2. not likely to need offline help 3. solution has a chance to succeed with some modifications
- 52. underscore user differences MARKET COMPARISON LARGE medium SMALL i 1. catastrophic usability findings 2. Lower rate of adoption 3. most likely to need offline help 4. solution doesn't work in this locale
- 55. so biz managers could see market reactions Key business managers were present during the tests, directly observing poten;al customers interac;ng with the new e-‐commerce solu;on.
- 56. the time machine
- 57. the time machine just getting to work
- 58. the time machine just getting to work last test of the day
- 59. the time machine just getting to work last test of the day where to eat dinner?
- 60. this is no time for a vacation All resources working on the prototypes should not be on vacation or tied up with other commitments during the time of testing. Consider moving to an alternate schedule.
- 61. technical difficulties Sorry. The streaming video has timed out. ! See your System Administrator for help. 0:00 / 4:59
- 62. daily conference calls Daily conference calls were setup to discuss test findings and build consensus on the severity of each finding.
- 63. allow for travel time
- 64. it's just a browser, right?
- 65. 4 ways to test overseas
- 66. 4 ways to test overseas plan it all and hire help
- 67. 4 ways to test overseas keynote (formerly vividence) study
- 68. 44 waysto test overseas ways to test abroad outsource it completely
- 69. 44 waysto test overseas ways to test abroad rogue or ad hoc testing
- 70. and so, in conclusion ... OVERSEAS TESTING PRE-‐FLIGHT CHECKLIST
- 71. questions?
- 73. resources iStockphoto Flickr UCD Stencil by Zazelenchuk & Boling Glyphish Icons My Wacom tablet and two hands Created while listening to Master of Puppets by Metallica and Are You Experienced by Jimi Hendrix
Editor's Notes
- stop and make the crowd smile for photo for the blog hand camera to julee
- managed all aspects of testing an e-commerce solution prototype in three carefully chosen markets in Europe: London, Paris, and Milan
- this slide will give you a sense of the logistics for such test
- Words to describe my trip: sleep-deprived, always-moving
- tasked to test your site or service internationally? are there any usability professionals here today?
- given the responsibility of creating success in a foreign market? any biz people here?
- Are you at all curious about usability testing in other countries?
- so, why are we talking about overseas testing as being any different from domestic testing?
- Do you know what a Value Added Tax is? Test in Europe and you’ll learn all about it.
- rates are a bit better these days but who knows where they might go ...
- The old stereotype of cheap rail in Europe is gone.
- most testing centers are equipped to handle the native language only
- I have just shared with you four major ways testing overseas is $ So, if you are going to do it, you should get your money’s worth
- the following is a comedy of errors it is my hope to entertain and enlighten so you not make the similar mistakes
- this we did right. We wrote our test plan, scripts and success criteria all before we hired an external agency for testing
- we conducted scenario based tasks. we selected the right number but poor execution made some tasks rushed or incomplete results
- the agency that was hired was well versed in bench mark testing. that is, testing of live sites to get both objective and subjective feedback this competed with time to complete what we needed to evaluate our ecommerce solution
- contact was made with the facilitators within just a few weeks of testing perhaps if we had sooner, we could have uncovered their lack of experience in formative testing
- Looking back, I am not sure they were listening to our requirements much anyway
- our agency was highly recommended to us and therefore additional bids were not sought further, we did not speak the same language. we said task based testing, they heard benchmark testing communicate until they become one of you
- we created some maintenance issues for ourselves but we wanted to compare one solution across three different markets to see what worked and what didn’t
- parts of our model worked and some parts did not this inhibited natural learning curve confused users
- We had some help with the translations but some words were just not quite right and sometimes this can be pretty funny
- an actual sign in a paris hotel elevator we all know what this supposed to say let’s further illustrate the point so that hopefully you will never create something like this
- at a Bed & Breakfast in France i love the last line. reminds me of Gusteau from Ratatouille this one just sounds French
- Switzerland: in a Swiss mountain inn:
- also from Switzerland
- for the people who flew here, how would you like to see such a sign when checking your bags? Denmark airline ticket office
- In an Acapulco hotel a sign read
- this also extends to multimillion dollar marketing campaigns remember this slogan? (read aloud)
- ok, last one cigarette ad in Japan so regarding translations, have a native speaker with your industry experience complete your translations
- seriously, my french is so bad, my first meal in paris was so bad because i thought i knew what i was ordering. it was a plate of legs and eyeballs it was like i had dinner at a bait shop
- With all the languages and different facilitators we tried to normalize the presentation of the scenarios by handing users a card with instructions.
- Despite asking for our prototype days in advance to ‘prepare’ it was obvious that the facilitators did not explore the model themselves like they were almost seeing it for the first time at the pilot session (get to the next slide)
- for instance, search page was at the start of the prototype and all they needed was to click the button or push enter we told them we were not evaluating the page. it is a common page in the industry but instead many pointless questions were asked instead
- If this happens to you, Be firm and get them to listen if they are not conducting properly
- soon, we started providing feedback during the session instead of waiting for session breaks “what are you doing? i thought we told you to stop asking that question ...”
- let’s review how seemingly small issues can build up and cascade into larger problems
- it wasn’t the looking for an agency so much as it was the executive approval process appropriately so as it was a lot of money then a long legal review of the contract with our corp counsel
- leaving too little time to communicate with our highly recommended agency
- each of these elements chipped away at our results again, these tests were highly successful but we can always improve, right?
- You will most likely be tantalized with a ‘comprehensive (cough) expensive test reports probably filled with recommendations that may well be off target anyway take your own notes, the sessions are recorded if you want to focus
- Testing in three markets nearly at once helped underscore the attitudinal and behavioral differences that exist in each country or location. It also helped reduce travel costs.
- without specifics We learned Large market worked well for us we could launch right away
- Medium market a chance to succeed but more refinement and testing are needed to bear out the final solution
- not successful in Italy and it is recommend traditional shopping model be used
- testing three cities at once limited the flights over the atlantic and saved us a lot of money
- we had several stakeholders present for the tests this was a good thing. besides who doesn’t want a trip to europe?
- we brought our prototype on CD as we did not have it on a server Due to the differences in time zones and developer on PTO, it was difficult to make substantive changes to the model. i was up late a few nights making changes in NOTEPAD. not fun (get to next slide, let me illustrate this point)
- let me illustrate just how different the time was ... i know this seems obvious but it really turned out to be important and just another hurdle
- in London we are wrapping up for the day
- that says dinner but in reality it was a pub
- manage any stateside resources well. they might just save your neck
- test streaming was one of the reasons why we chose the agencies we did. This was a huge problem when stateside execs wanted to see the expensive tests they were paying for
- tremendously good idea. with so much testing going on, it can all run together and many details can be lost. consensus building was also very important.
- we left very little time to travel between the testing cities. (tell the story of the suitcase wheel)
- something else we did that was interesting we had Internet Explorer and Fire Fox open to a blank page. Several people were nearly unable to begin a browsing session without the familiar search engine. In fact, several people were simply unable to type any URL at all,lacking all sense of correct URL syntax.
- let’s look at 4 different ways you can test overseas i will start with most and end with least desirable it is also ranked most expensive to least expensive
- Travel overseas and conduct a formal usability study like we did
- if you are REALLY BRAVE, farm it out
- Have staff from local branches overseas conduct Ad-Hoc or Rogue usability tests, even if they are not trained in usability testing.