Localization Research:
If we can do it, so can you!
Jennifer Fabrizi
Anna Bradley
Boston UPA Conference
May 25, 2011
Introduction
Jennifer Fabrizi
Lead of User Experience Research and
Design, Massachusetts Mutual Life
Insurance Company
Ann...
Agenda
• Our story
• Background
• Study One: 2009
• Study Two: 2010
• Why the findings were different
• What we learned ov...
This is the story about how 2 user experience practitioners
figured out how to conduct usability tests with
in-language pa...
Background
http://geoffallan.blogspot.com/
New Multicultural Markets
http://online.wsj.com/article/SB10001424052748704604704576220603247344790.html#project%3DCENSUS_SHIFT_2011%26articleTabs%3...
Marketing Research showed…
Hispanic Americans
• Have a lack of trust in financial institutions
• Depend on family and frie...
Marketing Research showed…
Chinese Americans
• Are averse to debt accumulation & personal
gratification
• Planning for lon...
Why in-language studies?
Study One: 2009
Recruiting
Multicultural Testing = nuance of localization
• 5 Hispanic-Americans with Spanish as first language
• 5 Chines...
Conducting the Study
Testing Room:
Spanish or Chinese speaking
moderator and participant
Observation Room:
Court-certified...
Conducting the Study
Testing Room:
Spanish or Chinese speaking
moderator and participant
Observation Room:
Court-certified...
What we learned
Chinese Hispanic
decision-
making
premium, coverage, customer service, reputation, branch locations/conven...
What we learned
Chinese Hispanic
decision-
making
premium, coverage, customer service, reputation, branch locations/conven...
Recommendations
Recommendations
A Typical Usability Error?
Setting Users’ Expectations
The in-language links didn’t set users’ expectation...
Study Two: 2009
Design Changes Made
• As a result of the 2009 study the link text was changed
to be more in line with the true experience
...
Follow up study of site changes
Setup and Recruiting
• The recruiting criteria were the same for this study as
for the 2009 study though the recruiting wa...
In-language Considerations
• All text presented to the user was translated into their
language
• UserZoom has a feature wh...
Findings
• All Spanish participants and several Chinese participants noted
that the in-language links should be located in...
Why were the findings different?
• The un-moderated study had nearly reversed results
from the 2009 study
• In an environm...
What we learned
• In-language studies can be done on a reasonable
budget.
• There are tools out there that make it easy to...
Questions?
Thanks for coming! Contact us!
Jennifer Fabrizi
Lead of User Experience Research and
Design, Massachusetts Mutual Life
Ins...
Upcoming SlideShare
Loading in …5
×

Boston Mini Upa2011: Localization Research Presentation by Jennifer Fabrizi and Anna Bradley

955 views

Published on

Presentation given by myself and my colleague, Anna Bradley

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
955
On SlideShare
0
From Embeds
0
Number of Embeds
17
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • multicultural testing = nuance of localization
    Set the users’ expectations more appropriately to what the site provides:
    Both links said “In Spanish” or “In Chinese” leading users to think the link would take them to a fully translated site. Recommendation = change text to “More about MassMutual” in their own language.

    Move other links into the information architecture, leaving only the in-language links. We felt this would increase findability of the links themselves.

  • This study was un-moderated, done through UserZoom™
    5 participants who spoke Spanish natively and 5 participants who spoke Mandarin Chinese natively
    Participants were presented a prototype which consisted of the public webpage and a working link to the entry page of the in-language content
    All sessions were remote and un-moderated, to allow for broader geographic distribution of participants and to control costs.
    Participants were asked follow-up questions after they interacted with the prototype.
  • Chinese respondents considered the site very useful and informative—and generally appreciated the amount of informative content and tools they could use for themselves.

    Hispanics considered the website too visually simple and not colorful. The straightforward layout and design somehow gave Hispanic respondents the perception that Mass Mutual was fairly young and had not been around for many years (until they read it was founded in 1851)

    Chinese respondents appreciated the written Chinese content, even in limited form, while Hispanics preferred either a full English site to only a partially translated one. For them, the effort seemed incomplete and half-hearted effort to win over the Hispanic population.
  • Boston Mini Upa2011: Localization Research Presentation by Jennifer Fabrizi and Anna Bradley

    1. 1. Localization Research: If we can do it, so can you! Jennifer Fabrizi Anna Bradley Boston UPA Conference May 25, 2011
    2. 2. Introduction Jennifer Fabrizi Lead of User Experience Research and Design, Massachusetts Mutual Life Insurance Company Anna Bradley Currently Interaction Designer at Memento Security; previously at Massachusetts Mutual Life Insurance Company
    3. 3. Agenda • Our story • Background • Study One: 2009 • Study Two: 2010 • Why the findings were different • What we learned overall
    4. 4. This is the story about how 2 user experience practitioners figured out how to conduct usability tests with in-language participants and moderators and learned a lot in the process Our story Jen Fabrizi Anna Bradley
    5. 5. Background
    6. 6. http://geoffallan.blogspot.com/ New Multicultural Markets
    7. 7. http://online.wsj.com/article/SB10001424052748704604704576220603247344790.html#project%3DCENSUS_SHIFT_2011%26articleTabs%3Darticle
    8. 8. Marketing Research showed… Hispanic Americans • Have a lack of trust in financial institutions • Depend on family and friends for advice Relationship development with agent is key
    9. 9. Marketing Research showed… Chinese Americans • Are averse to debt accumulation & personal gratification • Planning for longer term: protecting future generations • Rely on experts for advice Agent as expert, company stability and financial performance are key
    10. 10. Why in-language studies?
    11. 11. Study One: 2009
    12. 12. Recruiting Multicultural Testing = nuance of localization • 5 Hispanic-Americans with Spanish as first language • 5 Chinese-Americans with Mandarin Chinese as first language • Pertinent pre-screener requirements: • Emigrated to U.S.A. prior to age 16 • Current age 35-65 • Vendor support: • Bentley Design and Usability Center for review • New American Dimensions in Los Angeles to recruit and hire moderators and translators
    13. 13. Conducting the Study Testing Room: Spanish or Chinese speaking moderator and participant Observation Room: Court-certified synchronized interpreter, note takers, and observers Morae recorder Morae recorder Synchronized Translator Moderator Participant One of us taking notes Hosts and observers
    14. 14. Conducting the Study Testing Room: Spanish or Chinese speaking moderator and participant Observation Room: Court-certified synchronized translator, note takers, and observers Morae recorder Morae recorder Synchronized Translator Moderator Participant One of us taking notes Hosts and observers Lots of Talking Points on this slide! Multicultural users = nuance of localization? Details of the study: • In-language moderator; usability script translated into Spanish and Mandarin • Used a court-certified synchronized interpreter to provide live, real-time translations into English into the control room • Traveled to Los Angeles; used our mobile Usability Lab (Morae on 2 laptops) to record the participant sessions and translations • Used Camtasia to remove the audio of the in-language sessions and replace it with the translation audio • Used Morae Manager to pull clips to play for stakeholders; also provided full in-language .wmv files • One glitch: Morae wouldn’t connect with a 3rd laptop running Observer via the vendor’s network – learned that our mobile lab had to have network ports opened!
    15. 15. What we learned Chinese Hispanic decision- making premium, coverage, customer service, reputation, branch locations/convenience expectancy expected link to lead to fully-translated site translation quality excellent with few exceptions (i.e., mutuality) awareness low general useful and informative findability of link high low translated content appreciated translated content as data for decision making partial translations seemed to be less appreciated; translations seemed to serve emotional need vs. functional role visual design looks professional; Asian faces evoked positive connection; liked clean colors and information-focus liked detailed information; colors were cold
    16. 16. What we learned Chinese Hispanic decision- making premium, coverage, customer service, reputation, branch locations/convenience expectancy expected link to lead to fully-translated site translation quality excellent with few exceptions (i.e., mutuality) awareness low general useful and informative findability of link high low translated content appreciated translated content as data for decision making partial translations seemed to be much more negative; translations seemed to serve emotional need vs. functional role visual design looks professional; Asian faces evoked positive connection; liked clean colors and information-focus liked detailed information; colors were cold Warning!! Our study was designed to test for users’ behavior when trying to find links and figure out what they meant. Participants also gave us feedback about attitudes toward aspects of the site, including quality of translation, translated content, use of color, and overall visual design. These are all important aspects of a localization strategy for a site. We didn’t have such a strategy, so we probed about these attitudes and presented these findings as directional only to our internal partners. If the original study requirements had been to do a behavioral and valid attitudinal study, we would have used different recruiting methods (i.e., a much larger sample size!!) in combination with a different study design (probably remote-unmoderated via UserZoom), followed by qualitative usability. Our advice is to always make sure to report your findings in the correct context!
    17. 17. Recommendations
    18. 18. Recommendations A Typical Usability Error? Setting Users’ Expectations The in-language links didn’t set users’ expectations well in terms of what the site provided. In-language content was not a full-translated site; it was simply high-level content to reach out to these specific markets in their own languages. The links said “Spanish” and “Chinese”. Recommendation 1 = change text to “More about MassMutual” in their own languages Recommendation 2 = change text of the title of this section from “Browse by Feature” to “In-language Information” Through a second parallel mass affluent study not covered in this presentation, we cleared out the “junk drawer” by finding locations in the information architecture for everything else. Because findings regarding other aspects of localized design were directional only, we could not recommend changes to visual design treatment, content voice (which we heard was good feedback on), how much of the site to translate, translated menus, menu design, etc. Our recommendations aimed at being reasonable, cautious, and inexpensive.
    19. 19. Study Two: 2009
    20. 20. Design Changes Made • As a result of the 2009 study the link text was changed to be more in line with the true experience • The text was changed to say “More about MassMutual” in Spanish, Mandarin Chinese and Korean • In addition other links were moved to the menus (based on a separate usability study with mass affluent participants)
    21. 21. Follow up study of site changes
    22. 22. Setup and Recruiting • The recruiting criteria were the same for this study as for the 2009 study though the recruiting was done via a UserZoom™ recommended recruiter • This study was un-moderated, done through UserZoom™ • Prototype had one working link and one follow-up page • Remote and un-moderated for broad geographic access • Questions after the clicks.
    23. 23. In-language Considerations • All text presented to the user was translated into their language • UserZoom has a feature which translates the default text and instructions • All questions were asked in-language • The Spanish-speaking participants replied in Spanish • The Chinese-speaking participants replied in English. We assume this was due to a lack of a Chinese keyboard. • Responses were then translated by the same team who translated the original text
    24. 24. Findings • All Spanish participants and several Chinese participants noted that the in-language links should be located in the upper right hand of screen. • The Chinese participants found it easier to locate the in-language link due to Chinese characters standing out more amongst the text. • All Spanish participants were fine with what they found when they clicked on the link. They said it met their expectations. • The Chinese participants were split. One participant mentioned wanting to see the same homepage translated. One thought they saw more fluff than content. Others did not comment.
    25. 25. Why were the findings different? • The un-moderated study had nearly reversed results from the 2009 study • In an environment without another person present the Spanish speakers stated they were fine with the site. The Chinese participants on the other hand felt free to state that they did not like the lack of a full translation. • We believe that the un-moderated nature of the study changed how both groups presented their opinion • The altered text of the link also helped set the expectation for the Spanish speakers, but it did not help the Chinese speakers
    26. 26. What we learned • In-language studies can be done on a reasonable budget. • There are tools out there that make it easy to splice the synchronized translation in. • Cultural differences have to be taken into account but are influenced by adaptation to American culture. • We were surprised by the different results but after understanding them in a cultural context they make sense. • If we can do it, so can you!
    27. 27. Questions?
    28. 28. Thanks for coming! Contact us! Jennifer Fabrizi Lead of User Experience Research and Design, Massachusetts Mutual Life Insurance Company Anna Bradley Currently Interaction Designer at Memento Security; previously at Massachusetts Mutual Life Insurance Company jfabrizi@massmutual.com www.linkedin.com/in/jenniferfabrizi twitter.com/@jenniferfabrizi abradley@mementosecurity.com http://www.linkedin.com/pub/anna- margulis-bradley/3/b1a/7b3

    ×