Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Effect of Machine Translation in Interlingual Conversation: Lessons from a Formative Study

1,187 views

Published on

Our talk at CHI2015 in Seoul, South Korea. Find more information at www.kotarohara.com .

YouTube: https://youtu.be/isqsYLkX9gA
Makeability Lab: http://www.cs.umd.edu/~jonf/
Microsoft Research: http://research.microsoft.com/

ABSTRACT
Language barrier is the primary challenge for effective cross-lingual conversations. Spoken language translation (SLT) is perceived as a cost-effective alternative to less affordable human interpreters, but little research has been done on how people interact with such technology. Using a prototype translator application, we performed a formative evaluation to elicit how people interact with the technology and adapt their conversation style. We conducted two sets of studies with a total of 23 pairs (46 participants). Participants worked on storytelling tasks to simulate natural conversations with 3 different interface settings. Our findings show that collocutors naturally adapt their style of speech production and comprehension to compensate for inadequacies in SLT. We conclude the paper with the design guidelines that emerged from the analysis.

  • Be the first to comment

Effect of Machine Translation in Interlingual Conversation: Lessons from a Formative Study

  1. 1. Kotaro Hara and Shamsi Iqbal makeability lab Effect of Machine Translation in Interlingual Conversation
  2. 2. こんにちは!
  3. 3. こんにちは!
  4. 4. Hello!
  5. 5. Successful intercultural communication is important in: Education Business Health-care Yet, a language barrier can be a challenge
  6. 6. “133/2011 euroscola” by rosipaw
  7. 7. Simultaneous interpretation is only for privileged few
  8. 8. “The infrequent use of interpreters* in the delivery ward was among the most important reasons for the reduced quality of [health] care.” Kale and Syed (2010) * Interpreter: a person who translates the words that someone is speaking into a different language (Merriam-Webster)
  9. 9. Automatic Spoken Language Translation
  10. 10. Speech Recognition & Machine Translation Performance Fügen et al. (2007) noted that “automatic systems can already provide usable information for people.” 2015
  11. 11. 2015 Little research has paid attention to how people interact with spoken language translation technology
  12. 12. Very little knowledge about Spoken Language Translation in HCI “Jigsaw puzzle” by James Petts
  13. 13. The void that we are trying to fill Very little knowledge about Spoken Language Translation in HCI
  14. 14. Background 背景
  15. 15. Hamon O., et al., End-to-End Evaluation in Simultaneous Translation, ECACL2009 Hello! Speech Text Spoken Language Translation System Spoken Language Translation Automatic speech recognition Machine translation Speech synthesis こんにちは a あ
  16. 16. Hamon O., et al., End-to-End Evaluation in Simultaneous Translation, ECACL2009 Hello! Speech Text Spoken Language Translation System Spoken Language Translation Automatic speech recognition Machine translation Speech synthesis こんにちは a あ
  17. 17. Danger. Do not enter Effect of machine translation on text-based communication Gao et al. 2014; Yamashita et al. 2009; Yamashita and Ishida 2006
  18. 18. Hello! Native English SpeakerNon-native English Speaker Hello! Effect of speech recognition on interlingual conversation Gao et al. CHI2014
  19. 19. Key Point Speech Recognition Text Output Machine Translation
  20. 20. Effect of spoken language translation system to interlingual communication is under-explored. Key Point
  21. 21. Evaluation of NESPOLE! Constantini et al. 2002 • No detailed discussion on how people adapted in using the system • Push-to-talk interface made it hard to analyze turn-taking behavior • Examined usability of spoken language translation system
  22. 22. Explore how spoken language translation affects a natural conversation between people speaking different languages Goal
  23. 23. Translator Tool Study Method Quantitative Analysis Content Analysis
  24. 24. Translator Tool
  25. 25. Video to show how the system work: One side speaks in English, another side in German Translator Tool | Demo Skype Translator Demo, English-German Conversation
  26. 26. Video to show how the system work: One side speaks in English, another side in German Translator Tool | Demo
  27. 27. Video to show how the system work: One side speaks in English, another side in German Video chat interface Translator Tool | Interface Components
  28. 28. Video to show how the system work: One side speaks in English, another side in German Closed Caption (CC) Translator Tool | Interface Components
  29. 29. Video to show how the system work: One side speaks in English, another side in German Translated Text-to-Speech (TTS) Audio Translator Tool | Interface Components
  30. 30. Video to show how the system work: One side speaks in English, another side in German Headset Microphone Translator Tool | Interface Components
  31. 31. Translator Tool Study Method Quantitative Analysis Content Analysis
  32. 32. Translator Tool Study Method Quantitative Analysis Content Analysis
  33. 33. Study Method | Participants 8 French-German pairs (N=16 participants) 15 English-German pairs (N=30 participants)
  34. 34. Study Method | Participants 8 French-German pairs (N=16 participants) 15 English-German pairs (N=30 participants) No common language With common language (English) All participants could speak English reasonably well.
  35. 35. Study Method | Participants 8 French-German pairs (N=16 participants) 15 English-German pairs (N=30 participants) With common language (English) I will mainly talk about this study No common language
  36. 36. Study Method | Conversation Task Olivia trainierte für den Tanzwettbewerb. (Olivia was practicing for the dance-off) Conversation Task German Speaker French Speaker
  37. 37. A starting sentence was provided to the participant Study Method | Conversation Task Olivia trainierte für den Tanzwettbewerb. (Olivia was practicing for the dance-off) Conversation Task German Speaker French Speaker Translate
  38. 38. Study Method | Conversation Task Olivia trainierte für den Tanzwettbewerb. (Olivia was practicing for the dance-off) Je ne savais pas Olivia dansé. Combien de temps elle s'y adonne ? (I didn’t know Olivia danced. How long has she been practicing?) Conversation Task German Speaker Translate French Speaker
  39. 39. Study Method | Conversation Task Olivia trainierte für den Tanzwettbewerb. (Olivia was practicing for the dance-off) Seit sechs Jahren (For six years) C'est une longue période. (That’s a long time.) Conversation Task German Speaker … French Speaker Je ne savais pas Olivia dansé. Combien de temps elle s'y adonne ? (I didn’t know Olivia danced. How long has she been practicing?)
  40. 40. We asked participants to perform 9 conversation tasks (~3 min ea.) in their respective languages and an additional conversation task in English The goal of each task was to collaboratively construct a coherent story Study Method | Conversation Task Tasks were conducted using three different settings of the translator tool Conversation Task
  41. 41. Study Method | Interface Settings Closed Caption & Text-to-Speech (CC & TTS) CC Closed Caption only (CC) CC Text-to-Speech only (TTS) Interface Settings
  42. 42. Closed Caption & Text-to-SpeechClosed Caption Text-to-Speech With Closed Caption and Text-to-Speech CC CC
  43. 43. Closed Caption & Text-to-SpeechClosed Caption Text-to-Speech CC CC
  44. 44. Closed Caption only Closed Caption & Text-to-SpeechClosed Caption Text-to-Speech CC CC
  45. 45. Closed Caption & Text-to-SpeechClosed Caption Text-to-Speech CC CC
  46. 46. Text-to-Speech only Closed Caption & Text-to-SpeechClosed Caption Text-to-Speech CC CC
  47. 47. Closed Caption & Text-to-SpeechClosed Caption Text-to-Speech CC CC
  48. 48. Closed Caption and Text-to-Speech CC Closed Caption CC Text-to-Speech Round 1 Study Method | Interface Settings Interface Settings Order was permuted for each pair for counter balancing
  49. 49. CC CCRound 1 Study Method | Interface Settings Interface Settings CC CC CC CC Round 2 Round 3 9 tasks with different interface settings and different starting sentences
  50. 50. CC CCRound 1 Study Method | Interface Settings Interface Settings CC CC CC CC Round 2 Round 3 Eng A baseline English task to compare with/without translation settings
  51. 51. CC CCRound 1 Study Method | Data Data CC CC CC CC Round 2 Round 3 Post-task survey (after every task) E.g., “We had a successful conversation.” Strongly Agree Strongly Disagree Eng
  52. 52. Eng CC CCRound 1 Study Method | Data Data CC CC CC CC Round 2 Round 3 Post-round survey (after every 3 tasks) Interface setting preference ranking CC & TTS: __________, CC: __________, TTS: __________Best Neutral Worst
  53. 53. CC CCRound 1 Study Method | Data Data CC CC CC CC Round 2 Round 3 Eng Post-session survey and Interview
  54. 54. Translator Tool Study Method Quantitative Analysis Content Analysis
  55. 55. Translator Tool Study Method Quantitative Analysis Content Analysis • Overall interface setting preferences • Whether people became used to using the translator tool
  56. 56. Three interface settings Three rounds (nine tasks) Two languages 3 x 3 x 2 mixed design study With-in subject With-in subject Between subject Quantitative Analysis | Analysis Method Analysis Method
  57. 57. Transformed ordinal data in survey responses with aligned rank transformation We analyzed the data with restricted maximum likelihood model Quantitative Analysis | Analysis Method Analysis Method
  58. 58. Interface Preference Results Quantitative Analysis | Interface Preference Results Which interface setting did you favor the most and least? Closed Caption & Text-to-SpeechClosed Caption only Text-to-Speech only CC CC
  59. 59. 59% 35% 6% 0% 25% 50% 75% 100% Closed Caption & Text-to- Speech Closed Caption only Text-to-Speech only Quantitative Analysis | Interface Preference Results Most Favored Interface Setting by French-German Group CC CC Interface Settings Percentage of people who favored the interface setting
  60. 60. 59% 35% 6% 0% 25% 50% 75% 100% Closed Caption & Text-to- Speech Closed Caption only Text-to-Speech only Quantitative Analysis | Interface Preference Results Most Favored Interface Setting by French-German Group Interface settings with closed caption were preferred CC CC
  61. 61. 59% 35% 6% 0% 25% 50% 75% 100% Closed Caption & Text-to- Speech Closed Caption only Text-to-Speech only Quantitative Analysis | Interface Preference Results Most Favored Interface Setting by French-German Group CC CC 24% more people preferred to have text-to-speech
  62. 62. 59% 35% 6% 0% 25% 50% 75% 100% Closed Caption & Text-to- Speech Closed Caption only Text-to-Speech only Quantitative Analysis | Interface Preference Results Most Favored Interface Setting by French-German Group CC CC Set the Closed Caption & Text-to-Speech to the default, but allow users to turn on/off setting
  63. 63. 2.2 2.2 2.7 4.8 1 2 3 4 5 Round 1 Round 2 Round 3 English Overall Conversation Quailty Perceived Conversation Quality over Rounds Round Average 5-point Likert scale rating Quantitative Analysis | Perceived Conversation Quality 5-point Likert scale rating (higher is better) Rating
  64. 64. 2.2 2.2 2.7 4.8 1 2 3 4 5 Round 1 Round 2 Round 3 English Overall Conversation Quailty p < 0.01 p < 0.01 Participants felt that their conversation quality improved Perceived Conversation Quality over Rounds Quantitative Analysis | Perceived Conversation Quality F2,112=6.275, p<0.01; Error bars are standard errors 5-point Likert scale score (higher is better) Rating
  65. 65. 2.2 2.2 2.7 4.8 1 2 3 4 5 Round 1 Round 2 Round 3 English Overall Conversation Quailty Perceived conversation quality is not as good as that of English task Perceived Conversation Quality over Rounds Quantitative Analysis | Perceived Conversation Quality 5-point Likert scale score (higher is better) Rating
  66. 66. Translator Tool Study Method Quantitative Analysis Content Analysis
  67. 67. Translator Tool Study Method Quantitative Analysis Content Analysis • Why people were satisfied or dissatisfied with the experience • How people adapted to using the translator tool
  68. 68. Interview transcripts and survey responses were coded with a content analysis method; recurring themes were extracted and grouped together Post-session interviews were transcribed by authors Content Analysis | Method Method
  69. 69. 62.5% of the participants noted that proper noun recognition errors hindered the conversation. Content Analysis | Proper Noun Recognition Errors Proper Noun Recognition Errors
  70. 70. ‘Ava’, there was no way it was getting ‘Ava’ no matter how many times I tried. So in German, when you say ‘but,’ it sounds very similar. So that’s what it was picking up. French-German Pair 4, German speaker Content Analysis | Proper Noun Recognition Errors
  71. 71. ‘Ava’, there was no way it was getting ‘Ava’ no matter how many times I tried. So in German, when you say ‘but,’ it sounds very similar. So that’s what it was picking up. French-German Pair 4, German speaker Content Analysis | Proper Noun Recognition Errors Lack of an adaptation technique
  72. 72. Content Analysis | Grammar and Word Order Errors 37.5% of the participants noted that grammatical errors and misplacement of words were problematic. Grammar and Word Order Errors
  73. 73. French-German Pair 3, German speaker Because [...] German sentence structure is so different, [translated] sentence wouldn’t make any sense when [synthesized speech] said it, but when you see it on the screen, you can kind of reorganize the words in the way it supposed to go. And then it makes sense. Content Analysis | Grammar and Word Order Errors
  74. 74. French-German Pair 3, German speaker Because [...] German sentence structure is so different [translated] sentence wouldn’t make any sense when [synthesized speech] said it, but when you see it on the screen, you can kind of reorganize the words in the way it supposed to go. And then it makes sense. Content Analysis | Grammar and Word Order Errors People could find a fallback strategy
  75. 75. Content Analysis | Information from Original Speech 31.3% of the participants noted that original speech was useful. Information from Original Speech
  76. 76. [Having partner’s original voice] humanizes the relationships as well, because if you only get the robotic voice, I think it just cuts you off from the person you are engaged with. Whereas you talk, then it feels much more human relationship. French-German Pair 1, French speaker Content Analysis | Information from Original Speech An original speech can be useful even when a person do not know the language
  77. 77. Content Analysis | Forgiving 31.3% of the participants mentioned that they forgave the errors of the translator tool Forgiving
  78. 78. My sentence became shorter, I pronounced more clearly, I was speaking ... I mean, if I were speaking to someone who’s not a native German, and if I didn’t have a translator, then I would also slow down, obviously. French-German Pair 5, German speaker Content Analysis | Forgiving
  79. 79. French-German Group English-German Group Difference in Language Groups
  80. 80. My partner and I would begin to speak then the translation from the previous statement would catch up, translation and speaker would then all be talking at the same time. English-German Pair 4, English speaker Content Analysis | Speech Overlap
  81. 81. Content Analysis | Speech Overlap Speech Overlap 25.0% of the participants mentioned that they speech overlap was a problem French-German Group
  82. 82. Content Analysis | Speech Overlap Speech Overlap 25.0% 50.0% Participants from the English-German group perceived speech overlap as a more severe problem French-German Group English-German Group This was partly because German speakers could respond without seeing/hearing translation.
  83. 83. Design Guidelines
  84. 84. Design Guidelines • Support Users’ Adaptation for Speech Production • Offer Fallback Strategies with Non-verbal Input • Support Comprehension of Messages • Support Users’ Turn Taking
  85. 85. Limitation German Speakers’ English Proficiency
  86. 86. Limitation Interpreter Baseline Task
  87. 87. Hi! 안녕! Future Work Analysis of conversation Conduct follow up summative studies
  88. 88. Summary We used Skype Translator to elicit interaction problems in using spoken language translation for interlingual conversation. Our analysis revealed how people adjusted (or could not adjust) their behaviors to overcome the problems. makeability lab
  89. 89. Kotaro Hara Shamsi T. Iqbal Microsoft Skype Translator team makeability lab
  90. 90. @kotarohara_en Questions? makeability lab
  91. 91. makeability lab Image and Icon Credit - Friendlies by Mo Riza (https://www.flickr.com/photos/moriza/2565606353) - 133/2011 euroscola by rosipaw (https://www.flickr.com/photos/rosipaw/5768035647/) - Japantimes (http://www.japantimes.co.jp/news/2014/05/08/world/social-issues-world/births-fall-economies-may-falter) - Windows8core.com (http://www.windows8core.com/bing-translator-app-for-windows-8rt-lands-in-store-review/) - Jigsaw puzzle by James Petts - Speech bubble from Wikimedia Commons (http://commons.wikimedia.org/wiki/File:Speech_bubble.svg) - Head silhouette from Channelweb (http://www.channelweb.co.uk/crn-uk/news/2244637/good-week-bad-week) - Microphone from Uxrepo.com (http://uxrepo.com/icon/mic-by-mfg-labs) - Document from IconArchive (http://www.iconarchive.com/show/mono-general-2-icons-by-custom-icon-design/document-icon.html) - Funny Signs from Fanpop (http://www.fanpop.com/clubs/picks/images/1771904/title/funny-signs-photo) - Compass Zoomed by Gwgs (https://www.flickr.com/photos/gwgs/346806882) - Beaker by Emily van den Heever (https://thenounproject.com/EmvdHeever/)

×