Download as PDF, PPTX
![Hierarchical SMT - phrase-table
[X] → shall be passing on X1 X2 | werde X1 X2 aush¨andigen
[X] → shall be passing on X3 | werde X3 aush¨andigen
[X] → to you | Ihnen
[X] → some comments | die entsprechenden Anmerkungen
[X] → to you some comments | Ihnen die entsprechenden Anmerkungen
Learn a bilingual (synchronous) set of context-free rules about how to
translate F into E
Porto (14/05/2013) SMT and PE 44 / 67](https://image.slidesharecdn.com/luciaspecia-smteps-edio-130528063914-phpapp02/85/Lucia-Specia-SMT-e-pos-edicao-44-320.jpg)
![Quality of post-editing
• Can MT+PE (possibly +revision) produce the same level of quality
as HT+revision?
• Dependent on the quality of MT systems: variable for language pair,
text type/genre & domain, quality of source
• Dependent on quality requirements
• Different outcomes:
Autodesk reports lower error rates with MT+PE [Aut11]
Tilde observed higher error rate with MT+PE [SPSV11] -
under-resourced highly-inflected languages: lower MT quality
Porto (14/05/2013) SMT and PE 51 / 67](https://image.slidesharecdn.com/luciaspecia-smteps-edio-130528063914-phpapp02/85/Lucia-Specia-SMT-e-pos-edicao-51-320.jpg)
![Aspects involved in post-editing effort
• Temporal: time to post-edit (variable across translators)
• Technical: no./% edits made, keys pressed
• Cognitive: time, pauses, movements
Studies on PE process:
• Is PEMT faster than HT? Yes
• Is PEMT less keyboard intensive than HT? Yes
• Is PEMT less cognitively demanding than HT?
Cognitive effort is the most difficult to estimate: active research area
[Kop12, KARS12]
• PE more tedious/tiring? Mixed opinions
Check work by
Michael Carl - CBS (Translog) and Sharon O’Brien - DCU (eye-tracking)
Porto (14/05/2013) SMT and PE 53 / 67](https://image.slidesharecdn.com/luciaspecia-smteps-edio-130528063914-phpapp02/85/Lucia-Specia-SMT-e-pos-edicao-53-320.jpg)
![What can we get from PET?
Comparison between HT and multiple MT systems [dSAS11]
• PET to compare 3 MT systems vs translation from scratch
• Sitcom and movie subtitles:
Translating from scratch can be 73% slower than post-editing
SMT systems (Google and Moses) performed the best
How often a system produced an output that was faster to PE than other systems:
System Google Moses Systran Trados
Google - 139 161 187
Moses 69 - 122 164
Systran 69 106 - 145
Trados 48 67 89 -
How often post-editing a system output was faster than translating:
System Faster than human translation
Google 94%
Moses 86.8%
Systran 81.20%
Trados 72.40%
Porto (14/05/2013) SMT and PE 62 / 67](https://image.slidesharecdn.com/luciaspecia-smteps-edio-130528063914-phpapp02/85/Lucia-Specia-SMT-e-pos-edicao-62-320.jpg)
![What can we get from PET?
Evaluation of an MT system for translation of subtitles (space constraints)
[AdSS12]
• Add compression constraints to an MT system to generate length
compliant subtitles
• PET to guide post-editing according to length and time requirements
for every unit
Show space limitation (attribute)
Change colour of translation if too long
Offer shorter paraphrases
• PET to evaluate edit distance and length
System
Dexter How I Met.. Terra Nova
TER ↓ LENGTH TER ↓ LENGTH TER ↓ LENGTH
Mosest 30.3 116.0 20.0 108.5 33.8 120.2
Google 63.6 156.5 52.8 144.3 63.1 152.1
MosesLP2 29.5 115.5 21.0 109.1 33.4 119.3
MosesLP1 28.3 115.8 20.7 110.0 34.8 119.8
Porto (14/05/2013) SMT and PE 63 / 67](https://image.slidesharecdn.com/luciaspecia-smteps-edio-130528063914-phpapp02/85/Lucia-Specia-SMT-e-pos-edicao-63-320.jpg)
Lucia Specia - SMT e pós-edição
The document discusses statistical machine translation (SMT), its challenges, and the process of post-editing as presented in a conference. It highlights various aspects of SMT including word-based and phrase-based models and the importance of understanding ambiguity, structural divergences, and contextual nuances in language. Additionally, it outlines the methodology for learning translation models and the significance of probabilistic approaches in generating accurate translations.
More Related Content
![[BOOK SAMPLES] MASTER THE ART OF NEC](https://cdn.slidesharecdn.com/ss_thumbnails/mastertheartofnecsamples-181015022810-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Lucia Specia - SMT e pós-edição
Lucia Specia - SMT e pós-edição
- 1. Statistical Machine Translationand Post-editing I Conferˆencia Internacional de Tradu¸c˜ao e Tecnologia Porto, 14 Maio 2013 Lucia Specia (Thanks to Wilker Aziz for many of the SMT slides) Department of Computer Science University of Sheffield http://www.dcs.shef.ac.uk/~lucia/ Porto (14/05/2013) SMT and PE 1 / 67
- 2. Overview • Machine Translation- task and challenges • Statistical Machine Translation Word-based models Phrase-based models Syntax-based models • Post-editing: Process and research Practice with PET Porto (14/05/2013) SMT and PE 2 / 67
- 3. The task ofMachine Translation (MT) The boy ate an apple O menino comeu uma ma¸ca BUT He said that the bottle floated into the cave ? Dijo que la botella entro a la cueva flotando 虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。 However , the sky remained clear under the strong north wind . Porto (14/05/2013) SMT and PE 3 / 67
- 4. Challenges in MT:Ambiguity • Lexical ambiguity: different words meanings may translate differently e.g. book the flight ⇒ reservar read the book ⇒ libro e.g. kill a man ⇒ matar kill a process ⇒ acabar • Syntactic ambiguity: e.g. John hit the dog with the stick ⇒ John golpeo el perro con el palo ⇒ John golpeo el perro que tenia el palo • Pronoun resolution: e.g. The computer outputs the data; it is fast ⇒ La computadora imprime los datos; es rapida VS. e.g. The computer outputs the data; it is stored in ascii ⇒ La computadora imprime los datos; estan almacendos en ascii Porto (14/05/2013) SMT and PE 4 / 67
- 5. Challenges in MT:Divergences • Structural divergences: e.g. The bottle floated into the cave ⇒ La botella entro a la cueva flotando (The bottle entered the cave floating) • Different word orders: English is: subject – verb – object Japanese is: subject – object – verb English: IBM bought Lotus yesterday Japanese: IBM Lotus bought yesterday Porto (14/05/2013) SMT and PE 5 / 67
- 6. Challenges: Non-literal meaning •Idioms e.g. He finally kicked the bucket at the hospital ⇒ Ele finalmente chutou o balde no hospital VS. ⇒ Ele finalmente bateu as botas no hospital • Multi-word expressions e.g. Do take the long waiting list for organ donation in this country into account ⇒ Considere a longa lista de espera para doa¸c˜ao de ´org˜aos neste pa´ıs Porto (14/05/2013) SMT and PE 6 / 67
- 7. Requires deep languageprocessing • One of the most challenging problems in NLP • Requires Full understanding of source text (analysis) Generation of a coherent, accurate, fluent target text (synthesis) Porto (14/05/2013) SMT and PE 7 / 67
- 8. Statistical Machine Translation P(French|English) •Statistical Machine Translation (SMT) learn how to generate translations using statistical methods • Early 1990s by IBM. Adopted by Google, etc. • Idea is older: Warren Weaver (1949) When I look at an article in Russian, I say: “This is really written in English, but it has been coded in some strange symbols. I will now proceed to decode.” Inspired by WWII code-breaking Approach not feasible with early computers – idea not pursued Porto (14/05/2013) SMT and PE 8 / 67
- 9. Probabilistic Model • Outputof decoder depends probabilistically on the input • To translate French (F) into English (E): Given a French sentence F search for English sentence E∗ that maximises P(E|F) Porto (14/05/2013) SMT and PE 9 / 67
- 10. Probabilistic Model • FindEnglish sentence that maximises P(E|F), i.e. E∗ = argmax E P(E|F) = argmax E P(E)·P(F|E) P(F) Bayes Rule P(F) constant across different E, so: E∗ = argmax E P(E) · P(F|E) drop constant divisor P(F) Porto (14/05/2013) SMT and PE 10 / 67
- 11. Noisy Channel Model& SMT (ctd) E∗ = argmax E P(E|F) = argmax E P(E) · P(F|E) • Why not model P(E|F) directly? Would need very good probability estimates Divide and conquer! • Decomposition to P(E) · P(F|E) allows one to use less reliable probabilities P(E) worries about good English — fluency • ensures that E has words in the right order and fit together P(F|E) worries about French that translates English — faithfulness • ensures that E has words that generally translate F P(E) and P(F|E) can be trained independently Porto (14/05/2013) SMT and PE 11 / 67
- 12. Main Components forTranslation (F → E) • Translation model (TM): P(F|E) Faithfulness: TMs learned by analysing large amounts of parallel text • Judge which (F, E) pairs are likely to be translations of each other • Language model (LM): P(E) Fluency: LMs created from large volume of (fluent) target language sentences • Learn which word sequences are common/more probable in E • A Decoder: (argmax) Search algorithm to find E∗ • V. large search space — heuristic search approach used Porto (14/05/2013) SMT and PE 12 / 67
- 13. Learning Translation Models- P(F|E) • Requires a bilingual corpus e.g. European/Canadian/Hong Kong parliaments, subtitles, Bible • Bilingual corpus must be sentence-aligned: • Can we estimate P(F|E) from whole sentences? Porto (14/05/2013) SMT and PE 13 / 67
- 14. Learning Translation Models- Word-based SMT • Break sentences into smaller units: words are a good starting point • Learn translation probabilities by word aligning a sentence-aligned corpus: Zenish Uh useh Uh jejje Yiguo useh English A home A garden I arrived home Porto (14/05/2013) SMT and PE 14 / 67
- 15. Learning Translation Models- Word-based SMT • Break sentences into smaller units: words are a good starting point • Learn translation probabilities by word aligning a sentence-aligned corpus: Zenish Uh useh Uh jejje Yiguo useh • The same word happens in source 1 and 3 English A home A garden I arrived home Porto (14/05/2013) SMT and PE 15 / 67
- 16. Learning Translation Models- Word-based SMT • Break sentences into smaller units: words are a good starting point • Learn translation probabilities by word aligning a sentence-aligned corpus: Zenish Uh useh Uh jejje Yiguo useh • Could we expect the same in the target side? English A home A garden I arrived home Porto (14/05/2013) SMT and PE 16 / 67
- 17. Learning Translation Models- Word-based SMT • Break sentences into smaller units: words are a good starting point • Learn translation probabilities by word aligning a sentence-aligned corpus: Zenish Uh useh Uh jejje Yiguo useh English A home A garden I arrived home • useh = home Porto (14/05/2013) SMT and PE 17 / 67
- 18. Learning Translation Models- Word-based SMT • Break sentences into smaller units: words are a good starting point • Learn translation probabilities by word aligning a sentence-aligned corpus: Zenish Uh useh Uh jejje Yiguo useh • What about the contexts? English A home A garden I arrived home Porto (14/05/2013) SMT and PE 18 / 67
- 19. Learning Translation Models- Word-based SMT • Break sentences into smaller units: words are a good starting point • Learn translation probabilities by word aligning a sentence-aligned corpus: Zenish Uh useh Uh jejje Yiguo useh English A home A garden I arrived home • We can align them: Yiguo = I; Yiguo = arrived; Uh = A Porto (14/05/2013) SMT and PE 19 / 67
- 20. Learning Translation Models- Word-based SMT • Break sentences into smaller units: words are a good starting point • Learn translation probabilities by word aligning a sentence-aligned corpus: Zenish Uh useh Uh jejje Yiguo useh English A home A garden I arrived home • And reuse this knowledge to align more sentences: Uh = A Porto (14/05/2013) SMT and PE 20 / 67
- 21. Learning Translation Models- Word-based SMT • Break sentences into smaller units: words are a good starting point • Learn translation probabilities by word aligning a sentence-aligned corpus: Zenish Uh useh Uh jejje Yiguo useh English A home A garden I arrived home • And again: jejje = garden Porto (14/05/2013) SMT and PE 21 / 67
- 22. Learning Translation Models- Word-based SMT (ctd) Word-alignment: • Identify correspondences between two languages at the word level • Basis for word-based SMT, first step for other approaches • Alignment can be learned using Expectation Maximization (EM) Treat all alternative word alignments as equally likely Observe across sentences that Zenish useh often links to English home • Increase probability of this word pair aligning • Has knock-on effect for alignment of other words Iteratively redistribute probabilities, until they identify most probable links for each word (convergence) Word alignment commonly done using the IBM Models 1-5 Porto (14/05/2013) SMT and PE 22 / 67
- 23. Learning Translation Models- Word-based SMT (ctd) • IBM models produce a probabilistic dictionary for the whole parallel corpus: contras contras 0.0285714 contras los 0.0000011 contras esos 0.0000812 richer enriquecida 0.0476190 richer mucho 0.0000502 richer enriqueci 0.0769231 richer variado 0.0066225 Porto (14/05/2013) SMT and PE 23 / 67
- 24. Learning Translation Models- Word-based SMT (ctd) Back to the translation model: At translation (decoding time), for a new sentence to translate, take the set of translations that jointly maximise the whole translation probability E∗ = argmax E P(F|E) • Generates alternative translations using different translation options for words and different word orders • Selects the one with maximum score: no notion of context. How about ambiguity? What about the context and the fluency in the target language? Porto (14/05/2013) SMT and PE 24 / 67
- 25. Learning Language Models- Word-based SMT Language model: P(E) E∗ = argmax E P(F|E) · P(E) • Takes different translation options, tests how “common” each of them are in the target language as a whole sentence (some context) • P(E) = probability of strings E based on relative frequencies in a large corpus of language E Google the house is small 144,000 the house is little 85,300 small house 2,330,000 little house 3,330,000 Porto (14/05/2013) SMT and PE 25 / 67
- 26. Learning Language Models- Word-based SMT (ctd) • Model P(E) by decomposing into single-word probabilities given previous words, under Markov assumption: only the previous n-1 words matter for predicting a word. E.g for trigram models, n = 3 P(e1) · P(e2|e1) · P(e3|e1, e2) · P(e4|e2, e3) · · · P(en|en−2, en−1) • Relative frequencies to compute these probabilities. E.g. trigrams: P(e3|e1, e2) = count(e1e2e3) count(e1e2) P(little|house, is) = count(house is little) count(house is) Porto (14/05/2013) SMT and PE 26 / 67
- 27. Word-based SMT –Limitations • It is difficult to do alignment, and hence learn a TM, for languages that are structurally quite different, with different words orders Considering all possible word orders – too costly Poor reordering model: flexibility vs noise • Some languages may have different notions of what counts as a word Donaydampfshiffahrtsgesellschaftskapitaenskajuetenschluesseloch The keyhole of the door of the cabin of the captain of a steamship company operating on the Danubea a http://www.chebucto.ns.ca/~waterbuf/longwrds.html Porto (14/05/2013) SMT and PE 27 / 67
- 28. Phrase-based SMT • Leadingapproach since 2003 No voy1 a la2 casa3 → I am not going1 to the2 house3 it seems to1 me2 → me1 parece2 Je1 ne vais pas2 `a la3 maison4 → I1 am not going2 to the3 house4 Eu1 sinto saudade de vocˆe2 → I1 miss you2 I1 miss you2 → Eu1 sinto sua falta2 natuerlich1 hat2 John3 spass am4 spiel5 → of course1 John2 has3 fun with the4 game5 • More intuitive and reliable alignments Account for reordering within the phrases Phrases can still be reordered • Can we learn phrase translation probability distribution from the parallel corpus using EM? 1 src-tgt sentence → O(n4 ) phrase pairs Porto (14/05/2013) SMT and PE 28 / 67
- 29. Phrase-based SMT -Phrases from word alignments • Extract phrase pairs that are consistent with the word alignment • Phrase: sequence of tokens, not linguistically motivated • Word alignment produced by any of the IBM Models - only done once reanudaci´on del per´ıodo de sesiones resumption of the session 1 resumption ↔ reanudaci´on 2 of the ↔ del 3 session ↔ per´ıodo de sesiones 4 resumption of the ↔ reanudaci´on del 5 of the session ↔ del sesiones 6 resumption of the session ↔ reanudaci´on del sesiones Porto (14/05/2013) SMT and PE 29 / 67
- 30. Phrase-based SMT -Computing phrase probability distributions 1 Extract phrase pairs from word aligned parallel corpus √ 2 Extract counts of those phrases from large parallel corpus: φ(¯f |¯e) = count(¯f , ¯e) count(¯e) 3 Store phrases and their probabilities in a phrase table Probabilistic dictionary of phrases Porto (14/05/2013) SMT and PE 30 / 67
- 31. Phrase-based SMT -Limitation of Noisy Channel Model E∗ = argmax E P(F|E) · P(E) • Which component is more important? P(F|E) ? P(E) ? • Depends on the corpus used (more parallel data makes P(F|E) more reliable), language-pair, etc. Porto (14/05/2013) SMT and PE 31 / 67
- 32. Phrase-based SMT -Log-linear Model (ctd) Assign weights λi to components (feature functions) hi (f, e): e∗ = argmax e n i=1 λi hi (f, e) Components 1 P(E) 2 P(F|E) Weights 1 λP(E) 2 λP(F|E) Feature Functions 1 h1 = logP(E) 2 h2 = logP(F|E) Benefits 1 Extensible: other components are used: e.g. reordering, word and phrase penalties 2 Weights can be tuned, i.e., learned from examples to minimise error in a subset of the parallel corpus, e.g. MERT Porto (14/05/2013) SMT and PE 32 / 67
- 33. Phrase-based SMT -Decoding (ctd) Search problem: segmenting source sentence and selecting phrase translations: • All phrases matching source words are selected from the phrase table J’ ai les yeux noirs . I have the eyes black . me has them eye dark , I have eyes espresso ! I am the eyes somber . I did some black eyes . I had black eyes . I have black eyes . black eyes I have . • The goal of the decoder is to select the phrases whose combination (in a given order) yields the highest score for the log-linear model Porto (14/05/2013) SMT and PE 33 / 67
- 34. Phrase-based SMT -Decoding (ctd) Searching = incrementally construct the translation hypotheses by trying out several possibilities • Generating target words in sequence, from left to right • Incrementally compute the overall log-linear model score for each hypothesis • Heuristics to prune search space, e.g. stack-based beam search Use one stack for of every length of partial hypotheses Limited space for partial hypotheses, only keep those that are promising, e.g. model score is close to that of the best partial hypothesis so far Porto (14/05/2013) SMT and PE 34 / 67
- 35. Phrase-based SMT -Decoding (ctd) Search space Hypothesis • Covered source words • Target (output) words • Model score (using the combined probabilities) Porto (14/05/2013) SMT and PE 35 / 67
- 36. Hierarchical and Syntax-basedSMT • PBSMT still has trouble with long-distance reorderings • Ways of bringing structure and linguistic knowledge into the transfer rules of the phrase table Porto (14/05/2013) SMT and PE 36 / 67
- 37. Hierarchical SMT -Motivation Introduce structure into phrase-based SMT models to deal with long-distance reordering Ich werde Ihnen die entsprechenden Anmerkungen aush¨andigen I shall be passing on to you some comments Porto (14/05/2013) SMT and PE 37 / 67
- 38. Hierarchical SMT -Motivation Introduce structure into phrase-based SMT models to deal with long-distance reordering Ich werde Ihnen die entsprechenden Anmerkungen aush¨andigen I shall be passing on to you some comments • How can we get a phrase for shall be passing on? Porto (14/05/2013) SMT and PE 38 / 67
- 39. Hierarchical SMT -Motivation Introduce structure into phrase-based SMT models to deal with long-distance reordering Ich werde Ihnen die entsprechenden Anmerkungen aush¨andigen I shall be passing on to X you X some X comments X • How can we get a phrase for shall be passing on? • We cannot, unless we get to you some comments along Porto (14/05/2013) SMT and PE 39 / 67
- 40. Hierarchical SMT -Motivation Introduce structure into phrase-based SMT models to deal with long-distance reordering Ich werde Ihnen die entsprechenden Anmerkungen aush¨andigen I shall be passing on to you some comments • How can we get a phrase for shall be passing on? • We cannot, unless we get to you some comments along • Unless we replace all those words by a variable Porto (14/05/2013) SMT and PE 40 / 67
- 41. Hierarchical SMT -Motivation (ctd) shall be passing on to you some comments werde Ihnen die entsprechenden Anmerkungen aush¨andigen shall be passing on to you some comments werde Ihnen die entsprechenden Anmerkungen aush¨andigen shall be passing on X werde X aush¨andigen Porto (14/05/2013) SMT and PE 41 / 67
- 42. Hierarchical SMT -basics Learnt from word-aligned parallel corpora in the same way as before • Based on the fact that language has recursive structures • Phrases within other phrases treated as nonterminals: replaced by X • No linguistic constraints added - yet, some structure Porto (14/05/2013) SMT and PE 42 / 67
- 43. Hierarchical SMT -basics (ctd) shall be passing on to you some comments werde Ihnen die entsprechenden Anmerkungen aush¨andigen shall be passing on X1 some comments werde X1 die entsprechenden Anmerkungen aush¨andigen shall be passing on X1 X2 werde X1 X2 aush¨andigen Porto (14/05/2013) SMT and PE 43 / 67
- 44. Hierarchical SMT -phrase-table [X] → shall be passing on X1 X2 | werde X1 X2 aush¨andigen [X] → shall be passing on X3 | werde X3 aush¨andigen [X] → to you | Ihnen [X] → some comments | die entsprechenden Anmerkungen [X] → to you some comments | Ihnen die entsprechenden Anmerkungen Learn a bilingual (synchronous) set of context-free rules about how to translate F into E Porto (14/05/2013) SMT and PE 44 / 67
- 45. Syntax-based SMT • Hierarchicalmodels are not very informative (Xs) and suffer from an exponential number of rules • Syntax-based SMT overcomes these limitations by using linguistic categorise to label nodes • A syntactic parser is required at pre-processing time on at least one language (the other could have Xs) • Idea: Learn a bilingual (synchronous) set of linguistically informed context-free rules about how to translate F into E Rules are learnt from the word-alignment, constrained by syntactic categories Porto (14/05/2013) SMT and PE 45 / 67
- 46. Syntax-based SMT (ctd) S PRP J’ VP VB ai NP DT les NN yeux JJ noirs S PRP I VP VB have NP JJ black NN eyes J’ai les yeux noirs I have black eyes Standard constraints on rule construction: • Single nonterminal on the left • Consistent with word-alignment • Nonterminals on the right must align one-to-one Porto (14/05/2013) SMT and PE 46 / 67
- 47. Syntax-based SMT (ctd) Grammar PRP→ J’ | I JJ → noirs | black NP → les yeux JJ | JJ eyes VP → ai NP | have NP S → PRP VP | PRP VP Decoding becomes a (probabilistic) parsing problem! Porto (14/05/2013) SMT and PE 47 / 67
- 48. Summary on SMT •Adding linguistic knowledge to SMT is a trend: Stronger theoretical motivation Deals with reordering, sparsity, ambiguity Depends on (quality of) parsers Higher complexity • Performance: Hard to quantify in absolute terms Most language pairs → phrase-based or hierarchical SMT does better Language pairs with long-distance reordering → syntax-based SMT does better • Other open issues: Limited use of linguistic information - at most syntax, some attempts to use shallow semantics - not yet there Limited use of context - sentence-level Domain discrepancies - mistranslations, unknown words • Overall, to reach publishable quality, human checking/revision necessary: post-editing Porto (14/05/2013) SMT and PE 48 / 67
- 49. Definitions Note: Some slidesare based on those by Sharon O’Brien (tutorial on MT post-editing at AMTA-2010). What is post-editing? • Same as translation? • Same as revision? • A new profession? New skills? Who does the post-editing? What are the quality expectations? Porto (14/05/2013) SMT and PE 49 / 67
- 50. Degrees/types of post-editing •Light (fast/rapid/gist) post-editing: essential corrections • Full post-editing: more corrections, higher quality, closer to revision • Monolingual post-editing: no access to source text Decided by: • User requirements • Volume • Quality expectations • Turn-around time, etc. Porto (14/05/2013) SMT and PE 50 / 67
- 51. Quality of post-editing •Can MT+PE (possibly +revision) produce the same level of quality as HT+revision? • Dependent on the quality of MT systems: variable for language pair, text type/genre & domain, quality of source • Dependent on quality requirements • Different outcomes: Autodesk reports lower error rates with MT+PE [Aut11] Tilde observed higher error rate with MT+PE [SPSV11] - under-resourced highly-inflected languages: lower MT quality Porto (14/05/2013) SMT and PE 51 / 67
- 52. Ways of measuringquality of MT for PE • Automatic MT evaluation metrics ? BLEU/TER/METEOR... - automatic, but do not correlate well with PE effort • Error counts: compare source with raw MT output - manual, costly Can we use the “standard” error typologies for HT? Do MT and HT make same mistakes? QTLaunchPad project Multidimensional Quality Metrics for MT and HT, for manual and (semi-)automatic evaluation (QE): http://www.qt21.eu/launchpad/ • % Changes made: compare post-edited text with raw MT output - automatic • Time to make changes: record time to fix raw MT - automatic • Estimated effort: take source text and raw MT output and estimate PE effort prior to post-editing (QE) - automatic, less precise Is this the same as fuzzy match level in TM systems? Porto (14/05/2013) SMT and PE 52 / 67
- 53. Aspects involved inpost-editing effort • Temporal: time to post-edit (variable across translators) • Technical: no./% edits made, keys pressed • Cognitive: time, pauses, movements Studies on PE process: • Is PEMT faster than HT? Yes • Is PEMT less keyboard intensive than HT? Yes • Is PEMT less cognitively demanding than HT? Cognitive effort is the most difficult to estimate: active research area [Kop12, KARS12] • PE more tedious/tiring? Mixed opinions Check work by Michael Carl - CBS (Translog) and Sharon O’Brien - DCU (eye-tracking) Porto (14/05/2013) SMT and PE 53 / 67
- 54. Pricing models Open question.Some common practices: • Paying as fuzzy segment matches (source-based) • Paying as % of edits performed (MT&PE-based) • Paying a fee based on time spent (less common) Porto (14/05/2013) SMT and PE 54 / 67
- 55. PE Guidelines • Nostandard guidelines, but good practices (e.g. TAUS/CNGL) • Guidelines need to clearly specify level of post-editing and quality expectations • Guidelines need to clearly differentiate between essential and preferential changes • Ideally, guidelines should be system-, language-, purpose-specific Porto (14/05/2013) SMT and PE 55 / 67
- 56. General PE Guidelinesfor “good enough” quality - TAUS/CNGL • Aim for semantically correct translation. • Ensure that no information has been accidentally added or omitted. • Edit any offensive, inappropriate or culturally unacceptable content. • Use as much of the raw MT output as possible. • Basic rules regarding spelling apply. • No need to implement corrections that are of a stylistic nature only. • No need to restructure sentences solely to improve the natural flow of the text. http://www.translationautomation.com/postediting/ machine-translation-post-editing-guidelines Porto (14/05/2013) SMT and PE 56 / 67
- 57. General PE Guidelinesfor HT quality - TAUS/CNGL • Aim for grammatically, syntactically and semantically correct translation. • Ensure that key terminology is correctly translated and that untranslated terms belong to the clients list of Do Not Translate terms. • Ensure that no information has been accidentally added or omitted. • Edit any offensive, inappropriate or culturally unacceptable content. • Use as much of the raw MT output as possible. • Basic rules regarding spelling, punctuation and hyphenation apply. • Ensure that formatting is correct. http://www.translationautomation.com/postediting/ machine-translation-post-editing-guidelines Porto (14/05/2013) SMT and PE 57 / 67
- 58. IT PE Guidelines- Midori Tatsumi’s PhD thesis • What needs to be fixed: Non-translatable items, such as command and variable names, that have been translated. Please put it back to English. Inappropriately translated general IT terms. Mistranslation. Word orders that are inappropriate to the level that the sentence has become impossible or difficult to comprehend. Comprehensible but extremely unnatural or inappropriate expressions. Inappropriate postpositions and conjugations. • What are preferred to be standardised: Repetitive items, such as procedures and steps. Styles of the section titles in a user manual and help files. • What does not need to be fixed: Ending styles of bulleted items. Placeholders, such as &ProductNameShort; and &ProductNameLong; Punctuation. http://doras.dcu.ie/16062/1/SAKURA_final_revised.pdf Porto (14/05/2013) SMT and PE 58 / 67
- 59. Monolingual PE Guidelines- ACCEPT project • Try and edit the text by making it more fluent and clearer based on how you interpret its meaning. For example, try to rectify word order and spelling when they are inappropriate to the extent that the text has become impossible or difficult to comprehend • If words, phrases, or punctuation in the text are completely acceptable, try and use them (unmodified) rather than substituting them with something new and different. http://www.accept.unige.ch/Products/D6.2.2_Seminar_ Material_on_Post-Editing_Edition_2.pdf Porto (14/05/2013) SMT and PE 59 / 67
- 60. PE Technology (tools) Amyriad of tools appeared in the marked in the last couple of years: • TM tools that now integrate MT, e.g. SDL TRADOS, WordFast • MT tools (with or without TM): e.g. Systran, Google Translate Toolkit, Caitra • PE tools that can use different MT systems: MemoQ, MemSource, SmartMate, Coach, ... More and more functionalities: spell/grammar checkers, multiple MT, MT+TM, interactive MT, terminology, word/phrase alignment, etc. Limited (if any) logging of translator’s activities for productivity analysis – not enough for research. Porto (14/05/2013) SMT and PE 60 / 67
- 61. PE Technology (tools) Research-orientedtools: • Translog (not open source) • OmegaT (TM only) • Appraise (MT eval through ranking, no detailed statistics for PE) • PET: http://pers-www.wlv.ac.uk/~in1676/pet/index.html • Coming soon: open source tool promised as part of EU projects CASMACAT & MATECAT Porto (14/05/2013) SMT and PE 61 / 67
- 62. What can weget from PET? Comparison between HT and multiple MT systems [dSAS11] • PET to compare 3 MT systems vs translation from scratch • Sitcom and movie subtitles: Translating from scratch can be 73% slower than post-editing SMT systems (Google and Moses) performed the best How often a system produced an output that was faster to PE than other systems: System Google Moses Systran Trados Google - 139 161 187 Moses 69 - 122 164 Systran 69 106 - 145 Trados 48 67 89 - How often post-editing a system output was faster than translating: System Faster than human translation Google 94% Moses 86.8% Systran 81.20% Trados 72.40% Porto (14/05/2013) SMT and PE 62 / 67
- 63. What can weget from PET? Evaluation of an MT system for translation of subtitles (space constraints) [AdSS12] • Add compression constraints to an MT system to generate length compliant subtitles • PET to guide post-editing according to length and time requirements for every unit Show space limitation (attribute) Change colour of translation if too long Offer shorter paraphrases • PET to evaluate edit distance and length System Dexter How I Met.. Terra Nova TER ↓ LENGTH TER ↓ LENGTH TER ↓ LENGTH Mosest 30.3 116.0 20.0 108.5 33.8 120.2 Google 63.6 156.5 52.8 144.3 63.1 152.1 MosesLP2 29.5 115.5 21.0 109.1 33.4 119.3 MosesLP1 28.3 115.8 20.7 110.0 34.8 119.8 Porto (14/05/2013) SMT and PE 63 / 67
- 64. Hands-on activity withPET • en-pt-hiding-and-assessing.pec – bigbang-pe-2.pej • en-pt-hiding-and-assessing.pec – bigbang-ht-2.pej • fapesp.pec – fapesp-annotated.pej • fapesp-mono.pec – fapesp-annotated.pej • subs.pec – greys.08x06.v1-v1.en-br.s2.1.pej Advanced use: • Configuration file: .pec • Input/output XML files • How to build a PEJ file from raw files: pej.bat • How to interpret the results: scripts/readper.pl Porto (14/05/2013) SMT and PE 64 / 67
- 65. For more onthe research-side Workshop on Post-Editing Technology and Practice: • AMTA 2012: https://sites.google.com/site/wptp2012/ • MT Summit 2013 (deadline for papers 28/05/2013): https://sites.google.com/site/mts2013wptp/ Porto (14/05/2013) SMT and PE 65 / 67
- 66. References I Wilker Aziz,Sheila Castilho Monteiro de Sousa, and Lucia Specia. Cross-lingual sentence compression for subtitles. In The 16th Annual Conference of the European Association for Machine Translation, EAMT ’12, pages 103–110, Trento, Italy, May 2012. Autodesk. Translation and Post-Editing Productivity. In http: // translate. autodesk. com/ productivity. html , 2011. Sheila C. M. de Sousa, Wilker Aziz, and Lucia Specia. Assessing the post-editing effort for automatic and semi-automatic translations of DVD subtitles. In Conference Recent Advances in Natural Language Processing 2011, pages 97–103, Hissar, Bulgaria, 2011. Porto (14/05/2013) SMT and PE 66 / 67
- 67. References II Maarit Koponen,Wilker Aziz, Luciana Ramos, and Lucia Specia. Post-editing time as a measure of cognitive effort . In AMTA 2012 Workshop on Post-Editing Technology and Practice (WPTP 2012), pages 11–20, San Diego, USA, October 2012. Association for Machine Translation in the Americas (AMTA). Maarit Koponen. Comparing human perceptions of post-editing effort with post-editing operations. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 181–190, Montr´eal, Canada, 2012. R. Skadins, M. Purins, I. Skadina, and A. Vasiljevs. An evaluation of SMT in localization to under-resourced inflected languages. In Proceedings of the 15th Conference of the European Association for Machine Translation, Leuven, 2011. Porto (14/05/2013) SMT and PE 67 / 67