SlideShare a Scribd company logo
Statistical Machine Translation and
Post-editing
I Conferˆencia Internacional de Tradu¸c˜ao e Tecnologia
Porto, 14 Maio 2013
Lucia Specia
(Thanks to Wilker Aziz for many of the SMT slides)
Department of Computer Science
University of Sheffield
http://www.dcs.shef.ac.uk/~lucia/
Porto (14/05/2013) SMT and PE 1 / 67
Overview
• Machine Translation - task and challenges
• Statistical Machine Translation
Word-based models
Phrase-based models
Syntax-based models
• Post-editing:
Process and research
Practice with PET
Porto (14/05/2013) SMT and PE 2 / 67
The task of Machine Translation (MT)
The boy ate an apple
O menino comeu uma ma¸ca
BUT
He said that the bottle floated into the cave
? Dijo que la botella entro a la cueva flotando
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
However , the sky remained clear under the strong north wind .
Porto (14/05/2013) SMT and PE 3 / 67
Challenges in MT: Ambiguity
• Lexical ambiguity: different words meanings may translate differently
e.g. book the flight ⇒ reservar
read the book ⇒ libro
e.g. kill a man ⇒ matar
kill a process ⇒ acabar
• Syntactic ambiguity:
e.g. John hit the dog with the stick
⇒ John golpeo el perro con el palo
⇒ John golpeo el perro que tenia el palo
• Pronoun resolution:
e.g. The computer outputs the data; it is fast
⇒ La computadora imprime los datos; es rapida VS.
e.g. The computer outputs the data; it is stored in ascii
⇒ La computadora imprime los datos; estan almacendos en ascii
Porto (14/05/2013) SMT and PE 4 / 67
Challenges in MT: Divergences
• Structural divergences:
e.g. The bottle floated into the cave
⇒ La botella entro a la cueva flotando
(The bottle entered the cave floating)
• Different word orders:
English is: subject – verb – object
Japanese is: subject – object – verb
English: IBM bought Lotus yesterday
Japanese: IBM Lotus bought yesterday
Porto (14/05/2013) SMT and PE 5 / 67
Challenges: Non-literal meaning
• Idioms
e.g. He finally kicked the bucket at the hospital
⇒ Ele finalmente chutou o balde no hospital VS.
⇒ Ele finalmente bateu as botas no hospital
• Multi-word expressions
e.g. Do take the long waiting list for organ donation in this country into
account
⇒ Considere a longa lista de espera para doa¸c˜ao de ´org˜aos neste pa´ıs
Porto (14/05/2013) SMT and PE 6 / 67
Requires deep language processing
• One of the most challenging problems in NLP
• Requires
Full understanding of source text (analysis)
Generation of a coherent, accurate, fluent target text (synthesis)
Porto (14/05/2013) SMT and PE 7 / 67
Statistical Machine Translation
P(French|English)
• Statistical Machine Translation (SMT) learn how to generate
translations using statistical methods
• Early 1990s by IBM. Adopted by Google, etc.
• Idea is older:
Warren Weaver (1949)
When I look at an article in Russian, I say: “This is really written in
English, but it has been coded in some strange symbols. I will now
proceed to decode.”
Inspired by WWII code-breaking
Approach not feasible with early computers – idea not pursued
Porto (14/05/2013) SMT and PE 8 / 67
Probabilistic Model
• Output of decoder depends probabilistically on the input
• To translate French (F) into English (E):
Given a French sentence F search for English sentence E∗ that maximises
P(E|F)
Porto (14/05/2013) SMT and PE 9 / 67
Probabilistic Model
• Find English sentence that maximises P(E|F), i.e.
E∗ = argmax
E
P(E|F)
= argmax
E
P(E)·P(F|E)
P(F) Bayes Rule
P(F) constant across different E, so:
E∗ = argmax
E
P(E) · P(F|E)
drop constant
divisor P(F)
Porto (14/05/2013) SMT and PE 10 / 67
Noisy Channel Model & SMT (ctd)
E∗
= argmax
E
P(E|F) = argmax
E
P(E) · P(F|E)
• Why not model P(E|F) directly?
Would need very good probability estimates
Divide and conquer!
• Decomposition to P(E) · P(F|E) allows one to use less reliable
probabilities
P(E) worries about good English — fluency
• ensures that E has words in the right order and fit together
P(F|E) worries about French that translates English — faithfulness
• ensures that E has words that generally translate F
P(E) and P(F|E) can be trained independently
Porto (14/05/2013) SMT and PE 11 / 67
Main Components for Translation (F → E)
• Translation model (TM): P(F|E)
Faithfulness: TMs learned by analysing large amounts of parallel text
• Judge which (F, E) pairs are likely to be translations of each other
• Language model (LM): P(E)
Fluency: LMs created from large volume of (fluent) target language
sentences
• Learn which word sequences are common/more probable in E
• A Decoder: (argmax)
Search algorithm to find E∗
• V. large search space — heuristic search approach used
Porto (14/05/2013) SMT and PE 12 / 67
Learning Translation Models - P(F|E)
• Requires a bilingual corpus
e.g. European/Canadian/Hong Kong parliaments, subtitles, Bible
• Bilingual corpus must be sentence-aligned:
• Can we estimate P(F|E) from whole sentences?
Porto (14/05/2013) SMT and PE 13 / 67
Learning Translation Models - Word-based SMT
• Break sentences into smaller units: words are a good starting point
• Learn translation probabilities by word aligning a sentence-aligned
corpus:
Zenish
Uh useh
Uh jejje
Yiguo useh
English
A home
A garden
I arrived home
Porto (14/05/2013) SMT and PE 14 / 67
Learning Translation Models - Word-based SMT
• Break sentences into smaller units: words are a good starting point
• Learn translation probabilities by word aligning a sentence-aligned
corpus:
Zenish
Uh useh
Uh jejje
Yiguo useh
• The same word happens in source 1 and 3
English
A home
A garden
I arrived home
Porto (14/05/2013) SMT and PE 15 / 67
Learning Translation Models - Word-based SMT
• Break sentences into smaller units: words are a good starting point
• Learn translation probabilities by word aligning a sentence-aligned
corpus:
Zenish
Uh useh
Uh jejje
Yiguo useh
• Could we expect the same in the target side?
English
A home
A garden
I arrived home
Porto (14/05/2013) SMT and PE 16 / 67
Learning Translation Models - Word-based SMT
• Break sentences into smaller units: words are a good starting point
• Learn translation probabilities by word aligning a sentence-aligned
corpus:
Zenish
Uh useh
Uh jejje
Yiguo useh
English
A home
A garden
I arrived home
• useh = home
Porto (14/05/2013) SMT and PE 17 / 67
Learning Translation Models - Word-based SMT
• Break sentences into smaller units: words are a good starting point
• Learn translation probabilities by word aligning a sentence-aligned
corpus:
Zenish
Uh useh
Uh jejje
Yiguo useh
• What about the contexts?
English
A home
A garden
I arrived home
Porto (14/05/2013) SMT and PE 18 / 67
Learning Translation Models - Word-based SMT
• Break sentences into smaller units: words are a good starting point
• Learn translation probabilities by word aligning a sentence-aligned
corpus:
Zenish
Uh useh
Uh jejje
Yiguo useh
English
A home
A garden
I arrived home
• We can align them: Yiguo = I; Yiguo = arrived; Uh = A
Porto (14/05/2013) SMT and PE 19 / 67
Learning Translation Models - Word-based SMT
• Break sentences into smaller units: words are a good starting point
• Learn translation probabilities by word aligning a sentence-aligned
corpus:
Zenish
Uh useh
Uh jejje
Yiguo useh
English
A home
A garden
I arrived home
• And reuse this knowledge to align more sentences: Uh = A
Porto (14/05/2013) SMT and PE 20 / 67
Learning Translation Models - Word-based SMT
• Break sentences into smaller units: words are a good starting point
• Learn translation probabilities by word aligning a sentence-aligned
corpus:
Zenish
Uh useh
Uh jejje
Yiguo useh
English
A home
A garden
I arrived home
• And again: jejje = garden
Porto (14/05/2013) SMT and PE 21 / 67
Learning Translation Models - Word-based SMT (ctd)
Word-alignment:
• Identify correspondences between two languages at the word level
• Basis for word-based SMT, first step for other approaches
• Alignment can be learned using Expectation Maximization (EM)
Treat all alternative word alignments as equally likely
Observe across sentences that Zenish useh often links to English home
• Increase probability of this word pair aligning
• Has knock-on effect for alignment of other words
Iteratively redistribute probabilities, until they identify most probable
links for each word (convergence)
Word alignment commonly done using the IBM Models 1-5
Porto (14/05/2013) SMT and PE 22 / 67
Learning Translation Models - Word-based SMT (ctd)
• IBM models produce a probabilistic dictionary for the whole parallel
corpus:
contras contras 0.0285714
contras los 0.0000011
contras esos 0.0000812
richer enriquecida 0.0476190
richer mucho 0.0000502
richer enriqueci 0.0769231
richer variado 0.0066225
Porto (14/05/2013) SMT and PE 23 / 67
Learning Translation Models - Word-based SMT (ctd)
Back to the translation model:
At translation (decoding time), for a new sentence to translate, take the
set of translations that jointly maximise the whole translation probability
E∗ = argmax
E
P(F|E)
• Generates alternative translations using different translation options
for words and different word orders
• Selects the one with maximum score: no notion of context. How
about ambiguity?
What about the context and the fluency in the target language?
Porto (14/05/2013) SMT and PE 24 / 67
Learning Language Models - Word-based SMT
Language model: P(E)
E∗ = argmax
E
P(F|E) · P(E)
• Takes different translation options, tests how “common” each of
them are in the target language as a whole sentence (some context)
• P(E) = probability of strings E based on relative frequencies in a
large corpus of language E
Google
the house is small 144,000
the house is little 85,300
small house 2,330,000
little house 3,330,000
Porto (14/05/2013) SMT and PE 25 / 67
Learning Language Models - Word-based SMT (ctd)
• Model P(E) by decomposing into single-word probabilities given
previous words, under Markov assumption: only the previous n-1
words matter for predicting a word. E.g for trigram models, n = 3
P(e1) · P(e2|e1) · P(e3|e1, e2) · P(e4|e2, e3) · · · P(en|en−2, en−1)
• Relative frequencies to compute these probabilities. E.g. trigrams:
P(e3|e1, e2) = count(e1e2e3)
count(e1e2)
P(little|house, is) = count(house is little)
count(house is)
Porto (14/05/2013) SMT and PE 26 / 67
Word-based SMT – Limitations
• It is difficult to do alignment, and hence learn a TM, for languages
that are structurally quite different, with different words orders
Considering all possible word orders – too costly
Poor reordering model: flexibility vs noise
• Some languages may have different notions of what counts as a word
Donaydampfshiffahrtsgesellschaftskapitaenskajuetenschluesseloch
The keyhole of the door of the cabin of the captain of a steamship
company operating on the Danubea
a
http://www.chebucto.ns.ca/~waterbuf/longwrds.html
Porto (14/05/2013) SMT and PE 27 / 67
Phrase-based SMT
• Leading approach since 2003
No voy1 a la2 casa3 → I am not going1 to the2 house3
it seems to1 me2 → me1 parece2
Je1 ne vais pas2 `a la3 maison4 → I1 am not going2 to the3 house4
Eu1 sinto saudade de vocˆe2 → I1 miss you2
I1 miss you2 → Eu1 sinto sua falta2
natuerlich1 hat2 John3 spass am4 spiel5 → of course1 John2 has3 fun with the4 game5
• More intuitive and reliable alignments
Account for reordering within the phrases
Phrases can still be reordered
• Can we learn phrase translation probability distribution from the
parallel corpus using EM?
1 src-tgt sentence → O(n4
) phrase pairs
Porto (14/05/2013) SMT and PE 28 / 67
Phrase-based SMT - Phrases from word alignments
• Extract phrase pairs that are consistent with the word alignment
• Phrase: sequence of tokens, not linguistically motivated
• Word alignment produced by any of the IBM Models - only done once
reanudaci´on del per´ıodo de sesiones
resumption
of
the
session
1 resumption ↔ reanudaci´on
2 of the ↔ del
3 session ↔ per´ıodo de sesiones
4 resumption of the ↔ reanudaci´on del
5 of the session ↔ del sesiones
6 resumption of the session ↔ reanudaci´on del sesiones
Porto (14/05/2013) SMT and PE 29 / 67
Phrase-based SMT - Computing phrase probability
distributions
1 Extract phrase pairs from word aligned parallel corpus
√
2 Extract counts of those phrases from large parallel corpus:
φ(¯f |¯e) =
count(¯f , ¯e)
count(¯e)
3 Store phrases and their probabilities in a phrase table
Probabilistic dictionary of phrases
Porto (14/05/2013) SMT and PE 30 / 67
Phrase-based SMT - Limitation of Noisy Channel Model
E∗ = argmax
E
P(F|E) · P(E)
• Which component is more important?
P(F|E) ?
P(E) ?
• Depends on the corpus used (more parallel data makes P(F|E) more
reliable), language-pair, etc.
Porto (14/05/2013) SMT and PE 31 / 67
Phrase-based SMT - Log-linear Model (ctd)
Assign weights λi to components (feature functions) hi (f, e):
e∗ = argmax
e
n
i=1 λi hi (f, e)
Components
1 P(E)
2 P(F|E)
Weights
1 λP(E)
2 λP(F|E)
Feature Functions
1 h1 = logP(E)
2 h2 = logP(F|E)
Benefits
1 Extensible: other components are used: e.g. reordering, word and
phrase penalties
2 Weights can be tuned, i.e., learned from examples to minimise error
in a subset of the parallel corpus, e.g. MERT
Porto (14/05/2013) SMT and PE 32 / 67
Phrase-based SMT - Decoding (ctd)
Search problem: segmenting source sentence and selecting phrase
translations:
• All phrases matching source words are selected from the phrase table
J’ ai les yeux noirs .
I have the eyes black .
me has them eye dark ,
I have eyes espresso !
I am the eyes somber .
I did some black eyes .
I had black eyes .
I have black eyes .
black eyes I have .
• The goal of the decoder is to select the phrases whose combination
(in a given order) yields the highest score for the log-linear model
Porto (14/05/2013) SMT and PE 33 / 67
Phrase-based SMT - Decoding (ctd)
Searching = incrementally construct the translation hypotheses by trying
out several possibilities
• Generating target words in sequence, from left to right
• Incrementally compute the overall log-linear model score for each
hypothesis
• Heuristics to prune search space, e.g. stack-based beam search
Use one stack for of every length of partial hypotheses
Limited space for partial hypotheses, only keep those that are
promising, e.g. model score is close to that of the best partial
hypothesis so far
Porto (14/05/2013) SMT and PE 34 / 67
Phrase-based SMT - Decoding (ctd)
Search space
Hypothesis
• Covered source words
• Target (output) words
• Model score (using the combined probabilities)
Porto (14/05/2013) SMT and PE 35 / 67
Hierarchical and Syntax-based SMT
• PBSMT still has trouble with long-distance reorderings
• Ways of bringing structure and linguistic knowledge into the transfer
rules of the phrase table
Porto (14/05/2013) SMT and PE 36 / 67
Hierarchical SMT - Motivation
Introduce structure into phrase-based SMT models to deal with
long-distance reordering
Ich
werde
Ihnen
die
entsprechenden
Anmerkungen
aush¨andigen
I
shall
be
passing
on
to
you
some
comments
Porto (14/05/2013) SMT and PE 37 / 67
Hierarchical SMT - Motivation
Introduce structure into phrase-based SMT models to deal with
long-distance reordering
Ich
werde
Ihnen
die
entsprechenden
Anmerkungen
aush¨andigen
I
shall
be
passing
on
to
you
some
comments
• How can we get a phrase for shall
be passing on?
Porto (14/05/2013) SMT and PE 38 / 67
Hierarchical SMT - Motivation
Introduce structure into phrase-based SMT models to deal with
long-distance reordering
Ich
werde
Ihnen
die
entsprechenden
Anmerkungen
aush¨andigen
I
shall
be
passing
on
to X
you X
some X
comments X
• How can we get a phrase for shall
be passing on?
• We cannot, unless we get to you
some comments along
Porto (14/05/2013) SMT and PE 39 / 67
Hierarchical SMT - Motivation
Introduce structure into phrase-based SMT models to deal with
long-distance reordering
Ich
werde
Ihnen
die
entsprechenden
Anmerkungen
aush¨andigen
I
shall
be
passing
on
to
you
some
comments
• How can we get a phrase for shall
be passing on?
• We cannot, unless we get to you
some comments along
• Unless we replace all those words
by a variable
Porto (14/05/2013) SMT and PE 40 / 67
Hierarchical SMT - Motivation (ctd)
shall be passing on to you some comments
werde Ihnen die entsprechenden Anmerkungen aush¨andigen
shall be passing on to you some comments
werde Ihnen die entsprechenden Anmerkungen aush¨andigen
shall be passing on X
werde X aush¨andigen
Porto (14/05/2013) SMT and PE 41 / 67
Hierarchical SMT - basics
Learnt from word-aligned parallel corpora in the same way as before
• Based on the fact that language has recursive structures
• Phrases within other phrases treated as nonterminals: replaced by X
• No linguistic constraints added - yet, some structure
Porto (14/05/2013) SMT and PE 42 / 67
Hierarchical SMT - basics (ctd)
shall be passing on to you some comments
werde Ihnen die entsprechenden Anmerkungen aush¨andigen
shall be passing on X1 some comments
werde X1 die entsprechenden Anmerkungen aush¨andigen
shall be passing on X1 X2
werde X1 X2 aush¨andigen
Porto (14/05/2013) SMT and PE 43 / 67
Hierarchical SMT - phrase-table
[X] → shall be passing on X1 X2 | werde X1 X2 aush¨andigen
[X] → shall be passing on X3 | werde X3 aush¨andigen
[X] → to you | Ihnen
[X] → some comments | die entsprechenden Anmerkungen
[X] → to you some comments | Ihnen die entsprechenden Anmerkungen
Learn a bilingual (synchronous) set of context-free rules about how to
translate F into E
Porto (14/05/2013) SMT and PE 44 / 67
Syntax-based SMT
• Hierarchical models are not very informative (Xs) and suffer from an
exponential number of rules
• Syntax-based SMT overcomes these limitations by using linguistic
categorise to label nodes
• A syntactic parser is required at pre-processing time on at least one
language (the other could have Xs)
• Idea:
Learn a bilingual (synchronous) set of linguistically informed
context-free rules about how to translate F into E
Rules are learnt from the word-alignment, constrained by syntactic
categories
Porto (14/05/2013) SMT and PE 45 / 67
Syntax-based SMT (ctd)
S
PRP
J’
VP
VB
ai
NP
DT
les
NN
yeux
JJ
noirs
S
PRP
I
VP
VB
have
NP
JJ
black
NN
eyes
J’ ai les yeux noirs
I
have
black
eyes
Standard constraints on rule construction:
• Single nonterminal on the left
• Consistent with word-alignment
• Nonterminals on the right must align
one-to-one
Porto (14/05/2013) SMT and PE 46 / 67
Syntax-based SMT (ctd)
Grammar
PRP → J’ | I
JJ → noirs | black
NP → les yeux JJ | JJ eyes
VP → ai NP | have NP
S → PRP VP | PRP VP
Decoding becomes a (probabilistic) parsing problem!
Porto (14/05/2013) SMT and PE 47 / 67
Summary on SMT
• Adding linguistic knowledge to SMT is a trend:
Stronger theoretical motivation
Deals with reordering, sparsity, ambiguity
Depends on (quality of) parsers
Higher complexity
• Performance:
Hard to quantify in absolute terms
Most language pairs → phrase-based or hierarchical SMT does better
Language pairs with long-distance reordering → syntax-based SMT
does better
• Other open issues:
Limited use of linguistic information - at most syntax, some attempts
to use shallow semantics - not yet there
Limited use of context - sentence-level
Domain discrepancies - mistranslations, unknown words
• Overall, to reach publishable quality, human checking/revision
necessary: post-editing
Porto (14/05/2013) SMT and PE 48 / 67
Definitions
Note: Some slides are based on those by Sharon O’Brien (tutorial on MT
post-editing at AMTA-2010).
What is post-editing?
• Same as translation?
• Same as revision?
• A new profession? New skills?
Who does the post-editing?
What are the quality expectations?
Porto (14/05/2013) SMT and PE 49 / 67
Degrees/types of post-editing
• Light (fast/rapid/gist) post-editing: essential corrections
• Full post-editing: more corrections, higher quality, closer to revision
• Monolingual post-editing: no access to source text
Decided by:
• User requirements
• Volume
• Quality expectations
• Turn-around time, etc.
Porto (14/05/2013) SMT and PE 50 / 67
Quality of post-editing
• Can MT+PE (possibly +revision) produce the same level of quality
as HT+revision?
• Dependent on the quality of MT systems: variable for language pair,
text type/genre & domain, quality of source
• Dependent on quality requirements
• Different outcomes:
Autodesk reports lower error rates with MT+PE [Aut11]
Tilde observed higher error rate with MT+PE [SPSV11] -
under-resourced highly-inflected languages: lower MT quality
Porto (14/05/2013) SMT and PE 51 / 67
Ways of measuring quality of MT for PE
• Automatic MT evaluation metrics ? BLEU/TER/METEOR... -
automatic, but do not correlate well with PE effort
• Error counts: compare source with raw MT output - manual, costly
Can we use the “standard” error typologies for HT? Do MT and HT
make same mistakes?
QTLaunchPad project
Multidimensional Quality Metrics for MT and HT, for manual and
(semi-)automatic evaluation (QE): http://www.qt21.eu/launchpad/
• % Changes made: compare post-edited text with raw MT output -
automatic
• Time to make changes: record time to fix raw MT - automatic
• Estimated effort: take source text and raw MT output and estimate
PE effort prior to post-editing (QE) - automatic, less precise
Is this the same as fuzzy match level in TM systems?
Porto (14/05/2013) SMT and PE 52 / 67
Aspects involved in post-editing effort
• Temporal: time to post-edit (variable across translators)
• Technical: no./% edits made, keys pressed
• Cognitive: time, pauses, movements
Studies on PE process:
• Is PEMT faster than HT? Yes
• Is PEMT less keyboard intensive than HT? Yes
• Is PEMT less cognitively demanding than HT?
Cognitive effort is the most difficult to estimate: active research area
[Kop12, KARS12]
• PE more tedious/tiring? Mixed opinions
Check work by
Michael Carl - CBS (Translog) and Sharon O’Brien - DCU (eye-tracking)
Porto (14/05/2013) SMT and PE 53 / 67
Pricing models
Open question. Some common practices:
• Paying as fuzzy segment matches (source-based)
• Paying as % of edits performed (MT&PE-based)
• Paying a fee based on time spent (less common)
Porto (14/05/2013) SMT and PE 54 / 67
PE Guidelines
• No standard guidelines, but good practices (e.g. TAUS/CNGL)
• Guidelines need to clearly specify level of post-editing and quality
expectations
• Guidelines need to clearly differentiate between essential and
preferential changes
• Ideally, guidelines should be system-, language-, purpose-specific
Porto (14/05/2013) SMT and PE 55 / 67
General PE Guidelines for “good enough” quality -
TAUS/CNGL
• Aim for semantically correct translation.
• Ensure that no information has been accidentally added or omitted.
• Edit any offensive, inappropriate or culturally unacceptable content.
• Use as much of the raw MT output as possible.
• Basic rules regarding spelling apply.
• No need to implement corrections that are of a stylistic nature only.
• No need to restructure sentences solely to improve the natural flow of
the text.
http://www.translationautomation.com/postediting/
machine-translation-post-editing-guidelines
Porto (14/05/2013) SMT and PE 56 / 67
General PE Guidelines for HT quality - TAUS/CNGL
• Aim for grammatically, syntactically and semantically correct
translation.
• Ensure that key terminology is correctly translated and that
untranslated terms belong to the clients list of Do Not Translate
terms.
• Ensure that no information has been accidentally added or omitted.
• Edit any offensive, inappropriate or culturally unacceptable content.
• Use as much of the raw MT output as possible.
• Basic rules regarding spelling, punctuation and hyphenation apply.
• Ensure that formatting is correct.
http://www.translationautomation.com/postediting/
machine-translation-post-editing-guidelines
Porto (14/05/2013) SMT and PE 57 / 67
IT PE Guidelines - Midori Tatsumi’s PhD thesis
• What needs to be fixed:
Non-translatable items, such as command and variable names, that
have been translated. Please put it back to English.
Inappropriately translated general IT terms.
Mistranslation.
Word orders that are inappropriate to the level that the sentence has
become impossible or difficult to comprehend.
Comprehensible but extremely unnatural or inappropriate expressions.
Inappropriate postpositions and conjugations.
• What are preferred to be standardised:
Repetitive items, such as procedures and steps.
Styles of the section titles in a user manual and help files.
• What does not need to be fixed:
Ending styles of bulleted items.
Placeholders, such as &ProductNameShort; and &ProductNameLong;
Punctuation.
http://doras.dcu.ie/16062/1/SAKURA_final_revised.pdf
Porto (14/05/2013) SMT and PE 58 / 67
Monolingual PE Guidelines - ACCEPT project
• Try and edit the text by making it more fluent and clearer based on
how you interpret its meaning. For example, try to rectify word order
and spelling when they are inappropriate to the extent that the text
has become impossible or difficult to comprehend
• If words, phrases, or punctuation in the text are completely
acceptable, try and use them (unmodified) rather than substituting
them with something new and different.
http://www.accept.unige.ch/Products/D6.2.2_Seminar_
Material_on_Post-Editing_Edition_2.pdf
Porto (14/05/2013) SMT and PE 59 / 67
PE Technology (tools)
A myriad of tools appeared in the marked in the last couple of years:
• TM tools that now integrate MT, e.g. SDL TRADOS, WordFast
• MT tools (with or without TM): e.g. Systran, Google Translate
Toolkit, Caitra
• PE tools that can use different MT systems: MemoQ, MemSource,
SmartMate, Coach, ...
More and more functionalities: spell/grammar checkers, multiple MT,
MT+TM, interactive MT, terminology, word/phrase alignment, etc.
Limited (if any) logging of translator’s activities for productivity analysis –
not enough for research.
Porto (14/05/2013) SMT and PE 60 / 67
PE Technology (tools)
Research-oriented tools:
• Translog (not open source)
• OmegaT (TM only)
• Appraise (MT eval through ranking, no detailed statistics for PE)
• PET: http://pers-www.wlv.ac.uk/~in1676/pet/index.html
• Coming soon: open source tool promised as part of EU projects
CASMACAT & MATECAT
Porto (14/05/2013) SMT and PE 61 / 67
What can we get from PET?
Comparison between HT and multiple MT systems [dSAS11]
• PET to compare 3 MT systems vs translation from scratch
• Sitcom and movie subtitles:
Translating from scratch can be 73% slower than post-editing
SMT systems (Google and Moses) performed the best
How often a system produced an output that was faster to PE than other systems:
System Google Moses Systran Trados
Google - 139 161 187
Moses 69 - 122 164
Systran 69 106 - 145
Trados 48 67 89 -
How often post-editing a system output was faster than translating:
System Faster than human translation
Google 94%
Moses 86.8%
Systran 81.20%
Trados 72.40%
Porto (14/05/2013) SMT and PE 62 / 67
What can we get from PET?
Evaluation of an MT system for translation of subtitles (space constraints)
[AdSS12]
• Add compression constraints to an MT system to generate length
compliant subtitles
• PET to guide post-editing according to length and time requirements
for every unit
Show space limitation (attribute)
Change colour of translation if too long
Offer shorter paraphrases
• PET to evaluate edit distance and length
System
Dexter How I Met.. Terra Nova
TER ↓ LENGTH TER ↓ LENGTH TER ↓ LENGTH
Mosest 30.3 116.0 20.0 108.5 33.8 120.2
Google 63.6 156.5 52.8 144.3 63.1 152.1
MosesLP2 29.5 115.5 21.0 109.1 33.4 119.3
MosesLP1 28.3 115.8 20.7 110.0 34.8 119.8
Porto (14/05/2013) SMT and PE 63 / 67
Hands-on activity with PET
• en-pt-hiding-and-assessing.pec – bigbang-pe-2.pej
• en-pt-hiding-and-assessing.pec – bigbang-ht-2.pej
• fapesp.pec – fapesp-annotated.pej
• fapesp-mono.pec – fapesp-annotated.pej
• subs.pec – greys.08x06.v1-v1.en-br.s2.1.pej
Advanced use:
• Configuration file: .pec
• Input/output XML files
• How to build a PEJ file from raw files: pej.bat
• How to interpret the results: scripts/readper.pl
Porto (14/05/2013) SMT and PE 64 / 67
For more on the research-side
Workshop on Post-Editing Technology and Practice:
• AMTA 2012: https://sites.google.com/site/wptp2012/
• MT Summit 2013 (deadline for papers 28/05/2013):
https://sites.google.com/site/mts2013wptp/
Porto (14/05/2013) SMT and PE 65 / 67
References I
Wilker Aziz, Sheila Castilho Monteiro de Sousa, and Lucia Specia.
Cross-lingual sentence compression for subtitles.
In The 16th Annual Conference of the European Association for Machine
Translation, EAMT ’12, pages 103–110, Trento, Italy, May 2012.
Autodesk.
Translation and Post-Editing Productivity.
In http: // translate. autodesk. com/ productivity. html , 2011.
Sheila C. M. de Sousa, Wilker Aziz, and Lucia Specia.
Assessing the post-editing effort for automatic and semi-automatic
translations of DVD subtitles.
In Conference Recent Advances in Natural Language Processing 2011, pages
97–103, Hissar, Bulgaria, 2011.
Porto (14/05/2013) SMT and PE 66 / 67
References II
Maarit Koponen, Wilker Aziz, Luciana Ramos, and Lucia Specia.
Post-editing time as a measure of cognitive effort .
In AMTA 2012 Workshop on Post-Editing Technology and Practice (WPTP
2012), pages 11–20, San Diego, USA, October 2012. Association for
Machine Translation in the Americas (AMTA).
Maarit Koponen.
Comparing human perceptions of post-editing effort with post-editing
operations.
In Proceedings of the Seventh Workshop on Statistical Machine Translation,
pages 181–190, Montr´eal, Canada, 2012.
R. Skadins, M. Purins, I. Skadina, and A. Vasiljevs.
An evaluation of SMT in localization to under-resourced inflected languages.
In Proceedings of the 15th Conference of the European Association for
Machine Translation, Leuven, 2011.
Porto (14/05/2013) SMT and PE 67 / 67

More Related Content

Viewers also liked

2n Premi de la categoria Power Point 2013_ 'Obre els ulls' (Institut de Puig-...
2n Premi de la categoria Power Point 2013_ 'Obre els ulls' (Institut de Puig-...2n Premi de la categoria Power Point 2013_ 'Obre els ulls' (Institut de Puig-...
2n Premi de la categoria Power Point 2013_ 'Obre els ulls' (Institut de Puig-...AjBerga
 
Programa Siria - Dirección Nacional de Migraciones de Argentina
Programa Siria - Dirección Nacional de Migraciones de ArgentinaPrograma Siria - Dirección Nacional de Migraciones de Argentina
Programa Siria - Dirección Nacional de Migraciones de Argentina
Nicolás Munilla Korzeniowski
 
Denuncia Sobre El Caso De Union Electrica
Denuncia Sobre El Caso De Union ElectricaDenuncia Sobre El Caso De Union Electrica
Denuncia Sobre El Caso De Union Electrica
Ismael Garcia
 
SMALL SIZE Europabio_report_print_
SMALL SIZE Europabio_report_print_SMALL SIZE Europabio_report_print_
SMALL SIZE Europabio_report_print_
cvpopa
 
Folder aquecedor solar_heliotek2012
Folder aquecedor solar_heliotek2012Folder aquecedor solar_heliotek2012
Folder aquecedor solar_heliotek2012
bene piscinas
 
Identidad segura en las redes sociales
Identidad segura en las redes socialesIdentidad segura en las redes sociales
Identidad segura en las redes sociales
Adriana Durato
 
Unidad tres estructura de datos
Unidad tres estructura de datosUnidad tres estructura de datos
Unidad tres estructura de datos
René Sosa Arana
 
Redes informáticas
Redes informáticas Redes informáticas
Redes informáticas
Tecnope
 
Saa s and cloud erp observations is cloud erp right for you
Saa s and cloud erp observations  is cloud erp right for youSaa s and cloud erp observations  is cloud erp right for you
Saa s and cloud erp observations is cloud erp right for you
Christophe Monnier
 
High Growth Businesses Its A Life Stage Problem Not A Journey To Shangri La
High Growth Businesses   Its A Life Stage Problem Not A Journey To Shangri LaHigh Growth Businesses   Its A Life Stage Problem Not A Journey To Shangri La
High Growth Businesses Its A Life Stage Problem Not A Journey To Shangri La
Ged Mirfin
 
360 feedback implementation steps to follow
360 feedback implementation   steps to follow360 feedback implementation   steps to follow
360 feedback implementation steps to follow
Angele Abi Khalil
 
Doc planificacion estrategica
Doc planificacion estrategicaDoc planificacion estrategica
Doc planificacion estrategica
Constanza Zúñiga Desfaur
 
Un mundo en movimiento
Un mundo en movimientoUn mundo en movimiento
Un mundo en movimiento
alba2000
 
Web Semianr Seamless Apps with Moovweb
Web Semianr Seamless Apps with Moovweb Web Semianr Seamless Apps with Moovweb
Web Semianr Seamless Apps with Moovweb
Moovweb
 
Loris malaguzzi
Loris malaguzziLoris malaguzzi
Loris malaguzzi
osaxti
 
Laminas de policarbonato, caracteristicas
Laminas de policarbonato, caracteristicasLaminas de policarbonato, caracteristicas
Laminas de policarbonato, caracteristicas
oscar manrique zavala
 
QR-Code für Anwender
QR-Code für AnwenderQR-Code für Anwender
QR-Code für Anwender
Frank H. Ritz
 
Enlace ionico quim fimaas
Enlace ionico quim fimaasEnlace ionico quim fimaas
Enlace ionico quim fimaas
Jenny Fernandez Vivanco
 
Problema costos de Mecanizado (procesos de fabricacion)
Problema costos de Mecanizado (procesos de fabricacion)Problema costos de Mecanizado (procesos de fabricacion)
Problema costos de Mecanizado (procesos de fabricacion)
Miguel Antonio Bula Picon
 
Fandom Quiz 2015 - Prelims
Fandom Quiz 2015 - PrelimsFandom Quiz 2015 - Prelims
Fandom Quiz 2015 - Prelims
Vishal Katariya
 

Viewers also liked (20)

2n Premi de la categoria Power Point 2013_ 'Obre els ulls' (Institut de Puig-...
2n Premi de la categoria Power Point 2013_ 'Obre els ulls' (Institut de Puig-...2n Premi de la categoria Power Point 2013_ 'Obre els ulls' (Institut de Puig-...
2n Premi de la categoria Power Point 2013_ 'Obre els ulls' (Institut de Puig-...
 
Programa Siria - Dirección Nacional de Migraciones de Argentina
Programa Siria - Dirección Nacional de Migraciones de ArgentinaPrograma Siria - Dirección Nacional de Migraciones de Argentina
Programa Siria - Dirección Nacional de Migraciones de Argentina
 
Denuncia Sobre El Caso De Union Electrica
Denuncia Sobre El Caso De Union ElectricaDenuncia Sobre El Caso De Union Electrica
Denuncia Sobre El Caso De Union Electrica
 
SMALL SIZE Europabio_report_print_
SMALL SIZE Europabio_report_print_SMALL SIZE Europabio_report_print_
SMALL SIZE Europabio_report_print_
 
Folder aquecedor solar_heliotek2012
Folder aquecedor solar_heliotek2012Folder aquecedor solar_heliotek2012
Folder aquecedor solar_heliotek2012
 
Identidad segura en las redes sociales
Identidad segura en las redes socialesIdentidad segura en las redes sociales
Identidad segura en las redes sociales
 
Unidad tres estructura de datos
Unidad tres estructura de datosUnidad tres estructura de datos
Unidad tres estructura de datos
 
Redes informáticas
Redes informáticas Redes informáticas
Redes informáticas
 
Saa s and cloud erp observations is cloud erp right for you
Saa s and cloud erp observations  is cloud erp right for youSaa s and cloud erp observations  is cloud erp right for you
Saa s and cloud erp observations is cloud erp right for you
 
High Growth Businesses Its A Life Stage Problem Not A Journey To Shangri La
High Growth Businesses   Its A Life Stage Problem Not A Journey To Shangri LaHigh Growth Businesses   Its A Life Stage Problem Not A Journey To Shangri La
High Growth Businesses Its A Life Stage Problem Not A Journey To Shangri La
 
360 feedback implementation steps to follow
360 feedback implementation   steps to follow360 feedback implementation   steps to follow
360 feedback implementation steps to follow
 
Doc planificacion estrategica
Doc planificacion estrategicaDoc planificacion estrategica
Doc planificacion estrategica
 
Un mundo en movimiento
Un mundo en movimientoUn mundo en movimiento
Un mundo en movimiento
 
Web Semianr Seamless Apps with Moovweb
Web Semianr Seamless Apps with Moovweb Web Semianr Seamless Apps with Moovweb
Web Semianr Seamless Apps with Moovweb
 
Loris malaguzzi
Loris malaguzziLoris malaguzzi
Loris malaguzzi
 
Laminas de policarbonato, caracteristicas
Laminas de policarbonato, caracteristicasLaminas de policarbonato, caracteristicas
Laminas de policarbonato, caracteristicas
 
QR-Code für Anwender
QR-Code für AnwenderQR-Code für Anwender
QR-Code für Anwender
 
Enlace ionico quim fimaas
Enlace ionico quim fimaasEnlace ionico quim fimaas
Enlace ionico quim fimaas
 
Problema costos de Mecanizado (procesos de fabricacion)
Problema costos de Mecanizado (procesos de fabricacion)Problema costos de Mecanizado (procesos de fabricacion)
Problema costos de Mecanizado (procesos de fabricacion)
 
Fandom Quiz 2015 - Prelims
Fandom Quiz 2015 - PrelimsFandom Quiz 2015 - Prelims
Fandom Quiz 2015 - Prelims
 

Similar to Lucia Specia - SMT e pós-edição

Latent Semantic Transliteration using Dirichlet Mixture
Latent Semantic Transliteration using Dirichlet MixtureLatent Semantic Transliteration using Dirichlet Mixture
Latent Semantic Transliteration using Dirichlet Mixture
Rakuten Group, Inc.
 
Templates in linguistics - Why Garbage Garbage
Templates in linguistics - Why Garbage GarbageTemplates in linguistics - Why Garbage Garbage
Templates in linguistics - Why Garbage Garbage
Hussein Ghaly
 
Interspeech 2017 s_miyoshi
Interspeech 2017 s_miyoshiInterspeech 2017 s_miyoshi
Interspeech 2017 s_miyoshi
Hiroyuki Miyoshi
 
Deep network notes.pdf
Deep network notes.pdfDeep network notes.pdf
Deep network notes.pdf
Ramya Nellutla
 
Hunter's final KUT lecture
Hunter's final KUT lectureHunter's final KUT lecture
Hunter's final KUT lecture
Lawrie Hunter
 
81985999 turkish-3-months
81985999 turkish-3-months81985999 turkish-3-months
81985999 turkish-3-months
sorinakader
 
Mt framework nagao_makoto
Mt framework nagao_makotoMt framework nagao_makoto
Mt framework nagao_makoto
Hiroshi Matsumoto
 
Comparative study of Text-to-Speech Synthesis for Indian Languages by using S...
Comparative study of Text-to-Speech Synthesis for Indian Languages by using S...Comparative study of Text-to-Speech Synthesis for Indian Languages by using S...
Comparative study of Text-to-Speech Synthesis for Indian Languages by using S...
ravi sharma
 
Empirical Study Incorporating Linguistic Knowledge on Filled Pauses for Pers...
Empirical Study Incorporating Linguistic Knowledge on Filled Pausesfor Pers...Empirical Study Incorporating Linguistic Knowledge on Filled Pausesfor Pers...
Empirical Study Incorporating Linguistic Knowledge on Filled Pauses for Pers...
Yuta Matsunaga
 
Hoc tieng anh theo chu de
Hoc tieng anh theo chu deHoc tieng anh theo chu de
Hoc tieng anh theo chu de
Hao Luong
 
Parameter setting
Parameter settingParameter setting
Parameter setting
Rick McKinnon
 
Comparing and contrasting English grammar
Comparing and contrasting English grammarComparing and contrasting English grammar
Comparing and contrasting English grammar
AniqAzman3
 
Pre-intermediate tutorial
Pre-intermediate tutorialPre-intermediate tutorial
Pre-intermediate tutorial
Jenny Yarely
 
Basic sentence structure
Basic sentence structureBasic sentence structure
Basic sentence structure
halima888
 
Basic sentence structure
Basic sentence structureBasic sentence structure
Basic sentence structure
halima888
 
Introduction to Companies of EM
Introduction to Companies of EMIntroduction to Companies of EM
Introduction to Companies of EM
Jan Stander
 
NLP_KASHK:POS Tagging
NLP_KASHK:POS TaggingNLP_KASHK:POS Tagging
NLP_KASHK:POS Tagging
Hemantha Kulathilake
 
07-Effect-Of-Machine-Translation-In-Interlingual-Conversation.pdf
07-Effect-Of-Machine-Translation-In-Interlingual-Conversation.pdf07-Effect-Of-Machine-Translation-In-Interlingual-Conversation.pdf
07-Effect-Of-Machine-Translation-In-Interlingual-Conversation.pdf
simonp16
 
Grammar for IELTS.pdf
Grammar for IELTS.pdfGrammar for IELTS.pdf
Grammar for IELTS.pdf
uztemirovv
 
Identification of Fertile Translations in Comparable Corpora: a Morpho-Compos...
Identification of Fertile Translations in Comparable Corpora: a Morpho-Compos...Identification of Fertile Translations in Comparable Corpora: a Morpho-Compos...
Identification of Fertile Translations in Comparable Corpora: a Morpho-Compos...
Estelle Delpech
 

Similar to Lucia Specia - SMT e pós-edição (20)

Latent Semantic Transliteration using Dirichlet Mixture
Latent Semantic Transliteration using Dirichlet MixtureLatent Semantic Transliteration using Dirichlet Mixture
Latent Semantic Transliteration using Dirichlet Mixture
 
Templates in linguistics - Why Garbage Garbage
Templates in linguistics - Why Garbage GarbageTemplates in linguistics - Why Garbage Garbage
Templates in linguistics - Why Garbage Garbage
 
Interspeech 2017 s_miyoshi
Interspeech 2017 s_miyoshiInterspeech 2017 s_miyoshi
Interspeech 2017 s_miyoshi
 
Deep network notes.pdf
Deep network notes.pdfDeep network notes.pdf
Deep network notes.pdf
 
Hunter's final KUT lecture
Hunter's final KUT lectureHunter's final KUT lecture
Hunter's final KUT lecture
 
81985999 turkish-3-months
81985999 turkish-3-months81985999 turkish-3-months
81985999 turkish-3-months
 
Mt framework nagao_makoto
Mt framework nagao_makotoMt framework nagao_makoto
Mt framework nagao_makoto
 
Comparative study of Text-to-Speech Synthesis for Indian Languages by using S...
Comparative study of Text-to-Speech Synthesis for Indian Languages by using S...Comparative study of Text-to-Speech Synthesis for Indian Languages by using S...
Comparative study of Text-to-Speech Synthesis for Indian Languages by using S...
 
Empirical Study Incorporating Linguistic Knowledge on Filled Pauses for Pers...
Empirical Study Incorporating Linguistic Knowledge on Filled Pausesfor Pers...Empirical Study Incorporating Linguistic Knowledge on Filled Pausesfor Pers...
Empirical Study Incorporating Linguistic Knowledge on Filled Pauses for Pers...
 
Hoc tieng anh theo chu de
Hoc tieng anh theo chu deHoc tieng anh theo chu de
Hoc tieng anh theo chu de
 
Parameter setting
Parameter settingParameter setting
Parameter setting
 
Comparing and contrasting English grammar
Comparing and contrasting English grammarComparing and contrasting English grammar
Comparing and contrasting English grammar
 
Pre-intermediate tutorial
Pre-intermediate tutorialPre-intermediate tutorial
Pre-intermediate tutorial
 
Basic sentence structure
Basic sentence structureBasic sentence structure
Basic sentence structure
 
Basic sentence structure
Basic sentence structureBasic sentence structure
Basic sentence structure
 
Introduction to Companies of EM
Introduction to Companies of EMIntroduction to Companies of EM
Introduction to Companies of EM
 
NLP_KASHK:POS Tagging
NLP_KASHK:POS TaggingNLP_KASHK:POS Tagging
NLP_KASHK:POS Tagging
 
07-Effect-Of-Machine-Translation-In-Interlingual-Conversation.pdf
07-Effect-Of-Machine-Translation-In-Interlingual-Conversation.pdf07-Effect-Of-Machine-Translation-In-Interlingual-Conversation.pdf
07-Effect-Of-Machine-Translation-In-Interlingual-Conversation.pdf
 
Grammar for IELTS.pdf
Grammar for IELTS.pdfGrammar for IELTS.pdf
Grammar for IELTS.pdf
 
Identification of Fertile Translations in Comparable Corpora: a Morpho-Compos...
Identification of Fertile Translations in Comparable Corpora: a Morpho-Compos...Identification of Fertile Translations in Comparable Corpora: a Morpho-Compos...
Identification of Fertile Translations in Comparable Corpora: a Morpho-Compos...
 

More from I Conferência Internacional de Tradução e Tecnologia

Bernardo Santos - memoQ
Bernardo Santos - memoQBernardo Santos - memoQ
Anabela Barreiro - Alinhamentos
Anabela Barreiro - AlinhamentosAnabela Barreiro - Alinhamentos
José Ramom Campos - RBMT e distâncias linguísticas
José Ramom Campos - RBMT e distâncias linguísticasJosé Ramom Campos - RBMT e distâncias linguísticas
José Ramom Campos - RBMT e distâncias linguísticas
I Conferência Internacional de Tradução e Tecnologia
 
Félix do Carmo e Luís Trigo - Tradutores e máquinas de tradução
Félix do Carmo e Luís Trigo - Tradutores e máquinas de traduçãoFélix do Carmo e Luís Trigo - Tradutores e máquinas de tradução
Félix do Carmo e Luís Trigo - Tradutores e máquinas de tradução
I Conferência Internacional de Tradução e Tecnologia
 
José Ramom Campos - Opentrad
José Ramom Campos - OpentradJosé Ramom Campos - Opentrad
Lucia Specia - Estimativa de qualidade em TA
Lucia Specia - Estimativa de qualidade em TALucia Specia - Estimativa de qualidade em TA
Lucia Specia - Estimativa de qualidade em TA
I Conferência Internacional de Tradução e Tecnologia
 
Hilário Fontes - Tradução automática na CE
Hilário Fontes - Tradução automática na CEHilário Fontes - Tradução automática na CE
Hilário Fontes - Tradução automática na CE
I Conferência Internacional de Tradução e Tecnologia
 
Anabela Barreiro - Hibridização de TA
Anabela Barreiro - Hibridização de TAAnabela Barreiro - Hibridização de TA
Anabela Barreiro - Hibridização de TA
I Conferência Internacional de Tradução e Tecnologia
 
Luísa Coheur - Projecto PT-STAR
Luísa Coheur - Projecto PT-STARLuísa Coheur - Projecto PT-STAR
Belinda Maia - Introdução à tradução automática
Belinda Maia - Introdução à tradução automáticaBelinda Maia - Introdução à tradução automática
Belinda Maia - Introdução à tradução automática
I Conferência Internacional de Tradução e Tecnologia
 

More from I Conferência Internacional de Tradução e Tecnologia (10)

Bernardo Santos - memoQ
Bernardo Santos - memoQBernardo Santos - memoQ
Bernardo Santos - memoQ
 
Anabela Barreiro - Alinhamentos
Anabela Barreiro - AlinhamentosAnabela Barreiro - Alinhamentos
Anabela Barreiro - Alinhamentos
 
José Ramom Campos - RBMT e distâncias linguísticas
José Ramom Campos - RBMT e distâncias linguísticasJosé Ramom Campos - RBMT e distâncias linguísticas
José Ramom Campos - RBMT e distâncias linguísticas
 
Félix do Carmo e Luís Trigo - Tradutores e máquinas de tradução
Félix do Carmo e Luís Trigo - Tradutores e máquinas de traduçãoFélix do Carmo e Luís Trigo - Tradutores e máquinas de tradução
Félix do Carmo e Luís Trigo - Tradutores e máquinas de tradução
 
José Ramom Campos - Opentrad
José Ramom Campos - OpentradJosé Ramom Campos - Opentrad
José Ramom Campos - Opentrad
 
Lucia Specia - Estimativa de qualidade em TA
Lucia Specia - Estimativa de qualidade em TALucia Specia - Estimativa de qualidade em TA
Lucia Specia - Estimativa de qualidade em TA
 
Hilário Fontes - Tradução automática na CE
Hilário Fontes - Tradução automática na CEHilário Fontes - Tradução automática na CE
Hilário Fontes - Tradução automática na CE
 
Anabela Barreiro - Hibridização de TA
Anabela Barreiro - Hibridização de TAAnabela Barreiro - Hibridização de TA
Anabela Barreiro - Hibridização de TA
 
Luísa Coheur - Projecto PT-STAR
Luísa Coheur - Projecto PT-STARLuísa Coheur - Projecto PT-STAR
Luísa Coheur - Projecto PT-STAR
 
Belinda Maia - Introdução à tradução automática
Belinda Maia - Introdução à tradução automáticaBelinda Maia - Introdução à tradução automática
Belinda Maia - Introdução à tradução automática
 

Recently uploaded

UI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentationUI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentation
Wouter Lemaire
 
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of GermanyPresentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
innovationoecd
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Hiroshi SHIBATA
 
Skybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoptionSkybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoption
Tatiana Kojar
 
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and MilvusGenerating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
Zilliz
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
shyamraj55
 
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Tatiana Kojar
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Safe Software
 
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
MichaelKnudsen27
 
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing InstancesEnergy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Alpen-Adria-Universität
 
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdfNunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
flufftailshop
 
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Jeffrey Haguewood
 
dbms calicut university B. sc Cs 4th sem.pdf
dbms  calicut university B. sc Cs 4th sem.pdfdbms  calicut university B. sc Cs 4th sem.pdf
dbms calicut university B. sc Cs 4th sem.pdf
Shinana2
 
A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024
Intelisync
 
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdfMonitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Tosin Akinosho
 
AWS Cloud Cost Optimization Presentation.pptx
AWS Cloud Cost Optimization Presentation.pptxAWS Cloud Cost Optimization Presentation.pptx
AWS Cloud Cost Optimization Presentation.pptx
HarisZaheer8
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Malak Abu Hammad
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
Chart Kalyan
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
Octavian Nadolu
 
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
saastr
 

Recently uploaded (20)

UI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentationUI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentation
 
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of GermanyPresentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
 
Skybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoptionSkybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoption
 
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and MilvusGenerating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
 
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
 
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
 
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing InstancesEnergy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
 
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdfNunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
 
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
 
dbms calicut university B. sc Cs 4th sem.pdf
dbms  calicut university B. sc Cs 4th sem.pdfdbms  calicut university B. sc Cs 4th sem.pdf
dbms calicut university B. sc Cs 4th sem.pdf
 
A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024
 
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdfMonitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
 
AWS Cloud Cost Optimization Presentation.pptx
AWS Cloud Cost Optimization Presentation.pptxAWS Cloud Cost Optimization Presentation.pptx
AWS Cloud Cost Optimization Presentation.pptx
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
 
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
 

Lucia Specia - SMT e pós-edição

  • 1. Statistical Machine Translation and Post-editing I Conferˆencia Internacional de Tradu¸c˜ao e Tecnologia Porto, 14 Maio 2013 Lucia Specia (Thanks to Wilker Aziz for many of the SMT slides) Department of Computer Science University of Sheffield http://www.dcs.shef.ac.uk/~lucia/ Porto (14/05/2013) SMT and PE 1 / 67
  • 2. Overview • Machine Translation - task and challenges • Statistical Machine Translation Word-based models Phrase-based models Syntax-based models • Post-editing: Process and research Practice with PET Porto (14/05/2013) SMT and PE 2 / 67
  • 3. The task of Machine Translation (MT) The boy ate an apple O menino comeu uma ma¸ca BUT He said that the bottle floated into the cave ? Dijo que la botella entro a la cueva flotando 虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。 However , the sky remained clear under the strong north wind . Porto (14/05/2013) SMT and PE 3 / 67
  • 4. Challenges in MT: Ambiguity • Lexical ambiguity: different words meanings may translate differently e.g. book the flight ⇒ reservar read the book ⇒ libro e.g. kill a man ⇒ matar kill a process ⇒ acabar • Syntactic ambiguity: e.g. John hit the dog with the stick ⇒ John golpeo el perro con el palo ⇒ John golpeo el perro que tenia el palo • Pronoun resolution: e.g. The computer outputs the data; it is fast ⇒ La computadora imprime los datos; es rapida VS. e.g. The computer outputs the data; it is stored in ascii ⇒ La computadora imprime los datos; estan almacendos en ascii Porto (14/05/2013) SMT and PE 4 / 67
  • 5. Challenges in MT: Divergences • Structural divergences: e.g. The bottle floated into the cave ⇒ La botella entro a la cueva flotando (The bottle entered the cave floating) • Different word orders: English is: subject – verb – object Japanese is: subject – object – verb English: IBM bought Lotus yesterday Japanese: IBM Lotus bought yesterday Porto (14/05/2013) SMT and PE 5 / 67
  • 6. Challenges: Non-literal meaning • Idioms e.g. He finally kicked the bucket at the hospital ⇒ Ele finalmente chutou o balde no hospital VS. ⇒ Ele finalmente bateu as botas no hospital • Multi-word expressions e.g. Do take the long waiting list for organ donation in this country into account ⇒ Considere a longa lista de espera para doa¸c˜ao de ´org˜aos neste pa´ıs Porto (14/05/2013) SMT and PE 6 / 67
  • 7. Requires deep language processing • One of the most challenging problems in NLP • Requires Full understanding of source text (analysis) Generation of a coherent, accurate, fluent target text (synthesis) Porto (14/05/2013) SMT and PE 7 / 67
  • 8. Statistical Machine Translation P(French|English) • Statistical Machine Translation (SMT) learn how to generate translations using statistical methods • Early 1990s by IBM. Adopted by Google, etc. • Idea is older: Warren Weaver (1949) When I look at an article in Russian, I say: “This is really written in English, but it has been coded in some strange symbols. I will now proceed to decode.” Inspired by WWII code-breaking Approach not feasible with early computers – idea not pursued Porto (14/05/2013) SMT and PE 8 / 67
  • 9. Probabilistic Model • Output of decoder depends probabilistically on the input • To translate French (F) into English (E): Given a French sentence F search for English sentence E∗ that maximises P(E|F) Porto (14/05/2013) SMT and PE 9 / 67
  • 10. Probabilistic Model • Find English sentence that maximises P(E|F), i.e. E∗ = argmax E P(E|F) = argmax E P(E)·P(F|E) P(F) Bayes Rule P(F) constant across different E, so: E∗ = argmax E P(E) · P(F|E) drop constant divisor P(F) Porto (14/05/2013) SMT and PE 10 / 67
  • 11. Noisy Channel Model & SMT (ctd) E∗ = argmax E P(E|F) = argmax E P(E) · P(F|E) • Why not model P(E|F) directly? Would need very good probability estimates Divide and conquer! • Decomposition to P(E) · P(F|E) allows one to use less reliable probabilities P(E) worries about good English — fluency • ensures that E has words in the right order and fit together P(F|E) worries about French that translates English — faithfulness • ensures that E has words that generally translate F P(E) and P(F|E) can be trained independently Porto (14/05/2013) SMT and PE 11 / 67
  • 12. Main Components for Translation (F → E) • Translation model (TM): P(F|E) Faithfulness: TMs learned by analysing large amounts of parallel text • Judge which (F, E) pairs are likely to be translations of each other • Language model (LM): P(E) Fluency: LMs created from large volume of (fluent) target language sentences • Learn which word sequences are common/more probable in E • A Decoder: (argmax) Search algorithm to find E∗ • V. large search space — heuristic search approach used Porto (14/05/2013) SMT and PE 12 / 67
  • 13. Learning Translation Models - P(F|E) • Requires a bilingual corpus e.g. European/Canadian/Hong Kong parliaments, subtitles, Bible • Bilingual corpus must be sentence-aligned: • Can we estimate P(F|E) from whole sentences? Porto (14/05/2013) SMT and PE 13 / 67
  • 14. Learning Translation Models - Word-based SMT • Break sentences into smaller units: words are a good starting point • Learn translation probabilities by word aligning a sentence-aligned corpus: Zenish Uh useh Uh jejje Yiguo useh English A home A garden I arrived home Porto (14/05/2013) SMT and PE 14 / 67
  • 15. Learning Translation Models - Word-based SMT • Break sentences into smaller units: words are a good starting point • Learn translation probabilities by word aligning a sentence-aligned corpus: Zenish Uh useh Uh jejje Yiguo useh • The same word happens in source 1 and 3 English A home A garden I arrived home Porto (14/05/2013) SMT and PE 15 / 67
  • 16. Learning Translation Models - Word-based SMT • Break sentences into smaller units: words are a good starting point • Learn translation probabilities by word aligning a sentence-aligned corpus: Zenish Uh useh Uh jejje Yiguo useh • Could we expect the same in the target side? English A home A garden I arrived home Porto (14/05/2013) SMT and PE 16 / 67
  • 17. Learning Translation Models - Word-based SMT • Break sentences into smaller units: words are a good starting point • Learn translation probabilities by word aligning a sentence-aligned corpus: Zenish Uh useh Uh jejje Yiguo useh English A home A garden I arrived home • useh = home Porto (14/05/2013) SMT and PE 17 / 67
  • 18. Learning Translation Models - Word-based SMT • Break sentences into smaller units: words are a good starting point • Learn translation probabilities by word aligning a sentence-aligned corpus: Zenish Uh useh Uh jejje Yiguo useh • What about the contexts? English A home A garden I arrived home Porto (14/05/2013) SMT and PE 18 / 67
  • 19. Learning Translation Models - Word-based SMT • Break sentences into smaller units: words are a good starting point • Learn translation probabilities by word aligning a sentence-aligned corpus: Zenish Uh useh Uh jejje Yiguo useh English A home A garden I arrived home • We can align them: Yiguo = I; Yiguo = arrived; Uh = A Porto (14/05/2013) SMT and PE 19 / 67
  • 20. Learning Translation Models - Word-based SMT • Break sentences into smaller units: words are a good starting point • Learn translation probabilities by word aligning a sentence-aligned corpus: Zenish Uh useh Uh jejje Yiguo useh English A home A garden I arrived home • And reuse this knowledge to align more sentences: Uh = A Porto (14/05/2013) SMT and PE 20 / 67
  • 21. Learning Translation Models - Word-based SMT • Break sentences into smaller units: words are a good starting point • Learn translation probabilities by word aligning a sentence-aligned corpus: Zenish Uh useh Uh jejje Yiguo useh English A home A garden I arrived home • And again: jejje = garden Porto (14/05/2013) SMT and PE 21 / 67
  • 22. Learning Translation Models - Word-based SMT (ctd) Word-alignment: • Identify correspondences between two languages at the word level • Basis for word-based SMT, first step for other approaches • Alignment can be learned using Expectation Maximization (EM) Treat all alternative word alignments as equally likely Observe across sentences that Zenish useh often links to English home • Increase probability of this word pair aligning • Has knock-on effect for alignment of other words Iteratively redistribute probabilities, until they identify most probable links for each word (convergence) Word alignment commonly done using the IBM Models 1-5 Porto (14/05/2013) SMT and PE 22 / 67
  • 23. Learning Translation Models - Word-based SMT (ctd) • IBM models produce a probabilistic dictionary for the whole parallel corpus: contras contras 0.0285714 contras los 0.0000011 contras esos 0.0000812 richer enriquecida 0.0476190 richer mucho 0.0000502 richer enriqueci 0.0769231 richer variado 0.0066225 Porto (14/05/2013) SMT and PE 23 / 67
  • 24. Learning Translation Models - Word-based SMT (ctd) Back to the translation model: At translation (decoding time), for a new sentence to translate, take the set of translations that jointly maximise the whole translation probability E∗ = argmax E P(F|E) • Generates alternative translations using different translation options for words and different word orders • Selects the one with maximum score: no notion of context. How about ambiguity? What about the context and the fluency in the target language? Porto (14/05/2013) SMT and PE 24 / 67
  • 25. Learning Language Models - Word-based SMT Language model: P(E) E∗ = argmax E P(F|E) · P(E) • Takes different translation options, tests how “common” each of them are in the target language as a whole sentence (some context) • P(E) = probability of strings E based on relative frequencies in a large corpus of language E Google the house is small 144,000 the house is little 85,300 small house 2,330,000 little house 3,330,000 Porto (14/05/2013) SMT and PE 25 / 67
  • 26. Learning Language Models - Word-based SMT (ctd) • Model P(E) by decomposing into single-word probabilities given previous words, under Markov assumption: only the previous n-1 words matter for predicting a word. E.g for trigram models, n = 3 P(e1) · P(e2|e1) · P(e3|e1, e2) · P(e4|e2, e3) · · · P(en|en−2, en−1) • Relative frequencies to compute these probabilities. E.g. trigrams: P(e3|e1, e2) = count(e1e2e3) count(e1e2) P(little|house, is) = count(house is little) count(house is) Porto (14/05/2013) SMT and PE 26 / 67
  • 27. Word-based SMT – Limitations • It is difficult to do alignment, and hence learn a TM, for languages that are structurally quite different, with different words orders Considering all possible word orders – too costly Poor reordering model: flexibility vs noise • Some languages may have different notions of what counts as a word Donaydampfshiffahrtsgesellschaftskapitaenskajuetenschluesseloch The keyhole of the door of the cabin of the captain of a steamship company operating on the Danubea a http://www.chebucto.ns.ca/~waterbuf/longwrds.html Porto (14/05/2013) SMT and PE 27 / 67
  • 28. Phrase-based SMT • Leading approach since 2003 No voy1 a la2 casa3 → I am not going1 to the2 house3 it seems to1 me2 → me1 parece2 Je1 ne vais pas2 `a la3 maison4 → I1 am not going2 to the3 house4 Eu1 sinto saudade de vocˆe2 → I1 miss you2 I1 miss you2 → Eu1 sinto sua falta2 natuerlich1 hat2 John3 spass am4 spiel5 → of course1 John2 has3 fun with the4 game5 • More intuitive and reliable alignments Account for reordering within the phrases Phrases can still be reordered • Can we learn phrase translation probability distribution from the parallel corpus using EM? 1 src-tgt sentence → O(n4 ) phrase pairs Porto (14/05/2013) SMT and PE 28 / 67
  • 29. Phrase-based SMT - Phrases from word alignments • Extract phrase pairs that are consistent with the word alignment • Phrase: sequence of tokens, not linguistically motivated • Word alignment produced by any of the IBM Models - only done once reanudaci´on del per´ıodo de sesiones resumption of the session 1 resumption ↔ reanudaci´on 2 of the ↔ del 3 session ↔ per´ıodo de sesiones 4 resumption of the ↔ reanudaci´on del 5 of the session ↔ del sesiones 6 resumption of the session ↔ reanudaci´on del sesiones Porto (14/05/2013) SMT and PE 29 / 67
  • 30. Phrase-based SMT - Computing phrase probability distributions 1 Extract phrase pairs from word aligned parallel corpus √ 2 Extract counts of those phrases from large parallel corpus: φ(¯f |¯e) = count(¯f , ¯e) count(¯e) 3 Store phrases and their probabilities in a phrase table Probabilistic dictionary of phrases Porto (14/05/2013) SMT and PE 30 / 67
  • 31. Phrase-based SMT - Limitation of Noisy Channel Model E∗ = argmax E P(F|E) · P(E) • Which component is more important? P(F|E) ? P(E) ? • Depends on the corpus used (more parallel data makes P(F|E) more reliable), language-pair, etc. Porto (14/05/2013) SMT and PE 31 / 67
  • 32. Phrase-based SMT - Log-linear Model (ctd) Assign weights λi to components (feature functions) hi (f, e): e∗ = argmax e n i=1 λi hi (f, e) Components 1 P(E) 2 P(F|E) Weights 1 λP(E) 2 λP(F|E) Feature Functions 1 h1 = logP(E) 2 h2 = logP(F|E) Benefits 1 Extensible: other components are used: e.g. reordering, word and phrase penalties 2 Weights can be tuned, i.e., learned from examples to minimise error in a subset of the parallel corpus, e.g. MERT Porto (14/05/2013) SMT and PE 32 / 67
  • 33. Phrase-based SMT - Decoding (ctd) Search problem: segmenting source sentence and selecting phrase translations: • All phrases matching source words are selected from the phrase table J’ ai les yeux noirs . I have the eyes black . me has them eye dark , I have eyes espresso ! I am the eyes somber . I did some black eyes . I had black eyes . I have black eyes . black eyes I have . • The goal of the decoder is to select the phrases whose combination (in a given order) yields the highest score for the log-linear model Porto (14/05/2013) SMT and PE 33 / 67
  • 34. Phrase-based SMT - Decoding (ctd) Searching = incrementally construct the translation hypotheses by trying out several possibilities • Generating target words in sequence, from left to right • Incrementally compute the overall log-linear model score for each hypothesis • Heuristics to prune search space, e.g. stack-based beam search Use one stack for of every length of partial hypotheses Limited space for partial hypotheses, only keep those that are promising, e.g. model score is close to that of the best partial hypothesis so far Porto (14/05/2013) SMT and PE 34 / 67
  • 35. Phrase-based SMT - Decoding (ctd) Search space Hypothesis • Covered source words • Target (output) words • Model score (using the combined probabilities) Porto (14/05/2013) SMT and PE 35 / 67
  • 36. Hierarchical and Syntax-based SMT • PBSMT still has trouble with long-distance reorderings • Ways of bringing structure and linguistic knowledge into the transfer rules of the phrase table Porto (14/05/2013) SMT and PE 36 / 67
  • 37. Hierarchical SMT - Motivation Introduce structure into phrase-based SMT models to deal with long-distance reordering Ich werde Ihnen die entsprechenden Anmerkungen aush¨andigen I shall be passing on to you some comments Porto (14/05/2013) SMT and PE 37 / 67
  • 38. Hierarchical SMT - Motivation Introduce structure into phrase-based SMT models to deal with long-distance reordering Ich werde Ihnen die entsprechenden Anmerkungen aush¨andigen I shall be passing on to you some comments • How can we get a phrase for shall be passing on? Porto (14/05/2013) SMT and PE 38 / 67
  • 39. Hierarchical SMT - Motivation Introduce structure into phrase-based SMT models to deal with long-distance reordering Ich werde Ihnen die entsprechenden Anmerkungen aush¨andigen I shall be passing on to X you X some X comments X • How can we get a phrase for shall be passing on? • We cannot, unless we get to you some comments along Porto (14/05/2013) SMT and PE 39 / 67
  • 40. Hierarchical SMT - Motivation Introduce structure into phrase-based SMT models to deal with long-distance reordering Ich werde Ihnen die entsprechenden Anmerkungen aush¨andigen I shall be passing on to you some comments • How can we get a phrase for shall be passing on? • We cannot, unless we get to you some comments along • Unless we replace all those words by a variable Porto (14/05/2013) SMT and PE 40 / 67
  • 41. Hierarchical SMT - Motivation (ctd) shall be passing on to you some comments werde Ihnen die entsprechenden Anmerkungen aush¨andigen shall be passing on to you some comments werde Ihnen die entsprechenden Anmerkungen aush¨andigen shall be passing on X werde X aush¨andigen Porto (14/05/2013) SMT and PE 41 / 67
  • 42. Hierarchical SMT - basics Learnt from word-aligned parallel corpora in the same way as before • Based on the fact that language has recursive structures • Phrases within other phrases treated as nonterminals: replaced by X • No linguistic constraints added - yet, some structure Porto (14/05/2013) SMT and PE 42 / 67
  • 43. Hierarchical SMT - basics (ctd) shall be passing on to you some comments werde Ihnen die entsprechenden Anmerkungen aush¨andigen shall be passing on X1 some comments werde X1 die entsprechenden Anmerkungen aush¨andigen shall be passing on X1 X2 werde X1 X2 aush¨andigen Porto (14/05/2013) SMT and PE 43 / 67
  • 44. Hierarchical SMT - phrase-table [X] → shall be passing on X1 X2 | werde X1 X2 aush¨andigen [X] → shall be passing on X3 | werde X3 aush¨andigen [X] → to you | Ihnen [X] → some comments | die entsprechenden Anmerkungen [X] → to you some comments | Ihnen die entsprechenden Anmerkungen Learn a bilingual (synchronous) set of context-free rules about how to translate F into E Porto (14/05/2013) SMT and PE 44 / 67
  • 45. Syntax-based SMT • Hierarchical models are not very informative (Xs) and suffer from an exponential number of rules • Syntax-based SMT overcomes these limitations by using linguistic categorise to label nodes • A syntactic parser is required at pre-processing time on at least one language (the other could have Xs) • Idea: Learn a bilingual (synchronous) set of linguistically informed context-free rules about how to translate F into E Rules are learnt from the word-alignment, constrained by syntactic categories Porto (14/05/2013) SMT and PE 45 / 67
  • 46. Syntax-based SMT (ctd) S PRP J’ VP VB ai NP DT les NN yeux JJ noirs S PRP I VP VB have NP JJ black NN eyes J’ ai les yeux noirs I have black eyes Standard constraints on rule construction: • Single nonterminal on the left • Consistent with word-alignment • Nonterminals on the right must align one-to-one Porto (14/05/2013) SMT and PE 46 / 67
  • 47. Syntax-based SMT (ctd) Grammar PRP → J’ | I JJ → noirs | black NP → les yeux JJ | JJ eyes VP → ai NP | have NP S → PRP VP | PRP VP Decoding becomes a (probabilistic) parsing problem! Porto (14/05/2013) SMT and PE 47 / 67
  • 48. Summary on SMT • Adding linguistic knowledge to SMT is a trend: Stronger theoretical motivation Deals with reordering, sparsity, ambiguity Depends on (quality of) parsers Higher complexity • Performance: Hard to quantify in absolute terms Most language pairs → phrase-based or hierarchical SMT does better Language pairs with long-distance reordering → syntax-based SMT does better • Other open issues: Limited use of linguistic information - at most syntax, some attempts to use shallow semantics - not yet there Limited use of context - sentence-level Domain discrepancies - mistranslations, unknown words • Overall, to reach publishable quality, human checking/revision necessary: post-editing Porto (14/05/2013) SMT and PE 48 / 67
  • 49. Definitions Note: Some slides are based on those by Sharon O’Brien (tutorial on MT post-editing at AMTA-2010). What is post-editing? • Same as translation? • Same as revision? • A new profession? New skills? Who does the post-editing? What are the quality expectations? Porto (14/05/2013) SMT and PE 49 / 67
  • 50. Degrees/types of post-editing • Light (fast/rapid/gist) post-editing: essential corrections • Full post-editing: more corrections, higher quality, closer to revision • Monolingual post-editing: no access to source text Decided by: • User requirements • Volume • Quality expectations • Turn-around time, etc. Porto (14/05/2013) SMT and PE 50 / 67
  • 51. Quality of post-editing • Can MT+PE (possibly +revision) produce the same level of quality as HT+revision? • Dependent on the quality of MT systems: variable for language pair, text type/genre & domain, quality of source • Dependent on quality requirements • Different outcomes: Autodesk reports lower error rates with MT+PE [Aut11] Tilde observed higher error rate with MT+PE [SPSV11] - under-resourced highly-inflected languages: lower MT quality Porto (14/05/2013) SMT and PE 51 / 67
  • 52. Ways of measuring quality of MT for PE • Automatic MT evaluation metrics ? BLEU/TER/METEOR... - automatic, but do not correlate well with PE effort • Error counts: compare source with raw MT output - manual, costly Can we use the “standard” error typologies for HT? Do MT and HT make same mistakes? QTLaunchPad project Multidimensional Quality Metrics for MT and HT, for manual and (semi-)automatic evaluation (QE): http://www.qt21.eu/launchpad/ • % Changes made: compare post-edited text with raw MT output - automatic • Time to make changes: record time to fix raw MT - automatic • Estimated effort: take source text and raw MT output and estimate PE effort prior to post-editing (QE) - automatic, less precise Is this the same as fuzzy match level in TM systems? Porto (14/05/2013) SMT and PE 52 / 67
  • 53. Aspects involved in post-editing effort • Temporal: time to post-edit (variable across translators) • Technical: no./% edits made, keys pressed • Cognitive: time, pauses, movements Studies on PE process: • Is PEMT faster than HT? Yes • Is PEMT less keyboard intensive than HT? Yes • Is PEMT less cognitively demanding than HT? Cognitive effort is the most difficult to estimate: active research area [Kop12, KARS12] • PE more tedious/tiring? Mixed opinions Check work by Michael Carl - CBS (Translog) and Sharon O’Brien - DCU (eye-tracking) Porto (14/05/2013) SMT and PE 53 / 67
  • 54. Pricing models Open question. Some common practices: • Paying as fuzzy segment matches (source-based) • Paying as % of edits performed (MT&PE-based) • Paying a fee based on time spent (less common) Porto (14/05/2013) SMT and PE 54 / 67
  • 55. PE Guidelines • No standard guidelines, but good practices (e.g. TAUS/CNGL) • Guidelines need to clearly specify level of post-editing and quality expectations • Guidelines need to clearly differentiate between essential and preferential changes • Ideally, guidelines should be system-, language-, purpose-specific Porto (14/05/2013) SMT and PE 55 / 67
  • 56. General PE Guidelines for “good enough” quality - TAUS/CNGL • Aim for semantically correct translation. • Ensure that no information has been accidentally added or omitted. • Edit any offensive, inappropriate or culturally unacceptable content. • Use as much of the raw MT output as possible. • Basic rules regarding spelling apply. • No need to implement corrections that are of a stylistic nature only. • No need to restructure sentences solely to improve the natural flow of the text. http://www.translationautomation.com/postediting/ machine-translation-post-editing-guidelines Porto (14/05/2013) SMT and PE 56 / 67
  • 57. General PE Guidelines for HT quality - TAUS/CNGL • Aim for grammatically, syntactically and semantically correct translation. • Ensure that key terminology is correctly translated and that untranslated terms belong to the clients list of Do Not Translate terms. • Ensure that no information has been accidentally added or omitted. • Edit any offensive, inappropriate or culturally unacceptable content. • Use as much of the raw MT output as possible. • Basic rules regarding spelling, punctuation and hyphenation apply. • Ensure that formatting is correct. http://www.translationautomation.com/postediting/ machine-translation-post-editing-guidelines Porto (14/05/2013) SMT and PE 57 / 67
  • 58. IT PE Guidelines - Midori Tatsumi’s PhD thesis • What needs to be fixed: Non-translatable items, such as command and variable names, that have been translated. Please put it back to English. Inappropriately translated general IT terms. Mistranslation. Word orders that are inappropriate to the level that the sentence has become impossible or difficult to comprehend. Comprehensible but extremely unnatural or inappropriate expressions. Inappropriate postpositions and conjugations. • What are preferred to be standardised: Repetitive items, such as procedures and steps. Styles of the section titles in a user manual and help files. • What does not need to be fixed: Ending styles of bulleted items. Placeholders, such as &ProductNameShort; and &ProductNameLong; Punctuation. http://doras.dcu.ie/16062/1/SAKURA_final_revised.pdf Porto (14/05/2013) SMT and PE 58 / 67
  • 59. Monolingual PE Guidelines - ACCEPT project • Try and edit the text by making it more fluent and clearer based on how you interpret its meaning. For example, try to rectify word order and spelling when they are inappropriate to the extent that the text has become impossible or difficult to comprehend • If words, phrases, or punctuation in the text are completely acceptable, try and use them (unmodified) rather than substituting them with something new and different. http://www.accept.unige.ch/Products/D6.2.2_Seminar_ Material_on_Post-Editing_Edition_2.pdf Porto (14/05/2013) SMT and PE 59 / 67
  • 60. PE Technology (tools) A myriad of tools appeared in the marked in the last couple of years: • TM tools that now integrate MT, e.g. SDL TRADOS, WordFast • MT tools (with or without TM): e.g. Systran, Google Translate Toolkit, Caitra • PE tools that can use different MT systems: MemoQ, MemSource, SmartMate, Coach, ... More and more functionalities: spell/grammar checkers, multiple MT, MT+TM, interactive MT, terminology, word/phrase alignment, etc. Limited (if any) logging of translator’s activities for productivity analysis – not enough for research. Porto (14/05/2013) SMT and PE 60 / 67
  • 61. PE Technology (tools) Research-oriented tools: • Translog (not open source) • OmegaT (TM only) • Appraise (MT eval through ranking, no detailed statistics for PE) • PET: http://pers-www.wlv.ac.uk/~in1676/pet/index.html • Coming soon: open source tool promised as part of EU projects CASMACAT & MATECAT Porto (14/05/2013) SMT and PE 61 / 67
  • 62. What can we get from PET? Comparison between HT and multiple MT systems [dSAS11] • PET to compare 3 MT systems vs translation from scratch • Sitcom and movie subtitles: Translating from scratch can be 73% slower than post-editing SMT systems (Google and Moses) performed the best How often a system produced an output that was faster to PE than other systems: System Google Moses Systran Trados Google - 139 161 187 Moses 69 - 122 164 Systran 69 106 - 145 Trados 48 67 89 - How often post-editing a system output was faster than translating: System Faster than human translation Google 94% Moses 86.8% Systran 81.20% Trados 72.40% Porto (14/05/2013) SMT and PE 62 / 67
  • 63. What can we get from PET? Evaluation of an MT system for translation of subtitles (space constraints) [AdSS12] • Add compression constraints to an MT system to generate length compliant subtitles • PET to guide post-editing according to length and time requirements for every unit Show space limitation (attribute) Change colour of translation if too long Offer shorter paraphrases • PET to evaluate edit distance and length System Dexter How I Met.. Terra Nova TER ↓ LENGTH TER ↓ LENGTH TER ↓ LENGTH Mosest 30.3 116.0 20.0 108.5 33.8 120.2 Google 63.6 156.5 52.8 144.3 63.1 152.1 MosesLP2 29.5 115.5 21.0 109.1 33.4 119.3 MosesLP1 28.3 115.8 20.7 110.0 34.8 119.8 Porto (14/05/2013) SMT and PE 63 / 67
  • 64. Hands-on activity with PET • en-pt-hiding-and-assessing.pec – bigbang-pe-2.pej • en-pt-hiding-and-assessing.pec – bigbang-ht-2.pej • fapesp.pec – fapesp-annotated.pej • fapesp-mono.pec – fapesp-annotated.pej • subs.pec – greys.08x06.v1-v1.en-br.s2.1.pej Advanced use: • Configuration file: .pec • Input/output XML files • How to build a PEJ file from raw files: pej.bat • How to interpret the results: scripts/readper.pl Porto (14/05/2013) SMT and PE 64 / 67
  • 65. For more on the research-side Workshop on Post-Editing Technology and Practice: • AMTA 2012: https://sites.google.com/site/wptp2012/ • MT Summit 2013 (deadline for papers 28/05/2013): https://sites.google.com/site/mts2013wptp/ Porto (14/05/2013) SMT and PE 65 / 67
  • 66. References I Wilker Aziz, Sheila Castilho Monteiro de Sousa, and Lucia Specia. Cross-lingual sentence compression for subtitles. In The 16th Annual Conference of the European Association for Machine Translation, EAMT ’12, pages 103–110, Trento, Italy, May 2012. Autodesk. Translation and Post-Editing Productivity. In http: // translate. autodesk. com/ productivity. html , 2011. Sheila C. M. de Sousa, Wilker Aziz, and Lucia Specia. Assessing the post-editing effort for automatic and semi-automatic translations of DVD subtitles. In Conference Recent Advances in Natural Language Processing 2011, pages 97–103, Hissar, Bulgaria, 2011. Porto (14/05/2013) SMT and PE 66 / 67
  • 67. References II Maarit Koponen, Wilker Aziz, Luciana Ramos, and Lucia Specia. Post-editing time as a measure of cognitive effort . In AMTA 2012 Workshop on Post-Editing Technology and Practice (WPTP 2012), pages 11–20, San Diego, USA, October 2012. Association for Machine Translation in the Americas (AMTA). Maarit Koponen. Comparing human perceptions of post-editing effort with post-editing operations. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 181–190, Montr´eal, Canada, 2012. R. Skadins, M. Purins, I. Skadina, and A. Vasiljevs. An evaluation of SMT in localization to under-resourced inflected languages. In Proceedings of the 15th Conference of the European Association for Machine Translation, Leuven, 2011. Porto (14/05/2013) SMT and PE 67 / 67