The seminar presentation by Hrishikesh B discusses the concept of machine translation (MT), focusing on statistical approaches that utilize previously translated texts to improve translation accuracy. It outlines techniques such as the transfer model and the use of parallel corpora to estimate translation probabilities and refine decoding processes. Additionally, the presentation highlights the challenges in statistical MT, including the need for large, aligned corpora and the under-exploration of these techniques for Indian languages.

1. SEMINAR PRESENTATION
PRESENTED BY:
HRISHIKESH B
S7 CSE ALPHA
Univ reg:11012288

2. Machine Translation (MT) can be defined as the use of
computers to automate some or all of the process of translating
from one language to another.
 MT is an area of applied research that draws ideas and
techniques from linguistics, computer science, Artificial
Intelligence (AI), translation theory, and statistics.
This seminar discusses the statistical approach to MT, which
was first suggested by Warren Weaver in 1949 [Weaver, 1949],
but has found practical relevance only in the last decade or so.

3. • The Direct Approach
• Consider the sentence,
We will demand peace in the country.
To translate this to Hindi, we do not need to identify the thematic
roles universal concepts. We just need to do morphological
analysis, identify constituents, reorder them according to the
constituent order in Hindi (SOV with pre-modifiers), lookup the
words in an English-Hindi dictionary, and inflect the Hindi words
appropriately! There seems to be more to do here, but these are
operations that can usually be performed more simply and reliably.

4. • The Transfer Approach
The transfer model involves three stages: analysis, transfer,
and generation. In the analysis stage, the source language
sentence is parsed, and the sentence structure and the
constituents of the sentence are identified. In the transfer
stage, transformations are applied to the source language
parse tree to convert the structure to that of the target
language. The generation stage translates the words and
expresses the tense, number, gender etc. in the target
language

5. The approaches that we have seen so far, all use human-encoded
linguistic knowledge to solve the translation problem.
We will now look at some approaches that do not explicitly use such
knowledge, but instead use a training corpus (plur. corpora) of already
translated texts — a parallel corpus — to guide the translation
process.
A parallel corpus consists of two collections of documents: a source
language collection, and a target language collection. Each document
in the source language collection has an identified counterpart in the
target language collection.

6. Statistical MT models take the view that every sentence in the
target language is a translation of the source language
sentence with some probability.
The best translation, of course, is the sentence that has the
highest probability.
The key problems in statistical MT are: estimating the
probability of a translation, and efficiently finding the
sentence with the highest probability.

7.  Every sentence in one language is a possible translation of any
sentence in the other.
Assign to every pair of sentences (S, T) a probability, Pr(T|S), to be
interpreted as the probability that a translator will produce T in the
target language when presented with S in the source language
Pr(T|S) to be very small for pairs like (Lematin je me brosse les dents
|President Lincoln was a good lawyer)
And relatively large for pairs like (Le president Lincoln btait un bon
avocat | President Lincoln was a good lawyer).

8. PROBLEM OF MACHINE TRANSLATION:
“Given a sentence T in the target language, we seek
the sentence S from which the translator produced
T.”
Chance of error is minimized by choosing that
sentence S that is most probable given T. Thus, we
wish to choose S so as to maximize Pr(S | T).

9.  The translation problem can be described as modeling the probability distribution
P(S|T), where S is a string in the source language and T is a string in the target
language.
 Using Bayes’ Rule, this can be rewritten
Pr(S|T) = Pr(T|S)Pr(S)/Pr(T)
= Pr(T|S)Pr(S) [The denominator on the right of this equation does not
depend on S, and so it suffices to choose the S that maximizes the product
Pr(S)Pr(TIS).]
 Pr(T|S) is called the “translation model” (TM).
 Pr(S) is called the “language model” (LM).
The LM should assign probability to sentences which are “good English”.

10. A Source Language Model and a Translation Model furnish a probability
distribution over source-target sentence pairs (S,T). The joint probability
Pz (S, T) of the pair (S, T) is the product of the probability Pr (S) computed
by the language model and the conditional probability Pr (T I S) computed
by the translation model. The parameters of these models are estimated
automatically from a large database of source-target sentence pairs using
a statistical algorithm which optimizes, in an appropriate sense, the fit
between the models and the data.

11. A Decoder performs the actual translation. Given a
sentence T in the target language, the decoder chooses a
viable translation by selecting that sentence in the
source langnage for which the probability Pr (S [ T) is
maximum.

12. Why not calculate Pr(S|T) directly ,rather than break Pr(S|T) into
two terms, Pr(S) and Pr(T|S), using Bayes’ rule.
Pr(T|S)Pr(S) decomposition allows us to be sloppy
Pr(S) worries about good English i.e the kind of sentences that are
likely in language S.
Pr(S|T) worries about match with words i.e match of English word
with French.
The two can be trained independently

15. What’s P(s)?
P(STRING s1s2s3….sn)
P(s1, s2, s3 … si)
Using the chain rule
Pr (s1s2 ...sn)
= Pr (s1) Pr (s2 ls1) ... Pr (sn |s1s2 ...Sn_,)

16. ( ) ( | ) ( | , ) ( | , , ) ( | , , ) 1 2 1 3 1 2 4 1 2 3 i 1 2 i1 P s P s s P s s s P s s s s P s s s s
• Because there are so many histories, we cannot simply treat
each of these probabilities as a separate parameter.
• One way to reduce the number of parameters is to place each
of the histories into an equivalence class in some way and
then to allow the probability of an object word to depend on
the history only through the equivalence class into which that
history falls.
• The choice of word si depends only on the n words before si
( | , , , , , ) ( | , , , ) 1 2 4 3 2 1 4 3 2 1  i i i i i i i i i i P s s s s s s s P s s s s s

17. In an N-gram model ,the probability of a word given all
previous words is approximated by the probability of the
word given the previous N-1 words.
The approximation thus works by putting all contexts that
agree in the last N-1 words into one equivalence class.
With N = 2, we have what is called the bigram model, and N
= 3 gives the trigram model.

18. N-gram probabilities can be computed in a straightforward manner
from a corpus. For example, bigram probabilities can be calculated as:
P(wn|wn−1) =count(wn−1wn)/Σw count(wn−1w)
Here count(wn−1wn) denotes the number of occurrences of the the
sequence wn−1wn. The denominator on the right hand side sums over all
word w in the corpus — the number of times wn−1 occurs before any
word.
 Since this is just the count of wn−1, we can write the above equation as,
P(wn|wn−1) =count(wn−1wn)/count(wn−1)

19.  For example, to calculate the probability of the sentence, “all men are equal”, we
split it up as,
P(all men are equal) = P(all|start)P(men|all)P(are|men)P(equal|are)
where start denotes the start of the sentence, and P(all|start) is the
probability that a sentence starts with the word all.
 Given the bigram probabilities in table,
Bigram Probability
start all 0.16
all men 0.09
men are 0.24
are equal 0.08
 The probability of the sentence is calculated as :
P(all men are equal) = 0.16 × 0.04 × 0.20 × 0.08 = 0.00028

20.  So, we’ve got P(S), let’s talk P(T|S)
P T S P S
P S T  
( | ) P T S P S
( | ) ( )
( | ) ( )
P T
( )

21.  For simple sentences, it is reasonable to think of the French translation of an
English sentence as being generated from the English sentence word by word.
 Thus, in the sentence pair (Jean aime Marie | John loves Mary) we feel that
John produces Jean, loves produces aime, and Mary produces Marie. We say
that a word is aligned with the word that it produces.
 Not all pairs of sentences are as simple as this example. In the pair (Jean
n'aime personne | john loves nobody), we can again align John with Jean and
loves with aime, but now, nobody aligns with both n' and personne.
 Sometimes, words in the English sentence of the pair align with nothing in
the French sentence, and similarly, occasionally words in the French member
of the pair do not appear to go with any of the words in the English sentence.

22. An alignment indicates the origin in the English sentence of
each of the words in the French sentence.
The number of French words that an English word produces in
a given alignment its fertility in that alignment.
Sometimes, a French word will appear quite far from the
English word that produced it. We call this effect distortion.
Distortions will, for example, allow adjectives to precede the
nouns that they modify in English but to follow them in
French

23. The Translation Model: P(T|S)
 Alignment model:
assume there is a transfer relationship between source and target
words
 not necessarily 1-to-1
Example
S = w1 w2 w3 w4 w5 w6 w7
 T = u1 u2 u3 u4 u5 u6 u7 u8 u9
w4 -> u3 u5
 fertility of w4 = 2
distortion w5 -> u9

24.  How to compute probability of an alignment?
 Need to estimate
 Fertility probabilities
 P(fertility=n|w) = probability word w has fertility n
 Distortion probabilities
 P(i|j,l) = probability target word is at position i given source word at position
j and l is the length of the target
 Example
 (Le chien est battu par Jean | John(6) does beat(3,4) the(1) dog(2))
 P(f=1|John)P(Jean|John) x
 P(f=0|does) x
 P(f=2|beat)P(est|beat)P(battu|beat) x
 P(f=1|the)P(Le|the) x
 P(f=1|dog)P(chien|dog) x
 P(f=1|<null>)P(par|<null>) x distortion probabilities…

25.  Not done yet
 Given T
 translation problem is to find S that maximizes P(S)P(T|S)
 can’t look for all possible S in the language
 Idea (Search):
 construct best S incrementally
 start with a highly likely word transfer
 and find a valid alignment
 extending candidate S at each step
 (Jean aime Marie | * )
 (Jean aime Marie | John(1) * )
 Failure?
 best S not a good translation
 language model failed or
 translation model failed
 couldn’t find best S
 search failure

26. • Parameter Estimation
– English/French
• from the Hansard corpus
– 100 million words
– bilingual Canadian parliamentary proceedings
– unaligned corpus
– Language Model
• P(S) from bigram model
– Translation Model
• how to estimate this with an unaligned corpus?
• Used EM (Estimation and Maximization) algorithm, an iterative algorithm for re-estimating probabilities
• Need
– P(u|w) for words u in T and w in S
– P(n|w) for fertility n and w in S
– P(i|j,l) for target position i and source position j and target length l

27.  Traditional MT techniques require large amounts of linguistic knowledge to be encoded
as rules. Statistical MT provides a way of automatically finding correlations between
the features of two languages from a parallel corpus, overcoming to some extent the
knowledge bottleneck in MT.
 A major drawback with the statistical model is that it presupposes the existence of a
sentence-aligned parallel corpus. For the translation model to work well, the corpus has
to be large enough that the model can derive reliable probabilities from it, and
representative enough of the domain or sub-domain (weather forecasts, match reports,
etc.) it is intended to work for.
 Statistical MT techniques have not so far been widely explored for Indian languages. It
would be interesting to find out to what extent these models can contribute to the huge
ongoing MT efforts in the country.

28. [1] Peter F. Brown, John Cocke, Stephen A. Della Pietra, Vincent J. Della Pietra, Frederick Jelinek,
John D. Lafferty, Robert L. Mercer, and Paul S. Roossin, A Statistical Approach to Machine
Translation, Computational Linguistics, 16(2), pages 79–85, June 1990.
[2] Weaver, W. 1955 Translation (1949). In: Machine Translation of Languages, MIT Press,
Cambridge, MA.
[3] Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer, The
Mathematics of Statistical Machine Translation: Parameter Estimation, Computational Linguistics,
19(2), pages 263–311, June 1993.
[4] Masahiko Haruno and Takefumi Yamazaki, High- Performance Bilingual Text Alignment using
Statistical and Dictionary Information, Proceedings of the 34th Conference of the Association for
Computational Linguistics, pages 131–138, 1996.
[5] John Hutchins and Harold L. Somers, An Introduction to Machine Translation, Academic Press,
1992.
[6] Lopez, A. 2008. Statistical machine translation. ACM Comput. Surv., 40, 3, Article 8 (August 2008).
[7] W. A. Gale and K. W. Church, A Program for Aligning Sentences in Bilingual Corpora,
Computational Linguistics, 19(1), pages 75–102,1993.