MT and Translator's Tools

Outline
Tools for Translators
Free Language Data
Machine Translation

Machine Translation and Translation Technology

Jimmy O’Regan

The Apertium Project

OSS Bar Camp, 19 September 2009

Jimmy O’Regan Machine Translation and Translation Technology

Outline
Free Language Data
Machine Translation

1 Free Language Data


Outline
Tools for Translators Terminology
Free Language Data Localisation vs. Translation
Machine Translation



Outline
Machine Translation

Some Terminology

Internationalisation
Giving software the capability to display text in another
language


Outline
Machine Translation

Some Terminology

language
In Open Source, this generally means adding support for
gettext.


Outline
Machine Translation

Some Terminology

language
gettext.
Localisation


Outline
Machine Translation

Some Terminology

language
gettext.
Localisation
Customising the messages displayed to the user to appear in
the manner most appropriate for them.


Outline
Machine Translation

Some Terminology

language
gettext.
Localisation
In their language, or their dialect.


Outline
Machine Translation

Some Terminology

language
gettext.
Localisation

Translation


Outline
Machine Translation

Some Terminology

language
gettext.
Localisation

Translation
Converting text from one language to another


Outline
Machine Translation

Localisation vs. Translation

Localisation and translation are sometimes, but not always, the
same.


Outline
Machine Translation


same.
Documents may need to be localised, but not translated:


Outline
Machine Translation


same.
Documents may need to be localised, but not translated:
A British company with an Irish office still needs to localise their
documents: any reference to “our London office” will need to be
changed to “our Dublin office”.


Outline
Machine Translation


Localised translations can also have additional requirements:


Outline
Machine Translation


gettext allows numbers to be specially treated: ‘‘%d
file(s)’’ ugliness is not necessary.


Outline
Machine Translation


English and Spanish need two forms of words for number


Outline
Machine Translation


English and Spanish need two forms of words for number: single
and plural


Outline
Machine Translation


and plural
Polish needs three


Outline
Machine Translation


and plural
Polish needs three: single, plural, and quantity (greater than 5)


Outline
Machine Translation


and plural
Slovenian needs four


Outline
Machine Translation


and plural
Slovenian needs four: single, dual, plural, and quantity.


Outline
Machine Translation

Localisation

Software localisation is a huge business area for proprietary
software.


Outline
Machine Translation

Localisation

software.
One that traditionally lags behind Open Source.


Outline
Machine Translation

Localisation

software.
That advantage is usually due to the eﬀorts of a handful of
dedicated volunteers for the majority of languages.


Outline
Machine Translation

Localisation

software.
That advantage is usually due to the eﬀorts of a handful of
dedicated volunteers for the majority of languages.
But they’re catching up: Facebook is using Open Source-like
eﬀorts for their translations.


Outline
Machine Translation

Localisation Tools

Unsurprisingly then, localisation is very well supported by Open
Source software:
Pootle (http://translate.sourceforge.net/wiki/pootle/index -
Web-based)
Virtaal (http://translate.sourceforge.net/wiki/virtaal/index -
cross platform)
poEdit (http://www.poedit.net/ - cross platform)
Lokalize (http://userbase.kde.org/Lokalize - KDE)
GTranslator (http://gtranslator.sourceforge.net/ - GNOME)


Outline
Machine Translation

Translation Tools

Unfortunately, there’s only one real equivalent tool for general
translation: OmegaT


Outline
Machine Translation

Common Features

All of these tools include these features:


Outline
Machine Translation

Common Features

Translation Memory


Outline
Machine Translation

Common Features

Translation Memory
Automatically reuse previous translations


Outline
Machine Translation

Common Features

Translation Memory
Fuzzy matching


Outline
Machine Translation

Common Features

Translation Memory
Fuzzy matching
Suggest translations similar to previously translated sentences


Outline
Machine Translation

Common Features

Translation Memory
Fuzzy matching
Terminology Management


Outline
Machine Translation

Common Features

Translation Memory
Fuzzy matching
Terminology Management
Give suggestions from a per-project dictionary


Outline
Machine Translation

Translate Toolkit

http://translate.sourceforge.net/
A set of common tools for translation/localisation:
Translation Memory server
Format conversion
Terminology management
Quality control tools
All brought to you by the wonderful people of translate.org.za


Outline
Free Language Data
Machine Translation

Free Language Software Needs Free Data

Just like “Free Software Needs Free Documentation”, so too does
Free Language Software need Free Data.


Outline
Free Language Data
Machine Translation


Usually, this means we have to make it ourselves.


Outline
Free Language Data
Machine Translation


Usually, this means we have to make it ourselves.
Unfortunately, the community of developers of free language
software, and thus free language data, is quite small.


Outline
Free Language Data
Machine Translation

The importance of spell checkers

Spell checking data packages are the absolute bare minimum of
support for a language with technology.
Usually, the people who develop them tend to be involved in other
areas of Free language software:


Outline
Free Language Data
Machine Translation


Kevin Scannell


Outline
Free Language Data
Machine Translation


Kevin Scannell
Makes the Irish spell checking data.


Outline
Free Language Data
Machine Translation


Kevin Scannell
Makes the Irish spell checking data. And An Gramad´ir, an Irish
o
language grammar checker.


Outline
Free Language Data
Machine Translation


Kevin Scannell
o
language grammar checker. And created a WordNet/thesaurus for
Irish.


Outline
Free Language Data
Machine Translation


Kevin Scannell
o
Irish. And contributed the language data for Apertium’s
Irish–Scots Gaelic translator. etc.


Outline
Free Language Data
Machine Translation


Kevin Scannell
o
Marcin Milkowski


Outline
Free Language Data
Machine Translation


Kevin Scannell
o
Marcin Milkowski
Heavily involved in the Polish spell checking data


Outline
Free Language Data
Machine Translation


Kevin Scannell
o
Marcin Milkowski
Heavily involved in the Polish spell checking data And
LanguageTool, a multilingual grammar checker that’s integrated
with Open Oﬃce.


Outline
Free Language Data
Machine Translation


Kevin Scannell
o
Marcin Milkowski
Heavily involved in the Polish spell checking data And
LanguageTool, a multilingual grammar checker that’s integrated
with Open Oﬃce. And maintains the open Polish thesaurus.


Outline
Free Language Data
Machine Translation


translate.org.za
Make the spelling checkers for several South African languages, as
well as many tools for translators already mentioned–Virtaal,
Translate Toolkit, Pootle. Much of Apertium’s English–Africans
translator was made directly by translate.org.za developers, as well
as Apertium’s dbus interface, and a GUI. (Virtaal allows translators
to use machine translations as a basis for their work).


Outline
Free Language Data
Machine Translation


Spell checkers are set to become even more important. Hunspell,
which is fast becoming the standard spell checker in Open Source
projects, now includes morphological analysis and generation. This
will greatly improve, among other things, terminology management
in translator’s tools.
At the moment, if you have “dog” in your terminology list, the
translation tool will see that and only that: “dogs” will go
unrecognised. With morphological analysis, the tool can know that
“dogs” is not only related to “dog”, but is the plural of a noun:
another assistance to the translator.


Outline Is Machine Translation a Translator’s Tool?
Tools for Translators N-Grams
Free Language Data Moses
Machine Translation Apertium

Machine Translation

Machine translation has a bad reputation.



Mechanical Translation

Mechanical translation, of any form, does tend, inevitably, to have
mistakes.




mistakes.
For centuries, paintings of Moses portrayed him as having horns.




mistakes.
A translator of the Latin Vulgate added the wrong vowel: he
thought that Moses had horns, not that his face was glowing.




mistakes.
And people were killed for wishing to correct that, and other
mistakes.




mistakes.
And people were killed for wishing to correct that, and other
mistakes.
Proofread translations, always.



Types of Machine Translation

Dictionary lookup




Dictionary lookup – the most basic form of MT




Translation Memory




Translation Memory – also, a basic form of MT




Example Based Machine Translation




Example Based Machine Translation – considered the most
accurate form of MT, but there are few if any examples “in
the wild”.




the wild”.
Statistical Machine Translation




the wild”.
Statistical Machine Translation – currently the darling of
research and the basis of Google Translate.




the wild”.
research and the basis of Google Translate. Solves a lot of old
problems, but introduces new one.




the wild”.
problems, but introduces new one. And breaks a lot of things
that “used to work”.




the wild”.
Rule Based Machine Translation




the wild”.
Rule Based Machine Translation – The oldest kind of MT,
dating back to the 1950s.




the wild”.
dating back to the 1950s. – The kind I work with, so
obviously it’s the best




the wild”.
dating back to the 1950s. – The kind I work with, so
obviously it’s the best!!!



Is Machine Translation a Translator’s Tool?

Yes.




Yes.
That might be hard to accept.




Yes.
That might be hard to accept. Particularly if you only speak
English.




Yes.
That might be hard to accept. Particularly if you only speak
English. But for closely-related, similar languages, machine
translation can be as eﬀective and accurate as a spelling checker.



Uses of Machine Translation




1 Assimilation




1 Assimilation Understanding a text




2 Dissemination




2 Dissemination Preparing a text for translation.




2 Dissemination Preparing a text for translation. That is; for
preparing a rough draft for a translator. Who then edits the
text.



“Phrase-Based” SMT

Most current research (and commercial use) of Statistical MT uses
“phrase-based” SMT.




The problem is it’s not phrase-based.




The problem is it’s not phrase-based.
It’s N-Gram based.



N-Grams

An n-gram is a collection of “n” amounts of tokens



N-Grams

For text, these are usually (not always!) words



N-Grams

For text, these are usually (not always!) words
...punctuation is counted as a “word”.



Unigrams

“This is John’s dog.”



Unigrams

Example
This
is
John
’s
dog
.



Bigrams




Bigrams

Example
This is
is John
John ’s
’s dog
dog .



Trigrams



Trigrams
Example
This is John
is John ’s
John ’s dog
’s dog .



N-Gram Language Models

An n-gram language model is a collection of n-grams




An n-gram language model is a collection of n-grams for n..1: a
trigram model includes trigrams, bigrams, and unigrams.




Each n-gram is listed along with its frequency




(According to a particular corpus)




(According to a particular corpus)
But, most importantly...




N-grams overlap.




When a sequence of words is queried against a language model,
the language model software computes the combined likelihood of
1..n combinations in that sequence.




Possibly the ﬁrst use of n-gram language models was in Automatic
Speech Recognition.




For basic uses of ASR, such as call centres, a custom grammar is
used.




For basic uses of ASR, such as call centres, a custom grammar is
used.
In a mobile phone, such a grammar could look like this:
Example
CALLWORD : phone call dial
ZEROWORD : zero oh
NUMBER : one two three four five six seven eight nine
ZEROWORD
NUMBERS : NUMBER* NUMBER
COMMAND : CALLWORD NUMBERS




However, for continuous dictation, building such grammars is an
almost inﬁnite task.




However, for continuous dictation, building such grammars is an
almost infinite task.
Instead of defining long, complicated grammars that define, for
example, when the sound /mi:t/ represents “meet” and when it
represents “meat”, n-gram language models allow the correct
sound to be chosen based on the context of the surrounding words.




This was an obvious progression for ASR, which uses statistical
modelling to choose in context which sound is most likely, based
on the surrounding sounds.




Now, language models are being used in all areas of language
technology.
The problem is: useful language models are huge, and can be
computationally costly to use




technology.
computationally costly to use unless you have a data centre.




technology.
computationally costly to use unless you have a data centre.
Google, for example, use language models for everything :
Spell checking (Search, Google Wave, GMail)
Search queries (“Did you mean?”)
Machine translation



N-Gram-Based SMT

Basic Statistical MT uses a probabilistic dictionary: each word pair
has a probability assigned.



N-Gram-Based SMT

The interesting part is how they get those dictionaries.



N-Gram-Based SMT

The interesting part is how they get those dictionaries.
A program, usually GIZA++ (Open Source), reads two pairs of
text: the source language, and the target language.



N-Gram-Based SMT

For each word in each sentence of the source language, the
probable translation is considered to be every word in the target; as
more words are seen, the translations are re-evaluated: the next
time word 1 is used, perhaps “possible translation 1” is present,
but “possible translation 2” is absent from the sentence: the
probability of the former is increased; the latter, decreased.
And so on, over the course of the text, the probabilities of each
word are re-evaluated; then the whole text is processed again, and
again, until a reasonable level of probability remains.
The result is a probabilistic dictionary.



N-Gram-Based SMT

N-grams come into the picture in two ways:



N-Gram-Based SMT

1 Evaluating multiple probable translations



N-Gram-Based SMT

Similarly to Speech Recognition, each choice is evaluated
against a language model



N-Gram-Based SMT

2 N-grams as “words”



N-Gram-Based SMT

As well as considering individual words, each n-gram is
considered as a possible “phrase”, and treated as an individual
word.



N-Gram-Based SMT

As well as considering individual words, each n-gram is
considered as a possible “phrase”, and treated as an individual
word. This helps to cut down on ambiguous terms:
“basketball coach” vs. “coach driver”.



Moses

Moses is an Open Source SMT system. Moses has a distinct
advantage over several other SMT systems:
1 It’s Open Source



Moses

Actively developed, and supported by a large community



Moses

2 Factored Models



Moses

2 Factored Models
Moses is able to make use of linguistic information.



Moses

2 Factored Models
3 Open Data



Moses

2 Factored Models
3 Open Data
The Moses developers also recognise the importance of Free
Linguistic Data, and have provided the EuroParl corpus so
that others may build a statistical MT system using it. Also,
the JRC Acquis – the corpus of EU legal text (and most of
the data behind Google Translate’s support for most oﬃcial
EU languages) have prepared their corpus for use with Moses.



Apertium
Apertium is an Open Source Machine Translation platform.



Apertium
Rule Based



Apertium
Rule Based
Statistical disambiguation



Apertium
Rule Based
Follows the UNIX philosophy



Apertium
Rule Based
Follows the UNIX philosophy:
The system is a pipeline



Apertium
Rule Based
Each piece “does one thing, and does it well”



Apertium
Rule Based
(Not quite: analysis and generation of words are performed by
separate modes of the same program)



Apertium
Rule Based
Each component can be easily replaced



Apertium
Rule Based
The apertium program itself is just a shell script that calls
the correct pipeline.
Several statistics-based tools for building data



Apertium
Rule Based
Several statistics-based tools for building data: dictionaries



Apertium
Rule Based
Several statistics-based tools for building data: dictionaries,
rules
Doesn’t run on Windows


Apertium
Rule Based
rules
Doesn’t run on Windows yet.


Apertium
Rule Based
rules
Doesn’t run on Windows yet. Shame on you for cheering!


Apertium
Rule Based
rules
Doesn’t run on Windows yet. Shame on you for cheering! ;)


Some Errors

(Spanish – English)



Some Errors

Fondo Monetario Internacional



Some Errors

International Monetary bottom



Some Errors

(Catalan – English)



Some Errors

Fidel Castro



Some Errors

Fidel Castro
Faithful Castrate



Apertium

Apertium was born from a set of translators developed in
Universitat d’Alacant, as part of the OpenTrad project.



Apertium

Universitat d’Alacant, as part of the OpenTrad project. Originally
designed to translate between the Romance languages of Spain, it
has been expanded over time to support more distant languages:



Apertium

First English–Catalan



Apertium

First English–Catalan More recently, Basque to Spanish



Community

As part of the OpenTrad project, Apertium had a community of
developers, but limited to university and business developments.
Thanks mostly to Francis Tyers, Apertium has in recent years
begun to also acquire a community of volunteer contributors.



Community

The ﬁrst release from the volunteer community was our Welsh to
English translator (mostly designed by Kevin Donnelly – who also
maintains the Welsh spell checking data).



Community

The ﬁrst release from the volunteer community was our Welsh to
English translator (mostly designed by Kevin Donnelly – who also
maintains the Welsh spell checking data).
This summer, we took part in Google’s Summer of Code
programme, with 8 successful students. One of the translators
developed during GSoC, for Norwegian Bokm˚l–Nynorsk, has
a
(within a month of release) been used to translate 30 articles on
the Nynorsk Wikipedia.



Language Pairs Supported

Spanish – Catalan Spanish – Romanian
French – Catalan Occitan – Catalan
English – Galician Occitan – Spanish
Spanish – Portuguese English – Catalan
English – Spanish English – Esperanto
Spanish – Galician French – Spanish
Esperanto – Spanish Welsh – English
Breton – French Esperanto – Catalan
Portuguese – Catalan Portuguese – Galician
Basque – Spanish Nynorsk – Bokm˚la


MT and Translator's Tools

More Related Content

What's hot

Similar to MT and Translator's Tools

More from Jim O'Regan

Recently uploaded

MT and Translator's Tools