This document discusses a hybrid machine translation method developed by Matīss Rikters at the University of Latvia, focused on combining outputs from multiple online translation APIs for English-Latvian translations. The study explores various machine translation systems, presents experiments evaluating translation quality, and reports on results from human evaluations of machine-generated outputs. Conclusions include the need for improvements in translation selection and future work possibilities such as incorporating larger language models and better quality estimation methods.

1. Multi-system machine translation
using online APIs for English-Latvian
Matīss Rikters
University of Latvia
ACL 2015 Fourth Workshop on
Hybrid Approaches to Translation
Beijing, 31.07.2015

2. Introduction
 Motivation:
 Doctoral studies at the University of Latvia
 A hybrid machine translation method, combining results of various machine translation systems
 Literature review
 Recent trends in Multi-System Machine Translation
 Nothing similar publically available was found

3. Introduction
 Goals:
 Combine output from multiple online MT APIs
 Keep it simple
 Make it work fast

4. Related work
 "Coupling Statistical Machine Translation with Rule-based Transfer and Generation",
A. Ahsan, and P. Kolachina.
 "Using language and translation models to select the best among outputs from
multiple MT systems", Y. Akiba, T. Watanabe, and E. Sumita.
 "MANY: Open source machine translation system combination", L. Barrault.
 "A program for automatically selecting the best output from multiple machine
translation engines", C. Callison-Burch and R. S. Flournoy.

5. Initial plan
 Use systems that support English – Latvian translation
 Found five such systems:

6. What worked
 Couldn`t get APIs of two of them to work
 Used the remaining three:

7. System description
Sentence tokenization
Translation with APIs
Google Translate Bing Translator LetsMT
Selection of the best
translation
Output

8. Selection of the best translation
Probabilities are calculated based on the observed entry with longest matching history 𝑤𝑓
𝑛
:
𝑝 𝑤 𝑛 𝑤1
𝑛−1
= 𝑝 𝑤 𝑛 𝑤𝑓
𝑛−1
𝑖=1
𝑓−1
𝑏(𝑤𝑖
𝑛−1
)
where the probability 𝑝 𝑤 𝑛 𝑤𝑓
𝑛−1
and backoff penalties 𝑏(𝑤𝑖
𝑛−1
) are given by an already-
estimated language model. Perplexity is then calculated using this probability:
where given an unknown probability distribution p and a proposed probability model q, it
is evaluated by determining how well it predicts a separate test sample x1, x2... xN drawn
from p.

9. System usage
 Get the code - https://github.com/M4t1ss/Multi-System-Hybrid-Translator
 Get API access
 Google - https://cloud.google.com/translate/
 Bing - http://www.bing.com/dev/en-us/translator
 LetsMT - https://www.letsmt.eu/Integration.aspx
 Add API keys to the configuration
 Prepare a language model
 You can use KenLM – https://kheafield.com/code/kenlm/
 Prepare input data
 Run
 php MSHT.php languageModel.binary inputSentances.txt

10. Experiments
 MT System APIs
 Google Translate
 Bing Translator
 TB2013 EN-LV v03 from LetsMT
 Language model
 JRC Acquis corpus version 2.2
 Input sentences
 JRC Acquis corpus version 2.2
 ACCURAT balanced test corpus for under resourced languages

11. Experiment results – JRC Acquis
System BLEU TER WER
Translations selected
Google Bing LetsMT Equal
Google Translate 16.92 47.68 58.55 100 % - - -
Bing Translator 17.16 49.66 58.40 - 100 % - -
LetsMT 28.27 36.19 42.89 - - 100 % -
Hybrid Google + Bing 17.28 48.30 58.15 50.09 % 45.03 % - 4.88 %
Hybrid Google + LetsMT 22.89 41.38 50.31 46.17 % - 48.39 % 5.44 %
Hybrid LetsMT + Bing 22.83 42.92 50.62 - 45.35 % 49.84 % 4.81 %
Hybrid Google + Bing + LetsMT 21.08 44.12 52.99 28.93 % 34.31 % 33.98 % 2.78 %

12. Experiment results – ACCURAT balanced
System BLEU
Google Translate 24.73
Bing Translator 22.07
LetsMT 32.01
Hybrid Google + Bing 23.75
Hybrid Google + LetsMT 28.94
Hybrid LetsMT + Bing 27.44
Hybrid Google + Bing + LetsMT 26.74

13. Human evaluation
 5 native Latvian speakers were given a random 2% - 32 sentences
 They were told to mark which of the three MT outputs is the best, worst and OK
 Having the option to select multiple answers for best, worst or OK

14. Human results
System User 1 User 2 User 3 User 4 User 5 AVG user Hybrid BLEU
Bing 21,88% 53,13% 28,13% 25,00% 31,25% 31,88% 28,93% 16.92
Google 28,13% 25,00% 25,00% 28,13% 46,88% 30,63% 34,31% 17.16
LetsMT 50,00% 21,88% 46,88% 46,88% 21,88% 37,50% 33,98% 28.27

15. Conclusion
 Simple to
 Build
 Use
 Add new MT APIs
 Works
 When used on similar systems
 Poor with one much superior system
 Needs
 Improvements for translation selection
 More configuration options

16. Future work
 Use a bigger & better language model?
 Tried it… about the same results
 Confusion networks?
 Too confusing for now
 Use MT quality estimation for selecting the best candidates
 QuEst or QuEst++
 Other quality estimation
 Chunk sentences in smaller parts, translate & recombine

17. Thank you!
http://ej.uz/MSHT-GITHUB
http://ej.uz/MSMT-EN-LV