Internal Evaluation for a MT System, German to English

Internal Evaluation German to English translation [email_address] , [email_address] Nervo Verdezoto D.

Outline
The Problem
Objectives
Literature Review
Baseline
Experiments
Results
Summary

The Problem
Word order differences
En glish is SVO language while German is SOV language.
Long distance distortions are observed frequently.

Outline
The Problem
Objectives
Literature Review
Software & Hardware
Baseline
Experiments
Results
Summary

Objetives
Improve the baseline
Get familiar with moses toolkit
Get familiar with scientific reporting

Outline
The Problem
Objectives
Literature Review
Software & Hardware
Baseline
Experiments
Results
Summary

Literature Review
Pre-processing.
Alternative word reordering and alignment mechanisms as optional phrase table modifications, using POS-based reordering model [1] [2] [4] [7].
Compound splitting, compound merging and part-of-speech/morphological sequence models [3] [4] [6].
Augmenting the training corpus with an extracted dictionary [3] [5].
Factored phrase-based statistical machine translation system, with different levels of linguistic information (e.g. lemmas, POS, etc.) [5] [8].
Post-processing.
Out-of-vocabulary words (e.g. proper names which do not get capitalized properly or numbers with different format in German and English) [3].
Others

Road Map
The Problem
Objectives
Literature Review
Software & Hardware
Baseline
Experiments
Results
Summary

Software & Hardware
Software
Moses, an open source MT toolkit.
The IRSTLM language modeling toolkit.
Giza ++, to perform word alignments.
C&C Language Processing Tools, for the CCG supertagger.
Word_error.pl script, (provided by CMU).
Hardware.
Intel(R) Core(TM) 2 CPU T7200 @ 2.00 GHz with 3GB of RAM and 32 bit Linux (ubuntu).
see MOSES: http://www.statmt.org/moses/
see SUPERTAGGER: http://svn.ask.it.usyd.edu.au/trac/candc/wiki
see WER: CMU: http://www.cs.cmu.edu/~roni/11761-s07/assignments/assignment6/word_error.pl

Outline
The Problem
Objectives
Software & Hardware
Baseline
Experiments
Results
Summary

Baseline Table 1. Statistics of the Dataset de: wiederaufnahme der sitzungsperiode en: resumption of the session de: ich erkläre die am freitag , dem 17. dezember unterbrochene sitzungsperiode des europäischen parlaments für wiederaufgenommen , wünsche ihnen nochmals en: i declare resumed the session of the european parliament adjourned on friday 17 december 1999 , and i would like once again to wish you a happy new year in de: alles gute zum jahreswechsel und hoffe , daß sie schöne ferien hatten . en: the hope that you enjoyed a pleasant festive period . Figure 1. Sample of the training corpus German < >English Training Sentences 78524 Words 1581042 1684639 Dev Sentences 2000 Words 55118 58761 Test Sentences 2000 Words 55580 59153

Baseline - Results
Evaluation Measures
BLUE
NIST
WER
PER
Tunning: MERT (1000 sentences)
Table 2. Performance of initial models Measure Model 0 -Baseline Model 1 -Tunning BLUE 23.24% 23.62% NIST 6.5426 6.4539 WER 69.09% 70.90% PER 18.82% 16.43%

Outline
The Problem
Objectives
Baseline
Sofware & Hardware
Experiments
Results
Summary

Experiments Setup Description Setup 1, 2 Filter sentences Baseline - 40 Setup 1 – 45 Setup 2 -35 Setup 3,4 and 5 Combination with baseline, setup1 and setup2 and lexicalized reordering model (reordering configuration msd-bidirectional-fe and distorsion limit 6 ). Setup 3 – filter(40) Setup 4 –filter (45) Setup 5 –filter (35) Setup 6 I tried to split source data but it does not work Setup 7 and 8 Adding Part Of Speech information using Factored translation mode in the target data (English) / LM: Setup 7 (3gram), Setup 8(5gram) Setup 9 I tried to used Moses for factored translation model in the source (German) but it does not work. I tried to train the suppertagger with a German corpus (TIGGER corpus) and I got a problem with the format of the files see http://www.ims.uni-stuttgart.de/projekte/TIGER/TIGERCorpus/annotation

Factored Translation model
Alexandra Birch Miles Osborne Philipp Koehn. CCG Supertags in Factored Statistical Machine TranslationProceedings of the Second Workshop on Statistical Machine Translation, pages 9–16,Prague, June 2007. c2007 Association for Computational Linguistics

Training
../moses/scripts/training/train-factored-phrase-model.perl -bin-dir /home/nervo/MOSESMT/bin/ -scripts-root-dir /home/nervo/MOSESMT/moses/scripts -root-dir . --corpus trainingcorpus/baseline.de-en.tok.clean.lowercased -f de -e en -first-step 1 -last-step 9 -max-phrase-length 5 -reordering msd-bidirectional-fe -lm 0:3 :/home/nervo/MOSESMT/SETUP7/lm/baseline.irstlm.bin:0 -lm 2:3 :/home/nervo/MOSESMT/SETUP7/lm/baseline.pos.irstlm.bin:0 >& logsetup7.train
../moses/scripts/training/train-factored-phrase-model.perl -bin-dir /home/nervo/MOSESMT/bin/ -scripts-root-dir /home/nervo/MOSESMT/moses/scripts -root-dir . -corpus trainingcorpus/baseline.de-en.tok.clean.lowercased -f de -e en -first-step 1 -last-step 9 -reordering msd-bidirectional-fe -lm 0:5 :/home/nervo/MOSESMT/SETUP8/lm/baseline.irstlm.bin:0 -lm 2:5 :/home/nervo/MOSESMT/SETUP8/lm/baseline.pos.irstlm.bin:0 >& logsetup8.train

Changes in moses.ini
# translation tables: source-factors, target-factors, number of scores, file
[ttable-file]
0 0,1,2 5 /home/nervo/MOSESMT/SETUP8/model/phrase-table.gz
# language models: type(srilm/irstlm), factors, order, file
[lmodel-file]
1 0 5 /home/nervo/MOSESMT/SETUP8/lm/baseline.irstlm.bin
1 2 5 /home/nervo/MOSESMT/SETUP8/lm/baseline.pos.irstlm.bin

SETUP7

Outline
The Problem
Objectives
Baseline
Sofware & Hardware
Experiments
Results
Summary

Experimental results Measure Baseline 40 SETUP 1 - 45 SETUP 2 35 BLUE 23.24% 23.20% 22.88% NIST 6.5426 6.5490 6.4742 WER 69.09% 69.08% 69.58% PER 18.82% 18.80% 18.84% Measure SETUP 3 40 SETUP 4 45 SETUP 5 35 BLUE 23.03% 23.06% 22.56% NIST 6.5168 6.5349 6.4485 WER 68.52% 68.29% 69.07% PER 19.46% 19.72% 19.46% Measure SETUP 7 40 - 3gram SETUP 8 40-5gram SETUP9 BLUE 21.51% 21.59% / NIST 6.1699 6.1754 / WER 73.29% 73.45% / PER 17.74% 17.39% /

MODEL EXAMPLE REFERENCE he wanted the presidency to outline the way forward at nice . BASELINE he HAS EXPRESSED THE WISH THAT the presidency IN NICE the way *** AHEAD AUFZEIGT TUNING he HAS EXPRESSED THE WISH THAT the presidency IN NICE the way *** AHEAD AUFZEIGT SETUP1 he HAS EXPRESSED THE WISH THAT the presidency IN NICE the way *** *** AHEAD SETUP2 he HAS EXPRESSED the WISH THAT THE PRESIDENCY IN NICE way AUFZEIGT THE FUTURE SETUP3 he HAS EXPRESSED THE WISH THAT the presidency IN NICE , the way *** AHEAD AUFZEIGT SETUP4 he HAS EXPRESSED THE WISH THAT the presidency IN NICE the way *** *** AHEAD SETUP5 he HAS EXPRESSED THE WISH THAT the presidency IN NICE , THE FUTURE PATH AUFZEIGT SETUP6 -- SETUP7 he HAS EXPRESSED THE WISH THAT the presidency IN NICE , THE FUTURE PATH SHOWS SETUP8 he HAS EXPRESSED THE WISH THAT the presidency IN NICE , the way *** AHEAD SHOWS

Outline
The Problem
Objectives
Baseline
Sofware & Hardware
Experiments
Results
Summary

Summary and Conclusion
The BLUE score after tuning is always higher than the rest of scores of the setups.
Manual analysis can hell to understand the role of postagging in the target data
The future direction in this way would be tried to develop another strategy in order to choose the correct parameters and I need to play more with the factored sequence model in order to find a good way to select and set these parameters taking into account the advantages of these systems.

REFERENCES
Niehues, Jan., Kolss, M. A POS-Based Model for Long-Range Reordering in SMT. Proceedings of the Fourth Workshop on Statistical Machine Translation , pages 206–214, Athens, Greece, 30 March – 31 March 2009. Association for Computational Linguistics
Niehues, Jan., Herrmann, T., Kolss, M. The Universitat Karlsruhe Translation System for the EACL-WMT 2009. Proceedings of the Fourth Workshop on Statistical Machine Translation, pages 80–84, Athens, Greece, 30 March – 31 March 2009. Association for Computational Linguistics
Holmqvist, M., Stymne, S., Foo, J. and Ahrenberg, L. The Improving alignment for SMT by reordering and augmenting the training corpus . Proceedings of the Fourth Workshop on Statistical Machine Translation , pages 120–124, Athens, Greece, 30 March – 31 March 2009. Association for Computational Linguistics
Popovic, M., Vilar, D., Stein, D., Matusov E., Ney, H. The RWTH Machine Translation System for WMT 2009 . Proceedings of the Fourth Workshop on Statistical Machine Translation , pages 66–69, Athens, Greece, 30 March – 31 March 2009. Association for Computational Linguistics
Stymme, S. A Comparison of Merging Strategies for Translation of German Compounds. Proceedings of the EACL 2009 Student Research Workshop , pages 61–69, Athens, Greece, 2 April 2009. Association for Computational Linguistics
Dyer, C. Using a maximum entropy model to build segmentation lattices for MT. Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the ACL , pages 406–414, Boulder, Colorado, June 2009. c2009 Association for Computational Linguistics
Fraser, A. Experiments in morphosyntactic processing for translating to and from German. Proceedings of the Fourth Workshop on Statistical Machine Translation , pages 115–119, Athens, Greece, 30 March – 31 March 2009. c2009 Association for Computational Linguistics
Holmqvist, M., Stymne, S., Ahrenberg, L. Getting to know Moses: Initial experiments on German–English factored translation. Proceedings of the Second Workshop on Statistical Machine Translation , pages 181–184, Prague, June 2007. Association for Computational Linguistics
Birch, A., Osborne, M., Koehn, P. CCG Supertags in Factored Statistical Machine Translation. Proceedings of the Second Workshop on Statistical Machine Translation , pages 9–16, Prague, June 2007. Association for Computational Linguistics.
K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the ACL , pages 311–318.
Federico, M., Bertoldi, N. A Word-to-Phrase Statistical Translation Model. ACM Transactions on Speech and Language Processing, Vol. 2, No. 2, December 2005, Pages 1–24.
Abdul-Rauf, S., Schwenk, H. Exploiting Comparable Corpora with TER and TERp . Proceedings of the 2nd Workshop on Building and Using Comparable Corpora, ACL-IJCNLP 2009 , pages 46–54, Suntec, Singapore, 6 August 2009. ACL and AFNLP
Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A. and Herbst, E. 2007. Moses: Open source toolkit for statistical machine translation . In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics (ACL ’07) – Companion Volume, June
Stolcke, A. 2002. SRILM – an extensible language modeling toolkit. In Proceedings of the Intl. Conf. on Spoken Language Processing .
Och, F. and Ney, H. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51, March.
Och, F. 2003. Minimum error rate training for statistical machine translation. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics (ACL-2003), Sapporo, Japan.
Clark, S. and Curran, J. 2004. Parsing the wsj using ccg and log-linear models . In Proceedings of the Association for Computational Linguistics , pages 103–110, Barcelona, Spain.
Clark, S. 2002. Supertagging for combinatory categorical grammar . In Proceedings of the International Workshop on Tree Adjoining Grammars , pages 19–24, Venice, Italy.
Callison-Burch, C., Osborne, M. and Koehn, P. 2006. Re-evaluating the role of Bleu in machine translation research. In Proceedings of the European Chapter of the Association for Computational Linguistics , Trento, Italy

THANK YOU
Nervo Verdezoto D. [email_address] , [email_address]

Internal Evaluation for a MT System, German to English

by Nervo Verdezoto

Accessibility

Categories

Upload Details

Usage Rights

2 Embeds 36

Statistics

Internal Evaluation for a MT System, German to English Presentation Transcript

https://jujo00obo2o234ungd3t8qjfcjrs3o6k-a-sites-opensocial.googleusercontent.com	20
http://jujo00obo2o234ungd3t8qjfcjrs3o6k-a-sites-opensocial.googleusercontent.com	16