The document summarizes the results of experiments comparing large pre-trained language models for machine translation. In a machine translation challenge, a smaller Marian model demonstrated better or similar results to much larger pretrained models, contradicting expectations. This suggests that very large models do not necessarily improve translation quality and that current automatic evaluation metrics are limited. Human evaluation remains important for fully assessing machine translation quality.

1. EXAMINING LARGE
PRE-TRAINED
LANGUAGE MODELS
FOR MACHINE
TRANSLATION:
WHAT YOU DON’T
KNOW ABOUT IT
BIOMEDICAL CLINSPEN
WMT22 CHALLENGE RESULTS
2022
lifeng.han@manchester.ac.uk
serge.gladkoff@logrusglobal.com

2. Rationale
• Magical technology to repro and generate new translations
• BUT error rate is far from 0%.

3. ThefunctionofMTquality
The quality of MT output depends on the
model, language pair, quality of training
data, type of input content and other
smaller things.
3

4. TheFact: Theerrorrateisneverzero.
4

5. Maybeextra-
largemodels?
Recently, extra large MT models were
increasingly coming out, with ever
increasing number of parameters and
multilingual capabilities
5

6. WMT21andNLLB
Two most recent extra large language models
WMT21
• 4.7 billion parameters
https://huggingface.co/facebook/wmt21-dense-
24-wide-en-x
NLLB
• 1.3 billion parameters
https://huggingface.co/docs/transformers/model_doc/nllb
6
Both extra large pretrained models can only be fine-tuned, and even for that they require supercomputer.

7. To answer this question we undertook
participation in WMT2022 Biomed2022 MT
challenge, with the aim to train several
models and then compare the results.
Experiment
7

8. Experimentalsetting
8
• Preliminary results:

9. BIOMEDICAL
WMT22
CLINSPEN
CHALLENGE
RESULTS

10. Resultsofthefine-tuning
Clinical-Marian wins clinical-NLLB in Task-1 (all metrics), Task-2 (METEOR, ROUGE), and Task-3 (METEOR,
COMET, ROUGE) on platform metrics.
10

11. All models were trained on the same data and tested on the
same test.
A lot of attention was given to the data preparation and cleaning
for fine-tuning. We finessed these data preparation methods and
tools for our Paralela commercial aligner product
(https://paralela.logrusglobal.com/home), we already had them.
Insufficient metrics of quality measurement
Accurate experiment setup and execution Unexpected result
Conclusions
11
Marian Helsinki demonstrated BETTER results than both xPLM
models!
COMET is clearly incorrect, because quality metrics cannot
exceed 1, therefore COMET metric score has not been normalized
correctly. Also, it does not even proportionally correspond with
other metrics.
Overall, we see here that the quality differences of these models
are not distinguishable with current automatic quality metrics.
The industry is now in situation when the training went ahead of
the ability to evaluate the results of the training. Further work in
the field of quality evaluation needs to be done.
Human evaluation is still a golden standard.
Production-wise, training of extra large language models does
not justify the cost and effort production wise, because the
output quality of smaller models is either better, or the same, or
very close. Consequently, we have reached another plateau of
MT quality with extra large models not fulfilling the promise of
the hype.
Practicality

12. THANKYOU
12
lifeng.han@manchester.ac.uk
serge.gladkoff@logrusglobal.com
[1] Marcin Junczys-Dowmunt and etc. Marian: Fast neural machine translation
in C++. In Proceedings of ACL 2018, System Demonstrations.
[2] NLLB Team. No language left behind: Scaling human-centered machine
translation, 2022. URL https://arxiv.org/abs/2207.04672.
BIBLIOGRAPHY