Panelists: Yoshiyasu Yamakawa (Intel), JP Barraza (Systran), Konstantin Dranch (Memsource), David Koot (TAUS) The focus of this session will be on predictions and risk management. What kind of things can you predict and how can you manage risks by by analyzing your translation data or monitoring your productivity and quality. Tracking translation data in different cycles of the translation process (translation, post-editing, review, proof-reading) offers tremendous value when it comes to predicting future trends or making informed choices. What type of data can be valuable and what kind of predictions can we make using this data? How can we make more efficient use of already available data? How can we use this type of data to improve machine translation, automatic QA, error-recognition, sampling or quality estimation? How can academia and industry work together towards a common goal?

1. www.systrangroup.com
2016
SYSTRAN
TAUS QE Summit

2. PAGE2
www.systransoft.com I C O N F I D E N T I A
L
MT-PE Translation Productivity
• Lessons learned with a Korean LSP project. The results were jointly
presented at the Technical Communicator Symposium in Kyoto.
• LPs: EN>ES, EN>DE
• Domain: Cell phone user manuals (narrow domain)
• Available TM: 20K each from similar manuals
Use-case example

3. PAGE3
www.systransoft.com I C O N F I D E N T I A
L
MT-PE gain with pure text
• MT quality  OK level, clearly see MT-PE gain with pure text
• MT-PE was expected to have a productivity gain given the metrics
BLEU TER
Spanish 56.43 28.76
German 41.53 45.44

4. PAGE4
www.systransoft.com I C O N F I D E N T I A
L
MT-PE productivity
• MT-PE productivity in terms of Words Per Hour (WPH) pure text
translation (removing effects from PE tools, process, and tag
handling, etc)
• Consistent WPH gain with MT-PE process at MT quality level with pure
text (averaged over 2 HTs, 3 MT-PE-ers.)
Word Per Hour
HT MT-PE +/-
Spanish 682 938 38%
German 480 851 77%

5. PAGE5
www.systransoft.com I C O N F I D E N T I A
L
Actual MT-PE Gain
• Actual MT-PE using PE Tool  Does not see MT-PE Gain
• In the actual MT-PE workflow, the results were affected by type of PE
tool used, HT familiarity with the PE tool, ease of use of the PE tool,
and translator attitude towards MT-PE itself
Word Per Hour
HT MT-PE +/-
Spanish 849 811 -5%
German 706 774 10%

6. PAGE6
www.systransoft.com I C O N F I D E N T I A
L
Lessons learned
• To better predict productivity and manage risks, it is important to
design and implement the QE process as close as the actual MT-
PE workflow environment
0
100
200
300
400
500
600
700
800
900
1000
Spanish Text Spanish Actual German Text German Actual
WPH: MT-PE Text vs Actual
HT MT-PE

7. PAGE7
www.systransoft.com I C O N F I D E N T I A
L
QE for Pure Neural™ MT
• For each language pair, 100 sentences “in domain” (*) were collected,
• These sentences were sent to human translation (**), and translated
with candidate models and using available online translation services
(***).
• Without indicating the mix of human and machine translation, a team
of 3 professional translators / linguists fluent in both source and
target languages was asked to rank 3 random outputs for each
sentence based on their preference of translation.
– Preference includes accuracy as a priority, but also fluency of the generated
translation.
– They had the choice to give 3 different ranks, or could decide to give 2 or 3
of them the same rank, if they couldn’t decide.
Human Ranking Evaluation for 1st PNMT release

8. PAGE8
www.systransoft.com I C O N F I D E N T I A
L
Initial findings – 1st PNMT engines
• Con: The most salient errors come from missing words or parts of
sentences.
– We are working to address these in the subsequent releases
• Pro: We observe that NMT drastically improves fluency, slightly
reduces meaning selection errors, and handles morphology better
• Cons: Multiple practical issues worth sharing
– Translating very long sentences
– Translating user input such as a short word or
– Title of a news article
– Cleaning the corpus
– Alignment
Comparative error classification – what are the pros and cons?

9. PAGE9
www.systransoft.com I C O N F I D E N T I A
L
Error Category NMT RB SMT
Entity
Major 7 5 0
Format 3 1 1
Morphology
Minor - Local 3 2 3
Minor - Sentence Level 3 3 5
Major 3 4 6
Meaning Selection
Minor 9 17 7
Major - Prep Choice 4 9 10
Major - Expression 3 7 1
Major - Not Translated 5 1 4
Major - Contextual Meaning 14 39 14
Word Ordering and Fluency
Minor 2 28 15
Major 3 16 15
Missing or Duplicated
Missing Minor 7 3 1
Missing Major 6 1 3
Duplicated Major 2 2 1
Misc. (Minor)
Quotes, Punctuations 2 0 0
Case 6 0 2
Total
Major
Minor
Minor & Major
47 84 54
36 55 35
83 139 89

10. PAGE
10
www.systransoft.com I C O N F I D E N T I A
L
Initial findings – 1st PNMT engines
The role of automatic scores in NMT
NMT engines were not
tuned to optimize any
metrics and yet still
score well!
Automatic metrics are
thus only interesting in
the training process and
not really for Production