How do we measure translation quality and the effectiveness of the localization activities?

1. Quality Translation 21 (QT21) has received funding from the EU’s Horizon 2020 research and innovation programme under grant no. 645452. META-QT has received funding
from the EU’s Horizon 2020 research and innovation programme through the contract CRACKER (grant agreement no.: 645357). Formerly co-funded by FP7 and ICT PSP through the
contracts T4ME (grant agreement no.: 249119), CESAR (grant agreement no.: 271022), METANET4U (grant agreement no.: 270893) and META-NORD (grant agreement no.: 270899).
The Multilingual Europe Technology Alliance
Harmonization of DQF and MQM
Sets the Stage for (a) Standard
Translation Quality Metrics
Aljoscha Burchardt (DFKI)
with
Arle Lommel (CSA), Attila Görög (TAUS) &
Hans Uszkoreit (DFKI)

2. Aljoscha Burchardt • TAUS Roundtable Vienna 2016
• Translation quality assessment was subjective
• 1990s: move to vendor-specific checklist-based criteria for
quality assessment based on error counts
• Next step: shared metrics
– LISA QA Model
– SAE J2450 (automotive industry)
– Often heavily modified
• Attempt in ISO to implement a universal metric for all
translation (cancelled in 2013)
A little history
2

3. Aljoscha Burchardt • TAUS Roundtable Vienna 2016
• Little agreement on precisely what constitutes a good
translation at the detailed level
• Different requirements
• Quality is multidimensional
Why no universal metric?
Good? Good?
3

4. Aljoscha Burchardt • TAUS Roundtable Vienna 2016
• Part of the broader Dynamic Quality Framework
(DQF) suite of tools
• Based on best practice in industry
• Designed to address common needs at relatively
low granularity for simple use
• Six error categories + four additional features (20+
subtypes)
• Recently TAUS has launched the Quality
Dashboard for tracking and benchmarking quality
What is TAUS DQF Error Typology?
4

5. Aljoscha Burchardt • TAUS Roundtable Vienna 2016
– MQM is a framework for defining translation quality metrics
– From EU project (QTLaunchPad)
– Designed as a non-strict superset of prominent metrics (LISA
QA Model (3 models), SAE J2450, ISO 14080 (cancelled), SDL
TMS Classic, ApSIC XBench, Okapi CheckMate, xliff:doc,
Yamagata QA Distiller, ATA certification
– 100+ issue types in a hierarchy
– Built around a “master vocabulary” for describing task-specific
metrics;
– Very early version standardized in ITS 2.0
– Open, heavily customizable
What is the Multideminsional Quality Metrics?
5

6. Aljoscha Burchardt • TAUS Roundtable Vienna 2016
Full MQM Structure
6

7. Aljoscha Burchardt • TAUS Roundtable Vienna 2016
MQM Core
7

8. Aljoscha Burchardt • TAUS Roundtable Vienna 2016
New MQM-Compliant DQF hierarchy
8

9. Aljoscha Burchardt • TAUS Roundtable Vienna 2016
• DQF error typology is now a subset of MQM
• DQF can be implemented in MQM tools
(translate5, XTM, etc.)
• Mapping for DQF to ITS 2.0 via MQM
• Agreement to keep MQM and DQF in sync
• MQM is used in QT21 project for MT error
analysis, error profiling, etc.
• DFKI, TAUS and other cooperate in QT21
Present state and ongoing work
9

10. Aljoscha Burchardt • TAUS Roundtable Vienna 2016
Error profiles 1/3
By languages By system types
10

11. Aljoscha Burchardt • TAUS Roundtable Vienna 2016
Error profiles 2/3
11
Example: ES->EN, R2, extra function words
Castigado por la vida, => Punished by the life,
SMT: 36 Segments
RbMT: 53 Segments

12. Aljoscha Burchardt • TAUS Roundtable Vienna 2016 12
Error profiles 3/3

13. Aljoscha Burchardt • TAUS Roundtable Vienna 2016
Towards (a) Standard Translation Quality Metrics
1. A set of relevant quality
dimensions and correspon-
ding issue types:
– Dimensions: Accuracy, Fluency,
Verity, Design, etc.
– Issue Types: Grammar,
Omission, Company Style, etc.
2. Assessment method(s):
– Analytic, holistic, task-based, etc.
3. Weights and thresholds
– For different (major/minor)
issues, show stopper, etc.
Metrics
Interpretation
13
For a given assesment task/benchmark/etc., we need:

14. Quality Translation 21 (QT21) has received funding from the EU’s Horizon 2020 research and innovation programme under grant no. 645452. META-QT has received funding
from the EU’s Horizon 2020 research and innovation programme through the contract CRACKER (grant agreement no.: 645357). Formerly co-funded by FP7 and ICT PSP through the
contracts T4ME (grant agreement no.: 249119), CESAR (grant agreement no.: 271022), METANET4U (grant agreement no.: 270893) and META-NORD (grant agreement no.: 270899).
The Multilingual Europe Technology Alliance
Thank you
Latest version of MQM is maintained at http://qt21.eu/mqm-definition