1. Preferential feedback needs a relief valve. Reviewers should be allowed to suggest improvements or personal preferences rather than classifying all feedback as errors. Around 30% of feedback is typically preferential. 2. Key performance indicators should measure errors per 1,000 words rather than simple pass/fail criteria. Errors per 1,000 words better indicates translator performance as 1,000 words is about half a day's work. 3. Quality measurements should not rely on numbers alone. Managers should supplement numbers with expert reviewer opinions to better understand factors like why a translation received a good score but the reviewer did not like it.

1. Copyright © 2015 SDL PLC www.sdl.com
Top Ten Best Practices about Translation Quality Measurement
Estimated Reading Time: 5 minutes (@ 200 words/minute)
1. Preferential feedback needs a relief valve
If you don’t give reviewers a way to suggest improvements or personal preferences, then you are
forcing them to classify their feedback as errors. Even if you instruct them not to make preferential
changes, they will. Studies have shown, on average, that 30% of feedback is preferential (sometimes,
even 100%). So, instead of fighting their personal preferences, allow their opinions. Why prevent them
from suggesting an improvement, even if it’s personal? Just don’t call it an error.
Don’t waste time debating if something is right or wrong: When in doubt, classify the reviewer’s
feedback as a personal preference, and move on.
2. KPIs: Use “errors per 1,000 words” instead of “Pass/Fail”
We all love Pass/Fail criteria: They’re simple, and one doesn’t have to know the target language to
understand that the translations are substandard. Dr. W. Edwards Deming, quality guru, loathed them.
Why? Because they stifle continuous process improvement. Once a supplier reaches the magical
“pass zone”, what’s their incentive to keep improving?
Metrics based on “errors per 1,000 words” make for a better Key Performance Indicator (KPI) of
translators’ performance. In case you’re wondering “why 1,000 words?”: 1,000 words indicate the
average throughput of a professional translator in half a business day’s work (4 hours)—a sufficiently
long time to assess performance. (For those who are Six Sigma savvy, you can use DPMO, Defects
Per Million Opportunities.)
3. Don’t manage by numbers alone
It’s very tempting for managers (especially when we don’t understand the target language), to simply
look at the numbers of the translation quality scores. Remember that numbers are a convenient way to
tell a story, but numbers are not the only source of information. It’s important to gather data from all
sources. Supplement the story with the expert opinion of the reviewers.
For example, a translation might get a good score, but your in-country reviewer doesn’t like it. Why is
that? You need to find out. In Six Sigma quality management, there’s a concept called gemba, or “go
and see.” Talk to your reviewers to better understand what’s going on. You might learn something that
the numbers alone cannot tell you.
4. Need for error arbitration / reconciliation
There are two sides to a story. Your reviewer might complain about the quality of the translations. But
what is the translator’s feedback? Did the translator receive everything necessary to do a good job?
Sometimes, errors are caused for reasons outside of the translator’s control, or the reviewer was
missing some key piece of information.
The so-called arbitration or reconciliation cycle is a fundamental step before conclusions can be
reached. Yes, it can lengthen the turnaround time. But, with so much depending on these scores, it
behooves us to ensure that they are accurate.

2. Copyright © 2015 SDL PLC www.sdl.com
5. Distinguish error types from error severities
Don’t pre-assign a severity weight to error categories. Severities indicate the consequence of an error.
Classifying errors types into categories helps managers to understand what areas require
improvement. These are two different things, with different objectives.
For example, not all typos are made equally. A typo will have a different consequence if found in the
title on the front cover of a manual, as opposed to inside the manual in a footnote. The former will
likely cause a reprint, while the latter will not. Same error type, but different consequence, so different
severity.
6. Reproducibility & repeatability
For measurements to be useful, they have to be both accurate and precise. They also need to be
reproducible and repeatable. Reproducible means that different reviewers will look at the same sample
and reach the same (or very similar) conclusions. Repeatable means that the same reviewer will
evaluate the same sample multiple times and reach the same conclusion.
Measurement System Analysis (MSA) and Gage R&R are some Six Sigma techniques used to make
sure that the differences in the data are due to actual differences in what is being measured and not
due to variation in measurement methods or with the reviewers themselves.
7. Ensure that reviewers are impartial
Not all lawyers make good judges. Ergo, a reviewer who is very opinionated might be a good
translator, but won’t make for a good reviewer. You might not always have a choice in the reviewer,
but it’s something to look out for. A biased reviewer will provide biased measurements.
Also, good reviewers will provide impartial feedback in a constructive and professional manner. They
help the translators to become better at what they do, as opposed to “beating them down with a stick.”
8. Refer to authoritative standards
Any measurement method is useless unless it relies upon standards of reference. Glossaries of
approved terminology, language style guides, project instructions and compliance guidelines all serve
this purpose. Generally agreed upon language conventions (think The Chicago Manual of Style and
the Merriam-Webster dictionary for American English) can also serve as authoritative references. If
something is wrong, one should be able to cite a reference.
9. Avoid measuring samples that are too small
You wouldn’t use a telescope to observe something up close. When samples are too small, the error
tolerance disappears, making it impossible to measure variance. Lean Six Sigma teaches that
inspection is a form of waste because it doesn’t add value. Instead of wasting time measuring small
samples (e.g. less than 1,000 words), spend that time for an extra round of proofreading.
10. Focus on what’s important
A good measurement system will look at Critical-to-Quality (CTQ) parameters. Often, translation
quality methods place too much emphasis on imperfections that the end users hardly notice. Is
measuring punctuation errors and double spaces (and then debating them) a good use of
everyone’s time?
If you have any questions about translation quality best practices, feel free to contact the SDL
Business Consulting team: Derek Patrick (Vice President, dpatrick@sdl.com) and Franco Zearo
(Business Consultant, fzearo@sdl.com). Both are based at SDL USA in Superior, Colorado.