Translation quality evaluation is challenging and costly for many companies, often representing the largest share of spend in the localization budget and having significant impact on time to market. Our own metrics help us understand and refine our own processes, but what could we learn from sharing our quality data – securely and anonymously of course? Could our collective metrics be leveraged as valuable business intelligence? In June of this year TAUS launched the Quality Dashboard, an industry-shared platform that helps providers and buyers of translation as well as translators to measure and benchmark quality, productivity and efficiency. Through an open API users and translation technology providers can connect with the Dynamic Quality Framework (DQF) tools and track performance on the Quality Dashboard. This panel will discuss the value and power of these shared metrics and the impact of the Quality Dashboard. The panelists are: Steve Richardson (LDS Church), Samuel Läubli (Autodesk), Hyunjoo Han (Autodesk) The goal of this session is to share best practices and experiences across the panelists in terms of metrics-driven quality management. Discuss the industry’s need for tools and technology to aggregate quality data and provide a standardized approach to quality measurement.

1. Treating Translation Quality Metrics as
Business Intelligence
Panelists
Steve Richardson, LDS Church
Samuel Läubli, AutoDesk
Hyunjoo Han, AutoDesk
Nancy Anderson, EMC
Moderators
Eduardo D’Antonio, VMware
Clove Lynch, VMware

2. Key Questions
• What can we learn from sharing our data?
• Specifically, what kind of business intelligence
could we get (that we don’t have today) from
sharing this data?
• How is technology helping you measure it
today?
• How could the TAUS DQF and Quality
Dashboard contribute?

3. LDS Church

4. Translation at the Church of
Jesus Christ of Latter-day Saints
• 15M+ members in 168 countries
• Approximately 100M translated words produced annually
• Materials in 90+ languages in dozens of domains are
translated on a regular basis
• MT systems deployed for English to/from 39 languages
• Materials in 100+ languages are available at LDS.ORG

5. Measuring Quality at the Church
• Challenges:
– 1,000+ full-time & contracted translators around
the world
– From paper to CAT tools w/high speed internet
• Until now:
– Policies and human processes focused on
individual translators’ work
– Tried SAE J2450; moved to simpler methods
– Never been able to measure the overall effect on
quality of published materials

6. Measuring Quality at the Church:
A current example

7. Measuring Quality at the Church:
What we want to know
• What is the quality of our final output?
• Does our focus on translators and the
processes we use equate to improved final
quality, decreased cost, and timeliness?
• What is the cost/benefit ratio of the various
steps in our translation/review processes?

8. Over the past two years, we have
standardized on these systems and tools:
We have used DQF a number of times to measure MT
quality, ranking, and PE productivity

9. DQF Quality Evaluation

10. DQF Ranking Evaluation
German Swedish Norwegian
Danish Polish

11. AA
BB
CC
DD
EE
AA BB CC DD EE
DQF Productivity Evaluation
CC
DD

12. Set up DQF Workflow in SDL WorldServer

13. Create DQF-enabled Project in WorldServer

14. Download Project to SDL Trados Studio

15. After Translation, View DQF Report

16. Integrating DQF into our
Translation Processes
• Continue to assess individual translator
performance to target training needs
• Assess quality and productivity across
translators, languages, and content types
• Evaluate effect of our process steps on final
output quality, cost, and timeliness
• Further analyze productivity and refine pay
rates for MT post-editing
• Compare our measurements to industry
averages to adjust and/or improve

17. EMC

18. Aggregate Quality per Language

19. Linguistic Quality per vendor

20. Autodesk