This document discusses Google's approach to managing translation quality. It outlines how Google selects vendors, provides training, conducts linguistic quality evaluations (LQE) on sample translations, shares LQE data with vendors, and tracks key performance indicators. The LQE tool provides instant feedback to help vendors and translators improve. Challenges include ensuring consistent reviewer quality criteria and obtaining high quality from the start of translations.

1. Confidential & ProprietaryConfidential & Proprietary
TAUS QE Summit 2015
Managing quality expectations
Sergio Pelino - Google

2. Confidential & ProprietaryConfidential & Proprietary
Vendor management:
● Vendor selection and SLAs (Quality, OTD, Cost KPIs)
● 4 Translation vendors and 2 review vendors
Training/standards:
● Training and onboarding programs
● Language Services (LS) team owns quality standards, terminology and training
LQE and continuous feedback:
● LQE: Automatic sampling of content for review based on historic performance and risk management
criteria
● Continuous feedback: LQE tool allows to score quality and send immediate feedback for every project
to vendor/translators (pass/fail, err/1k, error types, full change report, reviewers comments)
Data management and sharing:
● LQE data shared with all vendors (via API and UI dashboard)
● KPIs tracking for immediate follow-ups/RCAs
How do we manage quality:

3. Confidential & ProprietaryConfidential & Proprietary
● Quality rules for reviewers can be overly complex and subjective which makes it difficult to have all
reviewers aligned on same criteria when scoring.
● Working on getting quality at the start (context, product knowledge, great TMs and glossaries, QA and
integrity checks in translation tool).
● Stretch target is to have all content on a minimum % sampling cadence (needed for data).
● Vendors capacity, talent management and retention, allocating the right resource to the content type.
Challenges:

4. Confidential & ProprietaryConfidential & Proprietary
Automatic, self-adjusting QE sampling logic based on:
Vendor, Component, Language, Product
- Average errors/1k > 2
- Pass rate > 95%
- # of consecutive Passes > 3-6
Whitelisted Components:
- Marketing
- Legal
Sample 10% of projects
Send all to Review
Business rules*Projects
Vendor_1
Help, JA, Android
Vendor_2
UI, FR, Chrome
Vendor_3
Marketing, IT, Adwords
Sample 20% of projects
(y/n)
Send to review?
*Based on historic performance data on rolling
window of previous 90 days

5. Confidential & ProprietaryConfidential & Proprietary
Our LQE tool
Chrome extension: runs on top of Google Translator Toolkit
Produces instant feedback to Vendors and translators
LQE evaluation criteria
Pass/Fail: 3 error points/1,000 words

6. Confidential & ProprietaryConfidential & Proprietary
Instant, continuous feedback to vendor and translators

7. Confidential & ProprietaryConfidential & Proprietary
LQE Pass/Fail performance over time vs SLA (max fail rate = 5%)

8. Confidential & ProprietaryConfidential & Proprietary
Fail rate per
language.
SLA: 5%

9. Confidential & ProprietaryConfidential & Proprietary
Vendor_x quality vs LSA

10. Confidential & ProprietaryConfidential & Proprietary
Vendor_x error distribution - identify areas for improvement

11. Confidential & ProprietaryConfidential & Proprietary
All vendors VS On Time delivery SLA (Max: 7%)

12. Confidential & ProprietaryConfidential & Proprietary
Vendors training
tracking.
No translator
should touch
Google work
without
completing
training.