Online machine translation services offer affordable, high-quality translations. Translation providers want to use the best MT for their customers and freelancers. However, it is difficult to evaluate of black-box, ever changing MT services. In this talk I discussed best practices how to efficiently conduct an evaluation of MT services for the language pairs and content you care about. I introduced the MT Decider Benchmark – a vendor-independent, transparent evaluation of 4 major online MT services for 24 language pairs.

1. Not Ready for Custom Machine
Translation?
How to Find the Best MT Service
for Your Language Pairs and
Content
Achim Ruopp, Polyglot Technology LLC @
lingo systems’ LSP Tech Talk, September 28th 2022

2. Online Machine Translation Services Offer
Affordable, High-Quality Translations
Source: JHU Fall 2017 MT class © Philipp Koehn CC BY 3.0

3. LSPs Want to Use the Best MT for Their
Customers and Post-Editors
Source: Nimdzi Language Technology Atlas
2022 (used with permission)

4. Difficult Evaluation of Black-Box, Ever
Changing MT Services
• MT quality varies as much as
54% or more than 9 BLEU
points between different MT
services
• Over one fifth, or 21.6%, of
top rankings by BLEU score
change from quarter to
quarter
Source: Polyglot Technology MT Decider Benchmark Q2/2022

5. Needed: A Vendor-Independent, Transparent,
and Up-To-Date Evaluation of MT Services

6. MT Decider Provides Reliable, Detailed Data
for Decision Makers

7. Machine Translation Quality
Matters for LSPs

8. Machine Translation Use Cases for LSPs
• Post-editing
• Complimentary use of raw, un-edited machine translations
• eDiscovery
• Perishable, low impact content
• User generated content
• Support content
• …
• Has potential to create more demand for human-level quality translations!
• Think creatively about use cases for your customers
• More use cases become unlocked with increasing quality

9. Better Machine Translation Helps LSPs and
Post-Editors
• Better productivity
• Faster turn-around times
• More enjoyable work for post-editors
Machine Translation Quality and Post-Editor Productivity
(Sanchez-Torron & Koehn, AMTA 2016)

10. Machine Translation Choice in your TMS/CAT
Selected by TMS/CAT
• Routing services
• Phrase
• Intento
• Adaptive machine translation
• Lilt
• ModernMT
• Higher cost
• ~$45/project manager/month1
• Not transparent
• Up-to-date?
• Vendor lock-in?
Configure in your TMS/CAT
• Sign up for online MT service
with credit card
• Most TMS/CAT tools
• E.g. Translate5: DeepL, Globalese,
MS Translator, Google, SDL
LanguageCloud, Lucy, Moses
• Low-cost
• ~$17.50/million characters2
• Same quality when picking right
service
1: Phrase Translate pricing with similar feature set as online MT services
2: Average Amazon Translate, DeepL, Google Translate, Microsoft Translator without volume discounts

11. Scaling Machine Translation
Quality Evaluation is Key

12. Human and Automatic MT Quality Evaluation
Human Evaluation
• Human evaluators
• Error annotation
• Judgments on accuracy/fluency
• …
• Similar to Linguistic Quality
Assurance (LQA)
• Slow & expensive to scale across
• Language pairs
• Projects
• Not consistently repeatable
Automatic Evaluation
• Uses human reference translations
• Calculates scores that reflect
similarity to reference translations
• Syntactic (BLEU/chrF)
• Semantic (COMET)
• Reference translations for many
language pairs publicly available
• Repeatable

13. Making Machine Translation Evaluation Part
of LQA
• Use LQA process to
• Check final translation
• Check machine translations
• Retain high-quality, representative subset of translations
• As reference translations for automatic evaluation metrics
• For current evaluation
• For assessment of future similar projects

14. MT Decider Provides Automatic
Evaluation for All Use Cases

15. MT Decider Benchmark Provides Reliable, Detailed
Data and Analysis for Decision Makers
• Automatic quarterly evaluation with the industry quasi-standard BLEU and COMET scores
• High quality human reference translations from
• The Conference on Machine Translation (aka “WMT”) from the news/general domain
• The International Conference for Spoken Language Translation (aka “IWSLT”) from transcribed
speech
• 24 language pair reports
• Czech↔German, German↔English, German↔French, English↔Arabic, English↔Spanish,
English↔Estonian, English↔Finnish, English↔French, English↔German, English↔Hungarian,
English↔Italian, English↔Lithuanian, English↔Latvian, English↔Polish, English↔Romanian,
English↔Gujarati, English↔Hindi, English↔Tamil, English↔Japanese, English↔Chinese,
English↔Kazakh, English↔Turkish, English↔Pashto and English↔Russian
• 4 online machine translation services
• Amazon Translate, Google Translate, DeepL, Microsoft Translator
• Detailed summary report

16. MT Decider Benchmark Provides Reliable, Detailed
Data and Analysis for Decision Makers

17. MT Decider Provides Evaluation for All
Content Types
Diverse, New Project
Content
• Specific
domain/industry
vertical
• No reference
data
• Provided in
collaboration
with TAUS as
DeMT™ Evaluate
Your Customers Content
• Reference Data
• Existing for repeat
customers
• Establishing for larger
projects
• Automatic evaluation with
MT Decider OSS
(upcoming)
• Verify against MT Decider
Benchmark
• Supplement with human
evaluation/LQA
Industry Verticals
Content
• New content type
or generic
• Small projects
• Raw MT use
• Quick pick of MT
service with
MT Decider
Benchmark

18. Summary
• Choosing Affordable, High-Quality MT Services is Hard
• MT Decider provides reliable, detailed, transparent, up-to-date data
and tools to make the right choice in every scenario
• Sign up for updates at https://polyglottech.gumroad.com/follow
• Upcoming Q3/2022 MT Decider Benchmark
• Availability of MT Decider OSS to evaluate your own data
• Subscribe to MT Decider Benchmark reports at
https://www.polyglot.technology/p/mt-decider.html