Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
No Hardware. No Software. No Hassle MT.
Machine Translation & Quality
What we aim to cover?
 The MT & Quality Relationship
 What is quality?
 Possible ways of measuring it
 Automated evalu...
The Quality & MT Relationship

Machine Translation & Quality
Attributes of Quality
 Language Attributes
 Adequacy



Accuracy of generated texts
Based on word recall & precision

...
Automated Evaluations
 Many difference techniques available



All compute similarity of generated texts to reference t...
Who needs to measure Quality?
 The Localisation Stakeholder Dilemma
 Developers of MT Engines




Automated BLEU, METE...
Who needs to measure Quality?
 The Localisation Stakeholder Dilemma
 Production Teams (PMs, LEs and QEs)


Need segment...
F-Measure

TER

BLEU

GTM

METEOR

NIST

MT Developers

Production

The Quality & MT Relationship

Machine Translation & Q...
Conclusions
 There are many automated MT quality measurements




Mostly suitable for MT developers
Not optimal for pr...
Upcoming SlideShare
Loading in …5
×

What is Quality? A Machine Translation Perspective

492 views

Published on

Presentation given by Tony O'Dowd, Founder and Chief Architect, KantanMT at the Gala Roundtable in Carton House, Ireland.

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

What is Quality? A Machine Translation Perspective

  1. 1. No Hardware. No Software. No Hassle MT.
  2. 2. Machine Translation & Quality
  3. 3. What we aim to cover?  The MT & Quality Relationship  What is quality?  Possible ways of measuring it  Automated evaluation methods  Who needs to measure quality  Localisation stakeholders  Conclusion Machine Translation & Quality
  4. 4. The Quality & MT Relationship Machine Translation & Quality
  5. 5. Attributes of Quality  Language Attributes  Adequacy   Accuracy of generated texts Based on word recall & precision  Fluency    Comprehensibility of texts Readability, understandability Based on phrase reuse and assembly  Task-oriented Attributes  Productivity  Post-editing speed  Acceptability   Fit-for-purpose measurement Usable translations within the context of the end user Machine Translation & Quality
  6. 6. Automated Evaluations  Many difference techniques available   All compute similarity of generated texts to reference texts The smaller the difference => the better the quality! NIST Fluency Usability GTM F-Measure Productivity TER Adequacy BLEU Acceptability METEOR Language Task Machine Translation & Quality
  7. 7. Who needs to measure Quality?  The Localisation Stakeholder Dilemma  Developers of MT Engines   Automated BLEU, METEOR, F-MEASURE, TER ideal and practical No individual measurement has absolute meaning  but points quality curve in the right direction within a domain Machine Translation & Quality
  8. 8. Who needs to measure Quality?  The Localisation Stakeholder Dilemma  Production Teams (PMs, LEs and QEs)  Need segment measurements on quality and PE efforts   Determine tiered segment post-edit rate Distribution of post-editing tasks based on segment quality  Localisation Managers  Need productivity measurements to predict budget and schedule   Aka Project Segment Reports MT Measurements need to ‘fit’ business planning and charge models  Translators  Unfortunately, don’t get a fair deal  No segment information, just top level project Machine Translation & Quality
  9. 9. F-Measure TER BLEU GTM METEOR NIST MT Developers Production The Quality & MT Relationship Machine Translation & Quality
  10. 10. Conclusions  There are many automated MT quality measurements    Mostly suitable for MT developers Not optimal for production teams Of no use to translators  All rely on reference texts to compute measurements  What’s needed?  Segment level measurements   Drive project schedule and charge model High correlation to human effort  Do not rely on reference texts to compute measurements Machine Translation & Quality

×