What is Quality? A Machine Translation Perspective

202
-1

Published on

Presentation given by Tony O'Dowd, Founder and Chief Architect, KantanMT at the Gala Roundtable in Carton House, Ireland.

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
202
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

What is Quality? A Machine Translation Perspective

  1. 1. No Hardware. No Software. No Hassle MT.
  2. 2. Machine Translation & Quality
  3. 3. What we aim to cover?  The MT & Quality Relationship  What is quality?  Possible ways of measuring it  Automated evaluation methods  Who needs to measure quality  Localisation stakeholders  Conclusion Machine Translation & Quality
  4. 4. The Quality & MT Relationship Machine Translation & Quality
  5. 5. Attributes of Quality  Language Attributes  Adequacy   Accuracy of generated texts Based on word recall & precision  Fluency    Comprehensibility of texts Readability, understandability Based on phrase reuse and assembly  Task-oriented Attributes  Productivity  Post-editing speed  Acceptability   Fit-for-purpose measurement Usable translations within the context of the end user Machine Translation & Quality
  6. 6. Automated Evaluations  Many difference techniques available   All compute similarity of generated texts to reference texts The smaller the difference => the better the quality! NIST Fluency Usability GTM F-Measure Productivity TER Adequacy BLEU Acceptability METEOR Language Task Machine Translation & Quality
  7. 7. Who needs to measure Quality?  The Localisation Stakeholder Dilemma  Developers of MT Engines   Automated BLEU, METEOR, F-MEASURE, TER ideal and practical No individual measurement has absolute meaning  but points quality curve in the right direction within a domain Machine Translation & Quality
  8. 8. Who needs to measure Quality?  The Localisation Stakeholder Dilemma  Production Teams (PMs, LEs and QEs)  Need segment measurements on quality and PE efforts   Determine tiered segment post-edit rate Distribution of post-editing tasks based on segment quality  Localisation Managers  Need productivity measurements to predict budget and schedule   Aka Project Segment Reports MT Measurements need to ‘fit’ business planning and charge models  Translators  Unfortunately, don’t get a fair deal  No segment information, just top level project Machine Translation & Quality
  9. 9. F-Measure TER BLEU GTM METEOR NIST MT Developers Production The Quality & MT Relationship Machine Translation & Quality
  10. 10. Conclusions  There are many automated MT quality measurements    Mostly suitable for MT developers Not optimal for production teams Of no use to translators  All rely on reference texts to compute measurements  What’s needed?  Segment level measurements   Drive project schedule and charge model High correlation to human effort  Do not rely on reference texts to compute measurements Machine Translation & Quality
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×