Post-editors are an important partner in the MT flow – they are the last line of defense for a quality output and can provide valuable feedback for the improvement of the system. However, it is a known industry fact that their rate of attrition is much higher than for “traditional” translators. There are multiple reasons for this (throughput estimation, content difficulty, MT-specific requirements), and there are also new, innovative solutions based on real-life experience that we can implement to facilitate their work and keep them engaged.

1. ENGAGING
POST-EDITORS
JOSÉ LUIS BONILLA SÁNCHEZ
October 14, 2015

2. TOPICS
•BACKGROUND: MT AT EBAY
•THE CHALLENGES OF POST-EDITING (INDUSTRY AND EBAY)
•THE EBAY EXPERIENCE: PROBLEMS AND SOLUTIONS
•A LOOK AT THE FUTURE
•DISCUSSION
ENGAGING POST-EDITORS 2

3. MT AT EBAY
•Since 2013
•Home-grown, statistical (Moses) engines
•Covering 10 language pairs (FR IT ESES ESLA DE BRPT
•RU ZH USEN UKEN)
•Content translated: listings (item titles and descriptions), keywords
ENGAGING POST-EDITORS 3

4. CHALLENGES OF POST-EDITING
ENGAGING POST-EDITORS 4

5. INDUSTRY CHALLLENGES
ENGAGING POST-EDITORS 5
Post-Editing projects add extra complications to the regular L10n flow. For
instance:
- Quality expectations can be unclear (definitions of light and
full post-editing vary).
- No universal agreement on rates (Per hour? Percentage
discount? Edit distance?).
- Every project (and engine) comes with its own difficulties:
statistical vs rule-based engines, technical vs social content…

6. EBAY CHALLENGES
ENGAGING POST-EDITORS 6
At eBay, we deal with especially complex projects:
- 12k+ eBay categories, many with their own terminology
- User-generated content: Unpredictable quality, slang, non-
standard acronyms and abbreviations…
- Very specific requirements: Our goal is not polished content,
but content which can be understood and useful to train the
engine

7. Our Solutions

8. ENGAGING POST-EDITORS 8

9. Modular Guidelines…
ENGAGING POST-EDITORS 9

10. …Structured to Facilitate Learning
ENGAGING POST-EDITORS 10
General
introduction
PE-Specific
Instructions
Item Titles Languages
RU
BRPT
ESLA
FR
IT
ESES
DE
ZH
Item
Descriptions
Queries
General
Translation
Instructions

11. Recorded Trainings
ENGAGING POST-EDITORS 11
The more specialized the training…
…the more important to preserve the
information in recorded format so future
post-editors can refer to it.

12. Escalation as Needed
ENGAGING POST-EDITORS 12
3rd fail triggers a call with the vendor.
• Participants:
Linguists, PjM, Quality Manager
• Agenda:
- Diagnosis
- Vendor Action Plan
- Feedback for Client

13. 2014 RESULTS (FAILS VS PASSES)
ENGAGING POST-EDITORS 13
0
5
10
15
20
25
30
35
40
45
JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
Reviews Pass Fail

14. ENGAGING POST-EDITORS 14

15. EARLY SAMPLE REVIEWS
ENGAGING POST-EDITORS 15
In-progress review for most
problematic language combinations:
First 2k words of the project,
in the first project week

16. ADDITIONAL REFERENCE MATERIAL
Providing vendors with MT
translations from 2 engines
(generic and customized) so they
pick the best
Out of Vocabulary words are unknown to
the system, and left in English, which
often makes them easy to mistake for
brand names. By automatically tagging
OOV, we allow the vendor to focus on
them.
eBay listing titles can be difficult
to understand without context
(images, descriptions).
Alternative MT
translation
Tagging “suspect”
terms (OOV)
Providing full context
(HTML files)
ENGAGING POST-EDITORS 16

17. RESULTS - PRODUCTIVITY
ENGAGING POST-EDITORS 17
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
Productivity Increase
Alternative MT
Translation
Tagging
"suspect" terms
(OOV)
Providing full
context

18. RESULTS - QUALITY
ENGAGING POST-EDITORS 18
0
5
10
15
20
25
30
35
40
45
JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
Pass/Fail over Time - 2014
Reviews Pass Fail
0
10
20
30
40
50
60
70
JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
Pass/Fail over Time- 2015
Reviews Pass Fail

19. THE FUTURE
Divider sub-headline goes here

20. THE NEXT BREAKTHROUGHS
By predicting the quality of the MT output,
we could:
- Filter and send to our vendors the best (or
worst) translations.
- Map QE score to time spent, and use it to
calculate more accurate initial rates.
Tools like iOmegaT or Matecat offer time
tracking, edit distance analysis and even
action recording – usable to:
- Analyze post-editor behavior and identify
areas to help them improve.
- Calculate accurate rates.
The future is for online tools – they will allow
for more direct interaction (early sampling,
continuous communication, centralized
information repository), further integrating
the post-editors with the team.
Quality Estimation
Behavior Tracking
Online Collaboration
ENGAGING POST-EDITORS 20

21. DISCUSSION TIME
ENGAGING POST-EDITORS 21