TAUS OPEN SOURCE MACHINE TRANSLATION SHOWCASE, Paris, Kerstin Bier, Sybase, 4 June 2012

TAUS OPEN SOURCE MACHINE TRANSLATION SHOWCASEAn MT journey: MT in use atSybase, a SAP company14:20-14:40Monday 4 JuneKerstin BierSybase

An MT journey:Moses in use atSybase, an SAP companyKerstin BierSybase Technical Publications Solutions

Our way to MT – When we started in 2009...  Traditional TM technology almost fully exploited: Cost pressure  ca. 80% of costs spent on „new“ words  only 20% spent on recycling  No more improvements in turnaround times:  Average translator productivity 2400 Time to market words/day or less (JA) for years  Encouraging developments in the industry:  SMT success stories (Microsoft, Autodesk ...) (S)MT developments  Availability of Moses Open Source toolkit3 – Sybase Confidential – June 11, 2012

Our way to MT – The beginnings We were skeptical...  Costs - Volumes big, but not big enough to justify huge up-front investment: Is open source MT a viable option?  Knowledge - Small team, limited expertise in-house: Can we take that on?  Quality – Can we maintain quality level with post-editing? How it all began...  Joined TDA (TAUS Data Association)  Participated in Microsoft SMT pilot (see www.tausdata.org)  Built up relationship with MT partner:  First contact with Pangeanic in 2009  Trial project: Surprisingly good results for EN-DE engine4 – Sybase Confidential – June 11, 2012

Our way to MT: From trial to production5 – Sybase Confidential – June 11, 2012

Our Moses engine  Training Data: 5 million source words in TMX files  Small data volume, but we do not have more  Our own data to have better control  Moses phrase decoder with add-on for inline markup handling (PangeaMT)  Language pair: EN -> DE  Other languages? Planned but not started yet  Setup:  In-house: Reasonably powerful machine currently sufficient (64-bit 8 CPU)  Ubuntu Linux, Moses decoder, PangeaMT, several pre- and post-processing scripts6 – Sybase Confidential – June 11, 2012

Our challenges: What Moses does not offer  Engine training/data issues  Getting enough and the right data  Integration into translation workflow  WorldServer  Post-editing environment and translator resistance  Getting the data right: Handling special input and output requirements  Inline XML tags  „Hybrid“ content: Translations mixed with EN  New terminology  Metrics:  Engine performance: bad output vs. good output  Post-editing effort/productivity increase7 – Sybase Confidential – June 11, 2012

What Moses does not offer: Handling special output requirements Solutions for inline tag handling:  PangeaMT inline handler Inline Markup  Scripts to pre-process „notranslate“ tags Content  Scripts to resolve UI references Solutions for DNT handling:  product/component names mostly OK through engine training DNTs  domain-specific untranslatable terms: pre- (Do Not Translates) processing and Moses XML markup Solutions for unknown terminology:  Pre-processing: OOV analysis in Moses, feed in Unknown via XML markup terminology  Issues: English multi-word compounds and grammar (cases)8 – Sybase Confidential – June 11, 2012

What Moses does not offer: Measuring engine performance and productivity increase  Measuring engine performance is critical:  to evaluate how engine evolves  as basis for translator payment  Option 1: Scoring tools  Problem with BLEU & friends: Just averages  Need to find out how much good, average and bad output there is, ideally something like TM matching categories?  Option 2: Human evaluation  Ask translators to assess quality? We found that highly subjective („I had to re-translate everything“)  We still continue to interview them but treat feedback with caution9 – Sybase Confidential – June 11, 2012

Custom solution: Measuring engine performance with segment-based scoring  Developed a workflow to evaluate MT output with METEOR  Developed scripts to determine percentage of„crap“ the engine produces and categorized PE effort  Basis for payment: Below threshold = full price, over threshold = fixed price reduction METEOR score range 100-70 50- 69 40 - 49 30 - 39 0 – 29 Retraining10 – Sybase Confidential – June 11, 2012

Our results11 – Sybase Confidential – June 11, 2012

Moses - Lessons Learned ² Moses out-of-the box offers:  State-of-the-art SMT toolkit and lots of tools ² Moses MT output often better than expected  Often better than translators say  Output quality depends on your training data AND can be influenced greatly by pre- and post-processing ² Moses is still a toolkit „only“  You will likely need to develop your own process around MT  You need a partner or you need capable developer(s) – you do not need to know all the inside stuff  You definitely need to know your data & output requirements to customize the tools around Moses12 – Sybase Confidential – June 11, 2012

Moses – Limitations: What we think is needed² Handling of source formats and tags  Support of plain text data for training: Loss of valuable meta-data  Inline tag handler should be integrated (promising: m4loc!)² Limited/no support for Asian languages „out-of-the box“  Need for separate tokenizer/word breaker² Better integration of more useful metrics & confidence values  BLEU & friends are just averages and can only be applied when you have reference content  Integration of confidence values to filter out bad content: Critical to improve translator acceptance of MT13 – Sybase Confidential – June 11, 2012

Thank you! For questions, feel free to contact Kerstin.Bier@sap.com.14 – Sybase Confidential – June 11, 2012

Examples: MT output and PE effort Minimal PE effort Small PE effort Medium PE effort15 – Sybase Confidential – June 11, 2012

TAUS OPEN SOURCE MACHINE TRANSLATION SHOWCASE, Paris, Kerstin Bier, Sybase, 4 June 2012

by TAUS - Enabling better translation on Jun 11, 2012

Accessibility

Categories

Upload Details

Usage Rights

Statistics

TAUS OPEN SOURCE MACHINE TRANSLATION SHOWCASE, Paris, Kerstin Bier, Sybase, 4 June 2012 Presentation Transcript