TAUS MT SHOWCASE, The WeMT Program, Olga Beregovaya, Welocalize, 10 October 2013


Published on

This presentation is a part of the MosesCore project that encourages the development and usage of open source machine translation tools, notably the Moses statistical MT toolkit. 

MosesCore is supported by the European Commission Grant Number 288487 under the 7th Framework Programme.

For the latest updates go to http://www.statmt.org/mosescore/
or follow us on Twitter - #MosesCore

Published in: Technology, Business
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

TAUS MT SHOWCASE, The WeMT Program, Olga Beregovaya, Welocalize, 10 October 2013

  1. 1. TAUS  MACHINE  TRANSLATION  SHOWCASE   The WeMT Program 10:20 – 10:40 Thursday, 10 October 2013 Olga Beregovaya Welocalize
  2. 2. WeMT  Tools  and   Processes  
  3. 3. We’ll talk about: •  MT  Programs   •  Metrics   •  Engines   •  Language  Tools  
  4. 4. Current MT Programs   Dell  –  27  languages   Autodesk  –  11  languages   PayPal    -­‐  8  languages   Cisco  –  17  languages  between  3  Ders   Intuit  –  20+languages   MicrosoH  (pre-­‐project  support)     McAfee  (pilot)     …  many  more  in  pilot  stage  
  5. 5. MT Program: Path-to-Success Components   A  set  of  MT  engines  –  “mix  and  match”     TMT  SelecDon  Mechanisms   Post-­‐ediDng  Environment     Processes  and  metrics     Data  gathering  and  reporDng  tool  –  what,   how  much,  how  fast  and  at  what  effort     EDUCATION  EDUCATION  EDUCATION     CHANGE   The recipe for success
  6. 6. Process and Workflow All aspects of the localization ecosystem are taken into consideration MT KPIs: Selec3ng  the  right  MT  engine   By  using  our  MT  engine  selecDon  Scorecard  we  make  sure  all   important  KPIs  are  taken  into  consideraDon  at  selecDon  Dme     Empowerment  through  educa3on   Internal,  by  the  use  of  customized  Toolkits;  external,  through   specialised  Trainings.   The  feedback  loop   ConstrucDve  communicaDon  from  post-­‐editor  to  MT   provider   ü   Produc3vity:  Throughputs   ü   Produc3vity:  Delta     ü   Quality:  LQA     ü   Quality:  Automa3c  Scores   ü   Cost   ü   GlobalSight:  Connec3vity     ü   GlobalSight:  Tagging     ü   Human  Evalua3on   ü   Customiza3on:  Internal/External   ü   Customiza3on:  Time  
  7. 7. MT Program Design - Source o  o  o  o  o  o  Source  content  classificaDon  (i.e.  markeDng/UI/UA/UGC)   Length  of  the  source  segment   Source  segment  morpho-­‐syntacDc  complexity   Presence/absence  of  pre-­‐defined  glossary  terms  or  mulD-­‐word  glossary   elements,  UI  elements,  numeric  variables,  product  lists,  ‘do-­‐not-­‐translate’   and  transliteraDon  lists   Tag  density  -­‐  Metadata  aeributes  and  their  representaDon  in  localizaDon   industry  standard  formats  (“tags”)   ROC  –  quality  levels  based  on  content  use  (“impact”)   3D  Model:  Expected  producDvity  mapped  to  desired  quality  levels  and  source   content  complexity    
  8. 8. MT Engine Selection Scorecard Produc3vity  -­‐  Throughputs    Number  of  post-­‐edited  words  per  hour   Produc3vity  -­‐  Delta      Percentage  difference  between  translaDon  and  post-­‐                              ediDng  Dme   Cost    ExtrapolaDon,  cost  per  word   CMS  -­‐  Connec3vity     We have tested and used  Is  there  a  connector  in  place?   different engines so we’ve seen the good, the bad and the ugly; Quality/Nature  of  source   now we can better appreciate Quality  (Final)  -­‐  LQA     what we have  Internal  quality  verificaDon   Quality  (MT)  -­‐  Automa3c  Scores    A  set  of  automaDc  scoring  systems  is  used  
  9. 9. Scorecard - Metrics Overall  data     German KPIs #  1 #  2 #  3 #  4 Productivity 4 4 4 4 Productivity  Increase 5 4 1 3 Quality  -­‐  LQA 2 2 1 2 Quality  -­‐  Automatic  Scores 3 3 3 3 Cost 4 2 3 3 GlobalSight  -­‐  Connectivity   4 3 2 4 GlobalSight  -­‐  Tagging   4 2 4 2 Human  Evaluation 3 3 3 4 Customization  -­‐  Internal/External 4 2 3 3 Customization  -­‐  Time 3 1 2 1 Total 36 26 26 29 French KPIs #  1 #  2 #  3 #  4 Productivity 4 5 3 4 Productivity  Increase 5 5 1 4 Quality  -­‐  LQA 5 3 3 4 Quality  -­‐  Automatic  Scores 3 4 3 3 Cost 4 2 3 3 GlobalSight  -­‐  Connectivity   4 3 2 4 GlobalSight  -­‐  Tagging   4 2 2 2 Human  Evaluation 3 3 3 3 Customization  -­‐  Internal/External 4 2 3 3 Customization  -­‐  Time 3 1 2 1 Total 39 30 25 31 ProducDvity  metrics   AutomaDc  Scoring   Human  EvaluaDon  
  10. 10. Toolkits and Trainings Our  experience:       ü   Most  translators  know  and  have  experienced  post-­‐ediDng  but  they  have   limited  knowledge  of  any  other  related  aspect  (automaDc  scoring,  output   differences  between  RBMT  and  SMT...)   ü   The  majority  of  people  who  work  in  localizaDon  have  heard  about  MT  but   most  of  them  sDll  find  it  a  daunDng  subject.   Our  answer:     ü   ConDnuous  MT  and  PE  related  trainings  and  documentaDon  for  language   providers   ü   Customized  Toolkits  for  different  internal  departments  (ProducDon,  Quality,   Sales,  Vendor  Management)  
  11. 11. Transparency and Ownership Theory  –  knowledge  foundaDons     Prac3ce  –  customized  PE  sessions  for  different  client  accounts         Transparency  –  process,  engine  selecDon/customizaDon,  evaluaDons   Training  helps a lot - After I was told some of the background information and tips and tricks for certain engines/ outputs, I was much more relaxed and happy to give MT a go. Responsibility  –  valid  evaluaDons,  construcDve  feedback,  quality  ownership  
  12. 12. Legacy data – best prediction tool   >  StaDsDcs  from  legacy  knowledge  base  
  13. 13. The feedback loop For me the biggest advantage would be the possibility to implement a client terminology list [in SMT] I wish we could easily fix the corpus for outdated terminology and characters Teach the engine to properly cope with sentences containing more than one verb and/or verbs in progressive form engine retraining improved significantly the handling of tags and spaces around tags, this is a productive achievement as it saves us a lot of manual corrections.
  14. 14. Feedback and Engine Improvement
  15. 15. “Beyond the Engine” Tools •  Teaminology  -­‐  crowdsourcing  plamorm  for  centralized  term  governance;  simultaneous   concordance  search  of  TMs  and  term  bases  =>  clean  training  data   •  Dispatcher  -­‐  A  global  community  content  translaDon  applicaDon  that  connects  user   generated  content  (UGC)  including  live  chats,  social  media,  forums,  comments  and   knowledge  bases  to  customized  machine  translaDon  (MT)  engines  for  real-­‐Dme   translaDon   •  Source  Candidate  Scorer  –  scoring  of  candidate  sentences  against  historically  good  and   bad  sentences  based  on  POS  and  perplexity     •  Corpus  Prepara3on  Toolkit  –  set  of  applicaDon  to  maximize  data  preparaDon  for  MT   engine  training  
  16. 16. Teaminology Teaminology
  17. 17. Dispatcher
  18. 18. Source Candidate Scorer Source Candidate Scorer Compares  your  source  content  to  “the  good”  and  “the  bad”   legacy  segments  and  esDmates  potenDal  suitability  for  MT  
  19. 19. Corpus Preparation Suite   •  •  •  •  •  •  •  Variety  of  tools  to  prepare  corpus  for  training  MT  engines  such  as:   DeleDng  formaong  tags  from  TMX   Removing  double  spaces   Removing  duplicated  punctuaDon  (e.g.  commas)   DeleDng  segments  where  source  =  target   DeleDng  segments  containing  only  URLs   Escaping  characters   Removing  duplicate  sentences  
  20. 20. Corpus Preparation: TM Creator Aggregates  training  data  from  various  relevant  sources   TM Creator
  21. 21. Corpus Preparation: TMX Splitter Extracts  the  relevant  training  corpus   based  on  the  TMX  metadata    
  22. 22. Welocalize Moses Implementation •  Why?  Far  more  control  over  engine  quality  since  we  can  control  corpus   preparaDon  and  output  post-­‐processing   •  Control  over  metadata  handling   •  Ties  into  our  company  open-­‐source  philosophy   •  Have  experienced  personnel  in-­‐house   •  Can  extend  and  customize  Moses  funcDonality  as  necessary   •  Have  connector  to  TMS  (GlobalSight)       RESULTS:  In  our  internal  tests  with  Moses/DoMT,  we  are  geong  automated   scores  similar  to  commercial  engines  for  the  languages  into  which  we  localize   most.     Same  feedback  received  from  human  evaluators    
  23. 23. … And it works! We are in the position to offer realistic discounts and aggressive timelines providing quality levels appropriate for the content
  24. 24. “Work-in-progress” Projects •  Ongoing improvements to our adaptation of iOmegaT tool (Welocalize/CNGL) •  Industry Partner in CNGL “Source Content Profiler” project •  Adoption of TMTPrime (CNGL) - MT vs. Fuzzy Match selection mechanism •  Language and content-specific pre-processing for the inhouse Moses deployment •  Teaminology – adding linguistic intelligence
  25. 25. Questions Language_Tools_Group_all@welocalize.com   We  speak  MT  -­‐  the  language  of  the  future