TAUS MT SHOWCASE, Moses in the Mix. A Technology Agnostic Approach to a Winning MT Strategy, Lori Thicke. LexWorks


Published on

This presentation is a part of the MosesCore project that encourages the development and usage of open source machine translation tools, notably the Moses statistical MT toolkit.

MosesCore is supported by the European Commission Grant Number 288487 under the 7th Framework Programme.

For the latest updates, follow us on Twitter - #MosesCore

Published in: Technology
1 Like
  • As SlideShare comments do not retain formatting I have reposted the below post with formatting here: http://kv-emptypages.blogspot.co.uk/2013/02/dispelling-mt-misconceptions.html
    Are you sure you want to  Yes  No
    Your message goes here
  • The reason for this detailed post is to correct misinformation so that potential MT users can make informed decisions about all MT based products that are based on fact. This is not an attack on the author; rather it is a series of proof points where we and others have disagreed with the author’s perspective with links and references to third party information in support of the counter positions presented.

    At the end of this presentation 4 different organizations raised issues with the content presented. SMT and RBMT are approaches to machine translation and not products. The presentation author is referring to SMT and RMBT as if they are unique products and not a technology approach. There are many products based on either or both approaches that have different features.

    The author seems to have only considered the features available in Moses, Microsoft Translator and Systran, while ignoring the features of many other commercial MT products that have already resolved many of the issues raised. The author appears to have not performed data management and optimization of the training data when creating statistical models. As a result, in the authors experience, with a limited number of products and a subset of features that commercial SMT based products can offer, sweeping statements are made that cover a wide scope have been bundled as if there were just 1 product. Each vendor’s products, whether SMT or RBMT based, have a range of different features that are not being recognized by the author.

    Many of the assumptions that are presented may have been true several years ago. However many of the issues raised with SMT in particular have been recognized and addressed some time ago by commercial MT vendors and as such are no longer true. If Microsoft Translator and Moses do not support a feature, it does not mean that all SMT based products do not support a feature.

    The author makes many sweeping statements that are factually incorrect in the presentation and can readily be verified as such. Multiple individuals have pointed out these discrepancies, but the author has chosen to ignore these proof points and continue to disseminate misleading information. As noted above, several of these issues were raised directly at the end of the presentation where these slides were delivered.

    Examples include:

    1. SMT cannot handle software tags properly. This is incorrect. Moses cannot handle software tags, but many commercial MT platforms based on SMT such as Asia Online’s Language Studio handle tags very well.

    2. SMT does not retain correction to terminology. This is incorrect. If the data is managed properly then management of terminology becomes very easy. Moses and Microsoft Translator do not provide terminology management tools and processes, but products such as Language Studio provide tools to manage and normalize terminology, both when preparing data for training and at translation runtime.

    3. SMT does not have a rapid developed customization cycle. This is incorrect. In Andrew Rufener’s presentation (link below), he notes clearly that Asia Online system improved dramatically over 3 days. And that as they added further data, they had control and the system improved quickly.

    4. SMT output is not predictable. This is incorrect. If the data is managed properly and supported with data manufacturing such as within Language Studio, then the output can be very predictable.

    5. RBMT is better suited to documentation and software. This is incorrect. There are many published case studies to the contrary. As an example, the case study of Omnilingua on the Asia Online website shows 52% of raw MT required zero edits for their technical automotive documentation. There are many other examples from Asia Online and other vendors.

    6. RBMT is better suited to post editing. This is incorrect. As with the above mentioned case study from Omnilingua, engines based on SMT can deliver near perfect quality. The quality of an engine greatly comes down to pre and post processing technologies and the amount of suitable data / corpus that is available for the SMT customization process. With less data or low quality data, translation quality will be poor and the editing will be difficult. With more, high quality data that is in domain, the editing will be less and because it has learned from the clients own translation memories, editing will be significantly less.
    Are you sure you want to  Yes  No
    Your message goes here
  • 7. SMT is not effective with a limited training corpus. This is incorrect. Advances in data manufacturing technologies such as those available in Language Studio mean that even when no data at all is available an engine can still be customized to a high level of quality. The case study on Kirti Vashee’s blog (http://kv-emptypages.blogspot.co.uk/2013/04/pemt-case-study-advanced-language.html) shows how Advanced Language Translation was able to customize engines for their clients with no data at all and only using data manufacturing technologies from within Language Studio.

    8. SMT is not as good at languages like Russian, Japanese and German. This is incorrect. The quality of a SMT engine greatly depends on the quality of the data that is used for training. If the author is getting poor results this may be due to insufficient data / corpus, low quality data or insufficient skills to prepare and process the data in a manner that delivers high quality output (see skills comment below). There are many high quality engines based on SMT that excel over RBMT. Andrew Rufener presented “Implementing large scale Machine Translation in Patent Information” (http://dotsub.com/view/159ce97c-dbd4-4d6a-90c2-427a3a3e755f) where he shows metrics from many RBMT and SMT systems. He took a technology agnostic approach and performed detailed metrics before selecting Language Studio.

    It appears in the article that the author has not managed data well when creating SMT systems and has not used any data manufacturing and optimization technologies as they are never mentioned. This is evidenced by the authors incorrect assumptions that systems based on SMT cannot have managed terminology and are unpredictable.

    The author only considers the hybrid approach of RBMT + SMT based smoothing that is available in Systran and does not consider other hybrid approaches of other vendors such as the hybrid approach of SMT guided by rules and syntax that is offered in Language Studio.

    We strongly recommend that the author expand beyond the 3 MT products listed and undertake to learn about data management and data manufacturing for SMT approaches. In Language Studio, Asia Online undertakes these complex tasks for our customers so that the customer can focus on providing the right data for SMT to learn from without the need for skills and the understanding of the complexity of data optimization. Once the initial optimization and data manufacturing is complete, control is handed to the end customer to add and further refine terminology and other linguistic features.

    What the author has omitted in their presentation is information about the corpus that was used to train the SMT engines and the actual product used to support each specific assumption. The author also omits information on how metrics were performed, how many segments were compared, how productivity was measured in post editing and over what time period productivity was measured. Too few segments and too short a time period can dramatically impact and incorrectly skew results. Additionally while languages were referred to, domains were excluded. The complexity of a domain is an important factor that impacts metrics and quality. Comparatively, LexisNexis case studies listed 9 MT systems and the metrics performed, the data volumes used and the results.

    We agree that a technology agnostic approach to MT is very viable, but as Andrew Rufener points out in his presentation the integration costs and skill levels required to run multiple MT platforms were significant and can often outweigh the benefits of selecting multiple MT technology solutions. Adobe, PayPal and others have successfully deployed multiple MT technologies and some such as Autodesk have been very open with their metrics. However they have made significant investment in skills, time, data acquisition and data optimization, as well as software development. They also are focused on their own narrow domains, not a broad range of domains in multiple languages like an LSP. Thus they have their own existing language assets and do not need to perform as much management of data as an LSP would when receiving TMs from multiple sources such as TAUS and other LSP partners. Trying to make such data that is not in the domain fit a new purpose is very difficult and unlikely to deliver the optimal quality. This mixture of data approach is commonly referred to as “dirty data SMT”, which is very different from the focused domain “clean data SMT” approach that Asia Online takes in Language Studio.
    Are you sure you want to  Yes  No
    Your message goes here
  • We encourage measurement and publication of metrics with all the relevant information about how the metrics were performed and with what data. However only including a small number of products in the group that are evaluated, without considering many of the leading commercial products from multiple vendors and not performing data optimization and data manufacturing means that the results irrespective of product will be biased, skewed and limited to only what the raw corpus can provide. Modern commercial SMT systems go well beyond the capabilities of open source Moses and Microsoft Translator.

    What many LSPs seem to underestimate is the complexity of delivering high quality MT. Being able to install Moses and train your own engine does not mean that you will get high quality any more than owning a sewing machine and cloth make one an expert tailor. As per the LexisNexis presentation, a significant investment in skills is needed. Andrew Rufener notes in his presentation the significant effort that they put into learning SMT approaches and optimizing data. It is our position that there is no one individual or organization that has the necessary skills to be an expert in each of the technology and approach. Much like healthcare, it is too complex for one individual to be an expert in all fields. For this reason, specialist medical professionals are needed for cancer, brain and other treatments. While a general medical practitioner can deal with common low level issues, more complex issues are referred to a specialist. Machine translation is complex. There are few true specialists globally and even fewer that have solid experience in multiple technologies and approaches. Finding an expert in optimizing any of these technologies is difficult. Finding an expert that can deliver the optimal approach and quality from all or even multiple SMT and RBMT vendors’ products is not realistic.

    For this reason, when Asia Online works with LSPs and other customers, we hide the complexity and engage our linguists to execute data manufacturing and optimization processes on the clients behalf. This means that the required level of expertise for our customers and the effort is greatly reduced. The Asia Online team are experts in Language Studio, our SMT based hybrid platform, and know how to optimize it to deliver the best results. The Asia Online team claims no expertise in other SMT based products or RBMT based products as each have their own merits, approaches and optimal configurations.

    We strongly recommend that anyone looking at any MT technology or approach whether RBMT or SMT look for skills that are vendor and product specific. These are not easy to come by, but are the only way to deliver high quality and reduce risk of deployment. Any vendor stating that they are experts in SMT and RBMT without explicitly listing the specific vendors and products that they work with is not going to give the optimal results. Generic skills in SMT and RBMT approaches most certainly cannot deliver the optimal result. Specialists in individual products, not just an understanding of an approach, are required.

    Additional counter positions to more misconceptions raised by the author in this presentation and other previous publications can be found in Kirti Vashee’s blog (http://kv-emptypages.blogspot.co.uk/2013/02/dispelling-mt-misconceptions.html)
    Are you sure you want to  Yes  No
    Your message goes here
  • See the TAUS MT FAQ for an objective overview of rule-based vs. statistical MT. -

    What are the most significant factors driving manager's decisions to use rule-based or statistical MT engines?

    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

TAUS MT SHOWCASE, Moses in the Mix. A Technology Agnostic Approach to a Winning MT Strategy, Lori Thicke. LexWorks

  1. 1. TAUS  MACHINE  TRANSLATION  SHOWCASE  Moses in the Mix: A Technology Agnostic Approach to aWinning MT Strategy!!10:50 – 11:10!Wednesday, 12 June 2013!!Lori Thicke!LexWorks!
  2. 2. Moses  in  the  Mix:  A  Technology  Agnos-c  Approach  to    the  Winning  MT  Strategy  
  3. 3. •  McKinsey s definition of the T-shaped company•  Language Services Provider(Lexcelera, founded 1986; managingtranslators & post-editors)•  MT Services Provider (trainingengines, post-editing, etc.)•  Technology Agnostic!What is LexWorks?!
  4. 4. •  Developing new technologies to help MTwork better with community content!
  5. 5. Other  Technology  Agnos-cs  A good MT strategy should betechnology-agnostic and look for themost efficient solution on a case-by-casebasis. The type of technology that bestsuits your needs will change dependingon the language pair. !
  6. 6. All approaches - SMT, RBMT,Hybrid - are good when matchedto the course!
  7. 7. The  process  aims  to  define  best  of  breed  soluDons  for  superior  performance  MT is not a tool. MT is an industrialprocess.!
  8. 8. 1.  Best  of  breed  means  raw  MT  that  is  perfectly  understandable  MS Translator! Systran Hybrid!sentences:! %! %!not understandable! 15.65! 20.87!partly understandable! 20.00! 34.78!fully understandable! 64.35! 44.35!
  9. 9. Raw  MT  for  FAQs  and  Forum  Content    MS Translator! Systran Hybrid!Average score on FAQ article! 2.6! 2.4!Average score on forum! 2.31! 1.97!Overall score! 2.48! 2.23!
  10. 10. 2. Best of breed means managingpost-editing costs!
  11. 11. 3.  Best  of  breed  means  retaining  your  post-­‐editors  
  12. 12. 4.  Best  of  breed  means  clear  metrics  !Translation engine!!Engine Type!!BLEU Score!!GTM Score(SymEval)!!Systran !!Hybrid!!69.74!!72.69!!Moses!!Statistical!!50.46!!57.93!!Microsoft Translator!!Statistical!!54.01!!60.81!
  13. 13. 15!Area! Feature! RBMT! SMT!Capability!Add rare language pairs! !!Capability!Number of languages it can handle out of the box! 20! 50!Cost! Free or Open Source version exists! !! !!Quality! Respects grammatical rules! !!Quality! Handles software tags properly! !!Quality! Output is fluent! !!Quality! Can handle bad grammar! !!Quality! Quality improves with Controlled Authoring! !!Quality! Output is predictable! !!Quality!Retains corrections to terminology (and appliesthe correct grammar)!!!
  14. 14. 16!Area! Feature! RBMT! SMT!Suitability!Is better for User Generated Content and broaddomain material such as patents!!!Suitability!Is better suited to on-the-fly translations of shortshelf-life content!!!Suitability! Is better for documentation and even software! !!Suitability! Is suited for rare language pairs! !!Suitability! Is better suited to post-editing! !!Training! Learns automatically ! !!Training! Rapid development customization cycle! !!Training! Effective with limited training corpus! !!
  15. 15. 17!Languages! Online! Hybrid! RBMT! SMT!French, Spanish! !! !! !! !!Russian, Japanese, German! !! !!Norwegian, Danish, Thai! !! !!
  16. 16. 18!Content Type & Other Considerations! Online! Hybrid! RBMT! SMT!Documentation, reports, online help, UI! !! !!FAQs, forums, UGC, ! !! !!Patents, other broad domain! !! !!Marketing materials!Insufficient in-domain/out-of-domain data (I,me)!!! !! !!Poor grammar, spelling! !! !!
  17. 17. Choose  the  horse  that  will  win  on  your  course  19!