Association for Computational Linguistics, profile picture
Uploaded byAssociation for Computational Linguistics
PDF, PPTX357 views

Philippe Langlais - 2017 - Users and Data: The Two Neglected Children of Bilingual Natural Language Processing Research

The document discusses various approaches to identifying parallel material in comparable corpora, including identifying parallel documents, sentence pairs, and word pairs. It describes methods such as Paradocs for identifying parallel documents using light document representations and classification. It also discusses the MUNT approach for identifying parallel sentence pairs using alignment-based features and pre-filtering, and evaluates MUNT on Wikipedia data, finding that it identifies a high percentage of parallel and quasi-parallel sentences. Finally, it describes the BUCC 2017 shared task on detecting parallel sentences in Wikipedia data.

Embed presentation

Download as PDF, PPTX
NLP Research Today … Models, models, models, models (preferably deep ones) 1felipe@BUCC 2017
But User Matters too • Not much reflected in current evaluation protocols • e.g. how useful is a model that can identify the translation of frequent words at an accuracy of 40% ? • Smart evaluations do exist though • (Sharoff et al., 2006) for translating difficult MWEs 2felipe@BUCC 2017
3felipe@BUCC 2017
TransSearch •  Users are tolerant to alignment problems •  We implemented a detector of erroneous alignments but decided not to integrate it •  We tested a number of alignment models but used one of the simplest (IBM model 2 with contiguity constraint) •  Users care more about translation diversity in the top candidate list •  so clustering translations is of better use for a user •  See (Bourdaillet et al., 2010) 4felipe@BUCC 2017
5felipe@BUCC 2017
TransType •  Keystroke saving is a poor measure of the usefulness of the tool •  often, users do not watch the screen… (Langlais et al., 2002) •  Cognitive load for reading completions •  predicting when a user will find a completion useful is a plus (Foster et al., 2002) •  Users do not like repetitive errors •  adapting online is important (Nepveu et al., 2004) See (Gonzales-Rubio et al., 2012) for advances in targeted mediated interactive MT 6felipe@BUCC 2017
And Data Matters too • Often the main issue • once you have data, use your ``best hammer’’ • e.g. • domain specific MT • organizing a dataflow • Know-how in handling Users and Data is in need in practical settings • but not much rewarded academically 7felipe@BUCC 2017
Of course … • Better models are important ! • e.g. NMT (Sutskever et al., 2014; Cho et al., 2014; Bahdanau et al., 2014) • simpler (single model) • better results • (Bentivogli et al., 2016; Isabelle et al.; 2017) • good properties for further improvements • multilingual • continuously learning • multi-tasking, etc. • And have the potential to impact users 8felipe@BUCC 2017
Plan   Data Management (Acquisition and Organization) •  What has been done •  What (I feel) is still missing •  Domain specific bilingual parallel and comparable corpora •  Assessing the quality/usefulness of a pair of documents   Parallel Material Extraction from a Comparable Corpus •  Document pairs •  Sentence pairs •  Word pairs 9felipe@BUCC 2017
Data Management Overview Web felipe@BUCC 2017 10 crawling dispatching parallel quasi- comparable comparable sentence alignment parallel pair extraction parallel fragment extraction parallel material update Input •  seed terms/urls •  documents of interest (mono or bilingual) Objective •  e.g. improving a generic SMT engine
Data Management: What has been done •  Web2ParallelDoc/Sent •  BITS (Ma & Liberman, 1999) •  a sensible pipeline, leveraging lexicons, cognates and a list of webdomains •  PTMiner (Chen & Nie, 2000), STRAND (Resnik & Smith, 2003) •  rule-based URL matching, HTML-tag and lexicon-based pairing •  (Nadeau & Foster, 2004; Fry 2005) •  specific to newswire feeds •  WebMining (Tomas et al., 2005) •  pipeline which only requires a list of bilingual websites (dictionaries are trained online) •  (Fukushima, et al., 2006) •  leveraging a dictionary2graph algorithm, and paying attention to speed issues •  BABYLON (Mohler and Mihalcea, 2008) •  low-density language oriented, a pipeline which input is a source document •  BITEXTOR (Espla-Gomis et Forcada, 2010) •  pipeline of URL- and content-based matching •  See (Kulkarni, 2012) for a survey 11felipe@BUCC 2017
Data Management: What has been done •  Notable Large-scale Efforts •  (Callisson-Burch et al., 2009) •  10^9 word parallel corpus crawled from specific URLs, URL-based pairing, sentence alignment (Moore 2002) + cleaning: 105M sentence pairs •  (Uszkoreit et al., 2010) •  parallel document detection using MT and near-duplicate detection •  2.5B webpages crawled from the Web + 1.5M public-domain books •  24 hours on a 2000-node cluster •  (Ture & Lin, 2012) •  English-German parallel sentences extracted from Wikipedia tackling the cross-lingual pairwise similarity problem without heuristics •  (Smith et al., 2013) •  Open-source extension of STRAND, impressive deployment on the CommonCrawl corpus (32TB), using Amazon’s elastic map-reduce 12felipe@BUCC 2017
Data Management: Issues •  No systematic comparison of those systems •  Not much sensibility to specific domains •  Crawling •  which input? manual set of webdomains / urls / terms ? •  iterative or not ? •  concerns: efficiency, coverage, resources expected •  Dispatching •  grade a pair of documents (Li and Gaussier, 2010; Fung and Cheung, 2004; Babych and Hartley, 2014) •  measure of the usefulness of a resource (Barker and Gaizauskas, 2012) •  Bootstrapping •  model/lexicon update (Fung and Cheung, 2004) felipe@BUCC 2017 13
• Experiments in those directions would lead to : • a better know-how •  valuable in practical (industry) settings • a shared repository of domain specific collections •  as OPUS for parallel texts (Tiedmann, 2012) • (potentially) a reference bilingual collection where all parallel units are marked •  would help measuring progress •  ease reproducibility felipe@BUCC 2017 14 Data Management: Issues
About Domain Specific CC •  Domain specific crawling (monolingual) •  WebBootCat (Baroni and Bernardini, 2004) •  TerminoWeb (Barrière and Agbago, 2006) •  Babouk (De Groc, 2011) •  (Azoulay, 2017) considering only pdf files •  Specialized (bilingual) comparable corpora acquisition •  See the ACCURAT project (Pinnis et al., 2012; Aker et al., 2012) •  METRICC (Alonso et al., 2012) •  Not just a matter of quantity (Morin et al., 2007) 15felipe@BUCC 2017
Plan   Data Management (Acquisition and Organization) •  What has been done •  What (I feel) is still missing •  Domain specific bilingual parallel and comparable corpora •  Assessing the quality/usefulness of a pair of documents   Parallel Material Extraction from a Comparable Corpus •  Document pairs •  Sentence pairs •  Word pairs 16felipe@BUCC 2017
Identifying parallel documents •  Has received some attention (see previous slides) •  (Enright and Kondrak, 2010) •  nb of shared hapax words •  (Uszkoreit et al., 2010) •  machine translation + monolingual duplicates detection •  (Patry and Langlais, 2011) •  light doc. representation + IR + classifier •  Often just one component of a pipeline which is not evaluated as such 17felipe@BUCC 2017
Paradocs (Patry and Langlais, 2011) •  Light way of representing a document: •  sequence of numerical entities •  sequence of hapax words (of at least 4 symbols) •  sequence of punctuations . ! ? ( ) : as in (Nadeau and Foster 2004) •  Avoid Cartesian product thanks to IR •  no need for a bilingual lexicon •  Train a classifier to recognize parallel documents •  normalized edit-distance between each sequence representation •  count of each entity (numerals, punctuations, hapaxes) 18felipe@BUCC 2017
Paradocs: Avoiding the Cartesian product Europarl setting - 6000 bitexts per language pair - Dutch2English §  nodoc: no document returned by Lucene §  nogood: no good document returned (n=20) Ø  >40% of failure for very short documents (<10 sentences) Ø  15% for long documents (>64 sentences) 19felipe@BUCC 2017
WMT 2016 Shared Task on Bilingual Document Alignment •  Goal: better understanding best practices •  11 teams, 21 systems submitted (+ 1 baseline) •  Easy entrance task: •  train: 49 webdomains crawled (ex: virtualhospice.ca) •  only HTML pages considered •  text pre-extracted, duplicates removed, language pre-identified (en-fr), machine translation provided •  test: 203 other webdomains •  evaluation: •  retrieving 2400 pairs of documents known parallel (within webdomains) •  strict rule: a target document should be proposed at most once 20felipe@BUCC 2017
RALI’s participation (Jakubina and Langlais, 2016) •  motivation: simplicity ! •  no use of MT, eventually even no lexicon-based translation •  an IR approach (Lucene-based) comparing/combining •  IR (monolingual) •  CLIR (based on a bilingual lexicon for « translating ») •  a simple but efficient URL-based IR (tokenizing URLs) •  badLuc ended up with a recall of 79.3% •  the best system recorded 95% •  the organizers’ baseline 59.8% 21felipe@BUCC 2017
Results on train Strategy @1 text monolingual default 6.4 default+tok 35.4 best (w/o length) 64.9 best (w length) 76.6 text bilingual best (w length) 83.3 url baseline WMT16 67.9 best 80.1 badLuc w/o post-treat 88.6 w post-treat 92.1 •  Playing with metaparameters helps a lot •  Applying a length filter also helps a lot •  Involving translation is a must •  even with a simple lexicon (which covers ~half of the words in the collection) •  Our URL variant is performing impressively well •  outperforms the baseline •  useful on short documents •  Combining both indexes (text and urls) helps •  Post-filtering is a plus 22felipe@BUCC 2017
Plan   Data Management (Acquisition and Organization) •  What has been done •  What (I feel) is still missing •  Domain specific bilingual parallel and comparable corpora •  Assessing the quality/usefulness of a pair of documents   Parallel Material Extraction from a Comparable Corpus •  Document pairs •  Sentence pairs •  Word pairs 23felipe@BUCC 2017
MUNT (Bérard, 2014) A reimplementation of (Munteanu and Marcu, 2005) with features borrowed from (Smith et al., 2010) 24felipe@BUCC 2017
MUNT (Bérard, 2014) • classifier (logistic regression) •  31 features •  length based, alignment-based features, fertility, etc. • pre-filter (selecting interesting sentence pairs) •  ratio of sentence length no greater than 2 •  at least 50% of tokens with alignment in the other side •  removes ~98% of the Cartesian product !! 25felipe@BUCC 2017
MUNT on Wikipedia •  Done before (Smith et al., 2010) •  Wikipedia dump (2014) felipe@BUCC 2017 26 #articles #paired #sent. #tokens en 4.5M 919k 29.3M 630.8M fr 1.5M 919k 16.8M 354.5M •  Configuration (default) -  lexicon trained with GIZA++ on 100k sentence pairs of Europarl -  classifier trained on 800 positive and 800 negative examples of news data, threeshold: 0.8
MUNT on Wikipedia felipe@BUCC 2017 27 Manually evaluated 500 (random) sentence pairs -  parallel -  quasi-parallel (at least partial) -  not parallel At the cellular level, the nervous system is defined by the presence of a special type of cell, called the neuron, also known as a ”nerve cell” . À l’échelle cellulaire, le système nerveux est défini par la présence de cellules hautement spécialisées appelées neurones, qui ont la capacité, très particulière, de véhiculer un signal électrochimique .
MUNT on Wikipedia felipe@BUCC 2017 28 grade % parallel 71 quasi-parallel 15 not parallel 14 We tried something odd: •  We applied Yasa (Lamraoui and Langlais, 2013) on Wikipedia article pairs (pretending they were parallel) •  Asked MUNT to classify the sentence pairs identified grade % parallel 48 quasi-parallel 16 not parallel 36 -  11M sentence pairs identified by Yasa -  1.6M kept parallel by MUNT -  1.3M once duplicate removed -  86% of sentence pairs are (quasi-)parallel -  much faster ! -  15 hours (on a cluster of 8 nodes) -  MUNT detected 2.61M sentence pairs -  2.26M once duplicates removed -  1.92M after removing sentences shorter than 4 words -  64% of sentence pairs are (quasi-)parallel
Munt on Wikipedia felipe@BUCC 2017 29 •  50 comparable Wiki article pairs manually aligned at the sentence level (Rebout and Langlais, 2014) •  measured the performance of YASA and MUNT on those articles •  MUNT has a better precision, but a lower recall npara # parallel sentences nfr (nen) # sentences in fr (en) doc.
BUCC 2017 Shared Task (Zweigenbaum et al., 2016) •  Detecting parallel sentences in a large text collection •  2 sets of monolingual Wikipedia sentences (2014 dumps): •  1.4M French sentences •  1.9M English sentences •  + 17k parallel sentences from News Commentary (v9) •  Evaluation: precision, recall, F1 •  pros •  no metadata (text-based) •  Cartesian product is large (with few positive examples) •  Smartness in inserting parallel sentences (to avoid simple solutions) •  cons •  Artificial task •  True parallel sentences in Wikipedia EN-FR are not known 30felipe@BUCC 2017
RALI’s participation 31felipe@BUCC 2017 Will be presented this afternoon (Grégoire and Langlais, 2017)
RALI’s participation •  Training: Europarl v7 French-English •  first 500K sentence pairs •  negative sampling: random selection of sentence pairs •  Test: newstest2012 (out-domain) •  1000 first sentence pairs + noise •  Pre-processing •  maximum sentence length: 80 •  tokenization with Moses’ toolkit, lowercased •  mapping digits to 0 (e.g. 1982 -> 0000) 32felipe@BUCC 2017
RALI’s participation •  Embedding-based filter for avoiding the Cartesian product •  word-based embeddings computed with BILBOWA (Gouws, 2015) •  sentence representation = average word embeddings •  40 first target sentences for each source one •  Not paying attention to ``details’’ (only model matters, right?) •  digit preprocessing •  random negative sampling at training does not match testing condition •  The next slides summarize what we have learnt after our participation to BUCC 2017 (new material) model precision recall F1 12.1 70.9 20.7 official 33felipe@BUCC 2017
Influence of the decision threeshold •  Cartesian product •  1M ex., 1k positive •  BIRNN trained with 7 negative ex. •  MUNT trained with a balanced corpus Precision Recall F1 ρ BiRNN 83.0 69.6 75.7 0.99 MUNT 31.8 24.1 27.7 0.99 34felipe@BUCC 2017
Influence of the decision threeshold •  Pre-filtering •  8053 ex., 1k positive •  BIRNN trained with 7 negative ex. •  MUNT trained with a balanced corpus Precision Recall F1 ρ BiRNN 91.0 62.4 74.0 0.97 MUNT 73.3 57.0 64.1 0.91 35felipe@BUCC 2017
Influence of Post-filtering •  Each decision taken independently •  A src sent. may be associated to several target ones, and vice versa ²  post-filtering (greedy algorithm, Hungarian algo. too slow) huge boost in precision at a small recall loss for both approaches 36felipe@BUCC 2017
Plan   Data Management (Acquisition and Organization) •  What has been done •  What (I feel) is still missing •  Domain specific bilingual parallel and comparable corpora •  Assessing the quality/usefulness of a pair of documents   Parallel Material Extraction from a Comparable Corpus •  Document pairs •  Sentence pairs •  Word pairs 37felipe@BUCC 2017
Bilingual Lexicon Induction •  Received a lot of attention •  Pioneering works: (Rapp, 1995; Fung, 1995) •  See (Sharoff et al., 2013) •  Revisited as a way to measure the quality of word embeddings •  Seminal work of (Mikolov et al., 2013) •  Comprehensive comparisons •  (Levy et al., 2014, 2017; Upadhyay et al., 2016) 38felipe@BUCC 2017
Bilingual Lexicon Induction (Jakubina & Langlais; 2016) •  We thoroughly revisited those approaches: •  (Rapp, 1995) •  (Mikolov et al. 2013) •  training the projection matrix with the toolkit of (Dinu and Baroni, 2015) •  (Faruqui and Dyer, 2014) •  and a few others, but without success •  investigating their meta-parameters •  paying attention to the frequency of the terms •  after (Pekar et al, 2006) •  showing their complementarity 39felipe@BUCC 2017
Experiments •  Wikipedia (dumps of June 2013) •  EN: 7.3M token forms (1.2G tokens) •  FR: 3.6M token forms (330M tokens) •  Test sets •  Wiki≤25 English words occurring at most 25 times in Wiki-EN •  6.8M such tokens (92%) •  Wiki>25 English words seen more than 25 times in Wiki-EN •  Euro-5-6k top frequent 5000 to 6000 words of WMT2011 Frequency min max avg cov (%) Wiki≤25 1 25 10 100.0 Wiki>25 27 19.4k 2.8k 100.0 Euro5-6k 1 2.6M 33.6k 87.3 40felipe@BUCC 2017
Metaparameters explored •  Rapp •  window size (3,7,15,31) •  association measure (LLR, discontinuous odd-ratios) •  projecting UNK words (yes, no) •  similarity measure (fixed, cosine similarity) •  seed lexicon (fixed, in-house 107k entries) •  Word2Vec •  skip-gram versus continuous bag-of-word •  negative sampling (5 or 10) versus hierarchical softmax •  window size (6,10,20,30) •  dimension (from 50 up to 250 for SKG and 200 for CBOW) •  seed lexicon (2k-low, 5-high, 5k-rand) •  Faruqui & Dyer •  ratio (0.5, 0.8 and 1.0) •  fixed dimension (the best one for Word2Vec) 41felipe@BUCC 2017 No adhoc filtering as usually done, so very time and memory challenging for the Rapp approach (Prochasson and Fung, 2011) this work -  ~20k document pairs ~700k ones -  target voc. 128k words (nouns) 3M words We did apply a few filters: -  context vectors: 1000 top-ranked words -  50k first occurrences of a source term
Best variant (per test set) @1 @5 @20 Wiki>25 (ed@1 19.3) Rapp 20.0 33.0 43.0 Miko 17.0 32.6 41.6 Faru 13.3 26.0 33.3 Wiki≤25 (ed@1 17.6) Rapp 2.6 4.3 7.3 Miko 1.6 4.6 10.6 Faru 1.6 2.6 5.0 Euro5-6k (ed@1 8.0) Rapp 16.6 31.8 41.2 Miko 42.0 59.0 67.8 Faru 30.6 47.7 59.8 42felipe@BUCC 2017
Reranking Candidate Translations (Jakubina and Langlais, 2016) •  Reranking shown useful in a number of settings (Delpech et al., 2012; Harastani et al., 2013; Kontonatsios et al., 2014) •  Trained a reranker (random forest) with RankLib •  700 terms for training, remaining 300 terms for testing •  3-fold cross-validation •  Light features for each pair (s,t): •  frequency-based features •  freq of s,t and their difference •  string-based features •  length of s and t, their difference, their ratio, edit distance(s,t) •  rank-based features •  score and rank of t in the native list •  number of lists in which t appears (when several n-best lists are considered) 43felipe@BUCC 2017
Reranking Individual n-best Lists Individual 1-Reranked @1 @5 @20 @1 @5 @20 Wiki>25 Rapp 20.0 33.0 43.0 36.3 48.8 53.8 Miko 17.0 32.6 41.6 38.1 49.0 54.3 Faru 13.3 26.0 33.3 34.3 44.0 47.9 Wiki≤25 Rapp 2.6 4.3 7.3 8.6 9.4 10.2 Miko 1.6 4.6 10.6 16.6 19.0 20.1 Faru 1.6 2.6 5.0 7.9 8.7 8.9 Euro5-6k Rapp 16.6 31.8 41.2 34.6 48.6 51.9 Miko 42.0 59.0 67.8 47.0 68.1 73.0 Faru 30.6 47.7 59.8 41.2 58.0 66.0 44felipe@BUCC 2017
Reranking several n-best Lists 1-Reranked n-Reranked @1 @5 @20 @1 @5 @20 Wiki>25 Rapp 36.3 48.8 53.8 Miko 38.1 49.0 54.3 R+M 43.3 58.4 62.4 Faru 34.3 44.0 47.9 R+M+F 45.6 59.6 64.0 Wiki≤25 Rapp 8.6 9.4 10.2 Miko 16.6 19.0 20.1 R+M 18.9 22.0 23.6 Faru 7.9 8.7 8.9 R+M+F 21.3 24.4 25.7 Euro5-6k Rapp 34.6 48.6 51.9 Miko 47.0 68.1 73.0 R+M 49.5 68.7 76.1 Faru 41.2 58.0 66.0 R+M+F 47.6 68.5 76.2 45felipe@BUCC 2017
One-Slide Wrap up • Hyp: Model-centric research somehow hides the value of: •  know-how in managing bilingual data (parallel and comparable) •  evaluation protocols involving real users (or proxies) • Better handling data is part of the game • We should learn from users felipe@BUCC 2017 46
Thank you for your attention felipe@BUCC 2017 47

More Related Content

PDF
Approach to leverage Websites to APIs through Semantics
byIoannis Stavrakantonakis
 
PDF
Apply chinese radicals into neural machine translation: deeper than character...
byLifeng (Aaron) Han
 
PDF
Data wrangling week1
byFerdin Joe John Joseph PhD
 
PDF
Data Wrangling Week 4
byFerdin Joe John Joseph PhD
 
PPTX
Searching for the best translation combination
byMatīss ‎‎‎‎‎‎‎  
 
PDF
Data wrangling week 6
byFerdin Joe John Joseph PhD
 
PDF
Data wrangling week 5
byFerdin Joe John Joseph PhD
 
PDF
Data wrangling week3
byFerdin Joe John Joseph PhD
 
Approach to leverage Websites to APIs through Semantics
byIoannis Stavrakantonakis
 
Apply chinese radicals into neural machine translation: deeper than character...
byLifeng (Aaron) Han
 
Data wrangling week1
byFerdin Joe John Joseph PhD
 
Data Wrangling Week 4
byFerdin Joe John Joseph PhD
 
Searching for the best translation combination
byMatīss ‎‎‎‎‎‎‎  
 
Data wrangling week 6
byFerdin Joe John Joseph PhD
 
Data wrangling week 5
byFerdin Joe John Joseph PhD
 
Data wrangling week3
byFerdin Joe John Joseph PhD
 

What's hot

PDF
Data wrangling week2
byFerdin Joe John Joseph PhD
 
PDF
Data Wrangling Week 7
byFerdin Joe John Joseph PhD
 
PPTX
CORE Analytics Dashboard
bypetrknoth
 
PPTX
The Web of Data: do we actually understand what we built?
byFrank van Harmelen
 
PDF
NIPS 2016 Highlights - Sebastian Ruder
bySebastian Ruder
 
PPTX
Tutorial on Question Answering Systems
bySaeedeh Shekarpour
 
PPTX
From TREC to Watson: is open domain question answering a solved problem?
byConstantin Orasan
 
PDF
SSSW 2013 - Feeding Recommender Systems with Linked Open Data
byPolytechnic University of Bari
 
PPTX
Online Index Extraction from Linked Open Data Sources
byFabio Benedetti
 
PDF
Recommender Systems and Linked Open Data
byPolytechnic University of Bari
 
PDF
Tutorial - Recommender systems meet linked open data - ICWE 2016 - Lugano - 0...
byPolytechnic University of Bari
 
PPTX
The future of scholarly communications professionals
byNancy Pontika
 
PPTX
LODeX: Schema Summarization and automatic SPARQL query generation for Linked ...
byFabio Benedetti
 
PDF
Infrastructure crossroads... and the way we walked them in DKPro
byopenminted_eu
 
PPT
IOTA OpenURL Quality @ 2011 UKSG Conference
byRafal Kasprowski
 
PDF
Semantics2018 Zhang,Petrak,Maynard: Adapted TextRank for Term Extraction: A G...
byJohann Petrak
 
Data wrangling week2
byFerdin Joe John Joseph PhD
 
Data Wrangling Week 7
byFerdin Joe John Joseph PhD
 
CORE Analytics Dashboard
bypetrknoth
 
The Web of Data: do we actually understand what we built?
byFrank van Harmelen
 
NIPS 2016 Highlights - Sebastian Ruder
bySebastian Ruder
 
Tutorial on Question Answering Systems
bySaeedeh Shekarpour
 
From TREC to Watson: is open domain question answering a solved problem?
byConstantin Orasan
 
SSSW 2013 - Feeding Recommender Systems with Linked Open Data
byPolytechnic University of Bari
 
Online Index Extraction from Linked Open Data Sources
byFabio Benedetti
 
Recommender Systems and Linked Open Data
byPolytechnic University of Bari
 
Tutorial - Recommender systems meet linked open data - ICWE 2016 - Lugano - 0...
byPolytechnic University of Bari
 
The future of scholarly communications professionals
byNancy Pontika
 
LODeX: Schema Summarization and automatic SPARQL query generation for Linked ...
byFabio Benedetti
 
Infrastructure crossroads... and the way we walked them in DKPro
byopenminted_eu
 
IOTA OpenURL Quality @ 2011 UKSG Conference
byRafal Kasprowski
 
Semantics2018 Zhang,Petrak,Maynard: Adapted TextRank for Term Extraction: A G...
byJohann Petrak
 

Viewers also liked

PDF
Chelo Vargas-Sierra
byChelo Vargas
 
PDF
Extraction of domain-specific bilingual lexicon from comparable corpora: comp...
byEstelle Delpech
 
PPTX
Cross-lingual ontology lexicalisation, translation and information extraction...
byTobias Wunner
 
PDF
Bilingual Terminology Extraction based on Translation Patterns
byAlberto Simões
 
PDF
Embedded Human Computation for Knowledge Extraction and Evaluation
bywebLyzard technology
 
PDF
Challenges in the linguistic exploitation of specialized republishable web co...
byAdrien Barbaresi
 
PDF
Parallel text extraction from multimodal comparable corpora
byHaithem Afli
 
PPTX
Macro economische analyse van brazilië
byJan-Willem Lammens
 
PPTX
A cognitive view of the bilingual lexicon
byİrem Tümer
 
PDF
Bilingual terminology mining
byEstelle Delpech
 
PDF
Michael Bloodgood - 2017 - Acquisition of Translation Lexicons for Historical...
byAssociation for Computational Linguistics
 
PDF
Dealing with Lexicon Acquired from Comparable Corpora: post-edition and exchange
byEstelle Delpech
 
PDF
Meng Zhang - 2017 - Adversarial Training for Unsupervised Bilingual Lexicon I...
byAssociation for Computational Linguistics
 
PDF
Identification of Fertile Translations in Comparable Corpora: a Morpho-Compos...
byEstelle Delpech
 
PDF
Enriching Transliteration Lexicon Using Automatic Transliteration Extraction
bySarvnaz Karimi
 
PDF
Applicative evaluation of bilingual terminologies
byEstelle Delpech
 
PPT
Word Formation in English
byteflang
 
Chelo Vargas-Sierra
byChelo Vargas
 
Extraction of domain-specific bilingual lexicon from comparable corpora: comp...
byEstelle Delpech
 
Cross-lingual ontology lexicalisation, translation and information extraction...
byTobias Wunner
 
Bilingual Terminology Extraction based on Translation Patterns
byAlberto Simões
 
Embedded Human Computation for Knowledge Extraction and Evaluation
bywebLyzard technology
 
Challenges in the linguistic exploitation of specialized republishable web co...
byAdrien Barbaresi
 
Parallel text extraction from multimodal comparable corpora
byHaithem Afli
 
Macro economische analyse van brazilië
byJan-Willem Lammens
 
A cognitive view of the bilingual lexicon
byİrem Tümer
 
Bilingual terminology mining
byEstelle Delpech
 
Michael Bloodgood - 2017 - Acquisition of Translation Lexicons for Historical...
byAssociation for Computational Linguistics
 
Dealing with Lexicon Acquired from Comparable Corpora: post-edition and exchange
byEstelle Delpech
 
Meng Zhang - 2017 - Adversarial Training for Unsupervised Bilingual Lexicon I...
byAssociation for Computational Linguistics
 
Identification of Fertile Translations in Comparable Corpora: a Morpho-Compos...
byEstelle Delpech
 
Enriching Transliteration Lexicon Using Automatic Transliteration Extraction
bySarvnaz Karimi
 
Applicative evaluation of bilingual terminologies
byEstelle Delpech
 
Word Formation in English
byteflang
 

Similar to Philippe Langlais - 2017 - Users and Data: The Two Neglected Children of Bilingual Natural Language Processing Research

PPTX
Natural Language Processing and Search Intent Understanding C3 Conductor 2019...
byDawn Anderson MSc DigM
 
ODP
Text-mining and Automation
bybenosteen
 
PDF
NLP and Knowledge Graphs
byCarlos Badenes-Olmedo
 
PDF
The Standards Mosaic Opening the Way to New Technologies
byDave Lewis
 
PPTX
Query Understanding
byMatt Corkum
 
PPTX
Taming Text
byGrant Ingersoll
 
PPTX
Using topic modelling frameworks for NLP and semantic search
byDawn Anderson MSc DigM
 
PDF
ER1 Eduard Barbu - EXPERT Summer School - Malaga 2015
byRIILP
 
PPTX
Mining Web content for Enhanced Search
byRoi Blanco
 
PDF
NetIKX Semantic Search Presentation
byurvics
 
PPTX
Beyond document retrieval using semantic annotations
byRoi Blanco
 
PDF
Martin Voigt | Streaming-based Text Mining using Deep Learning and Semantics
bysemanticsconference
 
PDF
Streaming-based Text Mining using Deep Learning and Semantics
byLinked Enterprise Date Services
 
PDF
Ontos NLP Stack, Sep. 2016
byMartin Voigt
 
PPT
The impact of standardized terminologies and domain-ontologies in multilingua...
byAIMS (Agricultural Information Management Standards)
 
PDF
Machine learning-and-data-mining-19-mining-text-and-web-data
byitstuff
 
PDF
Aspects of NLP Practice
byVsevolod Dyomkin
 
PPTX
Deep Learning Automated Helpdesk
byPranav Sharma
 
PPTX
CoLing 2016
byMatīss ‎‎‎‎‎‎‎  
 
PPT
Text Analytics: Yesterday, Today and Tomorrow
byTony Russell-Rose
 
Natural Language Processing and Search Intent Understanding C3 Conductor 2019...
byDawn Anderson MSc DigM
 
Text-mining and Automation
bybenosteen
 
NLP and Knowledge Graphs
byCarlos Badenes-Olmedo
 
The Standards Mosaic Opening the Way to New Technologies
byDave Lewis
 
Query Understanding
byMatt Corkum
 
Taming Text
byGrant Ingersoll
 
Using topic modelling frameworks for NLP and semantic search
byDawn Anderson MSc DigM
 
ER1 Eduard Barbu - EXPERT Summer School - Malaga 2015
byRIILP
 
Mining Web content for Enhanced Search
byRoi Blanco
 
NetIKX Semantic Search Presentation
byurvics
 
Beyond document retrieval using semantic annotations
byRoi Blanco
 
Martin Voigt | Streaming-based Text Mining using Deep Learning and Semantics
bysemanticsconference
 
Streaming-based Text Mining using Deep Learning and Semantics
byLinked Enterprise Date Services
 
Ontos NLP Stack, Sep. 2016
byMartin Voigt
 
The impact of standardized terminologies and domain-ontologies in multilingua...
byAIMS (Agricultural Information Management Standards)
 
Machine learning-and-data-mining-19-mining-text-and-web-data
byitstuff
 
Aspects of NLP Practice
byVsevolod Dyomkin
 
Deep Learning Automated Helpdesk
byPranav Sharma
 
CoLing 2016
byMatīss ‎‎‎‎‎‎‎  
 
Text Analytics: Yesterday, Today and Tomorrow
byTony Russell-Rose
 

More from Association for Computational Linguistics

PDF
Muis - 2016 - Weak Semi-Markov CRFs for NP Chunking in Informal Text
byAssociation for Computational Linguistics
 
PDF
Castro - 2018 - A High Coverage Method for Automatic False Friends Detection ...
byAssociation for Computational Linguistics
 
PDF
Castro - 2018 - A Crowd-Annotated Spanish Corpus for Humour Analysis
byAssociation for Computational Linguistics
 
PDF
Muthu Kumar Chandrasekaran - 2018 - Countering Position Bias in Instructor In...
byAssociation for Computational Linguistics
 
PDF
Daniel Gildea - 2018 - The ACL Anthology: Current State and Future Directions
byAssociation for Computational Linguistics
 
PDF
Elior Sulem - 2018 - Semantic Structural Evaluation for Text Simplification
byAssociation for Computational Linguistics
 
PDF
Daniel Gildea - 2018 - The ACL Anthology: Current State and Future Directions
byAssociation for Computational Linguistics
 
PDF
Wenqiang Lei - 2018 - Sequicity: Simplifying Task-oriented Dialogue Systems w...
byAssociation for Computational Linguistics
 
PDF
Matthew Marge - 2017 - Exploring Variation of Natural Human Commands to a Rob...
byAssociation for Computational Linguistics
 
PDF
Venkatesh Duppada - 2017 - SeerNet at EmoInt-2017: Tweet Emotion Intensity Es...
byAssociation for Computational Linguistics
 
PDF
Satoshi Sonoh - 2015 - Toshiba MT System Description for the WAT2015 Workshop
byAssociation for Computational Linguistics
 
PDF
Chenchen Ding - 2015 - NICT at WAT 2015
byAssociation for Computational Linguistics
 
PDF
John Richardson - 2015 - KyotoEBMT System Description for the 2nd Workshop on...
byAssociation for Computational Linguistics
 
PDF
John Richardson - 2015 - KyotoEBMT System Description for the 2nd Workshop on...
byAssociation for Computational Linguistics
 
PDF
Zhongyuan Zhu - 2015 - Evaluating Neural Machine Translation in English-Japan...
byAssociation for Computational Linguistics
 
PDF
Zhongyuan Zhu - 2015 - Evaluating Neural Machine Translation in English-Japan...
byAssociation for Computational Linguistics
 
PDF
Hyoung-Gyu Lee - 2015 - NAVER Machine Translation System for WAT 2015
byAssociation for Computational Linguistics
 
PDF
Satoshi Sonoh - 2015 - Toshiba MT System Description for the WAT2015 Workshop
byAssociation for Computational Linguistics
 
PDF
Chenchen Ding - 2015 - NICT at WAT 2015
byAssociation for Computational Linguistics
 
PDF
Graham Neubig - 2015 - Neural Reranking Improves Subjective Quality of Machin...
byAssociation for Computational Linguistics
 
Muis - 2016 - Weak Semi-Markov CRFs for NP Chunking in Informal Text
byAssociation for Computational Linguistics
 
Castro - 2018 - A High Coverage Method for Automatic False Friends Detection ...
byAssociation for Computational Linguistics
 
Castro - 2018 - A Crowd-Annotated Spanish Corpus for Humour Analysis
byAssociation for Computational Linguistics
 
Muthu Kumar Chandrasekaran - 2018 - Countering Position Bias in Instructor In...
byAssociation for Computational Linguistics
 
Daniel Gildea - 2018 - The ACL Anthology: Current State and Future Directions
byAssociation for Computational Linguistics
 
Elior Sulem - 2018 - Semantic Structural Evaluation for Text Simplification
byAssociation for Computational Linguistics
 
Daniel Gildea - 2018 - The ACL Anthology: Current State and Future Directions
byAssociation for Computational Linguistics
 
Wenqiang Lei - 2018 - Sequicity: Simplifying Task-oriented Dialogue Systems w...
byAssociation for Computational Linguistics
 
Matthew Marge - 2017 - Exploring Variation of Natural Human Commands to a Rob...
byAssociation for Computational Linguistics
 
Venkatesh Duppada - 2017 - SeerNet at EmoInt-2017: Tweet Emotion Intensity Es...
byAssociation for Computational Linguistics
 
Satoshi Sonoh - 2015 - Toshiba MT System Description for the WAT2015 Workshop
byAssociation for Computational Linguistics
 
Chenchen Ding - 2015 - NICT at WAT 2015
byAssociation for Computational Linguistics
 
John Richardson - 2015 - KyotoEBMT System Description for the 2nd Workshop on...
byAssociation for Computational Linguistics
 
John Richardson - 2015 - KyotoEBMT System Description for the 2nd Workshop on...
byAssociation for Computational Linguistics
 
Zhongyuan Zhu - 2015 - Evaluating Neural Machine Translation in English-Japan...
byAssociation for Computational Linguistics
 
Zhongyuan Zhu - 2015 - Evaluating Neural Machine Translation in English-Japan...
byAssociation for Computational Linguistics
 
Hyoung-Gyu Lee - 2015 - NAVER Machine Translation System for WAT 2015
byAssociation for Computational Linguistics
 
Satoshi Sonoh - 2015 - Toshiba MT System Description for the WAT2015 Workshop
byAssociation for Computational Linguistics
 
Chenchen Ding - 2015 - NICT at WAT 2015
byAssociation for Computational Linguistics
 
Graham Neubig - 2015 - Neural Reranking Improves Subjective Quality of Machin...
byAssociation for Computational Linguistics
 

Recently uploaded

PDF
Unit-III pdf (Basic listening Skill, Effective Writing Communication & Writin...
bySayyadTarannum
 
PPTX
Pig- piggy bank in Big Data Analytics.ppt.pptx
byVarshini M
 
PPTX
Semester 6 unit 2 Atopic dermatitis.pptx
bysachin7989
 
PDF
NAVIGATE PHARMACY CAREER OPPORTUNITIES.pdf
byMd. Zakaria Faruki
 
PDF
Projecte de la porta d'i5B: Els animals marins
byeduardrubio1
 
PDF
Scalable-MADDPG-Based Cooperative Target Invasion for a Multi-USV System.pdf
byMan_Ebook
 
DOCX
Mobile applications Devlopment ReTest year 2025-2026
byDr. Mazin Mohamed alkathiri
 
PPTX
Pain. definition, causes, factor influencing pain & pain assessment.pptx
byPompi Begum
 
PDF
Digital Wellness in University Communities: Libraries as Guardians of Healthy...
bySylvester Ebhonu
 
PPTX
How to use search_read method in Odoo 18
byCeline George
 
PPTX
How to Manage Package Reservation in Odoo 18 Inventory
byCeline George
 
PPTX
How to Manage Line Discounts in Odoo 18 POS
byCeline George
 
PDF
Models of Teaching - TNTEU - B.Ed I Semester - Teaching and Learning - BD1TL ...
byDr Raja Mohammed T
 
PDF
FAMILY ASSESSMENT FORMAT - CHN practical
byDr. Sudhadevi Sadanandan
 
PDF
Projecte de la porta de primer B: L'antic Egipte
byeduardrubio1
 
PPTX
York "Collaboration for Research Support at U-M Library"
byNational Information Standards Organization (NISO)
 
PPTX
AI_in_Daily_Life_Presentation and more.pptx
bybantydansena
 
PPTX
10-12-2025 Francois Staring How can Researchers and Initial Teacher Educators...
byEduSkills OECD
 
PPTX
How to Configure Push & Pull Rule in Odoo 18 Inventory
byCeline George
 
PDF
Current Electricity for first year physiotherapy
byGanesh Jadhav
 
Unit-III pdf (Basic listening Skill, Effective Writing Communication & Writin...
bySayyadTarannum
 
Pig- piggy bank in Big Data Analytics.ppt.pptx
byVarshini M
 
Semester 6 unit 2 Atopic dermatitis.pptx
bysachin7989
 
NAVIGATE PHARMACY CAREER OPPORTUNITIES.pdf
byMd. Zakaria Faruki
 
Projecte de la porta d'i5B: Els animals marins
byeduardrubio1
 
Scalable-MADDPG-Based Cooperative Target Invasion for a Multi-USV System.pdf
byMan_Ebook
 
Mobile applications Devlopment ReTest year 2025-2026
byDr. Mazin Mohamed alkathiri
 
Pain. definition, causes, factor influencing pain & pain assessment.pptx
byPompi Begum
 
Digital Wellness in University Communities: Libraries as Guardians of Healthy...
bySylvester Ebhonu
 
How to use search_read method in Odoo 18
byCeline George
 
How to Manage Package Reservation in Odoo 18 Inventory
byCeline George
 
How to Manage Line Discounts in Odoo 18 POS
byCeline George
 
Models of Teaching - TNTEU - B.Ed I Semester - Teaching and Learning - BD1TL ...
byDr Raja Mohammed T
 
FAMILY ASSESSMENT FORMAT - CHN practical
byDr. Sudhadevi Sadanandan
 
Projecte de la porta de primer B: L'antic Egipte
byeduardrubio1
 
York "Collaboration for Research Support at U-M Library"
byNational Information Standards Organization (NISO)
 
AI_in_Daily_Life_Presentation and more.pptx
bybantydansena
 
10-12-2025 Francois Staring How can Researchers and Initial Teacher Educators...
byEduSkills OECD
 
How to Configure Push & Pull Rule in Odoo 18 Inventory
byCeline George
 
Current Electricity for first year physiotherapy
byGanesh Jadhav
 

Philippe Langlais - 2017 - Users and Data: The Two Neglected Children of Bilingual Natural Language Processing Research

  • 2.
    NLP Research Today… Models, models, models, models (preferably deep ones) 1felipe@BUCC 2017
  • 3.
    But User Matterstoo • Not much reflected in current evaluation protocols • e.g. how useful is a model that can identify the translation of frequent words at an accuracy of 40% ? • Smart evaluations do exist though • (Sharoff et al., 2006) for translating difficult MWEs 2felipe@BUCC 2017
  • 4.
  • 5.
    TransSearch •  Users aretolerant to alignment problems •  We implemented a detector of erroneous alignments but decided not to integrate it •  We tested a number of alignment models but used one of the simplest (IBM model 2 with contiguity constraint) •  Users care more about translation diversity in the top candidate list •  so clustering translations is of better use for a user •  See (Bourdaillet et al., 2010) 4felipe@BUCC 2017
  • 6.
  • 7.
    TransType •  Keystroke savingis a poor measure of the usefulness of the tool •  often, users do not watch the screen… (Langlais et al., 2002) •  Cognitive load for reading completions •  predicting when a user will find a completion useful is a plus (Foster et al., 2002) •  Users do not like repetitive errors •  adapting online is important (Nepveu et al., 2004) See (Gonzales-Rubio et al., 2012) for advances in targeted mediated interactive MT 6felipe@BUCC 2017
  • 8.
    And Data Matterstoo • Often the main issue • once you have data, use your ``best hammer’’ • e.g. • domain specific MT • organizing a dataflow • Know-how in handling Users and Data is in need in practical settings • but not much rewarded academically 7felipe@BUCC 2017
  • 9.
    Of course … • Bettermodels are important ! • e.g. NMT (Sutskever et al., 2014; Cho et al., 2014; Bahdanau et al., 2014) • simpler (single model) • better results • (Bentivogli et al., 2016; Isabelle et al.; 2017) • good properties for further improvements • multilingual • continuously learning • multi-tasking, etc. • And have the potential to impact users 8felipe@BUCC 2017
  • 10.
    Plan   Data Management(Acquisition and Organization) •  What has been done •  What (I feel) is still missing •  Domain specific bilingual parallel and comparable corpora •  Assessing the quality/usefulness of a pair of documents   Parallel Material Extraction from a Comparable Corpus •  Document pairs •  Sentence pairs •  Word pairs 9felipe@BUCC 2017
  • 11.
    Data Management Overview Web felipe@BUCC2017 10 crawling dispatching parallel quasi- comparable comparable sentence alignment parallel pair extraction parallel fragment extraction parallel material update Input •  seed terms/urls •  documents of interest (mono or bilingual) Objective •  e.g. improving a generic SMT engine
  • 12.
    Data Management: Whathas been done •  Web2ParallelDoc/Sent •  BITS (Ma & Liberman, 1999) •  a sensible pipeline, leveraging lexicons, cognates and a list of webdomains •  PTMiner (Chen & Nie, 2000), STRAND (Resnik & Smith, 2003) •  rule-based URL matching, HTML-tag and lexicon-based pairing •  (Nadeau & Foster, 2004; Fry 2005) •  specific to newswire feeds •  WebMining (Tomas et al., 2005) •  pipeline which only requires a list of bilingual websites (dictionaries are trained online) •  (Fukushima, et al., 2006) •  leveraging a dictionary2graph algorithm, and paying attention to speed issues •  BABYLON (Mohler and Mihalcea, 2008) •  low-density language oriented, a pipeline which input is a source document •  BITEXTOR (Espla-Gomis et Forcada, 2010) •  pipeline of URL- and content-based matching •  See (Kulkarni, 2012) for a survey 11felipe@BUCC 2017
  • 13.
    Data Management: Whathas been done •  Notable Large-scale Efforts •  (Callisson-Burch et al., 2009) •  10^9 word parallel corpus crawled from specific URLs, URL-based pairing, sentence alignment (Moore 2002) + cleaning: 105M sentence pairs •  (Uszkoreit et al., 2010) •  parallel document detection using MT and near-duplicate detection •  2.5B webpages crawled from the Web + 1.5M public-domain books •  24 hours on a 2000-node cluster •  (Ture & Lin, 2012) •  English-German parallel sentences extracted from Wikipedia tackling the cross-lingual pairwise similarity problem without heuristics •  (Smith et al., 2013) •  Open-source extension of STRAND, impressive deployment on the CommonCrawl corpus (32TB), using Amazon’s elastic map-reduce 12felipe@BUCC 2017
  • 14.
    Data Management: Issues • No systematic comparison of those systems •  Not much sensibility to specific domains •  Crawling •  which input? manual set of webdomains / urls / terms ? •  iterative or not ? •  concerns: efficiency, coverage, resources expected •  Dispatching •  grade a pair of documents (Li and Gaussier, 2010; Fung and Cheung, 2004; Babych and Hartley, 2014) •  measure of the usefulness of a resource (Barker and Gaizauskas, 2012) •  Bootstrapping •  model/lexicon update (Fung and Cheung, 2004) felipe@BUCC 2017 13
  • 15.
    • Experiments in thosedirections would lead to : • a better know-how •  valuable in practical (industry) settings • a shared repository of domain specific collections •  as OPUS for parallel texts (Tiedmann, 2012) • (potentially) a reference bilingual collection where all parallel units are marked •  would help measuring progress •  ease reproducibility felipe@BUCC 2017 14 Data Management: Issues
  • 16.
    About Domain SpecificCC •  Domain specific crawling (monolingual) •  WebBootCat (Baroni and Bernardini, 2004) •  TerminoWeb (Barrière and Agbago, 2006) •  Babouk (De Groc, 2011) •  (Azoulay, 2017) considering only pdf files •  Specialized (bilingual) comparable corpora acquisition •  See the ACCURAT project (Pinnis et al., 2012; Aker et al., 2012) •  METRICC (Alonso et al., 2012) •  Not just a matter of quantity (Morin et al., 2007) 15felipe@BUCC 2017
  • 17.
    Plan   Data Management(Acquisition and Organization) •  What has been done •  What (I feel) is still missing •  Domain specific bilingual parallel and comparable corpora •  Assessing the quality/usefulness of a pair of documents   Parallel Material Extraction from a Comparable Corpus •  Document pairs •  Sentence pairs •  Word pairs 16felipe@BUCC 2017
  • 18.
    Identifying parallel documents • Has received some attention (see previous slides) •  (Enright and Kondrak, 2010) •  nb of shared hapax words •  (Uszkoreit et al., 2010) •  machine translation + monolingual duplicates detection •  (Patry and Langlais, 2011) •  light doc. representation + IR + classifier •  Often just one component of a pipeline which is not evaluated as such 17felipe@BUCC 2017
  • 19.
    Paradocs (Patry andLanglais, 2011) •  Light way of representing a document: •  sequence of numerical entities •  sequence of hapax words (of at least 4 symbols) •  sequence of punctuations . ! ? ( ) : as in (Nadeau and Foster 2004) •  Avoid Cartesian product thanks to IR •  no need for a bilingual lexicon •  Train a classifier to recognize parallel documents •  normalized edit-distance between each sequence representation •  count of each entity (numerals, punctuations, hapaxes) 18felipe@BUCC 2017
  • 20.
    Paradocs: Avoiding theCartesian product Europarl setting - 6000 bitexts per language pair - Dutch2English §  nodoc: no document returned by Lucene §  nogood: no good document returned (n=20) Ø  >40% of failure for very short documents (<10 sentences) Ø  15% for long documents (>64 sentences) 19felipe@BUCC 2017
  • 21.
    WMT 2016 SharedTask on Bilingual Document Alignment •  Goal: better understanding best practices •  11 teams, 21 systems submitted (+ 1 baseline) •  Easy entrance task: •  train: 49 webdomains crawled (ex: virtualhospice.ca) •  only HTML pages considered •  text pre-extracted, duplicates removed, language pre-identified (en-fr), machine translation provided •  test: 203 other webdomains •  evaluation: •  retrieving 2400 pairs of documents known parallel (within webdomains) •  strict rule: a target document should be proposed at most once 20felipe@BUCC 2017
  • 22.
    RALI’s participation (Jakubina andLanglais, 2016) •  motivation: simplicity ! •  no use of MT, eventually even no lexicon-based translation •  an IR approach (Lucene-based) comparing/combining •  IR (monolingual) •  CLIR (based on a bilingual lexicon for « translating ») •  a simple but efficient URL-based IR (tokenizing URLs) •  badLuc ended up with a recall of 79.3% •  the best system recorded 95% •  the organizers’ baseline 59.8% 21felipe@BUCC 2017
  • 23.
    Results on train Strategy@1 text monolingual default 6.4 default+tok 35.4 best (w/o length) 64.9 best (w length) 76.6 text bilingual best (w length) 83.3 url baseline WMT16 67.9 best 80.1 badLuc w/o post-treat 88.6 w post-treat 92.1 •  Playing with metaparameters helps a lot •  Applying a length filter also helps a lot •  Involving translation is a must •  even with a simple lexicon (which covers ~half of the words in the collection) •  Our URL variant is performing impressively well •  outperforms the baseline •  useful on short documents •  Combining both indexes (text and urls) helps •  Post-filtering is a plus 22felipe@BUCC 2017
  • 24.
    Plan   Data Management(Acquisition and Organization) •  What has been done •  What (I feel) is still missing •  Domain specific bilingual parallel and comparable corpora •  Assessing the quality/usefulness of a pair of documents   Parallel Material Extraction from a Comparable Corpus •  Document pairs •  Sentence pairs •  Word pairs 23felipe@BUCC 2017
  • 25.
    MUNT (Bérard, 2014) Areimplementation of (Munteanu and Marcu, 2005) with features borrowed from (Smith et al., 2010) 24felipe@BUCC 2017
  • 26.
    MUNT (Bérard, 2014) • classifier(logistic regression) •  31 features •  length based, alignment-based features, fertility, etc. • pre-filter (selecting interesting sentence pairs) •  ratio of sentence length no greater than 2 •  at least 50% of tokens with alignment in the other side •  removes ~98% of the Cartesian product !! 25felipe@BUCC 2017
  • 27.
    MUNT on Wikipedia • Done before (Smith et al., 2010) •  Wikipedia dump (2014) felipe@BUCC 2017 26 #articles #paired #sent. #tokens en 4.5M 919k 29.3M 630.8M fr 1.5M 919k 16.8M 354.5M •  Configuration (default) -  lexicon trained with GIZA++ on 100k sentence pairs of Europarl -  classifier trained on 800 positive and 800 negative examples of news data, threeshold: 0.8
  • 28.
    MUNT on Wikipedia felipe@BUCC2017 27 Manually evaluated 500 (random) sentence pairs -  parallel -  quasi-parallel (at least partial) -  not parallel At the cellular level, the nervous system is defined by the presence of a special type of cell, called the neuron, also known as a ”nerve cell” . À l’échelle cellulaire, le système nerveux est défini par la présence de cellules hautement spécialisées appelées neurones, qui ont la capacité, très particulière, de véhiculer un signal électrochimique .
  • 29.
    MUNT on Wikipedia felipe@BUCC2017 28 grade % parallel 71 quasi-parallel 15 not parallel 14 We tried something odd: •  We applied Yasa (Lamraoui and Langlais, 2013) on Wikipedia article pairs (pretending they were parallel) •  Asked MUNT to classify the sentence pairs identified grade % parallel 48 quasi-parallel 16 not parallel 36 -  11M sentence pairs identified by Yasa -  1.6M kept parallel by MUNT -  1.3M once duplicate removed -  86% of sentence pairs are (quasi-)parallel -  much faster ! -  15 hours (on a cluster of 8 nodes) -  MUNT detected 2.61M sentence pairs -  2.26M once duplicates removed -  1.92M after removing sentences shorter than 4 words -  64% of sentence pairs are (quasi-)parallel
  • 30.
    Munt on Wikipedia felipe@BUCC2017 29 •  50 comparable Wiki article pairs manually aligned at the sentence level (Rebout and Langlais, 2014) •  measured the performance of YASA and MUNT on those articles •  MUNT has a better precision, but a lower recall npara # parallel sentences nfr (nen) # sentences in fr (en) doc.
  • 31.
    BUCC 2017 SharedTask (Zweigenbaum et al., 2016) •  Detecting parallel sentences in a large text collection •  2 sets of monolingual Wikipedia sentences (2014 dumps): •  1.4M French sentences •  1.9M English sentences •  + 17k parallel sentences from News Commentary (v9) •  Evaluation: precision, recall, F1 •  pros •  no metadata (text-based) •  Cartesian product is large (with few positive examples) •  Smartness in inserting parallel sentences (to avoid simple solutions) •  cons •  Artificial task •  True parallel sentences in Wikipedia EN-FR are not known 30felipe@BUCC 2017
  • 32.
    RALI’s participation 31felipe@BUCC 2017 Willbe presented this afternoon (Grégoire and Langlais, 2017)
  • 33.
    RALI’s participation •  Training:Europarl v7 French-English •  first 500K sentence pairs •  negative sampling: random selection of sentence pairs •  Test: newstest2012 (out-domain) •  1000 first sentence pairs + noise •  Pre-processing •  maximum sentence length: 80 •  tokenization with Moses’ toolkit, lowercased •  mapping digits to 0 (e.g. 1982 -> 0000) 32felipe@BUCC 2017
  • 34.
    RALI’s participation •  Embedding-basedfilter for avoiding the Cartesian product •  word-based embeddings computed with BILBOWA (Gouws, 2015) •  sentence representation = average word embeddings •  40 first target sentences for each source one •  Not paying attention to ``details’’ (only model matters, right?) •  digit preprocessing •  random negative sampling at training does not match testing condition •  The next slides summarize what we have learnt after our participation to BUCC 2017 (new material) model precision recall F1 12.1 70.9 20.7 official 33felipe@BUCC 2017
  • 35.
    Influence of thedecision threeshold •  Cartesian product •  1M ex., 1k positive •  BIRNN trained with 7 negative ex. •  MUNT trained with a balanced corpus Precision Recall F1 ρ BiRNN 83.0 69.6 75.7 0.99 MUNT 31.8 24.1 27.7 0.99 34felipe@BUCC 2017
  • 36.
    Influence of thedecision threeshold •  Pre-filtering •  8053 ex., 1k positive •  BIRNN trained with 7 negative ex. •  MUNT trained with a balanced corpus Precision Recall F1 ρ BiRNN 91.0 62.4 74.0 0.97 MUNT 73.3 57.0 64.1 0.91 35felipe@BUCC 2017
  • 37.
    Influence of Post-filtering • Each decision taken independently •  A src sent. may be associated to several target ones, and vice versa ²  post-filtering (greedy algorithm, Hungarian algo. too slow) huge boost in precision at a small recall loss for both approaches 36felipe@BUCC 2017
  • 38.
    Plan   Data Management(Acquisition and Organization) •  What has been done •  What (I feel) is still missing •  Domain specific bilingual parallel and comparable corpora •  Assessing the quality/usefulness of a pair of documents   Parallel Material Extraction from a Comparable Corpus •  Document pairs •  Sentence pairs •  Word pairs 37felipe@BUCC 2017
  • 39.
    Bilingual Lexicon Induction • Received a lot of attention •  Pioneering works: (Rapp, 1995; Fung, 1995) •  See (Sharoff et al., 2013) •  Revisited as a way to measure the quality of word embeddings •  Seminal work of (Mikolov et al., 2013) •  Comprehensive comparisons •  (Levy et al., 2014, 2017; Upadhyay et al., 2016) 38felipe@BUCC 2017
  • 40.
    Bilingual Lexicon Induction (Jakubina& Langlais; 2016) •  We thoroughly revisited those approaches: •  (Rapp, 1995) •  (Mikolov et al. 2013) •  training the projection matrix with the toolkit of (Dinu and Baroni, 2015) •  (Faruqui and Dyer, 2014) •  and a few others, but without success •  investigating their meta-parameters •  paying attention to the frequency of the terms •  after (Pekar et al, 2006) •  showing their complementarity 39felipe@BUCC 2017
  • 41.
    Experiments •  Wikipedia (dumpsof June 2013) •  EN: 7.3M token forms (1.2G tokens) •  FR: 3.6M token forms (330M tokens) •  Test sets •  Wiki≤25 English words occurring at most 25 times in Wiki-EN •  6.8M such tokens (92%) •  Wiki>25 English words seen more than 25 times in Wiki-EN •  Euro-5-6k top frequent 5000 to 6000 words of WMT2011 Frequency min max avg cov (%) Wiki≤25 1 25 10 100.0 Wiki>25 27 19.4k 2.8k 100.0 Euro5-6k 1 2.6M 33.6k 87.3 40felipe@BUCC 2017
  • 42.
    Metaparameters explored •  Rapp • window size (3,7,15,31) •  association measure (LLR, discontinuous odd-ratios) •  projecting UNK words (yes, no) •  similarity measure (fixed, cosine similarity) •  seed lexicon (fixed, in-house 107k entries) •  Word2Vec •  skip-gram versus continuous bag-of-word •  negative sampling (5 or 10) versus hierarchical softmax •  window size (6,10,20,30) •  dimension (from 50 up to 250 for SKG and 200 for CBOW) •  seed lexicon (2k-low, 5-high, 5k-rand) •  Faruqui & Dyer •  ratio (0.5, 0.8 and 1.0) •  fixed dimension (the best one for Word2Vec) 41felipe@BUCC 2017 No adhoc filtering as usually done, so very time and memory challenging for the Rapp approach (Prochasson and Fung, 2011) this work -  ~20k document pairs ~700k ones -  target voc. 128k words (nouns) 3M words We did apply a few filters: -  context vectors: 1000 top-ranked words -  50k first occurrences of a source term
  • 43.
    Best variant (pertest set) @1 @5 @20 Wiki>25 (ed@1 19.3) Rapp 20.0 33.0 43.0 Miko 17.0 32.6 41.6 Faru 13.3 26.0 33.3 Wiki≤25 (ed@1 17.6) Rapp 2.6 4.3 7.3 Miko 1.6 4.6 10.6 Faru 1.6 2.6 5.0 Euro5-6k (ed@1 8.0) Rapp 16.6 31.8 41.2 Miko 42.0 59.0 67.8 Faru 30.6 47.7 59.8 42felipe@BUCC 2017
  • 44.
    Reranking Candidate Translations (Jakubinaand Langlais, 2016) •  Reranking shown useful in a number of settings (Delpech et al., 2012; Harastani et al., 2013; Kontonatsios et al., 2014) •  Trained a reranker (random forest) with RankLib •  700 terms for training, remaining 300 terms for testing •  3-fold cross-validation •  Light features for each pair (s,t): •  frequency-based features •  freq of s,t and their difference •  string-based features •  length of s and t, their difference, their ratio, edit distance(s,t) •  rank-based features •  score and rank of t in the native list •  number of lists in which t appears (when several n-best lists are considered) 43felipe@BUCC 2017
  • 45.
    Reranking Individual n-bestLists Individual 1-Reranked @1 @5 @20 @1 @5 @20 Wiki>25 Rapp 20.0 33.0 43.0 36.3 48.8 53.8 Miko 17.0 32.6 41.6 38.1 49.0 54.3 Faru 13.3 26.0 33.3 34.3 44.0 47.9 Wiki≤25 Rapp 2.6 4.3 7.3 8.6 9.4 10.2 Miko 1.6 4.6 10.6 16.6 19.0 20.1 Faru 1.6 2.6 5.0 7.9 8.7 8.9 Euro5-6k Rapp 16.6 31.8 41.2 34.6 48.6 51.9 Miko 42.0 59.0 67.8 47.0 68.1 73.0 Faru 30.6 47.7 59.8 41.2 58.0 66.0 44felipe@BUCC 2017
  • 46.
    Reranking several n-bestLists 1-Reranked n-Reranked @1 @5 @20 @1 @5 @20 Wiki>25 Rapp 36.3 48.8 53.8 Miko 38.1 49.0 54.3 R+M 43.3 58.4 62.4 Faru 34.3 44.0 47.9 R+M+F 45.6 59.6 64.0 Wiki≤25 Rapp 8.6 9.4 10.2 Miko 16.6 19.0 20.1 R+M 18.9 22.0 23.6 Faru 7.9 8.7 8.9 R+M+F 21.3 24.4 25.7 Euro5-6k Rapp 34.6 48.6 51.9 Miko 47.0 68.1 73.0 R+M 49.5 68.7 76.1 Faru 41.2 58.0 66.0 R+M+F 47.6 68.5 76.2 45felipe@BUCC 2017
  • 47.
    One-Slide Wrap up • Hyp:Model-centric research somehow hides the value of: •  know-how in managing bilingual data (parallel and comparable) •  evaluation protocols involving real users (or proxies) • Better handling data is part of the game • We should learn from users felipe@BUCC 2017 46
  • 48.
    Thank you foryour attention felipe@BUCC 2017 47