SlideShare a Scribd company logo

Mirella Lapata - 2017 - Invited Keynote: Translating from Multiple Modalities to Text and Back

Invited Keynote by Professor Mirella Lapata (University of Edinburgh) for ACL 2017. 2 Aug 2017. Vancouver, Canada License: CC BY 4.0

1 of 95
Download to read offline
Motivation
Case Studies
Translating from Multiple Modalities
to Text and Back
Mirella Lapata
School of Informatics
University of Edinburgh
mlap@inf.ed.ac.uk
1 / 70
Motivation
Case Studies
Research Goal
Methodology
Global Internet User Survey
72% of Internet users find it frustrating to get irrelevant
information when web searching!
Source: www.internetsociety.org/survey
2 / 70
Motivation
Case Studies
Research Goal
Methodology
What Causes User Frustration?
User
frustration
Multimodal
information needs
Information
overload
Inaccessible
content
text, DB, SM
images, video
songs, code
formulas
too many hits
contradiction
duplication
no structure
children
2nd language
learners
laypersons
disabled users
3 / 70
Motivation
Case Studies
Research Goal
Methodology
NLP Comes to the Rescue!
By rendering information more accessible by translat-
ing within the same language and between language
and different modalities.
4 / 70
Motivation
Case Studies
Research Goal
Methodology
NLP Comes to the Rescue!
riding a horse
5 / 70
Motivation
Case Studies
Research Goal
Methodology
NLP Comes to the Rescue!
riding a horse
define function with argument n
if n is not an integer value,
throw a TypeError exception
5 / 70
Ad

Recommended

How the Context Matters Language and Interaction in Dialogues
How the Context Matters Language and Interaction in DialoguesHow the Context Matters Language and Interaction in Dialogues
How the Context Matters Language and Interaction in DialoguesYun-Nung (Vivian) Chen
 
VAW-GAN for disentanglement and recomposition of emotional elements in speech
VAW-GAN for disentanglement and recomposition of emotional elements in speechVAW-GAN for disentanglement and recomposition of emotional elements in speech
VAW-GAN for disentanglement and recomposition of emotional elements in speechKunZhou18
 
IEEE ICASSP 2021
IEEE ICASSP 2021IEEE ICASSP 2021
IEEE ICASSP 2021KunZhou18
 
Oral Qualification Examination_Kun_Zhou
Oral Qualification Examination_Kun_ZhouOral Qualification Examination_Kun_Zhou
Oral Qualification Examination_Kun_ZhouKunZhou18
 
Speaker specific feature based clustering and its applications in language in...
Speaker specific feature based clustering and its applications in language in...Speaker specific feature based clustering and its applications in language in...
Speaker specific feature based clustering and its applications in language in...IJECEIAES
 
Meetup 22/2/2018 - Artificiële Intelligentie & Human Resources
Meetup 22/2/2018 - Artificiële Intelligentie & Human ResourcesMeetup 22/2/2018 - Artificiële Intelligentie & Human Resources
Meetup 22/2/2018 - Artificiële Intelligentie & Human ResourcesDigipolis Antwerpen
 
Abstractive Text Summarization
Abstractive Text SummarizationAbstractive Text Summarization
Abstractive Text SummarizationTho Phan
 
Conversational transfer learning for emotion recognition
Conversational transfer learning for emotion recognitionConversational transfer learning for emotion recognition
Conversational transfer learning for emotion recognitionTakato Hayashi
 

More Related Content

Similar to Mirella Lapata - 2017 - Invited Keynote: Translating from Multiple Modalities to Text and Back

Feature Extraction and Analysis of Natural Language Processing for Deep Learn...
Feature Extraction and Analysis of Natural Language Processing for Deep Learn...Feature Extraction and Analysis of Natural Language Processing for Deep Learn...
Feature Extraction and Analysis of Natural Language Processing for Deep Learn...Sharmila Sathish
 
IRJET - Automatic Lip Reading: Classification of Words and Phrases using Conv...
IRJET - Automatic Lip Reading: Classification of Words and Phrases using Conv...IRJET - Automatic Lip Reading: Classification of Words and Phrases using Conv...
IRJET - Automatic Lip Reading: Classification of Words and Phrases using Conv...IRJET Journal
 
IRJET- Semantic Question Matching
IRJET- Semantic Question MatchingIRJET- Semantic Question Matching
IRJET- Semantic Question MatchingIRJET Journal
 
The Smart Way To Invest in AI and ML_SFStartupDay
The Smart Way To Invest in AI and ML_SFStartupDayThe Smart Way To Invest in AI and ML_SFStartupDay
The Smart Way To Invest in AI and ML_SFStartupDayAmazon Web Services
 
How can text-mining leverage developments in Deep Learning? Presentation at ...
How can text-mining leverage developments in Deep Learning?  Presentation at ...How can text-mining leverage developments in Deep Learning?  Presentation at ...
How can text-mining leverage developments in Deep Learning? Presentation at ...jcscholtes
 
ODSC East: Effective Transfer Learning for NLP
ODSC East: Effective Transfer Learning for NLPODSC East: Effective Transfer Learning for NLP
ODSC East: Effective Transfer Learning for NLPindico data
 
A comprehensive guide to prompt engineering.pdf
A comprehensive guide to prompt engineering.pdfA comprehensive guide to prompt engineering.pdf
A comprehensive guide to prompt engineering.pdfStephenAmell4
 
[Paper Reading] Unsupervised Learning of Sentence Embeddings using Compositi...
[Paper Reading]  Unsupervised Learning of Sentence Embeddings using Compositi...[Paper Reading]  Unsupervised Learning of Sentence Embeddings using Compositi...
[Paper Reading] Unsupervised Learning of Sentence Embeddings using Compositi...Hiroki Shimanaka
 
Bridging the gap between AI and UI - DSI Vienna - full version
Bridging the gap between AI and UI - DSI Vienna - full versionBridging the gap between AI and UI - DSI Vienna - full version
Bridging the gap between AI and UI - DSI Vienna - full versionLiad Magen
 
IRJET- Querying Database using Natural Language Interface
IRJET-  	  Querying Database using Natural Language InterfaceIRJET-  	  Querying Database using Natural Language Interface
IRJET- Querying Database using Natural Language InterfaceIRJET Journal
 
Parallel Support Vector Machines Is A Supervised Machine...
Parallel Support Vector Machines Is A Supervised Machine...Parallel Support Vector Machines Is A Supervised Machine...
Parallel Support Vector Machines Is A Supervised Machine...Jessica Reed
 
IRJET - Mobile Chatbot for Information Search
 IRJET - Mobile Chatbot for Information Search IRJET - Mobile Chatbot for Information Search
IRJET - Mobile Chatbot for Information SearchIRJET Journal
 
IDE Code Compiler for the physically challenged (Deaf, Blind & Mute)
IDE Code Compiler for the physically challenged (Deaf, Blind & Mute)IDE Code Compiler for the physically challenged (Deaf, Blind & Mute)
IDE Code Compiler for the physically challenged (Deaf, Blind & Mute)IRJET Journal
 
Deep Learning & NLP: Graphs to the Rescue!
Deep Learning & NLP: Graphs to the Rescue!Deep Learning & NLP: Graphs to the Rescue!
Deep Learning & NLP: Graphs to the Rescue!Roelof Pieters
 
Accessibility of Big & Broad Data
Accessibility of Big & Broad DataAccessibility of Big & Broad Data
Accessibility of Big & Broad DataSimon Harper
 
Text Summarization and Conversion of Speech to Text
Text Summarization and Conversion of Speech to TextText Summarization and Conversion of Speech to Text
Text Summarization and Conversion of Speech to TextIRJET Journal
 
SENTIMENT ANALYSIS IN MYANMAR LANGUAGE USING CONVOLUTIONAL LSTM NEURAL NETWORK
SENTIMENT ANALYSIS IN MYANMAR LANGUAGE USING CONVOLUTIONAL LSTM NEURAL NETWORKSENTIMENT ANALYSIS IN MYANMAR LANGUAGE USING CONVOLUTIONAL LSTM NEURAL NETWORK
SENTIMENT ANALYSIS IN MYANMAR LANGUAGE USING CONVOLUTIONAL LSTM NEURAL NETWORKijnlc
 

Similar to Mirella Lapata - 2017 - Invited Keynote: Translating from Multiple Modalities to Text and Back (20)

Feature Extraction and Analysis of Natural Language Processing for Deep Learn...
Feature Extraction and Analysis of Natural Language Processing for Deep Learn...Feature Extraction and Analysis of Natural Language Processing for Deep Learn...
Feature Extraction and Analysis of Natural Language Processing for Deep Learn...
 
IRJET - Automatic Lip Reading: Classification of Words and Phrases using Conv...
IRJET - Automatic Lip Reading: Classification of Words and Phrases using Conv...IRJET - Automatic Lip Reading: Classification of Words and Phrases using Conv...
IRJET - Automatic Lip Reading: Classification of Words and Phrases using Conv...
 
IRJET- Semantic Question Matching
IRJET- Semantic Question MatchingIRJET- Semantic Question Matching
IRJET- Semantic Question Matching
 
The Smart Way To Invest in AI and ML_SFStartupDay
The Smart Way To Invest in AI and ML_SFStartupDayThe Smart Way To Invest in AI and ML_SFStartupDay
The Smart Way To Invest in AI and ML_SFStartupDay
 
How can text-mining leverage developments in Deep Learning? Presentation at ...
How can text-mining leverage developments in Deep Learning?  Presentation at ...How can text-mining leverage developments in Deep Learning?  Presentation at ...
How can text-mining leverage developments in Deep Learning? Presentation at ...
 
team10.ppt.pptx
team10.ppt.pptxteam10.ppt.pptx
team10.ppt.pptx
 
ODSC East: Effective Transfer Learning for NLP
ODSC East: Effective Transfer Learning for NLPODSC East: Effective Transfer Learning for NLP
ODSC East: Effective Transfer Learning for NLP
 
A comprehensive guide to prompt engineering.pdf
A comprehensive guide to prompt engineering.pdfA comprehensive guide to prompt engineering.pdf
A comprehensive guide to prompt engineering.pdf
 
Plug play language_models
Plug play language_modelsPlug play language_models
Plug play language_models
 
[Paper Reading] Unsupervised Learning of Sentence Embeddings using Compositi...
[Paper Reading]  Unsupervised Learning of Sentence Embeddings using Compositi...[Paper Reading]  Unsupervised Learning of Sentence Embeddings using Compositi...
[Paper Reading] Unsupervised Learning of Sentence Embeddings using Compositi...
 
Bridging the gap between AI and UI - DSI Vienna - full version
Bridging the gap between AI and UI - DSI Vienna - full versionBridging the gap between AI and UI - DSI Vienna - full version
Bridging the gap between AI and UI - DSI Vienna - full version
 
IRJET- Querying Database using Natural Language Interface
IRJET-  	  Querying Database using Natural Language InterfaceIRJET-  	  Querying Database using Natural Language Interface
IRJET- Querying Database using Natural Language Interface
 
Parallel Support Vector Machines Is A Supervised Machine...
Parallel Support Vector Machines Is A Supervised Machine...Parallel Support Vector Machines Is A Supervised Machine...
Parallel Support Vector Machines Is A Supervised Machine...
 
IRJET - Mobile Chatbot for Information Search
 IRJET - Mobile Chatbot for Information Search IRJET - Mobile Chatbot for Information Search
IRJET - Mobile Chatbot for Information Search
 
IDE Code Compiler for the physically challenged (Deaf, Blind & Mute)
IDE Code Compiler for the physically challenged (Deaf, Blind & Mute)IDE Code Compiler for the physically challenged (Deaf, Blind & Mute)
IDE Code Compiler for the physically challenged (Deaf, Blind & Mute)
 
Deep Learning & NLP: Graphs to the Rescue!
Deep Learning & NLP: Graphs to the Rescue!Deep Learning & NLP: Graphs to the Rescue!
Deep Learning & NLP: Graphs to the Rescue!
 
Accessibility of Big & Broad Data
Accessibility of Big & Broad DataAccessibility of Big & Broad Data
Accessibility of Big & Broad Data
 
Julia text mining_inmobi
Julia text mining_inmobiJulia text mining_inmobi
Julia text mining_inmobi
 
Text Summarization and Conversion of Speech to Text
Text Summarization and Conversion of Speech to TextText Summarization and Conversion of Speech to Text
Text Summarization and Conversion of Speech to Text
 
SENTIMENT ANALYSIS IN MYANMAR LANGUAGE USING CONVOLUTIONAL LSTM NEURAL NETWORK
SENTIMENT ANALYSIS IN MYANMAR LANGUAGE USING CONVOLUTIONAL LSTM NEURAL NETWORKSENTIMENT ANALYSIS IN MYANMAR LANGUAGE USING CONVOLUTIONAL LSTM NEURAL NETWORK
SENTIMENT ANALYSIS IN MYANMAR LANGUAGE USING CONVOLUTIONAL LSTM NEURAL NETWORK
 

More from Association for Computational Linguistics

Castro - 2018 - A High Coverage Method for Automatic False Friends Detection ...
Castro - 2018 - A High Coverage Method for Automatic False Friends Detection ...Castro - 2018 - A High Coverage Method for Automatic False Friends Detection ...
Castro - 2018 - A High Coverage Method for Automatic False Friends Detection ...Association for Computational Linguistics
 
Muthu Kumar Chandrasekaran - 2018 - Countering Position Bias in Instructor In...
Muthu Kumar Chandrasekaran - 2018 - Countering Position Bias in Instructor In...Muthu Kumar Chandrasekaran - 2018 - Countering Position Bias in Instructor In...
Muthu Kumar Chandrasekaran - 2018 - Countering Position Bias in Instructor In...Association for Computational Linguistics
 
Daniel Gildea - 2018 - The ACL Anthology: Current State and Future Directions
Daniel Gildea - 2018 - The ACL Anthology: Current State and Future DirectionsDaniel Gildea - 2018 - The ACL Anthology: Current State and Future Directions
Daniel Gildea - 2018 - The ACL Anthology: Current State and Future DirectionsAssociation for Computational Linguistics
 
Daniel Gildea - 2018 - The ACL Anthology: Current State and Future Directions
Daniel Gildea - 2018 - The ACL Anthology: Current State and Future DirectionsDaniel Gildea - 2018 - The ACL Anthology: Current State and Future Directions
Daniel Gildea - 2018 - The ACL Anthology: Current State and Future DirectionsAssociation for Computational Linguistics
 
Wenqiang Lei - 2018 - Sequicity: Simplifying Task-oriented Dialogue Systems w...
Wenqiang Lei - 2018 - Sequicity: Simplifying Task-oriented Dialogue Systems w...Wenqiang Lei - 2018 - Sequicity: Simplifying Task-oriented Dialogue Systems w...
Wenqiang Lei - 2018 - Sequicity: Simplifying Task-oriented Dialogue Systems w...Association for Computational Linguistics
 
Matthew Marge - 2017 - Exploring Variation of Natural Human Commands to a Rob...
Matthew Marge - 2017 - Exploring Variation of Natural Human Commands to a Rob...Matthew Marge - 2017 - Exploring Variation of Natural Human Commands to a Rob...
Matthew Marge - 2017 - Exploring Variation of Natural Human Commands to a Rob...Association for Computational Linguistics
 
Venkatesh Duppada - 2017 - SeerNet at EmoInt-2017: Tweet Emotion Intensity Es...
Venkatesh Duppada - 2017 - SeerNet at EmoInt-2017: Tweet Emotion Intensity Es...Venkatesh Duppada - 2017 - SeerNet at EmoInt-2017: Tweet Emotion Intensity Es...
Venkatesh Duppada - 2017 - SeerNet at EmoInt-2017: Tweet Emotion Intensity Es...Association for Computational Linguistics
 
Satoshi Sonoh - 2015 - Toshiba MT System Description for the WAT2015 Workshop
Satoshi Sonoh - 2015 - Toshiba MT System Description for the WAT2015 WorkshopSatoshi Sonoh - 2015 - Toshiba MT System Description for the WAT2015 Workshop
Satoshi Sonoh - 2015 - Toshiba MT System Description for the WAT2015 WorkshopAssociation for Computational Linguistics
 
John Richardson - 2015 - KyotoEBMT System Description for the 2nd Workshop on...
John Richardson - 2015 - KyotoEBMT System Description for the 2nd Workshop on...John Richardson - 2015 - KyotoEBMT System Description for the 2nd Workshop on...
John Richardson - 2015 - KyotoEBMT System Description for the 2nd Workshop on...Association for Computational Linguistics
 
John Richardson - 2015 - KyotoEBMT System Description for the 2nd Workshop on...
John Richardson - 2015 - KyotoEBMT System Description for the 2nd Workshop on...John Richardson - 2015 - KyotoEBMT System Description for the 2nd Workshop on...
John Richardson - 2015 - KyotoEBMT System Description for the 2nd Workshop on...Association for Computational Linguistics
 
Zhongyuan Zhu - 2015 - Evaluating Neural Machine Translation in English-Japan...
Zhongyuan Zhu - 2015 - Evaluating Neural Machine Translation in English-Japan...Zhongyuan Zhu - 2015 - Evaluating Neural Machine Translation in English-Japan...
Zhongyuan Zhu - 2015 - Evaluating Neural Machine Translation in English-Japan...Association for Computational Linguistics
 
Zhongyuan Zhu - 2015 - Evaluating Neural Machine Translation in English-Japan...
Zhongyuan Zhu - 2015 - Evaluating Neural Machine Translation in English-Japan...Zhongyuan Zhu - 2015 - Evaluating Neural Machine Translation in English-Japan...
Zhongyuan Zhu - 2015 - Evaluating Neural Machine Translation in English-Japan...Association for Computational Linguistics
 
Satoshi Sonoh - 2015 - Toshiba MT System Description for the WAT2015 Workshop
Satoshi Sonoh - 2015 - Toshiba MT System Description for the WAT2015 WorkshopSatoshi Sonoh - 2015 - Toshiba MT System Description for the WAT2015 Workshop
Satoshi Sonoh - 2015 - Toshiba MT System Description for the WAT2015 WorkshopAssociation for Computational Linguistics
 
Graham Neubig - 2015 - Neural Reranking Improves Subjective Quality of Machin...
Graham Neubig - 2015 - Neural Reranking Improves Subjective Quality of Machin...Graham Neubig - 2015 - Neural Reranking Improves Subjective Quality of Machin...
Graham Neubig - 2015 - Neural Reranking Improves Subjective Quality of Machin...Association for Computational Linguistics
 

More from Association for Computational Linguistics (20)

Muis - 2016 - Weak Semi-Markov CRFs for NP Chunking in Informal Text
Muis - 2016 - Weak Semi-Markov CRFs for NP Chunking in Informal TextMuis - 2016 - Weak Semi-Markov CRFs for NP Chunking in Informal Text
Muis - 2016 - Weak Semi-Markov CRFs for NP Chunking in Informal Text
 
Castro - 2018 - A High Coverage Method for Automatic False Friends Detection ...
Castro - 2018 - A High Coverage Method for Automatic False Friends Detection ...Castro - 2018 - A High Coverage Method for Automatic False Friends Detection ...
Castro - 2018 - A High Coverage Method for Automatic False Friends Detection ...
 
Castro - 2018 - A Crowd-Annotated Spanish Corpus for Humour Analysis
Castro - 2018 - A Crowd-Annotated Spanish Corpus for Humour AnalysisCastro - 2018 - A Crowd-Annotated Spanish Corpus for Humour Analysis
Castro - 2018 - A Crowd-Annotated Spanish Corpus for Humour Analysis
 
Muthu Kumar Chandrasekaran - 2018 - Countering Position Bias in Instructor In...
Muthu Kumar Chandrasekaran - 2018 - Countering Position Bias in Instructor In...Muthu Kumar Chandrasekaran - 2018 - Countering Position Bias in Instructor In...
Muthu Kumar Chandrasekaran - 2018 - Countering Position Bias in Instructor In...
 
Daniel Gildea - 2018 - The ACL Anthology: Current State and Future Directions
Daniel Gildea - 2018 - The ACL Anthology: Current State and Future DirectionsDaniel Gildea - 2018 - The ACL Anthology: Current State and Future Directions
Daniel Gildea - 2018 - The ACL Anthology: Current State and Future Directions
 
Elior Sulem - 2018 - Semantic Structural Evaluation for Text Simplification
Elior Sulem - 2018 - Semantic Structural Evaluation for Text SimplificationElior Sulem - 2018 - Semantic Structural Evaluation for Text Simplification
Elior Sulem - 2018 - Semantic Structural Evaluation for Text Simplification
 
Daniel Gildea - 2018 - The ACL Anthology: Current State and Future Directions
Daniel Gildea - 2018 - The ACL Anthology: Current State and Future DirectionsDaniel Gildea - 2018 - The ACL Anthology: Current State and Future Directions
Daniel Gildea - 2018 - The ACL Anthology: Current State and Future Directions
 
Wenqiang Lei - 2018 - Sequicity: Simplifying Task-oriented Dialogue Systems w...
Wenqiang Lei - 2018 - Sequicity: Simplifying Task-oriented Dialogue Systems w...Wenqiang Lei - 2018 - Sequicity: Simplifying Task-oriented Dialogue Systems w...
Wenqiang Lei - 2018 - Sequicity: Simplifying Task-oriented Dialogue Systems w...
 
Matthew Marge - 2017 - Exploring Variation of Natural Human Commands to a Rob...
Matthew Marge - 2017 - Exploring Variation of Natural Human Commands to a Rob...Matthew Marge - 2017 - Exploring Variation of Natural Human Commands to a Rob...
Matthew Marge - 2017 - Exploring Variation of Natural Human Commands to a Rob...
 
Venkatesh Duppada - 2017 - SeerNet at EmoInt-2017: Tweet Emotion Intensity Es...
Venkatesh Duppada - 2017 - SeerNet at EmoInt-2017: Tweet Emotion Intensity Es...Venkatesh Duppada - 2017 - SeerNet at EmoInt-2017: Tweet Emotion Intensity Es...
Venkatesh Duppada - 2017 - SeerNet at EmoInt-2017: Tweet Emotion Intensity Es...
 
Satoshi Sonoh - 2015 - Toshiba MT System Description for the WAT2015 Workshop
Satoshi Sonoh - 2015 - Toshiba MT System Description for the WAT2015 WorkshopSatoshi Sonoh - 2015 - Toshiba MT System Description for the WAT2015 Workshop
Satoshi Sonoh - 2015 - Toshiba MT System Description for the WAT2015 Workshop
 
Chenchen Ding - 2015 - NICT at WAT 2015
Chenchen Ding - 2015 - NICT at WAT 2015Chenchen Ding - 2015 - NICT at WAT 2015
Chenchen Ding - 2015 - NICT at WAT 2015
 
John Richardson - 2015 - KyotoEBMT System Description for the 2nd Workshop on...
John Richardson - 2015 - KyotoEBMT System Description for the 2nd Workshop on...John Richardson - 2015 - KyotoEBMT System Description for the 2nd Workshop on...
John Richardson - 2015 - KyotoEBMT System Description for the 2nd Workshop on...
 
John Richardson - 2015 - KyotoEBMT System Description for the 2nd Workshop on...
John Richardson - 2015 - KyotoEBMT System Description for the 2nd Workshop on...John Richardson - 2015 - KyotoEBMT System Description for the 2nd Workshop on...
John Richardson - 2015 - KyotoEBMT System Description for the 2nd Workshop on...
 
Zhongyuan Zhu - 2015 - Evaluating Neural Machine Translation in English-Japan...
Zhongyuan Zhu - 2015 - Evaluating Neural Machine Translation in English-Japan...Zhongyuan Zhu - 2015 - Evaluating Neural Machine Translation in English-Japan...
Zhongyuan Zhu - 2015 - Evaluating Neural Machine Translation in English-Japan...
 
Zhongyuan Zhu - 2015 - Evaluating Neural Machine Translation in English-Japan...
Zhongyuan Zhu - 2015 - Evaluating Neural Machine Translation in English-Japan...Zhongyuan Zhu - 2015 - Evaluating Neural Machine Translation in English-Japan...
Zhongyuan Zhu - 2015 - Evaluating Neural Machine Translation in English-Japan...
 
Hyoung-Gyu Lee - 2015 - NAVER Machine Translation System for WAT 2015
Hyoung-Gyu Lee - 2015 - NAVER Machine Translation System for WAT 2015Hyoung-Gyu Lee - 2015 - NAVER Machine Translation System for WAT 2015
Hyoung-Gyu Lee - 2015 - NAVER Machine Translation System for WAT 2015
 
Satoshi Sonoh - 2015 - Toshiba MT System Description for the WAT2015 Workshop
Satoshi Sonoh - 2015 - Toshiba MT System Description for the WAT2015 WorkshopSatoshi Sonoh - 2015 - Toshiba MT System Description for the WAT2015 Workshop
Satoshi Sonoh - 2015 - Toshiba MT System Description for the WAT2015 Workshop
 
Chenchen Ding - 2015 - NICT at WAT 2015
Chenchen Ding - 2015 - NICT at WAT 2015Chenchen Ding - 2015 - NICT at WAT 2015
Chenchen Ding - 2015 - NICT at WAT 2015
 
Graham Neubig - 2015 - Neural Reranking Improves Subjective Quality of Machin...
Graham Neubig - 2015 - Neural Reranking Improves Subjective Quality of Machin...Graham Neubig - 2015 - Neural Reranking Improves Subjective Quality of Machin...
Graham Neubig - 2015 - Neural Reranking Improves Subjective Quality of Machin...
 

Recently uploaded

Plant Genetic Resources, Germplasm, gene pool - Copy.pptx
Plant Genetic Resources, Germplasm, gene pool - Copy.pptxPlant Genetic Resources, Germplasm, gene pool - Copy.pptx
Plant Genetic Resources, Germplasm, gene pool - Copy.pptxAKSHAYMAGAR17
 
Andreas Schleicher - 20 Feb 2024 - How pop music, podcasts, and Tik Tok are i...
Andreas Schleicher - 20 Feb 2024 - How pop music, podcasts, and Tik Tok are i...Andreas Schleicher - 20 Feb 2024 - How pop music, podcasts, and Tik Tok are i...
Andreas Schleicher - 20 Feb 2024 - How pop music, podcasts, and Tik Tok are i...EduSkills OECD
 
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...Nguyen Thanh Tu Collection
 
Practical Research 1, Lesson 5: DESIGNING A RESEARCH PROJECT RELATED TO DAILY...
Practical Research 1, Lesson 5: DESIGNING A RESEARCH PROJECT RELATED TO DAILY...Practical Research 1, Lesson 5: DESIGNING A RESEARCH PROJECT RELATED TO DAILY...
Practical Research 1, Lesson 5: DESIGNING A RESEARCH PROJECT RELATED TO DAILY...Katherine Villaluna
 
Nzinga Kika - The story of the queen
Nzinga Kika    -  The story of the queenNzinga Kika    -  The story of the queen
Nzinga Kika - The story of the queenDeanAmory1
 
Overview of Databases and Data Modelling-2.pdf
Overview of Databases and Data Modelling-2.pdfOverview of Databases and Data Modelling-2.pdf
Overview of Databases and Data Modelling-2.pdfChristalin Nelson
 
11 CI SINIF SINAQLARI - 10-2023-Aynura-Hamidova.pdf
11 CI SINIF SINAQLARI - 10-2023-Aynura-Hamidova.pdf11 CI SINIF SINAQLARI - 10-2023-Aynura-Hamidova.pdf
11 CI SINIF SINAQLARI - 10-2023-Aynura-Hamidova.pdfAynouraHamidova
 
Food Web SlideShare for Ecology Notes Quiz in Canvas
Food Web SlideShare for Ecology Notes Quiz in CanvasFood Web SlideShare for Ecology Notes Quiz in Canvas
Food Web SlideShare for Ecology Notes Quiz in CanvasAlexandraSwartzwelde
 
Practical Research 1: Nature of Inquiry and Research.pptx
Practical Research 1: Nature of Inquiry and Research.pptxPractical Research 1: Nature of Inquiry and Research.pptx
Practical Research 1: Nature of Inquiry and Research.pptxKatherine Villaluna
 
Odontogenesis and its related anomiles.pptx
Odontogenesis and its related anomiles.pptxOdontogenesis and its related anomiles.pptx
Odontogenesis and its related anomiles.pptxMennat Allah Alkaram
 
ACTIVIDAD DE CLASE No 1 sopa de letras.docx
ACTIVIDAD DE CLASE No 1 sopa de letras.docxACTIVIDAD DE CLASE No 1 sopa de letras.docx
ACTIVIDAD DE CLASE No 1 sopa de letras.docxMaria Lucia Céspedes
 
2.20.24 The March on Washington for Jobs and Freedom.pptx
2.20.24 The March on Washington for Jobs and Freedom.pptx2.20.24 The March on Washington for Jobs and Freedom.pptx
2.20.24 The March on Washington for Jobs and Freedom.pptxMaryPotorti1
 
2.20.24 Asian Americans and the Myth of the Model Minority.pptx
2.20.24 Asian Americans and the Myth of the Model Minority.pptx2.20.24 Asian Americans and the Myth of the Model Minority.pptx
2.20.24 Asian Americans and the Myth of the Model Minority.pptxMaryPotorti1
 
catch-up-friday-ARALING PNLIPUNAN SOCIAL JUSTICE AND HUMAN RIGHTS
catch-up-friday-ARALING PNLIPUNAN SOCIAL JUSTICE AND HUMAN RIGHTScatch-up-friday-ARALING PNLIPUNAN SOCIAL JUSTICE AND HUMAN RIGHTS
catch-up-friday-ARALING PNLIPUNAN SOCIAL JUSTICE AND HUMAN RIGHTSCarlaNicolas7
 
BÀI TẬP BỔ TRỢ 4 KĨ NĂNG TIẾNG ANH LỚP 8 - HK2 - GLOBAL SUCCESS - NĂM HỌC 202...
BÀI TẬP BỔ TRỢ 4 KĨ NĂNG TIẾNG ANH LỚP 8 - HK2 - GLOBAL SUCCESS - NĂM HỌC 202...BÀI TẬP BỔ TRỢ 4 KĨ NĂNG TIẾNG ANH LỚP 8 - HK2 - GLOBAL SUCCESS - NĂM HỌC 202...
BÀI TẬP BỔ TRỢ 4 KĨ NĂNG TIẾNG ANH LỚP 8 - HK2 - GLOBAL SUCCESS - NĂM HỌC 202...Nguyen Thanh Tu Collection
 
New Features in the Odoo 17 Sales Module
New Features in  the Odoo 17 Sales ModuleNew Features in  the Odoo 17 Sales Module
New Features in the Odoo 17 Sales ModuleCeline George
 
11 CI SINIF SINAQLARI - 2-2023-Aynura-Hamidova.pdf
11 CI SINIF SINAQLARI - 2-2023-Aynura-Hamidova.pdf11 CI SINIF SINAQLARI - 2-2023-Aynura-Hamidova.pdf
11 CI SINIF SINAQLARI - 2-2023-Aynura-Hamidova.pdfAynouraHamidova
 
Overview of Databases and Data Modelling-1.pdf
Overview of Databases and Data Modelling-1.pdfOverview of Databases and Data Modelling-1.pdf
Overview of Databases and Data Modelling-1.pdfChristalin Nelson
 
BTKi in Treatment Of Chronic Lymphocytic Leukemia
BTKi in Treatment Of Chronic Lymphocytic LeukemiaBTKi in Treatment Of Chronic Lymphocytic Leukemia
BTKi in Treatment Of Chronic Lymphocytic LeukemiaFaheema Hasan
 

Recently uploaded (20)

Plant Genetic Resources, Germplasm, gene pool - Copy.pptx
Plant Genetic Resources, Germplasm, gene pool - Copy.pptxPlant Genetic Resources, Germplasm, gene pool - Copy.pptx
Plant Genetic Resources, Germplasm, gene pool - Copy.pptx
 
first section physiology laboratory.pptx
first section physiology laboratory.pptxfirst section physiology laboratory.pptx
first section physiology laboratory.pptx
 
Andreas Schleicher - 20 Feb 2024 - How pop music, podcasts, and Tik Tok are i...
Andreas Schleicher - 20 Feb 2024 - How pop music, podcasts, and Tik Tok are i...Andreas Schleicher - 20 Feb 2024 - How pop music, podcasts, and Tik Tok are i...
Andreas Schleicher - 20 Feb 2024 - How pop music, podcasts, and Tik Tok are i...
 
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
BÀI TẬP BỔ TRỢ TIẾNG ANH 11 THEO ĐƠN VỊ BÀI HỌC - CẢ NĂM - CÓ FILE NGHE (GLOB...
 
Practical Research 1, Lesson 5: DESIGNING A RESEARCH PROJECT RELATED TO DAILY...
Practical Research 1, Lesson 5: DESIGNING A RESEARCH PROJECT RELATED TO DAILY...Practical Research 1, Lesson 5: DESIGNING A RESEARCH PROJECT RELATED TO DAILY...
Practical Research 1, Lesson 5: DESIGNING A RESEARCH PROJECT RELATED TO DAILY...
 
Nzinga Kika - The story of the queen
Nzinga Kika    -  The story of the queenNzinga Kika    -  The story of the queen
Nzinga Kika - The story of the queen
 
Overview of Databases and Data Modelling-2.pdf
Overview of Databases and Data Modelling-2.pdfOverview of Databases and Data Modelling-2.pdf
Overview of Databases and Data Modelling-2.pdf
 
11 CI SINIF SINAQLARI - 10-2023-Aynura-Hamidova.pdf
11 CI SINIF SINAQLARI - 10-2023-Aynura-Hamidova.pdf11 CI SINIF SINAQLARI - 10-2023-Aynura-Hamidova.pdf
11 CI SINIF SINAQLARI - 10-2023-Aynura-Hamidova.pdf
 
Food Web SlideShare for Ecology Notes Quiz in Canvas
Food Web SlideShare for Ecology Notes Quiz in CanvasFood Web SlideShare for Ecology Notes Quiz in Canvas
Food Web SlideShare for Ecology Notes Quiz in Canvas
 
Practical Research 1: Nature of Inquiry and Research.pptx
Practical Research 1: Nature of Inquiry and Research.pptxPractical Research 1: Nature of Inquiry and Research.pptx
Practical Research 1: Nature of Inquiry and Research.pptx
 
Odontogenesis and its related anomiles.pptx
Odontogenesis and its related anomiles.pptxOdontogenesis and its related anomiles.pptx
Odontogenesis and its related anomiles.pptx
 
ACTIVIDAD DE CLASE No 1 sopa de letras.docx
ACTIVIDAD DE CLASE No 1 sopa de letras.docxACTIVIDAD DE CLASE No 1 sopa de letras.docx
ACTIVIDAD DE CLASE No 1 sopa de letras.docx
 
2.20.24 The March on Washington for Jobs and Freedom.pptx
2.20.24 The March on Washington for Jobs and Freedom.pptx2.20.24 The March on Washington for Jobs and Freedom.pptx
2.20.24 The March on Washington for Jobs and Freedom.pptx
 
2.20.24 Asian Americans and the Myth of the Model Minority.pptx
2.20.24 Asian Americans and the Myth of the Model Minority.pptx2.20.24 Asian Americans and the Myth of the Model Minority.pptx
2.20.24 Asian Americans and the Myth of the Model Minority.pptx
 
catch-up-friday-ARALING PNLIPUNAN SOCIAL JUSTICE AND HUMAN RIGHTS
catch-up-friday-ARALING PNLIPUNAN SOCIAL JUSTICE AND HUMAN RIGHTScatch-up-friday-ARALING PNLIPUNAN SOCIAL JUSTICE AND HUMAN RIGHTS
catch-up-friday-ARALING PNLIPUNAN SOCIAL JUSTICE AND HUMAN RIGHTS
 
BÀI TẬP BỔ TRỢ 4 KĨ NĂNG TIẾNG ANH LỚP 8 - HK2 - GLOBAL SUCCESS - NĂM HỌC 202...
BÀI TẬP BỔ TRỢ 4 KĨ NĂNG TIẾNG ANH LỚP 8 - HK2 - GLOBAL SUCCESS - NĂM HỌC 202...BÀI TẬP BỔ TRỢ 4 KĨ NĂNG TIẾNG ANH LỚP 8 - HK2 - GLOBAL SUCCESS - NĂM HỌC 202...
BÀI TẬP BỔ TRỢ 4 KĨ NĂNG TIẾNG ANH LỚP 8 - HK2 - GLOBAL SUCCESS - NĂM HỌC 202...
 
New Features in the Odoo 17 Sales Module
New Features in  the Odoo 17 Sales ModuleNew Features in  the Odoo 17 Sales Module
New Features in the Odoo 17 Sales Module
 
11 CI SINIF SINAQLARI - 2-2023-Aynura-Hamidova.pdf
11 CI SINIF SINAQLARI - 2-2023-Aynura-Hamidova.pdf11 CI SINIF SINAQLARI - 2-2023-Aynura-Hamidova.pdf
11 CI SINIF SINAQLARI - 2-2023-Aynura-Hamidova.pdf
 
Overview of Databases and Data Modelling-1.pdf
Overview of Databases and Data Modelling-1.pdfOverview of Databases and Data Modelling-1.pdf
Overview of Databases and Data Modelling-1.pdf
 
BTKi in Treatment Of Chronic Lymphocytic Leukemia
BTKi in Treatment Of Chronic Lymphocytic LeukemiaBTKi in Treatment Of Chronic Lymphocytic Leukemia
BTKi in Treatment Of Chronic Lymphocytic Leukemia
 

Mirella Lapata - 2017 - Invited Keynote: Translating from Multiple Modalities to Text and Back

  • 1. Motivation Case Studies Translating from Multiple Modalities to Text and Back Mirella Lapata School of Informatics University of Edinburgh mlap@inf.ed.ac.uk 1 / 70
  • 2. Motivation Case Studies Research Goal Methodology Global Internet User Survey 72% of Internet users find it frustrating to get irrelevant information when web searching! Source: www.internetsociety.org/survey 2 / 70
  • 3. Motivation Case Studies Research Goal Methodology What Causes User Frustration? User frustration Multimodal information needs Information overload Inaccessible content text, DB, SM images, video songs, code formulas too many hits contradiction duplication no structure children 2nd language learners laypersons disabled users 3 / 70
  • 4. Motivation Case Studies Research Goal Methodology NLP Comes to the Rescue! By rendering information more accessible by translat- ing within the same language and between language and different modalities. 4 / 70
  • 5. Motivation Case Studies Research Goal Methodology NLP Comes to the Rescue! riding a horse 5 / 70
  • 6. Motivation Case Studies Research Goal Methodology NLP Comes to the Rescue! riding a horse define function with argument n if n is not an integer value, throw a TypeError exception 5 / 70
  • 7. Motivation Case Studies Research Goal Methodology NLP Comes to the Rescue! riding a horse define function with argument n if n is not an integer value, throw a TypeError exception Suggs rushed for 82 yards and scored a touchdown. 5 / 70
  • 8. Motivation Case Studies Research Goal Methodology NLP Comes to the Rescue! riding a horse define function with argument n if n is not an integer value, throw a TypeError exception Suggs rushed for 82 yards and scored a touchdown. The Port Authority gave per- mission to exterminate Snowy Owls at NY City airports. 5 / 70
  • 9. Motivation Case Studies Research Goal Methodology NLP Comes to the Rescue! riding a horse define function with argument n if n is not an integer value, throw a TypeError exception Suggs rushed for 82 yards and scored a touchdown. The Port Authority gave per- mission to exterminate Snowy Owls at NY City airports. Which animals eat owls? 5 / 70
  • 10. Motivation Case Studies Research Goal Methodology Big Data 90% of the data in the world today has been created in the last two years alone (Science Daily, 22 May, 2013). At least 2.5 quintillion bytes of data is produced every day! 6 / 70
  • 11. Motivation Case Studies Research Goal Methodology A Brief History of Neural Networks Source: http://qingkaikong.blogspot.com/ 7 / 70
  • 12. Motivation Case Studies Research Goal Methodology Encoder-Decoder Modeling Framework Kalchbrenner and Blunsom (2013); Cho et al. (2014); Sutskever et al. (2014); Karpathy and Fei-Fei (2015); Vinyals et al. (2015). Source: https://medium.com/@felixhill/ 8 / 70
  • 13. Motivation Case Studies Research Goal Methodology Encoder-Decoder Modeling Framework 1 End-to-end training All parameters are simultaneously optimized to minimize a loss function on the network’s output. 2 Distributed representations share strength Better exploitation of word and phrase similarities. 3 Better exploitation of context We can use a much bigger context – both source and partial target text – to translate more accurately. 9 / 70
  • 14. Motivation Case Studies Research Goal Methodology Encoder-Decoder Modeling Framework 1 End-to-end training All parameters are simultaneously optimized to minimize a loss function on the network’s output. 2 Distributed representations share strength Better exploitation of word and phrase similarities. 3 Better exploitation of context We can use a much bigger context – both source and partial target text – to translate more accurately. Essentially a Conditional Recurrent Language Model! 9 / 70
  • 15. Motivation Case Studies Research Goal Methodology Encoder-Decoder Modeling Framework The cat sat on the mat The cat sat on the mat 10 / 70
  • 16. Motivation Case Studies Research Goal Methodology Encoder-Decoder Modeling Framework Seven Days is aSeven Days is a Chinese restaurant Inform(name=Seven Days, food=Chinese) 11 / 70
  • 24. Motivation Case Studies The Simplication Task Language to Code Movie Summarization In the Remainder of the Talk We will look at the encoder-decoder framework across tasks and along these dimensions: Translation different modalities same modality Data comparable parallel Training Size S M L Model encoder decoder training objective 19 / 70
  • 25. Motivation Case Studies The Simplication Task Language to Code Movie Summarization The Simplication Task 20 / 70
  • 26. Motivation Case Studies The Simplication Task Language to Code Movie Summarization The Simplification Task Goal: to make text easier to read and understand. Task: involves a broad spectrum of rewrite operations including deletion, substitution, insertion and reordering. Source Previous calculations show that, due to the solar wind (which drops 30% of the sun’s mass), Earth could escape to a higher orbit. Target Previous calculations show that Earth could escape to a higher orbit. This is due to the solar wind, which drops 30% of the sun’s mass. 21 / 70
  • 27. Motivation Case Studies The Simplication Task Language to Code Movie Summarization The Simplification Task Goal: to make text easier to read and understand. Task: involves a broad spectrum of rewrite operations including deletion, substitution, insertion and reordering. Source Previous calculations show that, due to the solar wind (which drops 30% of the sun’s mass), Earth could escape to a higher orbit. Target Previous calculations show that Earth could escape to a higher orbit. This is due to the solar wind, which drops 30% of the sun’s mass. 21 / 70
  • 28. Motivation Case Studies The Simplication Task Language to Code Movie Summarization The Simplification Task Goal: to make text easier to read and understand. Task: involves a broad spectrum of rewrite operations including deletion, substitution, insertion and reordering. Source These alterations are humble, but assist in circumventing the difficulties of ascertaining the meaning of obfuscated sentences. Target These alterations are simple, but help in getting around the difficulties of finding the meaning of confusing sentences. 22 / 70
  • 29. Motivation Case Studies The Simplication Task Language to Code Movie Summarization The Simplification Task Goal: to make text easier to read and understand. Task: involves a broad spectrum of rewrite operations including deletion, substitution, insertion and reordering. Source These alterations are humble, but assist in circumventing the difficulties of ascertaining the meaning of obfuscated sentences. Target These alterations are simple, but help in getting around the difficulties of finding the meaning of confusing sentences. 22 / 70
  • 30. Motivation Case Studies The Simplication Task Language to Code Movie Summarization The Simplification Task Goal: to make text easier to read and understand. Task: involves a broad spectrum of rewrite operations including deletion, substitution, insertion and reordering. Source These alterations are humble, but assist in circumventing the difficulties of ascertaining the meaning of obfuscated sentences. Target These alterations are simple, but help in getting around the difficulties of finding the meaning of confusing sentences. Constraints: output must be simpler, grammatical, and preserve the meaning of the input. 22 / 70
  • 31. Motivation Case Studies The Simplication Task Language to Code Movie Summarization A Tale of Two Presidents 23 / 70
  • 32. Motivation Case Studies The Simplication Task Language to Code Movie Summarization How to Simplify? Zhuetal.(2010) 24 / 70
  • 33. Motivation Case Studies The Simplication Task Language to Code Movie Summarization How to Simplify? Zhuetal.(2010) https://en.wikipedia.org/wiki/Vancouver 24 / 70
  • 34. Motivation Case Studies The Simplication Task Language to Code Movie Summarization How to Simplify? Zhuetal.(2010) https://en.wikipedia.org/wiki/Vancouver https://simple.wikipedia.org/wiki/Vancouver 24 / 70
  • 35. Motivation Case Studies The Simplication Task Language to Code Movie Summarization How to Simplify?Xuetal.(2015) 25 / 70
  • 36. Motivation Case Studies The Simplication Task Language to Code Movie Summarization How to Simplify? Translation different modalities same modality Data comparable parallel Training Size S M L Model encoder decoder training objective 26 / 70
  • 37. Motivation Case Studies The Simplication Task Language to Code Movie Summarization How to Simplify? Translation different modalities same modality Data comparable parallel Training Size S M L Model encoder decoder training objective 27 / 70
  • 38. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Deep Reinforcement Learning X = x1 x2 x3 x4 x5 ˆY = ˆy1 ˆy2 28 / 70
  • 39. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Deep Reinforcement Learning X = x1 x2 x3 x4 x5 ˆY = ˆy1 ˆy2 ˆy3 28 / 70
  • 40. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Deep Reinforcement Learning X = x1 x2 x3 x4 x5 ˆY = ˆy1 ˆy2 ˆy3 28 / 70
  • 41. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Deep Reinforcement Learning X = x1 x2 x3 x4 x5 ˆY = ˆy1 ˆy2 ˆy3 Vanilla encoder-decoder model only learns to copy. 28 / 70
  • 42. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Deep Reinforcement Learning X = x1 x2 x3 x4 x5 ˆY = ˆy1 ˆy2 ˆy3 Vanilla encoder-decoder model only learns to copy. We enforce task-specific constraints via reinforcement learning (Ranzato et al., 2016; Li et al., 2016; Narashimhan et al., 2016; Zhang and Lapata, 2017; Williams et al. 2017). 28 / 70
  • 43. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Deep Reinforcement Learning X = x1 x2 x3 x4 x5 ˆY = ˆy1 ˆy2 ˆy3 View model as an agent which reads source X. Agent takes action ˆyt ∈ V according to policy PRL( ˆyt |ˆy1:t−1, X). Agent outputs ˆY = (ˆy1, ˆy2, . . . , ˆy|ˆY| ) and receives reward r. 29 / 70
  • 44. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Deep Reinforcement Learning X = x1 x2 x3 x4 x5 ˆY = ˆy1 ˆy2 ˆy3 View model as an agent which reads source X. Agent takes action ˆyt ∈ V according to policy PRL( ˆyt |ˆy1:t−1, X). Agent outputs ˆY = (ˆy1, ˆy2, . . . , ˆy|ˆY| ) and receives reward r. 29 / 70
  • 45. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Deep Reinforcement Learning X = x1 x2 x3 x4 x5 ˆY = ˆy1 ˆy2 ˆy3 Get Action Seq. ˆY View model as an agent which reads source X. Agent takes action ˆyt ∈ V according to policy PRL( ˆyt |ˆy1:t−1, X). Agent outputs ˆY = (ˆy1, ˆy2, . . . , ˆy|ˆY| ) and receives reward r. 29 / 70
  • 46. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Deep Reinforcement Learning X = x1 x2 x3 x4 x5 ˆY = ˆy1 ˆy2 ˆy3 Get Action Seq. ˆY Simplicity Model Relevance Model Fluency Model View model as an agent which reads source X. Agent takes action ˆyt ∈ V according to policy PRL( ˆyt |ˆy1:t−1, X). Agent outputs ˆY = (ˆy1, ˆy2, . . . , ˆy|ˆY| ) and receives reward r. 29 / 70
  • 47. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Deep Reinforcement Learning X = x1 x2 x3 x4 x5 ˆY = ˆy1 ˆy2 ˆy3 Get Action Seq. ˆY Update Agent Simplicity Model Relevance Model Fluency Model REINFORCE algorithm View model as an agent which reads source X. Agent takes action ˆyt ∈ V according to policy PRL( ˆyt |ˆy1:t−1, X). Agent outputs ˆY = (ˆy1, ˆy2, . . . , ˆy|ˆY| ) and receives reward r. 29 / 70
  • 48. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Deep Reinforcement Learning X = x1 x2 x3 x4 x5 ˆY = ˆy1 ˆy2 ˆy3 Get Action Seq. ˆY Update Agent Simplicity Model Relevance Model Fluency Model REINFORCE algorithm Simplicity SARI (Xu et al., 2016), arithmetic average of n-gram precision and recall of addition, copying, and deletion. Relevance cosine similarity between vectors representing source X and predicted target ˆY. Fluency normalized sentence probability assigned by an LSTM language model trained on simple sentences. 30 / 70
  • 49. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Results: Newsela Dataset 0 0.5 1 1.5 2 2.5 3 3.5 4 2.73 3.06 3.08 3.28 3.38 MeanHumanRatings(Newsela) Hybrid EncDec PBMT-R DRESS Reference Hybrid: Narayan and Garden (2014); EncDec: vanilla encoder-decoder, PBMT-R: Wubben et al. (2012); DRESS: Deep REinforcement Sentence Simplication 31 / 70
  • 50. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Results: Wikipedia Dataset 0 0.5 1 1.5 2 2.5 3 3.5 4 2.85 3.21 3.34 3.46 3.46 MeanHumanRatings(Wikipedia) Hybrid SBMT PBMT-R DRESS Reference Hybrid: Narayan and Gardent (2014); SBMT: Xu et al. (2016); PBMT-R: Wubben et al. (2012); DRESS: own model. 32 / 70
  • 51. Motivation Case Studies The Simplication Task Language to Code Movie Summarization System Output: Obama’s Farewell Speech My fellow Americans, it has been the honor of my life to serve you. I won’t stop. In fact, I’ll be right there with you as a citizen for all my remaining days. But for now, whether you are young or whether you’re young at heart, I do have one final ask of you as your president. My fellow Americans, I ’ll be right there with you as a citizen for all my remaining days, whether you are young or young at heart, I do have one final ask of you as your president. 33 / 70
  • 52. Motivation Case Studies The Simplication Task Language to Code Movie Summarization System Output: Obama’s Farewell Speech My fellow Americans, it has been the honor of my life to serve you. I won’t stop. In fact, I’ll be right there with you as a citizen for all my remaining days. But for now, whether you are young or whether you’re young at heart, I do have one final ask of you as your president. My fellow Americans, I ’ll be right there with you as a citizen for all my remaining days, whether you are young or young at heart, I do have one final ask of you as your president. I am asking you to hold fast to that faith written into our founding documents, that idea whispered by slaves and abolitionists, that spirit sung by immigrants and homesteaders and those who march for justice. Hold fast to that faith written into our founding documents, that idea whispered by slaves and abolitionists. 33 / 70
  • 53. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Take-Home Message Sequence-to-sequence model with task-specific objective. RL framework could be used for other rewriting tasks. Training data is not perfect, will never be huge. Simplifications are decent, system performs well-out of domain. 34 / 70
  • 54. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Language to Code 35 / 70
  • 55. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Language to Code Users tell computers what to do using normal language. Archive missed calls from Android to Google Drive Text me the weather every morning Every time you are tagged in a photo on Facebook, it will be sent to Dropbox Text me the weather every morning Archive missed calls from Android to Google Drive IFTT Dataset: Quirk et al. (2015) 36 / 70
  • 56. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Language to Code Archive missed calls from Android to Google DriveArchive missed calls from Android to Google Drive 37 / 70
  • 57. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Language to Code What is the highest mountain in Alaska? (argmax $0 (and (mountain:t $0) (loc:t $0 alaska:s)) (elevation:i $0)) GEOQUERY: https://www.cs.utexas.edu/users/ml/nldata/geoquery.html 38 / 70
  • 58. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Language to Code Dallas to San Francisco leaving after 4 in the afternoon please (lambda $0 e (and (>(departure time $0) 1600:ti) (from $0 dallas:ci) (to $0 san francisco:ci))) ATIS: https://github.com/mesnilgr/is13 39 / 70
  • 59. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Language to Code Translation different modalities same modality Data comparable parallel Training Size S M L Model encoder decoder training objective 40 / 70
  • 60. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Language to Code Translation different modalities same modality Data comparable parallel Training Size S M L Model encoder decoder training objective 41 / 70
  • 61. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Sequence-to-Sequence Model Dong and Lapata (2016); Jia and Liang (2016). Sequence Sequence/Tree LSTM answer(J,(compa ny(J,'microsoft'),j ob(J),not((req_de g(J,'bscs'))))) Attention Layer LSTM what microsoft jobs do not require a bscs? Input LogicalInput Sequence Sequence Logical Utterance Encoder Decoder Form Uses minimal domain (and grammar) knowledge General model, can be used across meaning representations It is not guaranteed to generate well-formed trees. 42 / 70
  • 62. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Sequence-to-Tree Model (lambda $0 e (and (>(departure time $0) 1600:ti) (from $0 dallas:ci))) root lambda $0 e < n > and < n > > < n > departure time $0 1600:ti < n > from $0 dallas:ci 43 / 70
  • 63. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Sequence-to-Tree Model (lambda $0 e (and (>(departure time $0) 1600:ti) (from $0 dallas:ci))) root lambda $0 e < n > and < n > > < n > departure time $0 1600:ti < n > from $0 dallas:ci 43 / 70
  • 64. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Sequence-to-Tree Model (lambda $0 e (and (>(departure time $0) 1600:ti) (from $0 dallas:ci))) root lambda $0 e < n > and < n > > < n > departure time $0 1600:ti < n > from $0 dallas:ci 43 / 70
  • 65. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Sequence-to-Tree Model (lambda $0 e (and (>(departure time $0) 1600:ti) (from $0 dallas:ci))) root lambda $0 e < n > and < n > > < n > departure time $0 1600:ti < n > from $0 dallas:ci 43 / 70
  • 66. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Sequence-to-Tree Model (lambda $0 e (and (>(departure time $0) 1600:ti) (from $0 dallas:ci))) root lambda $0 e < n > and < n > > < n > departure time $0 1600:ti < n > from $0 dallas:ci 43 / 70
  • 67. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Sequence-to-Tree Model (lambda $0 e (and (>(departure time $0) 1600:ti) (from $0 dallas:ci))) root lambda $0 e < n > and < n > > < n > departure time $0 1600:ti < n > from $0 dallas:ci 43 / 70
  • 68. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Sequence-to-Tree Model and LSTM LSTM LSTM LSTM <n> <n> </s> > LSTM LSTM LSTM LSTM <n> 1600:ti </s> from LSTM LSTM LSTM LSTM $0 dallas:ci </s> LSTM LSTM departure _time $0 LSTM </s> LSTM LSTM lambda $0 e <n> LSTM LSTM LSTM </s>LSTM LSTM 44 / 70
  • 69. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Sequence-to-Tree Modeltrees in decoder LSTM LSTM LSTM LSTM lambda $0 e and <n> LSTM LSTM LSTM LSTM LSTM LSTM <n> <n> </s> LSTM </s> from LSTM LSTM LSTM LSTM $0 dallas:ci </s>> LSTM LSTM LSTM LSTM <n> 1600:ti </s>LSTM LSTM departure _time $0 LSTM </s> ee >> LS LLLLLLLSS LS <<<<</<n>> nndddddddddddd LST LLLLLLLLSSSSSSSTT <<n< LLLLLSSSSSSTT LST > <<<<<<<<<<<//s LLLLLLLSSSSSSTT LLLLLLLSSSSSSTT >>> <<<<<nn>>>>>>>>>>>>>>> n><<<<<<<<<<n><<< >> LSTM LLLLLLLLLSSSSSSSSSTTTM < >> 1111111111166<n>< > 45 / 70
  • 70. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Results: JOBS 70 75 80 85 90 95 100 90.7 87.1 90 JOBS Dataset Accuracy(%) DCS Seq2Seq Seq2Tree 46 / 70
  • 71. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Results: GEOQUERY 70 75 80 85 90 95 100 88.9 84.6 87.1 GEOQUERY Dataset Accuracy(%) TISP Seq2Seq Seq2Tree 47 / 70
  • 72. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Results: IFTT 50 60 70 80 90 100 66.5 72.9 74.2 IFTT Dataset BalancedF1(%) possclass Seq2Seq Seq2Tree 48 / 70
  • 73. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Structure is Coming Back! Recurrent Neural Network Grammars. Chris Dyer, Adhiguna Kun- coro, Miguel Ballesteros and Noah A. Smith. NAACL, 2016. Incorporating Structural alignment Biases into an Attentional Neural Translation Model. Trevor Cohn, Cong Duy Vu Hoang, Ekaterina Vymolova, Kaisheng Yao, Chris Dyer, and Gholamreza Haffari. NAACL, 2016. Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling. Diego Marcheggiani, and Ivan Titov. EMNLP 2017. Structured Attention Networks. Yoon Kim, Carl Denton, Loung Hoang, Alexander M. Rush. ICLR 2017. 49 / 70
  • 74. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Take-home Message Sequence-to-tree model, hierarchical tree decoder. Training data small but strictly parallel. Scaling: cross-domain, cross-language training. Neural nets can generate structure even though they don’t know what it is. 50 / 70
  • 75. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Movie Summarization 51 / 70
  • 76. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Why Movies? The litmus test for understanding (from NLP and CV perspective) Movies are naturally multimodal and not sequential. Is it possible to encode a movie into a vector? 52 / 70
  • 77. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Why Summarization? Anyone can do it! Here’s eight-year old Tru, summarizing Zootopia. 53 / 70
  • 78. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Movie Profile Generation The Silence of the Lambs can be described as tense, captivating, and suspenseful. The plot revolves around special agents, mind games, and a psychopath. The main genres are thriller and crime. In terms of style, The Silence of the Lambs stars a strong female character. In approach, it is serious and re- alistic. It is located in Maryland and Virginia. The Silence of the Lambs takes place in the 1990s. It is based on a book. The movie has received attention for being a modern classic, an Oscar winner, and a blockbuster. Note that The Silence of the Lambs involves brief nudity and sexual content. 54 / 70
  • 79. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Movie Profile Generation Mood: Suspenseful, Captivating, Tense, Scary Plot: Serial Killer, Special Agents, Investigation, Mind Game, Psychopath, Crimes, Deadly, Law Enforcement, Mind and Soul, Rivalry Genre: Crime, Thriller Style: Strong Female Presence Attitude: Serious, Realistic Place: Maryland, USA, Virginia Period: 20th Century, 90s Based on: Based on Book Praise: Award Winner, Blockbuster, Critically Ac- claimed, Oscar Winner, Modern Classic, Prestigious Awards Flag: Brief Nudity, Sexual Content, Strong Violent Content 55 / 70
  • 80. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Movie Profile Generation Mood: Suspenseful, Captivating, Tense, Scary Plot: Serial Killer, Special Agents, Investigation, Mind Game, Psychopath, Crimes, Deadly, Law Enforcement, Mind and Soul, Rivalry Genre: Crime, Thriller Style: Strong Female Presence Attitude: Serious, Realistic Place: Maryland, USA, Virginia Period: 20th Century, 90s Based on: Based on Book Praise: Award Winner, Blockbuster, Critically Ac- claimed, Oscar Winner, Modern Classic, Prestigious Awards Flag: Brief Nudity, Sexual Content, Strong Violent Content 55 / 70
  • 81. Motivation Case Studies The Simplication Task Language to Code Movie Summarization How to Generate Movie Profiles? Scripts (1,280), taken from sources available on-line Drama (667), Thriller (452), Comedy (379), Action (288) Paired with corresponding movies (video) and closed captions Paired with Jinni meta data (movie attributes and profiles) SCRIPTBASE: Gorinski and Lapata (2015) 56 / 70
  • 82. Motivation Case Studies The Simplication Task Language to Code Movie Summarization How to Generate Movie Profiles? Translation different modalities same modality Data comparable parallel Training Size S M L Model encoder decoder training objective 57 / 70
  • 83. Motivation Case Studies The Simplication Task Language to Code Movie Summarization How to Generate Movie Profiles? Translation different modalities same modality Data comparable parallel Training Size S M L Model encoder decoder training objective 58 / 70
  • 84. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Database-to-Sequence Model LSTM Attention LayerLSTM at microsoft jobs do not require a The Silence of the Lambs can be described as tense, captivating, and suspenseful. The plot revolves around special agents, mind games, and a psychopath. The main genres are thriller and crime. In terms of style, The Silence of the Lambs stars a strong female character. In approach, it is serious and re- alistic. It is located in Maryland and Virginia. The Silence of the Lambs takes place in the 1990s. It is based on a book. The movie has received attention for being a modern classic, an Oscar winner, and a blockbuster. Note that The Silence of the Lambs involves brief nudity and sexual content. 59 / 70
  • 85. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Database-to-Sequence Model LSTM Attention LayerLSTM at microsoft jobs do not require a The Silence of the Lambs can be described as tense, captivating, and suspenseful. The plot revolves around special agents, mind games, and a psychopath. The main genres are thriller and crime. In terms of style, The Silence of the Lambs stars a strong female character. In approach, it is serious and re- alistic. It is located in Maryland and Virginia. The Silence of the Lambs takes place in the 1990s. It is based on a book. The movie has received attention for being a modern classic, an Oscar winner, and a blockbuster. Note that The Silence of the Lambs involves brief nudity and sexual content. 60 / 70
  • 86. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Database-to-Sequence Model LSTM Attention Layer LSTM at microsoft jobs o not require a The Silence of the Lambs can be described as tense, captivating, and suspenseful. The plot revolves around special agents, mind games, and a psychopath. The main genres are thriller and crime. In terms of style, The Silence of the Lambs stars a strong female character. In approach, it is serious and re- alistic. It is located in Maryland and Virginia. The Silence of the Lambs takes place in the 1990s. It is based on a book. The movie has received attention for being a modern classic, an Oscar winner, and a blockbuster. Note that The Silence of the Lambs involves brief nudity and sexual content. 61 / 70
  • 87. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Content Selection Wen et al. (2015, 2016) labelset(genre1=’action’,genre2=’sci-fi’,…) 0 ... 1 d0 rt 1 0 Ct ot it ft DA cell LSTM cell wt ht-1 wt ht-1 wt ht-1 wt ht-1 wt ht-1 dt-1 dt ht CS cell 62 / 70
  • 88. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Alignment between Video, Screenplay and CCs subshot1 DR. LECTER’S CELL is coming slowly INTO VIEW... Behind its barred front wall is a second barrier of stout nylon net... Sparse, bolted- down furniture, many softcover books and papers. On the walls, extraordinarily detailed, skillful drawings, mostly European cityscapes, in charcoal or crayon. Clarice stops, at a polite distance from her bars, clearing her throat. CLARICE Dr. Lecter... My name is Clarice Starling. May I talk with you? DR. LECTER Good morning. CUTING BETWEEN THEM as Clarice comes a measured distance closer. DR. LECTER You’re one of Jack Crawford’s, aren’t you? CLARICE I am, yes. Dr LECTER May I see your credentials? 00:04:38 –> 00:04:39 Dr. Lecter... My name is Clarice Starling. May I talk with you? 00:04:40 –> 00:04:41 Good morning. 00:04:45 –> 00:04:46 You’re one of Jack Crawford’s, aren’t you? 00:04:47 –> 00:04:48 I am, yes. 00:04:49 –> 00:04:50 May I see your credentials? closed captions closed captions screenplayvideo subshot2 63 / 70
  • 89. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Multi-label Assignment Using SPENs w1 w2 wn i1 i2 im x ......... F(x) g1 g2 ...g3 gL EglobalElocal .... . . . sum Structured Prediction Energy Networks (Belanger and McCallum 2016) First network generates f-dimensional feature vector F(x) for input x. Second network finds label assignment g s.t. min g Ex (g) = Elocal x (g) + E global x (g) Features: words, sentiment, character graph, interactions, objects in scene, scene duration, movement in scene. 64 / 70
  • 90. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Attribute Evaluation Mood Plot Genre Attitude Place Flag 0 20 40 60 80 100 52.2 38.9 55.7 71.4 50.4 47.6 59.5 44.6 65.1 72 49.6 55.9 AttributeIdentification(%F1) LibLinear SPEN 65 / 70
  • 91. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Profile Evaluation 30 40 50 60 70 80 90 100 47.5 55 60 89.17 Percentageofprofilesratedsuitable kNN Template LSTM Jinni 66 / 70
  • 92. Motivation Case Studies The Simplication Task Language to Code Movie Summarization System Output The Bourne identity can be described as suspense- ful, stylized, and rough. The plot centers around a deadly, themes of uncover truth, and investigation. The Bourne identity is an action thriller, and crime movie. In approach, the Bourne identity is realistic and serious. The Bourne identity is set, at least in part, in the Europe. Note that the Bourne identity in- volves violent content. 67 / 70
  • 93. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Take-Home Message Encoder performs content selection over set of attribute-value pairs. Neural networks are well suited to tasks involving multiple modalities. There is too much data, but not in terms of number of training examples. 68 / 70
  • 94. Motivation Case Studies The Simplication Task Language to Code Movie Summarization The Future The future is uncertain but this uncertainty is at the very heart of human creativity. Ilya Prigogine 69 / 70
  • 95. Motivation Case Studies The Simplication Task Language to Code Movie Summarization A Big Thank You to Philip Gorinski Xingxing Zhang Li Dong Siva Reddy Jianpeng Cheng Jonathan Mallinson Yang Liu Stefanos Angelidis Spandana Gella Carina Silberer Lea Frermann Laura Perez Mark Steedman Shay Cohen Rico Sennrich 70 / 70