Mirella Lapata - 2017 - Invited Keynote: Translating from Multiple Modalities to Text and Back

1. Motivation Case Studies Translating from Multiple Modalities to Text and Back Mirella Lapata School of Informatics University of Edinburgh mlap@inf.ed.ac.uk 1 / 70

2. Motivation Case Studies Research Goal Methodology Global Internet User Survey 72% of Internet users ﬁnd it frustrating to get irrelevant information when web searching! Source: www.internetsociety.org/survey 2 / 70

3. Motivation Case Studies Research Goal Methodology What Causes User Frustration? User frustration Multimodal information needs Information overload Inaccessible content text, DB, SM images, video songs, code formulas too many hits contradiction duplication no structure children 2nd language learners laypersons disabled users 3 / 70

4. Motivation Case Studies Research Goal Methodology NLP Comes to the Rescue! By rendering information more accessible by translating within the same language and between language and different modalities. 4 / 70

5. Motivation Case Studies Research Goal Methodology NLP Comes to the Rescue! riding a horse 5 / 70

6. Motivation Case Studies Research Goal Methodology NLP Comes to the Rescue! riding a horse define function with argument n if n is not an integer value, throw a TypeError exception 5 / 70

7. Motivation Case Studies Research Goal Methodology NLP Comes to the Rescue! riding a horse define function with argument n if n is not an integer value, throw a TypeError exception Suggs rushed for 82 yards and scored a touchdown. 5 / 70

8. Motivation Case Studies Research Goal Methodology NLP Comes to the Rescue! riding a horse define function with argument n if n is not an integer value, throw a TypeError exception Suggs rushed for 82 yards and scored a touchdown. The Port Authority gave per- mission to exterminate Snowy Owls at NY City airports. 5 / 70

9. Motivation Case Studies Research Goal Methodology NLP Comes to the Rescue! riding a horse define function with argument n if n is not an integer value, throw a TypeError exception Suggs rushed for 82 yards and scored a touchdown. The Port Authority gave per- mission to exterminate Snowy Owls at NY City airports. Which animals eat owls? 5 / 70

10. Motivation Case Studies Research Goal Methodology Big Data 90% of the data in the world today has been created in the last two years alone (Science Daily, 22 May, 2013). At least 2.5 quintillion bytes of data is produced every day! 6 / 70

11. Motivation Case Studies Research Goal Methodology A Brief History of Neural Networks Source: http://qingkaikong.blogspot.com/ 7 / 70

12. Motivation Case Studies Research Goal Methodology Encoder-Decoder Modeling Framework Kalchbrenner and Blunsom (2013); Cho et al. (2014); Sutskever et al. (2014); Karpathy and Fei-Fei (2015); Vinyals et al. (2015). Source: https://medium.com/@felixhill/ 8 / 70

13. Motivation Case Studies Research Goal Methodology Encoder-Decoder Modeling Framework 1 End-to-end training All parameters are simultaneously optimized to minimize a loss function on the network’s output. 2 Distributed representations share strength Better exploitation of word and phrase similarities. 3 Better exploitation of context We can use a much bigger context – both source and partial target text – to translate more accurately. 9 / 70

14. Motivation Case Studies Research Goal Methodology Encoder-Decoder Modeling Framework 1 End-to-end training All parameters are simultaneously optimized to minimize a loss function on the network’s output. 2 Distributed representations share strength Better exploitation of word and phrase similarities. 3 Better exploitation of context We can use a much bigger context – both source and partial target text – to translate more accurately. Essentially a Conditional Recurrent Language Model! 9 / 70

15. Motivation Case Studies Research Goal Methodology Encoder-Decoder Modeling Framework The cat sat on the mat The cat sat on the mat 10 / 70

16. Motivation Case Studies Research Goal Methodology Encoder-Decoder Modeling Framework Seven Days is aSeven Days is a Chinese restaurant Inform(name=Seven Days, food=Chinese) 11 / 70

17. Motivation Case Studies Research Goal Methodology Encoder-Decoder Modeling Framework 12 / 70

18. Motivation Case Studies Research Goal Methodology The Scream 13 / 70

24. Motivation Case Studies The Simplication Task Language to Code Movie Summarization In the Remainder of the Talk We will look at the encoder-decoder framework across tasks and along these dimensions: Translation different modalities same modality Data comparable parallel Training Size S M L Model encoder decoder training objective 19 / 70

25. Motivation Case Studies The Simplication Task Language to Code Movie Summarization The Simplication Task 20 / 70

26. Motivation Case Studies The Simplication Task Language to Code Movie Summarization The Simpliﬁcation Task Goal: to make text easier to read and understand. Task: involves a broad spectrum of rewrite operations including deletion, substitution, insertion and reordering. Source Previous calculations show that, due to the solar wind (which drops 30% of the sun’s mass), Earth could escape to a higher orbit. Target Previous calculations show that Earth could escape to a higher orbit. This is due to the solar wind, which drops 30% of the sun’s mass. 21 / 70

27. Motivation Case Studies The Simplication Task Language to Code Movie Summarization The Simpliﬁcation Task Goal: to make text easier to read and understand. Task: involves a broad spectrum of rewrite operations including deletion, substitution, insertion and reordering. Source Previous calculations show that, due to the solar wind (which drops 30% of the sun’s mass), Earth could escape to a higher orbit. Target Previous calculations show that Earth could escape to a higher orbit. This is due to the solar wind, which drops 30% of the sun’s mass. 21 / 70

28. Motivation Case Studies The Simplication Task Language to Code Movie Summarization The Simplification Task Goal: to make text easier to read and understand. Task: involves a broad spectrum of rewrite operations including deletion, substitution, insertion and reordering. Source These alterations are humble, but assist in circumventing the difficulties of ascertaining the meaning of obfuscated sentences. Target These alterations are simple, but help in getting around the difficulties of finding the meaning of confusing sentences. 22 / 70

29. Motivation Case Studies The Simplication Task Language to Code Movie Summarization The Simplification Task Goal: to make text easier to read and understand. Task: involves a broad spectrum of rewrite operations including deletion, substitution, insertion and reordering. Source These alterations are humble, but assist in circumventing the difficulties of ascertaining the meaning of obfuscated sentences. Target These alterations are simple, but help in getting around the difficulties of finding the meaning of confusing sentences. 22 / 70

30. Motivation Case Studies The Simplication Task Language to Code Movie Summarization The Simplification Task Goal: to make text easier to read and understand. Task: involves a broad spectrum of rewrite operations including deletion, substitution, insertion and reordering. Source These alterations are humble, but assist in circumventing the difficulties of ascertaining the meaning of obfuscated sentences. Target These alterations are simple, but help in getting around the difficulties of finding the meaning of confusing sentences. Constraints: output must be simpler, grammatical, and preserve the meaning of the input. 22 / 70

31. Motivation Case Studies The Simplication Task Language to Code Movie Summarization A Tale of Two Presidents 23 / 70

32. Motivation Case Studies The Simplication Task Language to Code Movie Summarization How to Simplify? Zhuetal.(2010) 24 / 70

33. Motivation Case Studies The Simplication Task Language to Code Movie Summarization How to Simplify? Zhuetal.(2010) https://en.wikipedia.org/wiki/Vancouver 24 / 70

34. Motivation Case Studies The Simplication Task Language to Code Movie Summarization How to Simplify? Zhuetal.(2010) https://en.wikipedia.org/wiki/Vancouver https://simple.wikipedia.org/wiki/Vancouver 24 / 70

35. Motivation Case Studies The Simplication Task Language to Code Movie Summarization How to Simplify?Xuetal.(2015) 25 / 70

36. Motivation Case Studies The Simplication Task Language to Code Movie Summarization How to Simplify? Translation different modalities same modality Data comparable parallel Training Size S M L Model encoder decoder training objective 26 / 70

37. Motivation Case Studies The Simplication Task Language to Code Movie Summarization How to Simplify? Translation different modalities same modality Data comparable parallel Training Size S M L Model encoder decoder training objective 27 / 70

38. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Deep Reinforcement Learning X = x1 x2 x3 x4 x5 ˆY = ˆy1 ˆy2 28 / 70

39. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Deep Reinforcement Learning X = x1 x2 x3 x4 x5 ˆY = ˆy1 ˆy2 ˆy3 28 / 70

40. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Deep Reinforcement Learning X = x1 x2 x3 x4 x5 ˆY = ˆy1 ˆy2 ˆy3 28 / 70

41. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Deep Reinforcement Learning X = x1 x2 x3 x4 x5 ˆY = ˆy1 ˆy2 ˆy3 Vanilla encoder-decoder model only learns to copy. 28 / 70

42. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Deep Reinforcement Learning X = x1 x2 x3 x4 x5 ˆY = ˆy1 ˆy2 ˆy3 Vanilla encoder-decoder model only learns to copy. We enforce task-speciﬁc constraints via reinforcement learning (Ranzato et al., 2016; Li et al., 2016; Narashimhan et al., 2016; Zhang and Lapata, 2017; Williams et al. 2017). 28 / 70

43. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Deep Reinforcement Learning X = x1 x2 x3 x4 x5 ˆY = ˆy1 ˆy2 ˆy3 View model as an agent which reads source X. Agent takes action ˆyt ∈ V according to policy PRL( ˆyt |ˆy1:t−1, X). Agent outputs ˆY = (ˆy1, ˆy2, . . . , ˆy|ˆY| ) and receives reward r. 29 / 70

44. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Deep Reinforcement Learning X = x1 x2 x3 x4 x5 ˆY = ˆy1 ˆy2 ˆy3 View model as an agent which reads source X. Agent takes action ˆyt ∈ V according to policy PRL( ˆyt |ˆy1:t−1, X). Agent outputs ˆY = (ˆy1, ˆy2, . . . , ˆy|ˆY| ) and receives reward r. 29 / 70

45. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Deep Reinforcement Learning X = x1 x2 x3 x4 x5 ˆY = ˆy1 ˆy2 ˆy3 Get Action Seq. ˆY View model as an agent which reads source X. Agent takes action ˆyt ∈ V according to policy PRL( ˆyt |ˆy1:t−1, X). Agent outputs ˆY = (ˆy1, ˆy2, . . . , ˆy|ˆY| ) and receives reward r. 29 / 70

46. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Deep Reinforcement Learning X = x1 x2 x3 x4 x5 ˆY = ˆy1 ˆy2 ˆy3 Get Action Seq. ˆY Simplicity Model Relevance Model Fluency Model View model as an agent which reads source X. Agent takes action ˆyt ∈ V according to policy PRL( ˆyt |ˆy1:t−1, X). Agent outputs ˆY = (ˆy1, ˆy2, . . . , ˆy|ˆY| ) and receives reward r. 29 / 70

47. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Deep Reinforcement Learning X = x1 x2 x3 x4 x5 ˆY = ˆy1 ˆy2 ˆy3 Get Action Seq. ˆY Update Agent Simplicity Model Relevance Model Fluency Model REINFORCE algorithm View model as an agent which reads source X. Agent takes action ˆyt ∈ V according to policy PRL( ˆyt |ˆy1:t−1, X). Agent outputs ˆY = (ˆy1, ˆy2, . . . , ˆy|ˆY| ) and receives reward r. 29 / 70

48. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Deep Reinforcement Learning X = x1 x2 x3 x4 x5 ˆY = ˆy1 ˆy2 ˆy3 Get Action Seq. ˆY Update Agent Simplicity Model Relevance Model Fluency Model REINFORCE algorithm Simplicity SARI (Xu et al., 2016), arithmetic average of n-gram precision and recall of addition, copying, and deletion. Relevance cosine similarity between vectors representing source X and predicted target ˆY. Fluency normalized sentence probability assigned by an LSTM language model trained on simple sentences. 30 / 70

49. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Results: Newsela Dataset 0 0.5 1 1.5 2 2.5 3 3.5 4 2.73 3.06 3.08 3.28 3.38 MeanHumanRatings(Newsela) Hybrid EncDec PBMT-R DRESS Reference Hybrid: Narayan and Garden (2014); EncDec: vanilla encoder-decoder, PBMT-R: Wubben et al. (2012); DRESS: Deep REinforcement Sentence Simplication 31 / 70

50. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Results: Wikipedia Dataset 0 0.5 1 1.5 2 2.5 3 3.5 4 2.85 3.21 3.34 3.46 3.46 MeanHumanRatings(Wikipedia) Hybrid SBMT PBMT-R DRESS Reference Hybrid: Narayan and Gardent (2014); SBMT: Xu et al. (2016); PBMT-R: Wubben et al. (2012); DRESS: own model. 32 / 70

51. Motivation Case Studies The Simplication Task Language to Code Movie Summarization System Output: Obama’s Farewell Speech My fellow Americans, it has been the honor of my life to serve you. I won’t stop. In fact, I’ll be right there with you as a citizen for all my remaining days. But for now, whether you are young or whether you’re young at heart, I do have one ﬁnal ask of you as your president. My fellow Americans, I ’ll be right there with you as a citizen for all my remaining days, whether you are young or young at heart, I do have one ﬁnal ask of you as your president. 33 / 70

52. Motivation Case Studies The Simplication Task Language to Code Movie Summarization System Output: Obama’s Farewell Speech My fellow Americans, it has been the honor of my life to serve you. I won’t stop. In fact, I’ll be right there with you as a citizen for all my remaining days. But for now, whether you are young or whether you’re young at heart, I do have one ﬁnal ask of you as your president. My fellow Americans, I ’ll be right there with you as a citizen for all my remaining days, whether you are young or young at heart, I do have one ﬁnal ask of you as your president. I am asking you to hold fast to that faith written into our founding documents, that idea whispered by slaves and abolitionists, that spirit sung by immigrants and homesteaders and those who march for justice. Hold fast to that faith written into our founding documents, that idea whispered by slaves and abolitionists. 33 / 70

53. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Take-Home Message Sequence-to-sequence model with task-speciﬁc objective. RL framework could be used for other rewriting tasks. Training data is not perfect, will never be huge. Simpliﬁcations are decent, system performs well-out of domain. 34 / 70

54. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Language to Code 35 / 70

55. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Language to Code Users tell computers what to do using normal language. Archive missed calls from Android to Google Drive Text me the weather every morning Every time you are tagged in a photo on Facebook, it will be sent to Dropbox Text me the weather every morning Archive missed calls from Android to Google Drive IFTT Dataset: Quirk et al. (2015) 36 / 70

56. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Language to Code Archive missed calls from Android to Google DriveArchive missed calls from Android to Google Drive 37 / 70

57. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Language to Code What is the highest mountain in Alaska? (argmax $0 (and (mountain:t $0) (loc:t $0 alaska:s)) (elevation:i $0)) GEOQUERY: https://www.cs.utexas.edu/users/ml/nldata/geoquery.html 38 / 70

58. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Language to Code Dallas to San Francisco leaving after 4 in the afternoon please (lambda $0 e (and (>(departure time $0) 1600:ti) (from $0 dallas:ci) (to $0 san francisco:ci))) ATIS: https://github.com/mesnilgr/is13 39 / 70

59. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Language to Code Translation different modalities same modality Data comparable parallel Training Size S M L Model encoder decoder training objective 40 / 70

60. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Language to Code Translation different modalities same modality Data comparable parallel Training Size S M L Model encoder decoder training objective 41 / 70

61. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Sequence-to-Sequence Model Dong and Lapata (2016); Jia and Liang (2016). Sequence Sequence/Tree LSTM answer(J,(compa ny(J,'microsoft'),j ob(J),not((req_de g(J,'bscs'))))) Attention Layer LSTM what microsoft jobs do not require a bscs? Input LogicalInput Sequence Sequence Logical Utterance Encoder Decoder Form Uses minimal domain (and grammar) knowledge General model, can be used across meaning representations It is not guaranteed to generate well-formed trees. 42 / 70

62. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Sequence-to-Tree Model (lambda $0 e (and (>(departure time $0) 1600:ti) (from $0 dallas:ci))) root lambda $0 e < n > and < n > > < n > departure time $0 1600:ti < n > from $0 dallas:ci 43 / 70

68. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Sequence-to-Tree Model and LSTM LSTM LSTM LSTM <n> <n> </s> > LSTM LSTM LSTM LSTM <n> 1600:ti </s> from LSTM LSTM LSTM LSTM $0 dallas:ci </s> LSTM LSTM departure _time $0 LSTM </s> LSTM LSTM lambda $0 e <n> LSTM LSTM LSTM </s>LSTM LSTM 44 / 70

69. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Sequence-to-Tree Modeltrees in decoder LSTM LSTM LSTM LSTM lambda $0 e and <n> LSTM LSTM LSTM LSTM LSTM LSTM <n> <n> </s> LSTM </s> from LSTM LSTM LSTM LSTM $0 dallas:ci </s>> LSTM LSTM LSTM LSTM <n> 1600:ti </s>LSTM LSTM departure _time $0 LSTM </s> ee >> LS LLLLLLLSS LS <<<<</<n>> nndddddddddddd LST LLLLLLLLSSSSSSSTT <<n< LLLLLSSSSSSTT LST > <<<<<<<<<<<//s LLLLLLLSSSSSSTT LLLLLLLSSSSSSTT >>> <<<<<nn>>>>>>>>>>>>>>> n><<<<<<<<<<n><<< >> LSTM LLLLLLLLLSSSSSSSSSTTTM < >> 1111111111166<n>< > 45 / 70

70. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Results: JOBS 70 75 80 85 90 95 100 90.7 87.1 90 JOBS Dataset Accuracy(%) DCS Seq2Seq Seq2Tree 46 / 70

71. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Results: GEOQUERY 70 75 80 85 90 95 100 88.9 84.6 87.1 GEOQUERY Dataset Accuracy(%) TISP Seq2Seq Seq2Tree 47 / 70

72. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Results: IFTT 50 60 70 80 90 100 66.5 72.9 74.2 IFTT Dataset BalancedF1(%) possclass Seq2Seq Seq2Tree 48 / 70

73. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Structure is Coming Back! Recurrent Neural Network Grammars. Chris Dyer, Adhiguna Kun- coro, Miguel Ballesteros and Noah A. Smith. NAACL, 2016. Incorporating Structural alignment Biases into an Attentional Neural Translation Model. Trevor Cohn, Cong Duy Vu Hoang, Ekaterina Vymolova, Kaisheng Yao, Chris Dyer, and Gholamreza Haffari. NAACL, 2016. Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling. Diego Marcheggiani, and Ivan Titov. EMNLP 2017. Structured Attention Networks. Yoon Kim, Carl Denton, Loung Hoang, Alexander M. Rush. ICLR 2017. 49 / 70

74. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Take-home Message Sequence-to-tree model, hierarchical tree decoder. Training data small but strictly parallel. Scaling: cross-domain, cross-language training. Neural nets can generate structure even though they don’t know what it is. 50 / 70

75. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Movie Summarization 51 / 70

76. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Why Movies? The litmus test for understanding (from NLP and CV perspective) Movies are naturally multimodal and not sequential. Is it possible to encode a movie into a vector? 52 / 70

77. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Why Summarization? Anyone can do it! Here’s eight-year old Tru, summarizing Zootopia. 53 / 70

78. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Movie Proﬁle Generation The Silence of the Lambs can be described as tense, captivating, and suspenseful. The plot revolves around special agents, mind games, and a psychopath. The main genres are thriller and crime. In terms of style, The Silence of the Lambs stars a strong female character. In approach, it is serious and realistic. It is located in Maryland and Virginia. The Silence of the Lambs takes place in the 1990s. It is based on a book. The movie has received attention for being a modern classic, an Oscar winner, and a blockbuster. Note that The Silence of the Lambs involves brief nudity and sexual content. 54 / 70

79. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Movie Proﬁle Generation Mood: Suspenseful, Captivating, Tense, Scary Plot: Serial Killer, Special Agents, Investigation, Mind Game, Psychopath, Crimes, Deadly, Law Enforcement, Mind and Soul, Rivalry Genre: Crime, Thriller Style: Strong Female Presence Attitude: Serious, Realistic Place: Maryland, USA, Virginia Period: 20th Century, 90s Based on: Based on Book Praise: Award Winner, Blockbuster, Critically Ac- claimed, Oscar Winner, Modern Classic, Prestigious Awards Flag: Brief Nudity, Sexual Content, Strong Violent Content 55 / 70

80. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Movie Proﬁle Generation Mood: Suspenseful, Captivating, Tense, Scary Plot: Serial Killer, Special Agents, Investigation, Mind Game, Psychopath, Crimes, Deadly, Law Enforcement, Mind and Soul, Rivalry Genre: Crime, Thriller Style: Strong Female Presence Attitude: Serious, Realistic Place: Maryland, USA, Virginia Period: 20th Century, 90s Based on: Based on Book Praise: Award Winner, Blockbuster, Critically Ac- claimed, Oscar Winner, Modern Classic, Prestigious Awards Flag: Brief Nudity, Sexual Content, Strong Violent Content 55 / 70

81. Motivation Case Studies The Simplication Task Language to Code Movie Summarization How to Generate Movie Proﬁles? Scripts (1,280), taken from sources available on-line Drama (667), Thriller (452), Comedy (379), Action (288) Paired with corresponding movies (video) and closed captions Paired with Jinni meta data (movie attributes and proﬁles) SCRIPTBASE: Gorinski and Lapata (2015) 56 / 70

82. Motivation Case Studies The Simplication Task Language to Code Movie Summarization How to Generate Movie Proﬁles? Translation different modalities same modality Data comparable parallel Training Size S M L Model encoder decoder training objective 57 / 70

83. Motivation Case Studies The Simplication Task Language to Code Movie Summarization How to Generate Movie Proﬁles? Translation different modalities same modality Data comparable parallel Training Size S M L Model encoder decoder training objective 58 / 70

84. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Database-to-Sequence Model LSTM Attention LayerLSTM at microsoft jobs do not require a The Silence of the Lambs can be described as tense, captivating, and suspenseful. The plot revolves around special agents, mind games, and a psychopath. The main genres are thriller and crime. In terms of style, The Silence of the Lambs stars a strong female character. In approach, it is serious and realistic. It is located in Maryland and Virginia. The Silence of the Lambs takes place in the 1990s. It is based on a book. The movie has received attention for being a modern classic, an Oscar winner, and a blockbuster. Note that The Silence of the Lambs involves brief nudity and sexual content. 59 / 70

85. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Database-to-Sequence Model LSTM Attention LayerLSTM at microsoft jobs do not require a The Silence of the Lambs can be described as tense, captivating, and suspenseful. The plot revolves around special agents, mind games, and a psychopath. The main genres are thriller and crime. In terms of style, The Silence of the Lambs stars a strong female character. In approach, it is serious and realistic. It is located in Maryland and Virginia. The Silence of the Lambs takes place in the 1990s. It is based on a book. The movie has received attention for being a modern classic, an Oscar winner, and a blockbuster. Note that The Silence of the Lambs involves brief nudity and sexual content. 60 / 70

86. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Database-to-Sequence Model LSTM Attention Layer LSTM at microsoft jobs o not require a The Silence of the Lambs can be described as tense, captivating, and suspenseful. The plot revolves around special agents, mind games, and a psychopath. The main genres are thriller and crime. In terms of style, The Silence of the Lambs stars a strong female character. In approach, it is serious and realistic. It is located in Maryland and Virginia. The Silence of the Lambs takes place in the 1990s. It is based on a book. The movie has received attention for being a modern classic, an Oscar winner, and a blockbuster. Note that The Silence of the Lambs involves brief nudity and sexual content. 61 / 70

87. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Content Selection Wen et al. (2015, 2016) labelset(genre1=’action’,genre2=’sci-fi’,…) 0 ... 1 d0 rt 1 0 Ct ot it ft DA cell LSTM cell wt ht-1 wt ht-1 wt ht-1 wt ht-1 wt ht-1 dt-1 dt ht CS cell 62 / 70

88. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Alignment between Video, Screenplay and CCs subshot1 DR. LECTER’S CELL is coming slowly INTO VIEW... Behind its barred front wall is a second barrier of stout nylon net... Sparse, bolted- down furniture, many softcover books and papers. On the walls, extraordinarily detailed, skillful drawings, mostly European cityscapes, in charcoal or crayon. Clarice stops, at a polite distance from her bars, clearing her throat. CLARICE Dr. Lecter... My name is Clarice Starling. May I talk with you? DR. LECTER Good morning. CUTING BETWEEN THEM as Clarice comes a measured distance closer. DR. LECTER You’re one of Jack Crawford’s, aren’t you? CLARICE I am, yes. Dr LECTER May I see your credentials? 00:04:38 –> 00:04:39 Dr. Lecter... My name is Clarice Starling. May I talk with you? 00:04:40 –> 00:04:41 Good morning. 00:04:45 –> 00:04:46 You’re one of Jack Crawford’s, aren’t you? 00:04:47 –> 00:04:48 I am, yes. 00:04:49 –> 00:04:50 May I see your credentials? closed captions closed captions screenplayvideo subshot2 63 / 70

89. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Multi-label Assignment Using SPENs w1 w2 wn i1 i2 im x ......... F(x) g1 g2 ...g3 gL EglobalElocal .... . . . sum Structured Prediction Energy Networks (Belanger and McCallum 2016) First network generates f-dimensional feature vector F(x) for input x. Second network ﬁnds label assignment g s.t. min g Ex (g) = Elocal x (g) + E global x (g) Features: words, sentiment, character graph, interactions, objects in scene, scene duration, movement in scene. 64 / 70

90. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Attribute Evaluation Mood Plot Genre Attitude Place Flag 0 20 40 60 80 100 52.2 38.9 55.7 71.4 50.4 47.6 59.5 44.6 65.1 72 49.6 55.9 AttributeIdentiﬁcation(%F1) LibLinear SPEN 65 / 70

91. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Proﬁle Evaluation 30 40 50 60 70 80 90 100 47.5 55 60 89.17 Percentageofproﬁlesratedsuitable kNN Template LSTM Jinni 66 / 70

92. Motivation Case Studies The Simplication Task Language to Code Movie Summarization System Output The Bourne identity can be described as suspenseful, stylized, and rough. The plot centers around a deadly, themes of uncover truth, and investigation. The Bourne identity is an action thriller, and crime movie. In approach, the Bourne identity is realistic and serious. The Bourne identity is set, at least in part, in the Europe. Note that the Bourne identity involves violent content. 67 / 70

93. Motivation Case Studies The Simplication Task Language to Code Movie Summarization Take-Home Message Encoder performs content selection over set of attribute-value pairs. Neural networks are well suited to tasks involving multiple modalities. There is too much data, but not in terms of number of training examples. 68 / 70

94. Motivation Case Studies The Simplication Task Language to Code Movie Summarization The Future The future is uncertain but this uncertainty is at the very heart of human creativity. Ilya Prigogine 69 / 70

95. Motivation Case Studies The Simplication Task Language to Code Movie Summarization A Big Thank You to Philip Gorinski Xingxing Zhang Li Dong Siva Reddy Jianpeng Cheng Jonathan Mallinson Yang Liu Stefanos Angelidis Spandana Gella Carina Silberer Lea Frermann Laura Perez Mark Steedman Shay Cohen Rico Sennrich 70 / 70

Mirella Lapata - 2017 - Invited Keynote: Translating from Multiple Modalities to Text and Back

Association for Computational Linguistics

Mirella Lapata - 2017 - Invited Keynote: Translating from Multiple Modalities to Text and Back