This paper - Proposed a fully data-driven approach for sentence abstractive summarization - Combined Attention-based probabilistic model with Beam Search to generate sentence summary - Proposed method outperforms several strong baselines on headline-generation task

1. A Neural Attention Model for
Sentence Summarization
Authors: Alexander M. Rush, Sumit Chopra, Jason Weston
Conference: EMNLP 2015
Presentor: Mengsay LOEM
May 21, 2021
1

2. Overview
● Proposed a fully data-driven approach for sentence
abstractive summarization
● Combined Attention-based probabilistic model with Beam
Search to generate sentence summary
● Proposed method outperforms several strong baselines
on headline-generation task
2

3. Sentence Summarization
russian defense minister ivanov called sunday for the
creation of a joint front for combating global terrorism
russia calls for joint front against terrorism
Input
Output
Approaches:
● Compressive : deletion
● Extractive : deletion and reordering
● Abstractive : arbitrary transformation
3

4. Problems
● Compressive/Extractive methods
○ Cannot perform various summary operations
■ Paraphrasing, generalization
● Abstractive methods
○ Require linguistically-inspired constraints
○ Require syntactic transformation on input text
Can we generate an abstractive summary with
a fully data-driven approach?
4
Mostly Heuristics
Approach

5. Solution
● Fully data-driven approach for abstractive summaries
generation
○ NO syntactic transformations/linguistic structure required
○ Can be trained directly on any document-summary pair
● Utilize Attention-based Neural Networks model
○ Inspired by Neural Machine Translation*
5
* Bahdanau et al.: Neural Machine Translation by Jointly Learning to Align and Translate

6. Proposed Method: outline
6
enc:
• Bag-of-words encoder
• Convolutional encoder
• Attention-based encoder
dec:
Neural Network Language Model +
Beam Search
enc
⋯
dec
⋯
russian
defense
minister
ivanov
terrorism
<s>
russia
called
calls
for
terrorism
⋯
russia
calls
for
joint
<s>

7. Attention-based Encoder
7
dec
⋯
russian defense
<s>
russia
calls
for
against
⋯
russian
calls
for
joint
terrorism
<s>
!
Weighting with Attention
⋯ terrorism
</s>
Smoothed
version of input

8. 8
Example of Attention
Output (summary)
Input sentence
dec
⋯
russian defense
<s>
russia
calls
for
against
⋯
russian
calls
for
joint
terrorism
<s>
!
Weighting with Attention
⋯ terrorism
</s>

9. 9
Example of Attention
Output (summary)
Input sentence
dec
⋯
russian defense
<s>
russia
calls
for
against
⋯
russian
calls
for
joint
terrorism
<s>
!
Weighting with Attention
⋯ terrorism
</s>

10. 10
Input sentence
Example of Attention
Output (summary)
Smoothing window
dec
⋯
russian defense
<s>
russia
calls
for
against
⋯
russian
calls
for
joint
terrorism
<s>
!
Weighting with Attention
⋯ terrorism
</s>

11. 11
Input sentence
Example of Attention
Output (summary)
dec
⋯
russian defense
<s>
russia
calls
for
against
⋯
russian
calls
for
joint
terrorism
<s>
!
Weighting with Attention
⋯ terrorism
</s>

12. 12
Input sentence
Example of Attention
Output (summary)
dec
⋯
russian defense
<s>
russia
calls
for
against
⋯
russian
calls
for
joint
terrorism
<s>
!
Weighting with Attention
⋯ terrorism
</s>

13. 13
Input sentence
Example of Attention
Output (summary)
dec
⋯
russian defense
<s>
russia
calls
for
against
⋯
russian
calls
for
joint
terrorism
<s>
!
Weighting with Attention
⋯ terrorism
</s>

14. 14
Input sentence
Example of Attention
Output (summary)
dec
⋯
russian defense
<s>
russia
calls
for
against
⋯
russian
calls
for
joint
terrorism
<s>
!
Weighting with Attention
⋯ terrorism
</s>

15. 15
Input sentence
Example of Attention
Output (summary)
dec
⋯
russian defense
<s>
russia
calls
for
against
⋯
russian
calls
for
joint
terrorism
<s>
!
Weighting with Attention
⋯ terrorism
</s>

16. 16
Input sentence
Example of Attention
Output (summary)
dec
⋯
russian defense
<s>
russia
calls
for
against
⋯
russian
calls
for
joint
terrorism
<s>
!
Weighting with Attention
⋯ terrorism
</s>

17. Generation: Beam Search
17
russia
for
country
calls
…
defense
terrorism
russian
for
country
calls
…
defense
terrorism
russia
for
country
calls
…
defense
terrorism
russia
for
country
calls
…
defense
terrorism
russia
for
country
calls
…
defense
terrorism
[ ]
join
join
join
join
join
russia russia calls
defense
defense terrorism
russia calls for
defense terrorism country

18. Extension : Extractive Tuning
18
● Fully abstractive model cannot find extractive word
matches when necessary
○ transferring unseen proper noun phrases from the input
● Solution:
○ tuning a small set of additional features

19. Experiment
● Evaluate on Headline generation task
○ Metrics: ROUGE-1, ROUGE-2, ROUGE-L
○ Data sets:
■ Training: Annotated Gigaword data set (3.8M )
■ Evaluation: DUC-2003, 2004 (500)
● Baselines
○ Compressive: Prefix, Compress
○ Abstractive: IR,Topiary, MOSES+
○ W&L (Quasi-synchronous grammar approach)
19

20. Result : Summary Tasks
ABS : Proposed model (Attention-based encoder)
ABS+ : ABS with Extractive Tuning 20
● Only input article or LM alone is not sufficient (IR, Compress)
● Full model ABS+ scores the best, but additional extractive
features bias the system toward inputs words (useful for the ROUGE metric)
Compressive
Winning system
on the task
Statistical MT

21. Result : Encoding Ablation
21
● NNLM with no encoder performs better than n-gram LM.
● Including proposed encoders improve the model
● Ignore word order
● No use of generated
context
● Averaging over input
words
● No use of generated
context
● Allow local interactions
between words in input
sentence

22. Result : Model and Decoding Ablation
22
Pure extractive
● Attention-based encoder with Beam search give the biggest
impact
● For ROUGE, using pure extractive generation is also effective

23. Example
ABS: Interesting rewording
Input: australian foreign minister stephen smith sunday congratulated
new zealand ’s new prime minister-elect john key as he praised ousted
leader helen clark as a “ gutsy ” and respected politician .
ABS: australian foreign minister congratulates new nz pm after election
ABS+:australian foreign minister congratulates smith new zealand as
leader
Head-line: time caught up with nz ’s gutsy clark says australian fm
23

24. Example
ABS: Interesting rewording, but making MISTAKE
Input: russia ’s gas and oil giant gazprom and us oil major chevron have set
up a joint venture based in resource-rich northwestern siberia , the
interfax news agency reported thursday quoting gazprom officials .
ABS: russian oil giant chevron set up siberia joint venture
ABS+:russia ’s gazprom set up joint venture in siberia
Head-line: gazprom chevron set up joint venture
24

25. Example
good at picking keywords, but reorder words in syntactically incorrect way
Input: the white house on thursday warned iran of possible new sanctions
after the un nuclear watchdog reported that tehran had begun sensitive
nuclear work at a key site in defiance of un resolutions .
ABS: iran warns of possible new sanctions on nuclear work
ABS+:un nuclear watchdog warns iran of possible new sanctions
Head-line: us warns iran of step backward on nuclear issue
25

26. Conclusion
● Proposed a Neural Attention-based model for Abstractive
Sentence Summarization
● Fully data-driven approach
● Improve baselines scores, but the performance is far from
human's
○ Repeating semantic elements, improper generalization
26