小町研究室　ACL2016読み会 担当：小平　知範 Conference: ACL 2016 Title: Neural Summarization by Extracting Sentences and Words Authors: Jianpeng Cheng and Mirela Lapata PaperLink: http://aclweb.org/anthology/P/P16/P16-1046.pdf

1. Neural Summarization by
Extracting Sentences
and Words
Jianpeng Cheng and Mirella Lapata
ACL 2016
Presentator: Tomonori Kodaira
1

2. Intro
• Task: Single document summarization (extracting
sentences or words)
• Model: a neural network-based hierarchical
document reader or encoder and an attention-
based content extractor.
• Data: DailyMail news
2

3. Problem Formulation
• Sentence Extraction
a summary from D by selecting a subset of j
sentences. (predicting label: yL ∈ {0, 1})
• Word Extraction
a language generation task with out put vocabulary
restricted to the original document.
3

4. Data
• They make two large-scale datasets.
•
4

5. Data
(sentence extraction)
• They retrieved hundreds of thousands of news
articles and their corresponding highlights from
DailyMail.
• They designed a rule-based system that determines
whether a document sentence matches a highlight.
(Woodsend and Lapata, 2010)
5

6. Data
(word extraction dataset)
• In cases where all highlights words come from the
original document, the pair is added the dataset.
• For OOV words, they check if a neighbor, represented
by pre-trained embeddings, is in the original document.
• If they cannot find any substitutes, they discard the pair.
• word extraction dataset containing 170K articles.
6

7. Neural Summarization Model
• Key components:
• neural network-based hierarchical document reader
• attention-based hierarchical content extractor.
7

8. 8
kernel K ∈ Rc x d of width c
W ∈ Rn x d
Document Reader
(Convolutional Sentence Encoder)
sum these sentence vectors

9. • Long Short-Term Memory (LSTM) activation unit for
ameliorating the vanishing gradient problem when
training long sequences (Hochreiter and
Schmidhuber, 1997)
9
Document Reader
(Recurrent Document Encoder)

10. • Their sentence extractor applies attention to directly extract salient
sentences after reading them.
• at the beginning, they set pt-t to the true label of the previous sentence;
as training goes on, they gradually shift its value to the predicted label.
10
Sentence Extractor

11. Word
Extractor
• a sequential labeling model
• use n-gram features collected from
the document to rerank candidate
summaries obtained via beam
decoding.
• incorporate the features in a log-
linear reranker whose feature
weights are optimized with
minimum error rate training (Och,
2003)
11

12. Experimental Setup
• Datasets:
• two datasets created from DailyMail news:
90% for training, 5% for validation and 5% for testing
• DUC-2002 single document summarization task.
12

13. • Parameters:
• Adam (learning rate 0.01)
• The two momentum parameters: 0.99 and 0.999.
• batch size of 20 documents
• The size of word, sentence, document embedding:
150, 300, and 750.(word embedding is pre-trained)
• Kernel sizes {1, 2, 3, 4, 5, 6, 7}
• drop out 0.5
• The depth of each LSTM module: 1
13
Experimental Setup

14. • LEAD (leading three sents.)
• LREG (logistic regression)
• ILP
• NN-ABS (Rush et al. 2015)
• TGRAPH (Parveen et al., 2015)
• URANK (Wan, 2010)
• NN-SE (Sentence extractor)
• NN-WE (Word extractor)
14
Results

15. Results
15
• evaluate the generated summaries by eliciting human judgments
for 20 randomly sampled DUC 2002 test documents.
• Subjects were asked to rank the summaries from best to wrost
(with ties allow)
• collect 5 responses per document.

16. Results
16

17. Conclusion
• They developed two classes of models based on
sentence and word extractor.
• Future Work:
• combining their model with a tree-based
algorithm (Cohn and Lapata, 2009)
• or phrase-based(Lebret et al., 2015).
17