SummaRuNNer is a simple recurrent neural network model for extractive text summarization. It treats summarization as a sequence classification problem, making binary decisions about whether to include each sentence. The model uses a bi-directional GRU to encode sentences. It is trained end-to-end using abstractive summaries, which allows the model to be trained on data where only abstractive and not extractive summaries are available. Experimental results show SummaRuNNer performs comparably or better than state-of-the-art extractive models on several datasets.

1. SummaRuNNer
Ramesh Nallapati, Feifei Zhai, Bowen Zhou
Presented by :
Sharath T.S
Shubhangi Tandon

2. Contributions of this paper
● SummaRuNNer, a simple recurrent network based sequence classifier
that outperforms or matches state-of-the-art models for extractive
summarization
● The simple formulation of model facilitates interpretable visualization of
its decisions
● A novel training mechanism that allows our extractive model to be trained
end-to-end using abstractive summaries.

3. SummaRuNNer
● Treat extractive summarization as a sequence classification problem
● Each sentence is visited sequentially in the original document order
● A binary decision is made (taking into account previous decisions)
● GRU based RNN basic building block of sequence classifier
● Recurrent network with two gates, u :update gate and r : reset gate

4. Recurrents neural networks
LSTMs:
● Input gate: Decides what fraction of the new input flowing into the LSTM
cell has to be updated.

5. LSTMs - Continued
● Update gate: Calculates what amount of current cell state to forget, and
updates the new information.

6. LSTMs - Continued
● Output gate: Evaluates the new cell state and decides what parts of the
information has to be output.
Refer: http://colah.github.io/posts/2015-08-Understanding-LSTMs/

7. GRU LSTMs
Modifications compared to LSTMs:
● It combines the forget(f) and input(i) gate into a single update gate.
● Merges the cell state and hidden state into one state.

8. The Model

9. SummaRuNNer
Model:
● Two-layer bi-directional GRU-RNN - The first layer of the RNN runs at the word level, computes
hidden state representations at each word position. Another RNN at the word level that runs
backwards from the last word to the first.
● second layer of bi-directional RNN that runs at the sentence-level and accepts the average-pooled,
concatenated hidden states of word-level RNNs.
● Document representation :
`

10. Computing Posterior - Logistic loss
(7)

11. Extractive Summary labels - Greedy Algorithm
Why is it needed?
● most summarization corpora only contain human written abstractive
summaries as ground truth.
● Algorithm
○ selected sentences from the document should be the ones that maximize the Rouge
score with respect to gold summaries.
○ Stop when none of the remaining candidate when added improve the ROUGE score.
● Train the network with labelled data.

12. Abstractive training - Decoder
● Apart from the sigmoid function present to compute the class a sentence belongs to,
the decoder in addition does the following
○ Takes embedding of a word(hidden state) as input from the previous state as xk
, s -1
is the value computed
at the last sentence of the RNN( Equation 7).
○ Computes softmax to output the most probable word.
○ Optimize the log likelihood of the word distribution in the abstractive summaries.(context captured by
RNN)
○ Predict using weights W, without the decoder on test samples.

13. Decoder - Continued
How does it work?
● The summary representation s−1
acts as an information channel between
the SummaRuNNer model and the decoder.
● Maximizing the probability of abstractive summary words as computed by
the decoder will require the model to learn a good summary
representation which in turn depends on accurate estimates of extractive
probabilities p(yj
).

14. SummaRuNNer Visualisation

15. Corpus used
● Daily Mail ( Cheng & Lapata) : 200k Tr, 12k Val , 10k Test
● Daily Mail/CNN (Nallapati) : 286k Tr, 13k Val, 11k Test
● DUC 2002 : 567 documents ( out of Domain Testing)
● Average statistics
○ 28 sentences/ doc
○ 3-4 sentences in reference summary
○ 802 word / doc
● Training Data Constraints
○ Vocab size : 150k
○ Maximum sentences/ doc : 100
○ Max Sentence Length : 50 words
○ Model hidden state : 200
○ Batch Size : 64

16. Experiments and Results : Daily Mail Corpus

17. Experiments and Results : Daily Mail /CNN data

18. Experiments and Results : DUC 2002 data

19. Future Work
● Pre-Train extractive model using abstractive training
● Construct a joint extractive-abstractive model where predictions of
extractive component form stochastic intermediate units to be consumed
by abstractive component.