The document describes 3 hierarchical LSTM models for generating coherent multi-sentence text: 1) Standard LSTM encodes/decodes a document as a single sequence. 2) Hierarchical LSTM encodes sentences then the document. 3) Hierarchical LSTM with attention encodes sentences then decodes with attention over encoded sentences. The models were evaluated on hotel reviews and Wikipedia using ROUGE, BLEU, and a coherence metric called L-value. The hierarchical LSTMs outperformed the standard LSTM, and hotel reviews were easier to generate than Wikipedia text.

1. A Hierarchical Neural Autoencoder for
Paragraphs and Documents
Jiwei Li, Minh-Thang Luong and Dan Jurafsky
ACL 2015

2. Introduction
● Generating the coherent text (document/paragraph)
● Using the 2 stacked LSTM per encoder/decoder
○ Lower: word sequence is compressed into one vector
○ Upper: sentence sequence is compressed into one vector
● Evaluating the coherence of the output from autoencoder.

3. Paragraph autoencoder, Model 1: Standard LSTM
● Input and output is the same D = {s1
, s2
, …, sND
, endD
}
● each s = {w1
, w2
, …, wNs
}
○ ND, Ns means the each length.
● e represents the embedding
● Standard: all sentences concatenated into one sequence.

4. Model 2: Hierarchical LSTM
● Encoder
● Decoder
(I think the lower one is LSTMword
)
● After generating </s> at each sentence,
last hidden state is used as a input of the
sentence decoder.

5. Model 3: Hierarchical LSTM with Attention
● Bahdanau-like attention model
● Attention function is used at only sentence decoding step.

6. Experiment
● LSTM: 4-layer (in total), 1000 dim
● SGD: Learning rate = 0.1, 7 epoch, batch_size = 32 doc.
They used crawled documents
● Hotel Review (from TripAdvisor’s page)
○ Train: 340,000 reviews (included at most 250 words in each reviews)
○ Test: 40,000 reviews
● Wikipedia (it is not domain-specific data)
○ Train: 500,000 paragraphs
○ Test: 50,000 paragraphs

7. Evaluation
● ROUGE, BLEU (it doesn’t use the log function)
● Following L-value (Lapata and Barzily 2005)
“Based on the assumption that human-generated texts are coherent.”
a. Make the feature vector consisting of verbs and nouns
b. Align the input and output sentences based on F1 score using this vector.
■ ROUGE and BLEU is used as the precision and recall value
c. Calculate the L-value
i denotes the position index, i’ denotes the position index at input side

8. Result and Conclusion
● Hotel Review task is easier than Wikipedia task
○ Open domain documents are difficult to generate ?
○ The sentences on Wikipedia are written in the fixed format ?
● Proposed method is useful to auto-encode the document.