- QRNN is a new type of recurrent neural network that can process sequential data in parallel like a CNN, making it up to 16 times faster than LSTM for training and testing. It approximates the hidden state of RNNs using previous input values rather than the previous hidden state, allowing parallel computation. - The paper proposes the QRNN model, which uses masked convolutions and pooling to enable parallel processing of sequences. It evaluates QRNN on sentiment classification, language modeling, and character-level machine translation, finding it can achieve similar performance to LSTMs while being much faster to train.

1. James Bradbury∗, Stephen Merity∗ , Caiming Xiong & Richard Socher
Salesforce Research Palo Alto, California
arXiv:1611.01576v2 [cs.NE] 21 Nov 2016
ICLR 2017 accepted
文献紹介
QRNN: QUASI-RECURRENT NEURAL
NETWORKS

2. Abstract
- QRNN = RNN processing like CNN
- can process sequential data in parallel
- up to 16 times faster than LSTM in train/test
- can make visual analysis of weights easy

3. Outline
- Introduction
- review of RNN/LSTM
- Model (QRNN)
- Variants
- Results
- sentiment classification
- language modeling
- character-level machine translation
- Conclusion
- Reference

4. Introduction (review of RNN)
- the standard model architecture for deep learning approaches to sequence
modeling tasks
- sentence classification | word- and character-level language modeling | machine translation |
question answering | image caption | time series forecasting

5. Introduction (review of RNN)
- the network which has loop
arhictectures
- RNN is very deep (causing
gradient vanising) word2vec(“私”）
昨日の株価
(“の”:0.2, “は”:0.3, ...)
今日の株価の予測値

6. Introduction (review of RNN)
- problem: not good at
learning very long
sequences
- document classification |
character-level
- why?: can’t deal with
sequential data in parallel

7. Introduction (review of LSTM)
- LSTM solves gradient vanising, using memory cell
- LSTM has 3 gates to control information flow

8. Introduction (review of LSTM)
- forget gate to control long-term information (in memory cell c)

9. Introduction (review of LSTM)
- input gate to control current+short-time information (in x and h(t-1))

10. Introduction (review of LSTM)
- update memory cell, mixing the current with the previos memory cell

11. - output gate to control current hidden-state information to the next layer
Introduction (review of LSTM)

12. - using a forget gate instead of an input gate
Introduction (variants of LSTM)

13. Model

14. Model (convolution component)
“ズン”, “ドコ”, “きよし”
( 1, 0, 0, )
=“ズン”
この例はone-hotだが
word2vecというもっといい変換
を使う
ズン, ズン, ズン, ドコ, きよし
時刻tの値を予測す
るのに未来の時刻
t+1のデータを用い
てはいけないので、
masked
convolution

15. Model

16. bottle-neckになっていた前の層のhidden
state h[t-1] を用いるのではなく、前の時刻
の入力x[t-1/2/...]を用いて並列処理を可能
にした。
Model (pooling component)
LSTM
さらに、hidden state h に重みをかけずに渡していくので、各要素
の情報がごっちゃにならないので可視化しやすい。
ここは従来のLSTMと同じく逐次
計算するが、そんなに大して時
間かからない。

17. Model (pooling component)
other type poolings （この論文では使われていない？）
f-pooling
ifo-pooling

18. Variants
- Zoneout: Dropout for LSTM
- skip-connection like DenseNet
- Attention for Encoder-Decoder

19. Experiments
- Sentiment Classification (document binary-classification)
- IMDb movie review
- 25,000 positive/negative reviews
- Language Modeling (word-level prediction)
- PTB: Penn Treebank
- Character-level Machine Translatoin
- IWST English-German spoken language translation task

20. Results (sentiment classification)
- 小 batch_size, 長 seq_len
に向いている（最大16倍早
かった。）
- training 時間は3倍早い

21. Results (sentiment classification)
final layer’s
hidden state

22. Results (language modeling)

23. Results (character-level machine translation)
BLEU: upper is better
http://unicorn.ike.tottori-u.ac.jp/2010/s072046/paper/graduation-thesis/node32.html

24. 考察
- LSTMに精度で少し負けてしまった理由は、隠れ層の状態 h[t-1] ではなく、直前の
入力 x[t-1|t-2|,...]を使って近似したからと考えられる。
- 入力で、隠れ層の状態を近似する場合、使う、前の時刻の filter size k を無限大ま
で長くすれば一致する。(sentiment classificationのtaskではkを大きくしたら精度
上がった）
- なので、filter-sizeを大きくすればいいが、そうすると、計算速度はどれほど落ちる
のかが問題。

25. Conclusion
- QRNN = RNN processing like CNN
- can process sequential data in parallel
- up to 16x faster than LSTM in train/test
- can make visual analysis of weights easy

26. Reference
- LSTM
- LSTMネットワークの概要 https://qiita.com/KojiOhki/items/89cd7b69a8a6239d67ca
- わかるLSTM ～ 最近の動向と共に https://qiita.com/KojiOhki/items/89cd7b69a8a6239d67ca
- ニューラルネットワーク勉強会
http://isw3.naist.jp/~neubig/student/2015/seitaro-s/161025neuralnet_study_LSTM.pdf
- conv の 3D図作成
- thinkercad https://www.tinkercad.com/
- QRNN
- LSTMを超える期待の新星、QRNN https://qiita.com/icoxfog417/items/d77912e10a7c60ae680e
- slideshare
https://www.slideshare.net/DeepLearningJP2016/dlquasirecurrent-neural-networks?qid=a4ead77d-d8dd-458b-965c-5e53723d7757
&v=&b=&from_search=1
- pytorchでの公式実装 https://github.com/salesforce/pytorch-qrnn/blob/master/torchqrnn/qrnn.py