Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

[DL輪読会]Adversarial Feature Matching for Text Generation

3,814 views

Published on

2017/7/7
Deep Learning JP:
http://deeplearning.jp/seminar-2/

Published in: Technology
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

[DL輪読会]Adversarial Feature Matching for Text Generation

  1. 1. Adversarial Feature Matching
 for Text Generation 2017/7/7 DL輪読会 松尾研 曽根岡侑也 1
  2. 2. メタ情報 • 著者 - Yizhe Zhang, Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Lawrence Carin - NIPS2016 ✕ 3, ICML ✕ 2のデューク大学PhD • Accepted by ICML2017(arXiv on 12 Jun 2017) • NIPS2016 Workshopの進化版 2
  3. 3. 概要 • 文書生成にGANを用いるTextGANを提案 - GeneratorはLSTM、DiscriminatorはCNN - FeatureMatchingと再構成の項を目的関数に追加し、
 ModeCollapse・勾配消失問題を軽減 - Soft-argmax近似、Pre-training、soft-labeling等の
 学習テクニック • SeqGANよりいい評価 & 現実的な文生成に成功 3
  4. 4. 背景:自然言語生成 • 自然言語生成の系譜 - 文書から確率分布を評価し、その分布からサンプリング - RNNを用いたAutoEncoder [Cho et al.(2014)] - RNNベースのVAE [Bowman et al.(2016)] • RNNベースではうまくいかない - 潜在空間の一部しかカバーできていない - Exposure Bias:文後半に向けてズレが蓄積 4
  5. 5. 背景:GAN • 本物らしく作るGenerator vs 偽物を見抜くDiscriminator - Dは最大化、Gは最小化するように最適化 - DはJSDを近似し、Gは近似されたJSDを最小化する方向に • GANの問題点 - ModeCollapsing:潜在変数から同じ結果を作る - Dが局所解に近づいた場合、勾配消失が起きる
 (EBGANのTVDも同様) 5
  6. 6. 提案手法:TextGAN • GはLSTM、DとEはCNNを使用 • Feature Machingを採用 [Salimans et al. (2016)] 6
  7. 7. TextGANの目的関数 7 (λr, λmはハイパーパラメータ) → GANと同じ → Gは最小化する → Dは最大化する → 潜在変数の再構成時の誤差
  8. 8. TextGANの目的関数 8 (λr, λmはハイパーパラメータ) ③ ② → GANと同じ → Gは最小化する → Dは最大化する → 潜在変数の再構成時の誤差 ③① ② ①
  9. 9. Maximum Mean Discrepancy(MMD) • Gaussianカーネルで再生核ヒルベルト空間(RKHS)へ写像し、
 平均の差を用いて一致度を測定 [Gretton et al (2012)] 9 → 今回はGaussian
  10. 10. TextGANの目的関数 10 (λr, λmはハイパーパラメータ) → GANと同じ → Gは最小化する → Dは最大化する → 潜在変数の再構成時の誤差 ③① ② ① 偽物を見抜く方向に ② 潜在変数を最大限保存する方向に ③ Generatorがあわせるのが難しい特徴量を見つける方向に
  11. 11. Discriminator/Encoder:CNN • 文を学習済みの埋め込み行列でk ✕ Tの行列に変換 • Windowサイズが異なるConvolutionのフィルタをかけ、フィル タ毎にMaxPooling(活性化関数はtanh) • DはMLPの後にSoftmaxで真偽を判定、EはMLPでzを復元 11 [Kim et al.(2014)]
  12. 12. Generator:LSTM • よくあるLSTM • yは生成された単語を埋め込みベクトルにしたもの • zは毎回渡す 12
  13. 13. データ効率①:Compressing Network • 課題
 GaussianカーネルMMDでは特徴ベクトルfの次元に応じて、
 ミニバッチのサイズを大きくする必要がある • Compressing Network - 特徴ベクトルfを圧縮するための全結合レイヤーを追加 - 変換後の次元数はデータ効率と表現力のトレードオフ 13
  14. 14. データ効率②:Gaussian covariance matching • カーネルトリックの代わりに下記を使う 14 : の共分散 : の平均
  15. 15. 学習テクニック①:Soft-argmax approximation • 離散変数を含むため、Gの学習での勾配評価は難しい • 下記の式で近似(Gumbel-Softmaxに近い?) 15 Soft-argmax近似元の式
  16. 16. 学習テクニック②:Pre-training • D/E(CNN) - Permutation training - テキストの2単語を入れ替えて偽の文を作り学習 - 単語追加・消去より難しいタスク • G(LSTM) - CNN-LSTM autoencoderを利用 [Gan et al. (2016)] 16
  17. 17. 学習テクニック③:Soft-labeling • 1 or 0とするのが普通であるが、正解=0.7-1.2、偽=0-0.3か らランダムにサンプルする[Salimans et al (2016)] • 本論文では、最大0.99, 最低0.01としている 17
  18. 18. 実験 • データ - BookCorpus(70m) + Arxiv(5m)から各50万文ずつ • 潜在変数zは900次元 • D/E(CNN) - Windowサイズは3,4,5で、各300個ずつのfilter - 活性化関数はSigmoid - D:900-200-2のMLPで真偽を判定z、出力層はSoftmax - E:900-900-900でzを再構成、出力層はtanh • LSTM:隠れ層500 18
  19. 19. 実験 • その他 - Gを5回更新したらDを1回更新 - Gaussianカーネルのσ:20前後 - Optimizer:Adam(学習率:5 ✕ 10^-5) - ミニバッチ:256 - GのLossとBLEU (正解と生成の類似度)でValidation - 50 epoch (3days) 19
  20. 20. 実験結果①:特徴ベクトルの分布 • 本物と偽物の2000文ずつの特徴ベクトル(900次元)の
 平均と共分散をプロット 20
  21. 21. 実験結果②:定量的比較 21 ※ SeqGANは金子さんの輪読参照(16/9/30) MM:Mean Matching, CM:Covariance Matching MMD-L:compressed to 200次元
  22. 22. 実験結果③:生成文 • 文法に関してはDがうまく機能しており、約95%で偽物を識別 - 丸括弧やクォーテーションをあわせて生成している - 文法的には正しいが20語以上になると意味がおかしくなる 22
  23. 23. 実験結果④:潜在特徴空間の軌道 • 文Aから文Bまで潜在変数を連続的に変更した際の変化 • AEより意味的にも文法的にも正しいが、大きな変化が起きる 23
  24. 24. まとめ • 文書生成にGANを用いるTextGANを提案 - GeneratorはLSTM、DiscriminatorはCNN - FeatureMatchingと再構成の項を目的関数に追加し、
 ModeCollapse・勾配消失問題を軽減 - Soft-argmax近似、Pre-training、soft-labeling等の
 学習テクニック • SeqGANよりいい評価 & 現実的な文生成に成功 24
  25. 25. Appendix 25
  26. 26. 【参考】Jensen-Shannonダイバージェンス(JSD) • KLダイバージェンス:分布と分布の差異の大きさ
 • JSダイバージェンス:KLに対称性を付加 26
  27. 27. 【参考】 MMDのPytorch実装 27
  28. 28. 【参考】TextCNN実装 • https://github.com/dennybritz/cnn-text-classification-tf 28
  29. 29. 関連研究:Generative Moment Matching Networks(GMMNs) • GANのDをMMDで代替するアイデアで同じ • 自然言語で使えない(NNで使う場合近似する) - 計算量が重く次元数が大きいケースでは厳しい - 単語ベースの類似度のため、文構造を考慮しない
 (boy is swimming と a boy is swimming) 29
  30. 30. 関連研究:その他 • Kernelized Stein Discrepancy(KSD) - MMDからKSDにするのはFuture Work • WGAN - JSDよりWassersteinのほうがよさそうだが、MMDを 使っているので勾配消失は軽減されている 30
  31. 31. 【参考】BLEU 31
  32. 32. FutureWork • KSDの導入 • DropOut • 強化学習の戦略を用いてLSTMをアップデート • Conditional GAN • reverse-order LSTM 32
  33. 33. 【参考】KSD 33

×