Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

[DL輪読会]Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models

3,074 views

Published on

2017/4/7
Deep Learning JP:
http://deeplearning.jp/seminar-2/

Published in: Technology
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

[DL輪読会]Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models

  1. 1. Batch Renormalization: Towards Reducing Minibatch Dependence in Batch- Normalized Models DL輪読会 松尾研D1 保住 純 2017/03/31 1
  2. 2. 書誌情報 ・タイトル: Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models ・著者: Sergey Ioffe (Google Inc.) [Batch Normalizationの生みの親の1人] ・公開日: 02/10/2017 (on arXiv) ・引用件数: 1 (Google Scholar) ・近い将来、Tensorflowにも導入されそう。 - 既にプルリクが出ていた 2 (注:以後のスライド内の式とグラフは、すべてこの論文から引用)
  3. 3. 背景 ・Batch Normalization(バッチ正規化、BN)はDLの標準的技術になった。 - 内部共変量シフトを防ぐことで、学習を高速化 - DropOutのような正則化効果もある - ResNetやGANの成立にBNが貢献 ・BNの特徴 - (主にCNNで)活性化関数やNNの構造によらずに使用できる。 - 学習時と推定時で、正規化のされ方が異なる。 (正規化に用いる平均・分散を計算する際に使われる事例が異なるから) - 学習時はミニバッチに、推定時は学習事例全体に影響される - 学習時から学習事例全体の統計量で正規化するのは、勾配爆発するのでNG 3
  4. 4. ・BNの特徴を踏まえると、BNは、①バッチサイズが大きく、②事例が i.i.d.(独立同分布)であれば、有効に機能する。 ・そうでない場合には、以下の問題が生じる。 ①バッチサイズが小さい → 統計量が不正確になる(母集団の統計量から乖離) - バッチが増えたり、多層になったりすると、悪影響は深刻化 ②事例がi.i.d.でない → バッチごとに過学習してしまう - 学習時と推定時でバッチごとに分布が大きく異なると、悪影響も増大 ・この問題に対処する正規化手法は(Normalization Propagation, Layer Norm, Weight Normなど)色々あるが、BNに比べて面倒もしくは計算コストがかかる。 4 BNの手軽さを維持しつつ、以上の問題を克服する Batch Renormalization(バッチ再正規化)を提案 BNの抱える問題
  5. 5. 復習: Batch Normalization ・バッチごとに平均と分散を計算し、それらを用いて事例を正規化する。 ・γ(スケール)とβ(シフト)は、学習されるパラメータ。 5 平均 分散 正規化 ← この式に注目! バッチ内での バッチ内での
  6. 6. Batch Renormalizationのアイデア 正規化時のバッチ内の平均・分散を、全体の平均・分散に置き換える。 そして、新たに変数r, dを導入して、正規化部分を書き換える。 バッチ内の平均と分散を、rとdで、全体のそれらに合うよう補正する。 (バッチ内の平均と分散がデータ全体のそれらと一致する時、元のBNの式と一致。) 6 正規化 (全体での) (全体での)
  7. 7. Batch Renormalization ハイパーパラメータとして、rmax, dmax, (rとdの値域を定めるパラメータ)とΔ(更新率) を与える。 - rmax, dmaxは、少しずつ大きくする(詳しくは後述) 7 勾配計算時には定数扱い 全体の平均と分散は、移動平均を計算して更新 ← 前ページの式 値域の制限
  8. 8. ポイント ・r,dは順伝播時にバッチごとに計算するが、逆伝播時は定数として扱う。 - BatchRenormは、BNの正規化に、ミニバッチの統計量を全体の統計量に移して 正しく活性化させるアフィン変換を追加したもの、とみなせる ・最初は(普通の)BNで学習させる。(つまり、最初しばらくはr=1, d=0に固定。) そして、徐々にrmax, dmax を大きくする。 - こうすることで勾配爆発を防ぎ、うまく収束できる ・学習事例全体の統計量の算出には、移動平均を用いる。 ・推定時は普通のBNと同様、学習事例全体のμ,σを用いる。 8
  9. 9. ・画像分類問題で、BatchRenormを評価する。 Baseline: Inception v3 - 畳み込み層とReLUの間にBN Model: Inception v3 (※BNの代わりにBatchRenorm) Data Set: ImageNet BatchSize : 32(×50バッチ) - 50バッチを並列処理 Optimizer: RMS Prop Model Training: - BN (for first 5k steps) - rmax → 3 (@40k steps)、 dmax → 5(@25k steps) 9 実験
  10. 10. 実験結果1 – MicroBatch (Small Minibatch) ・MiniBatch(事例32ずつ)では、精度にに大きな差はない。 ・MicroBatch(事例4ずつ)だと、精度に明確な差が現れる。 10 1バッチ当たり32事例の場合 (MiniBatch) 1バッチ当たり4事例の場合 (MicroBatch) ← ここ(steps=5k)まではどちらも普通のBN
  11. 11. 実験結果2- Non i.i.d. Minibatch ・各バッチを「(ランダム選択した)16クラス×2事例=32事例」にさせて学習。 - 全部で32クラスある分類問題なので、バッチごとにデータが偏る(Non i.i.d.) BNだと(学習時ですら)精度が低い。 (オーバーフィットが発生) 11 ① BNだと(学習時ですら)精度が低い。 (過学習が発生するから) ← ② 推定時に推定バッチの平均・分散を用いた場合。 ← ③ 「16クラス×1事例=16事例」バッチの場合。 ← ④ BatchRenormなら、そんな面倒なことしなくても精度が良い。
  12. 12. BNは学習と推定時とで正規化のされ方が異なるため、 MicrobatchやNon i.i.d. Minibatchでの学習では上手くいかなかった。 → 正規化のされ方がミニバッチに依らなくなるよう、BNを修整した。 → BNと同様の手軽さと速さを維持したまま、MicrobatchやNon i.i.d. Minibatchでの学習を大幅に改善できた。 ・今後の研究課題 - ハイパーパラメータ(rmax, dmax, Δ)の良い決め方 - GANで問題となっている「BNのもたらす非決定性」を解決できるか - RNNにも使えるか - BatchRenormなら全タイムステップを考慮できるからいける? 12 結論

×