深層生成モデルの理論と導出(Variational Auto-encoderからADGM)
•
1 like•761 views
Deep learningを使った生成モデルの中でも、VAEとその拡張であるCVAE、IWAE、ADGMの導出と理論の話。 ADGMについては、論文がラベルありについての導出がメインなので、ラベルなしの場合のグラフィカルモデルを作って定式化し導出過程を説明。 VAEは実装できるけどどういう理論でできてるかわからないとか、どうやって拡張するか、ADGMの実装をどうすればいいかわからない人向け。
Report
Share
Report
Share
Download to read offline
Recommended
[DL輪読会]NVAE: A Deep Hierarchical Variational Autoencoder
2020/11/13
Deep Learning JP:
http://deeplearning.jp/seminar-2/
Curriculum Learning （関東CV勉強会）
2015年5月30日第29回コンピュータビジョン勉強会@関東「有名論文を読む会」発表用資料。
Curriculum Learning [Bengio+, ICML 2009] とその後続を紹介。
[DL輪読会]Flow-based Deep Generative Models
2019/03/08
Deep Learning JP:
http://deeplearning.jp/seminar-2/
【DL輪読会】Scaling Laws for Neural Language Models
This document summarizes a research paper on scaling laws for neural language models. Some key findings of the paper include:
- Language model performance depends strongly on model scale and weakly on model shape. With enough compute and data, performance scales as a power law of parameters, compute, and data.
- Overfitting is universal, with penalties depending on the ratio of parameters to data.
- Large models have higher sample efficiency and can reach the same performance levels with less optimization steps and data points.
- The paper motivated subsequent work by OpenAI on applying scaling laws to other domains like computer vision and developing increasingly large language models like GPT-3.
生成モデルの Deep Learning
PFIセミナー(2016/02/25)で発表したスライドです。伝承サンプリング可能な生成モデルに関するDeep Learningの紹介です（キーワード：Wake-Sleep, 変分 AutoEncoder, Generative Adversarial Nets, Likelihood Ratio）
Triplet Loss 徹底解説
ClassificationとMetric Learningの違い、Contrastive Loss と Triplet Loss、Triplet Lossの改良の変遷など
Recommended
[DL輪読会]NVAE: A Deep Hierarchical Variational Autoencoder
2020/11/13
Deep Learning JP:
http://deeplearning.jp/seminar-2/
Curriculum Learning （関東CV勉強会）
2015年5月30日第29回コンピュータビジョン勉強会@関東「有名論文を読む会」発表用資料。
Curriculum Learning [Bengio+, ICML 2009] とその後続を紹介。
[DL輪読会]Flow-based Deep Generative Models
2019/03/08
Deep Learning JP:
http://deeplearning.jp/seminar-2/
【DL輪読会】Scaling Laws for Neural Language Models
This document summarizes a research paper on scaling laws for neural language models. Some key findings of the paper include:
- Language model performance depends strongly on model scale and weakly on model shape. With enough compute and data, performance scales as a power law of parameters, compute, and data.
- Overfitting is universal, with penalties depending on the ratio of parameters to data.
- Large models have higher sample efficiency and can reach the same performance levels with less optimization steps and data points.
- The paper motivated subsequent work by OpenAI on applying scaling laws to other domains like computer vision and developing increasingly large language models like GPT-3.
生成モデルの Deep Learning
PFIセミナー(2016/02/25)で発表したスライドです。伝承サンプリング可能な生成モデルに関するDeep Learningの紹介です（キーワード：Wake-Sleep, 変分 AutoEncoder, Generative Adversarial Nets, Likelihood Ratio）
Triplet Loss 徹底解説
ClassificationとMetric Learningの違い、Contrastive Loss と Triplet Loss、Triplet Lossの改良の変遷など
IIBMP2016 深層生成モデルによる表現学習
IIBMP2016（第五回生命医薬情報学連合大会）での招待講演の内容です。深層学習を生成過程の問題としてとらえて、なぜ表現を学習できるのかを説明した後に，深層生成モデル（VAE, GAN, 少しだけ自己回帰モデル, エネルギーモデル, モーメントマッチングモデルを紹介します。
【DL輪読会】Flow Matching for Generative Modeling
2023/5/19
Deep Learning JP
http://deeplearning.jp/seminar-2/
[DL輪読会]"Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,0...
2018/08/24
Deep Learning JP:
http://deeplearning.jp/seminar-2/
多様な強化学習の概念と課題認識
Shibuya Synapse #3 〜現在の強化学習に何が足りないのか？〜 の発表スライドです．
同イベントのレポート https://dena.ai/news/201812-shibuyasynapse3rd-report/
【DL輪読会】Efficiently Modeling Long Sequences with Structured State Spaces
This document summarizes a research paper on modeling long-range dependencies in sequence data using structured state space models and deep learning. The proposed S4 model (1) derives recurrent and convolutional representations of state space models, (2) improves long-term memory using HiPPO matrices, and (3) efficiently computes state space model convolution kernels. Experiments show S4 outperforms existing methods on various long-range dependency tasks, achieves fast and memory-efficient computation comparable to efficient Transformers, and performs competitively as a general sequence model.
猫でも分かるVariational AutoEncoder
生成モデルとかをあまり知らない人にもなるべく分かりやすい説明を心がけたVariational AutoEncoderのスライド
実装と簡単な補足は以下を参照
http://sh-tatsuno.com/blog/index.php/2016/07/30/variationalautoencoder/
Overcoming Catastrophic Forgetting in Neural Networks読んだ
Summary of "Overcoming Catastrophic Forgetting in Neural Networks"
【DL輪読会】Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
This document discusses a paper on visuomotor policy learning via action diffusion. The paper presents a method for training policies that map camera images directly to actions by incorporating action diffusion, which adds noise to actions during training. This helps explore the action space and avoid getting stuck in local optima during policy learning. The method can learn policies for complex manipulation tasks entirely from pixels using self-supervised reinforcement learning with image rewards.
SSII2020SS: グラフデータでも深層学習 〜 Graph Neural Networks 入門 〜
SSII2020 技術動向解説セッション SS1
6/11 (木) 14:00～14:30 メイン会場 (vimeo + sli.do)
グラフ構造をもつデータに対する DNN、すなわち Graph Neural Networks (GNNs) の研究はこの２、３年で参加する研究者が急増している。現状、様々なアーキテクチャの GNN が様々なドメインや様々なタスクで個別に提案され、概観を捉えるのも簡単ではない状態になっている。本チュートリアルは、広範に散らばった GNN 研究の現状についての概観と基盤技術を紹介するとともに、時間が許す範囲でコンピュータビジョン領域における応用例の紹介にも取り組みたい。
機械学習モデルのハイパパラメータ最適化
The document discusses hyperparameter optimization in machine learning models. It introduces various hyperparameters that can affect model performance, and notes that as models become more complex, the number of hyperparameters increases, making manual tuning difficult. It formulates hyperparameter optimization as a black-box optimization problem to minimize validation loss and discusses challenges like high function evaluation costs and lack of gradient information.
【DL輪読会】論文解説：Offline Reinforcement Learning as One Big Sequence Modeling Problem
2022/06/10
Deep Learning JP:
http://deeplearning.jp/seminar-2/
Iclr2016 vaeまとめ
This document summarizes a presentation about variational autoencoders (VAEs) presented at the ICLR 2016 conference. The document discusses 5 VAE-related papers presented at ICLR 2016, including Importance Weighted Autoencoders, The Variational Fair Autoencoder, Generating Images from Captions with Attention, Variational Gaussian Process, and Variationally Auto-Encoded Deep Gaussian Processes. It also provides background on variational inference and VAEs, explaining how VAEs use neural networks to model probability distributions and maximize a lower bound on the log likelihood.
[DL輪読会]Learning Latent Dynamics for Planning from Pixels
2018/12/21
Deep Learning JP:
http://deeplearning.jp/seminar-2/
[DL輪読会]近年のエネルギーベースモデルの進展
1. The document discusses energy-based models (EBMs) and how they can be applied to classifiers. It introduces noise contrastive estimation and flow contrastive estimation as methods to train EBMs.
2. One paper presented trains energy-based models using flow contrastive estimation by passing data through a flow-based generator. This allows implicit modeling with EBMs.
3. Another paper argues that classifiers can be viewed as joint energy-based models over inputs and outputs, and should be treated as such. It introduces a method to train classifiers as EBMs using contrastive divergence.
【DL輪読会】Self-Supervised Learning from Images with a Joint-Embedding Predictive...
2023/6/30
Deep Learning JP
http://deeplearning.jp/seminar-2/
More Related Content
What's hot
IIBMP2016 深層生成モデルによる表現学習
IIBMP2016（第五回生命医薬情報学連合大会）での招待講演の内容です。深層学習を生成過程の問題としてとらえて、なぜ表現を学習できるのかを説明した後に，深層生成モデル（VAE, GAN, 少しだけ自己回帰モデル, エネルギーモデル, モーメントマッチングモデルを紹介します。
【DL輪読会】Flow Matching for Generative Modeling
2023/5/19
Deep Learning JP
http://deeplearning.jp/seminar-2/
[DL輪読会]"Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,0...
2018/08/24
Deep Learning JP:
http://deeplearning.jp/seminar-2/
多様な強化学習の概念と課題認識
Shibuya Synapse #3 〜現在の強化学習に何が足りないのか？〜 の発表スライドです．
同イベントのレポート https://dena.ai/news/201812-shibuyasynapse3rd-report/
【DL輪読会】Efficiently Modeling Long Sequences with Structured State Spaces
This document summarizes a research paper on modeling long-range dependencies in sequence data using structured state space models and deep learning. The proposed S4 model (1) derives recurrent and convolutional representations of state space models, (2) improves long-term memory using HiPPO matrices, and (3) efficiently computes state space model convolution kernels. Experiments show S4 outperforms existing methods on various long-range dependency tasks, achieves fast and memory-efficient computation comparable to efficient Transformers, and performs competitively as a general sequence model.
猫でも分かるVariational AutoEncoder
生成モデルとかをあまり知らない人にもなるべく分かりやすい説明を心がけたVariational AutoEncoderのスライド
実装と簡単な補足は以下を参照
http://sh-tatsuno.com/blog/index.php/2016/07/30/variationalautoencoder/
Overcoming Catastrophic Forgetting in Neural Networks読んだ
Summary of "Overcoming Catastrophic Forgetting in Neural Networks"
【DL輪読会】Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
This document discusses a paper on visuomotor policy learning via action diffusion. The paper presents a method for training policies that map camera images directly to actions by incorporating action diffusion, which adds noise to actions during training. This helps explore the action space and avoid getting stuck in local optima during policy learning. The method can learn policies for complex manipulation tasks entirely from pixels using self-supervised reinforcement learning with image rewards.
SSII2020SS: グラフデータでも深層学習 〜 Graph Neural Networks 入門 〜
SSII2020 技術動向解説セッション SS1
6/11 (木) 14:00～14:30 メイン会場 (vimeo + sli.do)
グラフ構造をもつデータに対する DNN、すなわち Graph Neural Networks (GNNs) の研究はこの２、３年で参加する研究者が急増している。現状、様々なアーキテクチャの GNN が様々なドメインや様々なタスクで個別に提案され、概観を捉えるのも簡単ではない状態になっている。本チュートリアルは、広範に散らばった GNN 研究の現状についての概観と基盤技術を紹介するとともに、時間が許す範囲でコンピュータビジョン領域における応用例の紹介にも取り組みたい。
機械学習モデルのハイパパラメータ最適化
The document discusses hyperparameter optimization in machine learning models. It introduces various hyperparameters that can affect model performance, and notes that as models become more complex, the number of hyperparameters increases, making manual tuning difficult. It formulates hyperparameter optimization as a black-box optimization problem to minimize validation loss and discusses challenges like high function evaluation costs and lack of gradient information.
【DL輪読会】論文解説：Offline Reinforcement Learning as One Big Sequence Modeling Problem
2022/06/10
Deep Learning JP:
http://deeplearning.jp/seminar-2/
Iclr2016 vaeまとめ
This document summarizes a presentation about variational autoencoders (VAEs) presented at the ICLR 2016 conference. The document discusses 5 VAE-related papers presented at ICLR 2016, including Importance Weighted Autoencoders, The Variational Fair Autoencoder, Generating Images from Captions with Attention, Variational Gaussian Process, and Variationally Auto-Encoded Deep Gaussian Processes. It also provides background on variational inference and VAEs, explaining how VAEs use neural networks to model probability distributions and maximize a lower bound on the log likelihood.
[DL輪読会]Learning Latent Dynamics for Planning from Pixels
2018/12/21
Deep Learning JP:
http://deeplearning.jp/seminar-2/
[DL輪読会]近年のエネルギーベースモデルの進展
1. The document discusses energy-based models (EBMs) and how they can be applied to classifiers. It introduces noise contrastive estimation and flow contrastive estimation as methods to train EBMs.
2. One paper presented trains energy-based models using flow contrastive estimation by passing data through a flow-based generator. This allows implicit modeling with EBMs.
3. Another paper argues that classifiers can be viewed as joint energy-based models over inputs and outputs, and should be treated as such. It introduces a method to train classifiers as EBMs using contrastive divergence.
【DL輪読会】Self-Supervised Learning from Images with a Joint-Embedding Predictive...
2023/6/30
Deep Learning JP
http://deeplearning.jp/seminar-2/
What's hot (20)
[DL輪読会]"Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,0...
[DL輪読会]"Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,0...
【DL輪読会】Efficiently Modeling Long Sequences with Structured State Spaces
【DL輪読会】Efficiently Modeling Long Sequences with Structured State Spaces
Overcoming Catastrophic Forgetting in Neural Networks読んだ
Overcoming Catastrophic Forgetting in Neural Networks読んだ
【DL輪読会】Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
【DL輪読会】Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
SSII2020SS: グラフデータでも深層学習 〜 Graph Neural Networks 入門 〜
SSII2020SS: グラフデータでも深層学習 〜 Graph Neural Networks 入門 〜
【DL輪読会】論文解説：Offline Reinforcement Learning as One Big Sequence Modeling Problem
【DL輪読会】論文解説：Offline Reinforcement Learning as One Big Sequence Modeling Problem
[DL輪読会]Learning Latent Dynamics for Planning from Pixels
[DL輪読会]Learning Latent Dynamics for Planning from Pixels
【DL輪読会】Self-Supervised Learning from Images with a Joint-Embedding Predictive...
【DL輪読会】Self-Supervised Learning from Images with a Joint-Embedding Predictive...