• Save
Deep Learning via Semi-Supervised Embedding (第 7 回 Deep Learning 勉強会資料; 大澤)
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Deep Learning via Semi-Supervised Embedding (第 7 回 Deep Learning 勉強会資料; 大澤)

  • 1,789 views
Uploaded on

Deep Learning Japan @ 東大です ...

Deep Learning Japan @ 東大です
http://www.facebook.com/DeepLearning
https://sites.google.com/site/deeplearning2013/

More in: Technology , Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,789
On Slideshare
1,787
From Embeds
2
Number of Embeds
1

Actions

Shares
Downloads
0
Comments
0
Likes
6

Embeds 2

https://twitter.com 2

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. eep Learning via Semi-Supervised Embedding Summarized by Shohei Ohsawa The University of Tokyo ohsawa@weblab.t.u-tokyo.ac.jp D
  • 2. Summary • Deep Learning via Semi-Supervised Embedding • Jason Weston (NEC Labs America, USA) • Frédéric Ratle (IGAR, University of Lausanne, Switzerland) • Ronan Collobert (NEC Labs America, USA) • Proceedings of the 25th International Conference on Machine Learning (ICML2008) • Enhancing semi-supervised learning to support deep architecture • Citations: 88 1 Title Author Contents Venue Information
  • 3. ADGENDA 2 1. Introduction 2. Semi-supervised Embedding 3. Semi-supervised Embedding for Deep Learning 4. Existing Approaches to Deep Learning 5. Experimental Result 6. Conclusion
  • 4. ADGENDA 3 1. Introduction 2. Semi-supervised Embedding 3. Semi-supervised Embedding for Deep Learning 4. Existing Approaches to Deep Learning 5. Experimental Result 6. Conclusion
  • 5. 1. Introduction Background • Embedding data into a lower dimensional space are unsupervised dimensionality reduction techniques that have been intensively studied. • Most algorithms are developed with the motivation of producing a useful analysis and visualization tool. 4 Unlabelled data Manifold Embedding
  • 6. 1. Introduction Semi-supervised Learning • Recently, the field of semi-supervised learning [Chapelle 2006], which has the goal of improving generalization on supervised tasks using unlabeled data, has made use of many of these techniques. • Ex.) researchers have used nonlinear embedding or cluster representations (unsupervised) as features for a supervised classifier, with improved results. 5 Labelled data Unlabelled data
  • 7. 1. Introduction Issue of Existing Architectures • Most of these architectures are disjoint and shallow • Joint Methods • Transductive Support Vector Machines (TSVMs) [Vapnik 1998] • LapSVM [Belkin 2006] •   their architecture is still shallow. 6 The unsupervised dimensionality reduction algorithm is trained on unlabeled data separately as a first step a supervised classifier which has a shallow architecture such as a (kernelized) linear model. • [Chapelle 2003][Chapelle 2005] learn a clustering or a distance measure based on a nonlinear manifold embedding as a first step
  • 8. 1. Introduction Semi-supervised Learning • Deep architectures seem a natural choice in hard AI tasks which involve several sub-tasks which can be coded into the layers of the architecture. • As argued by several researchers [Hinton 2006][Bengio 2007] semi-supervised learning is also natural in such a setting as otherwise one is not likely to ever have enough labeled data to perform well. 7
  • 9. 1. Introduction Deep Architecture for Semi-supervised Learning • Several authors have recently proposed methods for using unlabeled data in deep neural network-based architectures. • These methods either perform a greedy layer-wise pre-training of weights using unlabeled data alone followed by supervised fine-tuning (which can be compared to the disjoint shallow techniques for semi-supervised learning described before), • Or learn unsupervised encodings at multiple levels of the architecture jointly with a supervised signal. The basic setup 8 1. Choose an unsupervised learning algorithm. 2. Choose a model with a deep architecture. 3. The unsupervised learning is plugged into any layers of the architecture 4. Train supervised and unsupervised tasks using the same architecture simultaneously
  • 10. 1. Introduction Semi-supervised Learning • The aim is that the unsupervised method will improve accuracy on the task at hand. • However, the unsupervised methods so far proposed for deep architectures are in our opinion somewhat complicated and restricted. • They include: • generative model (a restricted Boltzmann machine) [Hinton 2006] • autoencoders [Bengio 2007] • sparse encoding [Ranzato 2007] • Moreover, in all cases these methods are not compared with, and appear on the surface to be completely different to, algorithms developed by researchers in the field of semi- supervised learning. 9
  • 11. 1. Introduction Objective • In this presentation, we advocate simpler ways of performing deep learning by leveraging existing ideas from semi-supervised algorithms so far developed in shallow architectures. • In particular, we focus on the idea of combining an embedding-based regularizer with a supervised learner to perform semi-supervised learning • Laplacian SVMs [Belkin et al.,2006] • We show that this method can be: • (i) generalized to multi-layer networks and trained by stochastic gradient descent • (ii) is valid in the deep learning framework given above. 10
  • 12. ADGENDA 12 1. Introduction 2. Semi-supervised Embedding 3. Semi-supervised Embedding for Deep Learning 4. Existing Approaches to Deep Learning 5. Experimental Result 6. Conclusion
  • 13. 2. Semi-supervised Embedding Structure Assumption • Structure Assumption: points within the same structure (such as a cluster or a manifold) are likely to have the same label. • Algorithms • cluster kernels [Chapelle et al., 2003] • LDS [Chapelle & Zien,2005] • label propagation [Zhu & Ghahramani, 2002] • LapSVM [Belkin et al., 2006] • To understand these methods we will first review some relevant approaches to linear and nonlinear embedding. 13 Labelled data The labels can be estimated as soon as the points on a same manifold
  • 14. 2. Semi-supervised Embedding Embedding Algorithms 14 minimize s.t. Balancing Constant
  • 15. 2. Semi-supervised Embedding Embedding Algorithms: Multidimensional Scaling (MDS) • A classical algorithm that attempts to preserve the distance between points, whilst embedding them in a lower dimensional space • MDS is equivalent to PCA if the metric is Euclidean [Williams, 2001] 15 Manifold
  • 16. 2. Semi-supervised Embedding Embedding Algorithms: ISOMAP [Tenenbaum 2000] 16 W=1/8
  • 17. 2. Semi-supervised Embedding Embedding Algorithms: Laplacian Eigenmaps [Belkin & Niyogi 2003] 17 Laplacian Kernel
  • 18. 2. Semi-supervised Embedding Semi-superbised Algorithms 19 • L: Labelled Data • U: Unlabelled Data
  • 19. 2. Semi-supervised Embedding Semi-superbised Algorithms: Label Propagation [Zhu & Ghahramani, 2002] 20 RegularizeLoss
  • 20. 2. Semi-supervised Embedding Semi-superbised Algorithms: LapSVM [Belkin et al., 2006] 21 1 10
  • 21. ADGENDA 22 1. Introduction 2. Semi-supervised Embedding 3. Semi-supervised Embedding for Deep Learning 4. Existing Approaches to Deep Learning 5. Experimental Result 6. Conclusion
  • 22. 3. Semi-supervised Embedding for Deep Learning Overview • We would like to use the ideas developed in semi-supervised learning for deep learning. Deep learning consists of learning a model with several layers of non-linear mapping. • We will consider multi-layer networks with M layers of hidden units that give a C- dimensional output vector: • wO: the weights for the output layer • typically the kth layer is defined as • S: a non-linear squashing function such as tanh. 23
  • 23. 3. Semi-supervised Embedding for Deep Learning Overview • Here, we describe a standard fully connected multi-layer network but prior knowledge about a particular problem could lead one to other network designs. • For example in sequence and image recognition time delay and convolutional networks (TDNNs and CNNs) [Le-Cun et al., 1998] have been very successful. • In those approaches one introduces layers that apply convolutions on their input which take into account locality information in the data, i.e. they learn features from image patches or windows within a sequence. 24
  • 24. 3. Semi-supervised Embedding for Deep Learning Three Modes of Embedding in Deep Architectures • The general method we propose for semi-supervised deep learning is to add a semi- supervised regularizer in deep architectures in one of three different modes 25 (a) Output (b) Internal (c) Auxiliary
  • 25. 3. Semi-supervised Embedding for Deep Learning Three Modes of Embedding in Deep Architectures: (a) Output • Add a semi-supervised loss (regularizer) to the supervised loss on the entire network’s output • This is most similar to the shallow techniques described before 26
  • 26. 3. Semi-supervised Embedding for Deep Learning Three Modes of Embedding in Deep Architectures: (b) Internal • Regularize the k th hidden layer (7) directly: 27 • is the output of the network up to the hidden layer.
  • 27. 3. Semi-supervised Embedding for Deep Learning Three Modes of Embedding in Deep Architectures: (c) Auxiliary • Create an auxiliary network which shares the first k layers of the original network but has a new final set of weights: • We train this network to embed unlabeled data simultaneously as we train the original network on labeled data. 28
  • 28. 3. Semi-supervised Embedding for Deep Learning Algorithm 30
  • 29. 3. Semi-supervised Embedding for Deep Learning Labeling unlabeled data as Neighbors 31
  • 30. ADGENDA 36 1. Introduction 2. Semi-supervised Embedding 3. Semi-supervised Embedding for Deep Learning 4. Existing Approaches to Deep Learning 5. Experimental Result 6. Conclusion
  • 31. Deep Boltzmann Machine • ボルツマンマシンの一種 • RBM を多段に重ねたような形 37 中間層II 中間層I 入力層 ノード値 バイアス エネルギー関数
  • 32. Auto-encoder • Auto-encoder framework [Lecaum 1987][Bourland 1988][Hinton 1994] : unsupervised feature construction method の一つ。 • auto-: 「自己の」 auto-encoder を直訳すると自己符号器 • encoder, decoder, reconstruction error の 3 つの要素から構成。 • encoder と decoder の合成写像が入力値を再現するような学習を行う。 • 学習は入力値と出力値の誤差(reconstruction error)を最小化することで行われる。 • この操作によって、入力値をより適切な表現に写像する auto-encoder が得られる。 38 (Auto-)encoder Decoder Reconstruction Representation Vector t-th Input Vector Output Vector Reconstruction Error
  • 33. ADGENDA 39 1. Introduction 2. Semi-supervised Embedding 3. Semi-supervised Embedding for Deep Learning 4. Existing Approaches to Deep Learning 5. Experimental Result 6. Conclusion
  • 34. 5. Experimental Result 43
  • 35. ADGENDA 47 1. Introduction 2. Semi-supervised Embedding 3. Semi-supervised Embedding for Deep Learning 4. Existing Approaches to Deep Learning 5. Experimental Result 6. Conclusion
  • 36. 6. Conclusion • In this work, we showed how one can improve supervised learning for deep architectures if one jointly learns an embedding task using unlabeled data. • Our results both confirm previous findings and generalize them. • Researchers using shallow architectures already showed two ways of using embedding to improve generalization • (i) embedding unlabeled data as a separate pre-processing step (i.e., first layer training) • (ii) using embedding as a regularizer (i.e., at the output layer). • More importantly, we generalized these approaches to the case where we train a semi- supervised embedding jointly with a supervised deep multi-layer architecture on any (or all) layers of the network 48
  • 37. 研究内容: Like Prediction 49
  • 38. 研究内容: Like Prediction 50
  • 39. ディープ・アーキテクチャを用いたグラフの階層的クラスタリング 51 オートエンコーダ グラフのクラスタ(コミュニ ティ) グラフ
  • 40. DISCUSSION 52