ICML 2017 Meta network
•Download as PPTX, PDF•
2 likes•953 views
1) The Meta Network model proposes a two-level learning approach for few-shot learning. It includes a slow-learning meta-learner and a fast-learning base learner. 2) The meta-learner learns to generate fast weights for the base learner using gradient-based meta information from previous tasks. It stores these weights in a memory indexed by task embeddings. 3) Experiments on few-shot classification datasets like Omniglot and MiniImageNet demonstrate the model can learn new concepts from very few examples through fast adaptation of the base learner's weights.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to ICML 2017 Meta network
Similar to ICML 2017 Meta network (20)
Recently uploaded
Recently uploaded (20)
ICML 2017 Meta network
- 1. Meta Network ICML 2017 citation: 0 Tsendsuren Munkhdalai Hong Yu University of Massachusetts, MA, USA Katy Lee @ Datalab 2017.09.11 1
- 2. Background • two level learning on meta learning: • slow learning of a meta-level model preforming across tasks • rapid learning of a base-level model acting with each task 2
- 3. Motivation • We want the neural network to learn and to generalize a new task or concept from a single example on the fly. 3
- 4. Related Work • Santoro et al., Meta-learning with memory- augmented neural networks. ICML 2016. -> use the external memory as temporary memory 4
- 5. • Ravi, Sachin and Larochell, Hugo. Optimization as a model for few-shot learning. ICLR 2017 5
- 6. • Ravi, Sachin and Larochell, Hugo. Optimization as a model for few-shot learning. ICLR 2017 6
- 7. • Ravi, Sachin and Larochell, Hugo. Optimization as a model for few-shot learning. ICLR 2017 7
- 10. Base Learner b: overlook • Unlike standard neural nets, b is parameterized by slow weights W and example-level fast weights W* • slow weights: updated via a learning algorithm during training • fast weights: generated by the meta learner for every input. 29
- 11. How Base Learner b provides meta info • X’ belongs to support set • The meta information is derived from the base learner in form of the loss gradient information: 30
- 12. Model W W* u (Q, Q*) m(Z) d(G) index R 31
- 13. Meta Learner • function: collects meta-info. and produce per example fast weight W* for the base learner when training • fast weight W* generating function m with parameter Z • fast weight Q* generating function d with parameter G • a dynamic representation learning function u (Q, Q*) 32
- 14. • m learns the mapping from the loss gradient to the fast weights for the support set(not the final ones output to the base learner) • we store the fast weights in a memory M which is indexed with task dependent embeddings R = {ri}i=1 to n of the support examples, obtained by u Meta Learner: m 33
- 15. Memory index R r1 W*1 r2 W*2 r3 W*3 r4 W*4 34
- 16. Meta Learner:u • a dynamic representation learning network u (Q, Q*) • We generate the fast weights Q* on a per task basis as follows: 35
- 17. Meta Learner • Once the fast weights are generated, the task dependent supper set input representations are computed as: • r’i =u(Q,Q*,x’i) • Q and Q* are integrated using the layer augmentation method • => Memory is ready! 36
- 18. Meta Learner • After the fast weights Wi* are stored in the memory M and the index R is constructed • given input xi in the training set/test set 1. embed xi using the dynamic representation u, ri =u(Q,Q*,xi) 2. reads the memory with soft attention 37
- 20. Model W W* u (Q, Q*) m(Z) d(G) index R b 5 conv with 64 filter maxpool 2 fc u 5 conv with 64 filter maxpool 2 fc d, m 3 fc with 20 neurons 39
- 21. • W, W*: slow and fast weight for the base learner • Q, Q*: slow and fast weight for representational learning • Z: slow weight for m, a nn that generates fast weight W* • G: slow weight for m, a nn that generates fast weight Q* 40
- 22. 41
- 23. On support set memory ready now! N: size of support set 42
- 24. On training set W, Q, Z, G L: size of support set R: index of memory 43
- 25. One-shot Learning exp. • Omniglot previous split • Omniglot standard split. • Mini-ImageNet 44
- 26. • Omniglot previous split • Omniglot standard split. • Mini-ImageNet One-shot Learning exp. 45
- 27. Omniglot Previous Split • following matching network exp. settings • 1200 training classes, 423 testing classes, 20 examples per class • three variations of MetaNet • MetaNet+: additional task-level weight for base learner • MetaNet • MetaNet-: not task-level fast weight W* for meta-learner 46
- 29. • Omniglot previous split • Omniglot standard split • Mini-ImageNet One-shot Learning exp. 48
- 30. Omniglot stardard split • 1200 -> 964 training classes, 423 -> 659 testing classes 49
- 32. One-shot Learning exp. • Omniglot previous split • Omniglot standard split. • Mini-ImageNet 51
- 33. Mini-ImageNet • 64 training classes, 20 testing classes • 600 examples per class 52
- 34. Mini-ImageNet 53
- 35. Generalization Experiment • N-way training and K-way testing • Rapid Parameterization of Fixed Weight • Meta-level continual learning 54
- 36. Generalization Experiment • N-way training and K-way testing • Rapid Parameterization of Fixed Weight • Meta-level continual learning 55
- 37. N-way training and K-way testing 56
- 38. Generalization Experiment • N-way training and K-way testing • Rapid Parameterization of Fixed Weight • Meta-level continual learning 57
- 39. Generalization Experiment • replace the base learner with a new CNN during evaluation. • the fast weight of the new CNN is generated by the meta learner that is trained to paramerterize the old based leaner(target CNN) 58
- 40. Rapid Parameterization of Fixed Weight • small: 32 filters • target: 64 filters • big: 128 filters 59
- 41. Generalization Experiment • N-way training and K-way testing • Rapid Parameterization of Fixed Weight • Meta-level continual learning 60
- 42. Meta-level continual learning • train and test on Omniglot -> train on MNIST -> train and test on Omniglot again 61
- 43. Meta-level continual learning acc difference = after - before(on Omniglot) 62
- 44. Conclusion • pro: • Interesting model, slow and fast weight have different functions • Solid experiment • con: • It’s kinda hard to read 63
- 45. Future Work • other meta information other than gradient • detect task/domain automatically 64
Editor's Notes
- @author
- The goal of a meta-level learner is to acquire generic knowledge of different tasks. The knowledge can then be transferred to the base-level learner to provide generalization in the context of a single task. —————- The base and meta-level models can be framed in a single learner (Schmidhuber, 1987) (??) or in separate learners (Bengio et al., 1990; Hochreiter et al., 2001).
- It must learn to hold data samples in memory until the appropriate labels are presented at the next time-step, after which sample-class information can be bound and stored for later use @relation to this work @TODO put more related work
- we propose an LSTM- based meta-learner model to learn the exact optimization algorithm used to train another learner neural network classifier in the few-shot regime. They split the network to meta-train and meta-test. during meta-train phase, they train the meta-learner to optimise the optimizer @relation to this work
- They split the network to meta-train and meta-test. during meta-train phase, they train the meta-learner to optimise the optimizer @relation to this work
- when considering each dataset D ∈ D_meta−train, the training objective(for meta learner?) we use is the loss L_test of the produced classifier on D’s test set D_test. While iterating over the examples in D’s training set D_train, at each time step t the LSTM meta-learner receives (∇_θt−1 Lt , Lt ) from the learner (the classifier) and proposes the new set of parameters θt . The process repeats for T steps, after which the classifier and its final parameters are evaluated on the test set to produce the loss that is then used to train the meta-learner. M? ＠TODO shortly explain
- 非常重要
- working memory -> short term nunnery -> long term memory?
- @TODO 例子？
- fast weight A clears the states by the contextual overlay
- intermediate steps between the recurrent computation fast weight clears things(from the state?) up from the contextual overlay, storing the useful history information of **this sequence**
- instead of using iterative learning backdrop one shot learning borrowing from hop field network fast weight is retrieving information from the hop field network youtube video 06:30
- It was downsampled to 48 ⇥ 48 greyscale. The full dataset contains 15 views since facial expressions were not discernible from the more extreme viewpoints. The resulting dataset contained > 100, 000 images. 317 identities appeared in the training set with the remaining 20 identities in the test set. Given the input face image, the goal is to classify the subject’s facial expression into one of the six different categories: neutral, smile, surprise, squint, disgust and scream. - Not only does the dataset have unbalanced numbers of labels, some of the expressions, for example squint and disgust, are are very hard to dis- tinguish. In order to perform well on this task, the models need to generalize over different lighting conditions and viewpoints.
- where do we store the information when examining the eyebrow in convnet: in fast weght: stack like mechanism
- @TODO biological feasible? The term in square brackets is just the scalar product of an earlier hidden state vector, h(⌧ ), with the current hidden state vector, hs(t+1), during the iterative inner loop. So at each iteration of the inner loop, the fast weight matrix is exactly equivalent to attending to past hidden vectors in proportion to their scalar product with the current hidden vector, weighted by a decay factor. During the inner loop iterations, attention will become more focussed on past hidden states that manage to attract the current hidden state. -> attract 是長得像的意思嗎？
- hopeful net is not that efficient can only store log n of pattern compare with associative LSTM (train with backprop) -> use the hop field net more efficiently
- @TODO is this page necessary?
- There are three major components of this model, the memory, the meta-learner and the base learner the meta learner is used to produce the fast weight for the base learner when each training example comes in the meta learner will make use of the support set to set up its own per task fast weight Q* in lots of few shot learning paper, learns a network that maps a small labelled support set and an unlabelled example to its label in this setting, they use only one example in support set so it’s cheap to obtain
- Let’s take a look at the base learner here
- the difference between normal neural network and this base learner is that it has fast weight
- we will see how it make use of the support set to give meta-information to the meta learner The base learner uses a representation of meta information obtained by using a support set, to provide the meta learner with feedbacks about the new input task. 有一撇是support set Here Li is the loss for support examples {x0i,yi0}Ni=1. N is the number of support examples in the task set (typically a single instance per class in the one-shot learning setup). delta_i is the loss gradient with respect to parameters W and is our meta information. Note that the loss function loss_task is generic and can take any form, such as a cumulative reward in reinforcement learning. For our one-shot classification setup we use cross-entropy loss. The meta learner takes in the gradient information nabla_i and generates the fast parameters W’ as in Equation 1 (and store it in the memoery). - Alternatively the base learner can take as input the task specific representations {ri}Li=1 produced by the dynamic representation learning network, effectively reducing the number of MetaNet parameters and leveraging shared representations. In this case, the base learner is forced to operate in the dynamic task space constructed by u instead of building new representations from the raw inputs {xi}Li=1
- Now we can take a look at the meta learner, we can see how it receives the meta info, learns a good representation function to embed the example. and store the fast weight in the memory for future training use. @TODO: Question: memory是會一直累加嗎？每次new task會把舊的洗掉嗎？
- This is not the final fast weight delta i is from the support set 接下來講怎麼產生Q* The representation learning function u is a neural net parameterized by slow weights Q and task-level fast weights Q⇤. It uses the representation loss lossemb to capture a representation learning objective and to obtain the gradients as meta information. We generate the fast weights Q⇤ on a per task basis as follows:
- @external This is not the final fast weight delta i is from the support set The fast weights are then stored in a memory M = {Wi⇤ }Ni=1 . The memory M is indexed with task dependent embeddings R = {ri0 }Ni=1 of the support examples {x0i }Ni=1 , obtained by the dynamic representation learning function u.
- these are the fast weight generated from the support set with the representation of the supper set examples as the index @不同task的fast weight會一直累加？
- @hard to understand d denotes a neural net parameterized by G, that accepts variable sized input. First, we sample T examples (T < N ) {x0i , yi0 }Ti=1 from the support set and obtain the loss gradient as meta information. Then d observes the gradient corresponding to each sampled example and summarizes into the task specific parameters
- the way it computes Q* , connects the knowledge of Q* and Q
- attention: cosine similarity norm: softmax Q and W will learn gradually across task Q* focus on task W* focus on example
- @TODO double check Intuitively, the fast and slow weights in the layer augmented neural net can be seen as feature detectors operating in two distinct numeric domains. The application of the non-linearity maps them into the same domain, which is [0, 1) in the case of ReLU so that the activations can be aggregated and processed further. Our aggregation func- tion here is element-wise sum.
- network architectue Omniglot we used a CNN with 64 filters as the base learner b. This CNN has 5 convolutional layers, each of which is a 3 x 3 convolution with 64 filters, followed by a ReLU non-linearity, a 2 x 2 max-pooling layer, a fully connected (FC) layer, and a softmax layer. Another CNN with the same architecture is used to define the dynamic representation learning function u, from which we take the output of the FC layer as the task dependent representation r. We trained a similar CNNs architecture with 32 filters for the experiment on Mini-ImageNet. However for computational efficiency as well as to demonstrate the flexibility of MetaNet, the last three layers of these CNN models were augmented by fast weights. For the networks d and m, we used a single-layer LSTM with 20 hidden units and a three-layer MLP with 20 hidden units and ReLU non-linearity. As in Andrychowicz et al. (2016), the parameters G and Z of d and m are shared across the coordinates of the gradients ∇ and the gradients are normalized using the same preprocessing rule (with p = 7). The MetaNet parameters θ are optimized with ADAM. The initial learning rate was set to 10−3. The model parameters θ were randomly initialized from the uniform distribution over [-0.1, 0.1).
- train W, Q, Z, G i from 1 to N: support set
- those with star are fast weight
- with respect to W @put images here @why sometimes sample, sometimes not support set those with star are fast weight
- those with star are fast weight
- @TODO MetaNet+ details three variations of MetaNet as ablation exp.
- @TODO MetaNet+ details @TODO read two papers Q* is useful additional task level weight is not that helpful
- @TODO MetaNet+ details three variations of MetaNet as ablation exp.
- @TODO MetaNet+ details three variations of MetaNet as ablation exp.
- @TODO MetaNet+ details three variations of MetaNet as ablation exp.
- train < test, decrease test > train, increase - how to augment the softmax?
- @TODO MetaNet+ details three variations of MetaNet as ablation exp.
- We replaced the entire base learner with a new CNN during evaluation. The slow weights of this network remained fixed. The fast weights are generated by the meta learner that is trained to parameterize the old base learner and used to augmented the fixed slow weights. The small CNN has 32 filters and the large CNN has 128 filters. target: 64
- @TODO during evaluation? based on support set -> predict big(orange) and small CNN (blue) are new base learner target CNN (filter # 64) is the original base learner that is learned along The performance difference between these models is large in earlier training iterations. However, as the meta learner sees more one-shot learning trials, the test ac- curacies of the base learners converge. This results show that MetaNet effectively learns to parameterize a neural net with fixed weights.
- **500 classes MNIST**, acc: 72% after 2400 MNIST trials
- **500 classes MNIST**, acc: 72% after 2400 MNIST trials reverse transfer learning train on omniglot -> mnist The positive values indicate that the training on the second problem automatically improves the performance of the earlier task exhibiting the reverse transfer property. Therefore, we can conclude that MetaNet successfully performs reverse transfer. At the same time, it is skilled on MNIST one-shot classification. The MNIST training accuracy reaches over 72% after 2400 MNIST trials. However, reverse transfer happens only up to a certain point in MNIST training (2400 trials). After that, the meta weights start to forget the Omniglot information. As a result from 2800 trials onwards, the Omniglot test accuracy drops. Nevertheless even after 7600 MNIST trials, at which point the MNIST training ac- curacy reached over 90%, the Omniglot performance drop was only 1.7%.
- ablation study on all components?