Many gradient-based meta-learning methods assume a set of parameters that do not participate in inner-optimization, which can be considered as hyperparameters. Although such hyperparameters can be optimized using the existing gradient-based hyperparameter optimization (HO) methods, they suffer from the following issues. Unrolled differentiation methods do not scale well to high-dimensional hyperparameters or horizon length, Implicit Function Theorem (IFT) based methods are restrictive for online optimization, and short horizon approximations suffer from short horizon bias. In this work, we propose a novel HO method that can overcome these limitations, by approximating the second-order term with knowledge distillation. Specifically, we parameterize a single Jacobian-vector product (JVP) for each HO step and minimize the distance from the true second-order term. Our method allows online optimization and also is scalable to the hyperparameter dimension and the horizon length. We demonstrate the effectiveness of our method on two different meta-learning methods and three benchmark datasets.

1. Online Hyperparameter Meta-Learning
with Hypergradient Distillation
Hae Beom Lee1, Hayeon Lee1, Jaewoong Shin3,
Eunho Yang1,2, Timothy Hospedales4,5, Sung Ju Hwang1,2
KAIST1, AITRICS2, Lunit3, University of Edinburgh4, Samsung AI Centre Cambridge5
ICLR 2022 spotlight

2. Meta-Learning
Unseen tasks
Knowledge
Transfer !
Meta-test
Test
Test
Training Test
Training
Training
Meta-training
S. Ravi, H. Larochelle, Optimization as a Model for Few-shot Learning, ICLR 2017
C. Finn, P, Abbeel, S. Levine, Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks, ICML 2017
MAML (Finn et al., ‘17)
• Humans generalize well because we never learn from scratch.
• Learn a model that can generalize over a distribution of tasks.

3. Hyperparameters in Meta-Learning
Whole feature extractor
Element-wise learning rates Interleaved (e.g. “Warp”) layers
A. Raghu*, M. Raghu*, S. Bengio, O. Vinyals, Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML, ICLR 2020
S. Flennerhag, A. A. Rusu, R. Pascanu, F. Visin, H. Yin, R. Hadsell, Meta-Learning with Warped Gradient Descent, ICLR 2020
Z. Li, F. Zhou, F. Chen, H. Li, Meta-SGD: Learning to Learn Quickly for Few-Shot Learning, 2017
• The parameters that do not participate in inner-optimization → Hyperparameters in meta-learning.
• They are usually high-dimensional.

4. Hyperparameter Optimization (HO)
D. Maclaurin, D. Duvenaud, R. P. Adams, Gradient-based Hyperparameter Optimization through Reversible Learning, ICML 2015
https://towardsdatascience.com/shallow-understanding-on-bayesian-optimization-324b6c1f7083
J. Bergstra, Y. Bengio. Random search for hyper-parameter optimization, 2012
• Hyperparameter optimization (HO): a problem of choosing a set of optimal hyperparameters for
a learning algorithm.
• Which method should we use for such high-dimensional hyperparams?
Random Search Bayesian Optimization Gradient-based HO
Not scalable to
hyperparameter
dimension
Whole feature extractor
Element-wise learning rates Interleaved (e.g. “Warp”) layers

5. In Case of Few-shot Learning
• In case of few-shot learning, computing the exact gradient w.r.t. the hyperparameter (i.e.
hypergradient) is not too expensive.
• A few-gradient steps are sufficient for each task.
Shared
initialization
5 steps
5 steps
5 steps
few-shot task
few-shot task
few-shot task

6. In Case of Few-shot Learning
• In case of few-shot learning, computing the exact gradient w.r.t. the hyperparameter (i.e.
hypergradient) is not too expensive.
• A few-gradient steps are sufficient for each task.
Shared
initialization
backprop
backprop
backprop
few-shot task
few-shot task
few-shot task

7. • Many-shot learning → Only a few gradient steps?
Shared
initialization
HO Does Matter when Horizon Gets Longer
SVHN
Flowers
→ Meta-learner may suffer from the short-horizon bias (Wu et al. ‘18).
J. Shin*, H. B. Lee*, B. Gong, S. J. Hwang, Large-Scale Meta-Learning with Continual Trajectory Shifting, ICML 2021
Cars
Y. Wu*, M. Ren*, R. Liao, R. Grosse, Understanding Short-Horizon Bias in Stochastic Meta-Optimization, ICLR 2018
?
5 steps
5 steps
5 steps

8. • Many-shot learning → requires longer inner-learning trajectory
Shared
initialization
HO Does Matter when Horizon Gets Longer
SVHN
Flowers
Cars

9. Shared
initialization
b….a….c….k….p….r….o….p....
HO Does Matter when Horizon Gets Longer
→ Computing a single hypergradient becomes too expensive!
• Many-shot learning → requires longer inner-learning trajectory
SVHN
Flowers
Cars

10. HO Does Matter when Horizon Gets Longer
→ Offline method: interval between two adjacent meta-updates is too long…
→ Meta-convergence is poor.
• Many-shot learning → requires longer inner-learning trajectory
1000 step forward
1000 step backward
meta-update meta-update meta-update
Shared
initialization
SVHN
Flowers
Cars
new trajectory
new trajectory
new
trajectory

11. HO Does Matter when Horizon Gets Longer
→ Online method: update hyperparamer every inner-grad step!
• Many-shot learning → requires longer inner-learning trajectory
Shared
initialization
Flowers

12. HO Does Matter when Horizon Gets Longer
→ Online method: update hyperparamer every inner-grad step!
• Many-shot learning → requires longer inner-learning trajectory
Shared
initialization
Flowers update

13. HO Does Matter when Horizon Gets Longer
→ Online method: update hyperparamer every inner-grad step!
• Many-shot learning → requires longer inner-learning trajectory
Shared
initialization
Flowers

14. HO Does Matter when Horizon Gets Longer
→ Online method: update hyperparamer every inner-grad step!
• Many-shot learning → requires longer inner-learning trajectory
Shared
initialization
Flowers
update

15. HO Does Matter when Horizon Gets Longer
→ Online method: update hyperparamer every inner-grad step!
• Many-shot learning → requires longer inner-learning trajectory
Shared
initialization
Flowers

16. HO Does Matter when Horizon Gets Longer
→ Online method: update hyperparamer every inner-grad step!
• Many-shot learning → requires longer inner-learning trajectory
Shared
initialization
Flowers update

17. HO Does Matter when Horizon Gets Longer
→ Online method: update hyperparamer every inner-grad step!
• Many-shot learning → requires longer inner-learning trajectory
Shared
initialization
Flowers much faster
meta-convergence!

18. Shared
initialization
SVHN
Flowers
?
Criteria of Good HO Algorithm for Meta-Learning
3. Computing a single hypergradient
should not be too expensive
4. Update hyperparam every inner-grad step
i.e. online optimization
1. Scalable to hyperparameter dimension 2. Less or no short-horizon bias
Shared
initialization
b….a….c….k….p….r….o….p....
b….a….c….k….p….r….o….p….
b
…
.
a
…
.
c
…
.
k
…
.
p….r….o….p….
SVHN
Flowers
Cars
Whole feature extractor
Element-wise learning rates Interleaved (e.g. “Warp”) layers
Shared
initialization
Flowers
…
much faster
meta-convergence!

19. Limitations of Existing Grad-based HO Algs
Unfortunately, the existing gradient-based HO algorithms do not satisfy all the criteria simultaneously.
Criteria FMD
1. Scalable to hyperparam dim
2. Less or no short horizon bias
3. Constant memory cost
4. Online optimization
o
o
o
x
L. Franceschi, M. Donini, P. Frasconi, M. Pontil, Forward and Reverse Gradient-Based Hyperparameter Optimization, ICML 2017

20. Limitations of Existing Grad-based HO Algs
Unfortunately, the existing gradient-based HO algorithms do not satisfy all the criteria simultaneously.
Criteria FMD RMD
1. Scalable to hyperparam dim
2. Less or no short horizon bias
3. Constant memory cost
4. Online optimization
o
o
o
x
o
x
x
o
L. Franceschi, M. Donini, P. Frasconi, M. Pontil, Forward and Reverse Gradient-Based Hyperparameter Optimization, ICML 2017

21. Limitations of Existing Grad-based HO Algs
Unfortunately, the existing gradient-based HO algorithms do not satisfy all the criteria simultaneously.
Criteria FMD RMD IFT
1. Scalable to hyperparam dim
2. Less or no short horizon bias
3. Constant memory cost
4. Online optimization
o
o
o
x
o
x
x
o
o
o
△
o
J. Lorraine, P. Vicol, D. Duvenaud, Optimizing Millions of Hyperparameters by Implicit Differentiation, AISTATS 2020

22. Limitations of Existing Grad-based HO Algs
Unfortunately, the existing gradient-based HO algorithms do not satisfy all the criteria simultaneously.
Criteria FMD RMD IFT 1-step
1. Scalable to hyperparam dim
2. Less or no short horizon bias
3. Constant memory cost
4. Online optimization
o
o
o
x
o
x
x
o
o
o
△
o
x
o
o
o
Shared
initialization
J. Luketina, M. Berglund, K. Greff, T. Raiko, Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters, ICML 2016

23. Goal of This Paper
This paper aims to overcome all the aforementioned limitations at the same time.
Criteria FMD RMD IFT 1-step Ours
1. Scalable to hyperparam dim
2. Less or no short horizon bias
3. Constant memory cost
4. Online optimization
o
o
o
x
o
x
x
o
o
o
△
o
x
o
o
o
o
o
o
o
Hypergradient distillation

24. Hypergradient Distillation
Shared
initialization

25. Shared
initialization
Hypergradient Distillation
requires 2t – 1 JVP computations (e.g. RMD)
respose Jacobian
hypergradient

26. Hypergradient Distillation
Requires only 1 JVP computation
But it suffers from short horizon bias
Shared
initialization
hypergradient respose Jacobian

27. Hypergradient Distillation
2t – 1 JVP
hypergradient
Shared
initialization
a single JVP
distilled weight and dataset
→ hypergrad direction
scaling factor
→ hypergrad size
𝑤𝑡, 𝐷𝑡
𝑤2, 𝐷2
𝑤1, 𝐷1
• it does not require computing the actual 𝑔𝑡
𝑆𝑂
.
• we only need to keep updating a moving
average of 𝑤𝑡
∗
and 𝐷𝑡
∗
.
• the scaling factor 𝜋∗
is also efficiently estimated
with a function approximator.
• we can approximately solve the distillation
problem efficiently.
Please read the main paper for
the technical details !
For each online HO step 𝒕,
distill
𝑤𝑡
∗
, 𝐷𝑡
∗

28. Experimental Setup
J. Shu, Q. Xie, L. Yi, Q. Zhao, S. Zhou, Z. Xu, D. Meng, Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting, NeurIPS 2019
ANIL (Raghu et al. ‘20) WarpGrad (Flennerhag et al. ‘20) Meta-Weight-Net (Shu et al. ‘20)
A. Nichol, J. Achiam, J. Schulman, On First-Order Meta-Learning Algorithms, 2018
Meta-learning
models
Other details
CIFAR100
tinyImageNet
• Inner-grad step = 100
• Use Reptile for learning
shared initialization
Task distribution
• 10-way 250-shot

29. Experimental Results
Q1. Does HyperDistill provide faster convergence?
Meta-training convergence (Test Loss)

30. Experimental Results
Q2. Does HyperDistill provide better generalization performance?
Meta-validation performance (Test Acc)
Meta-test performance (Test Acc)

31. Experimental Results
Q3. Is HyperDistill a reasonable approximation to the true hypergradient?
Cosine similarity to the true hypergradient
Q4. Is HyperDistill computationally efficient?
GPU memory consumption and wall-clock runtime

32. Conclusion
• The existing gradient-based HO algorithms do not satisfy the four criteria that should be met for
their practical use in meta-learning.
• In this paper, we showed that for each online HO step, it is possible to efficiently distill the whole
hypergradient indirect term into a single JVP, satisfying the four criteria simultaneously.
• Thank to the accurate hypergradient approximation, HyperDistill could improve meta-training
convergence and meta-testing performance, in a computationally efficient manner.
github.com/haebeom-lee/hyperdistill