Group meeting

1. HyperPrompt
HyperPrompt:
Prompt-based Task-Conditioning of Transformers
Yun He, Huaixiu Steven Zheng, Yi Tay et al.
National Yang Ming Chiao Tung University, Hsinchu
Speaker: Po-Chuan Chen
March 7, 2023
1 / 37

2. HyperPrompt
Table of contents
1 Abstract
2 Introduction
3 Problem Statement
4 Methods
5 Experiments
6 Related Work
7 Conclusion
2 / 37

3. HyperPrompt
Abstract
Table of contents
1 Abstract
2 Introduction
3 Problem Statement
4 Methods
5 Experiments
6 Related Work
7 Conclusion
3 / 37

4. HyperPrompt
Abstract
Abstract
This paper proposes HyperPrompt, a novel architecture for
prompt-based task-conditioning of self-attention in Transformers.
HyperPrompt allows the network to learn task-specific feature maps
where the hyper-prompts serve as task global memories for the queries
to attend to, at the same time enabling flexible information sharing
among tasks.
HyperPrompt is competitive against strong multi-task learning
baselines with as few as 0.14% of additional task-conditioning
parameters, achieving great parameter and computational
efficiency.
4 / 37

5. HyperPrompt
Introduction
Introduction
Prompts are lightly tuned, allowing the model to be trained quickly
since the main body of the pretrained model is kept frozen.
HyperPrompt, a natural but novel extension of Prompt-Tuning to
multi-task learning (MTL) for language.
Hyper-prompts are injected to the keys and values in the
self-attention module, reminiscent of memory augmented
Transformers.
5 / 37

6. HyperPrompt
Introduction
Introduction
This mitigates the cost of having prompts pass through the standard
FFN layers in Transformers and serves as additional task-specific
memory tokens for queries to attend to.
Also, introducing task-aware and layer-aware HyperNetworks, it
imbues the model with the necessary flexibility and expressiveness
Meanwhile, HyperPrompt remains very parameter and computational
efficient and friendly to multi-task scaling.
6 / 37

7. HyperPrompt
Introduction
Contribution
1 HyperPrompt Transformer architecture with learnable
hyper-prompts for multi-task fine-tuning with great parameter
and computational efficiency
2 HyperNetworks as a prompt generator, and inject hyper-prompts
into the self-attention module as global task memory tokens.
3 HyperPrompt outperforms state-of-the-art parameter-efficient T5
models using Prompt-Tuning or adapters on well-established
benchmarks such as SuperGLUE and GLUE, across all explored
model sizes.
7 / 37

8. HyperPrompt
Problem Statement
Table of contents
1 Abstract
2 Introduction
3 Problem Statement
4 Methods
5 Experiments
6 Related Work
7 Conclusion
8 / 37

9. HyperPrompt
Problem Statement
Problem Statement
Designing task-conditioned parameterization of Transformer models
to achieve greater parameter and computational efficiency as well
as Pareto efficiency for multi-task learning.
Some related topics:
Multi-task learning
Transformer-based pre-trained LMs
9 / 37

10. HyperPrompt
Problem Statement
Multi-task learning
A set of tasks {D𝜏 } = {x(n)
𝜏 , y(n)
𝜏 }N𝜏
n=1 indicates the corresponding
training set of the 𝜏-th task with N𝜏 samples.
To tackle such multi-task learning problem with pre-trained
Transformer model f𝜃 (·), it needs to minimize the objective function
L(𝜃) =
T
∑︁
𝜏=1
N𝜏
∑︁
n=1
C
f𝜃
x(n)
𝜏
, y(n)
𝜏
where C(·, ·) is typically the cross-entropy loss and f𝜃 (x(n)
𝜏 ) is the
output for training sample x(n)
𝜏 .
10 / 37

11. HyperPrompt
Problem Statement
Transformer-based pre-trained LMs
For example, T5 and BART are unified text-to-text frameworks where
all tasks share the same encoder-decoder architecture.
{{x(n)
𝜏 }N𝜏
n=1}T
𝜏=1 are fed into the same encoder
{{ŷ(n)
𝜏 }N𝜏
n=1}T
𝜏=1 are generated by the same decoder.
For such universal modules, multi-task learning simply corresponds to
mixing task data sets and there is no task-specific classification or
regression networks for each task as in encoder-only modules.
11 / 37

12. HyperPrompt
Problem Statement
Problem Statement
To improve the performance, they introduce a set of task-conditioned
parameters {𝛿𝜏 }T
𝜏=1 into f𝜃 (·).
Such that, the objective function can be updated as
L
𝜃, {𝛿𝜏 }T
𝜏=1
=
T
∑︁
𝜏=1
N𝜏
∑︁
n=1
C
f𝜃,𝛿𝜏
x(n)
𝜏
, y(n)
𝜏
During training, both 𝜃 and {𝛿𝜏 }T
𝜏=1 are updated via back-propagation
because they observe a large performance drop in SuperGLUE when
backbone model 𝜃 is frozen and only task-conditioned parameters are
tuned.
12 / 37

13. HyperPrompt
Methods
Table of contents
1 Abstract
2 Introduction
3 Problem Statement
4 Methods
5 Experiments
6 Related Work
7 Conclusion
13 / 37

14. HyperPrompt
Methods
Methods
Prompt-Based Task-Conditioned Transformer
HyperPrompt
HyperPrompt-Global
Parameter Efficiency of HyperPrompt
14 / 37

15. HyperPrompt
Methods
Methods
15 / 37

16. HyperPrompt
Methods
Prompt-Based Task-Conditioned Transformer
To learn the task-specific information for the 𝜏-th task, having l
trainable d-dimensional vectors as the hyper-prompts for the key and
the value respectively denoted as P𝜏,k ∈ Rl×h×dh and P𝜏,v ∈ Rl×h×dh .
Then, the hyper-prompts are concatenated with the original key and
value:
K′
𝜏 = concat(P𝜏,k, K𝜏) (1)
V′
𝜏 = concat(P𝜏,v, V𝜏) (2)
where the new key (value) K′
𝜏 (V′
𝜏) ∈ R(l+L)×h×dh are used to compute
the multihead self-attention.
16 / 37

17. HyperPrompt
Methods
Prompt-Based Task-Conditioned Transformer
The hyper-prompts benefit Transformers for multitask learning in two
ways:
Prompt for key P𝜏,k directly interacts (matrix multiplication)
with the original query Q𝜏 allowing tokens to acquire
task-specific semantics.
Prompt for value P𝜏,v can serve as task-specific memories for
multihead attention to retrieve information.
After that, the multihead self-attention can be operated:
O𝜏 = Attention Q𝜏, K′
𝜏, V′
𝜏
= softmax(Q𝜏K′T
𝜏 V′
𝜏)
where O𝜏 ∈ RL×d is the output of multihead attention.
17 / 37

18. HyperPrompt
Methods
How to obtain the prompts for the m-th Transformer block?
Global Prompts, which initialize a set of global prompts {P𝜏 }T
𝜏=1,
where P𝜏 ∈ Rl×d is a trainable matrix to learn the task-specific
information of the 𝜏-th task, d is the model dimension and l is the
length of the prompt.
Local HyperNetworks, that applying two local HyperNetworks hm
k
and hm
v to transform the global prompt P𝜏 into layer-specific and
task-specific prompts
Pm
𝜏,k = hm
k (P𝜏) = Um
k Relu Dm
k (P𝜏)
(3)
Pm
𝜏,v = hm
v (P𝜏) = Um
v Relu Dm
v (P𝜏)
(4)
where Dm
v and Um
v are down-projection and up-projection matrices.
18 / 37

19. HyperPrompt
Methods
HyperPrompt-Share share the same two local HyperNetworks
defined by the down-project matrices Dm
v and Dm
k , and the up-project
matrices Um
v and Um
k .
HyperPrompt-Sep each task can have its own local HyperNetworks
hm
𝜏,k(P𝜏) and hm
𝜏,v(P𝜏)
Pm
𝜏,k = hm
𝜏,k (P𝜏) = Um
𝜏,k
Relu
Dm
𝜏,k (P𝜏)
(5)
Pm
𝜏,v = hm
𝜏,v (P𝜏) = Um
𝜏,v Relu Dm
𝜏,v (P𝜏)
(6)
19 / 37

20. HyperPrompt
Methods
HyperPrompt-Global
It can flexibly share information and knowledge among tasks and
blocks while maintaining a low parameter cost.
Layer-Aware Task Embedding
Im
𝜏 = ht (k𝜏, zm), where ht is a MLP consisting of two feed-forward
layers and a ReLU non-linearity and k𝜏 ∈ Rt′
denote the task
embedding, and zm ∈ Rt′
is the layer embedding.
20 / 37

21. HyperPrompt
Methods
HyperPrompt-Global
Global HyperNetworks
Having Hk(·) to generates the weight matrices for key hyper-prompts,
and Hv(·) to generate for value hyper-prompts.
Um
𝜏,k, Dm
𝜏,k
= Hk Im
𝜏
=
WUk
, WDk
Im
𝜏 (7)
Um
𝜏,v, Dm
𝜏,v
= Hv Im
𝜏
=
WUv
, WDv
Im
𝜏 (8)
After Um
𝜏,k/v and Dm
𝜏,k/v are generated, the method can project the
global prompts P𝜏,k/v into into hyper-promtps Pm
𝜏,k/v following (5)
and (6). And then, calculating the task-conditioned attention scores
by using Figure 2(a).
21 / 37

22. HyperPrompt
Methods
Parameter Efficiency of HyperPrompt
The total number of additional parameters from HyperPrompt-Global
dlT + 4(bdt) + Tt′
+ Mt′
+ (2t′
+ t)e
where T is the total number of tasks, b is the bottleneck dimension of
the weight matrices of the local HyperNetworks, t′/t is the dimension
of the raw/final layer-aware task embedding, and e is the hidden
dimension of hk/v.
Therefore, the space complexity is O(d(lT + 4bt)) or further
simplified as O(bdt).
22 / 37

23. HyperPrompt
Experiments
Table of contents
1 Abstract
2 Introduction
3 Problem Statement
4 Methods
5 Experiments
6 Related Work
7 Conclusion
23 / 37

24. HyperPrompt
Experiments
Experiments
Experimental Setup
Key Results
Tuning all vs Task-Conditioned Parameters
Computational Efficiency
Ablation Study
Peeking into Hyper-Prompts
Impact of Hyper-Prompt Length
Encoder vs Decoder
24 / 37

25. HyperPrompt
Experiments
Experimental Setup
Datasets: GLUE and SuperGLUE
Transformers: Transformer model T5
Evaluation:
Testing the ability of transfer learning among the co-trained tasks.
Baselines:
1 MTL-Prompt and HyperPrompt-Share/Sep/Global with vanilla
T5 models
2 Vanilla Adapter
3 HyperFormer++
4 Prompt-Tuning
25 / 37

26. HyperPrompt
Experiments
26 / 37

27. HyperPrompt
Experiments
Tuning all vs Task-Conditioned Parameters
The experiments show that only tuning the task-conditioned
parameters is not enough to achieve competitive results as full model
training for multi-task learning, since the rest of the experiments are
conducted with tuning all model parameters.
27 / 37

28. HyperPrompt
Experiments
Computational Efficiency
This shows the computational efficiency of HyperPrompt for both
training and inference.
28 / 37

29. HyperPrompt
Experiments
Ablation Study
HyperPrompt-Global can adjust the degree of information sharing for
better multi-task generalization.
29 / 37

30. HyperPrompt
Experiments
For the top graph, the network has lower attention mass on
hyper-prompts in the lower layers and gradually increases attention
mass for higher layers.
30 / 37

31. HyperPrompt
Experiments
For the bottom graph, knowing that injecting hyper-prompts
encourages a more diverse attention distribution, which seems to be
beneficial to model generalization.
31 / 37

32. HyperPrompt
Experiments
Impact of Hyper-Prompt Length
As shoen in Figure, l = 6 is the best for the decoder, and l = 16 for the
encoder with l = 6 fixed for the decoder.
32 / 37

33. HyperPrompt
Experiments
Encoder vs Decoder
Based on this experiment, this paper adds task-conditioned parameters
to the decoder for SuperGLUE in their experiments.
33 / 37

34. HyperPrompt
Related Work
Table of contents
1 Abstract
2 Introduction
3 Problem Statement
4 Methods
5 Experiments
6 Related Work
7 Conclusion
34 / 37

35. HyperPrompt
Related Work
Related Work
1 Prompt-Tuning
1 Discrete prommpting words
2 Soft prompts
2 Adapter-Tuning
3 Multi-task Natural Language Understanding
35 / 37

36. HyperPrompt
Conclusion
Table of contents
1 Abstract
2 Introduction
3 Problem Statement
4 Methods
5 Experiments
6 Related Work
7 Conclusion
36 / 37

37. HyperPrompt
Conclusion
Conclusion
This paper propose a novel architecture for prompt-based
task-conditioning of self-attention in Transformers.
The hyper-prompts are generated by a HyperNetwork to enable
flexible information sharing among tasks while remain efficient in
parameters and computation.
37 / 37