The document discusses multi-task learning in transformer-based architectures for natural language processing, highlighting techniques such as shared encoders, adapters, and hypernetworks. It emphasizes the advantages of these methods in terms of data efficiency, parameter modularity, and performance gains over traditional single-task learning. Various models and their training implications, along with the challenges of aligning datasets and managing gradient directions, are also explored.

1. Multi-Task Learning in
Transformer-Based
Architectures for NLP

2. ABOUT ME
Data Science Student at FER, Zagreb
NLP Researcher at Doxray
TIN FERKOVIĆ

3. TABLE OF CONTENTS
SINGLE TASK LEARNING, STL
N tasks = N models
SHARED ENCODER
+ task-specific heads
ADAPTERS
small, modular
components
HYPERNETWORKS
generate parameters
0 1
2 3

4. STL
N tasks = N models
0

5. BERT (large)
345 M (1.34 GB)
5.65 GB
64 TPUs
4 days
~$7,000
4 GB
PARAMETERS
GPU MEMORY
PRE-TRAINING
CHECKPOINT
284t of CO2
(average transatlantic flight)
CO2 EMISSION
SINGLE TASK LEARNING
Devlin, Jacob, et al. "Bert: Pre-training of deep bidirectional transformers for language understanding." (2018).

6. 6,000
TPU chips
And for today’s SOTA LLMs…
SINGLE TASK LEARNING

7. 50 days
Training time
And for today’s SOTA LLMs…
SINGLE TASK LEARNING

8. $10M
Estimated training cost
And for today’s SOTA LLMs…
SINGLE TASK LEARNING

9. SHARED
ENCODER
+ TASK-SPECIFIC HEADS
1

10. MOTIVATION
SINGLE MODEL
N times storage
reduction
DATA EFFICIENCY
Low-resource tasks
benefit
KNOWLEDGE SHARING
Gradient updates
from other tasks
SHARED ENCODERS

11. ARCHITECTURE
SHARED ENCODERS

12. Batches
Sequential
Homogeneous
Heterogeneous
DECISIONS
Catastrophic forgetting
1 batch = 1 task
Multiple tasks in a batch
SHARED ENCODERS

13. Sampling
Proportional/
Random
Temperature
Annealed
DECISIONS
Simple, causes underfitting
Ease difference in dataset
sizes
Train equally towards the
end of training
Uncertainty Active learning
SHARED ENCODERS

14. Loss weighting
Uncertainty
Learning speed
Performance
DECISIONS
Logit - ground truth
Convergence
Is it the same metric?
Normalized Diving point-loss by log(n)
SHARED ENCODERS

15. INTERFERENCE
Different gradient
directions
SAMPLING
Difficult
PERFORMANCE
Easily
outperformed
UNDER-/OVERFITTING
Problems with
different dataset sizes
LOSS WEIGHTING
Difficult
TASK GROUPING
Identify compatible
tasks
DRAWBACKS
SHARED ENCODERS

16. ADAPTERS
small, modular components
02

17. MOTIVATION
0.5 %
Trainable
parameters
MODULARITY
Train, save, inject
PERFORMANCE
On-par with STL
ADAPTERS

18. Augment the base model with
new task-specific sub-functions
FUNCTION
Augment function’s input
by concatenating the
parameter vector
INPUT
Directly augment
parameters of the base
model
PARAMETER
COMPOSITIONS
ADAPTERS

19. FUNCTION
BOTTLENECK ADAPTER
PARAMETER
LoRA
INPUT
PREFIX-TUNING
ADAPTERS

20. ADAPTERS
Houlsby, Neil, et al. "Parameter-efficient transfer learning for NLP." International Conference on Machine Learning. PMLR, 2019.
FUNCTION
BOTTLENECK ADAPTER

21. ADAPTERS
Li, Xiang Lisa, and Percy Liang. "Prefix-Tuning: Optimizing Continuous Prompts for Generation.". 2021.
INPUT
PREFIX-TUNING

22. ADAPTERS
Hu, Edward J., et al. "LoRA: Low-Rank Adaptation of Large Language Models.". 2021.
PARAMETER
LoRA

23. ADAPTERS
FUNCTION
BOTTLENECK ADAPTER
PARAMETER
LoRA
INPUT
PREFIX-TUNING

24. PARAMETER
EFFICIENCY
TRAINING
EFFICIENCY
INFERENCE
EFFICIENCY
PERFORMANCE COMPOSITIONALITY
FUNCTION
COMPOSITION X ✓ X ✓✓ ✓
INPUT
COMPOSITION ✓✓ X X X ✓
PARAMETER
COMPOSITION ✓ X ✓✓ ✓ ✓
ADAPTERS
ADAPTERS

25. METHOD
COMBINATIONS
Bottleneck
ADAPTERS
Mao, Yuning, et al. "UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning.. 2022.

26. ADAPTER FUSION
ADAPTERS
Pfeiffer, Jonas, et al. "AdapterFusion: Non-Destructive Task Composition for Transfer Learning." 2021.

27. ADAPTER FUSION
ADAPTERS
Pfeiffer, Jonas, et al. "AdapterFusion: Non-Destructive Task Composition for Transfer Learning." 2021.

28. HYPERNETWORKS
generate parameters
03

29. MOTIVATION
KNOWLEDGE SHARING
Hypernetwork
TASK-SPECIFIC
Generated
components
HYPERNETWORKS

30. ARCHITECTURE
HYPERNETWORKS

31. ARCHITECTURE
HYPERNETWORKS
Üstün, Ahmet, et al. "Hyper-X: A Unified Hypernetwork for Multi-Task Multilingual Transfer." (2022).

32. INTERFERENCE
Different gradient
directions
SAMPLING
Difficult
UNDER-/OVERFITTING
Problems with
different dataset sizes
LOSS WEIGHTING
Difficult
DRAWBACKS
HYPERNETWORKS

33. “
“
"MULTI-TASK LEARNING: BECAUSE
DOING ONE THING AT A TIME IS SO
LAST YEAR."
— ChatGPT

34. THANKS
Do you have any questions?
tin.ferkovic@doxray.com
linkedin.com/in/tinferkovic/
doxray.com
CREDITS: Slidesgo, Flaticon, Freepik