NLU at Scale: Methods
and Applications of Few-
Shot Learning in
Conversational AI
Mahnoosh Mehrabani, PhD
Senior Principal Scientist

Mahnoosh
Mehrabani
Interactions LLC
Senior Principal Scientist
mahnoosh@interactions.com
/in/mahnoosh-mehrabani

ABSTRACT SUMMARY
Today, the best NLU models rely on deep neural networks (DNN). The billions of parameters powering
these highly accurate state-of-the-art NLU models are trained using gigantic volumes of data that
produce semantic outputs such as intent or sentiment. While these systems are incredibly effective,
they require expensive, and often unsustainable, amounts of supervised data. In contrast, few-shot
learning, which is a new generation of scalable machine learning methods, produces NLU models of
comparable quality without the dependence on large datasets.
In this talk, I will review some of the existing methods for few-shot learning and highlight their potential
applications for rapid NLU model development. I will also discuss drawbacks to current methods and
future research directions. As we grow NLU models and their applications, few-shot learning is an
indelible part of rapidly delivering better experiences to conversational AI end users, and I will discuss
the technical details behind this emerging technology.

Outline
 Problem specification: what is few-shot learning and why is it
important?
 Few-shot learning in Conversational AI
 How to learn with less data? (a high-level review of few-shot
learning methods and algorithms)
 Applications of few-shot learning in customer service industry
 Challenges and future work

Do machines learn like humans?
[1] “Human vs. supervised machine learning: Who learns patterns faster?’’, N Kuhl, et al., Karlsruhe Institute of Technology (KIT), 2020

Comparing ML performance with
humans
Symmetry Rule Numbers Rule

Learning to predict with only a few
examples
 Compared to ML algorithms, humans use prior knowledge to acquire new
skills and can learn more out of a few training samples.
 Few-shot learning attempts to use just a handful of examples to learn.
 Despite some breakthrough results, the drawbacks of DNN models include:
 Relying on lots of labeled training data to learn a large number of
parameters (sometimes billions)
 Require validation sets for hyperparameter tuning
 Overfitting and lacking generalization
 Bias depending on train data
 Inefficiency in computing power, redundancy, and over-parametrization

Conversational AI
Automatic Speech
Recognition (ASR)
Natural Language
Understanding
(NLU)
Dialogue Manager
(DM)
Natural Language
Generation (NLG)
Text to Speech
(TTS)
Speech
Text Semantic
Information
Response
I forgot my password
Sure, I can help you
reset your password

Pre-training and fine-tuning
Pre-training: self-supervised (leverages the large amounts of unlabeled text)
Fine-tuning: supervised (uses labeled data to train the model for a specific task)
Language Model
A quick brown fox jumps
over the [MASK] dog. lazy
Language Model
When will the product
that I ordered yesterday
be delivered?
Order Status
Classifier
Embedding

Scalability in Conversational AI
 Fast creation of dialogue systems with limited samples
 Adding new domains and/or languages quickly
 Data annotation is costly and time-consuming
 Data privacy

Methodology of learning with less
data
 Use prior knowledge and auxiliary information
 Task descriptions
 Generalization to new tasks based on similar tasks with lots of data
 Domain knowledge
 Take advantage of self-supervised and unsupervised learning
 Efficient learning algorithms
 Diversify: ensemble and boosting
 Human in the loop

Few-shot learning algorithms for
NLU
 Large-scale pre-trained models
 Data augmentation
 Pattern-based learning (task description, prompt-based)
 Rule-based methods (keyword spotting, programmatic approach)
 Hybrid approaches: combining human intelligence with ML (active learning, feature
engineering, human adaptive understanding)
 Meta learning, episodic learning, distance-based methods, transfer learning, multi-task
learning, and ensemble learning

Meta learning: learning to learn
 According to Wikipedia, As of 2017 the term had no standard interpretation
 “Meta-learning is the science of systematically observing how different machine
learning approaches perform on a wide range of learning tasks, and learning from
this experience, or meta-data, to learn new tasks.’’ [2]
 “The meta-learning framework treats tasks as training examples assuming that the
training tasks and the new task are from the same distribution of tasks” [3]
 Motivation: when learning new skills, we don’t start from scratch, and we use every skill
that we have previously learned: we learn how to learn across tasks
[2] “Meta-learning: a survey’’, J. Vanschoren , Eindhoven University of Technology, 2018
[3] “Meta-learning for Few-shot Natural Language Processing: A Survey”, W. Yin, Salesforce Research, 2020

Meta-learning example
[4] “Learning from few examples: a
summary if approaches to few-shot
learning’’, A. Parnami and M. Lee,
The University of North Carolina at
Charlotte, 2022

Multi-task learning and transfer
learning
Model1
TN
.
.
.
T2
T1
Model2
ModelN T
.
.
.
Meta-learning Transfer learning
Model1
T
T1
Model
Multi-task learning
TN
.
.
.
T2
T1
T
Model
Model

Case study: customer care IVA
Database
Verticals
Retail
Banking
Telecomm
. . .
New Client
Intent set
Definitions
Examples
. . .
Domain
Experts
Intent Model
Intelligent
Virtual
Assistant
Review
Train Data
Train Data

Active learning
NLU
Input Utterance
ASR
Semantic Information
Confidence
Score
>Threshold
Text
Annotation
No

Challenge#1: joint classification
Data-Driven Model
Few-Shot Model
C1
CN
.
.
.
CN+1
CN+K
.
.
.

Challenge#2: evaluation and
benchmarking
 The progress in NLU has been mainly based on public leaderboards
 Current large-scale models have shown to exceed “human-level” performance on
benchmarks with large amounts of task-specific labeled data.
 Lack of standardized few-shot evaluation benchmarks: different experimental settings in
different papers.
 Some studies overestimate the few-shot ability of language models with wrong
assumptions:
 Train and test data are sampled from the same distribution
 Test (and validation) data are available
[5] “CLUES: Few-Shot Learning Evaluation in Natural Language Understanding”, S. Mukherjee, et al., NeurIPS 2021
[6] “True Few-Shot Learning with Language Models”, E. Perez , et al., NeurIPS 2021

A few-shot NLU benchmark
Performance comparison of humans vs. PLMs on few-shot text classification. FT, PT and ICL stand for classic fine-tuning,
prompt-based fine-tuning and in-context learning, respectively. Model variance is reported across five splits for each setting.
 Task complexity
 Model capacity
 Performance variance

Conclusions
 Few-shot learning aims to use efficient algorithms to learn new tasks based
on a limited number of training samples.
 Few-shot NLU plays an important role in the scalability of Conversational AI
systems by allowing fast expansion to new domains and languages.
 Recently, an increasing number of studies have focused on few-shot NLU,
however, there is still a significant gap between the performance of state-
of-the-art methods and “human-level” performance.

QUESTIONS?

THANK YOU!
mahnoosh@interactions.com
/in/mahnoosh-mehrabani