Natural Language Understanding is a key component of modern conversational applications. Omilia is at the forefront of the global NLP community, providing state-of-the-art conversational AI services to high-profile clients around the world. Python has been an invaluable tool in this endeavor, enabling us to make use of open-source software, for data processing and machine learning applications, such as spaCy, scikit-learn and PyTorch. The utilization of these frameworks, and their contribution to the development of sophisticated deep learning architectures for commercial products and services, is going to be presented in detail by the Omilia NLU R&D team. The presentation will focus on the landscape of conversational AI and NLU from a data science perspective, featuring key computational tasks and use cases, as well as engineering solutions.

1. 27 September, 2019
IΝTENT CLASSIFICATION ON DIALOGS

2. Outline
๏ Introduction
๏ Architecture
๏ Performance
๏ Conclusions

3. INTRODUCTION

4. ๏ Typical interaction components:
‣ Automatic Speech Recognition (ASR), Natural Language Understanding (NLU), Dialogue
Manager (DM), Natural Language Generation (NLG), Text To Speech (TTS)
๏ Challenges: ASR errors, nature of the language, e.g., spontaneous speech, ambiguity, ground-
truth labels etc.
๏ NLU tries to understand user utterances. It breaks them into task specific semantic
representations, e.g., intents
๏ Dialogue Manager updates the dialogue flow based on these representations
!4
Introduction: NLU in Dialogue Systems
ASR NLU
DM
NLGTTS

5. ๏ NLU typically includes both intent classification and slot filling
๏ Intent: The intentions of the user during interaction
๏ Entity (slot): Semantic concepts. An entity modifies an intent
๏ Most common application: chatbot
!5
Introduction: NLU components
How may I help
you?
I want to book a
flight to New York
intent: bookFlight
utterance
Slot (destination): New York

6. ๏ Goal : Intent classification on utterance level using the dialogue context (dialogue
history)
๏ Motivation : Dialogue context contains useful information that is expected to
contribute to a more accurate (intent) decision
๏ NLU model can be :
‣ Rule-based: match manually design pattern sequences
‣ Statistical: Feature extraction (e.g., ngrams, POS tags) + Machine learning
model (e.g., Naive Bayes, Decision trees)
‣ Neural Networks: End-to-end approach without manual feature extraction
!6
Introduction: Goal & Approach
Dialogs
NLU
modelUtterances
Intents
Dialogue History

7. ARCHITECTURE

8. !8
ARCHITECTURE: Problem definition
๏ Language: US- English
๏ Domain: Banking, American Bank
๏ Tasks: Intent classification & slot filling
๏ NLU model: Neural networks: Deep Recurrent Neural Network (RNN) architecture
‣ Features: Word Embeddings
‣ RNNs: Long-short Term Memory (LSTMs), Gated Recurrent unit (GRUs)
๏ Dialogues: ~ 40K
๏ Intents: 67

9. ๏ Challenging task due to supporting natural interaction between humans and
computers
๏ Users pick up their phone, ask something and NLU model detects the intent based
on the dialog
๏ Intent classes:
‣ Can overlap
‣ Can be ambiguous
‣ Can have very close meaning between each other
‣ Are not equally frequent
!9
Intent classification challenge

10. Thank you for calling the bank.
How can i help you ?
Please check my charges on my card
Ok. Please tell me the last 4 digits of your card.
****
Got it thanks . Are you interested in the transactions posted on your
last statement or the most recent ones?
*** Would you like me to repeat or help you with something else?
Tell me the automatic payments
*** Would you like me to repeat or help you with something else?
more information
most recent ones
๏ Dialogue context contains
information that can contribute to
a more accurate prediction
๏ In the example we observe that
the utterance “more information”
is too generic without the dialog
context
๏ However, providing the model
with the dialogue context we
help it to understand that “more
information” refers to the
automatic payments
๏ Related intents are: Information,
Payment-Automatic_Information
Importance of dialogue context

11. ! /11
ARCHITECTURE: Intent Classification - One utterance
When was my last payment?
WORD EMBEDDINGS LAYER
RNN LAYERS
SOFTMAX LAYERS
ENTITY CLASSIFICATION INTENT CLASSIFICATION

12. !12
ARCHITECTURE: Intent Classification - Dialog Encoding
When was my last payment?
WORD EMBEDDINGS LAYER
RNN LAYERS
SOFTMAX LAYERS
ENTITY CLASSIFICATION INTENT CLASSIFICATION
DIALOG
ENCODING LAST RNN LAYER

13. !13
ARCHITECTURE: Dialog Encoder
Thank you for calling the bank. How can i help you ?
Balance and available credit
Very well. To check your available credit, i will first need you
to say or enter the last four digits of your ssn.
****
Got it, thanks. Let's see. Your available credit is ****
Should i repeat or is there something else i can help
you with ?
…
Dialog
History
When was my last payment?
Current
Utterance
W
O
R
D
E
M
B
E
D
D
I
N
G
S
L
A
Y
E
R
RNN
RNN
History
representation
Current ut.
representation
+
Dialogue
Context
RNN
DIALOG
ENCODING

14. PERFORMANCE

15. !15
PERFORMANCE: Training requirements
๏ Training ..
‣ for 40 epochs
‣ on 1 GPU
‣ for ~50min per epoch
๏ Tools & libraries

16. 92.6 %92
93
94
95
๏ Intent Classification Accuracy / Error, F-Score
๏ Dialogue encoder achieves ~ 92.6 %
classification accuracy (~7.4 error)
๏ With extra features used at Omilia reached
96.2 classification accuracy (~ 3.8 error )
PERFORMANCE: Evaluation
Payment
Loans
Bank Accounts
93
91
90
89
88
87
86
85
92
94
95
96
97
98
99
No - Dialogue history
Scikit-learn
No - Dialogue history
Transformers (BERT)
Dialogue encoder

17. CONCLUSIONS

18. !18
Conclusions
๏ NLU is a very challenging NLP subfield
๏ Intent classification is a crucial task for successful communication between humans
and machines
๏ Advanced NLU models like Neural networks are required in order to handle the
challenging
๏ Dialogue context helps