SlideShare a Scribd company logo
1 of 16
Download to read offline
xyang45@illinois.edu y.v.chen@ieee.org
End-to-End Joint Learning of
Natural Language Understanding
and Dialogue Manager
ICASSP 2017
March 9th
XUESONG YANG YUN-NUNG (VIVIAN) CHEN DILEK HAKKANI-TÜR
HLT-L2: Spoken Language Understanding I
PAUL CROOK
XIUJUN LI
JIANFENG GAO
LI DENG
What is a Dialogue System?
2
Apple Siri (2011)
Facebook M & Bot (2015) Google Home (2016)
Microsoft Cortana (2014) Amazon Alexa/Echo (2014)
Category:
• Chit Chat
• Task-Oriented
[1 ] Zue, Victor W., and James R. Glass. “Conversational interfaces: Advances and challenges.” Proceedings of the IEEE 88.8 (2000): 1166-1180.
A dialogue system is a computer agent that interacts with human via natural languages.
Pipelined Task-Oriented Dialogue System
Automatic Speech
Recognition
Natural Language Understanding (NLU)
• User Intent Detection
• Slot Filling
Dialogue Management (DM)
• Dialogue State Tracking (DST)
• Dialogue Policy Decision
Natural Language
Generation (NLG)
Hypothesis
Any action movies recommended
this weekend?
Semantic Frame
request_movie
(genre=action, date=this weekend)
System Action/Policy
request_location
Text response
Which theater do you prefer?
Text Input
Any action movies recommended this weekend?
Speech Signal
Motivation: The pipelined system (NLU  DM) results in error propagation issues.
3
Text-To-Speech
Proposed Approach
 End-to-end model
 Mitigate the effects of noisy output from NLU
 Refine NLU by supervised signals from DM
 Multi-task jointly learning
 NLU - User intent classification
 NLU - User slot tagging
 DM - System action prediction
 Contextual understanding
 Access to the user history
 Monitor user behavior states over turns
4
Human-Human Dialogue Interaction
Hi, how may I help you?
Do you want to have a backpack type of hotel?
So you don’t mind if it is not roomy, right?
Okay. These hotels are available for you: …
Thanks, goodbye.
Are there any cheap rate hotels to put my bags?
Yes. Just gonna leave our things there and stay out the
whole day.
Yes.
Ok, thank you, bye!
Idea: predicting the next system action given the current user
utterance together with the aggregated observations
Guide
Agent
Tourist
User
5
Natural Language Understanding (NLU)
6
Utterance: BOS % um how much is a taxi cab there ? EOS
Slot Tags: O O O B-det_PRICE I-det_PRICE O O B-trsp_TYPE I-trsp_TYPE B-area_CITY O O
User Intents: QST_HOW_MUCH; QST_INFO
System Actions: RES_EXPLAIN; RES_INFO; FOL_EXPLAIN; FOL_HOW_MUCH; FOL_INFO
Task 1: Slot Tagging Task 2: Multi-Label User Intent Prediction
Sharedweights
Task 1+2: Natural Language Understanding
NLU+DM 1: Pipelined BLSTMs
7
single user turn current user turn w/ contextual history
Utterance: BOS % um how much is a taxi cab there ? EOS
Slot Tags: O O O B-det_PRICE I-det_PRICE O O B-trsp_TYPE I-trsp_TYPE B-area_CITY O O
User Intents: QST_HOW_MUCH; QST_INFO
System Actions: RES_EXPLAIN; RES_INFO; FOL_EXPLAIN; FOL_HOW_MUCH; FOL_INFO
Task 3: Multi-Label System Action Prediction
NLU+DM 2: End-to-End Model (JointModel)
8
Utterance: BOS % um how much is a taxi cab there ? EOS
Slot Tags: O O O B-det_PRICE I-det_PRICE O O B-trsp_TYPE I-trsp_TYPE B-area_CITY O O
User Intents: QST_HOW_MUCH; QST_INFO
System Actions: RES_EXPLAIN; RES_INFO; FOL_EXPLAIN; FOL_HOW_MUCH; FOL_INFO
Task 1+2+3: End-to-End Joint NLU+DM
supervised learning with three tasks
DM output signal refines NLU
for better robustness
DM
Data
 Dialogue State Tracking Challenge 4
 Human-human dialogues: 21-hour dialogue sessions on touristic
information collected via Skype between tour guides and tourists
User Intent = Speech Act + Attributes
System Action = Speech Act + Attributes
Domains
Accommodation
Attraction
Food
Shopping
Transportation
Speech Act
QST (QUESTION)
RES (RESPONSE)
INI (INITIATIVE)
FOL (FOLLOW)
Speech Act Attributes
ACK CLOSING COMMIT THANK
CANCEL CONFIRM ENOUGH WHAT
EXPLAIN HOW_MUCH HOW_TO WHEN
INFO NEGATIVE OPENING WHERE
POSITIVE PERFERENCE RECOMMEND WHO
Train Dev Test
#utter 5,648 1,939 3,178
#intent 68 54 58
9
DM Result – System Action Prediction (SAP)
10
 Metric: frame-level accuracy (FrmAcc)
Model FrmAcc
Baseline (CRF+SVMs) 7.7
Pipeline-BLSTM 12.0
JointModel 22.8
Pipeline-BLSTM and JointModel outperform the baseline
JointModel improves Pipeline-BLSTM about 10% accuracy, indicating the importance of
mitigating downside of pipeline
Human-human conversations are complicated, so predicting system actions for DM is difficult
DM Result – System Action Prediction (SAP)
11
 Metric: frame-level accuracy (FrmAcc)
Model FrmAcc
Baseline (CRF+SVMs) 7.7
Pipeline-BLSTM 12.0
JointModel 22.8
Oracle-SAP (SVM) 7.7
Oracel-SAP (BLSTM) 19.7
Oracle models show the upper-bound of the SAP performance,
since it transfers the errors from NLU to SAP
Contextual user turns make significant contribution to DM performance
JointModel achieves the best DM performance (FrmAcc) with richer latent representations
Fully correct
NLU output
NLU Result – Slot Filling & Intent Prediction
12
 Metrics: frame-level accuracy (FrmAcc)
Models Slot Intent NLU
NLU-Baseline (CRF+SVM) 77.3 37.2 33.1
NLU-Pipeline-BLSTM 76.8 40.0 36.4
NLU-JointModel 76.5 42.2 37.4
CRF+SVMs baseline maintains strong NLU performance with 33.1%
Pipeline-BLSTM and JointModel outperformed the baseline
DM signal (system action prediction) helps more on user intent prediction
than slot filling, and NLU is significantly improved
Extra supervised DM signal helps refine the NLU by back-propagating the associated errors
Conclusion
13
 First propose an end-to-end deep hierarchical model for joint
NLU and DM with limited contextual dialogue memory
 Leverage multi-task learning using three supervised signals
 NLU: User intent classification
 NLU: Slot tagging
 DM: System action prediction
 Outperform the state-of-the-art pipelined NLU and DM models
 Better DM due to the contextual dialogue memory
 Robust NLU fine-tuned by supervised signal from DM
Code Available at
https://github.com/XuesongYang/end2end_dialog
Thanks for Your Attention! 14
Appendix - Baseline Model
NLU: CRF for slot tagging, and
One-Vs-All SVMs for intent classification
SAP: One-Vs-All SVMs
[1] Raymond, Christian, and Giuseppe Riccardi. “Generative and discriminative algorithms for spoken language
understanding." In INTERSPEECH, pp. 1605-1608. 2007.
Predicting system actions at the next
turn as responses to the current user
behaviors by pipelining NLU and SAP
together
15
Appendix: Configuration
 Optimizer: a mini-batch stochastic gradient descent method
Adam
 Contextual history: five user turns
 Dimension of word embedding: 512
 Dropout ratio: 0.5
 No early stopping, but use 300 training epochs
 Best models for three tasks are selected individually under
different metrics
 Token-level micro-average F1 score is used for slot filling
 frame-level accuracy (it counts only when the whole frame parse is
correct) is used for both user intent prediction and system action
prediction
 the decision thresholds are tuned on dev set
16

More Related Content

What's hot

Unsupervised Learning and Modeling of Knowledge and Intent for Spoken Dialogu...
Unsupervised Learning and Modeling of Knowledge and Intent for Spoken Dialogu...Unsupervised Learning and Modeling of Knowledge and Intent for Spoken Dialogu...
Unsupervised Learning and Modeling of Knowledge and Intent for Spoken Dialogu...Yun-Nung (Vivian) Chen
 
Leveraging Behavioral Patterns of Mobile Applications for Personalized Spoken...
Leveraging Behavioral Patterns of Mobile Applications for Personalized Spoken...Leveraging Behavioral Patterns of Mobile Applications for Personalized Spoken...
Leveraging Behavioral Patterns of Mobile Applications for Personalized Spoken...Yun-Nung (Vivian) Chen
 
"Sorry, I didn't get that!" - Statistical Learning from Dialogues for Intelli...
"Sorry, I didn't get that!" - Statistical Learning from Dialogues for Intelli..."Sorry, I didn't get that!" - Statistical Learning from Dialogues for Intelli...
"Sorry, I didn't get that!" - Statistical Learning from Dialogues for Intelli...Yun-Nung (Vivian) Chen
 
Detecting Actionable Items in Meetings by Convolutional Deep Structured Seman...
Detecting Actionable Items in Meetings by Convolutional Deep Structured Seman...Detecting Actionable Items in Meetings by Convolutional Deep Structured Seman...
Detecting Actionable Items in Meetings by Convolutional Deep Structured Seman...Yun-Nung (Vivian) Chen
 
Statistical Learning from Dialogues for Intelligent Assistants
Statistical Learning from Dialogues for Intelligent AssistantsStatistical Learning from Dialogues for Intelligent Assistants
Statistical Learning from Dialogues for Intelligent AssistantsYun-Nung (Vivian) Chen
 
An Intelligent Assistant for High-Level Task Understanding
An Intelligent Assistant for High-Level Task UnderstandingAn Intelligent Assistant for High-Level Task Understanding
An Intelligent Assistant for High-Level Task UnderstandingYun-Nung (Vivian) Chen
 
How the Context Matters Language and Interaction in Dialogues
How the Context Matters Language and Interaction in DialoguesHow the Context Matters Language and Interaction in Dialogues
How the Context Matters Language and Interaction in DialoguesYun-Nung (Vivian) Chen
 
One Day for Bot 一天搞懂聊天機器人
One Day for Bot 一天搞懂聊天機器人One Day for Bot 一天搞懂聊天機器人
One Day for Bot 一天搞懂聊天機器人Yun-Nung (Vivian) Chen
 
Unsupervised Learning and Modeling of Knowledge and Intent for Spoken Dialogu...
Unsupervised Learning and Modeling of Knowledge and Intent for Spoken Dialogu...Unsupervised Learning and Modeling of Knowledge and Intent for Spoken Dialogu...
Unsupervised Learning and Modeling of Knowledge and Intent for Spoken Dialogu...Yun-Nung (Vivian) Chen
 
2017 Tutorial - Deep Learning for Dialogue Systems
2017 Tutorial - Deep Learning for Dialogue Systems2017 Tutorial - Deep Learning for Dialogue Systems
2017 Tutorial - Deep Learning for Dialogue SystemsMLReview
 
Deep Dialog System Review
Deep Dialog System ReviewDeep Dialog System Review
Deep Dialog System ReviewNguyen Quang
 
Multiskill Conversational AI
Multiskill Conversational AIMultiskill Conversational AI
Multiskill Conversational AIDaniel Kornev
 
Dialogue systems and personal assistants
Dialogue systems and personal assistantsDialogue systems and personal assistants
Dialogue systems and personal assistantsNatalia Konstantinova
 
God Mode for designing scenario-driven skills for DeepPavlov Dream
God Mode for designing scenario-driven skills for DeepPavlov DreamGod Mode for designing scenario-driven skills for DeepPavlov Dream
God Mode for designing scenario-driven skills for DeepPavlov DreamDaniel Kornev
 
Managing Dialog Strategy in Multiskill AI Assistant with Discourse Management
Managing Dialog Strategy in Multiskill AI Assistant with Discourse ManagementManaging Dialog Strategy in Multiskill AI Assistant with Discourse Management
Managing Dialog Strategy in Multiskill AI Assistant with Discourse ManagementDaniel Kornev
 
Managing Dialog Strategy In Multiskill AI Assistant.pdf
Managing Dialog Strategy In Multiskill AI Assistant.pdfManaging Dialog Strategy In Multiskill AI Assistant.pdf
Managing Dialog Strategy In Multiskill AI Assistant.pdfDaniel Kornev
 

What's hot (17)

Unsupervised Learning and Modeling of Knowledge and Intent for Spoken Dialogu...
Unsupervised Learning and Modeling of Knowledge and Intent for Spoken Dialogu...Unsupervised Learning and Modeling of Knowledge and Intent for Spoken Dialogu...
Unsupervised Learning and Modeling of Knowledge and Intent for Spoken Dialogu...
 
Leveraging Behavioral Patterns of Mobile Applications for Personalized Spoken...
Leveraging Behavioral Patterns of Mobile Applications for Personalized Spoken...Leveraging Behavioral Patterns of Mobile Applications for Personalized Spoken...
Leveraging Behavioral Patterns of Mobile Applications for Personalized Spoken...
 
"Sorry, I didn't get that!" - Statistical Learning from Dialogues for Intelli...
"Sorry, I didn't get that!" - Statistical Learning from Dialogues for Intelli..."Sorry, I didn't get that!" - Statistical Learning from Dialogues for Intelli...
"Sorry, I didn't get that!" - Statistical Learning from Dialogues for Intelli...
 
Detecting Actionable Items in Meetings by Convolutional Deep Structured Seman...
Detecting Actionable Items in Meetings by Convolutional Deep Structured Seman...Detecting Actionable Items in Meetings by Convolutional Deep Structured Seman...
Detecting Actionable Items in Meetings by Convolutional Deep Structured Seman...
 
Statistical Learning from Dialogues for Intelligent Assistants
Statistical Learning from Dialogues for Intelligent AssistantsStatistical Learning from Dialogues for Intelligent Assistants
Statistical Learning from Dialogues for Intelligent Assistants
 
An Intelligent Assistant for High-Level Task Understanding
An Intelligent Assistant for High-Level Task UnderstandingAn Intelligent Assistant for High-Level Task Understanding
An Intelligent Assistant for High-Level Task Understanding
 
How the Context Matters Language and Interaction in Dialogues
How the Context Matters Language and Interaction in DialoguesHow the Context Matters Language and Interaction in Dialogues
How the Context Matters Language and Interaction in Dialogues
 
One Day for Bot 一天搞懂聊天機器人
One Day for Bot 一天搞懂聊天機器人One Day for Bot 一天搞懂聊天機器人
One Day for Bot 一天搞懂聊天機器人
 
Unsupervised Learning and Modeling of Knowledge and Intent for Spoken Dialogu...
Unsupervised Learning and Modeling of Knowledge and Intent for Spoken Dialogu...Unsupervised Learning and Modeling of Knowledge and Intent for Spoken Dialogu...
Unsupervised Learning and Modeling of Knowledge and Intent for Spoken Dialogu...
 
2017 Tutorial - Deep Learning for Dialogue Systems
2017 Tutorial - Deep Learning for Dialogue Systems2017 Tutorial - Deep Learning for Dialogue Systems
2017 Tutorial - Deep Learning for Dialogue Systems
 
Deep Dialog System Review
Deep Dialog System ReviewDeep Dialog System Review
Deep Dialog System Review
 
Multiskill Conversational AI
Multiskill Conversational AIMultiskill Conversational AI
Multiskill Conversational AI
 
Dialogue systems and personal assistants
Dialogue systems and personal assistantsDialogue systems and personal assistants
Dialogue systems and personal assistants
 
God Mode for designing scenario-driven skills for DeepPavlov Dream
God Mode for designing scenario-driven skills for DeepPavlov DreamGod Mode for designing scenario-driven skills for DeepPavlov Dream
God Mode for designing scenario-driven skills for DeepPavlov Dream
 
Managing Dialog Strategy in Multiskill AI Assistant with Discourse Management
Managing Dialog Strategy in Multiskill AI Assistant with Discourse ManagementManaging Dialog Strategy in Multiskill AI Assistant with Discourse Management
Managing Dialog Strategy in Multiskill AI Assistant with Discourse Management
 
Managing Dialog Strategy In Multiskill AI Assistant.pdf
Managing Dialog Strategy In Multiskill AI Assistant.pdfManaging Dialog Strategy In Multiskill AI Assistant.pdf
Managing Dialog Strategy In Multiskill AI Assistant.pdf
 
Blenderbot
BlenderbotBlenderbot
Blenderbot
 

Viewers also liked

Colonial economy (2)
Colonial economy (2)Colonial economy (2)
Colonial economy (2)Dody Rangai
 
Presentación
PresentaciónPresentación
Presentacióngrcmanu20
 
Encontro: Práticas no Ensino Profissional na área das TI,
Encontro: Práticas no Ensino Profissional na área das TI,Encontro: Práticas no Ensino Profissional na área das TI,
Encontro: Práticas no Ensino Profissional na área das TI,Marco Neves
 
Automatic Key Term Extraction and Summarization from Spoken Course Lectures
Automatic Key Term Extraction and Summarization from Spoken Course LecturesAutomatic Key Term Extraction and Summarization from Spoken Course Lectures
Automatic Key Term Extraction and Summarization from Spoken Course LecturesYun-Nung (Vivian) Chen
 
Automatic Key Term Extraction from Spoken Course Lectures
Automatic Key Term Extraction from Spoken Course LecturesAutomatic Key Term Extraction from Spoken Course Lectures
Automatic Key Term Extraction from Spoken Course LecturesYun-Nung (Vivian) Chen
 
VIPRE Operations Brief - 2017
VIPRE Operations Brief - 2017 VIPRE Operations Brief - 2017
VIPRE Operations Brief - 2017 Larry Rice
 
3Com 3C9I-PCI-25
3Com 3C9I-PCI-253Com 3C9I-PCI-25
3Com 3C9I-PCI-25savomir
 
strategic and change management organization
strategic and change management organizationstrategic and change management organization
strategic and change management organizationsainath balasani
 

Viewers also liked (16)

Colonial economy (2)
Colonial economy (2)Colonial economy (2)
Colonial economy (2)
 
WIP Egyptica
WIP EgypticaWIP Egyptica
WIP Egyptica
 
Ganglios basales
Ganglios basalesGanglios basales
Ganglios basales
 
Acoplamiento magnético
Acoplamiento magnéticoAcoplamiento magnético
Acoplamiento magnético
 
Presentación
PresentaciónPresentación
Presentación
 
Encontro: Práticas no Ensino Profissional na área das TI,
Encontro: Práticas no Ensino Profissional na área das TI,Encontro: Práticas no Ensino Profissional na área das TI,
Encontro: Práticas no Ensino Profissional na área das TI,
 
2 nd export & import
2 nd export & import2 nd export & import
2 nd export & import
 
Biology of nerve injury and repair
Biology of nerve injury and repairBiology of nerve injury and repair
Biology of nerve injury and repair
 
Automatic Key Term Extraction and Summarization from Spoken Course Lectures
Automatic Key Term Extraction and Summarization from Spoken Course LecturesAutomatic Key Term Extraction and Summarization from Spoken Course Lectures
Automatic Key Term Extraction and Summarization from Spoken Course Lectures
 
Automatic Key Term Extraction from Spoken Course Lectures
Automatic Key Term Extraction from Spoken Course LecturesAutomatic Key Term Extraction from Spoken Course Lectures
Automatic Key Term Extraction from Spoken Course Lectures
 
Circuitos trifasicos
Circuitos trifasicosCircuitos trifasicos
Circuitos trifasicos
 
Project solar
Project solar Project solar
Project solar
 
VIPRE Operations Brief - 2017
VIPRE Operations Brief - 2017 VIPRE Operations Brief - 2017
VIPRE Operations Brief - 2017
 
3Com 3C9I-PCI-25
3Com 3C9I-PCI-253Com 3C9I-PCI-25
3Com 3C9I-PCI-25
 
Practica innovatic
Practica innovaticPractica innovatic
Practica innovatic
 
strategic and change management organization
strategic and change management organizationstrategic and change management organization
strategic and change management organization
 

Similar to End-to-End Joint Learning of Natural Language Understanding and Dialogue Manager

Feature Extraction and Analysis of Natural Language Processing for Deep Learn...
Feature Extraction and Analysis of Natural Language Processing for Deep Learn...Feature Extraction and Analysis of Natural Language Processing for Deep Learn...
Feature Extraction and Analysis of Natural Language Processing for Deep Learn...Sharmila Sathish
 
Deep convolutional neural networks-based features for Indonesian large vocabu...
Deep convolutional neural networks-based features for Indonesian large vocabu...Deep convolutional neural networks-based features for Indonesian large vocabu...
Deep convolutional neural networks-based features for Indonesian large vocabu...IAESIJAI
 
End-to-end Speech Recognition with Recurrent Neural Networks (D3L6 Deep Learn...
End-to-end Speech Recognition with Recurrent Neural Networks (D3L6 Deep Learn...End-to-end Speech Recognition with Recurrent Neural Networks (D3L6 Deep Learn...
End-to-end Speech Recognition with Recurrent Neural Networks (D3L6 Deep Learn...Universitat Politècnica de Catalunya
 
Li Deng at AI Frontiers: Three Generations of Spoken Dialogue Systems (Bots)
Li Deng at AI Frontiers: Three Generations of Spoken Dialogue Systems (Bots)Li Deng at AI Frontiers: Three Generations of Spoken Dialogue Systems (Bots)
Li Deng at AI Frontiers: Three Generations of Spoken Dialogue Systems (Bots)AI Frontiers
 
Natural Language Processing - Research and Application Trends
Natural Language Processing - Research and Application TrendsNatural Language Processing - Research and Application Trends
Natural Language Processing - Research and Application TrendsShreyas Suresh Rao
 
HCI BASED APPLICATION FOR PLAYING COMPUTER GAMES | J4RV4I1014
HCI BASED APPLICATION FOR PLAYING COMPUTER GAMES | J4RV4I1014HCI BASED APPLICATION FOR PLAYING COMPUTER GAMES | J4RV4I1014
HCI BASED APPLICATION FOR PLAYING COMPUTER GAMES | J4RV4I1014Journal For Research
 
BIDIRECTIONAL LONG SHORT-TERM MEMORY (BILSTM)WITH CONDITIONAL RANDOM FIELDS (...
BIDIRECTIONAL LONG SHORT-TERM MEMORY (BILSTM)WITH CONDITIONAL RANDOM FIELDS (...BIDIRECTIONAL LONG SHORT-TERM MEMORY (BILSTM)WITH CONDITIONAL RANDOM FIELDS (...
BIDIRECTIONAL LONG SHORT-TERM MEMORY (BILSTM)WITH CONDITIONAL RANDOM FIELDS (...kevig
 
BIDIRECTIONAL LONG SHORT-TERM MEMORY (BILSTM)WITH CONDITIONAL RANDOM FIELDS (...
BIDIRECTIONAL LONG SHORT-TERM MEMORY (BILSTM)WITH CONDITIONAL RANDOM FIELDS (...BIDIRECTIONAL LONG SHORT-TERM MEMORY (BILSTM)WITH CONDITIONAL RANDOM FIELDS (...
BIDIRECTIONAL LONG SHORT-TERM MEMORY (BILSTM)WITH CONDITIONAL RANDOM FIELDS (...ijnlc
 
A technical paper presentation on Evaluation of Deep Learning techniques in S...
A technical paper presentation on Evaluation of Deep Learning techniques in S...A technical paper presentation on Evaluation of Deep Learning techniques in S...
A technical paper presentation on Evaluation of Deep Learning techniques in S...VarshaR19
 
E0502 01 2327
E0502 01 2327E0502 01 2327
E0502 01 2327IJMER
 
70 C o m m u n i C at i o n s o f t h E a C m j u.docx
70    C o m m u n i C at i o n s  o f  t h E  a C m       j u.docx70    C o m m u n i C at i o n s  o f  t h E  a C m       j u.docx
70 C o m m u n i C at i o n s o f t h E a C m j u.docxevonnehoggarth79783
 
A survey on Enhancements in Speech Recognition
A survey on Enhancements in Speech RecognitionA survey on Enhancements in Speech Recognition
A survey on Enhancements in Speech RecognitionIRJET Journal
 
Speech enhanced gesture based navigation for Google Maps
Speech enhanced gesture based navigation for Google MapsSpeech enhanced gesture based navigation for Google Maps
Speech enhanced gesture based navigation for Google MapsHimanshu Bansal
 
Long Short Term Memory (Neural Networks)
Long Short Term Memory (Neural Networks)Long Short Term Memory (Neural Networks)
Long Short Term Memory (Neural Networks)Olusola Amusan
 
90seconds Presentation
90seconds Presentation90seconds Presentation
90seconds Presentationpbihler
 
DHWANI- THE VOICE OF DEAF AND MUTE
DHWANI- THE VOICE OF DEAF AND MUTEDHWANI- THE VOICE OF DEAF AND MUTE
DHWANI- THE VOICE OF DEAF AND MUTEIRJET Journal
 
DHWANI- THE VOICE OF DEAF AND MUTE
DHWANI- THE VOICE OF DEAF AND MUTEDHWANI- THE VOICE OF DEAF AND MUTE
DHWANI- THE VOICE OF DEAF AND MUTEIRJET Journal
 

Similar to End-to-End Joint Learning of Natural Language Understanding and Dialogue Manager (20)

Feature Extraction and Analysis of Natural Language Processing for Deep Learn...
Feature Extraction and Analysis of Natural Language Processing for Deep Learn...Feature Extraction and Analysis of Natural Language Processing for Deep Learn...
Feature Extraction and Analysis of Natural Language Processing for Deep Learn...
 
Deep convolutional neural networks-based features for Indonesian large vocabu...
Deep convolutional neural networks-based features for Indonesian large vocabu...Deep convolutional neural networks-based features for Indonesian large vocabu...
Deep convolutional neural networks-based features for Indonesian large vocabu...
 
End-to-end Speech Recognition with Recurrent Neural Networks (D3L6 Deep Learn...
End-to-end Speech Recognition with Recurrent Neural Networks (D3L6 Deep Learn...End-to-end Speech Recognition with Recurrent Neural Networks (D3L6 Deep Learn...
End-to-end Speech Recognition with Recurrent Neural Networks (D3L6 Deep Learn...
 
Li Deng at AI Frontiers: Three Generations of Spoken Dialogue Systems (Bots)
Li Deng at AI Frontiers: Three Generations of Spoken Dialogue Systems (Bots)Li Deng at AI Frontiers: Three Generations of Spoken Dialogue Systems (Bots)
Li Deng at AI Frontiers: Three Generations of Spoken Dialogue Systems (Bots)
 
Natural Language Processing - Research and Application Trends
Natural Language Processing - Research and Application TrendsNatural Language Processing - Research and Application Trends
Natural Language Processing - Research and Application Trends
 
Priyank cv
Priyank cvPriyank cv
Priyank cv
 
Priyank cv
Priyank cvPriyank cv
Priyank cv
 
HCI BASED APPLICATION FOR PLAYING COMPUTER GAMES | J4RV4I1014
HCI BASED APPLICATION FOR PLAYING COMPUTER GAMES | J4RV4I1014HCI BASED APPLICATION FOR PLAYING COMPUTER GAMES | J4RV4I1014
HCI BASED APPLICATION FOR PLAYING COMPUTER GAMES | J4RV4I1014
 
BIDIRECTIONAL LONG SHORT-TERM MEMORY (BILSTM)WITH CONDITIONAL RANDOM FIELDS (...
BIDIRECTIONAL LONG SHORT-TERM MEMORY (BILSTM)WITH CONDITIONAL RANDOM FIELDS (...BIDIRECTIONAL LONG SHORT-TERM MEMORY (BILSTM)WITH CONDITIONAL RANDOM FIELDS (...
BIDIRECTIONAL LONG SHORT-TERM MEMORY (BILSTM)WITH CONDITIONAL RANDOM FIELDS (...
 
BIDIRECTIONAL LONG SHORT-TERM MEMORY (BILSTM)WITH CONDITIONAL RANDOM FIELDS (...
BIDIRECTIONAL LONG SHORT-TERM MEMORY (BILSTM)WITH CONDITIONAL RANDOM FIELDS (...BIDIRECTIONAL LONG SHORT-TERM MEMORY (BILSTM)WITH CONDITIONAL RANDOM FIELDS (...
BIDIRECTIONAL LONG SHORT-TERM MEMORY (BILSTM)WITH CONDITIONAL RANDOM FIELDS (...
 
A technical paper presentation on Evaluation of Deep Learning techniques in S...
A technical paper presentation on Evaluation of Deep Learning techniques in S...A technical paper presentation on Evaluation of Deep Learning techniques in S...
A technical paper presentation on Evaluation of Deep Learning techniques in S...
 
E0502 01 2327
E0502 01 2327E0502 01 2327
E0502 01 2327
 
70 C o m m u n i C at i o n s o f t h E a C m j u.docx
70    C o m m u n i C at i o n s  o f  t h E  a C m       j u.docx70    C o m m u n i C at i o n s  o f  t h E  a C m       j u.docx
70 C o m m u n i C at i o n s o f t h E a C m j u.docx
 
A survey on Enhancements in Speech Recognition
A survey on Enhancements in Speech RecognitionA survey on Enhancements in Speech Recognition
A survey on Enhancements in Speech Recognition
 
Speech enhanced gesture based navigation for Google Maps
Speech enhanced gesture based navigation for Google MapsSpeech enhanced gesture based navigation for Google Maps
Speech enhanced gesture based navigation for Google Maps
 
Long Short Term Memory (Neural Networks)
Long Short Term Memory (Neural Networks)Long Short Term Memory (Neural Networks)
Long Short Term Memory (Neural Networks)
 
90seconds Presentation
90seconds Presentation90seconds Presentation
90seconds Presentation
 
ijeter35852020.pdf
ijeter35852020.pdfijeter35852020.pdf
ijeter35852020.pdf
 
DHWANI- THE VOICE OF DEAF AND MUTE
DHWANI- THE VOICE OF DEAF AND MUTEDHWANI- THE VOICE OF DEAF AND MUTE
DHWANI- THE VOICE OF DEAF AND MUTE
 
DHWANI- THE VOICE OF DEAF AND MUTE
DHWANI- THE VOICE OF DEAF AND MUTEDHWANI- THE VOICE OF DEAF AND MUTE
DHWANI- THE VOICE OF DEAF AND MUTE
 

Recently uploaded

TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024Lonnie McRorey
 
Scale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterScale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterMydbops
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity PlanDatabarracks
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch TuesdayIvanti
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
Decarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityDecarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityIES VE
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxLoriGlavin3
 
Data governance with Unity Catalog Presentation
Data governance with Unity Catalog PresentationData governance with Unity Catalog Presentation
Data governance with Unity Catalog PresentationKnoldus Inc.
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsRavi Sanghani
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024TopCSSGallery
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxLoriGlavin3
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxLoriGlavin3
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesThousandEyes
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality AssuranceInflectra
 
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...itnewsafrica
 
Zeshan Sattar- Assessing the skill requirements and industry expectations for...
Zeshan Sattar- Assessing the skill requirements and industry expectations for...Zeshan Sattar- Assessing the skill requirements and industry expectations for...
Zeshan Sattar- Assessing the skill requirements and industry expectations for...itnewsafrica
 
A Framework for Development in the AI Age
A Framework for Development in the AI AgeA Framework for Development in the AI Age
A Framework for Development in the AI AgeCprime
 
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS:  6 Ways to Automate Your Data IntegrationBridging Between CAD & GIS:  6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integrationmarketing932765
 
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Varsha Sewlal- Cyber Attacks on Critical Critical InfrastructureVarsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructureitnewsafrica
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsPixlogix Infotech
 

Recently uploaded (20)

TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024
 
Scale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterScale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL Router
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity Plan
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch Tuesday
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
Decarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityDecarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a reality
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
 
Data governance with Unity Catalog Presentation
Data governance with Unity Catalog PresentationData governance with Unity Catalog Presentation
Data governance with Unity Catalog Presentation
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and Insights
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
 
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
 
Zeshan Sattar- Assessing the skill requirements and industry expectations for...
Zeshan Sattar- Assessing the skill requirements and industry expectations for...Zeshan Sattar- Assessing the skill requirements and industry expectations for...
Zeshan Sattar- Assessing the skill requirements and industry expectations for...
 
A Framework for Development in the AI Age
A Framework for Development in the AI AgeA Framework for Development in the AI Age
A Framework for Development in the AI Age
 
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS:  6 Ways to Automate Your Data IntegrationBridging Between CAD & GIS:  6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
 
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Varsha Sewlal- Cyber Attacks on Critical Critical InfrastructureVarsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and Cons
 

End-to-End Joint Learning of Natural Language Understanding and Dialogue Manager

  • 1. xyang45@illinois.edu y.v.chen@ieee.org End-to-End Joint Learning of Natural Language Understanding and Dialogue Manager ICASSP 2017 March 9th XUESONG YANG YUN-NUNG (VIVIAN) CHEN DILEK HAKKANI-TÜR HLT-L2: Spoken Language Understanding I PAUL CROOK XIUJUN LI JIANFENG GAO LI DENG
  • 2. What is a Dialogue System? 2 Apple Siri (2011) Facebook M & Bot (2015) Google Home (2016) Microsoft Cortana (2014) Amazon Alexa/Echo (2014) Category: • Chit Chat • Task-Oriented [1 ] Zue, Victor W., and James R. Glass. “Conversational interfaces: Advances and challenges.” Proceedings of the IEEE 88.8 (2000): 1166-1180. A dialogue system is a computer agent that interacts with human via natural languages.
  • 3. Pipelined Task-Oriented Dialogue System Automatic Speech Recognition Natural Language Understanding (NLU) • User Intent Detection • Slot Filling Dialogue Management (DM) • Dialogue State Tracking (DST) • Dialogue Policy Decision Natural Language Generation (NLG) Hypothesis Any action movies recommended this weekend? Semantic Frame request_movie (genre=action, date=this weekend) System Action/Policy request_location Text response Which theater do you prefer? Text Input Any action movies recommended this weekend? Speech Signal Motivation: The pipelined system (NLU  DM) results in error propagation issues. 3 Text-To-Speech
  • 4. Proposed Approach  End-to-end model  Mitigate the effects of noisy output from NLU  Refine NLU by supervised signals from DM  Multi-task jointly learning  NLU - User intent classification  NLU - User slot tagging  DM - System action prediction  Contextual understanding  Access to the user history  Monitor user behavior states over turns 4
  • 5. Human-Human Dialogue Interaction Hi, how may I help you? Do you want to have a backpack type of hotel? So you don’t mind if it is not roomy, right? Okay. These hotels are available for you: … Thanks, goodbye. Are there any cheap rate hotels to put my bags? Yes. Just gonna leave our things there and stay out the whole day. Yes. Ok, thank you, bye! Idea: predicting the next system action given the current user utterance together with the aggregated observations Guide Agent Tourist User 5
  • 6. Natural Language Understanding (NLU) 6 Utterance: BOS % um how much is a taxi cab there ? EOS Slot Tags: O O O B-det_PRICE I-det_PRICE O O B-trsp_TYPE I-trsp_TYPE B-area_CITY O O User Intents: QST_HOW_MUCH; QST_INFO System Actions: RES_EXPLAIN; RES_INFO; FOL_EXPLAIN; FOL_HOW_MUCH; FOL_INFO Task 1: Slot Tagging Task 2: Multi-Label User Intent Prediction Sharedweights
  • 7. Task 1+2: Natural Language Understanding NLU+DM 1: Pipelined BLSTMs 7 single user turn current user turn w/ contextual history Utterance: BOS % um how much is a taxi cab there ? EOS Slot Tags: O O O B-det_PRICE I-det_PRICE O O B-trsp_TYPE I-trsp_TYPE B-area_CITY O O User Intents: QST_HOW_MUCH; QST_INFO System Actions: RES_EXPLAIN; RES_INFO; FOL_EXPLAIN; FOL_HOW_MUCH; FOL_INFO Task 3: Multi-Label System Action Prediction
  • 8. NLU+DM 2: End-to-End Model (JointModel) 8 Utterance: BOS % um how much is a taxi cab there ? EOS Slot Tags: O O O B-det_PRICE I-det_PRICE O O B-trsp_TYPE I-trsp_TYPE B-area_CITY O O User Intents: QST_HOW_MUCH; QST_INFO System Actions: RES_EXPLAIN; RES_INFO; FOL_EXPLAIN; FOL_HOW_MUCH; FOL_INFO Task 1+2+3: End-to-End Joint NLU+DM supervised learning with three tasks DM output signal refines NLU for better robustness DM
  • 9. Data  Dialogue State Tracking Challenge 4  Human-human dialogues: 21-hour dialogue sessions on touristic information collected via Skype between tour guides and tourists User Intent = Speech Act + Attributes System Action = Speech Act + Attributes Domains Accommodation Attraction Food Shopping Transportation Speech Act QST (QUESTION) RES (RESPONSE) INI (INITIATIVE) FOL (FOLLOW) Speech Act Attributes ACK CLOSING COMMIT THANK CANCEL CONFIRM ENOUGH WHAT EXPLAIN HOW_MUCH HOW_TO WHEN INFO NEGATIVE OPENING WHERE POSITIVE PERFERENCE RECOMMEND WHO Train Dev Test #utter 5,648 1,939 3,178 #intent 68 54 58 9
  • 10. DM Result – System Action Prediction (SAP) 10  Metric: frame-level accuracy (FrmAcc) Model FrmAcc Baseline (CRF+SVMs) 7.7 Pipeline-BLSTM 12.0 JointModel 22.8 Pipeline-BLSTM and JointModel outperform the baseline JointModel improves Pipeline-BLSTM about 10% accuracy, indicating the importance of mitigating downside of pipeline Human-human conversations are complicated, so predicting system actions for DM is difficult
  • 11. DM Result – System Action Prediction (SAP) 11  Metric: frame-level accuracy (FrmAcc) Model FrmAcc Baseline (CRF+SVMs) 7.7 Pipeline-BLSTM 12.0 JointModel 22.8 Oracle-SAP (SVM) 7.7 Oracel-SAP (BLSTM) 19.7 Oracle models show the upper-bound of the SAP performance, since it transfers the errors from NLU to SAP Contextual user turns make significant contribution to DM performance JointModel achieves the best DM performance (FrmAcc) with richer latent representations Fully correct NLU output
  • 12. NLU Result – Slot Filling & Intent Prediction 12  Metrics: frame-level accuracy (FrmAcc) Models Slot Intent NLU NLU-Baseline (CRF+SVM) 77.3 37.2 33.1 NLU-Pipeline-BLSTM 76.8 40.0 36.4 NLU-JointModel 76.5 42.2 37.4 CRF+SVMs baseline maintains strong NLU performance with 33.1% Pipeline-BLSTM and JointModel outperformed the baseline DM signal (system action prediction) helps more on user intent prediction than slot filling, and NLU is significantly improved Extra supervised DM signal helps refine the NLU by back-propagating the associated errors
  • 13. Conclusion 13  First propose an end-to-end deep hierarchical model for joint NLU and DM with limited contextual dialogue memory  Leverage multi-task learning using three supervised signals  NLU: User intent classification  NLU: Slot tagging  DM: System action prediction  Outperform the state-of-the-art pipelined NLU and DM models  Better DM due to the contextual dialogue memory  Robust NLU fine-tuned by supervised signal from DM
  • 15. Appendix - Baseline Model NLU: CRF for slot tagging, and One-Vs-All SVMs for intent classification SAP: One-Vs-All SVMs [1] Raymond, Christian, and Giuseppe Riccardi. “Generative and discriminative algorithms for spoken language understanding." In INTERSPEECH, pp. 1605-1608. 2007. Predicting system actions at the next turn as responses to the current user behaviors by pipelining NLU and SAP together 15
  • 16. Appendix: Configuration  Optimizer: a mini-batch stochastic gradient descent method Adam  Contextual history: five user turns  Dimension of word embedding: 512  Dropout ratio: 0.5  No early stopping, but use 300 training epochs  Best models for three tasks are selected individually under different metrics  Token-level micro-average F1 score is used for slot filling  frame-level accuracy (it counts only when the whole frame parse is correct) is used for both user intent prediction and system action prediction  the decision thresholds are tuned on dev set 16