SlideShare a Scribd company logo

Sequential Labeling for Tracking Dynamic Dialog States

Sequential Labeling for Tracking Dynamic Dialog States. Seokhwan Kim, Rafael E. Banchs. The 15th Annual SIGdial Meeting on Discourse and Dialogue (SIGDIAL 2014), Philadelphia, Jun 2014

1 of 1
Download to read offline
Sequential Labeling for Tracking Dynamic Dialog States
Seokhwan Kim, Rafael E. Banchs
Natural Language Understanding Lab., Human Language Technology Department,
Institute for Infocomm Research (I2
R), Singapore
Introduction
Dialog state tracking is a sub-task of dialog management that
analyzes and maintains the dialog state at each moment
The major obstacle to dialog state tracking is the errors
produced by ASR and SLU
This work focuses on the second dialog state tracking
challenge (DSTC 2) with three subtasks:
Goals Tracking
Method Tracking
Requested Slots Tracking
Problem Definition: Goals Tracking
Goals represent the constraint values which are truly intended
by a user at each moment
A problem of finding the distributions over the hypotheses for
the following four categories:
AREA
FOOD
NAME
PRICE RANGE
The models for DSTC 2 should be able to handle goal changes
during a session
An example of goal change in DSTC 2
Utterance
Goals
Food Area
S1 Hello, How may I help you?
U1 I need a Persian restaurant in the south part
of town.
Persian South
S2 What kind of food would you like?
U2 Persian. Persian South
S3 I’m sorry but there is no restaurant serving
persian food
U3 How about Portuguese food? Portuguese South
S4 Peking restaurant is a nice place in the south
of town.
U4 Is that Portuguese? Portuguese South
S5 Nandos is a nice place in the south of town
serving tasty Portuguese food.
U5 Alright. Whats the phone number? Portuguese South
S6 The phone number of nandos is 01223
327908 .
U6 And the address? Portuguese South
S7 Sure, nandos is on Cambridge Leisure Park
Clifton Way.
U7 Thank you good bye.
Problem Definition: Method Tracking
To classify the way of requesting information by a user into the
following four categories: ‘by constraints’, ‘by alternatives’, ‘by
name’, ‘finished’
The probability distribution is computed for each turn
Problem Definition: Requested Slots Tracking
To specify the slots requested by a user
The binary distributions are computed with the probabilities
whether each slot is requested or not
Requestable slots: AREA, FOOD, NAME, PRICERANGE,
ADDR, PHONE, POSTCODE, SIGNATURE
Method: Sequential Labeling of Dialog States
To produce the most probable label sequence y = {y1, · · · , yn}
of a given input sequence x = {x1, · · · , xn}
BIO tagging scheme
To detect the boundaries of the label chunks
Considering discourse coherences in conversation
An example of goal chain on the food slot
Linear Chain CRFs
Conditional probability distributions over the label sequences y
conditioned on the input sequence x
p (y|x) =
1
Z (x)
n
t=1
Ψ(yt, yt−1, x),
Ψ(yt, yt−1, x) = Ψ1(yt, x) · Ψ2(yt, yt−1)
Ψ1(yt, x) = exp ( k λkfk(yt, x))
Ψ2(yt, yt−1) = exp ( k λkfk(yt, yt−1))
Experimental Settings
DSTC 2 dataset
3,235 dialog sessions on restaurant information domain
TRAINING: 1,612 sessions
DEVELOPMENT: 506 sessions
TEST: 1,117 sessions
The results of ASR and SLU are annotated for every turn in the dataset,
as well as the gold standard annotations are also provided for evaluation
Models
CRF (Conditional Random Fields): with sequential labeling
ME (Maximum Entropy): without sequential labeling
Features
SLU Hypothesis: inform, confirm, deny, affirm, negate, request, reqalts
System Action: expl-conf, impl-conf, request, select, canthelp
Evaluation Metrics
Features metrics: Accuracy, L2 norm, ROC CA 5
On Joint Goals, Method, and Requested Slots
Experimental Results
Comparisons of dialog state tracking performances
Dev set Test set
Acc L2 ROC Acc L2 ROC
Joint Goals
ME 0.638 0.551 0.144 0.596 0.671 0.036
CRF 0.644 0.545 0.103 0.601 0.649 0.064
Method
ME 0.839 0.260 0.398 0.877 0.204 0.397
CRF 0.875 0.202 0.181 0.904 0.155 0.187
Requested Slots
ME 0.946 0.099 0.000 0.957 0.081 0.000
CRF 0.942 0.107 0.000 0.960 0.073 0.000
CRF models produced better joint goals and method in
accuracy and L2 norm on both development and test sets
For the requested slots task, our proposed approach achieved
better results than the baseline on the test set
Conclusion
This paper presented a sequential labeling approach for dialog
state tracking
Experimental results show the merits of our proposed approach
with the improved performances on all the sub-tasks of DSTC 2
If we discover more advanced features that help to track the
proper dialog states, they can raise the overall performances
further
1 Fusionopolis Way, #21-01 Connexis (South Tower), Singapore 138632 Email: kims@i2r.a-star.edu.sg WWW: http://hlt.i2r.a-star.edu.sg/

Recommended

A Graph-based Cross-lingual Projection Approach for Spoken Language Understan...
A Graph-based Cross-lingual Projection Approach for Spoken Language Understan...A Graph-based Cross-lingual Projection Approach for Spoken Language Understan...
A Graph-based Cross-lingual Projection Approach for Spoken Language Understan...Seokhwan Kim
 
IIT-TUDA at SemEval-2016 Task 5: Beyond Sentiment Lexicon: Combining Domain ...
IIT-TUDA at SemEval-2016 Task 5: Beyond Sentiment Lexicon: Combining Domain ...IIT-TUDA at SemEval-2016 Task 5: Beyond Sentiment Lexicon: Combining Domain ...
IIT-TUDA at SemEval-2016 Task 5: Beyond Sentiment Lexicon: Combining Domain ...Alexander Panchenko
 
End-to-End Joint Learning of Natural Language Understanding and Dialogue Manager
End-to-End Joint Learning of Natural Language Understanding and Dialogue ManagerEnd-to-End Joint Learning of Natural Language Understanding and Dialogue Manager
End-to-End Joint Learning of Natural Language Understanding and Dialogue ManagerYun-Nung (Vivian) Chen
 
Automated Parameterization of Performance Models from Measurements
Automated Parameterization of Performance Models from MeasurementsAutomated Parameterization of Performance Models from Measurements
Automated Parameterization of Performance Models from MeasurementsWeikun Wang
 
Change Impact Analysis for Natural Language Requirements
Change Impact Analysis for Natural Language RequirementsChange Impact Analysis for Natural Language Requirements
Change Impact Analysis for Natural Language RequirementsLionel Briand
 
Steven K Allott - Effective Testing - SoftTest Ireland
Steven K Allott - Effective Testing - SoftTest IrelandSteven K Allott - Effective Testing - SoftTest Ireland
Steven K Allott - Effective Testing - SoftTest IrelandDavid O'Dowd
 
Master's Degree thesis presentation
Master's Degree thesis presentationMaster's Degree thesis presentation
Master's Degree thesis presentationDavide De Chiara
 

More Related Content

Similar to Sequential Labeling for Tracking Dynamic Dialog States

Interpretability and Reproducibility in Production Machine Learning Applicat...
 Interpretability and Reproducibility in Production Machine Learning Applicat... Interpretability and Reproducibility in Production Machine Learning Applicat...
Interpretability and Reproducibility in Production Machine Learning Applicat...Swaminathan Sundararaman
 
"Test Design Techniques"
"Test Design Techniques" "Test Design Techniques"
"Test Design Techniques" HYS Enterprise
 
Towards better software quality assurance by providing intelligent support
Towards better software quality assurance by providing intelligent supportTowards better software quality assurance by providing intelligent support
Towards better software quality assurance by providing intelligent supportConcordia University
 
HARE: An Engine for Enhancing Answer Completeness of SPARQL Queries via Crowd...
HARE: An Engine for Enhancing Answer Completeness of SPARQL Queries via Crowd...HARE: An Engine for Enhancing Answer Completeness of SPARQL Queries via Crowd...
HARE: An Engine for Enhancing Answer Completeness of SPARQL Queries via Crowd...Maribel Acosta Deibe
 
Noise Pollution in Hospital Readmission Prediction: Long Document Classificat...
Noise Pollution in Hospital Readmission Prediction: Long Document Classificat...Noise Pollution in Hospital Readmission Prediction: Long Document Classificat...
Noise Pollution in Hospital Readmission Prediction: Long Document Classificat...Jinho Choi
 
OCLR: A More Expressive, Pattern-Based Temporal Extension of OCL
OCLR: A More Expressive, Pattern-Based Temporal Extension of OCLOCLR: A More Expressive, Pattern-Based Temporal Extension of OCL
OCLR: A More Expressive, Pattern-Based Temporal Extension of OCLLionel Briand
 
Rethinking action spaces for reinforcement learning in end-to-end dialog agen...
Rethinking action spaces for reinforcement learning in end-to-end dialog agen...Rethinking action spaces for reinforcement learning in end-to-end dialog agen...
Rethinking action spaces for reinforcement learning in end-to-end dialog agen...Jeong-Gwan Lee
 
Thesis defense 2017
Thesis defense 2017Thesis defense 2017
Thesis defense 2017yashiitr
 
Isolated word recognition using lpc & vector quantization
Isolated word recognition using lpc & vector quantizationIsolated word recognition using lpc & vector quantization
Isolated word recognition using lpc & vector quantizationeSAT Journals
 
Isolated word recognition using lpc & vector quantization
Isolated word recognition using lpc & vector quantizationIsolated word recognition using lpc & vector quantization
Isolated word recognition using lpc & vector quantizationeSAT Publishing House
 
Studies On Traffic Management Models for Wireless Communication Network
Studies On Traffic Management Models for Wireless Communication NetworkStudies On Traffic Management Models for Wireless Communication Network
Studies On Traffic Management Models for Wireless Communication NetworkNeetaSingh38
 
Holistic Analysis and Optimization of Heterogeneous Fault-Tolerant Embedded S...
Holistic Analysis and Optimization of Heterogeneous Fault-Tolerant Embedded S...Holistic Analysis and Optimization of Heterogeneous Fault-Tolerant Embedded S...
Holistic Analysis and Optimization of Heterogeneous Fault-Tolerant Embedded S...paupo
 
Gas station competition_model
Gas station competition_modelGas station competition_model
Gas station competition_modelAri Pramono
 
Simple vs. Compound Mark Hierarchical Marking Menus
Simple vs. Compound Mark Hierarchical Marking MenusSimple vs. Compound Mark Hierarchical Marking Menus
Simple vs. Compound Mark Hierarchical Marking MenusKimberly Aguada
 
Face recognition v1
Face recognition v1Face recognition v1
Face recognition v1San Kim
 
A tale of experiments on bug prediction
A tale of experiments on bug predictionA tale of experiments on bug prediction
A tale of experiments on bug predictionMartin Pinzger
 

Similar to Sequential Labeling for Tracking Dynamic Dialog States (20)

Interpretability and Reproducibility in Production Machine Learning Applicat...
 Interpretability and Reproducibility in Production Machine Learning Applicat... Interpretability and Reproducibility in Production Machine Learning Applicat...
Interpretability and Reproducibility in Production Machine Learning Applicat...
 
"Test Design Techniques"
"Test Design Techniques" "Test Design Techniques"
"Test Design Techniques"
 
Ngs part iii 2013
Ngs part iii 2013Ngs part iii 2013
Ngs part iii 2013
 
Towards better software quality assurance by providing intelligent support
Towards better software quality assurance by providing intelligent supportTowards better software quality assurance by providing intelligent support
Towards better software quality assurance by providing intelligent support
 
HARE: An Engine for Enhancing Answer Completeness of SPARQL Queries via Crowd...
HARE: An Engine for Enhancing Answer Completeness of SPARQL Queries via Crowd...HARE: An Engine for Enhancing Answer Completeness of SPARQL Queries via Crowd...
HARE: An Engine for Enhancing Answer Completeness of SPARQL Queries via Crowd...
 
Noise Pollution in Hospital Readmission Prediction: Long Document Classificat...
Noise Pollution in Hospital Readmission Prediction: Long Document Classificat...Noise Pollution in Hospital Readmission Prediction: Long Document Classificat...
Noise Pollution in Hospital Readmission Prediction: Long Document Classificat...
 
Chi Square & Anova
Chi Square & AnovaChi Square & Anova
Chi Square & Anova
 
OCLR: A More Expressive, Pattern-Based Temporal Extension of OCL
OCLR: A More Expressive, Pattern-Based Temporal Extension of OCLOCLR: A More Expressive, Pattern-Based Temporal Extension of OCL
OCLR: A More Expressive, Pattern-Based Temporal Extension of OCL
 
Rethinking action spaces for reinforcement learning in end-to-end dialog agen...
Rethinking action spaces for reinforcement learning in end-to-end dialog agen...Rethinking action spaces for reinforcement learning in end-to-end dialog agen...
Rethinking action spaces for reinforcement learning in end-to-end dialog agen...
 
Thesis defense 2017
Thesis defense 2017Thesis defense 2017
Thesis defense 2017
 
Isolated word recognition using lpc & vector quantization
Isolated word recognition using lpc & vector quantizationIsolated word recognition using lpc & vector quantization
Isolated word recognition using lpc & vector quantization
 
Isolated word recognition using lpc & vector quantization
Isolated word recognition using lpc & vector quantizationIsolated word recognition using lpc & vector quantization
Isolated word recognition using lpc & vector quantization
 
Studies On Traffic Management Models for Wireless Communication Network
Studies On Traffic Management Models for Wireless Communication NetworkStudies On Traffic Management Models for Wireless Communication Network
Studies On Traffic Management Models for Wireless Communication Network
 
Holistic Analysis and Optimization of Heterogeneous Fault-Tolerant Embedded S...
Holistic Analysis and Optimization of Heterogeneous Fault-Tolerant Embedded S...Holistic Analysis and Optimization of Heterogeneous Fault-Tolerant Embedded S...
Holistic Analysis and Optimization of Heterogeneous Fault-Tolerant Embedded S...
 
50320130403005 2
50320130403005 250320130403005 2
50320130403005 2
 
Gas station competition_model
Gas station competition_modelGas station competition_model
Gas station competition_model
 
CORRECT-ICSE2016
CORRECT-ICSE2016CORRECT-ICSE2016
CORRECT-ICSE2016
 
Simple vs. Compound Mark Hierarchical Marking Menus
Simple vs. Compound Mark Hierarchical Marking MenusSimple vs. Compound Mark Hierarchical Marking Menus
Simple vs. Compound Mark Hierarchical Marking Menus
 
Face recognition v1
Face recognition v1Face recognition v1
Face recognition v1
 
A tale of experiments on bug prediction
A tale of experiments on bug predictionA tale of experiments on bug prediction
A tale of experiments on bug prediction
 

More from Seokhwan Kim

The Eighth Dialog System Technology Challenge (DSTC8)
The Eighth Dialog System Technology Challenge (DSTC8)The Eighth Dialog System Technology Challenge (DSTC8)
The Eighth Dialog System Technology Challenge (DSTC8)Seokhwan Kim
 
Deep Recurrent Neural Networks with Layer-wise Multi-head Attentions for Punc...
Deep Recurrent Neural Networks with Layer-wise Multi-head Attentions for Punc...Deep Recurrent Neural Networks with Layer-wise Multi-head Attentions for Punc...
Deep Recurrent Neural Networks with Layer-wise Multi-head Attentions for Punc...Seokhwan Kim
 
Dynamic Memory Networks for Dialogue Topic Tracking
Dynamic Memory Networks for Dialogue Topic TrackingDynamic Memory Networks for Dialogue Topic Tracking
Dynamic Memory Networks for Dialogue Topic TrackingSeokhwan Kim
 
The Fifth Dialog State Tracking Challenge (DSTC5)
The Fifth Dialog State Tracking Challenge (DSTC5)The Fifth Dialog State Tracking Challenge (DSTC5)
The Fifth Dialog State Tracking Challenge (DSTC5)Seokhwan Kim
 
Natural Language in Human-Robot Interaction
Natural Language in Human-Robot InteractionNatural Language in Human-Robot Interaction
Natural Language in Human-Robot InteractionSeokhwan Kim
 
Exploring Convolutional and Recurrent Neural Networks in Sequential Labelling...
Exploring Convolutional and Recurrent Neural Networks in Sequential Labelling...Exploring Convolutional and Recurrent Neural Networks in Sequential Labelling...
Exploring Convolutional and Recurrent Neural Networks in Sequential Labelling...Seokhwan Kim
 
The Fourth Dialog State Tracking Challenge (DSTC4)
The Fourth Dialog State Tracking Challenge (DSTC4)The Fourth Dialog State Tracking Challenge (DSTC4)
The Fourth Dialog State Tracking Challenge (DSTC4)Seokhwan Kim
 
Wikification of Concept Mentions within Spoken Dialogues Using Domain Constra...
Wikification of Concept Mentions within Spoken Dialogues Using Domain Constra...Wikification of Concept Mentions within Spoken Dialogues Using Domain Constra...
Wikification of Concept Mentions within Spoken Dialogues Using Domain Constra...Seokhwan Kim
 
Towards Improving Dialogue Topic Tracking Performances with Wikification of C...
Towards Improving Dialogue Topic Tracking Performances with Wikification of C...Towards Improving Dialogue Topic Tracking Performances with Wikification of C...
Towards Improving Dialogue Topic Tracking Performances with Wikification of C...Seokhwan Kim
 
A Composite Kernel Approach for Dialog Topic Tracking with Structured Domain ...
A Composite Kernel Approach for Dialog Topic Tracking with Structured Domain ...A Composite Kernel Approach for Dialog Topic Tracking with Structured Domain ...
A Composite Kernel Approach for Dialog Topic Tracking with Structured Domain ...Seokhwan Kim
 
Wikipedia-based Kernels for Dialogue Topic Tracking
Wikipedia-based Kernels for Dialogue Topic TrackingWikipedia-based Kernels for Dialogue Topic Tracking
Wikipedia-based Kernels for Dialogue Topic TrackingSeokhwan Kim
 
A Graph-based Cross-lingual Projection Approach for Weakly Supervised Relatio...
A Graph-based Cross-lingual Projection Approach for Weakly Supervised Relatio...A Graph-based Cross-lingual Projection Approach for Weakly Supervised Relatio...
A Graph-based Cross-lingual Projection Approach for Weakly Supervised Relatio...Seokhwan Kim
 
MMR-based active machine learning for Bio named entity recognition
MMR-based active machine learning for Bio named entity recognitionMMR-based active machine learning for Bio named entity recognition
MMR-based active machine learning for Bio named entity recognitionSeokhwan Kim
 
A semi-supervised method for efficient construction of statistical spoken lan...
A semi-supervised method for efficient construction of statistical spoken lan...A semi-supervised method for efficient construction of statistical spoken lan...
A semi-supervised method for efficient construction of statistical spoken lan...Seokhwan Kim
 
A spoken dialog system for electronic program guide information access
A spoken dialog system for electronic program guide information accessA spoken dialog system for electronic program guide information access
A spoken dialog system for electronic program guide information accessSeokhwan Kim
 
An alignment-based approach to semi-supervised relation extraction including ...
An alignment-based approach to semi-supervised relation extraction including ...An alignment-based approach to semi-supervised relation extraction including ...
An alignment-based approach to semi-supervised relation extraction including ...Seokhwan Kim
 
An Alignment-based Pattern Representation Model for Information Extraction
An Alignment-based Pattern Representation Model for Information ExtractionAn Alignment-based Pattern Representation Model for Information Extraction
An Alignment-based Pattern Representation Model for Information ExtractionSeokhwan Kim
 
EPG 정보 검색을 위한 예제 기반 자연어 대화 시스템
EPG 정보 검색을 위한 예제 기반 자연어 대화 시스템EPG 정보 검색을 위한 예제 기반 자연어 대화 시스템
EPG 정보 검색을 위한 예제 기반 자연어 대화 시스템Seokhwan Kim
 
A Cross-Lingual Annotation Projection Approach for Relation Detection
A Cross-Lingual Annotation Projection Approach for Relation DetectionA Cross-Lingual Annotation Projection Approach for Relation Detection
A Cross-Lingual Annotation Projection Approach for Relation DetectionSeokhwan Kim
 
A Cross-lingual Annotation Projection-based Self-supervision Approach for Ope...
A Cross-lingual Annotation Projection-based Self-supervision Approach for Ope...A Cross-lingual Annotation Projection-based Self-supervision Approach for Ope...
A Cross-lingual Annotation Projection-based Self-supervision Approach for Ope...Seokhwan Kim
 

More from Seokhwan Kim (20)

The Eighth Dialog System Technology Challenge (DSTC8)
The Eighth Dialog System Technology Challenge (DSTC8)The Eighth Dialog System Technology Challenge (DSTC8)
The Eighth Dialog System Technology Challenge (DSTC8)
 
Deep Recurrent Neural Networks with Layer-wise Multi-head Attentions for Punc...
Deep Recurrent Neural Networks with Layer-wise Multi-head Attentions for Punc...Deep Recurrent Neural Networks with Layer-wise Multi-head Attentions for Punc...
Deep Recurrent Neural Networks with Layer-wise Multi-head Attentions for Punc...
 
Dynamic Memory Networks for Dialogue Topic Tracking
Dynamic Memory Networks for Dialogue Topic TrackingDynamic Memory Networks for Dialogue Topic Tracking
Dynamic Memory Networks for Dialogue Topic Tracking
 
The Fifth Dialog State Tracking Challenge (DSTC5)
The Fifth Dialog State Tracking Challenge (DSTC5)The Fifth Dialog State Tracking Challenge (DSTC5)
The Fifth Dialog State Tracking Challenge (DSTC5)
 
Natural Language in Human-Robot Interaction
Natural Language in Human-Robot InteractionNatural Language in Human-Robot Interaction
Natural Language in Human-Robot Interaction
 
Exploring Convolutional and Recurrent Neural Networks in Sequential Labelling...
Exploring Convolutional and Recurrent Neural Networks in Sequential Labelling...Exploring Convolutional and Recurrent Neural Networks in Sequential Labelling...
Exploring Convolutional and Recurrent Neural Networks in Sequential Labelling...
 
The Fourth Dialog State Tracking Challenge (DSTC4)
The Fourth Dialog State Tracking Challenge (DSTC4)The Fourth Dialog State Tracking Challenge (DSTC4)
The Fourth Dialog State Tracking Challenge (DSTC4)
 
Wikification of Concept Mentions within Spoken Dialogues Using Domain Constra...
Wikification of Concept Mentions within Spoken Dialogues Using Domain Constra...Wikification of Concept Mentions within Spoken Dialogues Using Domain Constra...
Wikification of Concept Mentions within Spoken Dialogues Using Domain Constra...
 
Towards Improving Dialogue Topic Tracking Performances with Wikification of C...
Towards Improving Dialogue Topic Tracking Performances with Wikification of C...Towards Improving Dialogue Topic Tracking Performances with Wikification of C...
Towards Improving Dialogue Topic Tracking Performances with Wikification of C...
 
A Composite Kernel Approach for Dialog Topic Tracking with Structured Domain ...
A Composite Kernel Approach for Dialog Topic Tracking with Structured Domain ...A Composite Kernel Approach for Dialog Topic Tracking with Structured Domain ...
A Composite Kernel Approach for Dialog Topic Tracking with Structured Domain ...
 
Wikipedia-based Kernels for Dialogue Topic Tracking
Wikipedia-based Kernels for Dialogue Topic TrackingWikipedia-based Kernels for Dialogue Topic Tracking
Wikipedia-based Kernels for Dialogue Topic Tracking
 
A Graph-based Cross-lingual Projection Approach for Weakly Supervised Relatio...
A Graph-based Cross-lingual Projection Approach for Weakly Supervised Relatio...A Graph-based Cross-lingual Projection Approach for Weakly Supervised Relatio...
A Graph-based Cross-lingual Projection Approach for Weakly Supervised Relatio...
 
MMR-based active machine learning for Bio named entity recognition
MMR-based active machine learning for Bio named entity recognitionMMR-based active machine learning for Bio named entity recognition
MMR-based active machine learning for Bio named entity recognition
 
A semi-supervised method for efficient construction of statistical spoken lan...
A semi-supervised method for efficient construction of statistical spoken lan...A semi-supervised method for efficient construction of statistical spoken lan...
A semi-supervised method for efficient construction of statistical spoken lan...
 
A spoken dialog system for electronic program guide information access
A spoken dialog system for electronic program guide information accessA spoken dialog system for electronic program guide information access
A spoken dialog system for electronic program guide information access
 
An alignment-based approach to semi-supervised relation extraction including ...
An alignment-based approach to semi-supervised relation extraction including ...An alignment-based approach to semi-supervised relation extraction including ...
An alignment-based approach to semi-supervised relation extraction including ...
 
An Alignment-based Pattern Representation Model for Information Extraction
An Alignment-based Pattern Representation Model for Information ExtractionAn Alignment-based Pattern Representation Model for Information Extraction
An Alignment-based Pattern Representation Model for Information Extraction
 
EPG 정보 검색을 위한 예제 기반 자연어 대화 시스템
EPG 정보 검색을 위한 예제 기반 자연어 대화 시스템EPG 정보 검색을 위한 예제 기반 자연어 대화 시스템
EPG 정보 검색을 위한 예제 기반 자연어 대화 시스템
 
A Cross-Lingual Annotation Projection Approach for Relation Detection
A Cross-Lingual Annotation Projection Approach for Relation DetectionA Cross-Lingual Annotation Projection Approach for Relation Detection
A Cross-Lingual Annotation Projection Approach for Relation Detection
 
A Cross-lingual Annotation Projection-based Self-supervision Approach for Ope...
A Cross-lingual Annotation Projection-based Self-supervision Approach for Ope...A Cross-lingual Annotation Projection-based Self-supervision Approach for Ope...
A Cross-lingual Annotation Projection-based Self-supervision Approach for Ope...
 

Recently uploaded

Endocrine.pptx Organs/Glands in Endocrine System
Endocrine.pptx Organs/Glands in Endocrine SystemEndocrine.pptx Organs/Glands in Endocrine System
Endocrine.pptx Organs/Glands in Endocrine SystemJhonatanGarciaMendez
 
Seminario Biologia Molecular Nicole Michel Rojas Torres
Seminario Biologia Molecular Nicole Michel Rojas TorresSeminario Biologia Molecular Nicole Michel Rojas Torres
Seminario Biologia Molecular Nicole Michel Rojas Torresnicoledoc2004
 
2024 Insilicogen Company English Brochure
2024 Insilicogen Company English Brochure2024 Insilicogen Company English Brochure
2024 Insilicogen Company English BrochureInsilico Gen
 
Weak-lensing detection of intracluster filaments in the Coma cluster
Weak-lensing detection of intracluster filaments in the Coma clusterWeak-lensing detection of intracluster filaments in the Coma cluster
Weak-lensing detection of intracluster filaments in the Coma clusterSérgio Sacani
 
5FSHB- TANDOOR MURDER CASE (GROUP 1).pdf
5FSHB- TANDOOR MURDER CASE (GROUP 1).pdf5FSHB- TANDOOR MURDER CASE (GROUP 1).pdf
5FSHB- TANDOOR MURDER CASE (GROUP 1).pdfSubranjitaPriyadarsh
 
LIGHT Community Medicine LIGHT IS A SOURCE OF ENERGY THERE ARE TWO TYPE OF S...
LIGHT  Community Medicine LIGHT IS A SOURCE OF ENERGY THERE ARE TWO TYPE OF S...LIGHT  Community Medicine LIGHT IS A SOURCE OF ENERGY THERE ARE TWO TYPE OF S...
LIGHT Community Medicine LIGHT IS A SOURCE OF ENERGY THERE ARE TWO TYPE OF S...Abhinav S
 
Elbow joint - Anatomy of the Elbow joint
Elbow joint - Anatomy of the Elbow jointElbow joint - Anatomy of the Elbow joint
Elbow joint - Anatomy of the Elbow jointTELISHA2
 
Open Access Publishing in Astrophysics and the Open Journal of Astrophysics
Open Access Publishing in Astrophysics and the Open Journal of AstrophysicsOpen Access Publishing in Astrophysics and the Open Journal of Astrophysics
Open Access Publishing in Astrophysics and the Open Journal of AstrophysicsPeter Coles
 
REJUVENATION THROUGH PROGENY ORCHAD AND SCION BANK
REJUVENATION THROUGH PROGENY ORCHAD AND SCION BANKREJUVENATION THROUGH PROGENY ORCHAD AND SCION BANK
REJUVENATION THROUGH PROGENY ORCHAD AND SCION BANKAmanDohre
 
Earth and Planetary Science | Volume 01 | Issue 01 | April 2022
Earth and Planetary Science | Volume 01 | Issue 01 | April 2022Earth and Planetary Science | Volume 01 | Issue 01 | April 2022
Earth and Planetary Science | Volume 01 | Issue 01 | April 2022Nan Yang Academy of Sciences
 
Rootstock scion and Interstock Relationship Selection of Elite Mother Plants
Rootstock scion and Interstock Relationship Selection of Elite Mother PlantsRootstock scion and Interstock Relationship Selection of Elite Mother Plants
Rootstock scion and Interstock Relationship Selection of Elite Mother PlantsAmanDohre
 
Salesforce Starter Package Presentation.
Salesforce Starter Package Presentation.Salesforce Starter Package Presentation.
Salesforce Starter Package Presentation.Naresh Gupta
 
Chemical Bonding and it's Types 001.pptx
Chemical Bonding and it's Types 001.pptxChemical Bonding and it's Types 001.pptx
Chemical Bonding and it's Types 001.pptxperiyar arts college
 
Introduction to Chromatography (Column chromatography)
Introduction to Chromatography (Column chromatography)Introduction to Chromatography (Column chromatography)
Introduction to Chromatography (Column chromatography)Ahmed Metwaly
 
Hydro-Thermal Liquefaction Of Lignocellulosic biomass to produce Bio-Crude oil
Hydro-Thermal Liquefaction Of Lignocellulosic biomass to produce Bio-Crude oilHydro-Thermal Liquefaction Of Lignocellulosic biomass to produce Bio-Crude oil
Hydro-Thermal Liquefaction Of Lignocellulosic biomass to produce Bio-Crude oilZeeshan Nazir
 
Seminario biología molecular Lina Charris
Seminario biología molecular Lina CharrisSeminario biología molecular Lina Charris
Seminario biología molecular Lina CharrisLinaMarcelaCharrisRa
 
green chemistry, clean sustainable environment.ppt
green chemistry, clean sustainable environment.pptgreen chemistry, clean sustainable environment.ppt
green chemistry, clean sustainable environment.pptRashmiSanghi1
 
Genetic Code. A comprehensive overview..pdf
Genetic Code. A comprehensive overview..pdfGenetic Code. A comprehensive overview..pdf
Genetic Code. A comprehensive overview..pdfmughalgumar440
 
commercial production of cellulase enzyme and its uses
commercial production of cellulase enzyme and its usescommercial production of cellulase enzyme and its uses
commercial production of cellulase enzyme and its usesSilpa Selvaraj
 
Study of plant growth regulators in fruit culture – structure, biosynthesis ...
Study of plant growth regulators in fruit culture – structure, biosynthesis  ...Study of plant growth regulators in fruit culture – structure, biosynthesis  ...
Study of plant growth regulators in fruit culture – structure, biosynthesis ...AmanDohre
 

Recently uploaded (20)

Endocrine.pptx Organs/Glands in Endocrine System
Endocrine.pptx Organs/Glands in Endocrine SystemEndocrine.pptx Organs/Glands in Endocrine System
Endocrine.pptx Organs/Glands in Endocrine System
 
Seminario Biologia Molecular Nicole Michel Rojas Torres
Seminario Biologia Molecular Nicole Michel Rojas TorresSeminario Biologia Molecular Nicole Michel Rojas Torres
Seminario Biologia Molecular Nicole Michel Rojas Torres
 
2024 Insilicogen Company English Brochure
2024 Insilicogen Company English Brochure2024 Insilicogen Company English Brochure
2024 Insilicogen Company English Brochure
 
Weak-lensing detection of intracluster filaments in the Coma cluster
Weak-lensing detection of intracluster filaments in the Coma clusterWeak-lensing detection of intracluster filaments in the Coma cluster
Weak-lensing detection of intracluster filaments in the Coma cluster
 
5FSHB- TANDOOR MURDER CASE (GROUP 1).pdf
5FSHB- TANDOOR MURDER CASE (GROUP 1).pdf5FSHB- TANDOOR MURDER CASE (GROUP 1).pdf
5FSHB- TANDOOR MURDER CASE (GROUP 1).pdf
 
LIGHT Community Medicine LIGHT IS A SOURCE OF ENERGY THERE ARE TWO TYPE OF S...
LIGHT  Community Medicine LIGHT IS A SOURCE OF ENERGY THERE ARE TWO TYPE OF S...LIGHT  Community Medicine LIGHT IS A SOURCE OF ENERGY THERE ARE TWO TYPE OF S...
LIGHT Community Medicine LIGHT IS A SOURCE OF ENERGY THERE ARE TWO TYPE OF S...
 
Elbow joint - Anatomy of the Elbow joint
Elbow joint - Anatomy of the Elbow jointElbow joint - Anatomy of the Elbow joint
Elbow joint - Anatomy of the Elbow joint
 
Open Access Publishing in Astrophysics and the Open Journal of Astrophysics
Open Access Publishing in Astrophysics and the Open Journal of AstrophysicsOpen Access Publishing in Astrophysics and the Open Journal of Astrophysics
Open Access Publishing in Astrophysics and the Open Journal of Astrophysics
 
REJUVENATION THROUGH PROGENY ORCHAD AND SCION BANK
REJUVENATION THROUGH PROGENY ORCHAD AND SCION BANKREJUVENATION THROUGH PROGENY ORCHAD AND SCION BANK
REJUVENATION THROUGH PROGENY ORCHAD AND SCION BANK
 
Earth and Planetary Science | Volume 01 | Issue 01 | April 2022
Earth and Planetary Science | Volume 01 | Issue 01 | April 2022Earth and Planetary Science | Volume 01 | Issue 01 | April 2022
Earth and Planetary Science | Volume 01 | Issue 01 | April 2022
 
Rootstock scion and Interstock Relationship Selection of Elite Mother Plants
Rootstock scion and Interstock Relationship Selection of Elite Mother PlantsRootstock scion and Interstock Relationship Selection of Elite Mother Plants
Rootstock scion and Interstock Relationship Selection of Elite Mother Plants
 
Salesforce Starter Package Presentation.
Salesforce Starter Package Presentation.Salesforce Starter Package Presentation.
Salesforce Starter Package Presentation.
 
Chemical Bonding and it's Types 001.pptx
Chemical Bonding and it's Types 001.pptxChemical Bonding and it's Types 001.pptx
Chemical Bonding and it's Types 001.pptx
 
Introduction to Chromatography (Column chromatography)
Introduction to Chromatography (Column chromatography)Introduction to Chromatography (Column chromatography)
Introduction to Chromatography (Column chromatography)
 
Hydro-Thermal Liquefaction Of Lignocellulosic biomass to produce Bio-Crude oil
Hydro-Thermal Liquefaction Of Lignocellulosic biomass to produce Bio-Crude oilHydro-Thermal Liquefaction Of Lignocellulosic biomass to produce Bio-Crude oil
Hydro-Thermal Liquefaction Of Lignocellulosic biomass to produce Bio-Crude oil
 
Seminario biología molecular Lina Charris
Seminario biología molecular Lina CharrisSeminario biología molecular Lina Charris
Seminario biología molecular Lina Charris
 
green chemistry, clean sustainable environment.ppt
green chemistry, clean sustainable environment.pptgreen chemistry, clean sustainable environment.ppt
green chemistry, clean sustainable environment.ppt
 
Genetic Code. A comprehensive overview..pdf
Genetic Code. A comprehensive overview..pdfGenetic Code. A comprehensive overview..pdf
Genetic Code. A comprehensive overview..pdf
 
commercial production of cellulase enzyme and its uses
commercial production of cellulase enzyme and its usescommercial production of cellulase enzyme and its uses
commercial production of cellulase enzyme and its uses
 
Study of plant growth regulators in fruit culture – structure, biosynthesis ...
Study of plant growth regulators in fruit culture – structure, biosynthesis  ...Study of plant growth regulators in fruit culture – structure, biosynthesis  ...
Study of plant growth regulators in fruit culture – structure, biosynthesis ...
 

Sequential Labeling for Tracking Dynamic Dialog States

  • 1. Sequential Labeling for Tracking Dynamic Dialog States Seokhwan Kim, Rafael E. Banchs Natural Language Understanding Lab., Human Language Technology Department, Institute for Infocomm Research (I2 R), Singapore Introduction Dialog state tracking is a sub-task of dialog management that analyzes and maintains the dialog state at each moment The major obstacle to dialog state tracking is the errors produced by ASR and SLU This work focuses on the second dialog state tracking challenge (DSTC 2) with three subtasks: Goals Tracking Method Tracking Requested Slots Tracking Problem Definition: Goals Tracking Goals represent the constraint values which are truly intended by a user at each moment A problem of finding the distributions over the hypotheses for the following four categories: AREA FOOD NAME PRICE RANGE The models for DSTC 2 should be able to handle goal changes during a session An example of goal change in DSTC 2 Utterance Goals Food Area S1 Hello, How may I help you? U1 I need a Persian restaurant in the south part of town. Persian South S2 What kind of food would you like? U2 Persian. Persian South S3 I’m sorry but there is no restaurant serving persian food U3 How about Portuguese food? Portuguese South S4 Peking restaurant is a nice place in the south of town. U4 Is that Portuguese? Portuguese South S5 Nandos is a nice place in the south of town serving tasty Portuguese food. U5 Alright. Whats the phone number? Portuguese South S6 The phone number of nandos is 01223 327908 . U6 And the address? Portuguese South S7 Sure, nandos is on Cambridge Leisure Park Clifton Way. U7 Thank you good bye. Problem Definition: Method Tracking To classify the way of requesting information by a user into the following four categories: ‘by constraints’, ‘by alternatives’, ‘by name’, ‘finished’ The probability distribution is computed for each turn Problem Definition: Requested Slots Tracking To specify the slots requested by a user The binary distributions are computed with the probabilities whether each slot is requested or not Requestable slots: AREA, FOOD, NAME, PRICERANGE, ADDR, PHONE, POSTCODE, SIGNATURE Method: Sequential Labeling of Dialog States To produce the most probable label sequence y = {y1, · · · , yn} of a given input sequence x = {x1, · · · , xn} BIO tagging scheme To detect the boundaries of the label chunks Considering discourse coherences in conversation An example of goal chain on the food slot Linear Chain CRFs Conditional probability distributions over the label sequences y conditioned on the input sequence x p (y|x) = 1 Z (x) n t=1 Ψ(yt, yt−1, x), Ψ(yt, yt−1, x) = Ψ1(yt, x) · Ψ2(yt, yt−1) Ψ1(yt, x) = exp ( k λkfk(yt, x)) Ψ2(yt, yt−1) = exp ( k λkfk(yt, yt−1)) Experimental Settings DSTC 2 dataset 3,235 dialog sessions on restaurant information domain TRAINING: 1,612 sessions DEVELOPMENT: 506 sessions TEST: 1,117 sessions The results of ASR and SLU are annotated for every turn in the dataset, as well as the gold standard annotations are also provided for evaluation Models CRF (Conditional Random Fields): with sequential labeling ME (Maximum Entropy): without sequential labeling Features SLU Hypothesis: inform, confirm, deny, affirm, negate, request, reqalts System Action: expl-conf, impl-conf, request, select, canthelp Evaluation Metrics Features metrics: Accuracy, L2 norm, ROC CA 5 On Joint Goals, Method, and Requested Slots Experimental Results Comparisons of dialog state tracking performances Dev set Test set Acc L2 ROC Acc L2 ROC Joint Goals ME 0.638 0.551 0.144 0.596 0.671 0.036 CRF 0.644 0.545 0.103 0.601 0.649 0.064 Method ME 0.839 0.260 0.398 0.877 0.204 0.397 CRF 0.875 0.202 0.181 0.904 0.155 0.187 Requested Slots ME 0.946 0.099 0.000 0.957 0.081 0.000 CRF 0.942 0.107 0.000 0.960 0.073 0.000 CRF models produced better joint goals and method in accuracy and L2 norm on both development and test sets For the requested slots task, our proposed approach achieved better results than the baseline on the test set Conclusion This paper presented a sequential labeling approach for dialog state tracking Experimental results show the merits of our proposed approach with the improved performances on all the sub-tasks of DSTC 2 If we discover more advanced features that help to track the proper dialog states, they can raise the overall performances further 1 Fusionopolis Way, #21-01 Connexis (South Tower), Singapore 138632 Email: kims@i2r.a-star.edu.sg WWW: http://hlt.i2r.a-star.edu.sg/