SlideShare a Scribd company logo
1 of 1
Download to read offline
Towards Improving Dialogue Topic Tracking Performances
with Wikification of Concept Mentions
Seokhwan Kim, Rafael E. Banchs, Haizhou Li
Human Language Technology Department, Institute for Infocomm Research (I2
R), Singapore
Dialogue Topic Tracking
Detecting topic transitions and predicting topic categories
In on-going dialogues which address more than a single topic
Classification problem f(xt) = (yt−1, yt)
Examples of dialogue topic tracking
Speaker Utterance Topic Transition
Guide How can I help you? NONE→NONE
Tourist Can you recommend some good places to visit in Sin-
gapore?
NONE→ATTR
Guide Well if you like to visit an icon of Singapore, Merlion
park will be a nice place to visit.
Tourist That is a symbol for your country, right? ATTR→ATTR
Guide Yes, we use that to symbolise Singapore.
Tourist Okay. ATTR→ATTR
Guide The lion head symbolised the founding of the island and
the fish body just symbolised the humble fishing village.
Tourist How can I get there from Orchard Road? ATTR→TRSP
Guide You can take the red line train from Orchard and stop
at Raffles Place.
Tourist Is this walking distance from the station to the destina-
tion?
TRSP→TRSP
Guide Yes, it’ll take only ten minutes on foot.
Tourist Alright. TRSP→FOOD
Guide Well, you can also enjoy some seafoods at the riverside
near the place.
Tourist What food do you have any recommendations to try
there?
FOOD→FOOD
Guide If you like spicy foods, you must try chilli crab which is
one of our favourite dishes here.
Tourist Great! I’ll try that. FOOD→FOOD
Wikipedia-based Dialogue Topic Tracking
To leverage on Wikipedia as an external knowledge source
Obtained without significant effort toward building resources
Kernel methods based on dialogue segment-level correspondences to
Wikipedia articles obtained with vector space models
Kim et al. (ICASSP 2014, ACL 2014)
Limitations
Mismatches between spoken dialogues and written encyclopedia
Multiple relevances from a single dialogue segment to more than one concept
Lack of semantic or discourse aspects in concept retrieval
Wikification of Concept Mentions in Spoken Dialogues
Wikification aims at linking mentions to the relevant entries in Wikipedia
Examples of Wikification results on the previous example
Singapore, Merlion Park, Orchard Road, North South MRT Line, Raffles
Place MRT Station Singapore River, Chilli crab
Dealing with co-references, ambiguities, and variabilities of mentions
Every noun phrase is defined as a single mention
For each mention, candidates are retrieved from a Lucene index
Ranking SVM: Supervised learning to rank algorithm
s(m, c) =



4 if c is the exactly same as g(m),
3 if c is the parent article of g(m),
2 if c belongs to the same article
but different section of g(m),
1 otherwise.
m: a mention
c: a candidate concept
g(m): the manual annotation for the most relevant concept of m
The top-ranked item in the list of candidates scored by this model is
considered as the result of Wikification for a given mention
Wikification-based Features for Topic Tracking
Baseline based on vector space model
φ(x) = α1, α2, · · · , α|W| ∈ R|W|
αi =
h
j=0
λj
· tfidf(wi, u(t−j))
Wikipedia-based features (ICASSP 2014, ACL 2014)
φ (x) = α1, α2, · · · , α|W|, β1, β2, · · · , β|D|
βi = sim(x, ci) = cos (θ) =
φ(x) · φ(ci)
|φ(x)||φ(ci)|
Wikification-based features
γi =
h
j=0
λj
· mk ∈ u(t−j)|g(mk) = ci
Experimental Setup
Singapore tour guide dialogues
Human-human mixed initiative dialogues
35 sessions, 21 hours, 31,034 utterances
Manually annotated with nine topic categories
Wikipedia collection
4,797,927 articles and 25,577,464 sections in total
Collected from Wikipedia database dump as of January 2015
Indexed into a Lucene index
Wikification of Concept Mentions
Kim et al. (EMNLP 2015)
Preprocessed by Stanford CoreNLP toolkit
For every noun phrase, top 100 candidates are retrieved from Lucene index
SVMrank
was used to train a ranking function
Wikification Performances
Five-fold cross validation
38.04/31.97/34.74 in Precision/Recall/F-measure
Dialogue Topic Tracking
SVM models were tranined using SVMlight
3 schedules were considered: every turns, only tourist turns, and only guide turns
All the evaluations were done in five-fold cross validation
Evaluation metrics
Accuracy of the predicted topic label for every turn
Precision/Recall/F-measure for each event of topic transition occurred
Experimental Results
Schedule Features
Transition Turn
P R F ACC
all
α 42.08 53.48 47.10 67.97
α + β 42.12 53.38 47.08 67.98
α + γ 47.36 50.19 48.73 72.38
α + β + γ 47.35 50.24 48.75 72.43
α + γ 50.77 49.36 50.06 79.12
α + β + γ 50.82 49.41 50.10 79.15
tourist turns
α 41.88 52.59 46.63 67.15
α + β 41.84 52.75 46.67 67.08
α + γ 46.58 51.09 48.73 71.99
α + β + γ 46.57 51.09 48.72 71.99
α + γ 50.51 49.58 50.04 81.10
α + β + γ 50.43 49.58 50.00 81.10
guide turns
α 41.96 52.11 46.49 67.13
α + β 41.91 52.03 46.42 67.13
α + γ 47.10 48.44 47.76 71.94
α + β + γ 47.02 48.21 47.61 71.93
α + γ 50.94 49.10 50.00 78.92
α + β + γ 50.98 49.02 49.98 78.92
γ is the oracle features with the manual annotations for Wikification
1 Fusionopolis Way, #21-01 Connexis (South Tower), Singapore 138632 Email: kims@i2r.a-star.edu.sg

More Related Content

Similar to Towards Improving Dialogue Topic Tracking Performances with Wikification of Concept Mentions

Wikification of Concept Mentions within Spoken Dialogues Using Domain Constra...
Wikification of Concept Mentions within Spoken Dialogues Using Domain Constra...Wikification of Concept Mentions within Spoken Dialogues Using Domain Constra...
Wikification of Concept Mentions within Spoken Dialogues Using Domain Constra...Seokhwan Kim
 
[slide] A Compare-Aggregate Model with Latent Clustering for Answer Selection
[slide] A Compare-Aggregate Model with Latent Clustering for Answer Selection[slide] A Compare-Aggregate Model with Latent Clustering for Answer Selection
[slide] A Compare-Aggregate Model with Latent Clustering for Answer SelectionSeoul National University
 
NLP Project: Paragraph Topic Classification
NLP Project: Paragraph Topic ClassificationNLP Project: Paragraph Topic Classification
NLP Project: Paragraph Topic ClassificationEugene Nho
 
Learning from meaningful, purposive interaction
Learning from meaningful, purposive interactionLearning from meaningful, purposive interaction
Learning from meaningful, purposive interactionfridolin.wild
 
Information extraction for Free Text
Information extraction for Free TextInformation extraction for Free Text
Information extraction for Free Textbutest
 
Interpreting asia europe-handbook - 2005+content page
Interpreting asia europe-handbook - 2005+content pageInterpreting asia europe-handbook - 2005+content page
Interpreting asia europe-handbook - 2005+content pagetdbt_123
 
transfer.pptx
transfer.pptxtransfer.pptx
transfer.pptxHaibinSu2
 
Harnessing Linked Knowledge Sources for Topic Classification in Social Media
Harnessing Linked Knowledge Sources for Topic Classification in Social MediaHarnessing Linked Knowledge Sources for Topic Classification in Social Media
Harnessing Linked Knowledge Sources for Topic Classification in Social MediaAmparo Elizabeth Cano Basave
 
[Paper Introduction] Translating into Morphologically Rich Languages with Syn...
[Paper Introduction] Translating into Morphologically Rich Languages with Syn...[Paper Introduction] Translating into Morphologically Rich Languages with Syn...
[Paper Introduction] Translating into Morphologically Rich Languages with Syn...NAIST Machine Translation Study Group
 
Moore_slides.ppt
Moore_slides.pptMoore_slides.ppt
Moore_slides.pptbutest
 
Writing a scientific manuscript
Writing a scientific manuscriptWriting a scientific manuscript
Writing a scientific manuscriptMartin McMorrow
 
Analyzing MUVE Tasks in Action - XVIIth International CALL Research Conferenc...
Analyzing MUVE Tasks in Action - XVIIth International CALL Research Conferenc...Analyzing MUVE Tasks in Action - XVIIth International CALL Research Conferenc...
Analyzing MUVE Tasks in Action - XVIIth International CALL Research Conferenc...Cristina Palomeque
 
AsiaCALL 2017 presentation
AsiaCALL 2017 presentationAsiaCALL 2017 presentation
AsiaCALL 2017 presentationTakeshi Sato
 
(Final) cidoc 2009 chinese lang translation of the aat
(Final) cidoc 2009 chinese lang translation of the aat(Final) cidoc 2009 chinese lang translation of the aat
(Final) cidoc 2009 chinese lang translation of the aatAAT Taiwan
 
(Final) cidoc 2009 chinese lang translation of the aat
(Final) cidoc 2009 chinese lang translation of the aat(Final) cidoc 2009 chinese lang translation of the aat
(Final) cidoc 2009 chinese lang translation of the aatAAT Taiwan
 
The Ins and Outs of Preposition Semantics:
 Challenges in Comprehensive Corpu...
The Ins and Outs of Preposition Semantics:
 Challenges in Comprehensive Corpu...The Ins and Outs of Preposition Semantics:
 Challenges in Comprehensive Corpu...
The Ins and Outs of Preposition Semantics:
 Challenges in Comprehensive Corpu...Seth Grimes
 
Preposition Semantics: Challenges in Comprehensive Corpus Annotation and Auto...
Preposition Semantics: Challenges in Comprehensive Corpus Annotation and Auto...Preposition Semantics: Challenges in Comprehensive Corpus Annotation and Auto...
Preposition Semantics: Challenges in Comprehensive Corpus Annotation and Auto...Seth Grimes
 
EvoPat - Pattern-Based Evolution and Refactoring of RDF Knowledge Bases
EvoPat - Pattern-Based Evolution and Refactoring of RDF Knowledge BasesEvoPat - Pattern-Based Evolution and Refactoring of RDF Knowledge Bases
EvoPat - Pattern-Based Evolution and Refactoring of RDF Knowledge BasesSebastian Tramp
 

Similar to Towards Improving Dialogue Topic Tracking Performances with Wikification of Concept Mentions (20)

Wikification of Concept Mentions within Spoken Dialogues Using Domain Constra...
Wikification of Concept Mentions within Spoken Dialogues Using Domain Constra...Wikification of Concept Mentions within Spoken Dialogues Using Domain Constra...
Wikification of Concept Mentions within Spoken Dialogues Using Domain Constra...
 
[slide] A Compare-Aggregate Model with Latent Clustering for Answer Selection
[slide] A Compare-Aggregate Model with Latent Clustering for Answer Selection[slide] A Compare-Aggregate Model with Latent Clustering for Answer Selection
[slide] A Compare-Aggregate Model with Latent Clustering for Answer Selection
 
NLP Project: Paragraph Topic Classification
NLP Project: Paragraph Topic ClassificationNLP Project: Paragraph Topic Classification
NLP Project: Paragraph Topic Classification
 
Learning from meaningful, purposive interaction
Learning from meaningful, purposive interactionLearning from meaningful, purposive interaction
Learning from meaningful, purposive interaction
 
Information extraction for Free Text
Information extraction for Free TextInformation extraction for Free Text
Information extraction for Free Text
 
Chounta@paws
Chounta@pawsChounta@paws
Chounta@paws
 
Interpreting asia europe-handbook - 2005+content page
Interpreting asia europe-handbook - 2005+content pageInterpreting asia europe-handbook - 2005+content page
Interpreting asia europe-handbook - 2005+content page
 
transfer.pptx
transfer.pptxtransfer.pptx
transfer.pptx
 
Harnessing Linked Knowledge Sources for Topic Classification in Social Media
Harnessing Linked Knowledge Sources for Topic Classification in Social MediaHarnessing Linked Knowledge Sources for Topic Classification in Social Media
Harnessing Linked Knowledge Sources for Topic Classification in Social Media
 
[Paper Introduction] Translating into Morphologically Rich Languages with Syn...
[Paper Introduction] Translating into Morphologically Rich Languages with Syn...[Paper Introduction] Translating into Morphologically Rich Languages with Syn...
[Paper Introduction] Translating into Morphologically Rich Languages with Syn...
 
Moore_slides.ppt
Moore_slides.pptMoore_slides.ppt
Moore_slides.ppt
 
Writing a scientific manuscript
Writing a scientific manuscriptWriting a scientific manuscript
Writing a scientific manuscript
 
Analyzing MUVE Tasks in Action - XVIIth International CALL Research Conferenc...
Analyzing MUVE Tasks in Action - XVIIth International CALL Research Conferenc...Analyzing MUVE Tasks in Action - XVIIth International CALL Research Conferenc...
Analyzing MUVE Tasks in Action - XVIIth International CALL Research Conferenc...
 
AsiaCALL 2017 presentation
AsiaCALL 2017 presentationAsiaCALL 2017 presentation
AsiaCALL 2017 presentation
 
(Final) cidoc 2009 chinese lang translation of the aat
(Final) cidoc 2009 chinese lang translation of the aat(Final) cidoc 2009 chinese lang translation of the aat
(Final) cidoc 2009 chinese lang translation of the aat
 
(Final) cidoc 2009 chinese lang translation of the aat
(Final) cidoc 2009 chinese lang translation of the aat(Final) cidoc 2009 chinese lang translation of the aat
(Final) cidoc 2009 chinese lang translation of the aat
 
The Ins and Outs of Preposition Semantics:
 Challenges in Comprehensive Corpu...
The Ins and Outs of Preposition Semantics:
 Challenges in Comprehensive Corpu...The Ins and Outs of Preposition Semantics:
 Challenges in Comprehensive Corpu...
The Ins and Outs of Preposition Semantics:
 Challenges in Comprehensive Corpu...
 
Preposition Semantics: Challenges in Comprehensive Corpus Annotation and Auto...
Preposition Semantics: Challenges in Comprehensive Corpus Annotation and Auto...Preposition Semantics: Challenges in Comprehensive Corpus Annotation and Auto...
Preposition Semantics: Challenges in Comprehensive Corpus Annotation and Auto...
 
Chapter -7.pptx
Chapter -7.pptxChapter -7.pptx
Chapter -7.pptx
 
EvoPat - Pattern-Based Evolution and Refactoring of RDF Knowledge Bases
EvoPat - Pattern-Based Evolution and Refactoring of RDF Knowledge BasesEvoPat - Pattern-Based Evolution and Refactoring of RDF Knowledge Bases
EvoPat - Pattern-Based Evolution and Refactoring of RDF Knowledge Bases
 

More from Seokhwan Kim

The Eighth Dialog System Technology Challenge (DSTC8)
The Eighth Dialog System Technology Challenge (DSTC8)The Eighth Dialog System Technology Challenge (DSTC8)
The Eighth Dialog System Technology Challenge (DSTC8)Seokhwan Kim
 
Deep Recurrent Neural Networks with Layer-wise Multi-head Attentions for Punc...
Deep Recurrent Neural Networks with Layer-wise Multi-head Attentions for Punc...Deep Recurrent Neural Networks with Layer-wise Multi-head Attentions for Punc...
Deep Recurrent Neural Networks with Layer-wise Multi-head Attentions for Punc...Seokhwan Kim
 
Dynamic Memory Networks for Dialogue Topic Tracking
Dynamic Memory Networks for Dialogue Topic TrackingDynamic Memory Networks for Dialogue Topic Tracking
Dynamic Memory Networks for Dialogue Topic TrackingSeokhwan Kim
 
The Fifth Dialog State Tracking Challenge (DSTC5)
The Fifth Dialog State Tracking Challenge (DSTC5)The Fifth Dialog State Tracking Challenge (DSTC5)
The Fifth Dialog State Tracking Challenge (DSTC5)Seokhwan Kim
 
Exploring Convolutional and Recurrent Neural Networks in Sequential Labelling...
Exploring Convolutional and Recurrent Neural Networks in Sequential Labelling...Exploring Convolutional and Recurrent Neural Networks in Sequential Labelling...
Exploring Convolutional and Recurrent Neural Networks in Sequential Labelling...Seokhwan Kim
 
The Fourth Dialog State Tracking Challenge (DSTC4)
The Fourth Dialog State Tracking Challenge (DSTC4)The Fourth Dialog State Tracking Challenge (DSTC4)
The Fourth Dialog State Tracking Challenge (DSTC4)Seokhwan Kim
 
Sequential Labeling for Tracking Dynamic Dialog States
Sequential Labeling for Tracking Dynamic Dialog StatesSequential Labeling for Tracking Dynamic Dialog States
Sequential Labeling for Tracking Dynamic Dialog StatesSeokhwan Kim
 
Wikipedia-based Kernels for Dialogue Topic Tracking
Wikipedia-based Kernels for Dialogue Topic TrackingWikipedia-based Kernels for Dialogue Topic Tracking
Wikipedia-based Kernels for Dialogue Topic TrackingSeokhwan Kim
 
A Graph-based Cross-lingual Projection Approach for Spoken Language Understan...
A Graph-based Cross-lingual Projection Approach for Spoken Language Understan...A Graph-based Cross-lingual Projection Approach for Spoken Language Understan...
A Graph-based Cross-lingual Projection Approach for Spoken Language Understan...Seokhwan Kim
 
A Graph-based Cross-lingual Projection Approach for Weakly Supervised Relatio...
A Graph-based Cross-lingual Projection Approach for Weakly Supervised Relatio...A Graph-based Cross-lingual Projection Approach for Weakly Supervised Relatio...
A Graph-based Cross-lingual Projection Approach for Weakly Supervised Relatio...Seokhwan Kim
 
MMR-based active machine learning for Bio named entity recognition
MMR-based active machine learning for Bio named entity recognitionMMR-based active machine learning for Bio named entity recognition
MMR-based active machine learning for Bio named entity recognitionSeokhwan Kim
 
A semi-supervised method for efficient construction of statistical spoken lan...
A semi-supervised method for efficient construction of statistical spoken lan...A semi-supervised method for efficient construction of statistical spoken lan...
A semi-supervised method for efficient construction of statistical spoken lan...Seokhwan Kim
 
A spoken dialog system for electronic program guide information access
A spoken dialog system for electronic program guide information accessA spoken dialog system for electronic program guide information access
A spoken dialog system for electronic program guide information accessSeokhwan Kim
 
An alignment-based approach to semi-supervised relation extraction including ...
An alignment-based approach to semi-supervised relation extraction including ...An alignment-based approach to semi-supervised relation extraction including ...
An alignment-based approach to semi-supervised relation extraction including ...Seokhwan Kim
 
An Alignment-based Pattern Representation Model for Information Extraction
An Alignment-based Pattern Representation Model for Information ExtractionAn Alignment-based Pattern Representation Model for Information Extraction
An Alignment-based Pattern Representation Model for Information ExtractionSeokhwan Kim
 
EPG 정보 검색을 위한 예제 기반 자연어 대화 시스템
EPG 정보 검색을 위한 예제 기반 자연어 대화 시스템EPG 정보 검색을 위한 예제 기반 자연어 대화 시스템
EPG 정보 검색을 위한 예제 기반 자연어 대화 시스템Seokhwan Kim
 
A Cross-Lingual Annotation Projection Approach for Relation Detection
A Cross-Lingual Annotation Projection Approach for Relation DetectionA Cross-Lingual Annotation Projection Approach for Relation Detection
A Cross-Lingual Annotation Projection Approach for Relation DetectionSeokhwan Kim
 
A Cross-lingual Annotation Projection-based Self-supervision Approach for Ope...
A Cross-lingual Annotation Projection-based Self-supervision Approach for Ope...A Cross-lingual Annotation Projection-based Self-supervision Approach for Ope...
A Cross-lingual Annotation Projection-based Self-supervision Approach for Ope...Seokhwan Kim
 

More from Seokhwan Kim (18)

The Eighth Dialog System Technology Challenge (DSTC8)
The Eighth Dialog System Technology Challenge (DSTC8)The Eighth Dialog System Technology Challenge (DSTC8)
The Eighth Dialog System Technology Challenge (DSTC8)
 
Deep Recurrent Neural Networks with Layer-wise Multi-head Attentions for Punc...
Deep Recurrent Neural Networks with Layer-wise Multi-head Attentions for Punc...Deep Recurrent Neural Networks with Layer-wise Multi-head Attentions for Punc...
Deep Recurrent Neural Networks with Layer-wise Multi-head Attentions for Punc...
 
Dynamic Memory Networks for Dialogue Topic Tracking
Dynamic Memory Networks for Dialogue Topic TrackingDynamic Memory Networks for Dialogue Topic Tracking
Dynamic Memory Networks for Dialogue Topic Tracking
 
The Fifth Dialog State Tracking Challenge (DSTC5)
The Fifth Dialog State Tracking Challenge (DSTC5)The Fifth Dialog State Tracking Challenge (DSTC5)
The Fifth Dialog State Tracking Challenge (DSTC5)
 
Exploring Convolutional and Recurrent Neural Networks in Sequential Labelling...
Exploring Convolutional and Recurrent Neural Networks in Sequential Labelling...Exploring Convolutional and Recurrent Neural Networks in Sequential Labelling...
Exploring Convolutional and Recurrent Neural Networks in Sequential Labelling...
 
The Fourth Dialog State Tracking Challenge (DSTC4)
The Fourth Dialog State Tracking Challenge (DSTC4)The Fourth Dialog State Tracking Challenge (DSTC4)
The Fourth Dialog State Tracking Challenge (DSTC4)
 
Sequential Labeling for Tracking Dynamic Dialog States
Sequential Labeling for Tracking Dynamic Dialog StatesSequential Labeling for Tracking Dynamic Dialog States
Sequential Labeling for Tracking Dynamic Dialog States
 
Wikipedia-based Kernels for Dialogue Topic Tracking
Wikipedia-based Kernels for Dialogue Topic TrackingWikipedia-based Kernels for Dialogue Topic Tracking
Wikipedia-based Kernels for Dialogue Topic Tracking
 
A Graph-based Cross-lingual Projection Approach for Spoken Language Understan...
A Graph-based Cross-lingual Projection Approach for Spoken Language Understan...A Graph-based Cross-lingual Projection Approach for Spoken Language Understan...
A Graph-based Cross-lingual Projection Approach for Spoken Language Understan...
 
A Graph-based Cross-lingual Projection Approach for Weakly Supervised Relatio...
A Graph-based Cross-lingual Projection Approach for Weakly Supervised Relatio...A Graph-based Cross-lingual Projection Approach for Weakly Supervised Relatio...
A Graph-based Cross-lingual Projection Approach for Weakly Supervised Relatio...
 
MMR-based active machine learning for Bio named entity recognition
MMR-based active machine learning for Bio named entity recognitionMMR-based active machine learning for Bio named entity recognition
MMR-based active machine learning for Bio named entity recognition
 
A semi-supervised method for efficient construction of statistical spoken lan...
A semi-supervised method for efficient construction of statistical spoken lan...A semi-supervised method for efficient construction of statistical spoken lan...
A semi-supervised method for efficient construction of statistical spoken lan...
 
A spoken dialog system for electronic program guide information access
A spoken dialog system for electronic program guide information accessA spoken dialog system for electronic program guide information access
A spoken dialog system for electronic program guide information access
 
An alignment-based approach to semi-supervised relation extraction including ...
An alignment-based approach to semi-supervised relation extraction including ...An alignment-based approach to semi-supervised relation extraction including ...
An alignment-based approach to semi-supervised relation extraction including ...
 
An Alignment-based Pattern Representation Model for Information Extraction
An Alignment-based Pattern Representation Model for Information ExtractionAn Alignment-based Pattern Representation Model for Information Extraction
An Alignment-based Pattern Representation Model for Information Extraction
 
EPG 정보 검색을 위한 예제 기반 자연어 대화 시스템
EPG 정보 검색을 위한 예제 기반 자연어 대화 시스템EPG 정보 검색을 위한 예제 기반 자연어 대화 시스템
EPG 정보 검색을 위한 예제 기반 자연어 대화 시스템
 
A Cross-Lingual Annotation Projection Approach for Relation Detection
A Cross-Lingual Annotation Projection Approach for Relation DetectionA Cross-Lingual Annotation Projection Approach for Relation Detection
A Cross-Lingual Annotation Projection Approach for Relation Detection
 
A Cross-lingual Annotation Projection-based Self-supervision Approach for Ope...
A Cross-lingual Annotation Projection-based Self-supervision Approach for Ope...A Cross-lingual Annotation Projection-based Self-supervision Approach for Ope...
A Cross-lingual Annotation Projection-based Self-supervision Approach for Ope...
 

Recently uploaded

Scootsy Overview Deck - Pan City Delivery
Scootsy Overview Deck - Pan City DeliveryScootsy Overview Deck - Pan City Delivery
Scootsy Overview Deck - Pan City Deliveryrishi338139
 
Understanding Post Production changes (PPC) in Clinical Data Management (CDM)...
Understanding Post Production changes (PPC) in Clinical Data Management (CDM)...Understanding Post Production changes (PPC) in Clinical Data Management (CDM)...
Understanding Post Production changes (PPC) in Clinical Data Management (CDM)...soumyapottola
 
Sunlight Spectacle 2024 Practical Action Launch Event 2024-04-08
Sunlight Spectacle 2024 Practical Action Launch Event 2024-04-08Sunlight Spectacle 2024 Practical Action Launch Event 2024-04-08
Sunlight Spectacle 2024 Practical Action Launch Event 2024-04-08LloydHelferty
 
Basic overview of nerve conduction studies
Basic overview of nerve conduction studiesBasic overview of nerve conduction studies
Basic overview of nerve conduction studiesDrAbdulAli1
 
Principles of Management Touchstone 4 Template APPLE INC.ppt.pptx
Principles of Management Touchstone 4 Template APPLE INC.ppt.pptxPrinciples of Management Touchstone 4 Template APPLE INC.ppt.pptx
Principles of Management Touchstone 4 Template APPLE INC.ppt.pptxvirginiagaddafi
 
Testing with Fewer Resources: Toward Adaptive Approaches for Cost-effective ...
Testing with Fewer Resources:  Toward Adaptive Approaches for Cost-effective ...Testing with Fewer Resources:  Toward Adaptive Approaches for Cost-effective ...
Testing with Fewer Resources: Toward Adaptive Approaches for Cost-effective ...Sebastiano Panichella
 
05.02 MMC - Assignment 4 - Image Attribution Lovepreet.pptx
05.02 MMC - Assignment 4 - Image Attribution Lovepreet.pptx05.02 MMC - Assignment 4 - Image Attribution Lovepreet.pptx
05.02 MMC - Assignment 4 - Image Attribution Lovepreet.pptxerickamwana1
 
Personal Branding Lecture for Advanced Digital & Social Media Strategy at UCL...
Personal Branding Lecture for Advanced Digital & Social Media Strategy at UCL...Personal Branding Lecture for Advanced Digital & Social Media Strategy at UCL...
Personal Branding Lecture for Advanced Digital & Social Media Strategy at UCL...Valters Lauzums
 

Recently uploaded (8)

Scootsy Overview Deck - Pan City Delivery
Scootsy Overview Deck - Pan City DeliveryScootsy Overview Deck - Pan City Delivery
Scootsy Overview Deck - Pan City Delivery
 
Understanding Post Production changes (PPC) in Clinical Data Management (CDM)...
Understanding Post Production changes (PPC) in Clinical Data Management (CDM)...Understanding Post Production changes (PPC) in Clinical Data Management (CDM)...
Understanding Post Production changes (PPC) in Clinical Data Management (CDM)...
 
Sunlight Spectacle 2024 Practical Action Launch Event 2024-04-08
Sunlight Spectacle 2024 Practical Action Launch Event 2024-04-08Sunlight Spectacle 2024 Practical Action Launch Event 2024-04-08
Sunlight Spectacle 2024 Practical Action Launch Event 2024-04-08
 
Basic overview of nerve conduction studies
Basic overview of nerve conduction studiesBasic overview of nerve conduction studies
Basic overview of nerve conduction studies
 
Principles of Management Touchstone 4 Template APPLE INC.ppt.pptx
Principles of Management Touchstone 4 Template APPLE INC.ppt.pptxPrinciples of Management Touchstone 4 Template APPLE INC.ppt.pptx
Principles of Management Touchstone 4 Template APPLE INC.ppt.pptx
 
Testing with Fewer Resources: Toward Adaptive Approaches for Cost-effective ...
Testing with Fewer Resources:  Toward Adaptive Approaches for Cost-effective ...Testing with Fewer Resources:  Toward Adaptive Approaches for Cost-effective ...
Testing with Fewer Resources: Toward Adaptive Approaches for Cost-effective ...
 
05.02 MMC - Assignment 4 - Image Attribution Lovepreet.pptx
05.02 MMC - Assignment 4 - Image Attribution Lovepreet.pptx05.02 MMC - Assignment 4 - Image Attribution Lovepreet.pptx
05.02 MMC - Assignment 4 - Image Attribution Lovepreet.pptx
 
Personal Branding Lecture for Advanced Digital & Social Media Strategy at UCL...
Personal Branding Lecture for Advanced Digital & Social Media Strategy at UCL...Personal Branding Lecture for Advanced Digital & Social Media Strategy at UCL...
Personal Branding Lecture for Advanced Digital & Social Media Strategy at UCL...
 

Towards Improving Dialogue Topic Tracking Performances with Wikification of Concept Mentions

  • 1. Towards Improving Dialogue Topic Tracking Performances with Wikification of Concept Mentions Seokhwan Kim, Rafael E. Banchs, Haizhou Li Human Language Technology Department, Institute for Infocomm Research (I2 R), Singapore Dialogue Topic Tracking Detecting topic transitions and predicting topic categories In on-going dialogues which address more than a single topic Classification problem f(xt) = (yt−1, yt) Examples of dialogue topic tracking Speaker Utterance Topic Transition Guide How can I help you? NONE→NONE Tourist Can you recommend some good places to visit in Sin- gapore? NONE→ATTR Guide Well if you like to visit an icon of Singapore, Merlion park will be a nice place to visit. Tourist That is a symbol for your country, right? ATTR→ATTR Guide Yes, we use that to symbolise Singapore. Tourist Okay. ATTR→ATTR Guide The lion head symbolised the founding of the island and the fish body just symbolised the humble fishing village. Tourist How can I get there from Orchard Road? ATTR→TRSP Guide You can take the red line train from Orchard and stop at Raffles Place. Tourist Is this walking distance from the station to the destina- tion? TRSP→TRSP Guide Yes, it’ll take only ten minutes on foot. Tourist Alright. TRSP→FOOD Guide Well, you can also enjoy some seafoods at the riverside near the place. Tourist What food do you have any recommendations to try there? FOOD→FOOD Guide If you like spicy foods, you must try chilli crab which is one of our favourite dishes here. Tourist Great! I’ll try that. FOOD→FOOD Wikipedia-based Dialogue Topic Tracking To leverage on Wikipedia as an external knowledge source Obtained without significant effort toward building resources Kernel methods based on dialogue segment-level correspondences to Wikipedia articles obtained with vector space models Kim et al. (ICASSP 2014, ACL 2014) Limitations Mismatches between spoken dialogues and written encyclopedia Multiple relevances from a single dialogue segment to more than one concept Lack of semantic or discourse aspects in concept retrieval Wikification of Concept Mentions in Spoken Dialogues Wikification aims at linking mentions to the relevant entries in Wikipedia Examples of Wikification results on the previous example Singapore, Merlion Park, Orchard Road, North South MRT Line, Raffles Place MRT Station Singapore River, Chilli crab Dealing with co-references, ambiguities, and variabilities of mentions Every noun phrase is defined as a single mention For each mention, candidates are retrieved from a Lucene index Ranking SVM: Supervised learning to rank algorithm s(m, c) =    4 if c is the exactly same as g(m), 3 if c is the parent article of g(m), 2 if c belongs to the same article but different section of g(m), 1 otherwise. m: a mention c: a candidate concept g(m): the manual annotation for the most relevant concept of m The top-ranked item in the list of candidates scored by this model is considered as the result of Wikification for a given mention Wikification-based Features for Topic Tracking Baseline based on vector space model φ(x) = α1, α2, · · · , α|W| ∈ R|W| αi = h j=0 λj · tfidf(wi, u(t−j)) Wikipedia-based features (ICASSP 2014, ACL 2014) φ (x) = α1, α2, · · · , α|W|, β1, β2, · · · , β|D| βi = sim(x, ci) = cos (θ) = φ(x) · φ(ci) |φ(x)||φ(ci)| Wikification-based features γi = h j=0 λj · mk ∈ u(t−j)|g(mk) = ci Experimental Setup Singapore tour guide dialogues Human-human mixed initiative dialogues 35 sessions, 21 hours, 31,034 utterances Manually annotated with nine topic categories Wikipedia collection 4,797,927 articles and 25,577,464 sections in total Collected from Wikipedia database dump as of January 2015 Indexed into a Lucene index Wikification of Concept Mentions Kim et al. (EMNLP 2015) Preprocessed by Stanford CoreNLP toolkit For every noun phrase, top 100 candidates are retrieved from Lucene index SVMrank was used to train a ranking function Wikification Performances Five-fold cross validation 38.04/31.97/34.74 in Precision/Recall/F-measure Dialogue Topic Tracking SVM models were tranined using SVMlight 3 schedules were considered: every turns, only tourist turns, and only guide turns All the evaluations were done in five-fold cross validation Evaluation metrics Accuracy of the predicted topic label for every turn Precision/Recall/F-measure for each event of topic transition occurred Experimental Results Schedule Features Transition Turn P R F ACC all α 42.08 53.48 47.10 67.97 α + β 42.12 53.38 47.08 67.98 α + γ 47.36 50.19 48.73 72.38 α + β + γ 47.35 50.24 48.75 72.43 α + γ 50.77 49.36 50.06 79.12 α + β + γ 50.82 49.41 50.10 79.15 tourist turns α 41.88 52.59 46.63 67.15 α + β 41.84 52.75 46.67 67.08 α + γ 46.58 51.09 48.73 71.99 α + β + γ 46.57 51.09 48.72 71.99 α + γ 50.51 49.58 50.04 81.10 α + β + γ 50.43 49.58 50.00 81.10 guide turns α 41.96 52.11 46.49 67.13 α + β 41.91 52.03 46.42 67.13 α + γ 47.10 48.44 47.76 71.94 α + β + γ 47.02 48.21 47.61 71.93 α + γ 50.94 49.10 50.00 78.92 α + β + γ 50.98 49.02 49.98 78.92 γ is the oracle features with the manual annotations for Wikification 1 Fusionopolis Way, #21-01 Connexis (South Tower), Singapore 138632 Email: kims@i2r.a-star.edu.sg