Cooking an ontology-based spoken dialogue system

Cooking an ontology-based
coaching dialogue system
— based on my PhD proposal —

Joana Paulo Pardal
Quality and Usability Colloquium
December 7th, 2009
Berlin, Germany

Monday, December 7, 2009

Who am I?

Researcher, PhD Student and Lecturer
RESEARCHER:
- Spoken Language Systems Laboratory (L2F) @ INESC-ID, Lisbon

PHD STUDENT:
- IST, Technical University of Lisbon
- Ontology-based (spoken) Dialogue Systems

LECTURER:
- IST, Technical University of Lisbon


cooking a coaching
ontology-based
(spoken) dialogue systems


Cooking


House of the future


Part of our daily life


With a lot of inspiration


And interested people


Eyes and hands busy


Dialogue Systems

Typical Architecture

User

Speech Speech
Synthesis Recognition

Dialogue
Management
Natural Natural
Language Language
Generation Understanding

back-end
system


Typical Architecture

User
prosody acoustic
models
Speech Speech
coded
speech grammar

Dialogue
Management
lexicon lexicon
Natural Natural
Language Language
grammar
Generation Understanding grammar

back-end
system


DigA L2F INESC-ID
Dialogue Manager

Device Core
IO Manager
Core
ASR Business
Parser

Service Manager
JVoiceXML & JSpeechAPI 2.0 interface
(Audimus)

Interpretation

Dialogue Manager Interface
Microphone
Manager
Array

Task Manager

Device Core
Behavioral
Agent Data
GUI base
Generation
Manager
User
TTS Surface Access
(DIXI) Generation Manager

Device Core
Discourse Context Device QA@L2F
Face
Event Manager Manager


TRIPS Univ Rochester


Olympus Carnegie Mellon
#$%&'()*+,-&./0$12%02,/%
<%0=;9&:6;16%
!"$'%()&

0$1),"(#()" !"#$%&%$#'#()" !"#$%&'(&
)01&#.$(*$( !"#$%&'
)#*+,&

-.#"&/0 !"#$%'1#()"-.,% *('+),-.'"',$% 3405%67
1&234#5 )!#,,# ()*$%+,)-

23"#4$5(5 /$"$%'#()"
2),,&#!$ (#.$//)

8)62$9&:6;16%
!"#$%&

!"


Practical
dialogue

http://ﬂickr.com/photos/carpathian/1656738696/in/photostream/

Practical Dialogue Hypothesis
The conversational competence required
for practical dialogues, while still complex,
is significantly simpler to achieve than
general human conversational competence.

Domain-Independence Hypothesis
Within the genre of practical dialogue,
the bulk of the complexity in the language
interpretation and dialogue management
is independent of the task being performed.

Types of dialogue systems:
- Information providing
- Appliances control
- Planning & scheduling
- Tutoring
...


Tehran Times Sports Desk

Coaching

Simple but funcional
dialogue systems

How different levels of knowledge
stored in domain-specific ontologies
can be used to facilitate the creation
of new coaching dialogue systems
capable of domain reasoning?

http://www.ﬂickr.com/photos/jhhwild/1781000505/

short introduction to
ONTOLOGIES

Definition adapted from http://en.wiktionary.org/wiki/ontology

on·tol·o·gy
ORIGIN early 18th cent.:
from modern Latin ontologia,
from Greek ONT- ‘being’ -LOGY.

(philosophy)
the branch of metaphysics that addresses the nature or
essential characteristics of being and of things that exist;
the study of being in the capacity of being.

(computer science)
a structure of concepts or entities within a domain, organized by
relationships; a system model.


What is an Ontology?

is a description of the existing concepts and
relationships in a given domain

“An ontology is a specification of a conceptualization.”
[Gruber, 1993]


http://www.w3.org/2000/Talks/1206-xml2k-tbl/slide10-0.html

Tim Berners-Lee’s Semantic Web
Machine processable (not natural language),
for data, for the future.


machine processable
and
interoperable systems

http://ﬂickr.com/photos/ul_marga/755378645/

sharing knowledge

http://ﬂickr.com/photos/kgoldenchild/2223459035/

easier communication
http://ﬂickr.com/photos/hchalkley/92120879/

easier integration
http://ﬂickr.com/photos/stevehopson/1794895311

Data organization
when a large amount is available
http://ﬂickr.com/photos/shetha/1561376494/

Ontologies
in dialogue systems


Ontology in a dialog system

User
prosody acoustic
models
Speech Speech
coded
speech grammar

Dialogue
Management
lexicon lexicon
Natural Natural
Language Language
grammar
Generation Understanding grammar

back-end
system

Domain
Ontology


TRIPS Univ Rochester
!"#"$%&'' 01*$)2'2-."/3'' @5""&D''
@5""&D'' ()#*+)*"'' 4)3".'-#'
>"&-*#%E"$' @=#6D"3%3'
,-."/' &-$5+3'.)6)'
C-$.3'
C-$.3'
!$)22)$'
8$-5"$'
("7%&-#' 0-+#39' @+$F)&"''
:6&;' !"#"$)G-#'
8)$3"$'
<#6-/-*='
(B3' <#6-/-*=' ,)55%#*' (B3'
>+/"3'
J#6"$5$"6)G-#'
,)#)*"$!
H%3&-+$3"'' H-2)%#1@5"&%I&'
@6)6"' ?-25-#"#63'

?8@'A&63'


McGuinness’ spectrum


My work
so far...


OntoChef
a cooking ontology

joint work with R. Ribeiro, F. Batista, N. Mamede and P. Vaz

Main goal
Recipe: Chocolate Cake
Ingredients
4 oz of sweet dark chocolate

5 eggs

...

Procedure
separate egg whites and egg yolks
with an electric mixer whisk the egg
whites until forming stiff peaks
put egg whites in a container

...
video file

...


Competency Questions
Recipes:
- How do I make recipe R?
- What are the quantities to make recipe R for 4 persons?
Actions:
- How do I do A1?
Times:
- Which recipes that take less than 10 minutes to do?
Food:
- Which recipes have food item F1, but not F2?
- Which recipes have as main ingredient food item F1?
Utensils:
- Which utensils are used in recipe R1?
- Which recipes can be made using the microwave?
Equivalencies:
- How many liters is a cup?


OntoChef cooking domain model


several modules

Equivalences Actions
Food

Units & Recipes Utensils
Measures


Formalization overview


Resulting Ontology
4 main (reusable) modules:
- Food
- Utensils
- Actions
- Recipes
3 auxiliary (reusable) modules:
- Measures
- Equivalences
- Types of dishes
Metrics:
- 1151 classes
- 92 slots
- 52 relations between classes
- 311 instances


OntoChef building methodology


Main activities:
- Knowledge acquisition
- Conceptualization
- Formalization
- Evaluation


Food

Measures

separate teams for bulk work

weekly brainstorm sessions

hardest: reach agreement

OntoChef knowledge sources


professional websites
Web Site #Recipes #Photos
www.vaqueiro.pt 3136 2790

www.continente.pt 467 467

sabores.sapo.pt 2760 1038

comezainas.clix.pt 1646 0

www.entrepratos.com 189 51

Total: 8198 4346
Recipes collected from professional websites.


personal blogs
Web Site #Recipes
paladares.do.sapo.pt 329

receitasdeculinaria.blogs.sapo.pt 10

culinaria.no.sapo.pt 236

Total: 575
Recipes collected from personal blogs.


CMU’s Recipe Database
<recipe version="1.1">
<line>
<originaltext>1 (18 ounce) package small chocolate chip
cookies</originaltext>
<annotation>create_ing(ing0, "1 (18 ounce) package small
chocolate chip cookies")</annotation>
</line>
<line>
<originaltext>1 cup hot fudge topping</originaltext>
<annotation>create_ing(ing2, "1 cup hot fudge topping")</
annotation>
</line>
...
<line>
<originaltext>Spread 3/4 cup fudge topping over crust. </
originaltext>
<annotation>separate(ing2, ing12, "3/4 cup fudge", ing13,
"1/4 cup fudge", "")</annotation>
</line>
...


USENET Cookbook
.RH MOD.RECIPES-SOURCE CHOC-CAKE-1 D "20 May 86" 1986
.RZ "CHOCOLATE FANTASY CAKE" "A no-bake chocolate cake"
This is a very simple, quick, no-bake cake.
.IH "Serves 6-8"
.SH! ! CAKE
.IG "2(12 cups" "cream" "600 ml"
.IG "3 cups" "milk" "750 ml"
.IG "3 pkgs" "instant chocolate pudding"
.PH
.SK 1
Arrange a layer of biscuits at the bottom of a large baking
pan.
...
.SH RATING
.I Difficulty:
easy.
.I Time:
10 minutes preparation, several hours chilling.
.I Precision:
measure the ingredients.
.WR
Mike Trachtman, Weizmann Institute, Rehovot, Israel,

OntoChef conceptualization


OntoChef database population

joint work with T. Machado and N. Mamede

Manual input


Ontology-based
semi-automatic input
through natural language processing
OntoChef
Process

(cooking ontology)
Internet
Recipes
e-Books Database
Cookbooks (in text) (ontology intances)


OntoChef evaluation


OntoChef future work


additional languages

Knowledge sources
Book’s Cover Book’s Bibliographic Reference

English:
“Ingredients”, by Loukie Werle, Ullmann Publishing, 2009, 384 pp.

Portuguese:
“Ingredientes”, by Loukie Werle, Könemann, 2009, 384 pp.

English:
“The Cooks' Bible”, by Le Cordon Bleu, Carroll & Brown Publishers, 2005, 352 pp.

Portuguese:
“O Livro das Técnicas Culinárias”, by E. Treuille & J. Wright, Livros e Livros, 2006, 352 p.

“Dicionário Visual Português/Inglês”, Civilização Editora, 2006, 360 pgs.
“Dicionário Visual Português/Espanhol”, Civilização Editora, 2006, 360 pgs.
“Dicionário Visual Português/Françês”, Civilização Editora, 2006, 360 pgs.
“Dicionário Visual Português/Italiano”, Civilização Editora, 2006, 360 pgs.
“Dicionário Visual Português/Alemão”, Civilização Editora, 2006, 360 pgs.


Collected information
Languages # pairs PT-EN
Portuguese — English Ingredients 2112
Portuguese — English Actions 1658 5927
Portuguese — English 2157
Portuguese — Spanish 2157
Portuguese — French 2157
Portuguese — Italian 2157
Portuguese — German 2157
Total: 14555

wine information


Cooking Coach
an ontology-based
coaching dialogue systems


Recipe description


Available ‘tasks’
- next
- previous
- repeat
- how? (down)
- continue (up)


Cooking Corpus


Corpus annotation

Food

Measures


Cooking Coach
next steps...

Explore semantic spectrum
without automated Reasoning with automated Reasoning
Complexity

Catalog Terms / Term Formal Formal Frames Value Description General
Thesaurus
/ ID Glossary Hierarchy Taxonomy Instance (props) Restrictions Logics Logic


Reasoning

‘Caracóis’ (snails): broiled animal, a cake with snail shape


Tackle different users


Different coaching styles


System’s evaluation

a lot
Prior experience with
the Dialog System
some
none

none some a lot
Knowledge of the Domain
(Cooking prior experience)

MIT’s Counter


DFKI’s Smart Kitchen


Coaching another tasks


Questions

http://ﬂickr.com/photos/jazz_dalek/1845705965/

Thank you!

http://ﬂickr.com/photos/42304632@N00/232004616/

Joana Paulo Pardal

joana.paulo.pardal@gmail.com
ladydoctorbug.blogspot.com
www.l2f.inesc-id.pt ⁄!joana


Cooking an ontology-based
coaching dialogue system

Joana Paulo Pardal
Quality and Usability Colloquium
December 7th, 2009
Berlin, Germany


Cooking an ontology-based spoken dialogue system

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (19)

Similar to Cooking an ontology-based spoken dialogue system

Similar to Cooking an ontology-based spoken dialogue system (20)

More from Joana Paulo Pardal

More from Joana Paulo Pardal (6)

Recently uploaded

Recently uploaded (20)

Cooking an ontology-based spoken dialogue system