Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
1 / 28
http://www.flickr.com/photos/joshmichtom/4311110421/
Guardian:
A Crowd-Powered Spoken Dialog
System for Web APIs
Ti...
2 / 28
What time is it?
It’s 9:30.
Kenneth’s apartment.
3 / 28
How was the Pirates
game last night?
!
Kenneth’s apartment.
4 / 28
How was the Steelers
game yesterday?
!
Kenneth’s apartment.
5 / 28
Is the movie Martian
still playing in theaters?
!
Kenneth’s apartment.
6 / 28
Use Web APIs to Empower
Dialog Systems
7 / 28
Gap between User & Machine
?
8 / 28
A Crowdsourcing Solution
9 / 28
Two Challenges
term
location
Hi, I’m in San Diego.
Any Chinese restaurants here?
Define Parameters
Extract Paramete...
10 / 28
How Do Dialog Systems Usually Do?
term
location
Hi, I’m in San Diego.
Any Chinese restaurants here?
Define Paramet...
11 / 28
Bridging this Gap is Expensive
• Define Parameters requires Experts
– Experts are expensive.
– Most services are n...
12 / 28
Can the Crowd Do It?
term
location
Hi, I’m in San Diego.
Any Chinese restaurants here?
Define Parameters
Extract P...
13 / 28
Define Parameters
term
location
Hi, I’m in San Diego.
Any Chinese restaurants here?
Define Parameters
Extract Para...
14 / 28
How machines understand a Web API?
1. Use which parameters ?
2. Ask user what questions
to elicit these parameters...
15 / 28
Parameter Rating Problem
offset
term
location
sw_latitude
sw_longitude
category_filter
accuracy
deals_filter
radiu...
16 / 28
How about just do a survey?
Task
Parameter Name / Desc
17 / 28
Baselines
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
MAP MRR
Not Unnatural
Ask Siri
Ask a Friend
Average results of 8...
18 / 28
Match Questions with Parameters
offset
I like Chinese food.
What do you want to eat?
? !
I’m in Pittsburgh.
Which ...
19 / 28
Evaluation on Parameter Ranking
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
MAP MRR
Question Matching
Not Unnatural
As...
20 / 28
Questions Collected Already!
1. Use which parameters ?
2. Ask user what questions
to elicit these parameters?
Yelp...
21 / 28
Extract Parameters
term
location
Define Parameters
Extract Parameters
Hi, I’m in San Diego.
Any Chinese restaurant...
22 / 28
Dialog ESP Game
Hi, I’m in San Diego.
Answer
Aggregate
Location =
San Diego
RecruitedPlayers
Time Constraint
23 / 28
Guardian: A Crowd-Powered Spoken
Dialog System for Web APIs
3
2 Call Web APIHi, I’m in San Diego.
Any Chinese rest...
24 / 28
Engineering Challenges
• Real-time Response ……..…..……… Retainer Model
• Converse with User ……………………………….. Chorus
•...
25 / 28
System Evaluation
Web API
Task
Find Chinese
restaurants in
Pittsburgh.
Check current
weather
by using a zip
code.
...
26 / 28
Guardian: A Hybrid Framework
Annotate Data on the Fly !
27 / 28
What’s next?
• More Automations
– Slot Filling / Entity Recognition
– Dialog Management
– Response Generation
• 1,...
28 / 28
Thank you!
http://www.flickr.com/photos/joshmichtom/4311110421/
Upcoming SlideShare
Loading in …5
×

Guardian: A Crowd-Powered Spoken Dialog System for Web APIs

974 views

Published on

Ting-Hao K. Huang, Walter S Lasecki, Jeffrey P Bigham. (2015). Guardian: A Crowd-Powered Spoken Dialog System for Web APIs. Conference on Human Computation & Crowdsourcing (HCOMP 2015), November, 2015, San Diego, USA.

Published in: Technology
  • Be the first to comment

Guardian: A Crowd-Powered Spoken Dialog System for Web APIs

  1. 1. 1 / 28 http://www.flickr.com/photos/joshmichtom/4311110421/ Guardian: A Crowd-Powered Spoken Dialog System for Web APIs Ting-Hao (Kenneth) Huang Carnegie Mellon University Walter S. Lasecki University of Michigan Jeffrey P. Bigham Carnegie Mellon University
  2. 2. 2 / 28 What time is it? It’s 9:30. Kenneth’s apartment.
  3. 3. 3 / 28 How was the Pirates game last night? ! Kenneth’s apartment.
  4. 4. 4 / 28 How was the Steelers game yesterday? ! Kenneth’s apartment.
  5. 5. 5 / 28 Is the movie Martian still playing in theaters? ! Kenneth’s apartment.
  6. 6. 6 / 28 Use Web APIs to Empower Dialog Systems
  7. 7. 7 / 28 Gap between User & Machine ?
  8. 8. 8 / 28 A Crowdsourcing Solution
  9. 9. 9 / 28 Two Challenges term location Hi, I’m in San Diego. Any Chinese restaurants here? Define Parameters Extract Parameters
  10. 10. 10 / 28 How Do Dialog Systems Usually Do? term location Hi, I’m in San Diego. Any Chinese restaurants here? Define Parameters Extract Parameters
  11. 11. 11 / 28 Bridging this Gap is Expensive • Define Parameters requires Experts – Experts are expensive. – Most services are not designed for dialog systems. – Unsupervised Slot Induction • Extract Parameters requires Data – (Which we don’t have.) – Supervised Slot Filling • Slot Filling / Entity Recognition – No labeled data • State Tracking – No dialogue data – Unsupervised Slot Filling
  12. 12. 12 / 28 Can the Crowd Do It? term location Hi, I’m in San Diego. Any Chinese restaurants here? Define Parameters Extract Parameters
  13. 13. 13 / 28 Define Parameters term location Hi, I’m in San Diego. Any Chinese restaurants here? Define Parameters Extract Parameters
  14. 14. 14 / 28 How machines understand a Web API? 1. Use which parameters ? 2. Ask user what questions to elicit these parameters? Yelp Search API 2.0 has 22 parameters.
  15. 15. 15 / 28 Parameter Rating Problem offset term location sw_latitude sw_longitude category_filter accuracy deals_filter radius_filter ... offset term location sw_latitude sw_longitude category_filter accuracy deals_filter radius_filter ... Pick good parameters for the dialog system.
  16. 16. 16 / 28 How about just do a survey? Task Parameter Name / Desc
  17. 17. 17 / 28 Baselines 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 MAP MRR Not Unnatural Ask Siri Ask a Friend Average results of 8 Web APIs’ parameters Results are not so good...
  18. 18. 18 / 28 Match Questions with Parameters offset I like Chinese food. What do you want to eat? ? ! I’m in Pittsburgh. Which city are you in? ? ! Dinner. Is it dinner or lunch? ? ! ... location ? ! term ? ! ! ? ! ? ! ? ! ? ! category_filter ? ! ? ! ? ! ? ! ? ! ? ! ? ! ? ! ? ! ? ! ? ! ? ! ? ! ? ! ? ! ? ! ? !? ! ? ! ? ! ? ! ? ! ? !? ! ? ! ? ! ? ! ? !? ! ? ! ? ! ? ! ? ! ? ! ? ! term location sw_latitude sw_longitude category_filter BetterParameter Yelp API Question Collection Parameter Filtering Qestion-Parameter Matching
  19. 19. 19 / 28 Evaluation on Parameter Ranking 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 MAP MRR Question Matching Not Unnatural Ask Siri Ask a Friend Question Matching outperforms all baselines. Average results of 8 Web APIs’ parameters
  20. 20. 20 / 28 Questions Collected Already! 1. Use which parameters ? 2. Ask user what questions to elicit these parameters? Yelp Search API 2.0 has 22 parameters.
  21. 21. 21 / 28 Extract Parameters term location Define Parameters Extract Parameters Hi, I’m in San Diego. Any Chinese restaurants here?
  22. 22. 22 / 28 Dialog ESP Game Hi, I’m in San Diego. Answer Aggregate Location = San Diego RecruitedPlayers Time Constraint
  23. 23. 23 / 28 Guardian: A Crowd-Powered Spoken Dialog System for Web APIs 3 2 Call Web APIHi, I’m in San Diego. Any Chinese restaurants here? 1 Talk and Extract Parameter Interpret Result to User Mandarin Wok Restaurant is good ! It’s on 4227 Balboa Ave. term = Chinese location = San Diego Yelp Search API 2.0 { ... "name": "Mandarin Wok Restaurant”,... "address":["4227 Balboa Ave”,...], …} JSON
  24. 24. 24 / 28 Engineering Challenges • Real-time Response ……..…..……… Retainer Model • Converse with User ……………………………….. Chorus • Speech Recognition ………………... Web Speech API • Parameter Extraction ………..…… Dialog ESP-Game • JSON Visualization ………………..….. JSON Visualizer • Response Generation Assistant ………………. jQuery • Workflow Control ………………. Game-like Interface • Dialog Management …………. Finite-state Machine • Crowdsourcing Platform …………. Mechanical Turk
  25. 25. 25 / 28 System Evaluation Web API Task Find Chinese restaurants in Pittsburgh. Check current weather by using a zip code. Find information of “Titanic”. Valid JSON 9 / 10 9 / 10 6 / 10 Task Completion 10 / 10 9 / 10 10 / 10 Domain Referenced TCR 0.96 0.94 0.88
  26. 26. 26 / 28 Guardian: A Hybrid Framework Annotate Data on the Fly !
  27. 27. 27 / 28 What’s next? • More Automations – Slot Filling / Entity Recognition – Dialog Management – Response Generation • 1,000+ APIs? • Future of Dialog Systems – What if you can really talk to a machine… – On wearable device?
  28. 28. 28 / 28 Thank you! http://www.flickr.com/photos/joshmichtom/4311110421/

×