Conversational Internet - Creating a natural language interface for web pages

4,725 views

Published on

Presenting a paper to the 10th International Cross-Disciplinary Conference on Web Accessibility (W4A 2013)

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
4,725
On SlideShare
0
From Embeds
0
Number of Embeds
97
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Conversational Internet - Creating a natural language interface for web pages

  1. 1. © 2009 IBM CorporationConversational Internet:A natural language interface for webpagesDale Lane – IBM United Kingdom14 May 2013dale.lane@uk.ibm.com
  2. 2. © 2012 IBM Corporation2Challenge
  3. 3. © 2012 IBM Corporation3Challenge
  4. 4. © 2012 IBM Corporation4Motivation
  5. 5. © 2012 IBM Corporation5Understanding the page : Identifying type
  6. 6. © 2012 IBM Corporation6Understanding the page : Identifying navigation options
  7. 7. © 2012 IBM Corporation7Understanding the page : Identifying calls-to-action
  8. 8. © 2012 IBM Corporation8Understanding the user : Retrieving information
  9. 9. © 2012 IBM Corporation9Understanding the user : Mouse actions
  10. 10. © 2012 IBM Corporation10Understanding the user : Mouse actions
  11. 11. © 2012 IBM Corporation11Understanding the user : Keyboard actions
  12. 12. © 2012 IBM Corporation12Understanding the user : Keyboard actions
  13. 13. © 2012 IBM Corporation13Understanding the user : Keyboard actions
  14. 14. © 2012 IBM Corporation14Conversational Internet Presenting an early-stage prototype being developed to explore the potential forquestion answering as an alternative approach to screen-readers for retrievinginformation from web pages Architecture and approach inspired by active area of research and development inquestion answering on a knowledge derived from a corpus of documents(Ferucci, Lally, Chu-Carroll, et al)http://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=6177717
  15. 15. © 2012 IBM Corporation15ImplementationbrowserextensionUIMALanguageWareJavaclient server
  16. 16. © 2012 IBM Corporation16Implementationnew request new requestresponsewithconv. idresponsefollow-uprequestrequestwithconv. id“what can I do?”“the options are...”“I want to do...”
  17. 17. © 2012 IBM Corporation17Implementation : Client Firefox extension Submits current state ofthe page to server foranalysis Training mode
  18. 18. © 2012 IBM Corporation18Implementation : Server Analyzing the page Processing user queries
  19. 19. © 2012 IBM Corporation19Implementation : Server : Understanding the page
  20. 20. © 2012 IBM Corporation20Implementation : Server : Understanding the pageWhat type of site is this? Machine learning classifiers Whitelists of known domains
  21. 21. © 2012 IBM Corporation21Implementation : Server : Understanding the pageWhat can we infer from markup used? Semantic tags ARIA CSS class names
  22. 22. © 2012 IBM Corporation22Implementation : Server : Understanding the pageWhat does the structure of the page layout suggest? Machine learning models for common page elements
  23. 23. © 2012 IBM Corporation23Implementation : Server : Understanding the pageWhat does the text of the page tell us? Natural Language Processing using LanguageWare to recognize common formsof call-to-action
  24. 24. © 2012 IBM Corporation24Implementation : Server : Responding to queries
  25. 25. © 2012 IBM Corporation25Implementation : Server : Responding to queriesInterpreting the query NLP rules created with LanguageWare to map to closest known command type WordNet to attempt matches using synonyms of unknown terms
  26. 26. © 2012 IBM Corporation26Implementation : Server : Responding to queriesExtracting the requested information Information requested from page extracted from serialized CAS created by firstpipeline, by retrieving sections with relevant annotations
  27. 27. © 2012 IBM Corporation27Implementation : Server : Responding to queriesPreparing a response Speech generated using Nuance NDev and streamed to client
  28. 28. © 2012 IBM Corporation28Future work / Limitations Use cases Usability testing RIA / AJAX sites
  29. 29. © 2012 IBM Corporation29 Paper submitted to W4A– http://goo.gl/3X2iv Overview presentation– http://youtu.be/uS6oquJdgbw Demonstration of the prototype– http://youtu.be/tSGyPCcO-bYDale Lanedale.lane@uk.ibm.com@dalelane

×