2<br />
imagine<br />
imagine when<br />
meets <br />Farm Helper<br />
with this<br />Latitude: 38° 57’36” N<br />Longitude: 95° 15’12” W<br />Date: 10-9-2007<br />Time: 1345h<br />
that is sent to <br />Sensor Data Resource<br />Structured Data Resource<br />Weather Resource<br />Agri DB<br />Soil Surv...
and<br />
Six billion brains <br />
imagination today <br />
impacts our experience tomorrow<br />
Computing For Human Experience<br />ASWC 2008 Keynote<br />Amit P. Sheth,<br />Lexis Nexis Eminent Scholar and <br />Direc...
Technology that fits right in<br />“The most profound technologies are those that disappear. They weave themselves into th...
(c) 2007 Thomas Gruber<br />15<br />But we are not talking about<br />Intelligent Design<br />Ubicomp: Mark Wisner and oth...
What is CHE? Beyond better human interaction<br />Focus in the past (egUbicomp): How humans interact with the system (comp...
Principals of CHE<br />Human is the master, system is the slave<br />Human sees minimal changes to normal behavior and act...
Learning from a number of exciting visions<br />18<br />
Evolution of the Web (and associated computing)<br />Web of people<br />   - social networks, user-created casual<br />   ...
Sensing, Observing, Perceptual, Semantic, Social, Experiential<br />CHE components and enablers<br />
Consider <br />that all objects, events and activities in the physical world have a counterpart in the Cyberworld(IoT)<br ...
appropriate reasoning and human/social interaction are available and applied, insights extracted (semantic web, social sem...
answers obtained/ decisions reached/communicated/applied</li></ul>21<br />
Paradigm shift …<br />Where humans act as sensors or observers<br />Around them is a network of sensors, computing and com...
Today’s Sensor Network Types<br />Inert, fixed sensors<br />Carried on moving objects<br />Vehicles, pedestrians (asthma r...
Today’s Network of Sensors<br />Are sensing, computing, transmitting <br />Are acting in concert <br />Sharing data <br />...
Machine sensing<br />25<br />
26<br />Semantic Sensor Web<br />
<swe:component name="time"><br />	<swe:Time definition="urn:ogc:def:phenomenon:time" uom="urn:ogc:def:unit:date-time"><br ...
Semantic Sensor ML – Adding Ontological Metadata<br />Domain<br />Ontology<br />Person<br />Company<br />Spatial<br />Onto...
29<br />Semantic Query<br />Semantic Temporal Query<br />Model-references from SML to OWL-Time ontology concepts provides ...
Citizen Sensors<br />Human beings<br />6 billion intelligent sensors<br />informed observers<br />rich local knowledge<br ...
Citizen Science<br />Networks of amateur observers<br />possibly trained, skilled<br />Christmas Bird Count http://www.aud...
Citizen Sensor – Humans Actively Engage<br />In connecting, searching, processing, stitching together information<br />Ask...
Recent example - #Mumbai<br />33<br />
34<br />
35<br />
36<br />
37<br />
38<br />
Twitter In Controversial Spotlight Amid Mumbai Attacks Posted by Alexander Wolfe, Nov 29, 2008 11:27 AM <br />Never before...
In fairness, no one did. There were so many tentacles to these heinous attacks, and multiple hot spots (the Taj and Oberoi...
Alexander Wolfe continues …<br />I'd add that Mumbai is likely to be viewed in hindsight as the first instance of the para...
Analyzing Citizen Data<br />42<br />Semantic Data Store<br />Cloud/ Web based Services<br />Model Enrichment<br />Semantic...
Online and offline worlds<br />Computational abstractions to represent the physical world’s dynamic nature <br />Merging o...
Enriching Human Experience<br />Recognition of objects (IOT) and models of object<br />Understanding of objects and conten...
Understanding content … informal text<br />I say: “Your music is wicked” <br />What I really mean: “Your music is good” <b...
Urban Dictionary<br />Sentiment expression: Rocks <br />Transliterates to: cool, good<br />Structured text (biomedical lit...
Example: Pulse of a Community<br />Imagine millions of such informal opinions<br />Individual expressions to mass opinions...
How  do you get comprehensive situational awareness by merging “human sensing” and “machine sensing”?<br />48<br />
Synthetic but realistic scenario<br />an image taken from a raw satellite feed<br />an image taken by a camera phone with ...
Kno.e.sis’ Semantic Sensor Web<br />50<br />
DATA-TO-KNOWLEDGE ARCHITECTURE<br />Knowledge<br /><ul><li> Object-Event Relations
 Spatiotemporal Associations
 Provenance/Context</li></ul>Data Storage<br />(Raw Data, XML, RDF)<br />Semantic Analysis and Query<br />Information<br /...
 Feature Metadata</li></ul>Feature Extraction and Entity Detection<br />Semantic<br />Annotation<br />Data<br /><ul><li> R...
 Time Ontology
 Domain Ontology</li></ul>=|<br />RHO<br />DELTA<br />SIGMA<br />&<br />51<br />
On Our Way.. Multimodal interfaces<br />We are already seeing efforts toward this larger goal<br />Video visors - computer...
Challenges – Multiple Modalities<br />Multiple modalities of objects and events<br />How do we organize and access multimo...
Objects to Events<br />If we move from this object mode to an event mode<br />A single user action or request or sensory o...
We are already seeing efforts toward this larger goal<br />Social connections, interests, locations, alerts, comment<br />...
On our way…Internet of Things<br />Internet of Things: “A world where inanimate objects communicate with us and one anothe...
Building models…seed word to hierarchy creation using WIKIPEDIA<br />Query: “cognition”<br />
Learn Patterns that indicate Relationships<br />Query:<br />Australia  Sidney<br />Australia  Canberra<br /><ul><li>in S...
Sydney is the most populous city in Australia
Canberra, the Australian capital city
Canberra is the capital city of the Commonwealth of Australia
Canberra, the Australian capital
in Sydney, New South Wales, Australia
Sydney is the most populous city in Australia
Canberra, the Australian capital city
Canberra is the capital city of the Commonwealth of Australia
Canberra, the Australian capital</li></ul>We know that countries have capitals.<br />Which one is Australia’s?<br />
Experience<br />Direct Observation of<br />or Participation in<br />Events <br />as a basis of knowledge<br />
Entities and Events<br />Event <br />Entity <br />Name<br />Duration<br />Name<br />Location<br />Attributes<br />Data-str...
Strategic Inflection Points<br />Events on Web<br />(Experience)<br />Documents on Web<br />(Information)<br />Immersive<b...
Thanks – Ramesh Jain<br />
EventWeb [Jain], RelationshipWeb [Sheth]<br />Suppose that we create a Web in which<br />Each node is an event or object <...
is_advised_by<br />publishes<br />Researcher<br />Ph.D Student<br />Research Paper<br />published_in<br />published_in<br ...
Search<br />Integration<br />Domain Models<br />Structured text (biomedical literature)<br />Analysis<br />Discovery<br />...
HOw are Harry Potter and Dan Brown related?<br />
Challenges – Complex Events<br />Formal framework to model complex situations and composite events <br />Those consisting ...
However, today<br />68<br />Sensors capture and process uni-modal information. Bringing multiple modalities together is up...
THE SEMANTICS OF OBSERVATION<br />Observation is about capturing (or measuring) phenomena.<br />Perception is about explai...
ABDUCTION<br />	A formal model of inference which centers on cause-effect relationships and tries to find the best or most...
FORMS OF REASONING<br />Deduction (Prediction)<br />fact a, rule a => b  <br />INFER  b  (First-order logic)<br />Reasonin...
Perception as Abduction<br />The task of abductive perception is to find a consistent set of perceived objects and events ...
Upcoming SlideShare
Loading in …5
×

Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond

3,906
-1

Published on

Keynote at the 3rd Asian Semantic Web Conference (ASWC2008), Bangkok, Thailand, Feb 2-5, 2009. http://aswc2008.ait.ac.th/invitedspeaker2.html

More details: http://wiki.knoesis.org/index.php/Computing_For_Human_Experience

Published in: Education
1 Comment
7 Likes
Statistics
Notes
No Downloads
Views
Total Views
3,906
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
150
Comments
1
Likes
7
Embeds 0
No embeds

No notes for slide
  • Weiser’s human-centered vision of ubiquitous computing… new mode ofhuman–computer interaction.
  • Humanist computing: combination of observed human behaviour and conventional computing as humanist computing (perception based fuzzy feature, …particle classification and facial feature discovery).Experential computing: Seeking insight from data, users experience, explore, and experiment, no longer limited to just generating lists of possible answers to simplistic queries.
  • Going from syntactic to semantic metadata – Semantic Sensor MLInsert model references into tagsDomain Ontology e.g. people, companiesSpatial Ontology e.g. coordinate systems and coordinatesTemporal Ontology e.g. time units, time zones
  • Video visor - So as you&apos;re walking down a busy city street you will be able to see reviews of shops and restaurants, adverts for services, other people who have similar interests to you, or whatever.
  • Video visor - So as you&apos;re walking down a busy city street you will be able to see reviews of shops and restaurants, adverts for services, other people who have similar interests to you, or whatever.
  • Video visor - So as you&apos;re walking down a busy city street you will be able to see reviews of shops and restaurants, adverts for services, other people who have similar interests to you, or whatever.Internet– connecting computers to mobile devices to sensors.
  • Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond

    1. 1.
    2. 2. 2<br />
    3. 3. imagine<br />
    4. 4. imagine when<br />
    5. 5. meets <br />Farm Helper<br />
    6. 6. with this<br />Latitude: 38° 57’36” N<br />Longitude: 95° 15’12” W<br />Date: 10-9-2007<br />Time: 1345h<br />
    7. 7. that is sent to <br />Sensor Data Resource<br />Structured Data Resource<br />Weather Resource<br />Agri DB<br />Soil Survey <br />Weather Data<br />Services Resource<br />Location<br />Date /Time<br />Geocoder<br />Weather data<br />Lawrence, KS<br />Lat-Long<br />Farm Helper<br />Soil Information<br />Pest information …<br />
    8. 8. and<br />
    9. 9. Six billion brains <br />
    10. 10. imagination today <br />
    11. 11. impacts our experience tomorrow<br />
    12. 12.
    13. 13. Computing For Human Experience<br />ASWC 2008 Keynote<br />Amit P. Sheth,<br />Lexis Nexis Eminent Scholar and <br />Director, kno.e.sis center<br />Knoesis.org<br />
    14. 14. Technology that fits right in<br />“The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it. <br />Machines that fit the human environment instead of forcing humans to enter theirs will make using a computer as refreshing as a walk in the woods.” Mark Weiser, The Computer for the 21st Century (Ubicomp vision)<br />“We're crying out for technology that will allow us to combine what we can do on the Internet with what we do in the physical world.” <br /> Ian Pearson in Big data: The next Google <br />14<br />http://tarakash.com/guj/tool/hug2.html<br />Get citation<br />
    15. 15. (c) 2007 Thomas Gruber<br />15<br />But we are not talking about<br />Intelligent Design<br />Ubicomp: Mark Wisner and others<br />Intelligence @ Interface: Gruber –<br />Your life - on-line: <br />search, chat, music, photos, <br />videos, email, <br />multi-touch, mobile phone<br />
    16. 16. What is CHE? Beyond better human interaction<br />Focus in the past (egUbicomp): How humans interact with the system (computer, Internet)<br />Our focus—almost the reverse of the past (and both are needed)<br />Computing for Human Experience is about:How computing serves, assists and collaborates with humans to complement and enrich their normal activities <br />nondestructively and unobtrusively, with minimal explicit concern or effort on part of humans<br />anticipatory, knowledgeable, intelligent, ubiquitous<br />Computing that encompasses semantic, social, service, sensor and mobile Web<br />16<br />
    17. 17. Principals of CHE<br />Human is the master, system is the slave<br />Human sees minimal changes to normal behavior and activity, system is there to serve/assist/support in human’s natural condition<br />Search, browsing, etc are not primary; HCI is not the focus<br />Getting the assistance and answers are important, improving experience is key<br />Multimodal and multisensory environment<br />Integrated and contextual application of (not just access to) sensor data, databases, collective intelligence, wisdom of the crowd, conceptual models, reasoning<br />17<br />
    18. 18. Learning from a number of exciting visions<br />18<br />
    19. 19. Evolution of the Web (and associated computing)<br />Web of people<br /> - social networks, user-created casual<br /> content<br /> - Twine, GeneRIF, Connotea<br />Web of resources<br /> - data, service, data, mashups<br /> - ubiquitous/mobile computing<br />Web of databases<br /> - dynamically generated pages<br /> - web query interfaces<br />Web of pages<br /> - text, manually created links<br /> - extensive navigation<br />Computing for Human Experience<br />Web as an oracle / assistant / partner<br /> - “ask the Web”: using semantics to leverage text + data + services <br /> - Powerset <br />2007<br />Semantic TechnologyUsed<br />1997<br />
    20. 20. Sensing, Observing, Perceptual, Semantic, Social, Experiential<br />CHE components and enablers<br />
    21. 21. Consider <br />that all objects, events and activities in the physical world have a counterpart in the Cyberworld(IoT)<br />multi-facted context of real world is captured in the cyberworld(sensor web, citizen sensor)<br />each object, event and activity is represented <br />with semantic annotations (semantic sensor web)<br /><ul><li>for a chosen context, with an ability to explicate and associate variety of relationships and events (Relationship Web, EventWeb)
    22. 22. appropriate reasoning and human/social interaction are available and applied, insights extracted (semantic web, social semantic web, experiential computing)
    23. 23. answers obtained/ decisions reached/communicated/applied</li></ul>21<br />
    24. 24. Paradigm shift …<br />Where humans act as sensors or observers<br />Around them is a network of sensors, computing and communicating with each other<br />Processing and delivering multi-modal information <br />Collective Intelligence<br />Information-centric to Experience-centric era<br />Modeling, processing, retrieving event level information<br />Use of domain knowledge <br />….<br />Understanding of casual text<br />22<br />
    25. 25. Today’s Sensor Network Types<br />Inert, fixed sensors<br />Carried on moving objects<br />Vehicles, pedestrians (asthma research)<br />anonymous data from GPS-enabled vehicles, toll tags, and cellular signaling<br /> to mark how fast objects are moving – and overlaying that information with location data and maps (traffic.com, Nokia experiment, …)<br />23<br />Text from http://www.geog.ucsb.edu/~good/presentations/icsc.pdf<br />Images credit – flickr.com, cnet.com<br />
    26. 26. Today’s Network of Sensors<br />Are sensing, computing, transmitting <br />Are acting in concert <br />Sharing data <br />Processing them into meaningful digital representations of the world <br />Researchers using 'sensor webs' to ask new questions or test hypotheses<br />24<br />
    27. 27. Machine sensing<br />25<br />
    28. 28. 26<br />Semantic Sensor Web<br />
    29. 29. <swe:component name="time"><br /> <swe:Time definition="urn:ogc:def:phenomenon:time" uom="urn:ogc:def:unit:date-time"><br /> <sa:swe rdfa:about="?time" rdfa:instanceof="time:Instant"><br /> <sa:sml rdfa:property="xs:date-time"/><br /> </sa:swe><br /> </swe:Time><br /></swe:component><br /><swe:component name="measured_air_temperature"><br /> <swe:Quantity definition="urn:ogc:def:phenomenon:temperature“ uom="urn:ogc:def:unit:fahrenheit"><br /> <sa:swe rdfa:about="?measured_air_temperature“ rdfa:instanceof=“senso:TemperatureObservation"><br /> <sa:swe rdfa:property="weather:fahrenheit"/><br /> <sa:swe rdfa:rel="senso:occurred_when" resource="?time"/><br /> <sa:swe rdfa:rel="senso:observed_by" resource="senso:buckeye_sensor"/><br /> </sa:sml> <br /> </swe:Quantity><br /></swe:component><br /><swe:value name=“weather-data"><br /> 2008-03-08T05:00:00,29.1<br /></swe:value><br />Semantically Annotated O&M<br />27<br />
    30. 30. Semantic Sensor ML – Adding Ontological Metadata<br />Domain<br />Ontology<br />Person<br />Company<br />Spatial<br />Ontology<br />Coordinates<br />Coordinate System<br />Temporal<br />Ontology<br />Time Units<br />Timezone<br />28<br />Mike Botts, "SensorML and Sensor Web Enablement," <br />Earth System Science Center, UAB Huntsville<br />
    31. 31. 29<br />Semantic Query<br />Semantic Temporal Query<br />Model-references from SML to OWL-Time ontology concepts provides the ability to perform semantic temporal queries<br />Supported semantic query operators include:<br />contains: user-specified interval falls wholly within a sensor reading interval (also called inside)<br />within: sensor reading interval falls wholly within the user-specified interval (inverse of contains or inside)<br />overlaps: user-specified interval overlaps the sensor reading interval<br />Example SPARQL query defining the temporal operator ‘within’<br />
    32. 32. Citizen Sensors<br />Human beings<br />6 billion intelligent sensors<br />informed observers<br />rich local knowledge<br />uplink technology<br />broadband Internet<br />mobile phone<br />30<br />Christmas Bird Count<br />
    33. 33. Citizen Science<br />Networks of amateur observers<br />possibly trained, skilled<br />Christmas Bird Count http://www.audubon.org/bird/citizen/index.html , http://www.audubon.org/bird/cbc/index.html<br />thousands of volunteer participants<br />Protocols<br />Project GLOBE<br />an international network of school children<br />reporting environmental conditions<br />central integration and redistribution<br />31<br />
    34. 34. Citizen Sensor – Humans Actively Engage<br />In connecting, searching, processing, stitching together information<br />Asks, gets.. Asks again, gets again…<br />32<br />Images credit – flickr.com <br />
    35. 35. Recent example - #Mumbai<br />33<br />
    36. 36. 34<br />
    37. 37. 35<br />
    38. 38. 36<br />
    39. 39. 37<br />
    40. 40. 38<br />
    41. 41. Twitter In Controversial Spotlight Amid Mumbai Attacks Posted by Alexander Wolfe, Nov 29, 2008 11:27 AM <br />Never before has a crisis unleashed so much raw data -- and so little interpretation -- than what we saw as the deadly terrorist attacks in Mumbai, India unfolded. Amid the real-time video feeds (kudos to CNN International), cellphone pictures, and tweets, we were able to keep abreast of what seemed to be happening, and where it was going down, all the while not really knowing those other key, canonical components of journalistic information gathering -- namely, who or why.<br />39<br />
    42. 42. In fairness, no one did. There were so many tentacles to these heinous attacks, and multiple hot spots (the Taj and Oberoi hotels and the Chabad Jewish center, to name the three most prominent), that even the Indian government likely didn't have a handle on things until late in the game. My point here is not criticize, but simply to note that I was struck, as never before, by the ability of data to outstrip information.<br />40<br />
    43. 43. Alexander Wolfe continues …<br />I'd add that Mumbai is likely to be viewed in hindsight as the first instance of the paradigmatic shift in crisis coverage: namely, journalists will henceforth no longer be the first to bring us information. Rather, they will be a conduit for the stream of images and video shot by a mix of amateurs and professionals on scene.<br />You've got to add to this the immense influence of Twitter. In the past few days, I've seen a slew of stories pointing out how Twitter was a key source of real-time updates on the attacks, to the point that Indian authorities asked people to stop posting to the microblogging service (for fear that they might be giving away strategic information to the terrorists).<br />41<br />
    44. 44. Analyzing Citizen Data<br />42<br />Semantic Data Store<br />Cloud/ Web based Services<br />Model Enrichment<br />Semantic Analysis and Annotation<br />Term Extraction<br />Twitter: “loud bang near the Taj” [ location:mumbai]<br />Location based aggregation<br />loud bangnear the Taj<br />Event:explosion<br />Location:Taj Hotel, Mumbai<br />DueTo: terrorist activity<br />CNN News feed: “terrorist activity reported in Mumbai Hotel” [ location:mumbai]<br />Citizen-Sensor Stream Aggregation<br />Typeof:noise, relatedto:explosion<br />Typeof:hotel , part of:Hotel_chain<br />Citizen Sensor Streams<br />Twitter RSS/ATOM feeds<br />Blogs<br />Image Courtesy: Chemical Brothers, Galvanize<br />
    45. 45. Online and offline worlds<br />Computational abstractions to represent the physical world’s dynamic nature <br />Merging online and offline activities<br />Connecting the physical world naturally with the online world<br />What are natural operations on these abstractions? <br />How do we detect these abstractions based on other abstractions and multimodal data sources?<br />43<br />
    46. 46. Enriching Human Experience<br />Recognition of objects (IOT) and models of object<br />Understanding of objects and content<br />Multimodal interfaces<br />Multi(level) sensing and perception<br />From keywords and entities to events and rich sets of relationships; spatio-temporal-thematic computing<br />Models (ontologies, folkonomies, taxonomies, classification, nomenclature)– time, location, sensors, domain <br />More powerful reasoning: paths, patterns, subgraphs that connect related things; deductive and abductive reasoning, ….<br />44<br />
    47. 47. Understanding content … informal text<br />I say: “Your music is wicked” <br />What I really mean: “Your music is good” <br />45<br />
    48. 48. Urban Dictionary<br />Sentiment expression: Rocks <br />Transliterates to: cool, good<br />Structured text (biomedical literature)<br />Semantic Metadata: Smile is a Track<br />Lil transliterates to Lilly Allen<br />Lilly Allen is an Artist<br />MusicBrainz Taxonomy<br />Informal Text (Social Network chatter)<br />Artist: Lilly Allen<br />Track: Smile<br /> Your smile rocks Lil<br />Multimedia Content and Web data<br />Web Services<br />
    49. 49. Example: Pulse of a Community<br />Imagine millions of such informal opinions<br />Individual expressions to mass opinions<br />“Popular artists” lists from MySpace comments<br />Lilly Allen <br />Lady Sovereign <br />Amy Winehouse<br />Gorillaz<br />Coldplay<br />Placebo<br />Sting<br />Kean<br />Joss Stone <br />
    50. 50. How do you get comprehensive situational awareness by merging “human sensing” and “machine sensing”?<br />48<br />
    51. 51. Synthetic but realistic scenario<br />an image taken from a raw satellite feed<br />an image taken by a camera phone with an associated label, “explosion.” These two images, one from a wide-area sensor and the other from a citizen-sensor, can be correlated with spatial and temporal attributes in order to provide comprehensive situational awareness.<br />49<br />
    52. 52. Kno.e.sis’ Semantic Sensor Web<br />50<br />
    53. 53. DATA-TO-KNOWLEDGE ARCHITECTURE<br />Knowledge<br /><ul><li> Object-Event Relations
    54. 54. Spatiotemporal Associations
    55. 55. Provenance/Context</li></ul>Data Storage<br />(Raw Data, XML, RDF)<br />Semantic Analysis and Query<br />Information<br /><ul><li> Entity Metadata
    56. 56. Feature Metadata</li></ul>Feature Extraction and Entity Detection<br />Semantic<br />Annotation<br />Data<br /><ul><li> Raw Phenomenological Data</li></ul>Sensor Observation<br />Ontologies<br /><ul><li> Space Ontology
    57. 57. Time Ontology
    58. 58. Domain Ontology</li></ul>=|<br />RHO<br />DELTA<br />SIGMA<br />&<br />51<br />
    59. 59. On Our Way.. Multimodal interfaces<br />We are already seeing efforts toward this larger goal<br />Video visors - computer image superimposed over the world around you. <br />52<br />
    60. 60. Challenges – Multiple Modalities<br />Multiple modalities of objects and events<br />How do we organize and access multimodal data? <br />How do we organize, index, search and aggregate events and multimedia experiences as effectively as modern search engines do for text using keywords? <br />53<br />
    61. 61. Objects to Events<br />If we move from this object mode to an event mode<br />A single user action or request or sensory observation could act as a cue for getting all (multi-modal) information associated with an event<br />If conditions change, systems could even modify their behavior to suit their changing view of the world <br />Today text is most prevalent, with increasing but disparate (non-integrated) image and video data, but human experience is event based (at higher levels of abstractions) formed based on multi-sensory, multi-perception (at lower level of abstraction) observations<br />54<br />
    62. 62. We are already seeing efforts toward this larger goal<br />Social connections, interests, locations, alerts, comment<br />Mobile phone to social compass: LOOPT.com<br />55<br />Image credit - www.movilae.com <br />On our way…<br />
    63. 63. On our way…Internet of Things<br />Internet of Things: “A world where inanimate objects communicate with us and one another over the network via tiny intelligent objects” - Jean Philippe Vasseur, NSSTG Systems<br />56<br />Image credit - www.forbes.com <br />
    64. 64. Building models…seed word to hierarchy creation using WIKIPEDIA<br />Query: “cognition”<br />
    65. 65. Learn Patterns that indicate Relationships<br />Query:<br />Australia  Sidney<br />Australia  Canberra<br /><ul><li>in Sydney, New South Wales, Australia
    66. 66. Sydney is the most populous city in Australia
    67. 67. Canberra, the Australian capital city
    68. 68. Canberra is the capital city of the Commonwealth of Australia
    69. 69. Canberra, the Australian capital
    70. 70. in Sydney, New South Wales, Australia
    71. 71. Sydney is the most populous city in Australia
    72. 72. Canberra, the Australian capital city
    73. 73. Canberra is the capital city of the Commonwealth of Australia
    74. 74. Canberra, the Australian capital</li></ul>We know that countries have capitals.<br />Which one is Australia’s?<br />
    75. 75. Experience<br />Direct Observation of<br />or Participation in<br />Events <br />as a basis of knowledge<br />
    76. 76. Entities and Events<br />Event <br />Entity <br />Name<br />Duration<br />Name<br />Location<br />Attributes<br />Data-streams<br />Attributes<br />Processes<br />Adjacent States<br />Related Links<br />Processes<br />(Services)<br />Objects and Entities<br />are static.<br />Events are dynamic.<br />Thanks – Ramesh Jain<br />
    77. 77. Strategic Inflection Points<br />Events on Web<br />(Experience)<br />Documents on Web<br />(Information)<br />Immersive<br />Experience<br />Contextual<br />Search<br />Ubiquitous <br />Devices<br />Semantic<br />Search<br />Updates <br />and alerts<br />Keyword <br />Search<br />1995<br />2000<br />2005<br />2010<br />Thanks – Ramesh Jain<br />
    78. 78. Thanks – Ramesh Jain<br />
    79. 79. EventWeb [Jain], RelationshipWeb [Sheth]<br />Suppose that we create a Web in which<br />Each node is an event or object <br />Each node may be connected to other nodes using <br />Referential: similar to common links that refer to other related information. <br />Spatial and temporal relationships.<br />Causal: establishing causality among relationships. <br />Relational: giving similarity or any other relationship. <br />Semantic or Domain specific:<br />Familial <br />Professional<br />Genetics,…<br />63<br />Adapted from a talk by Ramesh Jain<br />
    80. 80. is_advised_by<br />publishes<br />Researcher<br />Ph.D Student<br />Research Paper<br />published_in<br />published_in<br />Assistant Professor<br />Professor<br />Journal<br />Conference<br />has_location<br />Location<br />Moscone Center, SFO<br />May 28-29, 2008<br />Spatio- temporal<br />Causal<br />Image Metadata<br />Attended <br />Google IO<br />Event<br />Domain Specific<br />AmitSheth<br />Karthik Gomadam<br />Is advised by<br />Relational<br />Directs<br />kno.e.sis<br />
    81. 81. Search<br />Integration<br />Domain Models<br />Structured text (biomedical literature)<br />Analysis<br />Discovery<br />Question Answering<br />Patterns / Inference / Reasoning<br />Informal Text (Social Network chatter)<br />Relationship Web<br />Meta data / Semantic Annotations<br />Metadata Extraction<br />Multimedia Content and Web data<br />Web Services<br />
    82. 82. HOw are Harry Potter and Dan Brown related?<br />
    83. 83. Challenges – Complex Events<br />Formal framework to model complex situations and composite events <br />Those consisting of interrelated events of varying spatial and temporal granularity, together with their multimodal experiences<br />What computational approaches will help to compute and reason with events and their associated experiences and objects ?<br />67<br />
    84. 84. However, today<br />68<br />Sensors capture and process uni-modal information. Bringing multiple modalities together is up to an application<br />Object centric environments – sensors understand objects from data. Events and not objects lend to holistic views of an experience <br />Where is the Domain knowledge!?<br />Multi-modal information effectively represents events<br />Human to machine to human to machine..<br />
    85. 85. THE SEMANTICS OF OBSERVATION<br />Observation is about capturing (or measuring) phenomena.<br />Perception is about explaining the observations.<br />When the human mind perceives what it observes<br />It uses what it already knows in addition to the context surrounding the observation<br />Cause-effect relationships play a vital role in how we reach conclusions<br />69<br />
    86. 86. ABDUCTION<br /> A formal model of inference which centers on cause-effect relationships and tries to find the best or most plausible explanations (causes) for a set of given observations (effects).<br />70<br />
    87. 87. FORMS OF REASONING<br />Deduction (Prediction)<br />fact a, rule a => b <br />INFER b (First-order logic)<br />Reasoning/inferring from causes to effects<br />Abduction (Explanation)<br />rule a => b, observe b <br />POSSIBLE EXPLANATION a (different formalizations)<br />Explaining effects by hypothesizing causes<br />Induction (Learning)<br />observe correlation between a1, b1, ... an, bn<br />LEARN a -> b<br />Learning connections/rules from observations<br />71<br />
    88. 88. Perception as Abduction<br />The task of abductive perception is to find a consistent set of perceived objects and events (DELTA), given a background theory (SIGMA) and a set of observations (RHO)<br />SIGMA && DELTA |= RHO<br />72<br />Murray Shanahan, "Perception as Abduction: Turning Sensor Data Into Meaningful Representation"<br />
    89. 89. WHAT WE KNOW + WHAT WE OBSERVE = EUREKA!<br />SIGMA & DELTA |= RHO<br />Sensor data to knowledge<br />Semantic annotation (DELTA) of sensor observations (RHO) using contextual domain knowledge (SIGMA)<br />73<br />
    90. 90. WHAT WE KNOW + WHAT WE OBSERVE = EUREKA!<br />SIGMA & DELTA |= RHO<br />Understanding casual text<br />Meaning of (DELTA) slang sentiment expressions (RHO) using dictionaries (SIGMA)<br />74<br />
    91. 91. WHAT WE KNOW + WHAT WE OBSERVE = EUREKA!<br />SIGMA & DELTA |= RHO<br />Discovering undiscovered knowledge<br />Connecting (DELTA) seemingly unrelated text (RHO) using domain knowledge (SIGMA)<br />75<br />
    92. 92. WHAT WE KNOW + WHAT WE OBSERVE = EUREKA!<br />SIGMA & DELTA |= RHO<br />Taxonomy Creation<br />Seed word (RHO) to hierarchy creation (DELTA) using Wikipedia (SIGMA)<br />76<br />
    93. 93. REVEALING CONNECTIONS IN BIOMEDICAL TEXT<br />RHO (observations) – Text from PubMed<br />SIGMA (what we know) - Background knowledge from UMLS (schema), Mesh (instances)<br />DELTA (perceived connections) – relationships connecting entities<br />
    94. 94. People Web<br />(human-centric)<br />Sensor Web<br />(machine-centric)<br />Observation<br />(senses)<br />Observation<br />(sensors)<br />Perception<br />(analysis)<br />Perception<br />(cognition)<br />Communication<br />(language)<br />Communication<br />(services)<br />
    95. 95. Enhanced Experience (humans & machines working in harmony)<br />Observation<br />Perception<br />Communication<br />Semantics for shared conceptualization and interoperability between machine and human<br />Ability to share common communication<br />
    96. 96. Example <br />1. Sensors observe environmental phenomena and nearby vegetation. <br />2. Observation analysis determines potential situation and effects. <br /><ul><li>Through abductive reasoning, observation analysis perceives a possible storm as the best explanation hypothesis for observed phenomena.
    97. 97. Through predictive deductive reasoning, observation analysis determines the effect on the crops, including the potential for the poisoning of the soil from salt carried from the ocean in the wind.
    98. 98. Through query against a knowledge base of the agriculture domain, observation analysis determines that the best remedy
    99. 99. for saline soil is to “leach” the soil with excess irrigation water in order to ‘push’ the salts below the crop root zone,
    100. 100. for sodic soil is to add gypsum before leaching.</li></li></ul><li>Example <br />1. Sensors observe environmental phenomena and nearby vegetation. <br />2. Observation analysis determines potential situation and effects. <br />3. System alerts nearby farmers of situation and possible remedy.<br />4. Farmer goes outside and looks at the sky and crops.<br />5. Farmer perceives high-winds and dark rain clouds over the ocean view and agrees with system perception. <br />6. Farmer calls children and neighbors to help take the necessary precautions to save the vegetables.<br />
    101. 101. Sensing, Observation, Perception, Semantic, Social Experiential<br />PARADIGM SHIFT<br />82<br />
    102. 102. From the Semantic Web Community<br />Several key contributing research areas<br />Operating Systems, networks, sensors, content management and processing, multimodal data integration, event modelling, high-dimensional data visualization ….<br />Semantics and Semantic technologies can play vital role <br />In the area of processing sensor observations, the Semantic Web is already making strides<br />Use of core SW capabilities: knowledge representation, use of knowledge bases (ontologies, folkonomies, taxonomy, nomenclature), semantic metadata extraction/annotation, exploiting relationships, reasoning<br />83<br />
    103. 103. EventWeb: Experience Age<br />Family<br />Sports<br />Fun<br />Office<br />Finance<br />Knowledge<br />I am borrowing from Experiential computing and EventWeb – credit Ramesh Jain<br />Personal<br />
    104. 104. THERE IS MORE HAPPENING AT KNO.E.SIS<br />http://knoesis.org<br />85<br />Thanks: NSF (SemDis, Spatio-temporal-thematic), NIH, AFRL,<br />and also Microsoft Research, HP Research, IBM Research. See<br />http://knoesis.wright.edu/projects/funding/<br />
    105. 105. Kno.e.sis Members – a subset<br />
    106. 106. Influential Works<br />V Bush, As We May Think, The Atlantic, July 1945. [Memex, trail blazing]<br />Mark Weiser, The Computer for the Twenty-First Century, Scientific American, Sept 1991, 94-10. [The original vision paper on ubicomp. Expansive vision albeit technical aspects focused on HCI with networked tabs, pads and boards.]<br />V. Kashyap and A. Sheth, Semantics-based information brokering. Third ACM Intl Conf on Information and Knowledge Management (CIKM94), Nov 29 - Dec 02, 1994. ACM, New York, NY. [semantics based query processing (involving multiple ontologies, context, semantic proximity) across a federated information sources across the Web]<br />Abowd, Mynatt, Rodden, The Human Experience, Pervasive computing, 2002. [explores Mark Wisner’s original ubicomp vision]<br />Jonathan Rossiter , Humanist Computing: Modelling with Words, Concepts, and Behaviours, in Modelling with Words, Springer, 2003, pp. 124-152 [modelling with words, concepts and behaviours defines a hierarchy of methods which extends from the low level data-driven modelling with words to the high level fusion of knowledge in the context of human behaviours]<br />Ramesh Jain, Experiential computing. Commun. ACM 46, 7, Jul. 2003, 48-55. <br />AmitSheth, Sanjeev Thacker, and Shuchi Patel, Complex Relationship and Knowledge Discovery Support in the InfoQuilt System, VLDB Journal, 12 (1), May 2003, 2–27. [complex semantic inter-domain (multi-ontology) relationships including causal relationships to enable human-assisted knowledge discovery and hypothesis testing over Web-accessible heterogeneous data]<br />87<br />
    107. 107. AmbjörnNaeve: The Human Semantic Web: Shifting from Knowledge Push to Knowledge Pull. Int. J. Semantic Web Inf. Syst. 1(3): 1-30 (2005) [discusses conceptual interface providing human-understandable semantics on top of the ordinary (machine) Semantic Web]<br />Ramesh Jain, Toward EventWeb. IEEE Distributed Systems Online 8, 9, Sep. 2007. [a web of temporally related events… informational attributes such as experiential data in the form of audio, images, and video can be associated with the events]<br />The Internet of Things, International Telecommunication Union, Nov 2005.<br />Other Closely Related publications<br />AmitSheth and MeenaNagarajan, Semantics empowered Social Computing, IEEE Internet Computing, Jan-Feb 2009.<br />AmitSheth, Cory Henson, and SatyaSahoo, "Semantic Sensor Web," IEEE Internet Computing, July/August 2008, p. 78-83. <br />AmitSheth and Matthew Perry, “Traveling the Semantic Web through Space, Time and Theme,” IEEE Internet Computing, 12, (no.2), February/March 2008, pp.81-86. <br />AmitSheth and CarticRamakrishnan, “Relationship Web: Blazing Semantic Trails between Web Resources,” IEEE Internet Computing, July–August 2007, pp. 84–88.<br />88<br />
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×