Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
1Amit ShethLexisNexis Ohio Eminent Scholar & Director,Kno.e.sis – Ohio Center of Excellence in Knowledge-enabled Computing...
2CHE emphasizes the unobtrusive, supportive, andassistive role of technology in improving humanexperienceCHE encompasses t...
3A glimpse at similar visions of computing…
4Man-Computer Symbiosis – J. C. R.LickliderHumans and machines cooperateto solve complex problems whilemachines formulate ...
5Augmenting Human Intellect – D.EngelbartHumans utilize machinesto solve ―insoluble‖problems
6Ubiquitous Computing – M. WeiserMaking machines disappearinto the fabric of our life
7Making Intelligent Machines –J.McCarthy
8J.McCarthyM.WeiserD.EngelbartJ. C. R. Licklider
9Imagine the role of computationaltechniques in solving bigchallengesSustainabilityCrime preventionWater managementTraffic...
10Grand challenges in the real-world arecomplexIf people do not believe that mathematics is simple, it isonly because they...
11What has changed now?About 2 billion of the 5+ billion have data connections – so they perform ―citizen sensinAnd there ...
12―The next wave of dramatic Internet growth will come through the confluence ofpeople, process, data, and things — the In...
13What has not changed?We need computational paradigms totap into the rich pulse of the humanpopulaceWe are still working ...
14Consider an example ofMark, who is diagnosed withhypertension
15How do I manage my hypertension?Search for suggestionsfrom resources on thewebThere are many suggestions but lessinsight...
16HCPs aren’t waiting to be detailed, they’re turning to thesocial web to educate themselves60% of physicians either use o...
17http://www.slideshare.net/IQLab/social-media-101-for-pharma-3494462People are turning to each other online to understand...
1883% of online adults search for health informationhttp://www.slideshare.net/IQLab/social-media-101-for-pharma-3494462Sea...
1983% of online adults search for health information60% of them look for the experience of “someone like me”http://www.sli...
20There are many suggestions but lessinsights that Mark can understand andtake action
21How do I manage my hypertension?Will this relevant informationtranslate into actions?Solution engine such asWolframAlpha...
22search vs. solution engineConventional search returns a set of documents forserving the information need expressed as a ...
23What do we need to helpMark?We need a human-centric computational paradigm that cansemantically integrate, correlate, un...
Physical-Cyber-Social ComputingAn early 21st century approach to Computing for Human Experience
25PCSPhysical:Weight, height, Activity, Heartrate, Blood PressureCyber: Medicalknowledge,Encyclopedia, NIHguidelinesSocial...
PCS ComputingPeople live in the physical world while interacting with the cyber and socialworldsPhysicalWorldCyber WorldSo...
27People consider their observations and experiences from physical, cyber, andsocial worlds for decision making – this is ...
28PCS computing is intended to address the seamlessintegration of cyber world, and human activities andinteractionsPCSPCS ...
29Increasingly, real-world events are:(a) Continuous: Observations are fine grained over time(b) Multimodal, multisensory:...
30PCS computing operatorsVertical operators facilitatetranscending from data-information-knowledge-wisdomusing background ...
31Let’s take progressive steps from existingcomputing paradigms toward PCScomputing…
http://www.nsf.gov/pubs/2013/nsf13502/nsf13502.htm32Physical-Cyber SystemsThe computational, communication, and control co...
33Physical-Cyber Systems Google’s autonomous car needsto sense, compute, and actuatecontinuously in order to navigatethrou...
Physical Systems Cyber SystemsRemote heath monitoringMobile cardiac outpatient telemetry and real time analyticsApplicatio...
Social SystemsCyber Systems35Sharing experiences and management of conditions and their treatmentsManaging risks and makin...
Social SystemsPhysical Systems Cyber SystemsSelf knowledge through numbersInvolves interactions between all the threecompo...
37Computations leverage observations formsensors, knowledge and experiences frompeople to understand, correlate, andperson...
Social SystemsPhysical Systems Cyber Systems38Semantics play a crucial role in bridging thesemantic gap between different ...
39DeterminesExperiencePhysical, Cyberand SocialExperiencesPerceptualInferenceBackground Knowledge(spanning Physical-Cyber-...
DeterminesExperienceExperiencesPerceptualInferenceBackground Knowledge(spanning Physical-Cyber-Social)EvolvesInfluencesInf...
44PCS Computing OperatorsLet’s look at machine perception, whichbelongs to the category of vertical operatorsPerception (s...
45Homo Digitus(Quantified Self)
46http://www.technologyreview.com/featuredstory/426968/the-patient-of-the-future/MIT Technology Review, 2012The Patient of...
DATAsensorobservationsKNOWLEDGEsituation awareness usefulfor decision making47Primary challenge is to bridge the gap betwe...
48What if we could automate this sense makingability?… and do it efficiently and at scale
49Making sense of sensor data withHenson et al An Ontological Approach to Focusing Attention and Enhancing Machine Percept...
50People are good at making sense of sensoryinputWhat can we learn from cognitive models of perception?The key ingredient ...
* based on Neisser’s cognitive model of perceptionObservePropertyPerceiveFeatureExplanationDiscrimination12Translating low...
52To enable machine perception,Semantic Web technology is used to integratesensor data with prior knowledge on the WebW3C ...
W3C Semantic SensorNetwork (SSN) Ontology Bi-partite Graph53Prior knowledge on theWeb
W3C Semantic SensorNetwork (SSN) Ontology Bi-partite Graph54Prior knowledge on theWeb
ObservePropertyPerceiveFeatureExplanation1Translating low-levelsignals into high-levelknowledge55ExplanationExplanation is...
Inference to the best explanation• In general, explanation is an abductive problem;and hard to computeFinding the sweet sp...
ExplanatoryFeature ≡ ∃ssn:isPropertyOf—.{p1} ⊓ … ⊓ ∃ssn:isPropertyOf—.{pn}elevated blood pressureclammy skinpalpitationsHy...
ObservePropertyPerceiveFeatureExplanationDiscrimination2Focusing attention on thoseaspects of the environmentthat provide ...
ExpectedProperty ≡ ∃ssn:isPropertyOf.{f1} ⊓ … ⊓ ∃ssn:isPropertyOf.{fn}elevated blood pressureclammy skinpalpitationsHypert...
NotApplicableProperty ≡ ¬∃ssn:isPropertyOf.{f1} ⊓ … ⊓ ¬∃ssn:isPropertyOf.{fn}elevated blood pressureclammy skinpalpitation...
DiscriminatingProperty ≡ ¬ExpectedProperty ⊓ ¬NotApplicablePropertyelevated blood pressureclammy skinpalpitationsHypertens...
Qualities-High BP-Increased WeightEntities-Hypertension-HypothyroidismkHealthMachine SensorsPersonal InputEMR/PHRComorbidi...
Use of OWL reasoner is resource intensive(especially on resource-constrained devices),in terms of both memory and time• Ru...
intelligence at the edgeApproach 1: Send all sensorobservations to the cloud forprocessing64Approach 2: downscalesemantic ...
010110001101001111001010110001101101101011000110100111100101011000110101100011010011165Efficient execution of machine perc...
O(n3) < x < O(n4) O(n)Efficiency Improvement• Problem size increased from 10’s to 1000’s ofnodes• Time reduced from minute...
2 Prior knowledge is the key to perceptionUsing SW technologies, machine perception can beformalized and integrated with p...
68Application of semantic perception to healthcare…
kHealthKnowledge-enabled Healthcare69
Through physical monitoring andanalysis, our cellphones could actas an early warning system todetect serious healthconditi...
Approach:• Use semantic perception inference• with data from cardio-related sensors• and curated medical background knowle...
• Symptoms: 284• Disorders: 173• Causal Relations: 1944Unified Medical Language SystemCausal Network72Cardiology Backgroun...
Weight ScaleHeart Rate MonitorBlood PressureMonitor73SensorsAndroid Device(w/ kHealth App)Total cost: < $500kHealth Kit fo...
74• Abnormal heart rate• High blood pressure• Panic Disorder• Hypoglycemia• Hyperthyroidism• Heart Attack• Septic ShockObs...
75Are you feeling lightheaded?Are you have trouble takingdeep breaths?yesyes• Abnormal heart rate• High blood pressure• Li...
76http://knoesis.org/healthApp/Paper at ISWC: C. Henson, K. Thirunarayan, A. Sheth, An Efficient Bit Vector Approach toSem...
Dr. William Abraham, M.D.Director of CardiovascularMedicine77Pre-clinical usability trial
78• Asthma, Stress, COPD, Obesity, GI,etc.Other Potential Applications
• Real Time Feature Streams:http://www.youtube.com/watch?v=_ews4w_eCpg• kHealth: http://www.youtube.com/watch?v=btnRi64hJp...
PCS Computing for Asthma80
811http://www.nhlbi.nih.gov/health/health-topics/topics/asthma/2http://www.lung.org/lung-disease/asthma/resources/facts-an...
82Specific AimsCan we detect asthma/allergy early?Using data from on-body sensors, and environmental sensorsUsing knowledg...
Asthma is a multifactorial disease with health signals spanningpersonal, public health, and population levels.83Real-time ...
Detection of events, such aswheezing sound, indoortemperature, humidity, dust, andCO2 levelWeather ApplicationAsthma Healt...
85PCS Computing ChallengesVariety: Health signals span heterogeneous sourcesVolume: Health signals are fine grainedVelocit...
86Sensordrone – for monitoringenvironmental air qualityWheezometer – for monitoringwheezing soundsCan I reduce my asthma a...
ICS= inhaled corticosteroid, LABA = inhaled long-acting beta2-agonist, SABA= inhaled short-acting beta2-agonist ;*consider...
88PersonalLevelSignalsSocietal LevelSignals(Personal Level Signals)(PersonalizedSocietal Level Signal)(Societal Level Sign...
89Population LevelPersonalWheeze – YesDo you have tightness of chest? –YesObservationsPhysical-Cyber-Social System Health ...
PCS Computing for Parkinson’sDisease90
911 http://www.pdf.org/en/parkinson_statisticsParkinson’s Disease (PD): Severity of theproblem10million 60,000$25billion$1...
Parkinson’s disease (PD) data from The Michael J. Fox Foundationfor Parkinson’s Research.921https://www.kaggle.com/c/predi...
Massive Amount of Data to ActionsInput: Sensor observations such asacceleration, audio, battery, compass, and GPS fromsmar...
Symptoms to possible manifestations in sensorobservationsMildTremorsrapid change in x,y, and z (unlike speed, the x, y, an...
Feature extraction(accelerometer)ControlgroupPD patientThe movement of the controlgroup person is not restrictedand exhibi...
ControlgroupPD patientFeature extraction (audio)The speech of the control groupperson is normal and exhibitsgood modulatio...
ControlgroupPD patientFeature extraction (Compass)The walking direction of thecontrol group person is well-balanced and ex...
Evaluation: run classification algorithms on carefully crafted features fromknowledge of PD
99ParkinsonMild(person) = Tremor(person) ∧ PoorBalance(person)ParkinsonModerate(person) = MoveSlow(person) ∧ PoorSleep(per...
100PCS Computing for Traffic Analytics
1011 http://www.ibm.com/smarterplanet/us/en/traffic_congestion/ideas/2The Crisis of Public Transport in IndiaTraffic manag...
Vehicular traffic data from San Francisco Bay Area aggregated from on-road sensors (numerical) and incident reports (textu...
103Heterogeneity leading to complementary observations
Slow movingtrafficLinkDescriptionScheduledEventScheduledEvent511.org511.orgSchedule Information511.org104
105Uncertainty in a Physical-Cyber-SocialSystemObservation: Slow Moving TrafficMultiple Causes (Uncertain about the cause)...
106Modeling Traffic EventsInternal (to the road network) observationsSpeed, volume, and travel time observationsCorrelatio...
AccidentMusiceventSportingevent Road WorkTheatre eventExternal events<ActiveEvents, ScheduledEvents>Internal observations<...
Domain ExpertsColdWeatherPoorVisibilitySlowTrafficIcyRoadDeclarative domain knowledgeCausalknowledgeLinked Open DataColdWe...
http://conceptnet5.media.mit.edu/web/c/en/traffic_jamDelaygo to baseball gametraffic jamtraffic accidenttraffic jamActiveE...
Traffic jamLinkDescriptionScheduledEventtraffic jambaseballgameAdd missing random variablesTime of daybad weather CapableO...
111ScheduledEventActiveEventDay ofweekTime of daydelayTraveltimespeedvolumeStructure extractedformtraffic observations(sen...
114PCS Computing for Intelligence
PhysicalCyberSocialHUMINT:Interpersonal interactionsSecret cellGEOINT:Satellite imagesSIGINT:Mobile CommunicationSIGINT:Se...
Threat to physical/cyber infrastructureSIGINT:Phone logs indicatingcalling patternsIMINT:Observations fromcontinuous monit...
What physical infrastructure may have threats?SIGINT:Phone logs indicatingcalling patternsIMINT:Observations fromcontinuou...
Prior knowledge fromhistorical events/threatsCharacterizing threats and their severity level using weights learned from da...
There are a variety of sensors used to monitorvitals of soldiers, location, and presence of poisonous gases.O2Heart rateCO...
120CPS Current State of Art: LimitationsCPS are stovepipe systems with narrow set of observations of the realworld.No know...
124ConclusionsTransition form search to solution to PCS computing engines for actionableinformation.Seamless integration o...
125J.McCarthyM.WeiserD.EngelbartJ. C. R. Licklider
Influential visions by Bush, Licklider, Eaglebert, and Weiser.Amit Sheth, Pramod Anantharam, Cory Henson, Physical-Cyber-S...
• OpenSource:http://knoesis.org/opensource• Showcase: http://knoesis.org/showcase• Vision: http://knoesis.org/vision• Publ...
• Collaborators: U-Surrey (Payam Barnaghi), UCI(Ramesh Jain), DERI, AFRL, Boonshoft Sch of Med –WSU (Dr. Forbis, …), OSU W...
129Physical Cyber Social ComputingAmit Sheth, Kno.e.sis, Wright State
Amit Sheth’sPHD studentsAshutosh JadhavHemantPurohitVinhNguyenLu ChenPavanKapanipathiPramodAnantharamSujanPereraAlan Smith...
131thank you, and please visit us athttp://knoesis.orgKno.e.sis – Ohio Center of Excellence in Knowledge-enabled Computing...
Upcoming SlideShare
Loading in …5
×

Physical Cyber Social Computing: An early 21st century approach to Computing for Human Experience

9,277 views

Published on

Keynote given at WiMS 2013 Conference, June 12-14 2013, Madrid, Spain. http://aida.ii.uam.es/wims13/keynotes.php

Video of this talk at: http://videolectures.net/wims2013_sheth_physical_cyber_social_computing/

More information at: More at: http://wiki.knoesis.org/index.php/PCS
and http://knoesis.org/projects/ssw/

Replacing earlier versions: http://www.slideshare.net/apsheth/physical-cyber-social-computing & http://www.slideshare.net/apsheth/semantics-empowered-physicalcybersocial-systems-for-earthcube

Abstract: The proper role of technology to improve human experience has been discussed by visionaries and scientists from the early days of computing and electronic communication. Technology now plays an increasingly important role in facilitating and improving personal and social activities and engagements, decision making, interaction with physical and social worlds, generating insights, and just about anything that an intelligent human seeks to do. I have used the term Computing for Human Experience (CHE) [1] to capture this essential role of technology in a human centric vision. CHE emphasizes the unobtrusive, supportive and assistive role of technology in improving human experience, so that technology “takes into account the human world and allows computers themselves to disappear in the background” (Mark Weiser [2]).

In this talk, I will portray physical-cyber-social (PCS) computing that takes ideas from, and goes significantly beyond, the current progress in cyber-physical systems, socio-technical systems and cyber-social systems to support CHE [3]. I will exemplify future PCS application scenarios in healthcare and traffic management that are supported by (a) a deeper and richer semantic interdependence and interplay between sensors and devices at physical layers, (b) rich technology mediated social interactions, and (c) the gathering and application of collective intelligence characterized by massive and contextually relevant background knowledge and advanced reasoning in order to bridge machine and human perceptions. I will share an example of PCS computing using semantic perception [4], which converts low-level, heterogeneous, multimodal and contextually relevant data into high-level abstractions that can provide insights and assist humans in making complex decisions. The key proposition is to explain that PCS computing will need to move away from traditional data processing to multi-tier computation along data-information-knowledge-wisdom dimension that supports reasoning to convert data into abstractions that humans are adept at using.

[1] A. Sheth, Computing for Human Experience
[2] M. Weiser, The Computer for 21st Century
[3] A. Sheth, Semantics empowered Cyber-Physical-Social Systems
[4] C. Henson, A. Sheth, K. Thirunarayan, Semantic Perception: Converting Sensory Observations to Abstractions

Published in: Education, Technology, Business

Physical Cyber Social Computing: An early 21st century approach to Computing for Human Experience

  1. 1. 1Amit ShethLexisNexis Ohio Eminent Scholar & Director,Kno.e.sis – Ohio Center of Excellence in Knowledge-enabled ComputingWright State University, Dayton, OH, USAhttp://knoesis.orgSpecial Thanks: Pramod Anantharam. Ack: Cory Henson, TK Prasad and Kno.e.sis Semantic Sensor Web teamExternal collaboration: Payam Barnaghi @ Surrey (IoT), many domain scientists/clinicians…Physical-Cyber-Social ComputingAn early 21st century approach to Computing for Human ExperienceKeynote, Madrid, Spain. June 2013.
  2. 2. 2CHE emphasizes the unobtrusive, supportive, andassistive role of technology in improving humanexperienceCHE encompasses the essentialrole of technology in a human-centric visionA. Sheth: Computing for Human Experience: 2008 -
  3. 3. 3A glimpse at similar visions of computing…
  4. 4. 4Man-Computer Symbiosis – J. C. R.LickliderHumans and machines cooperateto solve complex problems whilemachines formulate the problem(advancing beyond solving aformulated problem )
  5. 5. 5Augmenting Human Intellect – D.EngelbartHumans utilize machinesto solve ―insoluble‖problems
  6. 6. 6Ubiquitous Computing – M. WeiserMaking machines disappearinto the fabric of our life
  7. 7. 7Making Intelligent Machines –J.McCarthy
  8. 8. 8J.McCarthyM.WeiserD.EngelbartJ. C. R. Licklider
  9. 9. 9Imagine the role of computationaltechniques in solving bigchallengesSustainabilityCrime preventionWater managementTrafficmanagementHealthcareDrug developmentManaging chronicconditionsHospitalreadmissionsSecurityIntelligence &reconnaissanceBorder protectionCyberinfrastructureCriticalinfrastructure
  10. 10. 10Grand challenges in the real-world arecomplexIf people do not believe that mathematics is simple, it isonly because they do not realize how complicated life is.- John von NeumannComputational paradigms have always dealt witha simplified representations of the real-world…Algorithms work on these simplifiedrepresentationsSolutions from these algorithms are transcendedback to the real-world by humans as actions* Ack: Ramesh Jain: http://ngs.ics.uci.edu/blog/?p=1501*
  11. 11. 11What has changed now?About 2 billion of the 5+ billion have data connections – so they perform ―citizen sensinAnd there are more devices connected to the Internet than the entire human populationThese ~2 billion citizen sensors and 10 billion devices & objects connected to theInternetmakes it an era of IoT (Internet of Things) and Internet of Everything (IoE).http://www.cisco.com/web/about/ac79/docs/innov/IoT_IBSG_0411FINAL.pdf
  12. 12. 12―The next wave of dramatic Internet growth will come through the confluence ofpeople, process, data, and things — the Internet of Everything (IoE).‖- CISCO IBSG, 2013http://www.cisco.com/web/about/ac79/docs/innov/IoE_Economy.pdfBeyond the IoE based infrastructure, it is the possibility of developing applications thatspansPhysical, Cyber and the Social Worlds that is very exciting. Think of PCSComputingas the application layer for the IoE-based infrastructure.What has changed now?
  13. 13. 13What has not changed?We need computational paradigms totap into the rich pulse of the humanpopulaceWe are still working on the simpler representations of the real-world!Represent, capture, and compute with richerand fine-grained representations of real-worldproblemsWhat should change?
  14. 14. 14Consider an example ofMark, who is diagnosed withhypertension
  15. 15. 15How do I manage my hypertension?Search for suggestionsfrom resources on thewebThere are many suggestions but lessinsights that Mark can understand andfollowSearch basedapproach
  16. 16. 16HCPs aren’t waiting to be detailed, they’re turning to thesocial web to educate themselves60% of physicians either use or are interested in using social networks65% of docs plan to usesocial media forprofessional developmentManhattan Research 2009, 2010Sermo,comCompete.comThis doc-to-docblogger has53,000 readersthis month +20,000 Twitterfollowers112,000 docstalk to eachother on Sermo.http://www.slideshare.net/IQLab/social-media-101-for-pharma-349446260% of physicians either use or are interested in using social networksSearch basedapproach
  17. 17. 17http://www.slideshare.net/IQLab/social-media-101-for-pharma-3494462People are turning to each other online to understand their healthSearch basedapproach
  18. 18. 1883% of online adults search for health informationhttp://www.slideshare.net/IQLab/social-media-101-for-pharma-3494462Search basedapproach
  19. 19. 1983% of online adults search for health information60% of them look for the experience of “someone like me”http://www.slideshare.net/IQLab/social-media-101-for-pharma-3494462Search basedapproach
  20. 20. 20There are many suggestions but lessinsights that Mark can understand andtake action
  21. 21. 21How do I manage my hypertension?Will this relevant informationtranslate into actions?Solution engine such asWolframAlpha would providevery relevant informationSolution engine basedapproachThough Mark acquired knowledgeabout hypertension, it is notpersonalized and contextually relevantfor taking any action
  22. 22. 22search vs. solution engineConventional search returns a set of documents forserving the information need expressed as a searchquery.WolframAlpha (which calls itself ―Answer engine‖provides statistical information when available for aquery.Distribution of hypertension casesbased on age, weight, height andBMIChances of findinghypertension cases based onethnicity.
  23. 23. 23What do we need to helpMark?We need a human-centric computational paradigm that cansemantically integrate, correlate, understand, and reason overmultimodal and multisensory observations to provide actionableinformationBeyond Search and SolutionEngine!Search and solution engines do not providepersonalized, contextually relevant, and mostimportantly, actionable information
  24. 24. Physical-Cyber-Social ComputingAn early 21st century approach to Computing for Human Experience
  25. 25. 25PCSPhysical:Weight, height, Activity, Heartrate, Blood PressureCyber: Medicalknowledge,Encyclopedia, NIHguidelinesSocial: Knowledge shared oncommunities,Similar ethnicity, Socio-economicallysimilarAnswer to the Mark’squestion, ―How do I managemy hypertension?‖ lies herePCS Computing
  26. 26. PCS ComputingPeople live in the physical world while interacting with the cyber and socialworldsPhysicalWorldCyber WorldSocial World
  27. 27. 27People consider their observations and experiences from physical, cyber, andsocial worlds for decision making – this is not captured in current computingparadigmsPhysicalWorldCyber World Social WorldDecision makingPCS ComputingSubstituting spices for salt intakewould reduce sodium intakeaiding in controlling hypertension
  28. 28. 28PCS computing is intended to address the seamlessintegration of cyber world, and human activities andinteractionsPCSPCS Computing
  29. 29. 29Increasingly, real-world events are:(a) Continuous: Observations are fine grained over time(b) Multimodal, multisensory: Observations span PCSmodalities
  30. 30. 30PCS computing operatorsVertical operators facilitatetranscending from data-information-knowledge-wisdomusing background knowledgeHorizontal operators facilitate semanticintegration of multimodal and multisensoryobservations
  31. 31. 31Let’s take progressive steps from existingcomputing paradigms toward PCScomputing…
  32. 32. http://www.nsf.gov/pubs/2013/nsf13502/nsf13502.htm32Physical-Cyber SystemsThe computational, communication, and control components closelyinteract with physical components enabling better cyber-mediatedobservation of and interaction with physical components.CPS involvessensing, computing, and actuatingcomponents
  33. 33. 33Physical-Cyber Systems Google’s autonomous car needsto sense, compute, and actuatecontinuously in order to navigatethrough the city traffic
  34. 34. Physical Systems Cyber SystemsRemote heath monitoringMobile cardiac outpatient telemetry and real time analyticsApplications integrating telemedicine serviceRemote real-time patient monitoring of vitalsOffers sensor solutions for monitoring cardiovascular ailmentsProvides solutions to monitor cardio activityNon-invasive wearable monitoring of vital signalsCardiovascular patient monitoringContinuous monitoring of physiological parametershttp://quantifiedself.com/2010/01/non-invasive-health-monitoring/34Physical-Cyber SystemsHealth applications and tools thatmonitor a person physically andconnect them to care providers (e.g.doctors)
  35. 35. Social SystemsCyber Systems35Sharing experiences and management of conditions and their treatmentsManaging risks and making informed decisions based on gene sequencingCyber-Social SystemsObservations spanning Cyberand Social world – people sharetheiractivities, knowledge, experiences, opinions, and perceptions.
  36. 36. Social SystemsPhysical Systems Cyber SystemsSelf knowledge through numbersInvolves interactions between all the threecomponents.36Sensors collecting observationsform the physical worldhttp://www.fitbit.com/product/features#socialSocial aspect of sharingand friendly competitionQS conference where experiences oanalyzing and visualizing data fromphysiological sensors are presentedThis data is stove piped due tofragmentation in sensor data collectionservices.Data collected from physiological sensorsanalyzed in the social context of ―similar‖people.Needs significant human involvement ininterpretation of physiologicalobservations using their knowledge ofthe domain and social experiences.Integration and interaction betweenphysical, cyber, and social componentsfor computation is brittle.Physical-Cyber-SocialSystems
  37. 37. 37Computations leverage observations formsensors, knowledge and experiences frompeople to understand, correlate, andpersonalize solutions.Physical-CyberSocial-CyberPhysical-Cyber-SocialWhat if?
  38. 38. Social SystemsPhysical Systems Cyber Systems38Semantics play a crucial role in bridging thesemantic gap between different sensortypes, modalities, and observations to deriveinsights leading to a holistic solution.PCS Computing
  39. 39. 39DeterminesExperiencePhysical, Cyberand SocialExperiencesPerceptualInferenceBackground Knowledge(spanning Physical-Cyber-Social)EvolvesInfluencesInfluencesObservesInfluencesWe experience the world throughperceptions and actionsExperiences evolve ourbackground knowledgeBackground knowledge + newobservations will enhance ourPCS Computing
  40. 40. DeterminesExperienceExperiencesPerceptualInferenceBackground Knowledge(spanning Physical-Cyber-Social)EvolvesInfluencesInfluencesObservesInfluencesDiastolic blood pressure(BP) between 86 and 90Asian male has lower thresholdsfor hypertension23How are my peers of thesame socio-economic-cultural background doingw.r.t. BP?5Experiences in managing bloodpressure6Increase the use of herbsand spices instead of saltAsianmale1Shared physiologicalobservations from sensorsCorrective actions to betaken47PCS Computing: Scenario of high Blood Pressure(BP)
  41. 41. 44PCS Computing OperatorsLet’s look at machine perception, whichbelongs to the category of vertical operatorsPerception (sense making) is the act of engaging in a cyclicalprocess of observation and generation of explanations to transformmassive amount of raw data to actionable information in the form ofabstractionsVerticalOperatorsHorizontalOperators
  42. 42. 45Homo Digitus(Quantified Self)
  43. 43. 46http://www.technologyreview.com/featuredstory/426968/the-patient-of-the-future/MIT Technology Review, 2012The Patient of the Future
  44. 44. DATAsensorobservationsKNOWLEDGEsituation awareness usefulfor decision making47Primary challenge is to bridge the gap between data andknowledge
  45. 45. 48What if we could automate this sense makingability?… and do it efficiently and at scale
  46. 46. 49Making sense of sensor data withHenson et al An Ontological Approach to Focusing Attention and Enhancing Machine Perception on the Web, Applied Ont, 2011
  47. 47. 50People are good at making sense of sensoryinputWhat can we learn from cognitive models of perception?The key ingredient is prior knowledge
  48. 48. * based on Neisser’s cognitive model of perceptionObservePropertyPerceiveFeatureExplanationDiscrimination12Translating low-levelsignals into high-levelknowledgeFocusing attention on thoseaspects of the environmentthat provide usefulinformationPrior Knowledge51Perception Cycle*Convert large number of observations to semanticabstractions that provide insights and translate intodecisions
  49. 49. 52To enable machine perception,Semantic Web technology is used to integratesensor data with prior knowledge on the WebW3C SSN XG 2010-2011, SSN Ontology
  50. 50. W3C Semantic SensorNetwork (SSN) Ontology Bi-partite Graph53Prior knowledge on theWeb
  51. 51. W3C Semantic SensorNetwork (SSN) Ontology Bi-partite Graph54Prior knowledge on theWeb
  52. 52. ObservePropertyPerceiveFeatureExplanation1Translating low-levelsignals into high-levelknowledge55ExplanationExplanation is the act of choosing the objects or events that bestaccount for a set of observations; often referred to as hypothesisbuilding
  53. 53. Inference to the best explanation• In general, explanation is an abductive problem;and hard to computeFinding the sweet spot between abduction and OWL• Single-feature assumption* enables use ofOWL-DL deductive reasoner* An explanation must be a single feature which accounts forall observed properties56ExplanationExplanation is the act of choosing the objects or events that bestaccount for a set of observations; often referred to as hypothesisbuildingRepresentation of Parsimonious Covering Theory in OWL-DL
  54. 54. ExplanatoryFeature ≡ ∃ssn:isPropertyOf—.{p1} ⊓ … ⊓ ∃ssn:isPropertyOf—.{pn}elevated blood pressureclammy skinpalpitationsHypertensionHyperthyroidismPulmonary EdemaObserved Property Explanatory Feature57ExplanationExplanatory Feature: a feature that explains the set of observedproperties
  55. 55. ObservePropertyPerceiveFeatureExplanationDiscrimination2Focusing attention on thoseaspects of the environmentthat provide usefulinformation58DiscriminationDiscrimination is the act of finding those properties that, ifobserved, would help distinguish between multiple explanatoryfeatures
  56. 56. ExpectedProperty ≡ ∃ssn:isPropertyOf.{f1} ⊓ … ⊓ ∃ssn:isPropertyOf.{fn}elevated blood pressureclammy skinpalpitationsHypertensionHyperthyroidismPulmonary EdemaExpected Property Explanatory Feature59DiscriminationExpected Property: would be explained by every explanatoryfeature
  57. 57. NotApplicableProperty ≡ ¬∃ssn:isPropertyOf.{f1} ⊓ … ⊓ ¬∃ssn:isPropertyOf.{fn}elevated blood pressureclammy skinpalpitationsHypertensionHyperthyroidismPulmonary EdemaNot Applicable Property Explanatory Feature60DiscriminationNot Applicable Property: would not be explained by anyexplanatory feature
  58. 58. DiscriminatingProperty ≡ ¬ExpectedProperty ⊓ ¬NotApplicablePropertyelevated blood pressureclammy skinpalpitationsHypertensionHyperthyroidismPulmonary EdemaDiscriminating Property Explanatory Feature61DiscriminationDiscriminating Property: is neither expected nor not-applicable
  59. 59. Qualities-High BP-Increased WeightEntities-Hypertension-HypothyroidismkHealthMachine SensorsPersonal InputEMR/PHRComorbidity riskscoree.g., CharlsonIndexLongitudinal studiesof cardiovascularrisks- Find risk factors- Validation- domain knowledge- domain expertFind contribution ofeach risk factorRisk Assessment ModelCurrentObservations-Physical-Physiological-HistoryRisk Score(e.g., 1 => continue3 => contact clinic)Model CreationValidate correlationsHistoricalobservationse.g., EMR, sensorobservations62Risk Score: from Data to Abstraction and Actionable Information
  60. 60. Use of OWL reasoner is resource intensive(especially on resource-constrained devices),in terms of both memory and time• Runs out of resources with prior knowledge >> 15 nodes• Asymptotic complexity: O(n3)63How do we implement machine perceptionefficiently on aresource-constrained device?
  61. 61. intelligence at the edgeApproach 1: Send all sensorobservations to the cloud forprocessing64Approach 2: downscalesemantic processing so thateach device is capable ofmachine perception
  62. 62. 010110001101001111001010110001101101101011000110100111100101011000110101100011010011165Efficient execution of machine perceptionUse bit vector encodings and their operations to encode priorknowledge and execute semantic reasoningHenson et al. An Efficient Bit Vector Approach to Semantics-based Machine Perception in Resource-ConstrainedDevices, ISWC 2012.
  63. 63. O(n3) < x < O(n4) O(n)Efficiency Improvement• Problem size increased from 10’s to 1000’s ofnodes• Time reduced from minutes to milliseconds• Complexity growth reduced from polynomial tolinear66Evaluation on a mobile device
  64. 64. 2 Prior knowledge is the key to perceptionUsing SW technologies, machine perception can beformalized and integrated with prior knowledge on theWeb3 Intelligence at the edgeBy downscaling semantic inference, machineperception can execute efficiently on resource-constraineddevices1 Translate low-level data to high-level knowledgeMachine perception can be used to convert low-levelsensory signals into high-level knowledge useful fordecision making67Semantic Perception for smarter analytics: 3 ideas totakeaway
  65. 65. 68Application of semantic perception to healthcare…
  66. 66. kHealthKnowledge-enabled Healthcare69
  67. 67. Through physical monitoring andanalysis, our cellphones could actas an early warning system todetect serious healthconditions, and provide actionableinformationcanary in a coal minekHealth: knowledge-enabled healthcare70Our Motivation
  68. 68. Approach:• Use semantic perception inference• with data from cardio-related sensors• and curated medical background knowledge on theWebto ask the patient contextually relevant questions71kHealth – knowledge-enabled healthcare
  69. 69. • Symptoms: 284• Disorders: 173• Causal Relations: 1944Unified Medical Language SystemCausal Network72Cardiology Background Knowledge
  70. 70. Weight ScaleHeart Rate MonitorBlood PressureMonitor73SensorsAndroid Device(w/ kHealth App)Total cost: < $500kHealth Kit for the application for reducing ADHF readmission
  71. 71. 74• Abnormal heart rate• High blood pressure• Panic Disorder• Hypoglycemia• Hyperthyroidism• Heart Attack• Septic ShockObserved Property Explanatory Featurevia BluetoothExplanation in kHealth
  72. 72. 75Are you feeling lightheaded?Are you have trouble takingdeep breaths?yesyes• Abnormal heart rate• High blood pressure• Lightheaded• Trouble breathing• Panic Disorder• Hypoglycemia• Hyperthyroidism• Heart Attack• Septic ShockContextually dependent questioning based on prior observations(from 284 possible questions)Observed Property Explanatory FeatureFocus in kHealth
  73. 73. 76http://knoesis.org/healthApp/Paper at ISWC: C. Henson, K. Thirunarayan, A. Sheth, An Efficient Bit Vector Approach toSemantics-based Machine Perception in Resource-Constrained DevicesA kHealth application leveragelow-cost sensors toward reducinghospital readmissions of ADHFpatientskHealth will continue toward thevision of Physical-Cyber-Socialcomputing tounderstand, correlate, andpersonalize healthcarekHealth Summary
  74. 74. Dr. William Abraham, M.D.Director of CardiovascularMedicine77Pre-clinical usability trial
  75. 75. 78• Asthma, Stress, COPD, Obesity, GI,etc.Other Potential Applications
  76. 76. • Real Time Feature Streams:http://www.youtube.com/watch?v=_ews4w_eCpg• kHealth: http://www.youtube.com/watch?v=btnRi64hJp479Demos
  77. 77. PCS Computing for Asthma80
  78. 78. 811http://www.nhlbi.nih.gov/health/health-topics/topics/asthma/2http://www.lung.org/lung-disease/asthma/resources/facts-and-figures/asthma-in-adults.html3Akinbami et al. (2009). Status of childhood asthma in the United States, 1980–2007. Pediatrics,123(Supplement 3), S131-S145.Asthma: Severity of the problem25million300million$50billion155,000593,000People in the U.S. arediagnosed with asthma(7 million are children)1.People suffering fromasthma worldwide2.Spent on asthma alonein a year2Hospital admissions in20063Emergency departmentvisits in 20063
  79. 79. 82Specific AimsCan we detect asthma/allergy early?Using data from on-body sensors, and environmental sensorsUsing knowledge from an asthma ontology, generated from asthmaknowledge on the Web and domain expertiseGenerate a risk measure from collected data and background knowledgeCan we characterize asthma/allergy progression?State of asthma patient may change over timeIdentifying risky progressions before worsening of the patient stateDoes the early detection of asthma/allergy, and subsequentintervention/treatment, lead to improved outcomes?Improved outcomes could be improved health (less serioussymptoms), less need for invasive treatments, preventive measures (e.g.avoiding risky environmental conditions), less cost, etc.
  80. 80. Asthma is a multifactorial disease with health signals spanningpersonal, public health, and population levels.83Real-time health signals from personal level (e.g., Wheezometer, NO inbreath, accelerometer, microphone), public health (e.g., CDC, HospitalEMR), and population level (e.g., pollen level, CO2) arriving continuouslyin fine grained samples potentially with missing information and unevensampling frequencies.Variety VolumeVeracityVelocityValueCan we detect the asthma severity level?Can we characterize asthma control level?What risk factors influence asthma control?What is the contribution of each risk factor?semanticsUnderstanding relationships betweehealth signals and asthma attacksfor providing actionable informationMassive Amount of Data to Actions
  81. 81. Detection of events, such aswheezing sound, indoortemperature, humidity, dust, andCO2 levelWeather ApplicationAsthma Healthcare Application84Action in the Physical WorldAsthma Example of ActionableInformationClose the window at home duringday to avoid CO2 inflow, to avoidasthma attacks at nightPublic HealthPersonalPopulationLevel
  82. 82. 85PCS Computing ChallengesVariety: Health signals span heterogeneous sourcesVolume: Health signals are fine grainedVelocity: Real-time change in situationsVeracity: Reliability of health signals may be compromisedPublic HealthPersonalPopulationLevelValue: Can I reduce my asthma attacks atnight?Value: Decision support todoctors by providing them withdeeper insights into patientasthma care
  83. 83. 86Sensordrone – for monitoringenvironmental air qualityWheezometer – for monitoringwheezing soundsCan I reduce my asthma attacks at night?What are the triggers?What is the wheezing level?What is the propensity toward asthma?What is the exposure level over a day?What is the air quality indoors?Commute to WorkPersonalPublic HealthPopulation LevelClosing the window at homein the morning and taking analternate route to office maylead to reduced asthma attacksActionableInformationPCS Computing: Asthma Scenario
  84. 84. ICS= inhaled corticosteroid, LABA = inhaled long-acting beta2-agonist, SABA= inhaled short-acting beta2-agonist ;*consider referral to specialistAsthma Controland Actionable InformationSensors and their observationsfor understanding asthmaPersonal, Public Health, and Population Level Signals for MonitoringAsthma
  85. 85. 88PersonalLevelSignalsSocietal LevelSignals(Personal Level Signals)(PersonalizedSocietal Level Signal)(Societal Level Signals)Societal LevelSignals Relevant tothe Personal LevelPersonal Level Sensors(kHealth**) (EventShop*)Qualify QuantifyActionRecommendationWhat are the features influencing my asthma?What is the contribution of each of these features?How controlled is my asthma? (risk score)What will be my action plan to manage asthma?StorageSocietal Level SensorsAsthma Early Warning Model (AEWM)Query AEWMVerify & augmentdomain knowledgeRecommendedActionActionJustification*http://www.slideshare.net/jain49/eventshop-120721, ** http://www.youtube.com/watch?v=btnRi64hJp4Asthma Early Warning Model
  86. 86. 89Population LevelPersonalWheeze – YesDo you have tightness of chest? –YesObservationsPhysical-Cyber-Social System Health Signal Extraction Health Signal Understanding<Wheezing=Yes, time, location><ChectTightness=Yes, time, location><PollenLevel=Medium, time, location><Pollution=Yes, time, location><Activity=High, time, location>WheezingChectTightnessPollenLevelPollutionActivityWheezingChectTightnessPollenLevelPollutionActivityRiskCategory<PollenLevel, ChectTightness, Pollution,Activity, Wheezing, RiskCategory><2, 1, 1,3, 1, RiskCategory><2, 1, 1,3, 1, RiskCategory><2, 1, 1,3, 1, RiskCategory><2, 1, 1,3, 1, RiskCategory>...ExpertKnowledgeBackgroundKnowledgetweet reporting pollution leveland asthma attacksAcceleration readings fromon-phone sensorsSensor and personalobservationsSignals from personal, personalspaces, and community spacesRisk Category assigned bydoctorsQualifyQuantifyEnrichOutdoor pollen and pollutionPublic HealthWell Controlled - continueNot Well Controlled – contact nursePoor Controlled – contact doctorHealth Signal Extraction to Understanding
  87. 87. PCS Computing for Parkinson’sDisease90
  88. 88. 911 http://www.pdf.org/en/parkinson_statisticsParkinson’s Disease (PD): Severity of theproblem10million 60,000$25billion$100,0001 millionPeople worldwide areliving with Parkinsonsdisease1.Americans arediagnosed withParkinsons diseaseeach year1.Spent on Parkinson’salone in a year in theUS1Therapeutic surgerycan cost up to $100,000dollars per patient1.Americans live withParkinson’s Disease1
  89. 89. Parkinson’s disease (PD) data from The Michael J. Fox Foundationfor Parkinson’s Research.921https://www.kaggle.com/c/predicting-parkinson-s-disease-progression-with-smartphone-data8 weeks of data from 5 sensors on a smart phone, collected for 16patientsresulting in ~12 GB (with lot of missing data).Variety VolumeVeracityVelocityValueCan we detect the onset of Parkinson’s disease?Can we characterize the disease progression?Can we provide actionable information to the patientsemanticsMassive Amount of Data to ActionsRepresenting prior knowledge ofPDled to a focused exploration ofthismassive dataset
  90. 90. Massive Amount of Data to ActionsInput: Sensor observations such asacceleration, audio, battery, compass, and GPS fromsmart phonesOutput:Distinguish patients with and without (control group) Parkinson’sCategorize disease progression/evolution over timeCategorize the severity for actionable information
  91. 91. Symptoms to possible manifestations in sensorobservationsMildTremorsrapid change in x,y, and z (unlike speed, the x, y, and z will have highvariance)Poor balancezig-zag movement using GPS? + compassModerateMove slowlyaverage speed of movementMove intermittentlyx, y, and z coordinates do not change overtimeDisturbed sleepsounds in the nightSlower monotone speechlow energy in the voice recordingAdvancedFall pronerapid change in zcoordinate
  92. 92. Feature extraction(accelerometer)ControlgroupPD patientThe movement of the controlgroup person is not restrictedand exhibits active motion withhigh variance in accelerationreadingsThe movement of the PD patientis restricted and exhibits slowmotion with low variance inacceleration readings
  93. 93. ControlgroupPD patientFeature extraction (audio)The speech of the control groupperson is normal and exhibitsgood modulation in audio energylevelThe speech of the PD patient ismonotone and exhibits lowmodulation in audio energy level
  94. 94. ControlgroupPD patientFeature extraction (Compass)The walking direction of thecontrol group person is well-balanced and exhibits equalvariations in all directionsThe walking direction of the PDpatient is not well-balanced andexhibits ―sticky‖ behavior (stuckin one direction)
  95. 95. Evaluation: run classification algorithms on carefully crafted features fromknowledge of PD
  96. 96. 99ParkinsonMild(person) = Tremor(person) ∧ PoorBalance(person)ParkinsonModerate(person) = MoveSlow(person) ∧ PoorSleep(person) ∧MonotoneSpeech(person)ParkinsonAdvanced(person) = Fall(person)Knowledge Based Analytics of PD datasetDeclarative Knowledge of Parkinson’sDisease used to focus our attention onsymptom manifestations in sensorobservationsControlgroupPD patientMovements of an activeperson has a gooddistribution over X, Y, and ZaxisAudio is well modulatedwith good variations in theenergy of the voiceRestricted movements bya PD patient can be seenin the acceleration readingsAudio is not well modulatedrepresented a monotonespeech
  97. 97. 100PCS Computing for Traffic Analytics
  98. 98. 1011 http://www.ibm.com/smarterplanet/us/en/traffic_congestion/ideas/2The Crisis of Public Transport in IndiaTraffic management: Severity of the problem236%42%285million91%1 billionIncrease in traffic from1981 to 20011. Have stress relatedimplications due totraffic1.People lived in cities inIndia, greater than theentire population of US2Got stuck in traffic withan average delay of 1.3hours in last 3 years1.Cars on road and thisnumber may double by20201
  99. 99. Vehicular traffic data from San Francisco Bay Area aggregated from on-road sensors (numerical) and incident reports (textual)102http://511.org/Every minute update of speed, volume, travel time, and occupancyresulting in 178 million link status observations, 738 active events, and146 scheduled events with many unevenly sampled observationscollected over 3 months.Variety VolumeVeracityVelocityValueCan we detect the onset of traffic congestion?Can we characterize traffic congestion based on eventsCan we provide actionable information to decision maksemanticsMassive Amount of Data to Actions(traffic data analytics)Representing prior knowledge oftraffic lead to a focusedexplorationof this massive data stream
  100. 100. 103Heterogeneity leading to complementary observations
  101. 101. Slow movingtrafficLinkDescriptionScheduledEventScheduledEvent511.org511.orgSchedule Information511.org104
  102. 102. 105Uncertainty in a Physical-Cyber-SocialSystemObservation: Slow Moving TrafficMultiple Causes (Uncertain about the cause):- Scheduled Events: music events, fair, theatreevents, concerts, road work, repairs, etc.- Active Events: accidents, disabled vehicles, break down ofroads/bridges, fire, bad weather, etc.- Peak hour: e.g. 7 am – 9 am OR 4 pm – 6 pmEach of these events may have a varying impact on traffic.A delay prediction algorithm should process multimodal and multi-sensory observations.
  103. 103. 106Modeling Traffic EventsInternal (to the road network) observationsSpeed, volume, and travel time observationsCorrelations may exist between these variables across different partsof the networkExternal (to the road network) eventsAccident, music event, sporting event, and planned eventsExternal events and internal observations may exhibit correlations
  104. 104. AccidentMusiceventSportingevent Road WorkTheatre eventExternal events<ActiveEvents, ScheduledEvents>Internal observations<speed, volume, traveTime>WeatherTime of Day107Modeling Traffic Events: Pictorialrepresentation
  105. 105. Domain ExpertsColdWeatherPoorVisibilitySlowTrafficIcyRoadDeclarative domain knowledgeCausalknowledgeLinked Open DataColdWeather(YES/NO)IcyRoad (ON/OFF) PoorVisibility (YES/NO)SlowTraffic (YES/NO)1 0 1 11 1 1 01 1 1 11 0 1 0Domain ObservationsDomainKnowledgeStructure and parameters108WinterSeasonCorrelations to causations usingDeclarative knowledge on theSemantic WebCombining Data and Knowledge Graph
  106. 106. http://conceptnet5.media.mit.edu/web/c/en/traffic_jamDelaygo to baseball gametraffic jamtraffic accidenttraffic jamActiveEventScheduledEventCausestraffic jamCausestraffic jamCapableOfslow trafficCapableOfoccur twice each dayCausesis_abad weatherCapableOfslow trafficroad iceCausesaccidentTimeOfDaygo to concertHasSubeventcar crashaccidentRelatedTocar crashBadWeatherCausesCausesis_ais_ais_a is_a is_ais_ais_a109Declarative knowledge from ConceptNet5
  107. 107. Traffic jamLinkDescriptionScheduledEventtraffic jambaseballgameAdd missing random variablesTime of daybad weather CapableOf slow trafficbadweatherTraffic data from sensors deployed onroad network in San Francisco Bay Areatime of daytraffic jambaseball gametime of dayslow trafficThree Operations: Complementing graphical model structureextractionAdd missing links badweathertraffic jambaseball gametime of dayslow trafficAdd link directionbadweathertraffic jambaseball gametime of dayslow trafficgo to baseball game Causes traffic jamKnowledge from ConceptNet5traffic jam CapableOfoccur twice each daytraffic jam CapableOf slow traffic110
  108. 108. 111ScheduledEventActiveEventDay ofweekTime of daydelayTraveltimespeedvolumeStructure extractedformtraffic observations(sensors + textual)using statisticaltechniquesScheduledEventActiveEventDay ofweekTime of daydelayTraveltimespeedvolumeBadWeatherEnriched structure whichhas link directions andnew nodes such as “BadWeather” potentiallyleading to better delaypredictionsEnriched Probabilistic Models using ConceptNet 5Anantharam et al: Traffic Analytics using Probabilistic Graphical Models Enhanced with Knowledge Bases
  109. 109. 114PCS Computing for Intelligence
  110. 110. PhysicalCyberSocialHUMINT:Interpersonal interactionsSecret cellGEOINT:Satellite imagesSIGINT:Mobile CommunicationSIGINT:Sensors on the soldierOSINT:Textual message exchangesSIGINT:On-ground sensorsOSINT:News reports, knowledge bases,intelligence databases,history of undesirable eventsObservations span physical, cyber, and social spacegenerating massive, multimodal, and multisensory observationsPCS Computing for Intelligence
  111. 111. Threat to physical/cyber infrastructureSIGINT:Phone logs indicatingcalling patternsIMINT:Observations fromcontinuous monitoring (e.g., CCTV)OSINT:Communication and groupdynamics on socialmedia and emailsGEOINT:Satellite and drone imagescapturing activities in ageographical areaGEOINT:Satellite imagesSIGINT:Mobile CommunicationSIGINT:Sensors on the soldierOSINT:Textual message exchangesSIGINT:On-ground sensorsOSINT:News reports, knowledge bases,intelligence databases,history of undesirable eventsKnowledgebaseThreatHistoryMulti-Int:Prior knowledge fromhistorical events/threatsPower GridNatural Gas PipelinesWater ReservoirsGovernment FacilitiesData SourcesObservationsScenario: Physical Cyber Social Threat
  112. 112. What physical infrastructure may have threats?SIGINT:Phone logs indicatingcalling patternsIMINT:Observations fromcontinuous monitoring (e.g., CCTV)OSINT:Communication and groupdynamics on socialmedia and emailsGEOINT:Satellite and drone imagescapturing activities in ageographical areaMulti-Int:Prior knowledge fromhistorical events/threatsKnowledgebaseThreatHistoryLocation based knowledgePhysical Infrastructure at a locationSurveillance with high activity (e.g., logs, CCTV)Prior disposition of infrastructure to risksEvents of high risk near the location (e.g., raid of explosives)Who are the suspects?Location based knowledgePeople from watch list and making international callsPeople from watch list and found near infrastructure through surveillanceWhat are the locations currently at high threat levelLocation based knowledgeLocations with physical infrastructure + suspectsLocations where surveillance logs show frequent attacksAnswering these questions demands semanticintegration, annotation, mapping, andinterpretation of massive multi-modal andmulti-sensory observations.ObservationsScenario(along with the knowledge required to answer them)
  113. 113. Prior knowledge fromhistorical events/threatsCharacterizing threats and their severity level using weights learned from dataThreatLevel1 => InternationalCalls ∧ ForeignTravel ∧ WeaponMarket ∧ ExplosiveIngredients w1ThreatLevel2 => InternationalCalls ∧ ForeignTravel ∧ WeaponMarket ∧ ExplosiveIngredients ∧ InWatchList w2ThreatLevel3 => InternationalCalls ∧ ForeignTravel ∧ WeaponMarket ∧ ExplosiveIngredients ∧ InWatchList ∧ CaughtSurveillancew3Unified Semantic representationof observations across multi-modaland heterogeneous observationsKnown threats based on analyzinghistorical threat scenarios and theconditions that persisted during thesethreatsAnomalies in:- Communication- BehaviorSemantic fusion (horizontal operators) of MULTI-Int abstraction (vertical operators) for situation awarenessFilter: semantics-empowered threat detectionExpand: graph based anomaly detectionDecision supportAbnormal signaturesKnowledge baseThreatHistoryWhat physical infrastructure may be at risk?Who are the suspects?What are the locations currently at high threat level?Takeaway Points:- PCS computing is crucial for a holistic interpretation of observations- Horizontal and vertical operators empowers analytics spanning CPS modalities- Semantics-empowered graph based anomaly detection algorithms will detect more anomalie- Leveraging prior knowledge from experts and historical data will allow characterization of thrPhysicalCyberKnowledge baseSocialDomain ExpertObservationsSIGINT:Phone logs indicatingcalling patternsIMINT:Observations fromcontinuous monitoring (e.g., CCTV)OSINT:Communication and groupdynamics on socialmedia and emailsGEOINT:Satellite and drone imagescapturing activities in ageographical areaMulti-Int:Prior knowledge fromhistorical events/threatsPCS computing for detecting threats
  114. 114. There are a variety of sensors used to monitorvitals of soldiers, location, and presence of poisonous gases.O2Heart rateCOCO2GPSAccelerometer CompassFootstepsIntrusion DetectionPCS computing for soldier healthmonitoring- Each soldier is augmented with situation awareness of location, poisonous gases, and vitals.- Cohort health management is crucial for real-time efficient management of stress levels ofsoldiers- Data sent to a mobile control center after initial pre-processing and analysis- Dynamic allocation of units based on their physiological state and stress levels leads to informeddecisions
  115. 115. 120CPS Current State of Art: LimitationsCPS are stovepipe systems with narrow set of observations of the realworld.No knowledge support for informed decision making with Mark’s case.Social aspects crucial for decision making is ignoredThe vision of Physical-Cyber-Social Computing is to provide solutions forthese limitations.
  116. 116. 124ConclusionsTransition form search to solution to PCS computing engines for actionableinformation.Seamless integration of technology involving selective human involvement.Transition from reactive systems (humans initiating information need) toproactive systems (machines initiating information need).Sharing of knowledge, experiences, and observations across physical-cyber-social worlds lead to informed decision making.Physical-Cyber-Social Computing articulates how to achieve this vision.Semantic computing supports integration and reasoning capabilities neededfor PCS computing.
  117. 117. 125J.McCarthyM.WeiserD.EngelbartJ. C. R. Licklider
  118. 118. Influential visions by Bush, Licklider, Eaglebert, and Weiser.Amit Sheth, Pramod Anantharam, Cory Henson, Physical-Cyber-Social Computing: An Early21st Century Approach, IEEE Intelligent Systems, pp. 79-82, Jan./Feb. 2013Amit Sheth,‖Computing for Human Experience: Semantics-EmpoweredSensors, Services, and Social Computing on the Ubiquitous Web,‖ IEEE InternetComputing, vol. 14, no. 1, pp. 88-91, Jan.-Feb. 2010, doi:10.1109/MIC.2010.4A. Sheth, Semantics empowered Cyber-Physical-Social Systems, Semantic Web in 2012workshop at ISWC 2102.Cory Henson, Amit Sheth, Krishnaprasad Thirunarayan, Semantic Perception: ConvertingSensory Observations to Abstractions, IEEE Internet Computing, vol. 16, no. 2, pp. 26-34, Mar./Apr. 2012, doi:10.1109/MIC.2012.20Cory Henson, Krishnaprasad Thirunarayan, Amit Sheth. An Ontological Approach toFocusing Attention and Enhancing Machine Perception on the Web. Applied Ontology, vol.6(4), pp.345-376, 2011.Cory Henson, Krishnaprasad Thirunarayan, and Amit Sheth, An Efficient Bit Vector Approachto Semantics-based Machine Perception in Resource-Constrained Devices, In: Proceedingsof 11th International Semantic Web Conference (ISWC2012), Boston, Massachusetts, USA, November 11-25, 2012.126A bit more on this topic
  119. 119. • OpenSource:http://knoesis.org/opensource• Showcase: http://knoesis.org/showcase• Vision: http://knoesis.org/vision• Publications: http://knoesis.org/library• PCS computing:http://wiki.knoesis.org/index.php/PCS127Kno.e.sis resources
  120. 120. • Collaborators: U-Surrey (Payam Barnaghi), UCI(Ramesh Jain), DERI, AFRL, Boonshoft Sch of Med –WSU (Dr. Forbis, …), OSU Wexner (Dr. Abraham), andseveral more clinical experts, …• Funding: NSF (esp. IIS-1111183 “SoCS: Social MediaEnhanced Organizational Sensemaking in EmergencyResponse,”), AFRL, NIH, Industry….Acknowledgements
  121. 121. 129Physical Cyber Social ComputingAmit Sheth, Kno.e.sis, Wright State
  122. 122. Amit Sheth’sPHD studentsAshutosh JadhavHemantPurohitVinhNguyenLu ChenPavanKapanipathiPramodAnantharamSujanPereraAlan SmithPramod KoneruMaryam PanahiazarSarasi LalithsenaCory HensonKalpaGunaratnaDelroyCameronSanjayaWijeratneWenboWangKno.e.sis in 2012 = ~100 researchers (15 faculty, ~50 PhD students)
  123. 123. 131thank you, and please visit us athttp://knoesis.orgKno.e.sis – Ohio Center of Excellence in Knowledge-enabled ComputingWright State University, Dayton, Ohio, USAPhysical-Cyber-Social Computing

×