Physical Cyber Social Computing: An early 21st century approach to Computing for Human Experience


Published on

Keynote given at WiMS 2013 Conference, June 12-14 2013, Madrid, Spain.

Video of this talk at:

More information at: More at:

Replacing earlier versions: &

Abstract: The proper role of technology to improve human experience has been discussed by visionaries and scientists from the early days of computing and electronic communication. Technology now plays an increasingly important role in facilitating and improving personal and social activities and engagements, decision making, interaction with physical and social worlds, generating insights, and just about anything that an intelligent human seeks to do. I have used the term Computing for Human Experience (CHE) [1] to capture this essential role of technology in a human centric vision. CHE emphasizes the unobtrusive, supportive and assistive role of technology in improving human experience, so that technology “takes into account the human world and allows computers themselves to disappear in the background” (Mark Weiser [2]).

In this talk, I will portray physical-cyber-social (PCS) computing that takes ideas from, and goes significantly beyond, the current progress in cyber-physical systems, socio-technical systems and cyber-social systems to support CHE [3]. I will exemplify future PCS application scenarios in healthcare and traffic management that are supported by (a) a deeper and richer semantic interdependence and interplay between sensors and devices at physical layers, (b) rich technology mediated social interactions, and (c) the gathering and application of collective intelligence characterized by massive and contextually relevant background knowledge and advanced reasoning in order to bridge machine and human perceptions. I will share an example of PCS computing using semantic perception [4], which converts low-level, heterogeneous, multimodal and contextually relevant data into high-level abstractions that can provide insights and assist humans in making complex decisions. The key proposition is to explain that PCS computing will need to move away from traditional data processing to multi-tier computation along data-information-knowledge-wisdom dimension that supports reasoning to convert data into abstractions that humans are adept at using.

[1] A. Sheth, Computing for Human Experience
[2] M. Weiser, The Computer for 21st Century
[3] A. Sheth, Semantics empowered Cyber-Physical-Social Systems
[4] C. Henson, A. Sheth, K. Thirunarayan, Semantic Perception: Converting Sensory Observations to Abstractions

Published in: Education, Technology, Business
1 Comment
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • More at:
  • There are over 99.4% of physical devices that may one day be connected toThe Internet still unconnected. - CISCO IBSG, 2013
  • More at:
  • - Larry Smarr is a professor at the University of California, San DiegoAnd he was diagnosed with Chrones DiseaseWhat’s interesting about this case is that Larry diagnosed himselfHe is a pioneer in the area of Quantified-Self, which uses sensors to monitor physiological symptomsThrough this process he discovered inflammation, which led him to discovery of Chrones DiseaseThis type of self-tracking is becoming more and more common
  • - what if we could automate this sense making ability?- and what if we could do this at scale?
  • sense making based on human cognitive models
  • perception cycle contains two primary phasesexplanationtranslating low-level signals into high-level abstractions inference to the best explanationdiscriminationfocusing attention on those properties that will help distinguish between multiple possible explanationsused to intelligently task sensors and collect additional observations (rather than brute force approach of blindly collecting all observations)
  • perception cycle contains two primary phasesexplanationtranslating low-level signals into high-level abstractions inference to the best explanationdiscriminationfocusing attention on those properties that will help distinguish between multiple possible explanationsused to intelligently task sensors and collect additional observations (rather than brute force approach of blindly collecting all observations)
  • A single-feature (disease) assumption means that all the observed properties (symptoms) must be explained by a single feature.i.e., this framework is not expressive enough to model comorbidity where there may be more than one feature (disease) co-existing For example, if there are two diseases causing disjoint symptoms, and all the symptoms of both the diseases are observed, then this framework will not be able to find the coverage and returns no diseases.
  • perception cycle contains two primary phasesexplanationtranslating low-level signals into high-level abstractions inference to the best explanationdiscriminationfocusing attention on those properties that will help distinguish between multiple possible explanationsused to intelligently task sensors and collect additional observations (rather than brute force approach of blindly collecting all observations)
  • Intelligence distributed at the edge of the networkRequires resource-constrained devices (mobile phones, gateway notes, etc.) to be able to utilize SW technologies
  • Intelligence distributed at the edge of the networkRequires resource-constrained devices (mobile phones, gateway notes, etc.) to be able to utilize SW technologiesHenson et al. 'An Efficient Bit Vector Approach to Semantics-based Machine Perception in Resource-Constrained Devices, ISWC 2012.
  • compute machine perception inferences -- i.e., explanation and discrimination -- of high-complexity on a resource-constrained devices in milisecondsDifference between the other systems and what this system provides
  • Intelligence at the age. Shipping computation and domain models to the edge (Distributed)
  • - With this ability,many problems could be solved- For example: we could help solve health problems (before they become serious health problems) through monitoring symptoms and real-time sense making, acting as an early warning system to detect problematic health conditions
  • THROUGH real-time monitoring (sensor + social + experiences ) and analysis (perception) could our cell phones detect serious health conditions
  • RO1 in preparation for submission:Biomedical Computing and Health Informatics Study Section (BCHI)
  • Also, we have weather application which performs abstraction on weather sensory observations to identify blizzard conditions (food for actions!!) :--20,000 weather stations (with ~5 sensors per station)-- Real-Time Feature Streams - live demo: - video demo:
  • Smartphone and sensors collect lot of observations at a rate that is impossible to store / transport it over the network. In fact, storing/transporting such raw observations is in fact wasteful. PD patients and doctors are not interested in all the raw observations and it is intractable for them to go through all the observations. They are interested in finding answers to the questions they may have, which often is actionable.
  • The five sensors used sample acceleration, audio, compass, battery status of smartphone, and the GPS => observations form these sensors are our input and we need to provide actionable information as output. Actionable information can be such as characterizing disease progression and severity for well informed management of PD
  • Often in bottom-up analysis (data driven), the prior knowledge of the domain is ignored. While in top-down analysis (knowledge driven), we may restrict uncovering new patterns/symptoms telltale signs of the disease. Further, mapping symptoms to sensor manifestations is not well defined but crucial for understanding the behavior of a PD patient. We address this challenge by mapping each symptom to sensor observations which lead to focused exploration of the massive dataset – effectively, reducing the search space.
  • This is a plot of acceleration along x, y, and z axis of the phone when the person is carrying the phone with them for ~20 days during daily activity Move slowly (one of the symptom of moderate PD) is experienced by CROCUS, who is a PD patient. The control group person has relatively active motion and this distinguishes form the PD patient
  • This is a plot of audio energy (db) that the phone recorded when the person is carrying the phone with them for ~20 days during daily activity Slower monotone speech (one of the symptom of moderate PD), is experienced by the PD patient while the control group person has a well modulated speech
  • This is a plot of compass values that the phone recorded when the person is carrying the phone with them for ~20 days during daily activity Poor balance (one of the symptom of mild PD), is experienced by the PD patient where the the compass reading tends to take one direction (like a person is experiencing a drag in one direction) while the control group member has no such “drag”
  • Utilizing the knowledge empowered featured, we performed bottom-up analysis to classify PD patients and ended up with the following results.We got up to 80% accuracy in classification using the knowledge based approach.
  • We have transformed massive amount of multisensory data collected to insights This slide just shows the logic rules that captures the three levels of severity of PD which were mapped to sensor manifestations
  • Point of this slide: heterogeneity and uncertainty
  • Point of this slide: correlations
  • A single observation of slow moving traffic may have multiple explanations.
  • Internal observations are limited to whatever the on-road sensors can observe. In the data we have analyzed, the internal observations are mentioned above.External events are obtained from sources beyond the on-road sensors e.g., some agency like which reports traffic incidents.Note that: Internal observations are mostly machine sensors External events are mostly textual observationsThe analogy in healthcare will be:Internal observations: on body sensors such as heart rate, temperatureExternal events: jogging, walking, taking stairs
  • Some facts about the domain of traffic got from Conceptnet5The types of events are obtained by using the comprehensive subsumption relationship from 511.orgWe propose to use such a knowledge in complementing the PGM structure learning algorithmsCapableOf(traffic jam, occur twice each day)CapableOf(traffic jam, slow traffic)RelatedTo(accident, car crash)Causes(road ice, accident)CapableOf(bad weather, slow traffic)HasSubevent(go to concert, car crash)Causes(go to baseball game, traffic jam)Causes(traffic accident, traffic jam)BadWeather(road ice)BadWeather(bad weather)ScheduledEvent(go to concert)ScheduledEvent(go to baseball game)ActiveEvent(traffic accident)Delay(slow traffic)Delay(traffic jam)TimeOfDay(occur twice each day)
  • Declarative knowledge + statistical correlationThis slide illustrates the three operations to enrich the correlation structure extracted using statistical methods These operations utilize declarative knowledge form ConceptNet5 as shown in each step
  • Statistical correlation structure shown aboveThe enriched structure is shown belowThe enrichment of the graphical model will potentially allow us to capture the domain precisely and also improve our prediction as the model would get closer to the underlying probabilistic distribution in the real-worldLog-Likelihood score is one way of quantifying how good a structure is based on the observed data There may be many candidate structures extracted from data which result in the log likelihood scoreDeclarative knowledge will help us ground statistical models to reality which will allow us to pick one structure over the other Pramod Anantharam, KrishnaprasadThirunarayan and AmitSheth, 'Traffic Analytics using Probabilistic Graphical Models Enhanced with Knowledge Bases,' 2nd International Workshop on Analytics for Cyber-Physical Systems (ACS-2013) at SIAM International Conference on Data Mining (SDM13), pp. 13--20, Texas, USA, May 2-4, 2013.We stopped at structure extraction for our workshop paper (SIAM ACS workshop) since the declarative knowledge we used (ConceptNet5) and statistical model (nodes and edges) are at the same level of abstraction
  • Cia factbookContinuousKnowledgeSocialNight vision cameraIntrusion detectionGPS and RadioPoisonous gas sensorHeart rate sensorAccelerometerMicrophoneBatteries
  • Domain expert validation,threat definition, and refinementWhat physical infrastructure may be at risk?Who are the suspects?What are the locations currently at high threat level?
  • More at:
  • Physical Cyber Social Computing: An early 21st century approach to Computing for Human Experience

    1. 1. 1Amit ShethLexisNexis Ohio Eminent Scholar & Director,Kno.e.sis – Ohio Center of Excellence in Knowledge-enabled ComputingWright State University, Dayton, OH, USAhttp://knoesis.orgSpecial Thanks: Pramod Anantharam. Ack: Cory Henson, TK Prasad and Kno.e.sis Semantic Sensor Web teamExternal collaboration: Payam Barnaghi @ Surrey (IoT), many domain scientists/clinicians…Physical-Cyber-Social ComputingAn early 21st century approach to Computing for Human ExperienceKeynote, Madrid, Spain. June 2013.
    2. 2. 2CHE emphasizes the unobtrusive, supportive, andassistive role of technology in improving humanexperienceCHE encompasses the essentialrole of technology in a human-centric visionA. Sheth: Computing for Human Experience: 2008 -
    3. 3. 3A glimpse at similar visions of computing…
    4. 4. 4Man-Computer Symbiosis – J. C. R.LickliderHumans and machines cooperateto solve complex problems whilemachines formulate the problem(advancing beyond solving aformulated problem )
    5. 5. 5Augmenting Human Intellect – D.EngelbartHumans utilize machinesto solve ―insoluble‖problems
    6. 6. 6Ubiquitous Computing – M. WeiserMaking machines disappearinto the fabric of our life
    7. 7. 7Making Intelligent Machines –J.McCarthy
    8. 8. 8J.McCarthyM.WeiserD.EngelbartJ. C. R. Licklider
    9. 9. 9Imagine the role of computationaltechniques in solving bigchallengesSustainabilityCrime preventionWater managementTrafficmanagementHealthcareDrug developmentManaging chronicconditionsHospitalreadmissionsSecurityIntelligence &reconnaissanceBorder protectionCyberinfrastructureCriticalinfrastructure
    10. 10. 10Grand challenges in the real-world arecomplexIf people do not believe that mathematics is simple, it isonly because they do not realize how complicated life is.- John von NeumannComputational paradigms have always dealt witha simplified representations of the real-world…Algorithms work on these simplifiedrepresentationsSolutions from these algorithms are transcendedback to the real-world by humans as actions* Ack: Ramesh Jain:*
    11. 11. 11What has changed now?About 2 billion of the 5+ billion have data connections – so they perform ―citizen sensinAnd there are more devices connected to the Internet than the entire human populationThese ~2 billion citizen sensors and 10 billion devices & objects connected to theInternetmakes it an era of IoT (Internet of Things) and Internet of Everything (IoE).
    12. 12. 12―The next wave of dramatic Internet growth will come through the confluence ofpeople, process, data, and things — the Internet of Everything (IoE).‖- CISCO IBSG, 2013 the IoE based infrastructure, it is the possibility of developing applications thatspansPhysical, Cyber and the Social Worlds that is very exciting. Think of PCSComputingas the application layer for the IoE-based infrastructure.What has changed now?
    13. 13. 13What has not changed?We need computational paradigms totap into the rich pulse of the humanpopulaceWe are still working on the simpler representations of the real-world!Represent, capture, and compute with richerand fine-grained representations of real-worldproblemsWhat should change?
    14. 14. 14Consider an example ofMark, who is diagnosed withhypertension
    15. 15. 15How do I manage my hypertension?Search for suggestionsfrom resources on thewebThere are many suggestions but lessinsights that Mark can understand andfollowSearch basedapproach
    16. 16. 16HCPs aren’t waiting to be detailed, they’re turning to thesocial web to educate themselves60% of physicians either use or are interested in using social networks65% of docs plan to usesocial media forprofessional developmentManhattan Research 2009, 2010Sermo,comCompete.comThis doc-to-docblogger has53,000 readersthis month +20,000 Twitterfollowers112,000 docstalk to eachother on Sermo. of physicians either use or are interested in using social networksSearch basedapproach
    17. 17. 17 are turning to each other online to understand their healthSearch basedapproach
    18. 18. 1883% of online adults search for health information basedapproach
    19. 19. 1983% of online adults search for health information60% of them look for the experience of “someone like me” basedapproach
    20. 20. 20There are many suggestions but lessinsights that Mark can understand andtake action
    21. 21. 21How do I manage my hypertension?Will this relevant informationtranslate into actions?Solution engine such asWolframAlpha would providevery relevant informationSolution engine basedapproachThough Mark acquired knowledgeabout hypertension, it is notpersonalized and contextually relevantfor taking any action
    22. 22. 22search vs. solution engineConventional search returns a set of documents forserving the information need expressed as a searchquery.WolframAlpha (which calls itself ―Answer engine‖provides statistical information when available for aquery.Distribution of hypertension casesbased on age, weight, height andBMIChances of findinghypertension cases based onethnicity.
    23. 23. 23What do we need to helpMark?We need a human-centric computational paradigm that cansemantically integrate, correlate, understand, and reason overmultimodal and multisensory observations to provide actionableinformationBeyond Search and SolutionEngine!Search and solution engines do not providepersonalized, contextually relevant, and mostimportantly, actionable information
    24. 24. Physical-Cyber-Social ComputingAn early 21st century approach to Computing for Human Experience
    25. 25. 25PCSPhysical:Weight, height, Activity, Heartrate, Blood PressureCyber: Medicalknowledge,Encyclopedia, NIHguidelinesSocial: Knowledge shared oncommunities,Similar ethnicity, Socio-economicallysimilarAnswer to the Mark’squestion, ―How do I managemy hypertension?‖ lies herePCS Computing
    26. 26. PCS ComputingPeople live in the physical world while interacting with the cyber and socialworldsPhysicalWorldCyber WorldSocial World
    27. 27. 27People consider their observations and experiences from physical, cyber, andsocial worlds for decision making – this is not captured in current computingparadigmsPhysicalWorldCyber World Social WorldDecision makingPCS ComputingSubstituting spices for salt intakewould reduce sodium intakeaiding in controlling hypertension
    28. 28. 28PCS computing is intended to address the seamlessintegration of cyber world, and human activities andinteractionsPCSPCS Computing
    29. 29. 29Increasingly, real-world events are:(a) Continuous: Observations are fine grained over time(b) Multimodal, multisensory: Observations span PCSmodalities
    30. 30. 30PCS computing operatorsVertical operators facilitatetranscending from data-information-knowledge-wisdomusing background knowledgeHorizontal operators facilitate semanticintegration of multimodal and multisensoryobservations
    31. 31. 31Let’s take progressive steps from existingcomputing paradigms toward PCScomputing…
    32. 32. SystemsThe computational, communication, and control components closelyinteract with physical components enabling better cyber-mediatedobservation of and interaction with physical components.CPS involvessensing, computing, and actuatingcomponents
    33. 33. 33Physical-Cyber Systems Google’s autonomous car needsto sense, compute, and actuatecontinuously in order to navigatethrough the city traffic
    34. 34. Physical Systems Cyber SystemsRemote heath monitoringMobile cardiac outpatient telemetry and real time analyticsApplications integrating telemedicine serviceRemote real-time patient monitoring of vitalsOffers sensor solutions for monitoring cardiovascular ailmentsProvides solutions to monitor cardio activityNon-invasive wearable monitoring of vital signalsCardiovascular patient monitoringContinuous monitoring of physiological parameters SystemsHealth applications and tools thatmonitor a person physically andconnect them to care providers (e.g.doctors)
    35. 35. Social SystemsCyber Systems35Sharing experiences and management of conditions and their treatmentsManaging risks and making informed decisions based on gene sequencingCyber-Social SystemsObservations spanning Cyberand Social world – people sharetheiractivities, knowledge, experiences, opinions, and perceptions.
    36. 36. Social SystemsPhysical Systems Cyber SystemsSelf knowledge through numbersInvolves interactions between all the threecomponents.36Sensors collecting observationsform the physical world aspect of sharingand friendly competitionQS conference where experiences oanalyzing and visualizing data fromphysiological sensors are presentedThis data is stove piped due tofragmentation in sensor data collectionservices.Data collected from physiological sensorsanalyzed in the social context of ―similar‖people.Needs significant human involvement ininterpretation of physiologicalobservations using their knowledge ofthe domain and social experiences.Integration and interaction betweenphysical, cyber, and social componentsfor computation is brittle.Physical-Cyber-SocialSystems
    37. 37. 37Computations leverage observations formsensors, knowledge and experiences frompeople to understand, correlate, andpersonalize solutions.Physical-CyberSocial-CyberPhysical-Cyber-SocialWhat if?
    38. 38. Social SystemsPhysical Systems Cyber Systems38Semantics play a crucial role in bridging thesemantic gap between different sensortypes, modalities, and observations to deriveinsights leading to a holistic solution.PCS Computing
    39. 39. 39DeterminesExperiencePhysical, Cyberand SocialExperiencesPerceptualInferenceBackground Knowledge(spanning Physical-Cyber-Social)EvolvesInfluencesInfluencesObservesInfluencesWe experience the world throughperceptions and actionsExperiences evolve ourbackground knowledgeBackground knowledge + newobservations will enhance ourPCS Computing
    40. 40. DeterminesExperienceExperiencesPerceptualInferenceBackground Knowledge(spanning Physical-Cyber-Social)EvolvesInfluencesInfluencesObservesInfluencesDiastolic blood pressure(BP) between 86 and 90Asian male has lower thresholdsfor hypertension23How are my peers of thesame socio-economic-cultural background doingw.r.t. BP?5Experiences in managing bloodpressure6Increase the use of herbsand spices instead of saltAsianmale1Shared physiologicalobservations from sensorsCorrective actions to betaken47PCS Computing: Scenario of high Blood Pressure(BP)
    41. 41. 44PCS Computing OperatorsLet’s look at machine perception, whichbelongs to the category of vertical operatorsPerception (sense making) is the act of engaging in a cyclicalprocess of observation and generation of explanations to transformmassive amount of raw data to actionable information in the form ofabstractionsVerticalOperatorsHorizontalOperators
    42. 42. 45Homo Digitus(Quantified Self)
    43. 43. 46 Technology Review, 2012The Patient of the Future
    44. 44. DATAsensorobservationsKNOWLEDGEsituation awareness usefulfor decision making47Primary challenge is to bridge the gap between data andknowledge
    45. 45. 48What if we could automate this sense makingability?… and do it efficiently and at scale
    46. 46. 49Making sense of sensor data withHenson et al An Ontological Approach to Focusing Attention and Enhancing Machine Perception on the Web, Applied Ont, 2011
    47. 47. 50People are good at making sense of sensoryinputWhat can we learn from cognitive models of perception?The key ingredient is prior knowledge
    48. 48. * based on Neisser’s cognitive model of perceptionObservePropertyPerceiveFeatureExplanationDiscrimination12Translating low-levelsignals into high-levelknowledgeFocusing attention on thoseaspects of the environmentthat provide usefulinformationPrior Knowledge51Perception Cycle*Convert large number of observations to semanticabstractions that provide insights and translate intodecisions
    49. 49. 52To enable machine perception,Semantic Web technology is used to integratesensor data with prior knowledge on the WebW3C SSN XG 2010-2011, SSN Ontology
    50. 50. W3C Semantic SensorNetwork (SSN) Ontology Bi-partite Graph53Prior knowledge on theWeb
    51. 51. W3C Semantic SensorNetwork (SSN) Ontology Bi-partite Graph54Prior knowledge on theWeb
    52. 52. ObservePropertyPerceiveFeatureExplanation1Translating low-levelsignals into high-levelknowledge55ExplanationExplanation is the act of choosing the objects or events that bestaccount for a set of observations; often referred to as hypothesisbuilding
    53. 53. Inference to the best explanation• In general, explanation is an abductive problem;and hard to computeFinding the sweet spot between abduction and OWL• Single-feature assumption* enables use ofOWL-DL deductive reasoner* An explanation must be a single feature which accounts forall observed properties56ExplanationExplanation is the act of choosing the objects or events that bestaccount for a set of observations; often referred to as hypothesisbuildingRepresentation of Parsimonious Covering Theory in OWL-DL
    54. 54. ExplanatoryFeature ≡ ∃ssn:isPropertyOf—.{p1} ⊓ … ⊓ ∃ssn:isPropertyOf—.{pn}elevated blood pressureclammy skinpalpitationsHypertensionHyperthyroidismPulmonary EdemaObserved Property Explanatory Feature57ExplanationExplanatory Feature: a feature that explains the set of observedproperties
    55. 55. ObservePropertyPerceiveFeatureExplanationDiscrimination2Focusing attention on thoseaspects of the environmentthat provide usefulinformation58DiscriminationDiscrimination is the act of finding those properties that, ifobserved, would help distinguish between multiple explanatoryfeatures
    56. 56. ExpectedProperty ≡ ∃ssn:isPropertyOf.{f1} ⊓ … ⊓ ∃ssn:isPropertyOf.{fn}elevated blood pressureclammy skinpalpitationsHypertensionHyperthyroidismPulmonary EdemaExpected Property Explanatory Feature59DiscriminationExpected Property: would be explained by every explanatoryfeature
    57. 57. NotApplicableProperty ≡ ¬∃ssn:isPropertyOf.{f1} ⊓ … ⊓ ¬∃ssn:isPropertyOf.{fn}elevated blood pressureclammy skinpalpitationsHypertensionHyperthyroidismPulmonary EdemaNot Applicable Property Explanatory Feature60DiscriminationNot Applicable Property: would not be explained by anyexplanatory feature
    58. 58. DiscriminatingProperty ≡ ¬ExpectedProperty ⊓ ¬NotApplicablePropertyelevated blood pressureclammy skinpalpitationsHypertensionHyperthyroidismPulmonary EdemaDiscriminating Property Explanatory Feature61DiscriminationDiscriminating Property: is neither expected nor not-applicable
    59. 59. Qualities-High BP-Increased WeightEntities-Hypertension-HypothyroidismkHealthMachine SensorsPersonal InputEMR/PHRComorbidity riskscoree.g., CharlsonIndexLongitudinal studiesof cardiovascularrisks- Find risk factors- Validation- domain knowledge- domain expertFind contribution ofeach risk factorRisk Assessment ModelCurrentObservations-Physical-Physiological-HistoryRisk Score(e.g., 1 => continue3 => contact clinic)Model CreationValidate correlationsHistoricalobservationse.g., EMR, sensorobservations62Risk Score: from Data to Abstraction and Actionable Information
    60. 60. Use of OWL reasoner is resource intensive(especially on resource-constrained devices),in terms of both memory and time• Runs out of resources with prior knowledge >> 15 nodes• Asymptotic complexity: O(n3)63How do we implement machine perceptionefficiently on aresource-constrained device?
    61. 61. intelligence at the edgeApproach 1: Send all sensorobservations to the cloud forprocessing64Approach 2: downscalesemantic processing so thateach device is capable ofmachine perception
    62. 62. 010110001101001111001010110001101101101011000110100111100101011000110101100011010011165Efficient execution of machine perceptionUse bit vector encodings and their operations to encode priorknowledge and execute semantic reasoningHenson et al. An Efficient Bit Vector Approach to Semantics-based Machine Perception in Resource-ConstrainedDevices, ISWC 2012.
    63. 63. O(n3) < x < O(n4) O(n)Efficiency Improvement• Problem size increased from 10’s to 1000’s ofnodes• Time reduced from minutes to milliseconds• Complexity growth reduced from polynomial tolinear66Evaluation on a mobile device
    64. 64. 2 Prior knowledge is the key to perceptionUsing SW technologies, machine perception can beformalized and integrated with prior knowledge on theWeb3 Intelligence at the edgeBy downscaling semantic inference, machineperception can execute efficiently on resource-constraineddevices1 Translate low-level data to high-level knowledgeMachine perception can be used to convert low-levelsensory signals into high-level knowledge useful fordecision making67Semantic Perception for smarter analytics: 3 ideas totakeaway
    65. 65. 68Application of semantic perception to healthcare…
    66. 66. kHealthKnowledge-enabled Healthcare69
    67. 67. Through physical monitoring andanalysis, our cellphones could actas an early warning system todetect serious healthconditions, and provide actionableinformationcanary in a coal minekHealth: knowledge-enabled healthcare70Our Motivation
    68. 68. Approach:• Use semantic perception inference• with data from cardio-related sensors• and curated medical background knowledge on theWebto ask the patient contextually relevant questions71kHealth – knowledge-enabled healthcare
    69. 69. • Symptoms: 284• Disorders: 173• Causal Relations: 1944Unified Medical Language SystemCausal Network72Cardiology Background Knowledge
    70. 70. Weight ScaleHeart Rate MonitorBlood PressureMonitor73SensorsAndroid Device(w/ kHealth App)Total cost: < $500kHealth Kit for the application for reducing ADHF readmission
    71. 71. 74• Abnormal heart rate• High blood pressure• Panic Disorder• Hypoglycemia• Hyperthyroidism• Heart Attack• Septic ShockObserved Property Explanatory Featurevia BluetoothExplanation in kHealth
    72. 72. 75Are you feeling lightheaded?Are you have trouble takingdeep breaths?yesyes• Abnormal heart rate• High blood pressure• Lightheaded• Trouble breathing• Panic Disorder• Hypoglycemia• Hyperthyroidism• Heart Attack• Septic ShockContextually dependent questioning based on prior observations(from 284 possible questions)Observed Property Explanatory FeatureFocus in kHealth
    73. 73. 76 at ISWC: C. Henson, K. Thirunarayan, A. Sheth, An Efficient Bit Vector Approach toSemantics-based Machine Perception in Resource-Constrained DevicesA kHealth application leveragelow-cost sensors toward reducinghospital readmissions of ADHFpatientskHealth will continue toward thevision of Physical-Cyber-Socialcomputing tounderstand, correlate, andpersonalize healthcarekHealth Summary
    74. 74. Dr. William Abraham, M.D.Director of CardiovascularMedicine77Pre-clinical usability trial
    75. 75. 78• Asthma, Stress, COPD, Obesity, GI,etc.Other Potential Applications
    76. 76. • Real Time Feature Streams:• kHealth:
    77. 77. PCS Computing for Asthma80
    78. 78. 811 et al. (2009). Status of childhood asthma in the United States, 1980–2007. Pediatrics,123(Supplement 3), S131-S145.Asthma: Severity of the problem25million300million$50billion155,000593,000People in the U.S. arediagnosed with asthma(7 million are children)1.People suffering fromasthma worldwide2.Spent on asthma alonein a year2Hospital admissions in20063Emergency departmentvisits in 20063
    79. 79. 82Specific AimsCan we detect asthma/allergy early?Using data from on-body sensors, and environmental sensorsUsing knowledge from an asthma ontology, generated from asthmaknowledge on the Web and domain expertiseGenerate a risk measure from collected data and background knowledgeCan we characterize asthma/allergy progression?State of asthma patient may change over timeIdentifying risky progressions before worsening of the patient stateDoes the early detection of asthma/allergy, and subsequentintervention/treatment, lead to improved outcomes?Improved outcomes could be improved health (less serioussymptoms), less need for invasive treatments, preventive measures (e.g.avoiding risky environmental conditions), less cost, etc.
    80. 80. Asthma is a multifactorial disease with health signals spanningpersonal, public health, and population levels.83Real-time health signals from personal level (e.g., Wheezometer, NO inbreath, accelerometer, microphone), public health (e.g., CDC, HospitalEMR), and population level (e.g., pollen level, CO2) arriving continuouslyin fine grained samples potentially with missing information and unevensampling frequencies.Variety VolumeVeracityVelocityValueCan we detect the asthma severity level?Can we characterize asthma control level?What risk factors influence asthma control?What is the contribution of each risk factor?semanticsUnderstanding relationships betweehealth signals and asthma attacksfor providing actionable informationMassive Amount of Data to Actions
    81. 81. Detection of events, such aswheezing sound, indoortemperature, humidity, dust, andCO2 levelWeather ApplicationAsthma Healthcare Application84Action in the Physical WorldAsthma Example of ActionableInformationClose the window at home duringday to avoid CO2 inflow, to avoidasthma attacks at nightPublic HealthPersonalPopulationLevel
    82. 82. 85PCS Computing ChallengesVariety: Health signals span heterogeneous sourcesVolume: Health signals are fine grainedVelocity: Real-time change in situationsVeracity: Reliability of health signals may be compromisedPublic HealthPersonalPopulationLevelValue: Can I reduce my asthma attacks atnight?Value: Decision support todoctors by providing them withdeeper insights into patientasthma care
    83. 83. 86Sensordrone – for monitoringenvironmental air qualityWheezometer – for monitoringwheezing soundsCan I reduce my asthma attacks at night?What are the triggers?What is the wheezing level?What is the propensity toward asthma?What is the exposure level over a day?What is the air quality indoors?Commute to WorkPersonalPublic HealthPopulation LevelClosing the window at homein the morning and taking analternate route to office maylead to reduced asthma attacksActionableInformationPCS Computing: Asthma Scenario
    84. 84. ICS= inhaled corticosteroid, LABA = inhaled long-acting beta2-agonist, SABA= inhaled short-acting beta2-agonist ;*consider referral to specialistAsthma Controland Actionable InformationSensors and their observationsfor understanding asthmaPersonal, Public Health, and Population Level Signals for MonitoringAsthma
    85. 85. 88PersonalLevelSignalsSocietal LevelSignals(Personal Level Signals)(PersonalizedSocietal Level Signal)(Societal Level Signals)Societal LevelSignals Relevant tothe Personal LevelPersonal Level Sensors(kHealth**) (EventShop*)Qualify QuantifyActionRecommendationWhat are the features influencing my asthma?What is the contribution of each of these features?How controlled is my asthma? (risk score)What will be my action plan to manage asthma?StorageSocietal Level SensorsAsthma Early Warning Model (AEWM)Query AEWMVerify & augmentdomain knowledgeRecommendedActionActionJustification*, ** Early Warning Model
    86. 86. 89Population LevelPersonalWheeze – YesDo you have tightness of chest? –YesObservationsPhysical-Cyber-Social System Health Signal Extraction Health Signal Understanding<Wheezing=Yes, time, location><ChectTightness=Yes, time, location><PollenLevel=Medium, time, location><Pollution=Yes, time, location><Activity=High, time, location>WheezingChectTightnessPollenLevelPollutionActivityWheezingChectTightnessPollenLevelPollutionActivityRiskCategory<PollenLevel, ChectTightness, Pollution,Activity, Wheezing, RiskCategory><2, 1, 1,3, 1, RiskCategory><2, 1, 1,3, 1, RiskCategory><2, 1, 1,3, 1, RiskCategory><2, 1, 1,3, 1, RiskCategory>...ExpertKnowledgeBackgroundKnowledgetweet reporting pollution leveland asthma attacksAcceleration readings fromon-phone sensorsSensor and personalobservationsSignals from personal, personalspaces, and community spacesRisk Category assigned bydoctorsQualifyQuantifyEnrichOutdoor pollen and pollutionPublic HealthWell Controlled - continueNot Well Controlled – contact nursePoor Controlled – contact doctorHealth Signal Extraction to Understanding
    87. 87. PCS Computing for Parkinson’sDisease90
    88. 88. 911’s Disease (PD): Severity of theproblem10million 60,000$25billion$100,0001 millionPeople worldwide areliving with Parkinsonsdisease1.Americans arediagnosed withParkinsons diseaseeach year1.Spent on Parkinson’salone in a year in theUS1Therapeutic surgerycan cost up to $100,000dollars per patient1.Americans live withParkinson’s Disease1
    89. 89. Parkinson’s disease (PD) data from The Michael J. Fox Foundationfor Parkinson’s Research.921 weeks of data from 5 sensors on a smart phone, collected for 16patientsresulting in ~12 GB (with lot of missing data).Variety VolumeVeracityVelocityValueCan we detect the onset of Parkinson’s disease?Can we characterize the disease progression?Can we provide actionable information to the patientsemanticsMassive Amount of Data to ActionsRepresenting prior knowledge ofPDled to a focused exploration ofthismassive dataset
    90. 90. Massive Amount of Data to ActionsInput: Sensor observations such asacceleration, audio, battery, compass, and GPS fromsmart phonesOutput:Distinguish patients with and without (control group) Parkinson’sCategorize disease progression/evolution over timeCategorize the severity for actionable information
    91. 91. Symptoms to possible manifestations in sensorobservationsMildTremorsrapid change in x,y, and z (unlike speed, the x, y, and z will have highvariance)Poor balancezig-zag movement using GPS? + compassModerateMove slowlyaverage speed of movementMove intermittentlyx, y, and z coordinates do not change overtimeDisturbed sleepsounds in the nightSlower monotone speechlow energy in the voice recordingAdvancedFall pronerapid change in zcoordinate
    92. 92. Feature extraction(accelerometer)ControlgroupPD patientThe movement of the controlgroup person is not restrictedand exhibits active motion withhigh variance in accelerationreadingsThe movement of the PD patientis restricted and exhibits slowmotion with low variance inacceleration readings
    93. 93. ControlgroupPD patientFeature extraction (audio)The speech of the control groupperson is normal and exhibitsgood modulation in audio energylevelThe speech of the PD patient ismonotone and exhibits lowmodulation in audio energy level
    94. 94. ControlgroupPD patientFeature extraction (Compass)The walking direction of thecontrol group person is well-balanced and exhibits equalvariations in all directionsThe walking direction of the PDpatient is not well-balanced andexhibits ―sticky‖ behavior (stuckin one direction)
    95. 95. Evaluation: run classification algorithms on carefully crafted features fromknowledge of PD
    96. 96. 99ParkinsonMild(person) = Tremor(person) ∧ PoorBalance(person)ParkinsonModerate(person) = MoveSlow(person) ∧ PoorSleep(person) ∧MonotoneSpeech(person)ParkinsonAdvanced(person) = Fall(person)Knowledge Based Analytics of PD datasetDeclarative Knowledge of Parkinson’sDisease used to focus our attention onsymptom manifestations in sensorobservationsControlgroupPD patientMovements of an activeperson has a gooddistribution over X, Y, and ZaxisAudio is well modulatedwith good variations in theenergy of the voiceRestricted movements bya PD patient can be seenin the acceleration readingsAudio is not well modulatedrepresented a monotonespeech
    97. 97. 100PCS Computing for Traffic Analytics
    98. 98. 1011 Crisis of Public Transport in IndiaTraffic management: Severity of the problem236%42%285million91%1 billionIncrease in traffic from1981 to 20011. Have stress relatedimplications due totraffic1.People lived in cities inIndia, greater than theentire population of US2Got stuck in traffic withan average delay of 1.3hours in last 3 years1.Cars on road and thisnumber may double by20201
    99. 99. Vehicular traffic data from San Francisco Bay Area aggregated from on-road sensors (numerical) and incident reports (textual)102 minute update of speed, volume, travel time, and occupancyresulting in 178 million link status observations, 738 active events, and146 scheduled events with many unevenly sampled observationscollected over 3 months.Variety VolumeVeracityVelocityValueCan we detect the onset of traffic congestion?Can we characterize traffic congestion based on eventsCan we provide actionable information to decision maksemanticsMassive Amount of Data to Actions(traffic data analytics)Representing prior knowledge oftraffic lead to a focusedexplorationof this massive data stream
    100. 100. 103Heterogeneity leading to complementary observations
    101. 101. Slow movingtrafficLinkDescriptionScheduledEventScheduledEvent511.org511.orgSchedule Information511.org104
    102. 102. 105Uncertainty in a Physical-Cyber-SocialSystemObservation: Slow Moving TrafficMultiple Causes (Uncertain about the cause):- Scheduled Events: music events, fair, theatreevents, concerts, road work, repairs, etc.- Active Events: accidents, disabled vehicles, break down ofroads/bridges, fire, bad weather, etc.- Peak hour: e.g. 7 am – 9 am OR 4 pm – 6 pmEach of these events may have a varying impact on traffic.A delay prediction algorithm should process multimodal and multi-sensory observations.
    103. 103. 106Modeling Traffic EventsInternal (to the road network) observationsSpeed, volume, and travel time observationsCorrelations may exist between these variables across different partsof the networkExternal (to the road network) eventsAccident, music event, sporting event, and planned eventsExternal events and internal observations may exhibit correlations
    104. 104. AccidentMusiceventSportingevent Road WorkTheatre eventExternal events<ActiveEvents, ScheduledEvents>Internal observations<speed, volume, traveTime>WeatherTime of Day107Modeling Traffic Events: Pictorialrepresentation
    105. 105. Domain ExpertsColdWeatherPoorVisibilitySlowTrafficIcyRoadDeclarative domain knowledgeCausalknowledgeLinked Open DataColdWeather(YES/NO)IcyRoad (ON/OFF) PoorVisibility (YES/NO)SlowTraffic (YES/NO)1 0 1 11 1 1 01 1 1 11 0 1 0Domain ObservationsDomainKnowledgeStructure and parameters108WinterSeasonCorrelations to causations usingDeclarative knowledge on theSemantic WebCombining Data and Knowledge Graph
    106. 106. to baseball gametraffic jamtraffic accidenttraffic jamActiveEventScheduledEventCausestraffic jamCausestraffic jamCapableOfslow trafficCapableOfoccur twice each dayCausesis_abad weatherCapableOfslow trafficroad iceCausesaccidentTimeOfDaygo to concertHasSubeventcar crashaccidentRelatedTocar crashBadWeatherCausesCausesis_ais_ais_a is_a is_ais_ais_a109Declarative knowledge from ConceptNet5
    107. 107. Traffic jamLinkDescriptionScheduledEventtraffic jambaseballgameAdd missing random variablesTime of daybad weather CapableOf slow trafficbadweatherTraffic data from sensors deployed onroad network in San Francisco Bay Areatime of daytraffic jambaseball gametime of dayslow trafficThree Operations: Complementing graphical model structureextractionAdd missing links badweathertraffic jambaseball gametime of dayslow trafficAdd link directionbadweathertraffic jambaseball gametime of dayslow trafficgo to baseball game Causes traffic jamKnowledge from ConceptNet5traffic jam CapableOfoccur twice each daytraffic jam CapableOf slow traffic110
    108. 108. 111ScheduledEventActiveEventDay ofweekTime of daydelayTraveltimespeedvolumeStructure extractedformtraffic observations(sensors + textual)using statisticaltechniquesScheduledEventActiveEventDay ofweekTime of daydelayTraveltimespeedvolumeBadWeatherEnriched structure whichhas link directions andnew nodes such as “BadWeather” potentiallyleading to better delaypredictionsEnriched Probabilistic Models using ConceptNet 5Anantharam et al: Traffic Analytics using Probabilistic Graphical Models Enhanced with Knowledge Bases
    109. 109. 114PCS Computing for Intelligence
    110. 110. PhysicalCyberSocialHUMINT:Interpersonal interactionsSecret cellGEOINT:Satellite imagesSIGINT:Mobile CommunicationSIGINT:Sensors on the soldierOSINT:Textual message exchangesSIGINT:On-ground sensorsOSINT:News reports, knowledge bases,intelligence databases,history of undesirable eventsObservations span physical, cyber, and social spacegenerating massive, multimodal, and multisensory observationsPCS Computing for Intelligence
    111. 111. Threat to physical/cyber infrastructureSIGINT:Phone logs indicatingcalling patternsIMINT:Observations fromcontinuous monitoring (e.g., CCTV)OSINT:Communication and groupdynamics on socialmedia and emailsGEOINT:Satellite and drone imagescapturing activities in ageographical areaGEOINT:Satellite imagesSIGINT:Mobile CommunicationSIGINT:Sensors on the soldierOSINT:Textual message exchangesSIGINT:On-ground sensorsOSINT:News reports, knowledge bases,intelligence databases,history of undesirable eventsKnowledgebaseThreatHistoryMulti-Int:Prior knowledge fromhistorical events/threatsPower GridNatural Gas PipelinesWater ReservoirsGovernment FacilitiesData SourcesObservationsScenario: Physical Cyber Social Threat
    112. 112. What physical infrastructure may have threats?SIGINT:Phone logs indicatingcalling patternsIMINT:Observations fromcontinuous monitoring (e.g., CCTV)OSINT:Communication and groupdynamics on socialmedia and emailsGEOINT:Satellite and drone imagescapturing activities in ageographical areaMulti-Int:Prior knowledge fromhistorical events/threatsKnowledgebaseThreatHistoryLocation based knowledgePhysical Infrastructure at a locationSurveillance with high activity (e.g., logs, CCTV)Prior disposition of infrastructure to risksEvents of high risk near the location (e.g., raid of explosives)Who are the suspects?Location based knowledgePeople from watch list and making international callsPeople from watch list and found near infrastructure through surveillanceWhat are the locations currently at high threat levelLocation based knowledgeLocations with physical infrastructure + suspectsLocations where surveillance logs show frequent attacksAnswering these questions demands semanticintegration, annotation, mapping, andinterpretation of massive multi-modal andmulti-sensory observations.ObservationsScenario(along with the knowledge required to answer them)
    113. 113. Prior knowledge fromhistorical events/threatsCharacterizing threats and their severity level using weights learned from dataThreatLevel1 => InternationalCalls ∧ ForeignTravel ∧ WeaponMarket ∧ ExplosiveIngredients w1ThreatLevel2 => InternationalCalls ∧ ForeignTravel ∧ WeaponMarket ∧ ExplosiveIngredients ∧ InWatchList w2ThreatLevel3 => InternationalCalls ∧ ForeignTravel ∧ WeaponMarket ∧ ExplosiveIngredients ∧ InWatchList ∧ CaughtSurveillancew3Unified Semantic representationof observations across multi-modaland heterogeneous observationsKnown threats based on analyzinghistorical threat scenarios and theconditions that persisted during thesethreatsAnomalies in:- Communication- BehaviorSemantic fusion (horizontal operators) of MULTI-Int abstraction (vertical operators) for situation awarenessFilter: semantics-empowered threat detectionExpand: graph based anomaly detectionDecision supportAbnormal signaturesKnowledge baseThreatHistoryWhat physical infrastructure may be at risk?Who are the suspects?What are the locations currently at high threat level?Takeaway Points:- PCS computing is crucial for a holistic interpretation of observations- Horizontal and vertical operators empowers analytics spanning CPS modalities- Semantics-empowered graph based anomaly detection algorithms will detect more anomalie- Leveraging prior knowledge from experts and historical data will allow characterization of thrPhysicalCyberKnowledge baseSocialDomain ExpertObservationsSIGINT:Phone logs indicatingcalling patternsIMINT:Observations fromcontinuous monitoring (e.g., CCTV)OSINT:Communication and groupdynamics on socialmedia and emailsGEOINT:Satellite and drone imagescapturing activities in ageographical areaMulti-Int:Prior knowledge fromhistorical events/threatsPCS computing for detecting threats
    114. 114. There are a variety of sensors used to monitorvitals of soldiers, location, and presence of poisonous gases.O2Heart rateCOCO2GPSAccelerometer CompassFootstepsIntrusion DetectionPCS computing for soldier healthmonitoring- Each soldier is augmented with situation awareness of location, poisonous gases, and vitals.- Cohort health management is crucial for real-time efficient management of stress levels ofsoldiers- Data sent to a mobile control center after initial pre-processing and analysis- Dynamic allocation of units based on their physiological state and stress levels leads to informeddecisions
    115. 115. 120CPS Current State of Art: LimitationsCPS are stovepipe systems with narrow set of observations of the realworld.No knowledge support for informed decision making with Mark’s case.Social aspects crucial for decision making is ignoredThe vision of Physical-Cyber-Social Computing is to provide solutions forthese limitations.
    116. 116. 124ConclusionsTransition form search to solution to PCS computing engines for actionableinformation.Seamless integration of technology involving selective human involvement.Transition from reactive systems (humans initiating information need) toproactive systems (machines initiating information need).Sharing of knowledge, experiences, and observations across physical-cyber-social worlds lead to informed decision making.Physical-Cyber-Social Computing articulates how to achieve this vision.Semantic computing supports integration and reasoning capabilities neededfor PCS computing.
    117. 117. 125J.McCarthyM.WeiserD.EngelbartJ. C. R. Licklider
    118. 118. Influential visions by Bush, Licklider, Eaglebert, and Weiser.Amit Sheth, Pramod Anantharam, Cory Henson, Physical-Cyber-Social Computing: An Early21st Century Approach, IEEE Intelligent Systems, pp. 79-82, Jan./Feb. 2013Amit Sheth,‖Computing for Human Experience: Semantics-EmpoweredSensors, Services, and Social Computing on the Ubiquitous Web,‖ IEEE InternetComputing, vol. 14, no. 1, pp. 88-91, Jan.-Feb. 2010, doi:10.1109/MIC.2010.4A. Sheth, Semantics empowered Cyber-Physical-Social Systems, Semantic Web in 2012workshop at ISWC 2102.Cory Henson, Amit Sheth, Krishnaprasad Thirunarayan, Semantic Perception: ConvertingSensory Observations to Abstractions, IEEE Internet Computing, vol. 16, no. 2, pp. 26-34, Mar./Apr. 2012, doi:10.1109/MIC.2012.20Cory Henson, Krishnaprasad Thirunarayan, Amit Sheth. An Ontological Approach toFocusing Attention and Enhancing Machine Perception on the Web. Applied Ontology, vol.6(4), pp.345-376, 2011.Cory Henson, Krishnaprasad Thirunarayan, and Amit Sheth, An Efficient Bit Vector Approachto Semantics-based Machine Perception in Resource-Constrained Devices, In: Proceedingsof 11th International Semantic Web Conference (ISWC2012), Boston, Massachusetts, USA, November 11-25, 2012.126A bit more on this topic
    119. 119. • OpenSource:• Showcase:• Vision:• Publications:• PCS computing: resources
    120. 120. • Collaborators: U-Surrey (Payam Barnaghi), UCI(Ramesh Jain), DERI, AFRL, Boonshoft Sch of Med –WSU (Dr. Forbis, …), OSU Wexner (Dr. Abraham), andseveral more clinical experts, …• Funding: NSF (esp. IIS-1111183 “SoCS: Social MediaEnhanced Organizational Sensemaking in EmergencyResponse,”), AFRL, NIH, Industry….Acknowledgements
    121. 121. 129Physical Cyber Social ComputingAmit Sheth, Kno.e.sis, Wright State
    122. 122. Amit Sheth’sPHD studentsAshutosh JadhavHemantPurohitVinhNguyenLu ChenPavanKapanipathiPramodAnantharamSujanPereraAlan SmithPramod KoneruMaryam PanahiazarSarasi LalithsenaCory HensonKalpaGunaratnaDelroyCameronSanjayaWijeratneWenboWangKno.e.sis in 2012 = ~100 researchers (15 faculty, ~50 PhD students)
    123. 123. 131thank you, and please visit us athttp://knoesis.orgKno.e.sis – Ohio Center of Excellence in Knowledge-enabled ComputingWright State University, Dayton, Ohio, USAPhysical-Cyber-Social Computing