1) The document discusses a semantics-based approach to machine perception that uses semantic web technologies to derive abstractions from sensor data using background knowledge on the web.
2) It addresses three primary issues: annotation of sensor data, developing a semantic sensor web, and enabling semantic perception intelligence at the edge on resource-constrained devices.
3) The approach represents background knowledge and sensor observations using ontologies, and uses deductive and abductive reasoning over these representations to interpret sensor data at multiple levels of abstraction.
A Critique of the Proposed National Education Policy Reform
A Semantics-based Approach to Machine Perception
1. A SEMANTICS-BASED APPROACH
TO MACHINE PERCEPTION
Cory Andrew Henson
August 27, 2013
1
Committee:
Amit Sheth (advisor)
Krishnaprasad Thirunarayan
John Gallagher
Payam Barnaghi
Satya Sahoo
Ph.D. Dissertation Defense
Wright State University
Dept. of Computer Science and Engineering
3. 3
Thesis
Machine perception can be formalized using semantic web technologies
in order to derive abstractions from sensor data using background
knowledge on the Web, and efficiently executed on resource-constrained
devices.
4. 4
3 primary issues to be addressed
Annotation of
sensor data
Semantic
Sensor
Web
Semantic
Perception
Intelligence
at the Edge
Interpretation of
sensor data
Efficient execution on
resource-constrained devices
1 2 3
5. 5
lives in
has pet
is ahas pet
Person Animal
Concrete Facts
Resource Description Framework
Semantic Web
(according to Farside)
General Knowledge
Web Ontology Language
“Now! – That should clear up a few things around here!”
is a
11. 11
With freedom comes responsibility
1. discovery, access, and search
2. integration and interpretation
We want to set this data free
12. 12
RDF OWL
How are machines supposed to integrate and interpret sensor data?
Semantic Sensor Networks (SSN)
13. 13
W3C Semantic Sensor Network Ontology
Lefort, L., Henson, C., Taylor, K., Barnaghi, P., Compton, M., Corcho, O., Garcia-Castro, R., Graybeal, J., Herzog, A., Janowicz, K.,
Neuhaus, H., Nikolov, A., and Page, K.: Semantic Sensor Network XG Final Report, W3C Incubator Group Report (2011).
14. 14
W3C Semantic Sensor Network Ontology
Lefort, L., Henson, C., Taylor, K., Barnaghi, P., Compton, M., Corcho, O., Garcia-Castro, R., Graybeal, J., Herzog, A., Janowicz, K.,
Neuhaus, H., Nikolov, A., and Page, K.: Semantic Sensor Network XG Final Report, W3C Incubator Group Report (2011).
15. 15
W3C Semantic Sensor Network Ontology
Lefort, L., Henson, C., Taylor, K., Barnaghi, P., Compton, M., Corcho, O., Garcia-Castro, R., Graybeal, J., Herzog, A., Janowicz, K.,
Neuhaus, H., Nikolov, A., and Page, K.: Semantic Sensor Network XG Final Report, W3C Incubator Group Report (2011).
16. 16
Semantic Annotation of SWE
Lefort, L., Henson, C., Taylor, K., Barnaghi, P., Compton, M., Corcho, O., Garcia-Castro, R., Graybeal, J., Herzog, A., Janowicz, K.,
Neuhaus, H., Nikolov, A., and Page, K.: Semantic Sensor Network XG Final Report, W3C Incubator Group Report (2011).
17. Semantic Sensor Observation Service (SemSOS)
Cory Henson, Josh Pschorr,Amit Sheth, Krishnaprasad Thirunarayan, SemSOS: Semantic Sensor Observation Service, In Proceedings of the
2009 International Symposium on Collaborative Technologies and Systems (CTS 2009), Baltimore, MD, May 18-22, 2009. 17
18. Semantic Sensor Observation Service (SemSOS)
Joshua Pschorr, Cory Henson, Harshal Patni, and Amit P. Sheth. Sensor Discovery on Linked Data. Kno.e.sis Center Technical Report 2010. 18
19. 19
3 primary issues to be addressed
Annotation of
sensor data
Semantic
Sensor
Web
Semantic
Perception
Intelligence
at the Edge
Interpretation of
sensor data
Efficient execution on
resource-constrained devices1 2 3
20. 20
Semantic Perception
• The role of perception is to transform raw sensory
data into a meaningful and correct representation
of the external world.
• The systematic automation of this ability is the
focus of machine perception.
• For correct interpretation of this representation,
we need a formal account of how this is done.
21. 21
What can we learn from
cognitive models of perception?
People are good at
making sense
of sensory input
22. 22
Perception is an active,
cyclical process of exploration
and interpretation.
The perception cycle is driven by
prior knowledge, in order to generate
and test hypotheses.
Some observations are more
informative than others
(in order to effectively test
hypotheses*).
Ulric Neisser
Richard Gregory
Kenneth Norwich
1970’s 1980’s 1990’s
Cognitive Models of Perception
* Applies to machine perception within Intellego, NOT human perception.
26. The Patient of the Future
MIT Technology Review, 2012
26http://www.technologyreview.com/featuredstory/426968/the-patient-of-the-future/
27. Digital Doctor
Let’s provide people with the
tools needed to monitor and
manage their own health
http://worldofdtcmarketing.com/mobile-health-apps-a-new-opportunity-for-healthcare-marketers/mobile-healthcare-marketing-trends/
27
32. SSN
Ontology
2 Interpreted data
(deductive)
[in OWL]
e.g., threshold
1 Annotated Data
[in RDF]
e.g., label
0 Raw Data
[in TEXT]
e.g., number
Levels of Abstraction
3 Interpreted data
(abductive)
[in OWL]
e.g., diagnosis
Intellego
―150‖
Systolic blood pressure of 150 mmHg
Elevated
Blood
Pressure
Hyperthyroidism
……
32
33. Observed
Properties
Perceived
Features
33
Background knowledge
on the Web
Low-level observed properties suggest
explanatory hypotheses through abduction
Explanation
Focus
Ontology of Perception
An Ontological Approach to Focusing Attention and Enhancing Machine Perception on the Web (Applied Ontology, 2011)
34. 34
Semantics of Explanation
Abduction – or, inference to the best EXPLANATION
Task
• Given background knowledge of the environment (SIGMA), and
• given a set of sensor observation data (RHO),
• find a consistent explanation of the situation (DELTA)
Background
knowledge Features
(objects/events)
in the world
Sensor
observation data
35. 35
Semantics of Explanation
Background knowledge is represented as a causal network between features (objects
or events) in the world and the sensor observations they give rise to.
36. 36
Semantics of Explanation
Finding the sweet spot between abduction and OWL
• Simulation of Parsimonious Covering Theory in
OWL-DL (using the single-feature assumption*)
* An explanation must be a single feature which accounts for
all observed properties
Semantic Perception: Converting Sensory Observations to Abstractions (IEEE Internet Computing, 2012)
Theorem: Given a PCT problem P and its translation o(P) into OWL, Δ = {e} is a PCT
explanation if and only if ExplanatoryFeature(e) is deduced by an OWL-DL reasoner
— that is, if and only if o(P) ⊧ ExplanatoryFeature(e).
37. 37
Finding the Sweet Spot
minimize
explanations
degrade gracefully
w/ incomplete info
decidable
web reasoning
Abductive Logic (e.g., PCT)
high complexity
Deductive Logic (e.g., OWL)
(relatively) low complexity
Explanation
(in Intellego)
38. Explanatory Feature: A feature is explanatory w.r.t. a set of observed properties if it
causes each property in the set.
ExplanatoryFeature ≡ ∃isPropertyOf—.{p1} ⊓ … ⊓ ∃isPropertyOf—.{pn}
elevated blood pressure
clammy skin
palpitations
Hypertension
Hyperthyroidism
Pulmonary Edema
Observed Property Explanatory Feature
38
Semantics of Explanation
39. Observed
Properties
Perceived
Features
39
Background knowledge
on the Web
Hypotheses imply the informational value of
future observations through deduction
Explanation
Focus
Ontology of Perception
An Ontological Approach to Focusing Attention and Enhancing Machine Perception on the Web (Applied Ontology, 2011)
40. Universe of observable properties
40
Semantics of Focus
To predict which future observations have informational value, find those observable
properties that can discriminate between the set of hypotheses.
Expected
Properties
Not-applicable
Properties
Discriminating
Properties
41. Expected Property: A property is expected w.r.t. a set of features if it is caused by each
feature in the set.
ExpectedProperty ≡ ∃isPropertyOf.{f1} ⊓ … ⊓ ∃isPropertyOf.{fn}
elevated blood pressure
clammy skin
palpitations
Hypertension
Hyperthyroidism
Pulmonary Edema
Expected Property Explanatory Feature
41
Semantics of Focus
42. Not Applicable Property: A property is not-applicable w.r.t. a set of features if it is not
caused by any feature in the set.
NotApplicableProperty ≡ ¬∃isPropertyOf.{f1} ⊓ … ⊓ ¬∃isPropertyOf.{fn}
elevated blood pressure
clammy skin
palpitations
Hypertension
Hyperthyroidism
Pulmonary Edema
Not Applicable Property Explanatory Feature
42
Semantics of Focus
43. Discriminating Property: A property is discriminating w.r.t. a set of features if it is
neither expected nor not-applicable.
DiscriminatingProperty ≡ ¬ExpectedProperty ⊓ ¬NotApplicableProperty
elevated blood pressure
clammy skin
palpitations
Hypertension
Hyperthyroidism
Pulmonary Edema
Discriminating Property Explanatory Feature
43
Semantics of Focus
44. Off-the-shelf OWL-DL reasoners are too
resource intensive in terms of both
memory and time
• Runs out of resources with
background knowledge >> 20 nodes
• Asymptotic complexity: O(n3)
44
O(n3) < x < O(n4)
Semantic perception on resource-constrained devices
45. 45
3 primary issues to be addressed
Annotation of
sensor data
Semantic
Sensor
Web
Semantic
Perception
Intelligence
at the Edge
Interpretation of
sensor data
Efficient execution on
resource-constrained devices1 2 3
50. Approach 1: Send all sensor
observations to the cloud for
processing
50
Approach 2: downscale semantic
processing so that each device is
capable of machine perception
Intelligence at the Edge
51. Use bit vector encodings and their operations to encode background knowledge and
execute perceptual inference
51
Efficient execution of semantic perception
01100011
10001110
11001110
01010110
01110101
OWL-DL
An Efficient Bit Vector Approach to Semantics-based Machine Perception in Resource-Constrained Devices (ISWC, 2012)
53. 53
Efficient execution of semantic perception
Bit vector algorithms are provably equivalent to the OWL inference (i.e., semantics
preserving)
Intuition: discover and dismiss those features that
cannot explain the set of observed properties.
Intuition: discover and assemble those properties
that discriminate between the explanatory features
54. bp 1
cs 0
pa 1
HN HM PE
bp 1 1 1
cs 0 1 0
pa 1 1 0
HN HM PE
1 1 1
HN HM PE
1 1 1
1 1 0
AND =>
AND
1 1 1
Observed
Property Prior Knowledge
Previous
Explanatory Feature
Current
Explanatory Feature
=>
INTUITION: The strategy employed relies on the use of the bit vector AND operation to discover and dismiss
those features that cannot explain the set of observed properties.
54
Explanation: efficient algorithm
55. bp 1
cs 0
pa 1
HN HM PE
bp 0 1 1
cs 0 1 0
pa 1 1 0
HN HM PE
1 1 0
HN HM PE
0 1 0AND => 0 1 0
Observed
Property Prior Knowledge
Previous
Explanatory Feature
Current
Explanatory Feature
=
INTUITION: The strategy employed relies on the use of the bit vector AND operation to discover and assemble
those properties that discriminate between the explanatory features
bp 0
cs 0
pa 0
Discriminatin
g
Property
1 1 0
… expected?
0 0 0
… not-applicable?
ZERO Bit Vector
0 1 0
=
=> FALSE
=> FALSE
1
Is the property
discriminating?
55
Focus: efficient algorithm
56. O(n3) < x < O(n4) O(n)
56
Evaluation on a mobile device
Efficiency Improvement
• Problem size increased from 10’s to 1000’s of nodes
• Time reduced from minutes to milliseconds
• Complexity growth reduced from polynomial to linear
57. 57
Technical contributions in a nutshell
1. Semantic Sensor Web: Developed technologies for the semantic annotation
of sensor data on the Web
- Semantic Sensor Web (IEEE Internet Computing, 2008) – 276 citations (as of Aug. 2013)
- SemSOS: Semantic Sensor Observation Service (International Symposium on Collaborative Technologies and Systems, 2009)
- Semantic Sensor Network XG Final Report (W3C Incubator Group Report, 2011)
2. Semantic Perception: Designed a declarative specification of perception,
capable of utilizing an off-the-shelf OWL-DL reasoner
- An Ontological Approach to Focusing Attention and Enhancing Machine Perception on the Web (Applied Ontology Journal,
2011)
- Semantic Perception: Converting Sensory Observations to Abstractions (IEEE Internet Computing: Special Issue on Context-
Aware Computing, 2012) – most downloaded paid paper in IEEE-IC 2012
3. Intelligence at the Edge: Implemented efficient algorithms for executing
perceptual inference on resource-constrained devices
- An Efficient Bit Vector Approach to Semantics-based Machine Perception in Resource-Constrained Devices (International
Semantic Web Conference, 2012)
58. 58
W3C Reports
1. Semantic Sensor Network XG Final Report (W3C Incubator Group Report, 2011)
Journal Publications
1. Physical-Cyber-Social Computing: An Early 21st Century Approach (IEEE Intelligent Systems, 2013)
2. Semantic Perception: Converting Sensory Observations to Abstractions (IEEE Internet Computing, 2012)
3. Semantics for the Internet of Things: Early Progress and Back to the Future (International Journal on Semantic Web and Information Systems, 2012)
4. The SSN ontology of the W3C semantic sensor network incubator group (Web Semantics: Science, Services and Agents on the World Wide Web, 2012)
5. An Ontological Approach to Focusing Attention and Enhancing Machine Perception on the Web (Applied Ontology, 2011)
6. Semantic Sensor Web (IEEE Internet Computing, 2008)
Conference Publications
1. An Efficient Bit Vector Approach to Semantics-based Machine Perception in Resource-Constrained Devices (International Semantic Web Conference,
2012)
2. Computing Perception from Sensor Data (IEEE Sensors Conference, 2012)
3. SemSOS: Semantic Sensor Observation Service (International Symposium on Collaborative Technologies and Systems, 2009)
4. Situation Awareness via Abductive Reasoning for Semantic Sensor Data: A Preliminary Report (International Symposium on Collaborative
Technologies and Systems, 2009).
Workshop Publications
1. SECURE: Semantics Empowered Rescue Environment (International Workshop on Semantic Sensor Networks, 2011)
2. Representation of Parsimonious Covering Theory in OWL-DL (International Workshop on OWL: Experiences and Directions, 2011)
3. Provenance Aware Linked Sensor Data (Workshop on Trust and Privacy on the Social and Semantic Web, 2010)
4. Linked Sensor Data (International Symposium on Collaborative Technologies and Systems, 2010)
5. A Survey of the Semantic Specification of Sensors (International Workshop on Semantic Sensor Networks, 2009)
6. An Ontological Representation of Time Series Observations on the Semantic Sensor Web (International Workshop on the Semantic Sensor Web)
7. Video on the Semantic Sensor Web (W3C Video on the Web Workshop, 2007)
Relevant Publications
60. Heart disease is a critical issue
~815,000 (2011)
http://millionhearts.hhs.gov/abouthds/cost-consequences.html 60
61. Acute Decompensated Heart Failure (ADHF)
• Affects nearly 6 million people (in U.S.)
• 555,000 new cases are diagnosed each year
61U.S. Department of Health & Human Services. (2011). Hospital Compare. http://www.hospitalcompare.hhs.gov (Accessed on February 19, 2012).
62. • 4.8 million hospitalizations per year
• 50% are readmitted within 6 months
• 25% are readmitted within 30 days
• 70% due to worsening conditions
• Costing $17 billion per year
ADHF hospital readmission rates are too high
62U.S. Department of Health & Human Services. (2011). Hospital Compare. http://www.hospitalcompare.hhs.gov (Accessed on February 19, 2012).
63. Congress has incentivized hospitals to lower readmission rates
63U.S. Department of Health & Human Services. (2011). Hospital Compare. http://www.hospitalcompare.hhs.gov (Accessed on February 19, 2012).
65. Score (0: Not at all, 1: A little, 2: A great deal, 3: Extremely)
Heart Failure Somatic Awareness Scale (HFSAS)
Current state-of-the-art
65
66. kHealth – knowledge-enabled healthcare
Approach:
• Use semantic perception inference
• with data from cardio-related sensors
• and curated medical background knowledge on the Web
1. to monitor and abstract health conditions
2. to ask the patient contextually relevant questions
66
67. Cardiology Background Knowledge
• Symptoms: 284
• Disorders: 173
• Causal Relations: 1944
Unified Medical Language System
Causal Network
67
69. 69
Explanation in kHealth
• Abnormal heart rate
• High blood pressure
• Panic Disorder
• Hypoglycemia
• Hyperthyroidism
• Heart Attack
• Septic Shock
Observed Property Explanatory Feature
via Bluetooth
70. 70
Focus in kHealth
Are you feeling lightheaded?
Are you have trouble taking deep breaths?
yes
yes
• Abnormal heart rate
• High blood pressure
• Lightheaded
• Trouble breathing
• Panic Disorder
• Hypoglycemia
• Hyperthyroidism
• Heart Attack
• Septic Shock
Contextually dependent questioning based on prior observations
(from 284 possible questions)
Observed Property Explanatory Feature
71. 71
Evaluation of kHealth
Evaluate the ability to discriminate between sets of potential disorders
using:
1. HFSAS/WANDA’s restricted set of
observable symptoms (12)
2. kHealth’s more comprehensive set of
observable symptoms (284)
72. 72
Evaluation of kHealth
Evaluation Metrics:
1. Efficiency: How many observations (or questions) required to minimize the
set of explanations?
2. Specificity: How specific is the resulting minimum set of explanations (i.e.,
how many explanatory disorders in the set)?
Explanatory Disorders
(computed by Intellego)
Actual Disorder
(extracted from EMR)
Possible Disorders
(derived from cardiology KB)
73. 73
Evaluation of kHealth – Early Results
HFSAS/WANDA
Efficiency: ~7.45 (# questions asked)
Specificity: ~11.95* (# minimum explanations)
* Converged to 1 explanation 20% of the time
• 496 EMRs
• ~3.2 diagnosed disorders
per EMR
• 173 possible disorders
The approach utilized by kHealth is
more efficient and more specific
than HFSAS/WANDA.
kHealth
Efficiency: ~7.28 (# questions asked)
Specificity: 1 (# minimum explanations)
Explanatory Disorders
(computed by Intellego)
Actual Disorder
(extracted from EMR)
Possible Disorders
(derived from cardiology KB)
79. A SEMANTICS-BASED APPROACH
TO MACHINE PERCEPTION
Cory Andrew Henson
August 27, 2013
79
Thank you.
For additional information visit:
http://knoesis.org/researchers/cory
Editor's Notes
- Larry Smarr is a professor at the University of California, San DiegoAnd he was diagnosed with Chrones DiseaseWhat’s interesting about this case is that Larry diagnosed himselfHe is a pioneer in the area of Quantified-Self, which uses sensors to monitor physiological symptomsThrough this process he discovered inflamation, which led him to discovery of Chrones DiseaseThis type of self-tracking is becoming more and more common