Situation based analysis and control for supporting Event-web applications

Situation based analysis and control
for supporting Event-web
applications
Vivek Singh
Advisor: Professor Ramesh Jain

Outline
A. Background
B. E2E project
▫ Project overview
▫ Situation based control
▫ Current status/ example
▫ Research challenges
C. Situ-itter
▫ Overview
D. Research Plan

Event-web
• Events and objects as basic organization and
linking mechanism
▫ Multimodal
▫ Closer to real world
• Users gain insights and experiences

Events everywhere– (near future)
• Events are all around us.
▫ Ubiquitous sensors
▫ Excellent signal processing techniques
▫ Wide-spread information broadcast
▫ Excellent data management techniques
• Large volumes of event data, streaming
in real time.
• How can we use it? – machines don’t
understand them.

Motivation: From events to situations…
• Given a plethora of event data. How can we:
▫ Disambiguate relevant and irrelevant events?
▫ Combine events into meaningful representations ?
▫ Allow inference and cascading effects
▫ Support different interpretations based on
application domain
▫ Support Control & decision making

Situation based control: Motivations
1. Inherent support for event-based (temporal)
reasoning
2. The ability of the controller to reason based on
symbols (rather than just signals)
3. Explicit inclusion of domain semantics (to
support multiple applications)

Related Work
Area Sample Event- Symbolic Explicit Decision Focus area
work based inference Domain making
semantics
inclusion
Situation Endsley98 X X Defense/
Awareness Tactical
Situation Yan06 * X Databases
Modeling
Situation Jakobson07 * X X Defense/
Management Tactical
Situational Pospelov86 X Semiotics/
Control Linguistics
Event detection Jain03 X Vision/
Multimedia
Knowledge Sullivan86 X X X Intelligent
based systems systems
Discrete Event Ho89 X X Control theory
Control
Situation McCarthy69 X X * Logic
Calculus
Situation based This work X X X X Symbolic
control Control

Applications
• Energy efficient buildings:
▫ When to switch off air-conditioner?
• Telepresence:
▫ Which camera feed to send out?
• Business analysis:
▫ What should be the correct price for iPhone?
• Earthquake rescue effort:
▫ Where to send out the next fire-fighter engine?

E2E communication: Project Overview
Environment 1 Environment 2

Device to Device
Sentient Sentient
Information
communication
Web Information
System System

Towards Environment to Environment (E2E) multimedia communication systems, in
Multimedia Tools and Applications Journal, Springer Netherlands, 2009.
Also in: ACM Workshop on Semantic Ambient Media Experiences (SAME), ACM
Multimedia workshop, 2008.

Env. 2
Joint SM
Env. 1

JSM 1

JSM 2 Env. 3
Env. 5

Env. 4

Shared Visualization Spaces for Environment to Environment Communication , in
Workshop on Media, Arts, Science and Technology (MAST 09), 2009.

Design Principles
E2E Communication

Bi- Not depend
Natural Semantic Seamless directional on physical Handle
interaction interaction interaction connectivity similarity privacy

Design Implications
Event-based Multimodal Sensor Scalable No fixed Live and
architecture information abstraction architecture application archived
modes

Environment: Node Architecture

EventBase
Sensors

Situation
Physical Environment Environment Network/
MMDB based
Environment Model Server Transmis
controller
sion
Actuators /
Presentation Actuator /
Devices Presentation
Model

Situation Calculus: Quick overview
▫ enter(P1), startWork(P1)
▫ enter(P1), exit(P1), enter(P1), startWork(P1),
stopWork(P1), startWork(P1)
- isInRoom(P1, s(k))
- isWorking(P1, s(k))

isInRoom(P1, s) 1
0
isWorking(P1, s) 1
0

isInRoom(P1, s) ˄~isWorking(P1, s) →
IncreaseMusicVolume()
Situation = Not events , nor sequence of events,
but their assimilated descriptor

Situation calculus: Basics (1/3)
• Logic formalism designed for representing and
reasoning about dynamical domains.
• It builds upon traditional predicate, 1st and 2nd
order calculus, but is different because it allows
for truth values to change over time.
• Situation:
▫ “The set of necessary and sufficient world state
descriptors for undertaking control decision”.

Situation Calculus: Basics (2/3)
• Ω = {A, S, O, F}
▫ Actions (A) for actions i.e. those which change the
'state’ of the world. A= Aex U Asys
▫ Situation (S) for `history of events' ,
▫ Objects (O) as the default sort for everything else,
▫ Fluents (F) are predicates reified with situations.
(value assignments which change with time).
 Relational (give True/False answers) or
 Functional (return any value as computed)
• Do(action, situation): A X S → S

Situation Calculus: Basics (3/3)
• D = Dfnd U Duna U ε U Dap U Dss U D0
▫ Dap is a set of action precondition axioms, one per
action symbol A.
▫ Dss is a set of successor state axioms (SSAs), one
for each fluent symbol f, which characterizes all the
ways the value of a particular fluent can be changed.
Poss(a, s) → [F(x, do(a, s)) ↔ γ+F(x, a, s) ˄ (
(F(x,
s)˄
γ-F (x, a, s))]
▫ D0 is a set of axioms describing the initial
situation S0.

Control theoretic problem formulation

• •
• •
• •

Implementing the controller
• →
•

•Φ →α
•
•Φ →α
• Φ α Φ α
→

Implementing the controller
Situation Based Controller
A. Inference
Engine

B. Knowledge Base

C. System Goal

D’ = D U Dca
D’ = Dfnd U Duna U ε U Dap U Dss U D0 U Dca

Situation modeling
1. Identify the relevant Objects (O) , Actions (A)
and Fluents (F)
2. Identify the preconditions for each action (Dap)
3. Identify the after-effects of each action (Dss)
4. Describe the initial situation (D0)
5. Identify the goal state using action-condition
constraints (Dca)

Situation modeling: E2E application
Loc 1: Desk Loc2: Whiteboard Conditions Actions

Move to Activity Selected Desired
location Cam Volume

Desk WorkOn 1 1
Actions possible: PC
1. Work on PC
2. Work on Table Desk WorkOn 2 2
Table

Whitebo - 3 3
ard

User Model - 4 4
Loc 3: Engineering
Model

Situation based control for cyber physical environments, Accepted: IEEE
workshop on situation management, MILCOM, 2009

Step 1: Identify the relevant Objects,
Actions and Fluents.

Step 2: Identify the preconditions for
each action

Step 3: Identify the after-effects of
each action

Step 4: Describe the initial situation

Step 5: Identify the goal-state using
the action-condition constraints

Finding the optimal control action

Sample executions
DecreaseVolume, DecreaseVolume,
DecreaseVolume, S0

• Exogenous action: MoveToLoc(`Model’) at the
end of second cycle
IncreaseVolume, IncreaseVolume,
SelectCam(4) MoveToLoc(`Model’),
DecreaseVolume, DecreaseVolume, S0

Research Challenge 1: Generic
adaptability
• Tools to allow system designers to undertake
their domain’s situation modeling
• Necessary and sufficient details for handling
application
• Discrete, hybrid or continuous
• Current status:
▫ Dap U Dss U D0 U Dca
• To Do
▫ Providing easy tools for users to inscribe such
domain knowledge

Research Challenge 2: Enhanced
sensing based on feedback
• Top down+ bottom up sensing
▫ Sensing = F(current_state)
• Detect and discard noisy event data.
▫ Only allow valid sequences of input events
▫ Invalid(Seq) ↔(KB U S0 |= ¬Seq)
▫ Discard (WearSocks >(T) WearShoes)
• Anomaly detection using these techniques
▫ Event based (semantic) level not signal level

Research Challenge 3: Reasoning and
analysis
• Minimal representation: Find the minimal set of
events Emin which lead the situation changing
from S0 to SGoal.
• Handling un-observable systems:
▫ Can we find the unknown state S0, by looking at
patterns of events and the changes in the system
state (fluents) [e.g. in Chess]
• Approach:
▫ Using planning and projection operators of
situation Calculus

Research Challenge 4: Using Predictive
Analysis for control action
• Using estimates of future exogenous actions for
better control
• Signal based data
▫ Kalman Filter
▫ Model Predictive Control
• Symbolic data
▫ Semantic Kalman filter?

“Coopetitive multi-camera surveillance using Model Predictive Control”, Machine
Vision and Applications Journal, 2008.

Situ-itter: Looking beyond rooms…
• Can an entire city or country be considered a
cyber physical system.
• Humans as sensors:
▫ Everywhere !
▫ Perception, Censors, Rumors, Delays
• Applications
▫ Should iPhone price be increased/decreased?
▫ Detect swine flu in Mexico ->> Issue pork-import
health warnings in Alaska
▫ DEMO

Research Challenge 5: Scalability of
situation based control
• Number of Events and conditions to be considered
▫ Hierarchical approach
• Supporting multiple applications with different
complexity levels
▫ Creating models for different applications
• Approaches:
▫ Allow users to define models
▫ Learn patterns
▫ Use public knowledge/ Ontologies

Outline
A. Background
B. E2E project
▫ Project overview
▫ Situation based control
▫ Current status/ example
C. Situ-itter
▫ Overview
D. Summary and Plan ahead

Current status: Systems
• E2E project
▫ Working prototypes
 DBH2059, CalIT2
▫ Skype based lite-version
▫ Collaborative nodes
 National university of Singapore (Observation System)
 INRIA, France (emotion enhanced E2E)
• Situ-itter
▫ Proof-of-concept
• Multimodal observation systems, ACM Multimedia 2008.
• ObSys: A Generic Sensing Architecture for Multimodal Observation Systems, Submitted to
TOMCCAP: ACM Transactions on Multimedia Computing, Communications and Applications
• Toward Environment-to-Environment (E2E) Affective Sensitive Communication Systems,
submitted to: MTDL workshop, ACM-MM, 2009.

Future work: Systems
• Robust bi-directional E2E communication
between UCI, and Singapore
• Implementing situation controller into physical
sensors
• Building Twitter crawler/ real-time analysis tool

Area Challenges Status Type of Approach
contribution
(expected)
Overall Temporal + Symbolic Prelim. Tools Situation Calculus
Framework reasoning
Use domain semantics Prelim. Tools Situation Modeling

Generic & Support Multiple Prelim. Tools -User tools
Scalable applications /Plan -Learning
-Ontologies
Large number of events Plan Tools Hierarchical Control

Reasoning and Minimal event set Plan Logic-based Min (Seq) : Do(Seq, S0)
Analysis -> Sgoal
Partial Observability Plan Logic-based S0: Do(Seq, S0) -> Sgoal

Feedback Noisy event data , Plan Logic-based Invalid (Seq)<-> KB U
enhanced anomalies S0 |= ¬Seq
sensing
Top-down + bottom up Plan Optimality Sensing =F(S_curr)
sensing
Predictive Sensor/ device selection Plan Optimality Symbolic Kalman
Control Filter+ Model
Predictive Control

Research Plan
• In progressing order of importance for my work
• Year 3 --Tools
▫ Finalize overall framework
▫ Make it generic and scalable
• Year 4 – Logic based approaches
▫ Use inference, reasoning and analysis
▫ Feedback enhanced sensing
• Year 5 – Optimality based contributions
▫ Predictive Control

Publications
• E2E
1. {VKS, HP, IR, RJ}: Towards Environment to Environment
(E2E) multimedia communication systems, in Multimedia
Tools and Applications Journal, Springer Netherlands, 2009.
2. {VKS, HP, IR, RJ}: Also in: ACM Workshop on Semantic
Ambient Media Experiences (SAME), ACM Multimedia
workshop, 2008.
3. {VKS, IR, RJ}:User availability detection in E2E systems, in
Workshop on Media, Arts, Science and Technology (MAST 09),
2009.
4. {HP, VKS, AM, RJ}: Shared Visualization Spaces for
Environment to Environment Communication , in Workshop
on Media, Arts, Science and Technology (MAST 09), 2009.
5. {IR, VKS, HP, RJ}: Environment to Environment (E2E)
communication systems for collaborative work, Poster in
Computer Supported Cooperative Work (CSCW) 2008.
VKS=Vivek Singh, HP=Hamed Pirsiavash, IR=Ish Rishabh,
AM=Aditi Majumder, RJ=Ramesh Jain

Publications
• Situation based control
1. {VKS, RJ}: Situation based control for cyber physical environments,
Accepted: IEEE workshop on situation management, MILCOM, 2009

• With external collaborators
1. {MS,VKS, RJ, MK}: Multimodal observation systems, ACM
Multimedia 2008.
2. {MP,VKS, BH,RJ}:“Toward Environment-to-Environment (E2E)
Affective Sensitive Communication Systems”, MTDL workshop,
ACM-MM, 2009.
3. {MS,VKS, RJ, MK}: ObSys: A Generic Sensing Architecture for
Multimodal Observation Systems, Submitted to TOMCCAP: ACM
Transactions on Multimedia Computing, Communications and
Applications
4. {VKS, RJ, MK}: Motivating contributors in Social media networks,
submitted to: ACM MM workshop on Social media.
VKS=Vivek Singh, RJ=Ramesh Jain, MS=Mukesh Saini, MK=Mohan
Kankanhalli, MP=Marco Paleari, BH=Benoit Huet

Publications
• Prior work: Master’s thesis
1. “Coopetitive multi-camera surveillance using Model Predictive Control”.
Journal of Machine Vision and Applications, 2009.
2. Adversary aware surveillance systems, IEEE TIFS, Trans. Info. Forensics and
Security, 2009.
3. “Coopetitive Multimedia Surveillance”, International Conference on
Multimedia Modeling (MMM'2007).
4. "Towards adversary aware surveillance systems", IEEE International
Conference on Multimedia and Expo, (ICME-2007).
5. A Design Methodology for Selection and Placement of Sensors in Multimedia
Surveillance Systems”, ACM Multimedia Workshop on Video Surveillance and
Sensor Networks (ACM MM, workshop-VS SN 06)
6. “Coopetitive Visual Surveillance using Model Predictive Control”, (ACM-
Multimedia, workshop-VSSN 05)
• Journals (3 accepted, 1 submitted),
• Conferences (4),
• ACM-MM workshops (5),
• Other venues (3)

Situation based analysis and control for supporting Event-web applications

Recommended

Recommended

More Related Content

What's hot

What's hot (6)

Similar to Situation based analysis and control for supporting Event-web applications

Similar to Situation based analysis and control for supporting Event-web applications (20)

Recently uploaded

Recently uploaded (20)

Situation based analysis and control for supporting Event-web applications

Editor's Notes