1. SITUATION RECOGNITION:
AN EVOLVING PROBLEM FOR HETEROGENEOUS
DYNAMIC BIG MULTIMEDIA DATA
Vivek K. Singh1,2, Mingyan Gao1, Ramesh Jain1
1 University of California, Irvine
2 MIT Media Lab
Presenter: jain@ics.uci.edu

2. 2
Sandy in New York: Situation today
Weather
forecast
Imagine What do
This I do?

3. An Interesting Problem
When we were data poor
– we searched for words
in documents.
Now that we are data rich
– should we still search
for words?
Time has come for us to stop thinking data poor;
really start thinking and behaving data rich.

4. 4
Data, Information, Knowledge, Wisdom
Data is Essential.
But, we are really interested in products:
Information,
Knowledge, and
Wisdom.

5. 5
BIG DATA
Variety
Volume
Big Data offers Big Opportunities.

6. 6
The Grand Challenge
Sense making from multimodal
massive geo-social data-
streams.

7. Social Networks
Connecting
People

8. Fundamental Problem
Connecting People to Resources
effectively, efficiently, and promptly
in given situations.

9. 11/1/12 9
Social Life Networks
Connecting Information
People
Aggregation Situation Alerts
and Detection
CompositionAnd
Queries
Resources

10. 10
Concept Recognition: Last Century
Location Scenes
Environ Trajectories
Situations
Single Media
aware ments
Location Visual
Real world Visual
Objects
Objects Activities
Events
unaware
Static Dynamic
SPACE
TIME
Data = Text or Images or Video

11. Visual Concept Recognition: First research papers
Trajectory Situation
Object Scene Event
1960 1970 1980 1990 2000 2010
• 1963: Object Recognition [Lawrence + Roberts]
• 1967: Scene Analysis [Guzman]
• 1984: Trajectory detection [Ed Chang+ Kurz]
• 1986: Event Recognition [Haynes + Jain]
• 1988: Situation Recognition [Dickmanns]

12. 12
Concept Recognition: This Century
Heterogeneous Media
Location Environ
Situations
aware ments
Location Real world
Objects Activities
unaware
Static Dynamic
SPACE
TIME
Data is just Data.
Medium and sources do not matter.

13. 13
Situations: Definition
An actionable abstraction of
observed spatio-temporal
characteristics.

14. 11/1/12 14

15. Overall framework 15
A) Situation B) Situation C) Visualization,
Modeling Recognition Personalization, and Alerts
i) Visualization
C1
…
⊕
v2 v3 Personal
context ii) Personalization
Personal
v5 v6 ized
STT situation
Stream
Available
resources
Emage
iii) Alerts
Situation

16. 16
Challenge: Unifying Multimodal Big Data
• Spatio-temporal-thematic (STT) real-time streams
• E-mage as a unifying representation
(a) Pollen levels (Source: Visual) (b) Census data (Source: text file) (c) Reports on ‘Hurricanes’ (source: Twitter stream)
d) Cloud cover (Source: Satellite imagery) (e) Predicted hurricane path (source: KML) (f) Open shelters coverage(Source: KML)

17. 17
Situation Modeling
Get_components (v){ USA,
5 mins,
v ϵ { Low,
Mid,
1) Identify output state space 0.01x 0.01 f1 High}
2) Identify S-T bounds
3) Define component features: v2 v3 v4
v=f(v1, …, vk) @
f2 ∏
• If (type = imprecise)
• identify learning data source, method Emage
v5 v6
Emage
@
4) ForEach (feature vi) { Δ
∏
Δ
If (atomic)
• Identify Data source. D1 Emage Emage
D2
• Type, URL, ST bounds Δ Δ
• Identify highest Rep. level reqd.
D2 D3
• Identify operations
Else
Get_components(vi)
}
}

18. 18
Situation recognition: Workflow
Level
0:
Raw
data
streams
e.g.
tweets,
cameras,
traffic,
weather,
…
…
Level
1:
Unified
representa3on
Proper3es
(STT
Data)
STT Stream
Level
2:
Aggrega3on
Proper3es Emage
(Emage)
Operations
Level
3:
Symbolic
rep.
Proper3es Situation
(Situa3ons)

19. 11/1/12 19
Billions of data sources.
Selecting and combining appropriate sources to detect
situations.
Interactions with different types of Users
Decision Makers
Individuals
Want to use: Contact jain@ics.uci.edu

20. 11/1/12 20
Front
End
GUI
New New E-­‐mage Alert
Data Query Stream Request
Source
Back
End
Controller
E-­‐mage
Stream
Personalized
Registered Stream
Query
Processor
Queries Alert
Unit
E-­‐mage
Stream User
Info
Registered
Data Data
Ingestor Raw
Data
Storage
Sources
API
Calls Raw
Spatial
Data
Stream
Data
Cloud

21. 11/1/12 21

22. 22
Building Blocks: Operators
Supporting
Operator Type Data
parameter(s)
Output
1) Data into right
representation Transform …
Spatio-temporal
window
Filter +
Mask
Aggregate +
2) Analyze data to Classification
derive features Classification method
Characterization Property Growth Rate
required = 125%
Pattern Matching
+
Pattern 72%
{Features}
3) Use features to Learn f Learning
method f
evaluate situations {Situation}

23. Personalized Alerts

24. 24
Situation recognition and control
Aggregation, Operations
Alert level
= High
Date: 3rd Jun, 2011
STT data Situation Detection
User-Feedback
Tweet: 1) Classification
‘Please visit Dr. Cureit at
‘Urrgh… sinus’ 2) Control action 4th St immediately’
Loc: NYC,
Date: 3rd Jun, 2011
Theme: Allergy

25. Allergy System: Beyond Twitter

26. 26
EVENTSHOP:
Recognizing situations from web streams

27. 27
Thailand Flood Mitigation

28. 28
In New York:
https://twitter.com/researchrerere
Help today.

29. 29
Connecting resources: Problems and
Research Community
• Big Data is BIG in challenges and opportunities
particularly for Multimedia research community.
• Situation recognition is the challenge for NOW.
• IF PUBLICATIONS motivate you, THEN this is a an
opportunity to grab.
• IF you want to make an IMPACT, THEN this is an
opportunity for you.

30. 30
Thanks.
We Need Collaborators: EventShop.