The document discusses challenges in connecting people to essential resources through modern data processing, particularly in areas like computer vision, machine learning, and social networks. It emphasizes the need for effective resource management and the development of smart social systems capable of recognizing situations and needs. Additionally, it outlines the importance of handling large-scale data streams and personal contexts in creating responsive applications.

1. 8/21/2013 1

2. • Why am I excited about things?
• What are We doing?
• What are the challenges?

3. What is your research interest?
• Computer Vision
• Multimedia
• Machine Learning
• Social Media
• Intelligent Systems
• Big Data
• Stream Processing
Building a Better Society.

4. Events and Entities Exist in the real
world.
Events and entities result in Data and
Documents

5. What is the most Fundamental
Problem in Society?
Connecting People to Resources
Effectively, Efficiently, and Promptly
in given Situations.
Hint: Economics, Health Care, Politics, Computer Science,
Operations Research, …

6. Maslow: Basic Needs
http://www.theappgap.com/collaboration-whats-in-it-for-me.html

7. Maslow: Needs and Resources
http://www.theappgap.com/collaboration-whats-in-it-for-me.html
• Financial
• Natural (Food)
• Human Skills
• Health
• Rescue
• Transportation
• Education
• Production
• …

8. Social Life Networks
Connecting People to Resources

9. Modern Data

10. 8/21/2013 10
Knowledge
Data
Processing
Information/
Decision

11. Decisions
Food
Friend
What???
Information
Data
Knowledge
Observations

14. • Desired state (Goal)
• System model and Control Signal
(Actions)
• Current State (for Feedback)

15. Key Steps
1. Identify Situation
2. Determine Needs
3. Determine Resources
4. Develop best resource management
approach
5. Communicate/Actuate decisions
6. Go to 1.

16. Social Networks
Connecting
People

17. Current
Social
Networks
Important
Unsatisfied
Needs
8/21/2013 17

18. Motivation
Location Based
Mobile Applications
Ongoing Archived
Database System
satellite
Environmental
Sensor Devices
Internet of Things
Social Media
Social Life
Network
Experts
People
Governmental
Agencies
Situations

19. Social Life Network
Aggregation
and
Composition
Situation
Detection
Alerts
Queries
Information

20. Fundamental Problem
Connecting People to Resources
effectively, efficiently, and promptly
in given situations.

21. • Social observations are now possible with
little latency.
• We can design social systems with feedback.
• Situation Recognition and Need-Availability
identification of resources is a major
challenge.

22. Smart Social Systems Architecture
Situation
Recognition
Evolving
Situations
Available
Resources
Identified
Needs
Need-
Resource
Matcher
Communicati
on/Control
Unit
Resources
People
D
a
t
a
S
o
u
r
c
e
s
Database System
satellite
Environmental
Sensor Devices
Social Media

23. Concept Recognition: Last Century
23
Environm
ents
Real world
Objects
Situations
Activities
SingleMedia
SPACE
TIME
ScenesLocation
aware
Visual
Objects
Trajectories
Visual
Events
Location
unaware
Static Dynamic
Location
aware
Location
unaware
Static Dynamic
Data = Text or Images or Video

24. Visual Concept Recognition: Quick History
• 1963: Object Recognition [Lawrence + Roberts]
• 1967: Scene Analysis [Guzman]
• 1984: Trajectory detection [Ed Chang+ Kurz]
• 1986: Event Recognition [Haynes + Jain]
• 1988: Situation Recognition [Dickmanns]
1960 1970 1980 1990 2000 2010
Object Scene Trajectory Event Situation

25. Concept Recognition: This Century
25
Environm
ents
Real world
Objects
Situations
Activities
SPACE
TIME
Location
aware
Location
unaware
Static Dynamic
HeterogeneousMedia
Location
aware
Location
unaware
Static Dynamic
Data is just Data.
Medium and sources do not matter.

26. • relative position or combination of
circumstances at a certain moment.
• The combination of circumstances at a
given moment; a state of affairs.

27. Situations: Definition
An actionable abstraction of observed
spatio-temporal characteristics.
27

28. 8/21/2013 28

29. • Example 1:
– A person shouting.
– 1000 people shouting.
• In a contained building
• In main parts of a city
• Example 2:
– One person complaining about flu.
– Many people from different areas of a country
complaining about flu.

30. Micro-events:
Sensors detecting and chirping
(broadcasting) events
• Billions of disparate kinds of sensors being
placed everywhere.
• Each sensor detects ‘basic events’ and
broadcasts it in a simple form.
• Develop a system to process these micro-
events and make them useful.

31. Example: Cameras in a city
• ‘Chirps’ could be of different types
• Define behaviors like:
– Heavy traffic
– Popular event going on
– People leaving X area
– Violence starting
– . . .
• Use for Macro-behvior analysis

32. From micro events
to situations
Thermodynamics
provides a framework for relating the
microscopic properties of individual
atoms and molecules to the
macroscopic or bulk properties of
materials that can be observed in
everyday life.

33. Data Types in Situation Recognition
• Static Data:
– POIs (location of hospitals)
– Population
– Technical data of hospitals
– Contact persons in committees or health organization, and
– All information on support-potentials for personnel, material and
infrastructure
• Dynamic Data:
– Current disease data
– Twitter/Google Trends
– Environmental data
– Personal Individual data

34. Two Big Challenges
• Data Ingestion to efficiently extract data from
the Web and make them available for later
computation is not-trivial.
• Stream Processing Engine to bridge the
semantic gap between high level concept of
situations and low level data streams.

35. S
Social
Networks
2-D spatial
Grid at time
T
Database
Systems
Global
Sensors
Phone
Apps Internet of
Things
(S, T, T)
S Uses
Application
semantics to
combine
different data
items.

36. Observed State(Situations)
Intelligent Social Systems: Spatial Perspective
Real World Events
Observations
Control
Signals
Goal

37. Observed State(Situations)
Intelligent Social Systems: People Perspective
Real World Events
Observations
Control
Signals
Goal

38. • Spatio-temporal element
– STTPoint = {s-t-coord, theme, value, pointer}
• E-mage
– g = (x, {(tm, v(x))}|xϵ X = R2 , tm ϵ θ, and v(x) ϵ V = N)
• Temporal E-mage Stream
– TES=((ti, gi), ..., (tk, gk))
• Temporal Pixel Stream
– TPS = ((ti, pi), ..., (tk, pk))
38

39. 8/21/2013 39

40. 8/21/2013 40

41. 8/21/2013 41
Retail Store
Locations
Net Catchment
area

42. Level 1: Unified
representation
(STT Data)
Level 3:
Symbolic rep.
(Situations)
Properties
Properties
Properties
Level 0: Raw data streams
e.g. tweets, cameras, traffic, weather, …
Level 2:
Aggregation
(Emage)
…
STT Stream
Emage
Situation
42
Operations

43. 43

Pattern Matching
Aggregate

@ Characterization
∏ Filter
 Classification
72%
+
+
Growth Rate
= 125%
Data
Supporting
parameter(s)
OutputOperator Type
+
Classification
method
Property
required
Pattern
Mask

44. • IF 𝑢𝑖 𝑖𝑠𝑖𝑛 𝑧𝑗 𝑇𝐻𝐸𝑁 𝑐𝑜𝑛𝑛𝑒𝑐𝑡 (𝑢𝑖, 𝑛𝑒𝑎𝑟𝑒𝑠𝑡 𝑢𝑖, 𝑟𝑘 )
1) 𝑖𝑠𝑖𝑛 𝑢𝑖, 𝑧𝑗 → 𝑚𝑎𝑡𝑐ℎ(𝑢𝑖, 𝑟𝑘))
𝑓: (𝑈 × 𝑍) → (𝑈 × 𝑅)
• U = Users
• Z = Personalized Situations
• R = Resources
2) 𝑛𝑒𝑎𝑟𝑒𝑠𝑡 𝑢𝑖, 𝑟 𝑘 = 𝑎𝑟𝑔𝑚𝑖𝑛 (𝑢𝑖. 𝑙𝑜𝑐, 𝑟 𝑘. 𝑙𝑜𝑐𝑠)
44

45. 8/21/2013 45
Billions of data sources.
Environment for
Selecting, and
Combining
appropriate sources to detect situations.
Prediction for Pro-active actions
Interactions with different types of Users
Decision Makers
Individuals

46. OutputIngestor
EventSource
Parser
Data Adapter
Emage
Generator
(+resolution mapper)
Processing
EvShop Internal Storage
Query
Parser
Query
Rewriter
Event Stream Processing
Executor
ᴨ
ᴨ
Action Parser
Register EventSource Register Continuous Query
Situation
Emage
Visualization
(Dashboard)
Actuator
Communication
Action Control
Event Property &
Other Information
(e.g., spatio-temporal
pattern)
µ
Data Access Manager
Live Stream
Archived Stream
Situation Stream
Real-Time
DataSource
(e.g., sensor
streams, geo-image
streams)
Near Real-Time
DataSource
(e.g., preprocessing
data streams, social
media streams)
Raw Event

47. Input Manager
External Event
Preprocessing
(Collaboration)
47
Real-Time Sensor Streams
e.g., Cloud Satellite Images
Real-Time Sensor Streams
e.g., Wind Speed, Traffic Flow
RealTime DataSource
1D Data
Wrapper
STT to Emage
2D Data
Wrapper
Data Adapter Emage Generator
Emage
Emage
Factory
STT
Emage
Raw Social Media Streams
e.g., Twitter, News RSS Feed
NearRealTime DataSource
Event Model
Wrapper
STT to Emage
Data Adapter Emage Generator
Emage
Emage
Factory
STT
Topic Event
Detection
Abnormal
Event
Detection
Raw Sensor Streams
e.g., PM2.5 data
“EventModel” Streams
e.g., suddenly change
of data trend
within time window
Emage Store
STT Store
Metadata Store
EventSource
Parser Interface
(Optional)
RealTime
Emage Streams
NearRealTime
Emage Streams
Processing
Manager
ES Descriptor
ES Control
(Start/Stop/
View ES)
Users Input
Automatic Data/Events Flow
InitialResolution
AggregationFunc
Metadata
Theme
AdapterType
SourceURL
TimeWindow
Parameters

48. 48
Stream Processing Engine
Operators Manager
Built-in
Operators
User-Defined
Operators
ᴨ
ᴨ
µ
Data
Access
ᴨ
ᴨ
µ
Data
Access
ᴨ
ᴨ
µ
Data
Access
Input Manager
(Accessing
External Data)
Event Stream Executor
Operators
Nodes
Internal Storage
(Accessing Internal
Data)
AsterixDB, SciDB,
MongoDB
Emage Store
STT Store
Metadata Store
Query Parser
Interface
Query
Descriptor
Query Control
(Start/Stop/ View)
Real-time/
near real-time
Emage Streams
Archived
Emage Streams Situation Streams
Emage
Interpolation
Function
Emage
Conversion
Final
Resolution
Parameter Operators
Operators
Store Parameters
Retrieve Parameters
Query
Rewriter
Execution Plan

49. 8/21/2013 49

51. 8/21/2013 51
Flood level - Shelter
Flood Level
Shelter
Twitter
Classify (Flood level - Shelter)

52. 8/21/2013 52

53. Personalized Alerts
Disaster
Situation
Assimilation
and Control
Environmental
Resources and
Historic Data
Governmental Agencies
Internet of Things
Social Sources
Experts
Users
All Users are
not EQUAL.

54. What defines a person?

55. Activities
Communications
Media
Persona: Turning Disassociated Data into
Meaningful Information
AGGREGATED PERSONAL
ANALYTICS
Sensors
55
Persona

56. Personal EventShop: Micro Situation
Detection

57. HEALTH
PERSONA
Logical Sensor Physiological SensorsFitness Tracking Sensors
Life
Event
Kinetic
Event
Physiological
Event
Food
Event
Personicle
Health Persona Framework

58. Life Events Ontology

59. Meeting with
development group
Yoga with
Jena
Meeting with
test group
Mina’s birthday party
9:30-11:00 12 - 1
4:00-
5:00
Transportation
walking
walking
Transportation
Transportation
Transportation
Home HomeWork Caspian
HavingBreakfast
Doingthedishes
Commuting
Commuting
Walking
Walking
Meeting
Meeting
working
Working
Exercise
Exercise
Working
Working
Working
Working
Meeting
Meeting
Walking
Commuting
Commuting
Commuting
Shopping
Shopping
Commuting
Partying
Partying
Partying
Partying
Partying
Commuting
Commuting
Commuting
Sleeping
Sleeping
WatchingTV
Cleaning
Personicle
Commute Walk Meet work Exer. work Meet Commute shop party Commute Sleep
GPSlocationtracking
Calendar
Activity
levelPersonicle

60. SLN Architecture
EventShop
Personal
EventShop
Evolving
Situations
Available
Resources
Identified
Needs
Need
Resource
Matcher
Communicati
on/Control
Unit
Resources
People
D
a
t
a
S
o
u
r
c
e
s
Database System
satellite
Environmental
Sensor Devices
Social Media

61. Research Challenges
• Situation Recognition
• Persona and Personal Context
• Chronicle Analytics and Visualization
• Massive Geo-Spatial Heterogeneous
Stream Processing
• Dynamic Need-Resource Optimization

62. Situation Recognition
• Next Frontier in Concept Recognition
• Heterogeneous Geo-spatial Dynamic Data
• Social data and IoT become a key element
• Application and domain semantics
• Model definitions
• High dimensionality
• Unification of data: Social-Cyber-Physical

63. Persona and Personal Context
• Not only Logs of Keyboard and Surfing.
• You Log and explore every thing.
– Entity resolution on TURBO
• Many new data processing and unification
challenges.

64. Single Tri-axial
Accelerometer
Example: Activity Recognition
Multiple
Accelerometers
Activity
Classification
Features: mean, variance, standard
deviation, energy, entropy
Correlation between axis, discrete
FFT coefficients
Activities: running, walking,
lying, sitting, ascending stairs,
descending stairs, cycling
Location
Calendar
History
…..
Context
Activities: running, walking,
lying, sitting, ascending stairs,
descending stairs, cycling,
shopping, commuting, doing
housework, office work, lunch
routine.

65. MicroBlogs and Twitter:
LIMITATIONS
• Very LOW Signal-to-Noise ratio: High
Noise-to-Signal ratio
• Focus on being broad platform.
• Difficult to extract SIGNAL.
• Information must be extracted from
limited text.

66. Solution: Tweeting Applications
• Develop focused Apps: Focused MicroBlogs
• Get all information from ‘motivated’ and
collaborative users.
• Help them solve their problem.

67. WAZE: Outsmarting Traffic, Together

68. Chronicle Analytics
• Enterprise Warehouse were for late 20th
Century – Planetary Warehouses are defining
this century.
• Big data is important because it collects
everything that happens to build ‘Prediction
Machines’.
• Machine learning and visualization are the
key tools.

69. Massive Geo-Spatial
Heterogeneous Stream Processing
• Why does the DATA become so BIG?
• And it will keep getting BIGGER.
• We have to go beyond Batch Processing as
primary computing approach.
• Should be of great interest to Social Media
researchers.

70. Dynamic Need-Resource Optimization
• What are the fundamental problems in
Computer Science?
– Time and memory compexity
– Operating systems, networks, storage management,
algorithms, …
• What is the main concern in
– Economics?
– Healthcare?
– Politics?
– …

71. Live EventShop and Collaboration
• Live EventShop Demo
– http://auge.ics.uci.edu/eventshop/
• Current Collaborators & Plan
– Cyber-Physical Cloud Computing Project
• NICT, NIST
– SLN4MOP Project
• Sri Lanka Farmers; Prof. Ginige in Sydney leading
– Open Source EventShop by mid-year of 2013
• HCL

72. Thanks for your time and attention.
For questions: jain@ics.uci.edu