This document discusses using micro-reports and photos to gain insights about situations and needs through computational analysis. It presents Krumbs, a platform for capturing experiences through photos, video and other data to create structured reports. These micro-reports can then be analyzed using EventShop to detect situations in real-time and predict future needs by matching resources. The goal is to design social machines that effectively connect people's needs to resources.

1. © Ramesh JainSlide 1
Micro-Reports and
Situation Recognition
Ramesh Jain
Computer Science@UCI
(with Vivek Singh, Mingyan Gao, Siripen Pongpaichet, and
Mengfann Tan)
and
Krumbs Inc
(Asquith Baily, Neil Jain, Pinaki Sinha)
jain@ics.uci.edu

2. © Ramesh JainSlide 2
Society exists only as a mental concept;
in the real world there are only
individuals.
-- Oscar Wilde

3. © Ramesh JainSlide 3
Humans are Smart Sensors.
3
Humans are Smart Actuators
Humans are the goal as well as the
source of Technology.

4. © Ramesh JainSlide 4
The Magic Device: Mobile Phone
Middle 4 Billion
Top 1.5
Billion
Bottom 1.5 Billion
MOP: Improving
Information
Environment
TOP: Strong Information
Environment
BOP: Deprived of
Information
4

5. © Ramesh JainSlide 5
This century is different from the last.
Should we think differently???

6. © Ramesh JainSlide 6
In 20th century, we tolerated photos
in our textual documents.
In 21st century, you create visual
documents that tolerate text.

7. © Ramesh JainSlide 7
Major Disruption in Photos: From
Memories to Information Sources.
Photos are the most compelling source
of information.

8. © Ramesh JainSlide 8
Most Fundamental Problem:
Connecting People’s Needs to
Resources Effectively, Efficiently,
and Promptly in given Situations.
8

9. © Ramesh JainSlide 9
Major transformation in human
history are a chronicle of building
Social Machines for
How People’s need are
connected to Resources.
9

10. © Ramesh JainSlide 10
Hunter Gatherer:
You went to Food.
Now food comes to you.

11. © Ramesh JainSlide 11
Until 2000, you
went to make a
call.
Now call finds you.

12. © Ramesh JainSlide 12 12
Human Needs remain
the same.
Resources and
distribution methods
are invented.

13. © Ramesh JainSlide 13
Designing Social Machines:
• Needs
• Resources
• Connecting Needs to Resources

14. © Ramesh JainSlide 14
How can we identify Needs?
Need = f (person, context)
14

15. © Ramesh JainSlide 15
Designing Social Machines
IoT
Social Media
Human Sensors
Environmental
Sensors
Human/Actuators
Open Data

16. © Ramesh JainSlide 16
We Live in Dynamic World.
16

17. © Ramesh JainSlide 17
What is Cyber Space?
Who invented it?
Animals
Machines
Societies
Published first in 1942

18. © Ramesh JainSlide 18
• Desired state (Goal)
• System model
• Control Signal (Act)
• Current State
(observe)
18

19. © Ramesh JainSlide 19
Social Life Networks: Important
Factors
• The world we live in.
– Knowing current situation.
– Knowing where resources are.
• Where needs are.
– Knowing each individual’s situation.
– Knowing what they may need.
• Matching/recommendation Engine
• Action

20. © Ramesh JainSlide 20
Social Life Networks
Physical
World
And
Informa
tion
Systems
Environment and Resources
Information
Personal Situation and Needs
Information
Match
ing
Action Signals

21. © Ramesh JainSlide 21
EventShop : Geospatial Situation Detection
Situation
Recognition
Data Stream
Ingestion and
aggregation
Database
Predictive
Analytics
Personal EventShop: Life Event Detection
Personal
Situation
Recognition
Database
Personal
Data
Ingestion
Objective Self
Recommendation
Engine
Need- Resource Matcher
Identify Resources and Needs
Resources Needs
Evolving Global Situation
Evolving Personal Situation
Actionable Information

22. © Ramesh JainSlide 22
Billions of data sources.
Environment for
Selecting, and
Combining
appropriate sources to detect situations.
Prediction for Pro-active actions
Interactions with different types of Users
Inspired by Photoshop

23. © Ramesh JainSlide 23

24. © Ramesh JainSlide 24

25. © Ramesh JainSlide 25
Flood level - Shelter
Flood Level
Shelter
Twitter
Classify (Flood level - Shelter)

26. © Ramesh JainSlide 26

27. © Ramesh JainSlide 27
Microblogs: Participatory Sensing
• Microblogging is a broadcast medium that
uses typically smaller form of blogging.
• Twitter, Status updates, Instagram, …

28. © Ramesh JainSlide 28
Micro-Blog Mining Process
• Extracting Data From Data Providers
• Parsing, Integrating, and Storing the
data
• Extract Information of interest
• Earthquake Analysis; Flu; Trends

29. © Ramesh JainSlide 29
Problem with Micro-Blogs
• Noisy
• Subjective
• Poor context
• Great concept but has limitations.
• New technology to overcome these
limitations.

30. © Ramesh JainSlide 30
Waze

31. © Ramesh JainSlide 31
First Principle in Journalism
• Truthfulness, accuracy, objectivity,
impartiality, fairness and public accountability
• Journalists cannot always guarantee ‘truth’,
but getting the facts right is the cardinal
principle of journalism. We should always
strive for accuracy, give all the relevant facts
we have and ensure that they have been
checked. When we cannot corroborate
information we should say so.
• Seek Truth and Report it as Fully as Possible

32. © Ramesh JainSlide 32
Most Reports or Information are report
from an Event.
• ‘Kodak Moment’
• Each event has interesting and important
moments.
• How do we capture a moment?

33. © Ramesh JainSlide 33
What is a camera?

34. © Ramesh JainSlide 34
Is a Smartphone camera still a
camera?
Camera collects all metadata related to the Event.
• Exposure Time
• Aperture Diameter
• Flash
• Metering Mode
• ISO Ratings
• Focal Length
• Time
• Location
• Face
Smartphone camera captures events.

35. © Ramesh JainSlide 35
Micro-Reports: Requirements
• Objective (Subjective comments put
explicitly)
• Spontaneous
• Compelling
• Universal

36. © Ramesh JainSlide 36
Micro-Reports
• What (Information)
• Who (Information)
• When (Time)
• Where (Location)
• Why (Causality)
• How (Experiential)
Photo
What
Where
When
Who
Why
Sound

37. © Ramesh JainSlide 37
Krumbs: Capture and Report
experience of a moment.
What: Objects
Who: People
When: Events
Where: Location
Why: Intent/Emotions

38. © Ramesh JainSlide 38

39. © Ramesh JainSlide 39

40. © Ramesh JainSlide 40

41. © Ramesh JainSlide 41
Krumbs SDK

42. © Ramesh JainSlide 42
Dashboard

43. © Ramesh JainSlide 43
Micro Reports and Analytics SDK
1. Reports convey direct desire, feedback, and observations.
2. Real time Aggregation and Analytics for understanding reports.
Creation of reports Customer Reports 43

44. © Ramesh JainSlide 44
EVENT
Spatial
Temporal
Informa-
tional
Experien-
tial
Structural
Causal
{"micro_reports":[{
"where":{
"geo_location":{
"latitude":32.90233332316081,
"longitude":-
117.2441166718801},
"when":{
"start_time":"Jun 14, 2009
11:25:19 AM",
"end_time":"Jun 14, 2009 11:25:19
AM",
"time_zone":"America/Los_Angeles"}
,
"what":[{
"concept_name":"people",
"confidence":0.9836078882217407,
"visual_concept_provider":"CLARIFA
I"},
… {
"concept_name":"food",
"confidence":0.8526291847229004,
"visual_concept_provider":"CLARIFA
I"}],
"tag":”#niceday #summer",
"source":{"default_src":"https://….jpg"}},
"sub_event":[],
"why":[]},
…]}
MediaJSON for each micro-report

45. © Ramesh JainSlide 45
STT
Emage
EventShop
Emage Stream
Processing Engine
STTbase Others
Visual
Analytics
Query &
Feedback
STT Data
Ingestion
Trend and
Correlation Analysis
Micro-
Reports
Wrappe
r
Data Streams from
Other sources i.e.,
satellites, IoT, and
stationary sensors.
STT1 Stream
STT2 Stream
STTn Stream
…
Rule Engines and
Alert Units
Emage Generator
Dashboard and
External Applications
Notification

46. © Ramesh JainSlide 46
Krumbs SDK at work
UCI students in Next
Generation Search Class
Service Connect MyUCIVizNotes Places
46

47. © Ramesh JainSlide 47
• Routine operations
• Surge conditions
• Situational awareness
• Multi-objective
resource optimization
• Service Routing
• Changing uses
Management Expectations
• Clean
• Safe
• Walkable
• Reliable operations
• Special events
• Livability
• Healthy environment
• Fresh food
• Web sources and blogs
• Realtime information
(traffic, weather, transit,
…)
• Citizen participation
vs
Our Challenge

48. © Ramesh JainSlide 48
The relevant Sustainable DC goals to this project
include:
• Develop a Zero Waste plan for the city
• Ban polystyrene from the city
• Decrease all citywide waste streams
• Increase recycling bins in public realm
• Coordinating a city-wide education programs
The Sustainable DC Plan, developed with extensive citizen
input, established 2032 goals and actions designed to make
Washington DC the most sustainable city in North
America.
How to Improve Public Space
Waste Management
Our Goals

49. © Ramesh JainSlide 49
DowntownDC plans have so far focused on:
• Refining routes for personnel and equipment
• Gaining an understanding of the timing of services
• Cataloging major events and activities
• Mapping all public space elements in GIS
Micro-reports Event dataGIS data Route information
Our Approach

50. © Ramesh JainSlide 50
Real-time Trash Situations from
Sensors and Micro-Reports
Trash Bin Sensors Data
Micro Reports from Krumbs
Filter
Aggregate
Filter
Real-Time Trash Fill Level Situation
in EventShop

51. © Ramesh JainSlide 51
Prediction based on Events History
Events Data
Real-Time Trash Fill Level Situation
5 32
Now
Predicted Trash Fill Level
in 30 minutes at a given location
4
5
95
30 minutes
10
30
40
70
90
100
20
0
20
40
60
80
100
120
7:30 8:00 8:30 9:00 9:30 10:00 10:30
Projected Trash Fill Level at a given location based on Event History
0
35
50
90
0
20
40
60
80
100
7:30 8:00 8:30 9:00 9:30 10:00 10:30
Real-Time Fill Level Situations at a given location of an event

52. © Ramesh JainSlide 52
Current Status: In the hands of
employees this week.

53. © Ramesh JainSlide 53
Smart Communities: Community Relationship
Management
Most communities have Web Presence .
Improving community experience: facilities, local services,
participation.

54. © Ramesh JainSlide 54
Each Photo is a
Micro-Report
Today Flickr
Data.
Tomorrow All
Photos.

55. © Ramesh JainSlide 55
B
A
D
C
E

56. © Ramesh JainSlide 56
Number of Flickr Photos (in London)

57. © Ramesh JainSlide 57
Flickr Concepts Combined

58. © Ramesh JainSlide 58
#photos in each concepts
people
sport
running swimming
Can you solve the mystery?

59. © Ramesh JainSlide 59
Current Status
Krumbs SDK Ready: Being used by multiple
groups.
EventShop is open source.
Photos as a report is an exciting topic for research.
Looking for collaborators to join the
adventure in building Social Machines.

60. © Ramesh JainSlide 60
Dream!
5.5 Billion People Reporting and
Contributing to Solve Societal Problems.
Even in Remotest parts of a Developing
Country!

61. © Ramesh JainSlide 61
Thanks for your time and attention.
For questions: jain@ics.uci.edu