This document discusses the evolving role of photos from memories to sources of information. It argues that photos, when connected to associated metadata and context, can form the basis of a "visual web" that allows for more natural searching and knowledge sharing compared to a text-based web. The author envisions apps that can automatically extract context from photos using sensors and tags to organize all photos into a personal visual web for each user and enable new types of crowdsourced information through "focused microblogging" with photos.
1. Weaving
the
Visual
Web
Ramesh
Jain
Dept.
of
Computer
Science
University
of
California,
Irvine
jain@ics.uci.edu
2. • Memories
of
experiences.
• Explicit
contextual
communicaFon.
– Data
is
a
new
uFlity.
– Text
is
too
abstract
in
many
situaFons.
Duality
of
Photos
Why
do
we
take
photos?
3. Where
do
photos
come
from?
• Drawings
• Pain6ngs
• Cameras
• Smart
Cameras
• Graphics
10. Berners-‐Lee:
Suppose
all
the
informa6on
stored
on
computers
everywhere
were
linked.
Suppose
I
could
program
my
computer
to
create
a
space
in
which
anything
could
be
linked
to
anything.
11. Web: Human experiences, knowledge,
and understanding captured using
associative links in DOCUMENTS.
13. Imagine if every photo and video
captured were connected to every
other! And to other information!!
14. What is a camera?
Captures intensity from a point in the world.
15. Is
a
Smartphone
camera
sFll
a
camera?
Many
sensors
capture
metadata
related
to
the
moment
and
capture.
• Exposure Time
• Aperture Diameter
• Flash
• Metering Mode
• ISO Ratings
• Focal Length
• Time
• Location
• Face
Smartphone
camera
captures
events.
30. 30
Who
When
Where
What
Why
AssociaFons
related
to
these.
31. Marking
Moments:
Micro
Blogs
• Facebook
Status
and
Tweets
started
Micro
Blogs.
– Now
there
are
many
– Instagram,
Snapchat,
…
• Problem
with
Tweets:
More
Noise
less
Data.
• Time
to
add
Focused
Micro
Blogs
– Sensors
– Importance
of
marking
a
moment
35. A
Photo
is
a
Click
in
Real
World.
• Remember
Kodak
Moment!
• For
each
photo:
• Unique
ID,
• All
metadata
of
the
event,
• Tags,
• Links,
• AnnotaFons.
39. Krumbs
Used
as
Focused
Micro
Blog
• One
Click
upload
to
the
FMB-‐App
Server.
• The
client
sends:
– Sender
ID
– Photo
– GPS
and
Place
– Time
and
Event
– Emoji
based
Context
and
annota6on
– Any
addi6onal
comments
• As
JSON
40. FMB-‐App
Server
• FMB-‐App
Server
uses
EventShop
for
appropriate
aggrega6ons.
– Loca6on
– Geographic
area
(ward,
town,
city,
…)
– Also
computes
some
rates
of
changes.
– Determines
Trends
– Classifies
areas
based
on
evolving
and
current
situa6ons.
• Allows
drill-‐down
to
show
even
individual
photos.
41. Tweets
VS
Krumbs
Tweets
1. Tweet
require
thinking
and
typing.
2. Loca6on
of
a
tweet
and
the
corresponding
event
may
be
different.
3. Tweets
are
subjec6ve.
Krumbs
1. Krumbs
have
more
informa6on
and
are
spontaneous.
2. Krumbs
maintains
event
loca6on.
3. Krumbs
are
objec6ve.
44. Life
Events
relate
disparate
streams
to
life.
Personal
photos
on
smart
phones
TELL
a
lot
about
you.
45. Visual
Web
PlaZorm
Contextual
Reasoning
and
Event
ComputaFons
NavigaFon
and
Search
Krumbs
Next
App
Knowledge
Discovery:
Event
AnalyFcs
and
VisualizaFon
Agro-‐
Tech
Clean
India
Personal
Sharing
and
CommunicaFon
46. Your
Personal
Visual
Web
on
Smartphone
Photo
Cloud
Moment
Capture
Contextual
Reasoning
and
Event
ComputaFons
Event
AnalyFcs
And
VisualizaFon
NavigaFon
and
Search
• Available
on
Android
and
iOS.
• Your
data
remains
on
your
phone
unless
shared.
Sharing
47. Developing
An
App:
Agro-‐Tech
Contextual
Reasoning
and
Event
ComputaFons
NavigaFon
and
Search
Krumbs
for
Agro-‐Tech
Agro
Knowledge
Discovery:
Event
AnalyFcs
and
VisualizaFon
Agro-‐
Tech
Sharing
and
CommunicaFon
Popular
Selfie
Food
Agriculture
Shopping
48. Technology
for
Building
Visual
Web
• Visual
Authoring
Environment:
HTML
for
Visual
Data.
• Content
Analysis
• Deep
Contextual
Reasoning:
From
Smartphones,
sensors,
IoT,
personal
history,
social,
personal
and
all
other
events.
• Event
Clustering
and
recogniFon.
• Photo
Ranking
• Combine
with
Deep
Learning
based
Content
Analysis
• Visual
NavigaFon
• Cross
sharing
and
integraFon
with
other
PlaZorms.
• Combine
Algorithmic
and
InteracFve
tools.
49. For Visual Web, we need
• Addressing (URI)
• Transfer Protocol (HTTP)
• Authoring and Presentation (HTML)
• Ranking
• Contextual Processing
• Content Analysis
• Privacy and Security
• Information Vs Experience
Challenges
50. Sky
is
the
Limit.
ApplicaFons:
Lifestyle
Health
Commerce
Surveillance
and
Monitoring
Agriculture
Research
and
development
ConstrucFon
…
51. Thanks
for
your
Fme
and
a_enFon.
For
ques6ons:
jain@ics.uci.edu