Weaving	
  the	
  Visual	
  Web	
  
	
  
	
  
Ramesh	
  Jain	
  
Dept.	
  of	
  Computer	
  Science	
  
University	
  of	
...
•  Memories	
  of	
  experiences.	
  
•  Explicit	
  contextual	
  communicaFon.	
  
– Data	
  is	
  a	
  new	
  uFlity.	
...
Where	
  do	
  photos	
  come	
  from?	
  	
  
•  Drawings	
  	
  
•  Pain6ngs	
  
•  Cameras	
  
•  Smart	
  Cameras	
  
...
Camera	
  History	
  
The	
  Role	
  of	
  Photos	
  con6nues	
  to	
  Evolve	
  
•  Memory	
  
•  Informa6on	
  
– Notes	
  	
  
– Documenta6on...
In 20th century, we tolerated photos
in our textual documents.
In 21st century, you create visual
documents that tolerate ...
Major Disruption in Photos: From
Memories to Information Sources.
Photos are the most compelling source
of information.	
  
Experiences
Life =
Events
+
Progress: Knowledge Creation and
Propagation
Visual	
  
Visual	
  knowledge	
  
Oral	
  
Oral	
  knowledge	
  
Textual	
  ...
Berners-­‐Lee:	
  	
  Suppose	
  all	
  the	
  informa6on	
  stored	
  
on	
  computers	
  everywhere	
  were	
  linked.	
...
Web: Human experiences, knowledge,
and understanding captured using
associative links in DOCUMENTS.
Text	
  based	
  
documents	
  are	
  not	
  
NATURAL.	
  
	
  
	
  Language	
  and	
  
Literacy	
  come	
  in	
  
the	
  ...
Imagine if every photo and video
captured were connected to every
other! And to other information!!	
  
What is a camera?
Captures intensity from a point in the world.
Is	
  a	
  Smartphone	
  camera	
  sFll	
  a	
  
camera?	
  
Many	
  sensors	
  capture	
  metadata	
  related	
  to	
  th...
Computa6onal	
  Representa6on	
  of	
  an	
  
event.	
  
Experien6al	
  Data:	
  	
  	
  
•  Photos	
  
•  Video	
  
•  Au...
EMPT: Extractable Mobile Photo Tags
Exif	
  
Content	
  
Analysis	
  
EMPT	
  
	
  
People	
  
Place	
  
Objects	
  
Event...
Krumbs:	
  	
  Capture	
  and	
  Connect	
  
holisFc	
  experience	
  of	
  a	
  moment.	
  
What:  Objects	
Who:    Peopl...
All photos and
associated context
is automatically
Organized in a
Web, and
Shared (if desired).
Duality	
  of	
  Photos:	
  	
  Relevance	
  to	
  
MM	
  and	
  CompuFng	
  
•  Memories:	
  Search	
  and	
  Retrieval	
...
How	
  do	
  you	
  search	
  photos?	
  
What	
  
Who	
  
When	
  
Where	
  
	
  
Why	
  
AssociaFons	
  related	
  to	
 ...
Searching	
  ‘For	
  a	
  Photo’	
  	
  
and	
  	
  
Searching	
  ‘From	
  a	
  Photo’.	
  
28	
  
29	
  
30	
  
Who	
  
When	
  
Where	
  
What	
  
	
  
Why	
  
AssociaFons	
  related	
  to	
  these.	
  
Marking	
  Moments:	
  	
  Micro	
  Blogs	
  
•  Facebook	
  Status	
  and	
  Tweets	
  started	
  Micro	
  
Blogs.	
  
– ...
Waze	
  
Crowdsourced	
  SituaFons	
  
Crowdsourced	
  SituaFons	
  
A	
  Photo	
  is	
  a	
  Click	
  in	
  Real	
  World.	
  
•  Remember	
  Kodak	
  Moment!	
  
•  For	
  each	
  photo:	
 ...
Crowdsourced	
  SituaFons	
  
At	
  435	
  Main	
  
8:37	
  AM	
  
10/20/15	
   37	
  
10/20/15	
   38	
  
Flood level - Shelter
Flood Level
Shelter
Twitter
Classify (Flood level - Shelter)
Krumbs	
  Used	
  as	
  Focused	
  Micro	
  Blog	
  
•  One	
  Click	
  upload	
  to	
  the	
  FMB-­‐App	
  Server.	
  
• ...
FMB-­‐App	
  Server	
  
•  FMB-­‐App	
  Server	
  uses	
  EventShop	
  for	
  appropriate	
  
aggrega6ons.	
  
–  Loca6on	...
Tweets	
  VS	
  Krumbs	
  
Tweets	
  
1.  Tweet	
  require	
  thinking	
  and	
  
typing.	
  
2.  Loca6on	
  of	
  a	
  tw...
Seman6c	
  Links	
  for	
  a	
  Photo	
  
•  Photo	
  Level	
  
– Automa6c	
  crea6on	
  
– Manual	
  Annota6on	
  
– Manu...
ObjecFve	
  Self:	
  From	
  Personal	
  Big	
  Data	
  
Life	
  Events	
  relate	
  disparate	
  streams	
  to	
  life.	
  
Personal	
  photos	
  on	
  smart	
  phones	
  TELL	
 ...
Visual	
  Web	
  PlaZorm	
  	
  
Contextual	
  Reasoning	
  and	
  Event	
  ComputaFons	
  
NavigaFon	
  and	
  Search	
  ...
Your	
  Personal	
  Visual	
  Web	
  on	
  
Smartphone	
  
Photo	
  Cloud	
  
Moment	
  Capture	
  
Contextual	
  Reasonin...
Developing	
  An	
  App:	
  	
  Agro-­‐Tech	
  
Contextual	
  Reasoning	
  and	
  Event	
  ComputaFons	
  
NavigaFon	
  an...
Technology	
  for	
  Building	
  Visual	
  Web	
  
• Visual	
  Authoring	
  Environment:	
  HTML	
  for	
  Visual	
  
Data...
For Visual Web, we need
•  Addressing (URI)
•  Transfer Protocol (HTTP)
•  Authoring and Presentation (HTML)
•  Ranking
• ...
Sky	
  is	
  the	
  Limit.	
  
ApplicaFons:	
  
Lifestyle	
  
Health	
  
Commerce	
  
Surveillance	
  and	
  Monitoring	
 ...
Thanks	
  for	
  your	
  Fme	
  and	
  a_enFon.	
  
For	
  ques6ons:	
  jain@ics.uci.edu	
  
Visual Web keynote at MMSP 2015
Visual Web keynote at MMSP 2015
Visual Web keynote at MMSP 2015
Visual Web keynote at MMSP 2015
Visual Web keynote at MMSP 2015
Visual Web keynote at MMSP 2015
Upcoming SlideShare
Loading in …5
×

Visual Web keynote at MMSP 2015

1,349 views

Published on

The 21st century began with a major disruption: the rapid rise of smartphones meant that capturing, storing, and sharing photos and their context became easier than using text. Photos and videos communicate directly, without the need for language or literacy. Until recently, photos were used as compelling memories. Now, photos are increasingly used to convey intent and information.related to a moment. A photo may be linked to many other photos along different dimensions. One may also create explicit links among photos or objects in photos. All photos on the Web form a Visual Web that links photos with other photos and other information elements including all documents on the WWW. This Visual Web offers opportunities to address new societal issues and solve many difficult yet unsolved problems. We discuss nature of the Visual Web, technical challenges, and some interesting opportunities in this area.

Published in: Data & Analytics
0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,349
On SlideShare
0
From Embeds
0
Number of Embeds
28
Actions
Shares
0
Downloads
32
Comments
0
Likes
6
Embeds 0
No embeds

No notes for slide

Visual Web keynote at MMSP 2015

  1. 1. Weaving  the  Visual  Web       Ramesh  Jain   Dept.  of  Computer  Science   University  of  California,  Irvine   jain@ics.uci.edu  
  2. 2. •  Memories  of  experiences.   •  Explicit  contextual  communicaFon.   – Data  is  a  new  uFlity.   – Text  is  too  abstract  in  many  situaFons.     Duality  of  Photos   Why  do  we  take  photos?  
  3. 3. Where  do  photos  come  from?     •  Drawings     •  Pain6ngs   •  Cameras   •  Smart  Cameras   •  Graphics  
  4. 4. Camera  History  
  5. 5. The  Role  of  Photos  con6nues  to  Evolve   •  Memory   •  Informa6on   – Notes     – Documenta6on   •  Ephemeral    Photos  have  ‘half  life’!  
  6. 6. In 20th century, we tolerated photos in our textual documents. In 21st century, you create visual documents that tolerate text.  
  7. 7. Major Disruption in Photos: From Memories to Information Sources. Photos are the most compelling source of information.  
  8. 8. Experiences Life = Events +
  9. 9. Progress: Knowledge Creation and Propagation Visual   Visual  knowledge   Oral   Oral  knowledge   Textual  knowledge  
  10. 10. Berners-­‐Lee:    Suppose  all  the  informa6on  stored   on  computers  everywhere  were  linked.  Suppose   I  could  program  my  computer  to  create  a  space   in  which  anything  could  be  linked  to  anything.  
  11. 11. Web: Human experiences, knowledge, and understanding captured using associative links in DOCUMENTS.
  12. 12. Text  based   documents  are  not   NATURAL.      Language  and   Literacy  come  in   the  way.  
  13. 13. Imagine if every photo and video captured were connected to every other! And to other information!!  
  14. 14. What is a camera? Captures intensity from a point in the world.
  15. 15. Is  a  Smartphone  camera  sFll  a   camera?   Many  sensors  capture  metadata  related  to  the  moment   and  capture.   •  Exposure Time •  Aperture Diameter •  Flash •  Metering Mode •  ISO Ratings •  Focal Length •  Time •  Location •  Face   Smartphone  camera  captures  events.  
  16. 16. Computa6onal  Representa6on  of  an   event.   Experien6al  Data:       •  Photos   •  Video   •  Audio   •  Accelerometer   •  Heart  rate   •  …  
  17. 17. EMPT: Extractable Mobile Photo Tags Exif   Content   Analysis   EMPT     People   Place   Objects   Events   …  
  18. 18. Krumbs:    Capture  and  Connect   holisFc  experience  of  a  moment.   What:  Objects Who:    People When:  Events Where:  Location Why:  Intent/Emotions
  19. 19. All photos and associated context is automatically Organized in a Web, and Shared (if desired).
  20. 20. Duality  of  Photos:    Relevance  to   MM  and  CompuFng   •  Memories:  Search  and  Retrieval   •  InformaFon  communicaFon:  Web,  Big  Data  
  21. 21. How  do  you  search  photos?   What   Who   When   Where     Why   AssociaFons  related  to  these.  
  22. 22. Searching  ‘For  a  Photo’     and     Searching  ‘From  a  Photo’.   28  
  23. 23. 29  
  24. 24. 30   Who   When   Where   What     Why   AssociaFons  related  to  these.  
  25. 25. Marking  Moments:    Micro  Blogs   •  Facebook  Status  and  Tweets  started  Micro   Blogs.   – Now  there  are  many   – Instagram,  Snapchat,  …   •  Problem  with  Tweets:  More  Noise  less  Data.   •  Time  to  add  Focused  Micro  Blogs     – Sensors     – Importance  of  marking  a  moment  
  26. 26. Waze  
  27. 27. Crowdsourced  SituaFons  
  28. 28. Crowdsourced  SituaFons  
  29. 29. A  Photo  is  a  Click  in  Real  World.   •  Remember  Kodak  Moment!   •  For  each  photo:   •  Unique  ID,     •  All  metadata  of  the  event,   •  Tags,   •  Links,   •  AnnotaFons.  
  30. 30. Crowdsourced  SituaFons   At  435  Main   8:37  AM  
  31. 31. 10/20/15   37  
  32. 32. 10/20/15   38   Flood level - Shelter Flood Level Shelter Twitter Classify (Flood level - Shelter)
  33. 33. Krumbs  Used  as  Focused  Micro  Blog   •  One  Click  upload  to  the  FMB-­‐App  Server.   •  The  client  sends:   – Sender  ID   – Photo   – GPS  and  Place   – Time  and  Event   – Emoji  based  Context  and  annota6on   – Any  addi6onal  comments   •  As  JSON  
  34. 34. FMB-­‐App  Server   •  FMB-­‐App  Server  uses  EventShop  for  appropriate   aggrega6ons.   –  Loca6on   –  Geographic  area  (ward,  town,  city,  …)   –  Also  computes  some  rates  of  changes.   –  Determines  Trends   –  Classifies  areas  based  on  evolving  and  current   situa6ons.   •  Allows  drill-­‐down  to  show  even  individual  photos.  
  35. 35. Tweets  VS  Krumbs   Tweets   1.  Tweet  require  thinking  and   typing.   2.  Loca6on  of  a  tweet  and  the   corresponding  event  may   be  different.   3.  Tweets  are  subjec6ve.   Krumbs   1.  Krumbs  have  more   informa6on  and  are   spontaneous.   2.  Krumbs  maintains  event   loca6on.   3.  Krumbs  are  objec6ve.  
  36. 36. Seman6c  Links  for  a  Photo   •  Photo  Level   – Automa6c  crea6on   – Manual  Annota6on   – Manual  Crea6on   •  Segment  Level   – Automa6c  Crea6on   – Manual  Crea6on    
  37. 37. ObjecFve  Self:  From  Personal  Big  Data  
  38. 38. Life  Events  relate  disparate  streams  to  life.   Personal  photos  on  smart  phones  TELL  a  lot   about  you.  
  39. 39. Visual  Web  PlaZorm     Contextual  Reasoning  and  Event  ComputaFons   NavigaFon  and  Search   Krumbs     Next   App   Knowledge  Discovery:    Event  AnalyFcs  and  VisualizaFon   Agro-­‐   Tech   Clean   India   Personal   Sharing  and  CommunicaFon  
  40. 40. Your  Personal  Visual  Web  on   Smartphone   Photo  Cloud   Moment  Capture   Contextual  Reasoning   and   Event  ComputaFons   Event  AnalyFcs   And  VisualizaFon   NavigaFon  and  Search   • Available  on  Android  and  iOS.   • Your  data  remains  on  your  phone   unless  shared.     Sharing  
  41. 41. Developing  An  App:    Agro-­‐Tech   Contextual  Reasoning  and  Event  ComputaFons   NavigaFon  and  Search   Krumbs  for  Agro-­‐Tech     Agro  Knowledge  Discovery:    Event  AnalyFcs  and  VisualizaFon   Agro-­‐   Tech   Sharing  and  CommunicaFon   Popular    Selfie    Food      Agriculture    Shopping  
  42. 42. Technology  for  Building  Visual  Web   • Visual  Authoring  Environment:  HTML  for  Visual   Data.   • Content  Analysis   •  Deep  Contextual  Reasoning:  From  Smartphones,  sensors,  IoT,     personal  history,  social,  personal  and  all  other  events.   •  Event  Clustering  and  recogniFon.   •  Photo  Ranking   •  Combine  with  Deep  Learning  based  Content  Analysis   • Visual  NavigaFon   • Cross  sharing  and  integraFon  with  other  PlaZorms.   • Combine  Algorithmic  and  InteracFve  tools.  
  43. 43. For Visual Web, we need •  Addressing (URI) •  Transfer Protocol (HTTP) •  Authoring and Presentation (HTML) •  Ranking •  Contextual Processing •  Content Analysis •  Privacy and Security •  Information Vs Experience Challenges  
  44. 44. Sky  is  the  Limit.   ApplicaFons:   Lifestyle   Health   Commerce   Surveillance  and  Monitoring   Agriculture   Research  and  development   ConstrucFon   …    
  45. 45. Thanks  for  your  Fme  and  a_enFon.   For  ques6ons:  jain@ics.uci.edu  

×