Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Visual Web keynote at MMSP 2015


Published on

The 21st century began with a major disruption: the rapid rise of smartphones meant that capturing, storing, and sharing photos and their context became easier than using text. Photos and videos communicate directly, without the need for language or literacy. Until recently, photos were used as compelling memories. Now, photos are increasingly used to convey intent and information.related to a moment. A photo may be linked to many other photos along different dimensions. One may also create explicit links among photos or objects in photos. All photos on the Web form a Visual Web that links photos with other photos and other information elements including all documents on the WWW. This Visual Web offers opportunities to address new societal issues and solve many difficult yet unsolved problems. We discuss nature of the Visual Web, technical challenges, and some interesting opportunities in this area.

Published in: Data & Analytics
  • Be the first to comment

Visual Web keynote at MMSP 2015

  1. 1. Weaving  the  Visual  Web       Ramesh  Jain   Dept.  of  Computer  Science   University  of  California,  Irvine  
  2. 2. •  Memories  of  experiences.   •  Explicit  contextual  communicaFon.   – Data  is  a  new  uFlity.   – Text  is  too  abstract  in  many  situaFons.     Duality  of  Photos   Why  do  we  take  photos?  
  3. 3. Where  do  photos  come  from?     •  Drawings     •  Pain6ngs   •  Cameras   •  Smart  Cameras   •  Graphics  
  4. 4. Camera  History  
  5. 5. The  Role  of  Photos  con6nues  to  Evolve   •  Memory   •  Informa6on   – Notes     – Documenta6on   •  Ephemeral    Photos  have  ‘half  life’!  
  6. 6. In 20th century, we tolerated photos in our textual documents. In 21st century, you create visual documents that tolerate text.  
  7. 7. Major Disruption in Photos: From Memories to Information Sources. Photos are the most compelling source of information.  
  8. 8. Experiences Life = Events +
  9. 9. Progress: Knowledge Creation and Propagation Visual   Visual  knowledge   Oral   Oral  knowledge   Textual  knowledge  
  10. 10. Berners-­‐Lee:    Suppose  all  the  informa6on  stored   on  computers  everywhere  were  linked.  Suppose   I  could  program  my  computer  to  create  a  space   in  which  anything  could  be  linked  to  anything.  
  11. 11. Web: Human experiences, knowledge, and understanding captured using associative links in DOCUMENTS.
  12. 12. Text  based   documents  are  not   NATURAL.      Language  and   Literacy  come  in   the  way.  
  13. 13. Imagine if every photo and video captured were connected to every other! And to other information!!  
  14. 14. What is a camera? Captures intensity from a point in the world.
  15. 15. Is  a  Smartphone  camera  sFll  a   camera?   Many  sensors  capture  metadata  related  to  the  moment   and  capture.   •  Exposure Time •  Aperture Diameter •  Flash •  Metering Mode •  ISO Ratings •  Focal Length •  Time •  Location •  Face   Smartphone  camera  captures  events.  
  16. 16. Computa6onal  Representa6on  of  an   event.   Experien6al  Data:       •  Photos   •  Video   •  Audio   •  Accelerometer   •  Heart  rate   •  …  
  17. 17. EMPT: Extractable Mobile Photo Tags Exif   Content   Analysis   EMPT     People   Place   Objects   Events   …  
  18. 18. Krumbs:    Capture  and  Connect   holisFc  experience  of  a  moment.   What:  Objects Who:    People When:  Events Where:  Location Why:  Intent/Emotions
  19. 19. All photos and associated context is automatically Organized in a Web, and Shared (if desired).
  20. 20. Duality  of  Photos:    Relevance  to   MM  and  CompuFng   •  Memories:  Search  and  Retrieval   •  InformaFon  communicaFon:  Web,  Big  Data  
  21. 21. How  do  you  search  photos?   What   Who   When   Where     Why   AssociaFons  related  to  these.  
  22. 22. Searching  ‘For  a  Photo’     and     Searching  ‘From  a  Photo’.   28  
  23. 23. 29  
  24. 24. 30   Who   When   Where   What     Why   AssociaFons  related  to  these.  
  25. 25. Marking  Moments:    Micro  Blogs   •  Facebook  Status  and  Tweets  started  Micro   Blogs.   – Now  there  are  many   – Instagram,  Snapchat,  …   •  Problem  with  Tweets:  More  Noise  less  Data.   •  Time  to  add  Focused  Micro  Blogs     – Sensors     – Importance  of  marking  a  moment  
  26. 26. Waze  
  27. 27. Crowdsourced  SituaFons  
  28. 28. Crowdsourced  SituaFons  
  29. 29. A  Photo  is  a  Click  in  Real  World.   •  Remember  Kodak  Moment!   •  For  each  photo:   •  Unique  ID,     •  All  metadata  of  the  event,   •  Tags,   •  Links,   •  AnnotaFons.  
  30. 30. Crowdsourced  SituaFons   At  435  Main   8:37  AM  
  31. 31. 10/20/15   37  
  32. 32. 10/20/15   38   Flood level - Shelter Flood Level Shelter Twitter Classify (Flood level - Shelter)
  33. 33. Krumbs  Used  as  Focused  Micro  Blog   •  One  Click  upload  to  the  FMB-­‐App  Server.   •  The  client  sends:   – Sender  ID   – Photo   – GPS  and  Place   – Time  and  Event   – Emoji  based  Context  and  annota6on   – Any  addi6onal  comments   •  As  JSON  
  34. 34. FMB-­‐App  Server   •  FMB-­‐App  Server  uses  EventShop  for  appropriate   aggrega6ons.   –  Loca6on   –  Geographic  area  (ward,  town,  city,  …)   –  Also  computes  some  rates  of  changes.   –  Determines  Trends   –  Classifies  areas  based  on  evolving  and  current   situa6ons.   •  Allows  drill-­‐down  to  show  even  individual  photos.  
  35. 35. Tweets  VS  Krumbs   Tweets   1.  Tweet  require  thinking  and   typing.   2.  Loca6on  of  a  tweet  and  the   corresponding  event  may   be  different.   3.  Tweets  are  subjec6ve.   Krumbs   1.  Krumbs  have  more   informa6on  and  are   spontaneous.   2.  Krumbs  maintains  event   loca6on.   3.  Krumbs  are  objec6ve.  
  36. 36. Seman6c  Links  for  a  Photo   •  Photo  Level   – Automa6c  crea6on   – Manual  Annota6on   – Manual  Crea6on   •  Segment  Level   – Automa6c  Crea6on   – Manual  Crea6on    
  37. 37. ObjecFve  Self:  From  Personal  Big  Data  
  38. 38. Life  Events  relate  disparate  streams  to  life.   Personal  photos  on  smart  phones  TELL  a  lot   about  you.  
  39. 39. Visual  Web  PlaZorm     Contextual  Reasoning  and  Event  ComputaFons   NavigaFon  and  Search   Krumbs     Next   App   Knowledge  Discovery:    Event  AnalyFcs  and  VisualizaFon   Agro-­‐   Tech   Clean   India   Personal   Sharing  and  CommunicaFon  
  40. 40. Your  Personal  Visual  Web  on   Smartphone   Photo  Cloud   Moment  Capture   Contextual  Reasoning   and   Event  ComputaFons   Event  AnalyFcs   And  VisualizaFon   NavigaFon  and  Search   • Available  on  Android  and  iOS.   • Your  data  remains  on  your  phone   unless  shared.     Sharing  
  41. 41. Developing  An  App:    Agro-­‐Tech   Contextual  Reasoning  and  Event  ComputaFons   NavigaFon  and  Search   Krumbs  for  Agro-­‐Tech     Agro  Knowledge  Discovery:    Event  AnalyFcs  and  VisualizaFon   Agro-­‐   Tech   Sharing  and  CommunicaFon   Popular    Selfie    Food      Agriculture    Shopping  
  42. 42. Technology  for  Building  Visual  Web   • Visual  Authoring  Environment:  HTML  for  Visual   Data.   • Content  Analysis   •  Deep  Contextual  Reasoning:  From  Smartphones,  sensors,  IoT,     personal  history,  social,  personal  and  all  other  events.   •  Event  Clustering  and  recogniFon.   •  Photo  Ranking   •  Combine  with  Deep  Learning  based  Content  Analysis   • Visual  NavigaFon   • Cross  sharing  and  integraFon  with  other  PlaZorms.   • Combine  Algorithmic  and  InteracFve  tools.  
  43. 43. For Visual Web, we need •  Addressing (URI) •  Transfer Protocol (HTTP) •  Authoring and Presentation (HTML) •  Ranking •  Contextual Processing •  Content Analysis •  Privacy and Security •  Information Vs Experience Challenges  
  44. 44. Sky  is  the  Limit.   ApplicaFons:   Lifestyle   Health   Commerce   Surveillance  and  Monitoring   Agriculture   Research  and  development   ConstrucFon   …    
  45. 45. Thanks  for  your  Fme  and  a_enFon.   For  ques6ons: