Fall 2008 CS 212 Multimedia Systems and Applications Instructor: Ramesh Jain
Projects Based on Research in  Experiential Computing Laboratory <ul><li>Based on the concept of EventWeb </li></ul><ul><l...
Tim Berners-Lee  thought: <ul><li>Suppose all the information stored on computers everywhere were linked. </li></ul><ul><l...
That resulted in the Web <ul><li>DocumentWeb </li></ul><ul><ul><li>Each node is a ‘Page’ or a document. </li></ul></ul><ul...
Continuing the Evolution of the Web <ul><li>Consider a Web in which each node  </li></ul><ul><ul><li>Is an  event </li></u...
Different Facets of an Event <ul><li>What  </li></ul><ul><li>When </li></ul><ul><li>Where </li></ul><ul><li>Who </li></ul>...
Time 1- dimensional Space Events Happen
Time 1- dimensional Space EventWeb
Stage 1: Event Ingestion Atomic Event Entry Event Base
Stage 2: Event Clustering/Linking Event Grouping, Linking, Assimilation Event Base Atomic Event Entry
Stage 3: Event Presentation, Navigation Event Base Event Presentation/ Navigation Event Grouping, Linking, Assimilation At...
Multimedia Storytelling <ul><li>Collect information about events </li></ul><ul><ul><li>All relevant events </li></ul></ul>...
Time 1- dimensional Space Multimedia Storytelling  Text Video Photo
Experiential Media Management   Environment <ul><li>Event-based </li></ul><ul><li>Photo-taking is an event </li></ul><ul><...
EMME Event Cycle Event Base Event Presentation/ Navigation Event Grouping, Linking, Assimilation Atomic Event Entry EXIF F...
Modern Cameras <ul><li>Are more than ‘Camera Obscura’. </li></ul><ul><li>Many sensors capture scene context and store it a...
Examples: Photos are  Assigned tags only based on EXIF
EMME Event Cycle Event Base Event Presentation/ Navigation Event Grouping, Linking, Assimilation Atomic Event Entry EXIF F...
Photo Stream Segmentation <ul><li>Definition: given a photo stream </li></ul>Segment Event 1 Event 2 Event 3
EMME Event Cycle Event Base Event Presentation/ Navigation Event Grouping, Linking, Assimilation Atomic Event Entry EXIF F...
Using EMME <ul><li>Searching for photo  </li></ul><ul><ul><li>I2R visit  </li></ul></ul><ul><li>Creating Albums: </li></ul...
Sentient EventWeb <ul><li>Use sensors to create events. </li></ul><ul><ul><li>This class </li></ul></ul><ul><li>Infrastruc...
First Version: E2E <ul><li>Person to Person </li></ul><ul><li>Device to Device </li></ul><ul><ul><li>Telephone </li></ul><...
Event Cycle Event Base Event Presentation/ Navigation Event Grouping, Linking, Assimilation Atomic Event Entry
Natural interaction Semantic interaction Seamless interaction Bi-directional connectivity Not depend on physical similarit...
Environment <ul><li>Fixed objects </li></ul><ul><li>Active Objects </li></ul><ul><li>Events and Activities </li></ul><ul><...
Key Idea <ul><li>Put enough  sensors and devices  to make mediation ( sensors and devices ) disappear. </li></ul><ul><li>S...
System Architecture ES 2 ES 1 ES 3 ES 4 ES 5 JSM 2 JSM 1
Node Architecture Sensors Data Acquisition & Analysis Situation Model Env. Model + Security settings Event Server Multimed...
Event e.g. E1_Cam2, E1_Mic2 e.g. Lip-motion=true,  Volume=High e.g. Env.1,  Position: 102,145  Visitor Chair e.g. Telemedi...
Env. 1 Env. 4 JSM 2 JSM 1 Env. 5 Env. 3 Env. 2 Joint Situation Model
Telemedicine Application
Telemedicine Application
Many Potential Projects: Photos <ul><li>Features to classify images:  </li></ul><ul><ul><li>Detect specific objects (sky, ...
Many Potential Projects: E2E <ul><li>Tracking people using multiple sensors. </li></ul><ul><li>Mobile E2E mode: switching ...
Upcoming SlideShare
Loading in …5
×

© Ramesh Jain 1 Fall 2008

385 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
385
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
28
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

© Ramesh Jain 1 Fall 2008

  1. 1. Fall 2008 CS 212 Multimedia Systems and Applications Instructor: Ramesh Jain
  2. 2. Projects Based on Research in Experiential Computing Laboratory <ul><li>Based on the concept of EventWeb </li></ul><ul><li>Two majore research projects </li></ul><ul><ul><li>Personal Media Management </li></ul></ul><ul><ul><li>Environment to Environment Connectivity (E2E) </li></ul></ul>
  3. 3. Tim Berners-Lee thought: <ul><li>Suppose all the information stored on computers everywhere were linked. </li></ul><ul><li>Suppose I could program my computer to create a space in which anything could be linked to anything. </li></ul>Others – including Bush -- had that idea earlier but the technology was not ready.
  4. 4. That resulted in the Web <ul><li>DocumentWeb </li></ul><ul><ul><li>Each node is a ‘Page’ or a document. </li></ul></ul><ul><ul><li>Pages are linked through explicit referential links </li></ul></ul>
  5. 5. Continuing the Evolution of the Web <ul><li>Consider a Web in which each node </li></ul><ul><ul><li>Is an event </li></ul></ul><ul><ul><li>Has informational as well as experiential data </li></ul></ul><ul><ul><li>Is connected to other nodes using </li></ul></ul><ul><ul><ul><li>Referential links </li></ul></ul></ul><ul><ul><ul><li>Structural links </li></ul></ul></ul><ul><ul><ul><li>Relational links </li></ul></ul></ul><ul><ul><ul><li>Causal links </li></ul></ul></ul><ul><ul><li>Explicit links can be created by anybody </li></ul></ul><ul><li>This EventWeb is connected to other Webs. </li></ul>
  6. 6. Different Facets of an Event <ul><li>What </li></ul><ul><li>When </li></ul><ul><li>Where </li></ul><ul><li>Who </li></ul><ul><li>Why </li></ul><ul><li>How </li></ul>
  7. 7. Time 1- dimensional Space Events Happen
  8. 8. Time 1- dimensional Space EventWeb
  9. 9. Stage 1: Event Ingestion Atomic Event Entry Event Base
  10. 10. Stage 2: Event Clustering/Linking Event Grouping, Linking, Assimilation Event Base Atomic Event Entry
  11. 11. Stage 3: Event Presentation, Navigation Event Base Event Presentation/ Navigation Event Grouping, Linking, Assimilation Atomic Event Entry
  12. 12. Multimedia Storytelling <ul><li>Collect information about events </li></ul><ul><ul><li>All relevant events </li></ul></ul><ul><ul><li>Appropriate information </li></ul></ul><ul><ul><li>In right media </li></ul></ul><ul><li>Stories are sequence of events. </li></ul><ul><ul><li>Stories/Novels </li></ul></ul><ul><ul><li>Drama </li></ul></ul><ul><ul><li>Movies </li></ul></ul>Present right event information using right media in right order.
  13. 13. Time 1- dimensional Space Multimedia Storytelling Text Video Photo
  14. 14. Experiential Media Management Environment <ul><li>Event-based </li></ul><ul><li>Photo-taking is an event </li></ul><ul><li>Photos are taken of an event </li></ul><ul><li>Albums represents events and objects </li></ul><ul><li>Presentations are to share and communicate experiences </li></ul>First Photos then other media.
  15. 15. EMME Event Cycle Event Base Event Presentation/ Navigation Event Grouping, Linking, Assimilation Atomic Event Entry EXIF Features Tags/ Context
  16. 16. Modern Cameras <ul><li>Are more than ‘Camera Obscura’. </li></ul><ul><li>Many sensors capture scene context and store it along with intensity values. </li></ul><ul><li>EXIF data is all metadata related to the Event . </li></ul><ul><li>And we throw away all this valuable information!!! </li></ul>
  17. 17. Examples: Photos are Assigned tags only based on EXIF
  18. 18. EMME Event Cycle Event Base Event Presentation/ Navigation Event Grouping, Linking, Assimilation Atomic Event Entry EXIF Features Tags/ Context Photo stream Segment. Event Ontology User Annot- ations
  19. 19. Photo Stream Segmentation <ul><li>Definition: given a photo stream </li></ul>Segment Event 1 Event 2 Event 3
  20. 20. EMME Event Cycle Event Base Event Presentation/ Navigation Event Grouping, Linking, Assimilation Atomic Event Entry EXIF Features Tags/ Context Photo stream Segment. Event Ontology User Annot- ations Story Telling Search Explore
  21. 21. Using EMME <ul><li>Searching for photo </li></ul><ul><ul><li>I2R visit </li></ul></ul><ul><li>Creating Albums: </li></ul><ul><ul><li>Professional in Singapore </li></ul></ul><ul><ul><li>Tourism in Summer 2008 </li></ul></ul><ul><li>Telling stories </li></ul><ul><ul><li>What did I do in Summer 2008 </li></ul></ul><ul><li>Scenario: In December 2008, I have 20,000 pictures taken in 2008. How do I (semi-automatically) select 25 to send to </li></ul><ul><ul><li>My mother </li></ul></ul><ul><ul><li>The uncle that I hate </li></ul></ul><ul><ul><li>My personal friend </li></ul></ul><ul><ul><li>My professional friend </li></ul></ul><ul><ul><li>… </li></ul></ul>
  22. 22. Sentient EventWeb <ul><li>Use sensors to create events. </li></ul><ul><ul><li>This class </li></ul></ul><ul><li>Infrastructure to create and link all such events. </li></ul><ul><ul><li>Index </li></ul></ul><ul><ul><li>Search </li></ul></ul><ul><ul><li>Present </li></ul></ul><ul><li>Archive events and put them on the Web. </li></ul>
  23. 23. First Version: E2E <ul><li>Person to Person </li></ul><ul><li>Device to Device </li></ul><ul><ul><li>Telephone </li></ul></ul><ul><ul><li>Video Conferencing </li></ul></ul><ul><li>Environment to Environment (E2E) </li></ul>Communication Modes:
  24. 24. Event Cycle Event Base Event Presentation/ Navigation Event Grouping, Linking, Assimilation Atomic Event Entry
  25. 25. Natural interaction Semantic interaction Seamless interaction Bi-directional connectivity Not depend on physical similarity Handle privacy Event-based architecture Scalable architecture Sensor abstraction Multimodal information No fixed application Live and archived modes E2E Communication
  26. 26. Environment <ul><li>Fixed objects </li></ul><ul><li>Active Objects </li></ul><ul><li>Events and Activities </li></ul><ul><li>Sensor to capture Objects and Events </li></ul><ul><ul><li>Cameras, microphones, pressure, RFID, … </li></ul></ul><ul><li>Displays and Actuators </li></ul><ul><ul><li>Projectors, Displays, Speakers, … </li></ul></ul>
  27. 27. Key Idea <ul><li>Put enough sensors and devices to make mediation ( sensors and devices ) disappear. </li></ul><ul><li>System detects events and objects and sends appropriate data streams and displays/renders incoming streams on appropriate devices </li></ul><ul><li>Bring back Person to Person communication </li></ul>
  28. 28. System Architecture ES 2 ES 1 ES 3 ES 4 ES 5 JSM 2 JSM 1
  29. 29. Node Architecture Sensors Data Acquisition & Analysis Situation Model Env. Model + Security settings Event Server Multimedia Database Presentation Event Base Distribution/ Networking
  30. 30. Event e.g. E1_Cam2, E1_Mic2 e.g. Lip-motion=true, Volume=High e.g. Env.1, Position: 102,145 Visitor Chair e.g. Telemedicine App., Consultation room e.g. Patient talking Data Acquisition & Analysis Env. Model Situation Model Event Detection Stages
  31. 31. Env. 1 Env. 4 JSM 2 JSM 1 Env. 5 Env. 3 Env. 2 Joint Situation Model
  32. 32. Telemedicine Application
  33. 33. Telemedicine Application
  34. 34. Many Potential Projects: Photos <ul><li>Features to classify images: </li></ul><ul><ul><li>Detect specific objects (sky, grass, mountains, …) and label images. </li></ul></ul><ul><ul><li>Face detection and recognition. </li></ul></ul><ul><ul><li>Person recognition – faces to other body information to person. </li></ul></ul><ul><ul><li>Some simple ‘event’ (like a talk, or a wedding) recognition from photographs. </li></ul></ul><ul><li>Using calendar information to assign information to images. </li></ul><ul><li>Integrating Social Network Information In Personal Media Management </li></ul><ul><li>Mobile App for uploading Event information.  Can use Android APIs </li></ul><ul><li>Select 1 or a few images from a group to represent the group. This may be based on quality of images or content of images. </li></ul><ul><li>Prepare a collage of images to represent a group. Use some theme for selecting and displaying your images. </li></ul><ul><li>Voice annotation to photos and use of speech recognition to convert voice tags to text tags. </li></ul><ul><li>Summarize an Event from Photos from many sources. Many people shoot a photo for an event. How to create a good event representation from photos from multiple sources? </li></ul>
  35. 35. Many Potential Projects: E2E <ul><li>Tracking people using multiple sensors. </li></ul><ul><li>Mobile E2E mode: switching from a mobile phone to a much richer environment. </li></ul><ul><li>Creating ‘virtual camera’ for heterogeneous camera connections. </li></ul><ul><li>Anywhere projector </li></ul><ul><li>IP Audio streaming: Send and receive audio by just specifying the IP address – without knowing the devices connected there. </li></ul>

×