Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.



Published on

  • Be the first to comment

  • Be the first to like this


  1. 1. Pervasive Pixels (Columbia University Dept. of Computer Science) Henning Schulzrinne (PI) Steven K. Feiner Gail Kaiser John Kender Kathleen McKeown
  2. 2. Proposed Research <ul><li>Goal: seamless mobile multimedia collaboration across distance </li></ul><ul><li>Integrate advances across fields </li></ul><ul><ul><ul><li>Collaborative work </li></ul></ul></ul><ul><ul><ul><li>Graphical and visual interfaces </li></ul></ul></ul><ul><ul><ul><li>Spoken language understanding and generation </li></ul></ul></ul><ul><ul><ul><li>Vision sensing and understanding </li></ul></ul></ul><ul><ul><ul><li>Networking and security </li></ul></ul></ul>
  3. 3. Contributions <ul><li>Contextual information management </li></ul><ul><ul><ul><li>use workflow to determine display content </li></ul></ul></ul><ul><ul><ul><li>multimedia summaries of past and present sessions </li></ul></ul></ul><ul><li>Harmonizing physical and virtual environments </li></ul><ul><ul><ul><li>map changing virtual information onto physical displays </li></ul></ul></ul><ul><ul><ul><li>map layout of physical environment onto virtual space </li></ul></ul></ul><ul><li>Network services </li></ul><ul><ul><ul><li>clear, flexible interface to common services </li></ul></ul></ul><ul><ul><ul><li>authentication and privacy support </li></ul></ul></ul><ul><ul><ul><li>infrastructure for persistent large displays </li></ul></ul></ul>
  4. 4. Features of Research Infrastructure <ul><li>Large numbers of instrumented multi-display workspaces </li></ul><ul><li>Networked mobile devices of various capabilities </li></ul><ul><li>Transparent and automatic adaptability to changes of place, platform or group </li></ul><ul><li>Support for a wide range of hardware and software, from commercial to novel </li></ul>
  5. 5. Proposed Research Infrastructure <ul><li>Outfit informal areas for collaboration </li></ul><ul><ul><ul><li>Public areas for walk-by interaction </li></ul></ul></ul><ul><ul><ul><li>Multiple touch displays, cameras, audio </li></ul></ul></ul><ul><ul><ul><li>Portable units </li></ul></ul></ul><ul><li>Stationary setups </li></ul><ul><ul><ul><li>Multiple displays, video cameras, audio </li></ul></ul></ul><ul><ul><ul><li>Seminar room, meeting rooms </li></ul></ul></ul><ul><ul><ul><li>12 faculty offices </li></ul></ul></ul><ul><li>User-based personalization: user location </li></ul><ul><ul><ul><li>Triangulation on mobile devices </li></ul></ul></ul><ul><ul><ul><li>Visual tracking </li></ul></ul></ul><ul><ul><ul><li>Standard methods (e.g., active badge) </li></ul></ul></ul>
  6. 6. Public areas – walk by stations <ul><li>Multiple touch displays, video projectors and cameras, embedded computers, speakers and microphones </li></ul>
  7. 7. IR/RF badge network PC proj. camera card reader loudspeaker ceiling electronic whiteboard microphone array Design for Walk-by Collaboration Station
  8. 8. Public Areas – informal gatherings
  9. 9. Meeting Room <ul><li>Remote-controlled pan-tilt video cameras and projectors, Omnicam, conference table microphones, automatic audio mixer, ceiling speakers </li></ul>
  10. 10. Faculty Office <ul><li>Mimio electronic whiteboard, XGA video projector, Ethernet speaker phone, wall-mounted pan-tilt video camera, PocketPCs </li></ul>
  11. 11. Seminar room <ul><li>Omnicam omnidrectional audience camera, high-resolution DV video camera, 2 pan-tilt speaker cameras, ceiling mounted microphones, electronic whiteboard, XGA high-brightness video projectors </li></ul>
  12. 12. Functionalities <ul><li>Conferencing </li></ul><ul><ul><ul><li>Internet conferencing server to mix IP and PSTN audio streams </li></ul></ul></ul><ul><li>Interconnection with analog phone </li></ul><ul><ul><ul><li>Digital hybrid connects digital or analog sound to existing telephone system in classroom </li></ul></ul></ul><ul><ul><ul><li>Network voice-over-IP interface attached to Nortel Meridian PBX for 20 simultaneous conversations </li></ul></ul></ul><ul><li>Multi-processor servers and IA64 compute and database server </li></ul><ul><li>File storage </li></ul><ul><li>Face, speaker and fingerprint recognition </li></ul><ul><li>Backup facilities: 2 printers and tape library system </li></ul>
  13. 13. Initial results <ul><li>HCI: gesture-based user interface for public kiosk </li></ul><ul><ul><li>mouse replacement for pointing and selecting </li></ul></ul><ul><ul><li>uses frontal and side camera </li></ul></ul><ul><li>Security: disCFS and WebDAVA secure file systems </li></ul><ul><ul><li>disCFS: NFS with credentials instead of authorization </li></ul></ul><ul><ul><li>WebDAVA: grant restricted access to resources using HTTP and Java applets </li></ul></ul><ul><li>Web-based collaboration: </li></ul><ul><ul><li>content on all kinds of devices </li></ul></ul><ul><ul><li>pass DOM through a series of filters and transformations  HTML </li></ul></ul><ul><li>Ubiquitous multimedia communications infrastructure </li></ul><ul><ul><li>being commercialized; I2 demonstration </li></ul></ul><ul><ul><li>input into standardization (IETF) </li></ul></ul>
  14. 14. Ubiquitous Computing <ul><li>Traditionally, focus on closed environments </li></ul><ul><ul><li>proprietary protocols </li></ul></ul><ul><ul><li>single (trusted) user class </li></ul></ul><ul><ul><li>single site (room, lab, home, …) </li></ul></ul><ul><ul><li>stand-alone components (“video conferencing”) </li></ul></ul><ul><li>PP focuses on whole system and user experience </li></ul><ul><li>Pervasive Pixels networking component: </li></ul><ul><ul><li>standard protocols: </li></ul></ul><ul><ul><ul><li>SIP for media configuration, event notification, instant multimedia messaging </li></ul></ul></ul><ul><ul><ul><li>SLP for service discovery </li></ul></ul></ul><ul><ul><li>integration of presence and user context </li></ul></ul><ul><ul><ul><li>standardization in the IETF (RPID) </li></ul></ul></ul><ul><ul><li>location-based services </li></ul></ul><ul><ul><ul><li>user context </li></ul></ul></ul><ul><ul><ul><li>user authorization </li></ul></ul></ul><ul><ul><ul><li>service location </li></ul></ul></ul>
  15. 15. Mobility in Pervasive Pixels <ul><li>Terminal mobility </li></ul><ul><ul><li>application-layer mobility complements L3 mobility </li></ul></ul><ul><li>Session mobility </li></ul><ul><ul><li>move active sessions to devices found in the environment </li></ul></ul><ul><ul><li> service discovery </li></ul></ul><ul><li>Service mobility </li></ul><ul><ul><li>move configuration to new devices </li></ul></ul><ul><li>Personal mobility </li></ul><ul><ul><li>one user, many devices </li></ul></ul>
  16. 16. Location-based services <ul><li>Traditionally, focus on geospatial location (e.g., GPS) </li></ul><ul><li>But other aspects as important: </li></ul><ul><ul><li>civil location (often more intuitive) </li></ul></ul><ul><ul><li>type of place (home vs. office; outdoors vs. theatre) </li></ul></ul><ul><ul><li>behavioral: distraction, privacy, appropriateness </li></ul></ul><ul><li>Experimenting with low-complexity location mechanisms: </li></ul><ul><ul><li>IR/RF active badges with low installation cost (Ivistar) </li></ul></ul><ul><ul><li>BlueTooth location beacons </li></ul></ul><ul><ul><li>LAN backtracking and DHCP </li></ul></ul><ul><ul><li>swipe cards and i-buttons </li></ul></ul>DHCP server 458/17  Rm. 815 458/18  Rm. 816 DHCP answer: sta=DC loc=Rm815 lat=38.89868 long=77.03723 8:0:20:ab:d5:d CDP + SNMP 8:0:20:ab:d5:d  458/17
  17. 17. Some initial lessons learned <ul><li>Usage: remote presence from UKy during sabbatical </li></ul><ul><ul><li>research group meetings </li></ul></ul><ul><ul><li>departmental site visit </li></ul></ul><ul><ul><li>thesis proposals and defenses </li></ul></ul><ul><li>Perception: “Multimedia collaboration is a mature field” </li></ul><ul><li>Reality: It doesn’t work much better than in 1992 </li></ul><ul><ul><li>still fails in hard-to-diagnose ways </li></ul></ul><ul><ul><li>quality better, but echo, feedback and level issues remain </li></ul></ul><ul><li>Integration between synchronous and asynchronous collaboration </li></ul><ul><ul><li>integrating documents, minutes, … </li></ul></ul><ul><li>Transition from call-focused to presence-focused </li></ul><ul><ul><li>much larger use of asynchronous collaboration (email, bulletin boards, …) </li></ul></ul><ul><li>Working with start-up company: new IP-based departmental communication system to replace PBX </li></ul>
  18. 18. Columbia SIP servers (CINEMA) Internal Telephone Extn: 7040 SIP/PSTN Gateway Department PBX Web based configuration Telephone switch SQL database sipd: Proxy, redirect, registrar server Extn: 7134 [email_address] NetMeeting H.323 rtspd: media server sipum: Unified messaging Quicktime RTSP clients RTSP Extn: 7136 713x Single machine SNMP (Network Management) sipconf: Conference server siph323: SIP-H.323 translator Local/long distance 1-212-5551212 Web server
  19. 19. Larger lessons for multimedia systems research <ul><li>Software tool support for multimedia communications lacking </li></ul><ul><ul><li>most are applications, not building blocks </li></ul></ul><ul><ul><li>cross-platform research media tools are getting very old and creaky (vic, rat, etc.) </li></ul></ul><ul><ul><li>multi-party support very weak (multicast never happened) </li></ul></ul><ul><li>Components designed to be operated by humans </li></ul><ul><ul><li>IP phones only have HTTP/HTML interface </li></ul></ul><ul><ul><li>video projectors just proprietary configuration </li></ul></ul><ul><li>Lots of components, but hard to evaluate in real use </li></ul><ul><ul><li>still mostly barely demo quality: audio delay, echo, random failures </li></ul></ul><ul><ul><li>people will fall back to good ol’ PSTN quickly </li></ul></ul>
  20. 20. Conclusion <ul><li>Pervasive Pixels = attempt to integrate multiple modalities into system, not just grouping of components </li></ul><ul><li>Evaluation in real usage, not just demos </li></ul><ul><li>Spread throughout the department, not just lab </li></ul>