Olivier Scheer - Solutions for Immersive 3D Video Communication

1,295 views

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,295
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Olivier Scheer - Solutions for Immersive 3D Video Communication

  1. 1. Fraunhofer Image Processing Heinrich Hertz InstituteSolutions for Immersive 3D VideoCommunicationiMinds, Ghent, Belgium, 16th December 2010 Dr. Dr Oliver Schreer Nicole Atzpadin Wolfgang Waizenegger Ingo F ld I FeldmannFraunhofer Heinrich Hertz Institute, Einsteinufer 37, 10587 Berlin www.hhi.fraunhofer.de 16. Dec. 2010 1
  2. 2. Image Processing Outline  State of the art in telepresence systems and 3D videoconferencing  Existing drawbacks of current systems  Virtual i Vi t l view rendering d i  Conclusion© Dr. Oliver Schreer 16. Dec. 2010 2
  3. 3. Image Processing SoA of Telepresence Systems telepresence system by CISCO HP Halo telepresence Polycom TPX system system© Dr. Oliver Schreer 16. Dec. 2010 3
  4. 4. Image Processing SoA of 3D Videoconferencing Coliseum by HP, Palo Alto, CA, 2003 VIRTUE System by FP5-IST-VIRTUE, 2003 MultiView by Univ. of California, Berkeley, 2004 im.point by Fraunhofer HHI, 2004© Dr. Oliver Schreer 16. Dec. 2010 4
  5. 5. Image Processing Outline State of the art in telepresence systems and 3D videoconferencing Existing drawbacks of current systems Virtual view rendering Conclusion© Dr. Oliver Schreer 16. Dec. 2010 5
  6. 6. Image Processing Existing Drawbacks Misleading gesture representation in multi-point setups (who is pointing at whom?)© Dr. Oliver Schreer 16. Dec. 2010 6
  7. 7. Misleading Viewing Direction and Pointing Image Processing Gesture in Commercial Systems What view of the remote participant are you expecting from the position of this local participant ? View f Vi from l ft local t right remote left l l to i ht t View f Vi from right l i ht local t right remote l to i ht t© Dr. Oliver Schreer 16. Dec. 2010 7
  8. 8. Image Processing The Principle of the Shared Table 0 m 1 ,0 R emote C onferees  A 0 m 2 ,0 L ocal C onferees 120° R emote C onferees  B© Dr. Oliver Schreer 16. Dec. 2010 8
  9. 9. Image Processing Existing Drawbacks Misleading gesture representation in multi-point setups (who is pointing at whom?) shared table principle correct t camera mounting ti No stereoscopic viewing© Dr. Oliver Schreer 16. Dec. 2010 9
  10. 10. Image Processing Which Display Technologies for 3D? Stereoscopic S i displays Autostereoscopic flat displays p y© Dr. Oliver Schreer 16. Dec. 2010 10
  11. 11. Image Processing The Answer is Given by Our Chancellor© Dr. Oliver Schreer 16. Dec. 2010 11
  12. 12. Image Processing Existing Drawbacks Misleading gesture representation in multi-point setups (who is pointing at whom?) shared table principle correct t camera mounting ti No stereoscopic viewing auto-stereoscopic displays No multi-perspective viewing for multiple users at one site novel multi-perspective 3D display by Philips (see talk by Harm, Belt p p p y y p ( y , from Philips)© Dr. Oliver Schreer 16. Dec. 2010 12
  13. 13. Image Processing Example© Dr. Oliver Schreer 16. Dec. 2010 13
  14. 14. Image Processing Existing Drawbacks Misleading gesture representation in multi-point setups (who is pointing at whom?) shared table principle correct camera mounting No stereoscopic viewing auto-stereoscopic displays No multi-perspective viewing for multiple users at one site novel multi-perspective 3D display by Philips (see talk by Harm, Belt from Philips) Lack of eye contact© Dr. Oliver Schreer 16. Dec. 2010 14
  15. 15. Image Processing Problem of Eye Contact Problem Displacement angle Solution Virtual camera Realization • Di Disparity/depth it /d th estimation • Virtual rotation of conferee© Dr. Oliver Schreer 16. Dec. 2010 15
  16. 16. Image Processing Summary Misleading gesture representation in multi-point setups (who is pointing at whom?) shared table principle correct camera mounting No stereoscopic viewing auto-stereoscopic displays No multi-perspective viewing for multiple users at one site novel multi-perspective 3D display Lack of eye contact Real-time 3D video processing is required for  support of multi-view 3D di l t f lti i displays  provision of eye contact by virtual view rendering© Dr. Oliver Schreer 16. Dec. 2010 16
  17. 17. Image Processing Outline State of the art in telepresence systems and 3D videoconferencing Existing drawbacks of current systems Virtual view rendering Conclusion© Dr. Oliver Schreer 16. Dec. 2010 17
  18. 18. Image Processing Virtual View Rendering top camera© Dr. Oliver Schreer 16. Dec. 2010 18
  19. 19. Image Processing Virtual View Rendering video + depth for virtual camera warp virtual view© Dr. Oliver Schreer 16. Dec. 2010 19
  20. 20. Image Processing The approach horizontal  disparities horizontal small  baseline DE trifocal  4 video  consistency  robust  streams check vertical small  disparities baseline DE vertical  disparities patch sweeping  refinement video +  depth (n)  Initial disparity/depth estimation based on hybrid-recursive matching (g (HRM) )  Robust disparities after trifocal consistency check  GPU based patch sweeping for refinement in depth© Dr. Oliver Schreer 16. Dec. 2010 20
  21. 21. Image Processing Principle of patch sweeping  Change of orientation of a surflet in the reference camera to match best neighbour cameras  User d fi d d th resolution U defined depth l ti possible  High p g parallelization p possible  Well suited for graphic card implementation© Dr. Oliver Schreer 16. Dec. 2010 21
  22. 22. Image Processing Depth analysis results HRM result for pixel accuracy, sub-pel accuracy of factor 2, sub-pel accuracy of factor 4 Patch sweeping result: original view (left), synthesized view ( p g g ( ), y (middle), depth map (right) ), p p( g )© Dr. Oliver Schreer 16. Dec. 2010 22
  23. 23. Image Processing Depth-Image-Based Rendering M m1 m2 real view virtual view Cv C1© Dr. Oliver Schreer 16. Dec. 2010 23
  24. 24. Simulation Results on Virtual View Image Processing Synthesis top camera ground truth camera© Dr. Oliver Schreer 16. Dec. 2010 24
  25. 25. Simulation Results on Virtual View Image Processing Synthesis top camera interpolated view© Dr. Oliver Schreer 16. Dec. 2010 25
  26. 26. Image Processing Original Video + Depth Camera 1: original video and depth Camera 2 C 2: original video and depth© Dr. Oliver Schreer 16. Dec. 2010 26
  27. 27. Original Views and Target Ground Truth Image Processing Camera original view target view© Dr. Oliver Schreer 16. Dec. 2010 27
  28. 28. Image Processing Target View and Synthesized View target view synthesized view© Dr. Oliver Schreer 16. Dec. 2010 28
  29. 29. Video Example: Target View and Image Processing Synthesized View target view synthesized view© Dr. Oliver Schreer 16. Dec. 2010 29
  30. 30. Image Processing The Virtual Eye Contact Engine First demonstration to the public at 3D Stereo Media, Liège, 8.-10. December 2010!!© Dr. Oliver Schreer 16. Dec. 2010 30
  31. 31. Image Processing The Virtual Eye Contact Engine© Dr. Oliver Schreer 16. Dec. 2010 31
  32. 32. Image Processing Conclusion  Commercial video communication systems still lack with respect to true tele-immersion  Limited gesture awareness  No eye contact  Solutions for direct eye contact on prototype level available  Combined approach of multi-view stereo and patch sweeping pp p p g  Exploitation of graphic cards  High-resolution, high quality rendering achieved© Dr. Oliver Schreer 16. Dec. 2010 32
  33. 33. Image Processing Thank you for your attention! i ! Contact: Oliver.Schreer@hhi.fraunhofer.de© Dr. Oliver Schreer 16. Dec. 2010 33

×