0
Interactive Video Tours MSR Interactive Visual Media Group //msrweb/vision/IBR Rick Szeliski, Sing Bing Kang, Matt Uyttend...
Being There: Video-Based Reality Project <ul><li>A new computer-mediated interactive medium </li></ul><ul><li>Goal:   Comb...
Being There: Video-Based Reality Project <ul><li>How? </li></ul><ul><li>Capture real-world video footage with an omni-dire...
Virtual Tours <ul><li>Move camera along a rail (“dolly track”) and play back a 360   video </li></ul><ul><li>Applications...
Surround video acquisition system  OmniCam (six-camera head)
OmniCam <ul><li>Built by Point Grey Research ( Ladybug ) </li></ul><ul><li>Six camera head </li></ul><ul><li>Portable hard...
Acquisition platforms <ul><li>Robotic cart </li></ul><ul><li>Wearable </li></ul>
Calibration: Lens distortion Original (distorted) image Output (corrected, perspective) image
Calibration: Vignetting Before correction After correction
Stitching (Only 4 of 6 images shown here)
Problem with simple feathering
Multiperspective plane sweep With MPPS Without MPPS
Feature tracking for calibration and ego-motion Points  tracked in all frames in all 6 cameras Edges  tracked in all frame...
Stabilization Before motion stabilization After motion stabilization Align  frame- to-frame and distribute  heading
Augmented surround video A movie clip from the  original surround data. The augmented movie clip. Surround augmented with ...
Viewing Software <ul><li>D3D + DirectShow + XML scene graphs </li></ul><ul><li>Model: 6 texture mapped planes </li></ul><u...
Demo A Virtual Home Tour
<ul><li>Head-mounted outdoor scene acquisition </li></ul><ul><li>Map control </li></ul><ul><li>Localized audio for a riche...
Data acquisition <ul><li>Walk through the garden with the Ladybug camera mounted on helmet </li></ul><ul><li>Acquisition d...
Map control <ul><li>The garden map </li></ul><ul><li>Drawing the  </li></ul><ul><li>acquisition path </li></ul><ul><li>Map...
Audio processing and streaming <ul><li>All audio sources are: </li></ul><ul><li>Attenuated (based on  </li></ul><ul><li>di...
Bifurcation handling <ul><li>Hypothesize, align, choose best pair as bifurcation </li></ul><ul><li>Choice of path depends ...
Bifurcation handling <ul><li>Hypothesize, align, choose best pair as bifurcation </li></ul><ul><li>Choice of path depends ...
Demo 2 A walk in Bellevue Botanical Garden
<ul><li>High-Dynamic range (HDR) imaging </li></ul><ul><li>Complex indoor path navigation </li></ul>3 rd  scenario: High-E...
<ul><li>Combine multiple exposures to obtain a high dynamic range (HDR) image. </li></ul>High dynamic range imaging Overex...
High dynamic range imaging All frames Underexposed frames Overexposed frames <ul><li>HDR imaging is challenging for videos...
Demo 3 A High-End Home
Sample walkthroughs Garden Tour High-End Home Tour Simple Home Tour
Features and statistics <ul><li>Rapid scene acquisition </li></ul><ul><li>Map control </li></ul><ul><li>Localized audio fo...
Current/future work <ul><li>Scene Modeling </li></ul><ul><ul><li>Motion estimation and map generation </li></ul></ul><ul><...
Upcoming SlideShare
Loading in...5
×

*αí*ß

194

Published on

Published in: Business, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
194
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "*αí*ß"

  1. 1. Interactive Video Tours MSR Interactive Visual Media Group //msrweb/vision/IBR Rick Szeliski, Sing Bing Kang, Matt Uyttendaele, Simon Winder, Antonio Criminisi
  2. 2. Being There: Video-Based Reality Project <ul><li>A new computer-mediated interactive medium </li></ul><ul><li>Goal: Combine the interactivity of 3D games and virtual worlds with the realism and richness of film and video </li></ul>
  3. 3. Being There: Video-Based Reality Project <ul><li>How? </li></ul><ul><li>Capture real-world video footage with an omni-directional camera and add interactive video/graphic elements </li></ul><ul><li>Application: Video-Based Walkthroughs [IEEE CG&A May/June 2004] </li></ul>
  4. 4. Virtual Tours <ul><li>Move camera along a rail (“dolly track”) and play back a 360  video </li></ul><ul><li>Applications: </li></ul><ul><ul><li>Homes and architecture </li></ul></ul><ul><ul><li>Outdoor locations (tourist destinations) </li></ul></ul>
  5. 5. Surround video acquisition system OmniCam (six-camera head)
  6. 6. OmniCam <ul><li>Built by Point Grey Research ( Ladybug ) </li></ul><ul><li>Six camera head </li></ul><ul><li>Portable hard drives, fiber-optic link </li></ul><ul><li>Resolution per image: 1024 x 768 </li></ul><ul><li>FOV: ~100 o x ~80 o </li></ul><ul><li>Acquisition speed: 15 fps uncompressed </li></ul>
  7. 7. Acquisition platforms <ul><li>Robotic cart </li></ul><ul><li>Wearable </li></ul>
  8. 8. Calibration: Lens distortion Original (distorted) image Output (corrected, perspective) image
  9. 9. Calibration: Vignetting Before correction After correction
  10. 10. Stitching (Only 4 of 6 images shown here)
  11. 11. Problem with simple feathering
  12. 12. Multiperspective plane sweep With MPPS Without MPPS
  13. 13. Feature tracking for calibration and ego-motion Points tracked in all frames in all 6 cameras Edges tracked in all frames in all 6 cameras
  14. 14. Stabilization Before motion stabilization After motion stabilization Align frame- to-frame and distribute  heading
  15. 15. Augmented surround video A movie clip from the original surround data. The augmented movie clip. Surround augmented with video
  16. 16. Viewing Software <ul><li>D3D + DirectShow + XML scene graphs </li></ul><ul><li>Model: 6 texture mapped planes </li></ul><ul><ul><li>Rendered using D3D hardware </li></ul></ul><ul><li>Rendering software: </li></ul><ul><ul><li>Selective decompression </li></ul></ul><ul><ul><li>Joystick controller </li></ul></ul><ul><ul><li>Overlays support </li></ul></ul>
  17. 17. Demo A Virtual Home Tour
  18. 18. <ul><li>Head-mounted outdoor scene acquisition </li></ul><ul><li>Map control </li></ul><ul><li>Localized audio for a richer experience </li></ul><ul><li>Complex path navigation (greater freedom of motion) </li></ul>A second scenario: Bellevue Botanical Gardens
  19. 19. Data acquisition <ul><li>Walk through the garden with the Ladybug camera mounted on helmet </li></ul><ul><li>Acquisition done in about 15 minutes </li></ul><ul><li>44 Gb worth of video and audio data </li></ul>
  20. 20. Map control <ul><li>The garden map </li></ul><ul><li>Drawing the </li></ul><ul><li>acquisition path </li></ul><ul><li>Mapping video frames </li></ul><ul><li>positions on map </li></ul><ul><li>Placing sound sources </li></ul><ul><li>(background and dynamic) </li></ul><ul><li>Output is a descriptive </li></ul><ul><li>XML file </li></ul>
  21. 21. Audio processing and streaming <ul><li>All audio sources are: </li></ul><ul><li>Attenuated (based on </li></ul><ul><li>distance from current </li></ul><ul><li>position), </li></ul><ul><li>Mixed together, </li></ul><ul><li>Streamed to sound card </li></ul><ul><li>(DirectX Audio). </li></ul><ul><li>Two types of sound: </li></ul><ul><li>background and dynamic </li></ul><ul><li>Some audio sources </li></ul><ul><li>Current ego-position </li></ul>
  22. 22. Bifurcation handling <ul><li>Hypothesize, align, choose best pair as bifurcation </li></ul><ul><li>Choice of path depends on current heading and bifurcation point orientation </li></ul>Current heading
  23. 23. Bifurcation handling <ul><li>Hypothesize, align, choose best pair as bifurcation </li></ul><ul><li>Choice of path depends on current heading and bifurcation point orientation </li></ul>Current heading
  24. 24. Demo 2 A walk in Bellevue Botanical Garden
  25. 25. <ul><li>High-Dynamic range (HDR) imaging </li></ul><ul><li>Complex indoor path navigation </li></ul>3 rd scenario: High-End Home (architectural walkthrough)
  26. 26. <ul><li>Combine multiple exposures to obtain a high dynamic range (HDR) image. </li></ul>High dynamic range imaging Overexposed Combined Underexposed
  27. 27. High dynamic range imaging All frames Underexposed frames Overexposed frames <ul><li>HDR imaging is challenging for videos. </li></ul><ul><li>Image registration is necessary. </li></ul>
  28. 28. Demo 3 A High-End Home
  29. 29. Sample walkthroughs Garden Tour High-End Home Tour Simple Home Tour
  30. 30. Features and statistics <ul><li>Rapid scene acquisition </li></ul><ul><li>Map control </li></ul><ul><li>Localized audio for a richer experience </li></ul><ul><li>Complex path navigation </li></ul><ul><li>High data rates: </li></ul><ul><ul><li>44 Gb worth of video and audio data in 15 minutes </li></ul></ul>
  31. 31. Current/future work <ul><li>Scene Modeling </li></ul><ul><ul><li>Motion estimation and map generation </li></ul></ul><ul><ul><li>More interactive 3D elements </li></ul></ul><ul><li>Compression & selective decoding </li></ul><ul><ul><li>Tiled video representation </li></ul></ul><ul><li>Rendering </li></ul><ul><ul><li>Better use of GPU </li></ul></ul><ul><li>Interactive DVD deployment </li></ul>
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×