2D to 3D conversion at CRCA visual perception approach                           Carlos Vazquez and Wa James Tam          ...
2D to S3D video conversion       Why?                                      Where/When?                           How?     ...
Real-time vs. off-line conversion   On-line 2D-to-3D conversion     Fully automatic     Artefacts are not removed     ...
2D to S3D conversion: General framework   Strategy:      Define depth structure and      render new virtual view   Input...
How to get the depth        Manually assignment of                                                             • Human int...
Automatic generation of depth information   Two approaches are used:     Extract depth information from pictorial      d...
Visual perception: An active process   Content dependency:           Aoccdrnig to rscheearch at an Elingsh uinervtisy,  ...
Visual perception: Depth filling June 4, 2012   Advanced Video Systems -- 3D-TV Research   9
Surrogate depth mapsMotivation   Shadows offer relative depth    inside objects   Small image features (rain drops,    s...
CRC-i3D: Real-time 2D to 3D video conversion   Surrogate Depth Maps:       Single Cue – using the Chroma component of th...
Off-line conversion: depth-based approach          Depth generation                     Depth map editing                 ...
Depth-based approach: Main characteristics Intermediate depth master: Can be used to render the  stereoscopic content for...
CRC-DMEG: Depth editing toolJune 4, 2012   Advanced Video Systems -- 3D-TV Research   14
CRC-DMEG: Depth editing tool               Simple histogram equalization of green channelJune 4, 2012     Advanced Video S...
June 4, 2012   Advanced Video Systems -- 3D-TV Research   16
2D to 3D conversion at CRC: A visual perception approach.
Upcoming SlideShare
Loading in …5
×

2D to 3D conversion at CRC: A visual perception approach.

926 views

Published on

Invited panel presentation at the First Toronto International Stereoscopic 3D Conference

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
926
On SlideShare
0
From Embeds
0
Number of Embeds
17
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

2D to 3D conversion at CRC: A visual perception approach.

  1. 1. 2D to 3D conversion at CRCA visual perception approach Carlos Vazquez and Wa James Tam 3D-TV Research Advanced Video Systems
  2. 2. 2D to S3D video conversion Why? Where/When? How? •Many S3D cinema screens available •Post-production: For cinema and •Fully manual •Introduction of 3D-enabled TV sets TV production •Semi-automatic •Large amount of 2D content •Broadcasting: For live content and •Automatic off-line available legacy material •Automatic in real-time •Limited content in original •TV set, STB: Legacy content from stereoscopic format viewer library or 2D program •Still difficult to capture stereoscopic contentJune 4, 2012 Advanced Video Systems -- 3D-TV Research 2
  3. 3. Real-time vs. off-line conversion On-line 2D-to-3D conversion  Fully automatic  Artefacts are not removed  Speed of conversion is critical  Non-veridical depth information frequently used Off-line 2D-to-3D conversion  Automatic, semi-automatic or manual  Automatically generated content can be corrected  Normally human-assisted (post production)  High quality and high cost June 4, 2012 Advanced Video Systems -- 3D-TV Research 3
  4. 4. 2D to S3D conversion: General framework Strategy: Define depth structure and render new virtual view Input: 2D video sequence Output: Stereoscopic/multi-view video sequence June 4, 2012 Advanced Video Systems -- 3D-TV Research 4
  5. 5. How to get the depth Manually assignment of • Human interpretation of the scene depth by operator analysis of translates into depth information the scene Extracting depth from • Depth from static images monocular cues in images • Depth from motion related cues Assigning of depth from • Apply a know depth model automatic analysis of the • Use surrogates for the depth scene informationJune 4, 2012 Advanced Video Systems -- 3D-TV Research 5
  6. 6. Automatic generation of depth information Two approaches are used:  Extract depth information from pictorial depth cues in images  Assign depth based on general assumptions about scenes Pictorial depth cues:  Size, texture density, occlusion, atmospheric haze, shadows  Perspective  Focus/defocus General assumptions  Scene depth models Single image with several pictorial  Surrogate depth based on colour and depth cues shadings June 4, 2012 Advanced Video Systems -- 3D-TV Research 6
  7. 7. Visual perception: An active process Content dependency:  Aoccdrnig to rscheearch at an Elingsh uinervtisy, Past experience help in interpretation of visual content it deosnt mttaer in waht oredr the ltteers in a Auto-correction: wrod are, olny taht the frist and lsat ltteres are  Fill-in missing information or minimize conflicting information at the rghit pcleas. The rset can be a toatl mses and you can sitll raed it wouthit a porbelm. Tihs is bcuseae we do not raed ervey lteter by ilstef, but the wrod as a wlohe. http://kybele.psych.cornell.edu/~edelman/ June 4, 2012 Advanced Video Systems -- 3D-TV Research 8
  8. 8. Visual perception: Depth filling June 4, 2012 Advanced Video Systems -- 3D-TV Research 9
  9. 9. Surrogate depth mapsMotivation Shadows offer relative depth inside objects Small image features (rain drops, snow flakes, dust) have depth Good separation between objects in the scene Green and blue go back (Trees and sky) Red goes to the front (Skin tones) June 4, 2012 Advanced Video Systems -- 3D-TV Research 10
  10. 10. CRC-i3D: Real-time 2D to 3D video conversion Surrogate Depth Maps:  Single Cue – using the Chroma component of the images.  Minimal adjustments to original depth map Rendering:  Real-Time and no buffering required.  Avoids disocclusion by smoothing depth map.  Manual control of depth to ensure comfort according to viewing conditions. Advantages:  Very low processing requirements.  No storage requirements.  This all translates into less processing, heat and power requirements. June 4, 2012 Advanced Video Systems -- 3D-TV Research 11
  11. 11. Off-line conversion: depth-based approach Depth generation Depth map editing New image generation Generate Select Edit depth Render Fill-in2D depth regions values image holes S3D Generate initial depth: Automatic operation Edit depth map: Manual or semi-automatic  Select objects: Semi-automatic operation  Change depth of objects: Semi-automatic Render new view: Automatic operation Fill-in holes: Automatic operation with human supervision. June 4, 2012 Advanced Video Systems -- 3D-TV Research 12
  12. 12. Depth-based approach: Main characteristics Intermediate depth master: Can be used to render the stereoscopic content for different targets Depth relations are easy to evaluate Depth is continuous: Slow variations in depth are easily represented Depth information is represented as an image: Easy to store and transmit June 4, 2012 Advanced Video Systems -- 3D-TV Research 13
  13. 13. CRC-DMEG: Depth editing toolJune 4, 2012 Advanced Video Systems -- 3D-TV Research 14
  14. 14. CRC-DMEG: Depth editing tool Simple histogram equalization of green channelJune 4, 2012 Advanced Video Systems -- 3D-TV Research 15
  15. 15. June 4, 2012 Advanced Video Systems -- 3D-TV Research 16

×