3DTV - Past, Present and Future

1

3DTV
Past, Present and Future
Touradj Ebrahimi
Professor II at NTNU/Q2S
Professor at EPFL

Digitale telenett 2011
January 4th, 2010, Trondheim, Norway

2
Outline
• Introduction
– Motivation
– A bit of History
• 3D perception
– Human visual system
– Depth perception
• 3D processing
– Creation
– Representation
– Coding
– Visualization
• 3D quality
• Future of 3DTV
• Conclusion

Indicates that the anaglyph glasses should be used to
view the content of the slides


3

INTRODUCTION


4
Motivation

• Evolution of visual content towards greater realism
– Black and white
– Color
– High resolution
– 3DTV
• 3 major factors for success
– Technology: Ability to capture, process and show 3D
– Content: Availability of interesting 3D content
– Quality: Attractive to consumers


5
History of 3D
• 1840: Invention of stereoscopy and
stereoscope by C. Wheatstone
• 1890: First patent for 3D motion pictures
using stereoscope
• 1915: First 3D footage in cinema using
anaglyph glasses
• 1922: Invention of „Teleview“ a shutter
based technique
• 1936: First demonstration of polarization
based projection
• 1952: Golden era of 3D movies due to
invention of television
• 1961: Single film solution „Space-Vision
3D“ using polarization
• 1980: IMAX 70mm projectors for non-
fiction short films
• 2003: First full length 3D feature film for
IMAX screens by J. Cameron
• 2004: Animation „Polar Express“ makes
14 times more revenue in 3D than 2D

6
Example of most recent 3D successes
• Movies
– Beowulf (2007)
– Avatar (2009)
• Music
– U2 3D (2008)
– In Concert 3D (2009)
• Documentary
– Biodiversity (2009)
• Sports
– NBA All Star Game (2009)
– Six Nations Cup (2010)
– FIFA World Cup (2010)
• Games
– 19 PS3 titles in 2010
• 3D Blu-ray


7

3D Perception


8
3D perception

• Human visual system uses several depth cues
to build a mental model of a perceived 3D scene
– Oculomotor cues
– Monocular cues
– Binocular cues
• Understanding the human visual perception of
Text

3D is an important aspect to develop high
quality 3D systems and services


9
Visual perception
• Eye
– Optical part of the visual
perception
– Amount of light is controlled by
iris
– Image is brought into focus by
lens
– Retinal cells capture information
• Visual pathways
– Carry visual information from the
retina to the brain
• Visual cortex
– Primary visual cortex responds to
low level visual information such
as frequencies, color and
direction
– Dorsal and vental streams are
dealing with motions and objects


10
Eye movements
• Achieved by muscles attached to the
eye itself and the lens
• Accommodation
– Change to optical parameter of the
lens to bring an object into focus
• Vergence
– Movement of the eyes to opposite
directions to gaze at an object
• Version
– Movement of the eyes to same
direction to pursue movements
• Saccadic
– Rapid movement of the eyes to scan
the scene


11
Stereopsis
• Human eyes are separated
horizontally by approx. 6.3 cm
• Existence of different retinal
images leads to binocular
disparity
• Binocular disparity provides
cues about the relative depth
of objects and their
environment
• Very effective for large
disparities at close distances


12
Depth cues


13
Oculomotor depth cues

• Accommodation
– Change of the focal length of the lens
– Quite a weak depth cue and only effective for distances < 2 m
• Convergence
– Rotation of the eyes towards each other for closer objects
– Quite a weak depth cue and only effective for distances < 10 m
• Myosis
– Size of the pupil determines both amount of light and depth of
field (DOF)
– Very weak depth cue for short distances


14
Monocular depth cues
• Mainly experiential and learned over
time
• Shadow
• Illumination and shadow
• Relative sizes differences
• Motion parallax
• Aerial perspective
• Linear perspective
• Interposition
• Texture gradient
• Intensity gradient
• ...


15
Monocular depth cues


16
Binocular depth cues
• Two eyes observe scene from two slightly different angles
• Most important depth cue for medium viewing distances
• Basic idea behind any stereoscopic display technology
• Around 5% of the population have difficulties with binocular
depth


17
Motion parallax

• Motion cues are created
when the viewer moves
his eyes or head
• Relative object motion
around a fixation point
serves as depth cue
• Very important depth cue
for a large range of scene
depths


18
Depth perception layers


19

3D Processing


20
3D processing chain


21
3D creation

• Computer generated imagery (CGI)
• Stereoscopic cameras
• Video + depth sensor
• 2D to 3D conversion


22
Stereoscopic camera

• Matching cameras
• Matching photography
– White balance
– Sensitivity
– Shutter speed
– Aperture
• Matching optics
– Focal length
• Matching geometry
– Distance
– Angle
• Synchronization

23
Camera distance

• Single most important
parameter in stereoscopy
• Regulates strength of 3D
effect and object size
• Optimal camera distance
depends on many factors
– Near and far distance
– Focal length
– Maximum disparity


24
Camera distance


25
Camera convergence
• Second most important
parameter in stereoscopy
• Parallel camera setup
– All objects are in front of
screen
– Depth position can be set
later
• Converged camera setup
– Objects partially behind the
screen
– Depth position fixed to
objects where optical axes
converge
– May lead to keystone effect


26
Camera convergence


27
2D to 3D conversion

• Basic idea is to convert
available 2D content to 3D
• Exploit monocular depth cues
to generate 3D content
• Automatic conversion is
difficult and may cause
artifacts
• Semiautomatic conversion
based on computer vision
technologies for best tradeoff
between efficiency and quality
– Structure from Motion
– Monocular Depth Estimation


28
3D post-processing
• Geometric alignment
– Horizontal and vertical alignment of cameras may not be perfect
– Align images to avoid vertical disparities and adjust depth position
• Color adjustment
– White balance and exposure of individual cameras may differ
– Leads to color and brightness variations between stereo images
– Correction using histogram matching techniques


29
3D scene representation

• Stereoscopic
– Left-Right
– Above-Below
– Interlaced
• Image + Depth
– Synchronized 2D and depth
image or video
• Multiview
– Several synchronized 2D video
streams
– Also possible to include depth for
each view


30
3D compression standards
• Simulcast
– Possible with any video
coding standard
• Stereoscopic
– MPEG-4/AVC: Stereo SEI
message and different L/R
packings
• Image+depth
– MPEG-C Part A: Auxiliary data
representation for depth maps
• Multiview video coding (MVC)
– H.264/MVC: Extension of H.
264/AVC to multiview video
• Multiview+depth
– 3D Video Coding (3DVC)


31
Asymmetric coding
• Individual views are coded independently
• Each view is coded with a different quality
• According to binocular suppression theory the perceived
quality is dominated by the higher quality view
• Any video coding standard can be used


32
3D content displays (devices)

• Head Mounted Displays (HMD)
• Monitors
– Anaglyph
– Polarization
– Shutter
– Autostereoscopic
• Projectors
– Anaglyph
– Polarization
– Shutter


33
3D content display (technologies)

• Stereograms
• Passive anaglyph
• Active shutter
• Passive polarization
– Linear
– Circular
• Autostereoscopic
– Single view
– Multi view


34
Autostereoscopic displays
• Requires no glasses
• Sends the left and right views
directly to the eyes
• Lenticular lens
– Tiny cylindrical plastic lenses
attached to the front of the screen
– Does not support 2D mode
• Parallax barrier
– Fine grating of liquid crystals
placed in front of the screen
– Supports 2D/3D mode by
switching parallax barrier off/on


35
Sweet spot

• Correct 3D perception
depends largely on the
viewers position and the
used display technology
• Wrong distance to the
screen leads to crosstalk
between the left and the
right images
• Wrong horizontal position
leads to inverted images
and causes headache


36
Single vs. multi view displays

• Single view displays provide the same view for each position
• Multi view displays provide different views depending on position


37

3D Quality


38
Quality of Experience is the Key
• "As much as it pains me to say this - I love 3-D, I really do -
these films are unpleasant to watch.“

• "In any event, the 3D experience doesn’t at all feel natural,
much less immersive.”

• "I don't get motion sickness from a car or sea yet 3D movies
tent to give me headache.”

• "The 3D trend is annoying…what’s so bad about a really
beautiful 2D composition? Even the best 3D still darkens the
picture and muddies the color ever so slightly“

• "The 3D glasses offered to me ruined my movie-watching
experience to say the least. They were uncomfortable to wear
as they only had a one-size-fits-all.”

39
Good vs. Bad 3D Quality
• Avatar (2009) • Clash of the Titans (2010)
– Combination of 3D shooting, – Shot completely in 2D and
motion capture and and CGI converted to 3D in 8 weeks
– Budget of 237 million USD – Budget of 125 million USD
– "It was an absolute marvel and I – "The film redefines 3-D but in the
am left in awe after seeing it.“ wrong way.”


40
3D Quality
• Improved quality of experience (QoE) is an important aspect for
the success of any new technology such as 3DTV
• 2D image and video quality assessment is relatively mature
• Emerging 3D video quality assessment needs to consider
additional aspects such as depth perception, immersion and
presence


41
3D Quality

Nintendo Warns
Parents Not To Let
Children Under 6
Play 3DS
December 29th, 2010


42
3D processing chain

• Capture
– Special care needs to be taken when positioning cameras or
selecting rendering parameters
– Unnatural correspondences are source of many 3D artifacts
• Conversion
– Various representations for 3D video exist (e.g. multiview and
image+depth)
– Converting between them may introduce different artifacts

43
3D processing chain

• Coding
– Coding schemes exploit spatial, temporal and inter view
redundancies
– May lead to various 2D and 3D artifacts
• Transmission
– Digital wireless transmission are subject to packet losses
– Resilience and error concealment algorithms may
introduce artifacts

44
3D processing chain

• Visualization
– Various approaches exist for 3D scene visualization with
different degree of scene approximation
– Each 3D display technology may cause specific artifacts
and the amount of artifacts is largely scene dependent


45
2D artifacts
• Not specific to 3D video but also present in 2D video
• Mainly degrade the 2D quality but may also affect depth
perception
• Examples include blocking, ringing, blurring, color bleeding,
jerkiness


46
3D artifacts
• Distortions that lead to a perceptual difference between the
real 3D structure of a scene and its representation
• May have different effects on the depth perception from a
feeling of unnaturalness to visual discomfort and headache
• Includes keystone distortion, puppet theater effect, crosstalk,
cardboard effect, shear distortion, disocclusions


47
A key challenge in 3D

• Subjective and objective quality evaluation protocols
and metrics that measure:
– Artifacts of 3D
processing chain
– Naturalness
– Sense of presence
– Fatigue
– Eye strain
– Headache
– …


48
Holographic 3DTV

• 3DTV with and without glasses are
milestones towards holographic TV


49
Challenges in Holographic 3DTV

• Efficient and practical capture and display
• Compression, transmission and storage
• Conversion of legacy 2D and even multi-
view content
• Applications and viable business models


50

CONCLUSION


51
Conclusion
• 3DTV is among one of the emerging multimedia trends
• Success will largely depend on improved quality of experience
• Interesting content and good 3D quality provides novel experience
to users (e.g. Avatar)
• Bad 3D quality may limit the acceptance of 3D technologies (e.g.
Clash of the Titans)
• Subjective and objective quality assessment of 2D image and video
quite mature
• Methodologies and metrics need to be adopted for 3D considering
the special characteristics of 3D perception
• Huge opportunities in research, technology, art, and business


52
Thanks for your attention


53
Acknowledgements

• Many of the slides presented in this
seminar were collected/produced
diligently by Dr. Lutz Goldmann.
• Great efforts by Martin Rerabek for
preparation of appropriate 3D anaglyph
glasses are appreciated and
acknowledged.


3DTV - Past, Present and Future

More Related Content

What's hot

Viewers also liked

Similar to 3DTV - Past, Present and Future

More from Touradj Ebrahimi

Recently uploaded

3DTV - Past, Present and Future

Editor's Notes