1. 1
3DTV
Past, Present and Future
Touradj Ebrahimi
Professor II at NTNU/Q2S
Professor at EPFL
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
2. 2
Outline
• Introduction
– Motivation
– A bit of History
• 3D perception
– Human visual system
– Depth perception
• 3D processing
– Creation
– Representation
– Coding
– Visualization
• 3D quality
• Future of 3DTV
• Conclusion
Indicates that the anaglyph glasses should be used to
view the content of the slides
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
4. 4
Motivation
• Evolution of visual content towards greater realism
– Black and white
– Color
– High resolution
– 3DTV
• 3 major factors for success
– Technology: Ability to capture, process and show 3D
– Content: Availability of interesting 3D content
– Quality: Attractive to consumers
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
5. 5
History of 3D
• 1840: Invention of stereoscopy and
stereoscope by C. Wheatstone
• 1890: First patent for 3D motion pictures
using stereoscope
• 1915: First 3D footage in cinema using
anaglyph glasses
• 1922: Invention of „Teleview“ a shutter
based technique
• 1936: First demonstration of polarization
based projection
• 1952: Golden era of 3D movies due to
invention of television
• 1961: Single film solution „Space-Vision
3D“ using polarization
• 1980: IMAX 70mm projectors for non-
fiction short films
• 2003: First full length 3D feature film for
IMAX screens by J. Cameron
• 2004: Animation „Polar Express“ makes
14 times more revenue in 3D than 2D
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
6. 6
Example of most recent 3D successes
• Movies
– Beowulf (2007)
– Avatar (2009)
• Music
– U2 3D (2008)
– In Concert 3D (2009)
• Documentary
– Biodiversity (2009)
• Sports
– NBA All Star Game (2009)
– Six Nations Cup (2010)
– FIFA World Cup (2010)
• Games
– 19 PS3 titles in 2010
• 3D Blu-ray
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
7. 7
3D Perception
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
8. 8
3D perception
• Human visual system uses several depth cues
to build a mental model of a perceived 3D scene
– Oculomotor cues
– Monocular cues
– Binocular cues
• Understanding the human visual perception of
Text
3D is an important aspect to develop high
quality 3D systems and services
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
9. 9
Visual perception
• Eye
– Optical part of the visual
perception
– Amount of light is controlled by
iris
– Image is brought into focus by
lens
– Retinal cells capture information
• Visual pathways
– Carry visual information from the
retina to the brain
• Visual cortex
– Primary visual cortex responds to
low level visual information such
as frequencies, color and
direction
– Dorsal and vental streams are
dealing with motions and objects
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
10. 10
Eye movements
• Achieved by muscles attached to the
eye itself and the lens
• Accommodation
– Change to optical parameter of the
lens to bring an object into focus
• Vergence
– Movement of the eyes to opposite
directions to gaze at an object
• Version
– Movement of the eyes to same
direction to pursue movements
• Saccadic
– Rapid movement of the eyes to scan
the scene
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
11. 11
Stereopsis
• Human eyes are separated
horizontally by approx. 6.3 cm
• Existence of different retinal
images leads to binocular
disparity
• Binocular disparity provides
cues about the relative depth
of objects and their
environment
• Very effective for large
disparities at close distances
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
13. 13
Oculomotor depth cues
• Accommodation
– Change of the focal length of the lens
– Quite a weak depth cue and only effective for distances < 2 m
• Convergence
– Rotation of the eyes towards each other for closer objects
– Quite a weak depth cue and only effective for distances < 10 m
• Myosis
– Size of the pupil determines both amount of light and depth of
field (DOF)
– Very weak depth cue for short distances
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
14. 14
Monocular depth cues
• Mainly experiential and learned over
time
• Shadow
• Illumination and shadow
• Relative sizes differences
• Motion parallax
• Aerial perspective
• Linear perspective
• Interposition
• Texture gradient
• Intensity gradient
• ...
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
16. 16
Binocular depth cues
• Two eyes observe scene from two slightly different angles
• Most important depth cue for medium viewing distances
• Basic idea behind any stereoscopic display technology
• Around 5% of the population have difficulties with binocular
depth
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
17. 17
Motion parallax
• Motion cues are created
when the viewer moves
his eyes or head
• Relative object motion
around a fixation point
serves as depth cue
• Very important depth cue
for a large range of scene
depths
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
23. 23
Camera distance
• Single most important
parameter in stereoscopy
• Regulates strength of 3D
effect and object size
• Optimal camera distance
depends on many factors
– Near and far distance
– Focal length
– Maximum disparity
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
24. 24
Camera distance
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
25. 25
Camera convergence
• Second most important
parameter in stereoscopy
• Parallel camera setup
– All objects are in front of
screen
– Depth position can be set
later
• Converged camera setup
– Objects partially behind the
screen
– Depth position fixed to
objects where optical axes
converge
– May lead to keystone effect
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
26. 26
Camera convergence
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
27. 27
2D to 3D conversion
• Basic idea is to convert
available 2D content to 3D
• Exploit monocular depth cues
to generate 3D content
• Automatic conversion is
difficult and may cause
artifacts
• Semiautomatic conversion
based on computer vision
technologies for best tradeoff
between efficiency and quality
– Structure from Motion
– Monocular Depth Estimation
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
28. 28
3D post-processing
• Geometric alignment
– Horizontal and vertical alignment of cameras may not be perfect
– Align images to avoid vertical disparities and adjust depth position
• Color adjustment
– White balance and exposure of individual cameras may differ
– Leads to color and brightness variations between stereo images
– Correction using histogram matching techniques
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
29. 29
3D scene representation
• Stereoscopic
– Left-Right
– Above-Below
– Interlaced
• Image + Depth
– Synchronized 2D and depth
image or video
• Multiview
– Several synchronized 2D video
streams
– Also possible to include depth for
each view
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
30. 30
3D compression standards
• Simulcast
– Possible with any video
coding standard
• Stereoscopic
– MPEG-4/AVC: Stereo SEI
message and different L/R
packings
• Image+depth
– MPEG-C Part A: Auxiliary data
representation for depth maps
• Multiview video coding (MVC)
– H.264/MVC: Extension of H.
264/AVC to multiview video
• Multiview+depth
– 3D Video Coding (3DVC)
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
31. 31
Asymmetric coding
• Individual views are coded independently
• Each view is coded with a different quality
• According to binocular suppression theory the perceived
quality is dominated by the higher quality view
• Any video coding standard can be used
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
33. 33
3D content display (technologies)
• Stereograms
• Passive anaglyph
• Active shutter
• Passive polarization
– Linear
– Circular
• Autostereoscopic
– Single view
– Multi view
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
34. 34
Autostereoscopic displays
• Requires no glasses
• Sends the left and right views
directly to the eyes
• Lenticular lens
– Tiny cylindrical plastic lenses
attached to the front of the screen
– Does not support 2D mode
• Parallax barrier
– Fine grating of liquid crystals
placed in front of the screen
– Supports 2D/3D mode by
switching parallax barrier off/on
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
35. 35
Sweet spot
• Correct 3D perception
depends largely on the
viewers position and the
used display technology
• Wrong distance to the
screen leads to crosstalk
between the left and the
right images
• Wrong horizontal position
leads to inverted images
and causes headache
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
36. 36
Single vs. multi view displays
• Single view displays provide the same view for each position
• Multi view displays provide different views depending on position
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
37. 37
3D Quality
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
38. 38
Quality of Experience is the Key
• "As much as it pains me to say this - I love 3-D, I really do -
these films are unpleasant to watch.“
• "In any event, the 3D experience doesn’t at all feel natural,
much less immersive.”
• "I don't get motion sickness from a car or sea yet 3D movies
tent to give me headache.”
• "The 3D trend is annoying…what’s so bad about a really
beautiful 2D composition? Even the best 3D still darkens the
picture and muddies the color ever so slightly“
• "The 3D glasses offered to me ruined my movie-watching
experience to say the least. They were uncomfortable to wear
as they only had a one-size-fits-all.”
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
39. 39
Good vs. Bad 3D Quality
• Avatar (2009) • Clash of the Titans (2010)
– Combination of 3D shooting, – Shot completely in 2D and
motion capture and and CGI converted to 3D in 8 weeks
– Budget of 237 million USD – Budget of 125 million USD
– "It was an absolute marvel and I – "The film redefines 3-D but in the
am left in awe after seeing it.“ wrong way.”
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
40. 40
3D Quality
• Improved quality of experience (QoE) is an important aspect for
the success of any new technology such as 3DTV
• 2D image and video quality assessment is relatively mature
• Emerging 3D video quality assessment needs to consider
additional aspects such as depth perception, immersion and
presence
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
41. 41
3D Quality
Nintendo Warns
Parents Not To Let
Children Under 6
Play 3DS
December 29th, 2010
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
42. 42
3D processing chain
• Capture
– Special care needs to be taken when positioning cameras or
selecting rendering parameters
– Unnatural correspondences are source of many 3D artifacts
• Conversion
– Various representations for 3D video exist (e.g. multiview and
image+depth)
– Converting between them may introduce different artifacts
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
43. 43
3D processing chain
• Coding
– Coding schemes exploit spatial, temporal and inter view
redundancies
– May lead to various 2D and 3D artifacts
• Transmission
– Digital wireless transmission are subject to packet losses
– Resilience and error concealment algorithms may
introduce artifacts
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
44. 44
3D processing chain
• Visualization
– Various approaches exist for 3D scene visualization with
different degree of scene approximation
– Each 3D display technology may cause specific artifacts
and the amount of artifacts is largely scene dependent
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
45. 45
2D artifacts
• Not specific to 3D video but also present in 2D video
• Mainly degrade the 2D quality but may also affect depth
perception
• Examples include blocking, ringing, blurring, color bleeding,
jerkiness
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
46. 46
3D artifacts
• Distortions that lead to a perceptual difference between the
real 3D structure of a scene and its representation
• May have different effects on the depth perception from a
feeling of unnaturalness to visual discomfort and headache
• Includes keystone distortion, puppet theater effect, crosstalk,
cardboard effect, shear distortion, disocclusions
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
47. 47
A key challenge in 3D
• Subjective and objective quality evaluation protocols
and metrics that measure:
– Artifacts of 3D
processing chain
– Naturalness
– Sense of presence
– Fatigue
– Eye strain
– Headache
– …
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
48. 48
Holographic 3DTV
• 3DTV with and without glasses are
milestones towards holographic TV
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
49. 49
Challenges in Holographic 3DTV
• Efficient and practical capture and display
• Compression, transmission and storage
• Conversion of legacy 2D and even multi-
view content
• Applications and viable business models
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
51. 51
Conclusion
• 3DTV is among one of the emerging multimedia trends
• Success will largely depend on improved quality of experience
• Interesting content and good 3D quality provides novel experience
to users (e.g. Avatar)
• Bad 3D quality may limit the acceptance of 3D technologies (e.g.
Clash of the Titans)
• Subjective and objective quality assessment of 2D image and video
quite mature
• Methodologies and metrics need to be adopted for 3D considering
the special characteristics of 3D perception
• Huge opportunities in research, technology, art, and business
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
52. 52
Thanks for your attention
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
53. 53
Acknowledgements
• Many of the slides presented in this
seminar were collected/produced
diligently by Dr. Lutz Goldmann.
• Great efforts by Martin Rerabek for
preparation of appropriate 3D anaglyph
glasses are appreciated and
acknowledged.
Digitale telenett 2011
January 4th, 2010, Trondheim, Norway
BBC News - Quality warnings issued over 3DTV\n""'We don't think it's right to confuse consumers this early on with second-rate conversion technology,' Fabrice Estornel, product manager at Panasonic TV, told website Home Cinema Choice (HCC)."\nhttp://www.bbc.co.uk/news/10446419\n\nMichael Bay And James Cameron Skeptical Of 3D Conversions: &#x201C;The Jury Is Out&#x201D; &#x2013; Deadline.com\n"In any event, the 3D experience doesn&#x2019;t at all feel natural, much less &#x201C;immersive.&#x201D;"\nhttp://www.deadline.com/2010/03/michael-bay-james-cameron-skeptical-of-3d-conversions-the-jury-is-out/\n\nMichael Bay And James Cameron Skeptical Of 3D Conversions: &#x201C;The Jury Is Out&#x201D; &#x2013; Deadline.com\n"I was able to see 2D footage that was converted to 3D by one of these companies mentioned in the article. Long story short, it looked awful and cheesy. While yes, the images did seem to pop off of the screen, it still looked flat and lacked the depth that AVATAR has."\nhttp://www.deadline.com/2010/03/michael-bay-james-cameron-skeptical-of-3d-conversions-the-jury-is-out/\n\n3-D movies like Monsters vs. Aliens hurt your eyes. They always have, and they always will. - By Daniel Engber - Slate Magazine\n"(A Katzenbergian mantra: "Making your customers sick is not a recipe for success.")"\nhttp://www.slate.com/id/2215265/\n\n3-D movies like Monsters vs. Aliens hurt your eyes. They always have, and they always will. - By Daniel Engber - Slate Magazine\n"As much as it pains me to say this&#x2014;I love 3-D, I really do&#x2014;these films are unpleasant to watch."\nhttp://www.slate.com/id/2215265/\n\n3-D movies like Monsters vs. Aliens hurt your eyes. They always have, and they always will. - By Daniel Engber - Slate Magazine\n"Did the RealD projection give me a headache or was it the movie's lamebrained script?"\nhttp://www.slate.com/id/2215265/pagenum/2\n\n3D Sickness: 3D movies make me sick...literally!\n"I don't get motion sickness from a car or sea yet 3D movies tent to give me headache."\nhttp://www.squidoo.com/3dsickness\n\nAvatar 3D glasses of sub-standard quality? &#xAB; Fenil and Bollywood\n"&#x201C;The 3D glasses offered to me ruined my movie-watching experience to say the least. They were uncomfortable to wear as they only had a one-size-fits-all."\nhttp://fenilandbollywood.wordpress.com/2009/12/24/avatar-3d-glasses-of-bad-sub-standard-quality/\n
Why is the 3-D so bad in Clash of the Titans? - By Daniel Engber - Slate Magazine\n"The film "redefines 3-D but in the wrong way," reports the St. Petersburg Times"\nhttp://www.slate.com/id/2249527\n\nWhy is the 3-D so bad in Clash of the Titans? - By Daniel Engber - Slate Magazine\n"The film "redefines 3-D but in the wrong way," reports the St. Petersburg Times"\nhttp://www.slate.com/id/2249527\n\nToday3D Forum\n"Earlier this week James Cameron warned against such 2D to 3D conversions, saying specifically of the Clash of the Titans conversion that &#x201C;If you want to make a movie in 3-D, make the movie in 3-D.&#x201D;"\nhttp://www.today3d.com/2010/03/clash-of-titans-2d-to-3d-conversion-is.html\n\nMichael Bay And James Cameron Skeptical Of 3D Conversions: &#x201C;The Jury Is Out&#x201D; &#x2013; Deadline.com\n"The 3D trend is annoying&#x2026;what&#x2019;s so bad about a really beautiful 2D composition? Even the best 3D still darkens the picture and muddies the color ever so slightly"\nhttp://www.deadline.com/2010/03/michael-bay-james-cameron-skeptical-of-3d-conversions-the-jury-is-out/\n