3DTV CONTENTCAPTURE, ENCODINGAND TRANSMISSIONBUILDING THE TRANSPORTINFRASTRUCTURE FORCOMMERCIAL SERVICESDaniel MinoliA JOHN WILEY & SONS, INC., PUBLICATION
Copyright 2010 by John Wiley & Sons, Inc. All rights reservedPublished by John Wiley & Sons, Inc., Hoboken, New JerseyPublished simultaneously in CanadaNo part of this publication may be reproduced, stored in a retrieval system, or transmitted in anyform or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise,except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, withouteither the prior written permission of the Publisher, or authorization through payment of theappropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers,MA 01923, (978) 750-8400, fax (978) 750-4470, or on the web at www.copyright.com. Requeststo the Publisher for permission should be addressed to the Permissions Department, John Wiley &Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online athttp://www.wiley.com/go/permission.Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their bestefforts in preparing this book, they make no representations or warranties with respect to theaccuracy or completeness of the contents of this book and speciﬁcally disclaim any impliedwarranties of merchantability or ﬁtness for a particular purpose. No warranty may be created orextended by sales representatives or written sales materials. The advice and strategies containedherein may not be suitable for your situation. You should consult with a professional whereappropriate. Neither the publisher nor author shall be liable for any loss of proﬁt or any othercommercial damages, including but not limited to special, incidental, consequential, or otherdamages.For general information on our other products and services or for technical support, please contactour Customer Care Department within the United States at (800) 762-2974, outside the UnitedStates at (317) 572-3993 or fax (317) 572-4002.Wiley also publishes its books in a variety of electronic formats. Some content that appears in printmay not be available in electronic formats. For more information about Wiley products, visit ourweb site at www.wiley.com.Library of Congress Cataloging-in-Publication Data:Minoli, Daniel, 1952- 3DTV content capture, encoding and transmission : building thetransport infrastructure for commercial services / Daniel Minoli. p. cm. ISBN 978-0-470-64973-2 (cloth) 1. Stereoscopic television. I. Title. TK6643.M56 2010 621.388– dc22 2010008432Printed in Singapore10 9 8 7 6 5 4 3 2 1
CONTENTSPreface xiAbout the Author xiii1 Introduction 1 1.1 Overview 1 1.2 Background 6 1.2.1 Adoption of 3DTV in the Marketplace 6 1.2.2 Opportunities and Challenges for 3DTV 16 1.3 Course of Investigation 19 References 24 Appendix A1: Some Recent Industry Events Related to 3DTV 262 3DV and 3DTV Principles 29 2.1 Human Visual System 29 2.1.1 Depth/Binocular Cues 33 2.1.2 Accommodation 34 2.1.3 Parallax 34 2.2 3DV/3DTV Stereoscopic Principles 35 2.3 Autostereographic Approaches 42 References 453 3DTV/3DV Encoding Approaches 47 3.1 3D Mastering Methods 51 3.1.1 Frame Mastering for Conventional Stereo Video (CSV) 51 3.1.2 Compression for Conventional Stereo Video (CSV) 55 3.2 More Advanced Methods 59 3.2.1 Video Plus Depth (V + D) 60 vii
viii CONTENTS 3.2.2 Multi-View Video Plus Depth (MV + D) 63 3.2.3 Layered Depth Video (LDV) 65 3.3 Short-term Approach for Signal Representation and Compression 69 3.4 Displays 69 References 69 Appendix A3: Color Encoding 73 Appendix B3: Additional Details on Video Encoding Standards 74 B3.1 Multiple-View Video Coding (MVC) 75 B3.2 Scalable Video Coding (SVC) 78 B3.3 Conclusion 794 3DTV/3DV Transmission Approaches and Satellite Delivery 81 4.1 Overview of Basic Transport Approaches 81 4.2 DVB 90 4.3 DVB-H 95 References 99 Appendix A4: Brief Overview of MPEG Multiplexing and DVB Support 101 A4.1 Packetized Elementary Stream (PES) Packets and Transport Stream (TS) Unit(s) 101 A4.2 DVB (Digital Video Broadcasting)-Based Transport in Packet Networks 104 A4.3 MPEG-4 and/or Other Data Support 1055 3DTV/3DV IPTV Transmission Approaches 113 5.1 IPTV Concepts 114 5.1.1 Multicast Operation 115 5.1.2 Backbone 120 5.1.3 Access 125 5.2 IPv6 Concepts 132 References 135 Appendix A5: IPv6 Basics 138 A5.1 IPv6 Overview 138 A5.2 Advocacy for IPv6 Deployment—Example 157
CONTENTS ix6 3DTV Standardization and Related Activities 163 6.1 Moving Picture Experts Group (MPEG) 165 6.1.1 Overview 165 6.1.2 Completed Work 166 6.1.3 New Initiatives 178 6.2 MPEG Industry Forum (MPEGIF) 182 6.3 Society of Motion Picture and Television Engineers (SMPTE) 3D Home Entertainment Task Force 183 6.4 Rapporteur Group On 3DTV of ITU-R Study Group 6 184 6.5 TM-3D-SM Group of Digital Video Broadcast (DVB) 187 6.6 Consumer Electronics Association (CEA) 188 6.7 HDMI Licensing, LLC 189 6.8 Blu-ray Disc Association (BDA) 189 6.9 Other Advocacy Entities 190 6.9.1 3D@Home Consortium 190 6.9.2 3D Consortium (3DC) 190 6.9.3 European Information Society Technologies (IST) Project “Advanced Three-Dimensional Television System Technologies” (ATTEST) 191 6.9.4 3D4YOU 192 6.9.5 3DPHONE 196 References 198Glossary 201Index 225
PREFACE3 Dimensions TV (3DTV) became commercially available in the United Statesin 2010 and service in other countries was expected to follow soon thereafter.3DTV is a subset of a larger discipline known as 3D Video (3DV). There arenow many routine vendor announcements related to 3DTV/3DV, and there arealso conferences wholly dedicated to the topic. To highlight the commercial interest in this topic, note that ESPN announced inJanuary 2010 that it planned to launch what would be the world’s ﬁrst 3D sportsnetwork with the 2010 World Cup soccer tournament in June 2010, followed byan estimated 85 live sports events during its ﬁrst year of operation. DirecTV wasplanning to become the ﬁrst company to offer satellite-based 3D as announcedat the 2010 International Consumer Electronics Show. Numerous manufacturersshowed 3D displays at recent consumer electronics trade shows. Several standardsbodies and industry consortia are now working to support commercialization ofthe service. An increasing inventory of content is now also becoming availablein 3D. This text offers an overview of the content capture, encoding, and transmis-sion technologies that have emerged of late in support of 3DTV/3DV. It focuseson building the transport infrastructure for commercial services. The book isaimed at interested planners, researchers, and engineers who wish to get anoverview of the topic. Stakeholders involved with the rollout of the infrastructureinclude video engineers, equipment manufacturers, standardization committees,broadcasters, satellite operators, Internet Service Providers, terrestrial telecom-munications carriers, storage companies, content-development entities, designengineers, planners, college professors and students, and venture capitalists. While there is a lot of academic interest in various aspects of the overall sys-tem, service providers and the consumers ultimately tend to take a system-levelview. While service providers do to an extent take a constructionist bottom-upview to deploy the technological building blocks (such as encoders, encapsula-tors, IRDs, and set-top boxes), 3DTV stakeholders need to consider the overallarchitectural system-level view of what it will take to deploy an infrastructure thatis able to reliably and cost-effectively deliver a commercial-grade quality bundleof multiple 3DTV content channels to paying customers with high expectations. This text, therefore, takes such system-level view. Fundamental visual con-cepts supporting stereographic perception of 3DTV are reviewed. 3DTV technol-ogy and digital video principles are discussed. Elements of an end-to-end 3DTVsystem are covered. Compression and transmission technologies are assessed xi
xii PREFACEfor satellite and terrestrial (or hybrid) IPTV-based architecture. Standardizationactivities, critical to any sort of broad deployment, are identiﬁed. The focus of this text is how to actually deploy the technology. There is asigniﬁcant quantity of published material in the form of papers, reports, and tech-nical speciﬁcations. This published material forms the basis for this synthesis, butthe information is presented here in a self-contained, organized, tutorial fashion.
ABOUT THE AUTHORMr. Minoli has done extensive work in video engineering, design, and implemen-tation over the years. The results presented in this book are based on work donewhile at Bellcore/Telcordia, Stevens Institute of Technology, AT&T, and otherengineering ﬁrms, starting in the early 1990s and continuing to the present. Someof his video work has been documented in the books he has authored such as 3DTelevision (3DTV) Technology, Systems, and Deployment - Rolling out theInfrastructure for Next-Generation Entertainment (Francis and Taylor, 2010);IP Multicast with Applications to IPTV and Mobile DVB-H (Wiley/IEEEPress, 2008); Video Dialtone Technology: Digital Video over ADSL, HFC,FTTC, and ATM (McGraw-Hill, 1995); Distributed Multimedia ThroughBroadband Communication Services (co-authored) (Artech House, 1994); Dig-ital Video (4 chapters) in The Telecommunications Handbook, K. Terplan &P. Morreale Editors, IEEE Press, 2000; and, Distance Learning: Technologyand Applications (Artech House, 1996). Mr. Minoli has many years of technical hands-on and managerial experience inplanning, designing, deploying, and operating IP/IPv6-, telecom-, wireless-, andvideo networks, and data center systems and subsystems for global best-in-classcarriers and ﬁnancial companies. He has worked in ﬁnancial ﬁrms such as AIG,Prudential Securities, Capital One Financial, and service provider ﬁrms such asNetwork Analysis Corporation, Bell Telephone Laboratories, ITT, Bell Commu-nications Research (now Telcordia), AT&T, Leading Edge Networks Inc., andSES Engineering, where he is Director of Terrestrial Systems Engineering (SESis the largest satellite services company in the world). At SES, in addition to otherduties, Mr. Minoli has been responsible for the development and deployment ofIPTV systems, terrestrial and mobile IP-based networking services, and otherglobal networks. He also played a founding role in the launching of two com-panies through the high-tech incubator Leading Edge Networks Inc., which heran in the early 2000s: Global Wireless Services, a provider of secure broadbandhotspot mobile Internet and hotspot VoIP services; and, InfoPort Communica-tions Group, an optical and Gigabit Ethernet metropolitan carrier supporting datacenter/SAN/channel extension and cloud computing network access services. Forseveral years, he has been Session, Tutorial, and now overall Technical ProgramChair for the IEEE ENTNET (Enterprise Networking) conference; ENTNETfocuses on enterprise networking requirements for large ﬁnancial ﬁrms and othercorporate institutions. xiii
xiv ABOUT THE AUTHOR Mr. Minoli has also written columns for ComputerWorld, NetworkWorld,and Network Computing (1985–2006). He has taught at New York University(Information Technology Institute), Rutgers University, and Stevens Institute ofTechnology (1984–2006). Also, he was a Technology Analyst At-Large, for Gart-ner/DataPro (1985–2001); based on extensive hand-on work at ﬁnancial ﬁrmsand carriers, he tracked technologies and wrote CTO/CIO-level technical scansin the area of telephony and data systems, including topics on security, disas-ter recovery, network management, LANs, WANs (ATM and MPLS), wireless(LAN and public hotspot), VoIP, network design/economics, carrier networks(such as metro Ethernet and CWDM/DWDM), and e-commerce. Over the yearshe has advised Venture Capitals for investments of $150M in a dozen high-techcompanies. He has acted as Expert Witness in a (won) $11B lawsuit regarding aVoIP-based wireless air-to-ground communication system, and has been involvedas a technical expert in a number of patent infringement lawsuits (including twolawsuits on digital imaging).
CHAPTER 1Introduction1.1 OVERVIEWRecently, there has been a lot of interest on the part of technology suppliers,broadcasters, and content providers to bring 3 Dimension Video (3DV) to theconsumer. The year 2010 has been called the ﬁrst year of 3D Television (3DTV)by some industry players. 3DTV is the delivery of 3DV on a TV screen, typi-cally in the consumer’s home. The initial step in this commercialization endeavorwas to make 3D content available on Blu-ray Discs (BDs), for example with therelease of Titanic, Terminator, and Avatar. However, well beyond that stand-alonehome arrangement there has been a concerted effort to develop end-to-end sys-tems to bring 3DTV services to the consumer, supported by regular commercialprogramming that is delivered and made available on a routine scheduled basis.Broadcasters such as, but not limited to, ESPN, DIRECTV, Discovery Commu-nications, BSkyB, and British Channel 4 were planning to start 3D programmingin 2010. LG, Samsung, Panasonic, Sony, JVC, Vizio, Sharp, and Mitsubishi,among others, were actively marketing high quality TV display products at presstime, with some such as Samsung and Mitsubishi already shipping 3D-ready ﬂat-panel TVs as far back as 2008. Front Projection 3D systems for medium-sizedaudiences (5–25 people), for example for the “prosumer,” have been availablefor longer; of course, movie theater systems have been around for years. Thegoal of the 3DTV industry is to replicate to the degree possible the experienceachievable in a 3D movie theater, but in the home setting. A commercial 3DTV system is comprised of the following functionalelements: capture of 3D content, speciﬁcally moving scenes; encoding(representation) of content; content compression; content transport over satellite,cable, Internet Protocol Television (IPTV), or over-the-air channels1 ; and contentdisplay. Figure 1.1 depicts a logical, functional view of an end-to-end 3DTV1 Internet-based downloading and/or streaming is also a possibility for some applications or subsetof users.3DTV Content Capture, Encoding and Transmission: Building the TransportInfrastructure for Commercial Services, by Daniel MinoliCopyright 2010 John Wiley & Sons, Inc. 1
2 INTRODUCTION Scene replica 3D scene Capture Representation Compression Coding Transmission Signal Display conversion Figure 1.1 Basic 3DTV system—logical view.system. Figure 1.2 depicts graphically a system architecture that may see earlycommercial introduction—this system is known as stereoscopic ConventionalStereo Video (CSV) or Stereoscopic 3D (S3D). Figures 1.3 and 1.4 showexamples of 3D camera arrangements, while Fig. 1.5 illustrates a typical 3Ddisplay (this one using active glasses, also called eyewear). Finally, Fig. 1.6depicts what we call a pictorialization of 3D TV screens, as may be includedin vendor brochures. This text offers an overview of the content capture, encoding, and trans-mission subelements, speciﬁcally the technologies, standards, and infrastruc-ture required to support commercial real-time 3DTV/3DV services. It reviewsthe required standards and technologies that have emerged of late—or are justemerging—in support of such new services, with a focus on encoding and thebuild-out of the transport infrastructure. Stakeholders involved with the rolloutof this infrastructure include consumer and system equipment manufacturers,broadcasters, satellite operators, terrestrial telecommunications carriers, InternetService Providers (ISPs), storage companies, content-development entities, andstandardization committees. There is growing interest on the part of stakeholders to introduce 3DTV ser-vices, basically as a way to generate new revenues. There was major emphasis
OVERVIEW 3 5 - Transmission 2 - Mastering of two 4 - Digital (HD) frames encoder Satellite IPTV cable over the air 3 - 3D processor combines Internet two frames into single 1 - Dual camera capture (HDTV) frames system 3DTV 3DTV 6 - Decoding glasses display 7 - Displaying, viewing Figure 1.2 Basic 3DTV system—conventional stereo video. Figure 1.3 Illustrative 2-camera rig for 3D capture. Source: www.inition.co.uks.on 3DTV from manufacturers at various consumer shows taking place in therecent past. One in four consumers surveyed by the Consumer Electronics Asso-ciation (CEA) in a press time study indicated that they plan to buy a 3D TV setwithin the next three years . The research ﬁrm DisplaySearch has forecastedthat the 3D display market will grow to $22 billion by 2018 (this represents an
4 INTRODUCTIONFigure 1.4 Illustrative single 3D camcorder with dual lenses. Source: Panasonic CES2010 Press Kit. Figure 1.5 Illustrative 3D home display. Source: Panasonic CES 2010 Press Kit.
OVERVIEW 5 Figure 1.6 Pictorialization of 3D home display. Source: LG CES 2010 Press Kit.annual compound growth rate of about 50%2 ). When it comes to entertainment,especially for a compelling type of entertainment that 3D has the opportunity ofbeing, there may well be a reasonably high take rate, especially if the price pointis right for the equipment and for the service. Classical questions that are (and/or should be) asked by stakeholders includethe following: • Which competing 3D encoding and transmission technologies should an operator adopt? • What technological advancements are expected in 3D, say by 2012 or 2015? • Where do the greatest market opportunities exist in the 3D market? These and similar questions are addressed in this text.2 The company originally forecast a $1B industry for 2010, but recently lowered that forecast byabout 50%.
6 INTRODUCTION1.2 BACKGROUNDThis section provides an encapsulated assessment of the 3DTV industry landscapeto give the reader a sense of what some of the issues are. It provides a press timesnapshot of industry drivers that support the assertion just made: that there is alot of activity in this arena at this time.1.2.1 Adoption of 3DTV in the MarketplaceIt should be noted that 3D ﬁlm and 3DTV trials have a long history, as shownin Fig. 1.7 (based partially on Ref. 2). However, the technology has ﬁnally Stereoscopic 3D pictures—1838 (Wheatsone) Popular by 1844 in US and Europe 2D Photography—1839 2D Movies—1867 (Lincon) 3D stereoscopic cinema—early 1900 2D TV—1920 (Belin and Baird) Stereoscopic 3D TV—1920 (Baird) Stereoscopic 3D cinema popular by 1950s Stereoscopic 3DTV broadcast—1953 First commercial 3DTV broadcast—1980s Vendor buzz—2010 called “The Year of 3DTV” by some Timeline 1838 1867 1920 1950s 1980s 2010 (not on a linear scale) Analog broadcast of 3DTV (limited to single movie or specific event): first experimental broadcast in 1953 first commercial broadcast in 1980 first experimental broadcast in Europe in 1982 Digital broadcast of 3DTV (typically stereoscopic 3DTV) Japan, 1998; Korea, 2002 3D cinema becomes popular routine stereoscopic broadcast with anaglypth methods, especially for sports events 3DTV over IP networks: video streaming experiments and demonstrations assessment of streaming protocols (RTP over UDP, DCCP) research into Multiview video streaming Development of plethora of displays (ongoing) Development of standards (ongoing) Figure 1.7 History of 3D in ﬁlm and television.
BACKGROUND 7progressed enough at this juncture, for example with the deployment of digitaltelevision (DTV) and High Deﬁnition Television (HDTV), that regular commer-cial services will ﬁnally be introduced at this juncture. We start by noting that there are two general commercial-grade displayapproaches for 3DTV: (i) stereoscopic TV, which requires special glasses towatch 3D movies, and (ii) autostereoscopic TV, which displays 3D images insuch a manner that the user can enjoy the viewing experience without specialaccessories.3 Short-term commercial 3DTV deployment, and the focus of this book, ison stereoscopic 3D imaging and movie technology. The stereoscopic approachfollows the cinematic model, is simpler to implement, can be deployed morequickly (including the use of relatively simpler displays), can produce the bestresults in the short term, and may be cheaper in the immediate future. However,the limitations are the requisite use of accessories (glasses), somewhat limitedpositions of view, and physiological and/or optical limitations including possibleeye strain. In summary, (i) glasses may be cumbersome and expensive (especiallyfor a large family) and (ii) without the glasses, the 3D content is unusable. Autostereoscopic 3DTV eliminates the use of any special accessories: itimplies that the perception of 3D is in some manner automatic, and doesnot require devices—either ﬁlter-based glasses or shutter-based glasses.Autostereoscopic displays use additional optical elements aligned on the surfaceof the screen, to ensure that the observer sees different images with each eye.From a home screen hardware perspective the autostereoscopic approach is morechallenging, including the need to develop relatively more complex displays;also, more complex acquisition/coding algorithms may be needed to makeoptimal use of the technology. It follows that this approach is more complex toimplement, will require longer to be deployed, and may be more expensive inthe immediate future. However, this approach can produce the best results in thelong term, including accessories-free viewing, multi-view operation allowingboth movement and different perspective at different viewing positions, andbetter physiological and/or optical response to 3D. Table 1.1 depicts a larger set of possible 3DTV (display) systems than whatwe identiﬁed above. The expectation is that 3DTV based on stereoscopy willexperience earlier deployment compared with other technological alternatives.Hence, this text focuses principally on stereoscopy. Holography and integralimaging are relatively newer technologies in the 3DTV context compared tostereoscopy; holographic and/or integral imaging 3DTV may be feasible late inthe decade. There are a number of techniques to allow each eye to view theseparate pictures, as summarized in Table 1.2 (based partially on Ref. 3.) All ofthese techniques work in some manner, but all have some shortcomings. To highlight the commercial interest in 3DTV at press time, note that ESPNannounced in January 2010 that it planned to launch what would be the world’s3 Autostereoscopic technology may also (in particular) be appropriate for mobile 3D phones and thereare several initiatives to explore these applications and this 3D phone-display technology.
8 INTRODUCTIONTABLE 1.1 Various 3D Display Approaches and TechnologiesStereoscopic 3D A system where two photographs (or video streams) taken (S3D) from slightly different angles that appear three-dimensional when viewed together; this technology is likely to see the earliest implementation using specially designed equipment displays that support polarizationAutostereoscopic 3D displays that do not require glasses to see the stereoscopic image (using lenticular or parallax barrier technology). Whether stereoscopic or autostereoscopic, a 3D display (screen) needs to generate parallax that, in turn, creates a stereoscopic sense. Will ﬁnd use in cell phone 3D displays in the near futureMulti-viewpoint A system that provides a sensation of depth and motion 3D system parallax based on the position and motion of the viewer; at the display side new images are synthesized, based on the actual position of the viewerIntegral imaging A technique that provides autostereoscopic images with full (holoscopic parallax by using an array of microlenses to generate a imaging) collection of 2D elemental images; in the reconstruction/display subsystem, the set of elemental images is displayed in front of a far-end microlens arrayHolography A technique for generating an image (hologram) that conveys a sense of depth, but is not a stereogram in the usual sense of providing ﬁxed binocular parallax information; holograms appear to ﬂoat in space and they change perspective as one walks left or right; no special viewers or glasses are necessary (note, however, that holograms are monochromatic)Volumetric Systems that use geometrical principles of holography, in systems conjunction with other volumetric display methods. Volumetric displays form the image by projection within a volume of space without the use of a laser light reference, but have limited resolution. They are primarily targeted, at least at press time, at the Industrial, Scientiﬁc, and Medical (ISM) communityﬁrst 3D sports network with the 2010 World Cup soccer tournament in June 2010,followed by an estimated 85 live sports events during its ﬁrst year of operation.DIRECTV announced that they will start 3D programming in 2010. DIRECTV’snew HD 3D channels will deliver movies, sports, and entertainment contentfrom some of the world’s most renowned 3D producers. DIRECTV is currentlyworking with AEG/AEG Digital Media, CBS, Fox Sports/FSN, Golden BoyPromotions, HDNet, MTV, NBC Universal, and Turner Broadcasting System,Inc., to develop additional 3D programming that will debut in 2010–2011. Atlaunch, the new DIRECTV HD 3D programming platform will offer a 24/7 3Dpay per view channel focused on movies, documentaries, and other programming;
TABLE 1.2 Current Techniques to Allow Each Eye to View Distinct Pictures Streams With Orthogonal Uses orthogonal (different) polarization planes for each, with matching viewer glasses for each of the appliances polarization left and right eye pictures. Light from each picture is ﬁltered such that only one plane for the light (glasses) wave is available. This is easy to arrange in a cinema, but more difﬁcult to arrange in a television display. Test television systems have been developed on the basis of this method, either using two projection devices projecting onto the same surface, or two displays orthogonally placed so that a combined image can be seen using a semisilvered mirror. In either case, these devices are “nonstandard” television receivers. Of the systems with glasses, this is considered the “best” Colorimetric One approach is to use different colorimetric arrangements (anaglyth) for each of the two pictures, arrangements coupled with glasses that ﬁlter appropriately. A second, is a relatively new notch ﬁlter color separation (or anaglyth) technique that can be used in projection systems (advanced by Dolby)—described later in the chapter Time Sometimes also called “interlaced stereo”, content shown with consecutive left and right signals and multiplexing shuttered glasses. This technology is applicable to 3DTV. This technique is still used for movie of the display theaters today, such as the IMAX, and sometimes used in conjunction with polarization plane separation. In a Cathode Ray Tube (CRT) environment, a major shortcoming of the interlaced stereo was image ﬂicker, since each eye would see only 25 or 30 images per second, rather than 50 or 60. To overcome this, the display rate could be doubled to 100 or 120 Hz to allow ﬂicker-free reception “Virtual reality” Technique using immersion headgear/glasses often used for video games headset Without Lenticular This technique arranges for each eye’s view to be directed toward separate picture elements by lenses. appliances This is done by fronting the screen with a ribbed (lenticular) surface Barrier This technique arranges for the screen to be fronted with barrier slots that perform a similar function. In this system, two views (left and right), or more than two (multi-camera 3D) can be used. However, since each of the picture elements (stripes or points) have to be laid next to each other, the number of views impacts on the resolution available. There is a trade-off between resolution and ease of viewing. Arrangements can be made with this type of system to track head or eye movements, and thus change the barrier position, giving the viewer more freedom of head movement9
10 INTRODUCTIONa 24/7 3D DIRECTV on Demand channel; and a free 3D sampler demo channelfeaturing event programming such as sports, music, and other content. Comcasthas announced that its VOD (Video-On-Demand) service is offering a numberof movies in anaglyph 3D (as well as HD) form. Comcast customers can pickup 3D anaglyph glasses at Comcast payment centers and malls “while supplieslast” (anaglyph is a basic and inexpensive method of 3D transmission that relieson inexpensive colored glasses, but its drawback is the relatively low quality.)Verizon’s FiOS was expected to support 3DTV programming by Late 2010. SkyTV in the United Kingdom was planning to start broadcasting programs in 3D inthe fall of 2010 on a dedicated channel that will be available to anyone who hasthe Sky HD package; there are currently 1.6 million customers who have a SkyHD set-top box. Sky TV has not announced what programs will be broadcastin 3D, but it is expected to broadcast live the main Sunday afternoon soccergame from the Premiership in 3D from the 2011 season, along with some artsdocumentaries and performances of ballet . Sky TV has already invested ininstalling special twin-lens 3D cameras at stadiums. (Appendix A1 includes alisting of events during the year, prior to the publication of this text to furtherdocument the activity in this arena.) 3DTV television displays could be purchased in the United States and UnitedKingdom as of the spring of 2010 for $1000–5000 initially, depending ontechnology and approach. Liquid Crystal Display (LCD) systems with activeglasses tend to generally cost less. LG released its 3D model, a 47-in. LCDscreen, expected to cost about $3000; with this system, viewers will need towear polarized dark glasses to experience broadcasts in 3D. Samsung and Sonyalso announced they were bringing their own versions to market by the summerof 2010, along with 3D Blu-ray players, allowing consumers to enjoy 3D moviessuch as Avatar and Up, in their own homes . Samsung and Sony’s modelsuse LED (Light-Emitting Diode) screens which are considered to give a crisperpicture and are, therefore, expected to retail for about $5000 or possibly more.While LG is adopting the use of inexpensive polarizing dark glasses, Sonyand Samsung are using active shutter technology. This requires users to buyexpensive dark glasses, which usually cost more than $50 and are heavier thanthe $2–3 plastic polarizing ones. Active shutter glasses alternately darken overone eye, and then the other, in synchronization with the refresh rate of the screenusing shutters built into the glasses (using infrared or Bluetooth connections).Panasonic Corporation has developed a full HD 3D home theater systemconsisting of a plasma full HD 3D TVs, 3D Blu-ray player, and active shutter3D glasses. The 3D display was originally available in 50-in., 54-in., 58-in.and 65-in. class sizes. High-end systems are also being introduced; for examplePanasonic announced a 152-in. 4K × 2K (4096 × 2160 pixels)-deﬁnition fullHD 3D plasma display. The display features a new Plasma Display Panel(PDP) that uses self-illuminating technology. Self-illuminating plasma panelsoffer excellent response to moving images with full motion picture resolution,making them suitable for rapid 3D image display (its illuminating speed is aboutone-fourth the speed of conventional full HD panels). Each display approachhas advantages and disadvantages as shown in Table 1.3.
TABLE 1.3 Summary of Possible, Commercially Available TV Screen/System Choices for 3D 3D Display System Advantages Disadvantages 1 Projection-based Big-screen 3D effect similar to cinematic experience Needs a silver screen to retain polarization of light FPTV (polarized Excellent-to-good light intensity Alignment of two projectors should be such that they display) with Choice of projectors/cost are stacked on top of each other passive glasses Inexpensive, lightweight passive 3D glasses Not totally d´ cor-friendly e 2 Projection-based Option of using newer single DLP projectors that More expensive glasses FPTV (unpolarized support 120 Hz refresh rate (active-based system) Need battery-powered LCD shutter glasses display) with active No polarization-protecting screen needed glasses 3 Projection-based Integrated unit—easier to add to room d´ cor e Some light intensity loss at the display level RPTV (polarized To present stereoscopic content, two images are Not of the “ﬂat-panel-TV type”; cannot be hung on display) with projected superimposed onto the same screen walls passive glasses through different polarizing ﬁlters (either linear or circular polarizing ﬁlters can be used) Viewer wears low-cost eyeglasses that also contain a corresponding pair of different polarizing ﬁlters 4 LCD 3DTV Simple-to-use system, not requiring projection setup Some possible loss of resolution (polarized display) To present stereoscopic content, two images are Viewer wears low-cost eyeglasses which also contain with passive projected superimposed onto the same screen a pair of different polarizing ﬁlters glasses through interlacing techniques Some light intensity loss at the display level Relatively expensive ($3000–5000 in 2010) 5 3D plasma/LCD TV Simple-to-use system not requiring projection setup Delivers two images to the same screen pixels, but (unpolarized Flat-screen TV type, elegant d´ cor e alternates them such that two different images are display) with active alternating on the screen glasses Active shutter glasses can be expensive, particularly for a larger viewing group (continued overleaf)11
12 TABLE 1.3 (Continued ) 3D Display System Advantages Disadvantages Requires TV sets to be able to accept and display images at 120/240 Hz Glasses need power Some light intensity loss at the viewer (glasses level) Some loss of resolution Size limited to 60–80 in. at this time but larger systems being brought to market LCDs are relatively cheaper than other alternatives: a 42-in. set based on LCD and shutter glasses was selling for about US$1000 and a 50-in. set was selling for more than US$2000 (compare that with a 42-in. HD LCD TV which costs about US$600–700) LEDs and or plasma systems can be costly 6 Autostereoscopic No glasses needed Very few suppliers in 2010 screen (lenticular Further out in time in terms of development and or barrier) deployment Some key manufacturers have exited the business (for now) Content production is more complex Displays have a “sweet spot” that requires viewers to be within this viewing zone a FPTV, Front Projection Television; DLP, Digital Light Processing; RPTV, Rear Projection Television.
BACKGROUND 13 Figure 1.8 3D Blu-ray disc logo. It is to be expected that 3DTV for home use is likely to ﬁrst see penetration viastored media delivery. For content source, proponents make the case that BD “isthe ideal platform” for the initial penetration of 3D technology in the mainstreammarket because of the high quality of pictures and sound it offers ﬁlm producers.Many products are being introduced by manufacturers: for example at the 2010Consumer Electronics Show (CES) International Trade Show, vendors introducedeight home theater product bundles (one with 3D capability), 14 new players(four with 3D capability), three portable players, and a number of software titles.In 2010 the Blu-ray Disc Association (BDA) launched a new 3D Blu-ray logoto help consumers quickly discern 3D-capable Blu-ray players from 2D-onlyversions (Fig. 1.8) . The BDA makes note of the strong adoption rate of the Blu-ray format. In2009, the number of Blu-ray households increased by more than 75% over 2008totals. After four years in the market, total Blu-ray playback devices (includingboth set-top players and PlayStation3 consoles) numbered 17.6 million units,and 16.2 million US homes had one or more Blu-ray playback devices. Bycomparison, DVD playback devices (set-tops and PlayStation2 consoles) reached14.1 million units after four years, with 13.7 million US households having oneor more playback devices. The strong performance of the BD format is due toa number of factors, including the rapid rate at which prices declined due tocompetitive pressures and the economy; the rapid adoption pace of HDTV sets,which has generated a US DTV household penetration rate exceeding 50%; and,a superior picture and sound experience compared to standard deﬁnition and evenother HD sources. Another factor in the successful adoption pace has been thewillingness of movie studios to discount popular BD titles . Blu-ray softwareunit sales in 2009 reached 48 million, compared with 22.5 million in 2008, upby 113.4%. A number of movie classics were available at press time throughleading retailers at sale prices as low as $10. The BDA also announced (at the end of 2009) the ﬁnalization and releaseof the Blu-ray 3D speciﬁcation. These BD speciﬁcations for 3D allow for fullHD 1080p resolution to each eye. The speciﬁcations are display agnostic, mean-ing they apply equally to plasma, LCD, projector, and other display formatsregardless of the 3D systems those devices use to present 3D to viewers. The
14 INTRODUCTIONspeciﬁcations also allow the PlayStation3 gaming console to play back 3D con-tent. The speciﬁcations that represent the work of the leading Hollywood studiosand consumer electronic and computer manufacturers, will enable the home enter-tainment industry to bring stereoscopic 3D experience into consumers’ livingrooms on BD, but will require consumers to acquire new players, HDTVs, andshutter glasses. The speciﬁcations allow studios (but do not require them) topackage 3D Blu-ray titles with 2D versions of the same content on the samedisc. The speciﬁcations also support playback of 2D discs in forthcoming 3Dplayers and can enable 2D playback of Blu-ray 3D discs on an installed baseof BD. The Blu-ray 3D speciﬁcation encodes 3D video using the Multi-ViewVideo Coding (MVC) codec, an extension to the ITU-T H.264 Advanced VideoCoding (AVC) codec currently supported by all BD players. MPEG-4 (MovingPicture Experts Group 4)-MVC compresses both left and right eye views with atypical 50% overhead compared to equivalent 2D content, according to BDA andcan provide full 1080p resolution backward compatibility with current 2D BDplayers . The broadcast commercial delivery of 3DTV on a large scale—whetherover satellite/Direct-To-Home (DTH), over the air, over cable systems, or viaIPTV—may take some number of years because of the relatively large-scaleinfrastructure that has to be put in place by the service providers and the limitedavailability of 3D-ready TV sets in the home (implying a small subscriber,and so small revenue base). A handful of providers were active at press time,as described earlier, but general deployment by multiple providers serving ageographic market will come at a future time. Delivery of downloadable 3DTVﬁles over the Internet may occur at any point in the immediate future, but theprovision of a broadcast-quality service over the Internet is not likely for theforeseeable future. At the transport level, 3DTV will require more bandwidth of regular pro-gramming, perhaps even twice the bandwidth in some implementations (e.g.,simulcasting—the transmission of two fully independent channels4 ); some newerschemes such as “video + depth” may require only 25% more bandwidth com-pared to 2D, but these schemes are not the leading candidate technologies foractual deployment in the next 2–3 years. Other interleaving approaches use thesame bandwidth of a channel now in use, but at a compromise in resolution.Therefore, in principle, if HDTV programming is broadcast at high quality, say,12–15 Mbps using MPEG-4 encoding, 3DTV using the simplest methods of twoindependent streams will require 24–30 Mbps.5 This data rate does not ﬁt astandard over-the-air digital TV (DTV) channel of 19.2 Mbps, and will also be4 Inthe 3DTV context, the term “simulcasting” has been used with two meanings: one use is asimplied above—the coding and transmission of two channels (which is unlikely to occur in reality);the second use is in the traditional sense of transmitting, say a HDTV signal and also a 3DTV signalby some other means or on some other channel/system.5 Some HDTV content may be delivered at lower rates by same operators, say 8 Mbps; this rate,however, may not be adequate for sporting HDTV channels, and may be marginal for 3D TV at1080p/60 Hz per eye.
BACKGROUND 15a challenge for non-Fiber-To-The-Home (non-FTTH) broadband Internet con-nections. However, one expects to see the emergence of bandwidth reductiontechniques, as alluded to above. On the other hand, DTH satellite providers,terrestrial ﬁberoptic providers, and some cable TV ﬁrms should have adequatebandwidth to support the service. For example, the use of the Digital VideoBroadcast Satellite Second Generation (DVB-S2) allows a transponder to carry75 Mbps of content with modulation using an 8-point constellation and twice thatmuch with a 16-point constellation. The trade-off would be, however (if we usethe raw HD bandwidth just described as a point of reference), that a DVB-S2transponder that would otherwise carry 25 channels of standard deﬁnition videoor 6–8 channels of HD video would now only carry 2–3 3DTV channels. To bepragmatic about this issue, most 3DTV providers are not contemplating deliveringfull resolution as just described and/or the transmission of two fully independentchannels (simulcasting), but some compromise; for example, lowering the per eyedata rate such that a 3DTV program ﬁts into a commercial-grade HDTV channel(say 8–10 Mbps), using time interleaving or spatial compression—again, this isdoable but comes with the degradation of ultimate resolution quality. There are a number of alternative transport architectures for 3DTV signals,also depending on the underlying media. As noted, the service can be sup-ported by traditional broadcast structures including the DVB architecture, wireless3G/4G transmission such as DVB-H approaches, Internet Protocol (IP) in sup-port of an IPTV-based service (in which case it also makes sense to considerIPv6) and the IP architecture for internet-based delivery (both non–real timeand streaming). The speciﬁc approach used by each of these transport meth-ods will also depend on the video-capture approach. One should note that inthe United States, one has a well-developed cable infrastructure in all Tier 1 andTier 2 metropolitan and suburban areas; in Europe/Asia, this is less so, with moreDTH delivery (in the United States DTH tends to serve more exurban and ruralareas). A 3DTV rollout must take these differences into account and/or accom-modate both. In reference to possible cable TV delivery, CableLabs announcedat press time that it started to provide testing capabilities for 3D TV implemen-tation scenarios over cable; these testing capabilities cover a full range of tech-nologies including various frame-compatible, spatial multiplexing solutions fortransmission . Standards are critical to achieving interworking and are of great value to bothconsumers and service providers. The MPEG of the International Organization forStandardization/International Electrotechnical Commission (ISO/IEC) has beenworking on coding formats for 3D video (and has already completed some ofthem.) The Society of Motion Picture and Television Engineers (SMPTE) 3DHome Entertainment Task Force has been working on mastering standards. TheRapporteur Group on 3DTV of the International Telecommunications Union-Radiocommunications Sector (ITU-R) Study Group 6, and the TM-3D-SM groupof DVB were working on transport standards.
16 INTRODUCTION1.2.2 Opportunities and Challenges for 3DTVThe previous section highlighted that many of the components needed to supportan end-to-end commercial broadcast service are available or are becoming avail-able. Hence, proponents see a signiﬁcant market opportunity at this time. CEAestimates that more than 25% of sets sold in 2013 will be 3D-enabled. A handfulof representative quotes from proponents of the 3D technology are as follows: No one can escape the buzz and excitement around 3D. We’re witnessing the start of dramatic change in how we view TV—the dawn of a new dimension. And through Sky’s clear commitment to 3D broadcasting, 3D in the home is set to become a reality . . . . . . . The next killer application for the home entertainment industry—3DTV . . . [It] will drive new revenue opportunities for content creators and distributors by enabling 3D feature ﬁlms and other programming to be played on their home television and computer displays—regardless of delivery channels . . . . . . . The most buzzed about topics at CES: 3-D stereoscopic content creation . . . Several pivotal announcements [in] 2010 including 3D-TV releases from the major consumer electronics manufacturers and the launch of several dedicated 3D broadcast channels are driving the rapid increase in demand for 3-D content . . . . 3D technology is now positioned “to become a major force in future in-home enter- tainment.”. . . . 3DTV is one of the ‘hottest’ subjects today in broadcasting. The combination of the audience’s‘wow’ factor and the potential to launch completely new services, makes it an attractive subject for both consumer and professional. There have already been broadcasts of a conventional display-compatible system, and the ﬁrst HDTV channel compatible broadcasts are scheduled to start in Europe in the Spring of 2010 . . . . . . . In Europe, the EC is currently funding a large series of projects for 3DTV, including multiview, mobile 3D and 3D search . . . . Naturally, while there are proponents of the 3DTV technology, at the sametime, there are industry observers that take a more conservative view. Theseobservers make note that there are uncertainties about the availability of content,the technological readiness, and acceptance in the living room, especially giventhe requirement to use polarized or shutter glasses. A rational approach to marketpenetration is certainly in order; also, the powerful tool of statistically validmarket research can be used to truly measure user interest and willingness to pay.Some representative quotes for a more conservative view of the 3D technologyare given below: . . . In a wide range of demos, companies . . . claim . . . in January 2010 that stereo- scopic 3D is ready for the home. In fact, engineers face plenty of work hammering out
BACKGROUND 17 the standards and silicon for 3DTV products, most of which will ship for the holiday 2010 season . . . . It has proven somewhat difﬁcult to create a 3D system that does not cause ‘eye fatigue’ after a certain time. Most current-generation higher resolution systems also need spe- cial eyeglasses which can be inconvenient. Apart from eye-fatigue, systems developed so far can also have limitations such as constrained viewing positions. Multiple view- point television systems are intended to alleviate this. Stereoscopic systems also allow only limited ‘production grammar’ . . . One should not underestimate the difﬁculty, or the imagination and creativity required, to create a near ‘ideal’ 3DTV system that the public could enjoy in a relaxed way, and for a long period of time . . . . . . . The production process for 3D television requires a fundamental rethinking of the underlying technology. Scenes have to be recorded with multiple imaging devices that may be augmented with additional sensor technology to capture the three-dimensional nature of real scenes. In addition, the data format used in 3D television is a lot more complex. Rather than normal video streams, time-varying computational models of the recorded scenes are required that comprise of descriptions of the scenes’ shape, motion, and multiview appearance. The reconstruction of these models from the multiview sensor data is one of the major challenges that we face today. Finally, the captured scene descriptions have to be shown to the viewer in three-dimensions which requires completely new display technology . . . . . . . The conventional stereoscopic concept entails with two views: it relies on the basic concept of an end-to-end stereoscopic video chain, that is, on the capturing, trans- mission and display of two separate video streams, one for the left and one for the right eye. [Advocates for the autostereoscopic approach argue that] this conventional approach is not sufﬁcient for future 3DTV services. The objective of 3DTV is to bring 3D imaging to users at home. Thus, like conventional stereo production and 3D Cin- ema, 3DTV is based on the idea of providing a viewer with two individual perspective views—one for the left eye and one for the right eye. The difference in approaches, however, lies in the environment in which the 3D content is presented. While it seems to be acceptable for a user to wear special glasses in the darkened theatrical audito- rium of a 3D Digital Cinema, [many, perhaps] most people would refuse to wear such devices at home in the communicative atmosphere of their living rooms. Basically, auto-stereoscopic 3D displays are better suited for these kinds of applications . . . . The two greatest industry-wide concerns [are]: (1.) That poor quality stereoscopic TV will‘poison the water’ for everyone. Stereoscopic content that is poorly realized in grammar or technology will create a reputation of eyestrain which cannot be shaken off. This has happened before in the 30s, the 50s, and the 80s in the cinema. (2.) That fragmentation of technical standards will split and confuse the market, and prevent stereoscopic television from ever being successful . . . . . . . people may quickly tire of the novelty. I think it will be a gimmick. I suspect there will be a lot of people who say it’s sort of neat, but it’s not really comfortable . . . . The challenge for the stakeholder is to determine where the “true” situation is,whether it is at one extreme, at the other extreme, or somewhere in the middle. An
18 INTRODUCTIONabbreviated list of issues to be resolved in order to facilitate broad deploymentof 3DTV services, beyond pure technology and encoding issues, include thefollowing : • Production grammar (3D production for television still in infancy) • Compatibility of systems (also possibly different issues for pay TV and free-to-air operators) • Assessment of quality/suitability – Methodologies for the quality assessment of 3D TV systems; – Parameters that need to be measured that are speciﬁc to 3D TV; – Sensation of reality; – Ease of viewing. • Understanding what the user requirements are.In general, a technology introduction process spans three phases: • Phase 1: The technology becomes available in basic form to support a given service; • Phase 2: A full panoply of standards emerges to support widespread deploy- ment of the technology; • Phase 3: The technology becomes inexpensive enough to foster large-scale adoption by a large set of end-users. With reference to 3DTV, we ﬁnd ourselves at some early point in Phase 1.However, there are several retarding factors that could hold back short-termdeployment of the technology on a broad scale, including deployment and servicecost (overall status of the economy), standards, content, and quality. The previous assertion can be further elaborated as follows: ITU-R WP 6Cclassiﬁes 3D TV systems into two groups. The “ﬁrst generation” systems areessentially those based on “Plano-stereoscopic” display of single or multiplediscrete lateral left and right eye pictures. Recommendations for such systemsshould be possible in the near future. The “second generation” systems are thosewhich are based on object-wave recording (holography) or approximations ofobject-wave recording. Recommendations for such systems may be possible in theyears ahead. We reﬁne these observations by deﬁning the following generationsof 3DTV technology: • Generation 0: Anaglyth TV transmission; • Generation 1: 3DTV that supports plano-stereoscopic displays, which are stereoscopic (that is, require active or passive glasses); • Generation 2: 3DTV that supports plano-stereoscopic displays, which are autostereoscopic (do not require glasses);
COURSE OF INVESTIGATION 193DTV epoch 0 Generation 0:Yesterday anaglypth TV Generation 2.5: Generation 1: Generation 2:3DTV epoch 1 plano-stereoscopic displays plano-stereoscopic displays plano-stereoscopic displays autostereoscopicThis decade stereoscopic autostereoscopic multiple (N = 9) views 2010–2013 2013–2015 20163DTV epoch 2 Generation 3: Generation 4: Generation 5:Speculative integral imaging, volumetric displays, object–wave transmission> 2020? transmission, and displays transmission, and displays re-creationNote: These blocks intend to represent a deployed commercial service, not just prototypes and/or trials Figure 1.9 Three epochs of 3DTV commercial deployment. • Generation 2.5: 3DTV that supports plano-stereoscopic displays, which are autostereoscopic (do not require glasses) and support multiple (N = 9) views; • Generation 3: 3DTV that supports integral imaging, transmission, and dis- plays; • Generation 4: 3DTV that supports volumetric displays, transmission, and displays; • Generation 5: 3DTV that supports object-wave transmission.See Figs. 1.9 and 1.10 (partially based on Ref. 2). Whether and when we getbeyond Generation 2.5 in this decade remains to be seen. This text, and the cur-rent commercial industry efforts concentrate on Generation 1 services. At presstime, we ﬁnd ourselves in Phase 1 of Generation 1. The existing commercialvideo infrastructure can handle 3D video in a basic, developmental form; how-ever, providing HD 3D with motion graphics is not achievable without makingenhancements to such infrastructure. Existing infrastructures, including satelliteand/or terrestrial distribution networks for example, can handle what some havetermed “half-HD resolution” per eye, or frame formats of 1080i, 1080p24, and1080i60. Existing encoders and set-top boxes can be used as long as signalingissues are addressed and 3D content is of a consistent form. The drawback ofhalf-HD resolution is that images can be blurry, especially for sporting events andhigh-action movies . New set-top chip sets are required for higher resolution3DTV.1.3 COURSE OF INVESTIGATIONWhile there is a lot of academic interest in various aspects of the overall system,service providers and the consumers ultimately tend to take a system-level view.While service providers do, to an extent, take a constructionist bottom-up view to
20 INTRODUCTION • Generation 0: Anaglypth TV transmission • Anaglypth Analog Digital • Stereoscopy • earliest form: known for ~ 170 years • simplest, based on “perception” • two simultaneous video/image to two eyes – color-based filtering (anaglyphs) • Generation 1: 3DTV that supports – polarization-based filtering plano-stereoscopic displays, – shutter-based filtering which are stereoscopic • other forms exist (that is, require active or • pulfrich effect stereoscopy passive glasses) • Generation 2: 3DTV that supports • Autostereoscopic viewing • no special eye-wear plano-stereoscopic displays, • lenticular or barrier technologies which are autostereoscopic • sweet-spot phenomenon (do not need glasses) • Generation 2.5: 3DTV that supports • Mult-view autostereoscopy plano-stereoscopic displays, • many simultaneous horizontally which are autostereoscopic spaced views (do not need glasses) and support • usually 7-9 views; may multiple go up to ~ 50 (N = 9) views • some horizontal parallax • Integral imaging • Generation 3: 3DTV that supports integral • known since 1905 imaging, transmission, and • microlens arrays during capture displays and projection • 2D array of many elemental images • both vertical and horizontal parallax • light field renderer in the limit – replicates 3D physical light distribution: True 3D technique – “incoherent Holography” • Volumetric 3D displays • Generation 4: 3DTV that supports volumetric • sweeping 3D volume either mechanically displays, transmission, and or electronically displays • voxels – self-luminous pixels – moving projection screens • Holography • basic principle—1948, first 30-year timeline: 1995–2025 holograms—1960 • based on physics: duplication of light field: True 3D technique • recording on • Generation 5: 3DTV that supports object–wave – photographic films transmission – high resolution CCD arrays • 3D reconstruction by proper illumination of the recording • digital holographic techniques • experimental holographic motion pictures—1989 • still at basic research phase Figure 1.10 A 30-year timeline for 3DTV services (1995–2025).
COURSE OF INVESTIGATION 21 Millions... Each viewer 2D decoder IGMPing v1c1, v1c2, v1c3,... 2D display 2D SD, HD Millions... 2D Encoder and v1camera content production m content providers CSV decoder Hundreds 200–300 CSV Aggregator content 3D display channels Each viewerStereo Encoder and 3D v1camera content production v2 m content IGMPing v1c1+v2c1, v1c2+v2c2, v1c3+v2c3,... providers Hundreds Millions... V + meta Aggregator Encoder and 3D IPTV withStereo v1 V+D decoder meta m content service providercamera content production providers managed IP network 3D display Hundreds Aggregator Each viewer V+D (DVB transmission) IGMPing v1c1+d1c1, v1c2+d2c2, v1c3+d3c3,... Depth Encoder and 3D v1 m contentcamera content production d providers Millions... Aggregator Hundreds MV + D 2D display v1 m content 200–300 MV+D decoder Multi- Encoder and 3D providers content with DIBRcamera content production vn Aggregator channels Hundreds (3v, d) Each viewer IGMPing v1c1+v5c1+ v9c1, v1c2+v5c2+v9c3,... M-view 3D display (a) All channels: each viewer selecting v1c1, v1c2, v1c3,... DTH DTH 3D DTH DTH Millions... 2D decoder 3D 3D 2D display 2D SD, HD DTH Millions... 3D DTH 2D Encoder and DTH v1camera content production m content DTH providers CSV decoder Hundreds CSV Aggregator All channels: 3D displayStereo Encoder and 3D v1 each viewercamera content production v2 m content selecting v1c1+v2c1, v1c2+v2c2, v1c3+v2c3,... providers Millions... Hundreds DTH V + meta Aggregator DTH DTHStereo Encoder and 3D v1 DTH V+D decoder meta m contentcamera content production providers All channels: 3D display Hundreds Aggregator each viewer V+D selecting v1c1+d1c1, v1c2+d2c2, v1c3+d2c3,... Depth Encoder and 3D v1 m contentcamera content production d providers Aggregator Millions... Hundreds DTH DTH MV + D DTH 2D display m content DTH MV+D decoder Multi- Encoder and 3D v1 with DIBR providerscamera content production vn Aggregator All channels: Hundreds (3v, d) each viewer selecting v1c1+ v5c1+ v9c1, v1c2+v5c2+v9c3,... M-view 3D display (b) Figure 1.11 A system view of a fully developed 3DTV distribution ecosystem.
22 INTRODUCTIONdeploy the technological building blocks (such as encoders, encapsulators, IRDs[Integrated Receiver/Decoders], set-top boxes, and so on), 3DTV stakeholdersneed to consider the overall architectural system-level view of what it will taketo deploy an infrastructure that is able to reliably and cost-effectively deliver acommercial-grade quality bundle of multiple 3DTV content channels to payingcustomers with high expectations. This text, therefore, takes such a system-levelview, namely how to actually deploy the technology. Figure 1.11 depicts the3DTV distribution ecosystem that this text addresses. Fundamental visual concepts supporting stereographic perception of 3DTV arereviewed in Chapter 2. 3DTV technology and digital video compression princi-ples are discussed in Chapter 3. Elements of an end-to-end 3DTV system arecovered from a satellite deliver perspective in Chapter 4. Transmission technolo-gies are assessed for terrestrial and IPTV-based architecture in Chapter 5. Finally,Chapter 6 covers standardization activities that are critical to any sort of broaddeployment. This author recently published a companion text, “3D Television (3DTV)Technology, Systems, and Deployment—Rolling out the Infrastructure forNext-Generation Entertainment” (published by Francis and Taylor, 2010), whichaddresses the broader issues related to technologies listed in Table 1.1 andwith an emphasis on post-CSV systems. At this time, as noted earlier, theCSV approach is the mostly likely to see early deployment in commercial3DTV systems. Table 1.4 identiﬁes the approaches that have been advancedby researchers for the treatment of the video images after their immediatecapture by a stereo (or multi-view) set of cameras [21–23]. The most commonapproaches are CSV, video plus depth (V + D), multi-view video plus depth(MV + D), and layered depth video (LDV). We provide a brief discussion ofthese other systems in the chapters that follow, but we do not focus on them;the reader is referred to our companion text for a more detailed discussion ofthese systems.TABLE 1.4 Common Video Treatment ApproachesaVideo TreatmentApproach DescriptionConventional stereo CSV is the most well-known and in a way, the simplest type video (CSV) of 3D video representation. Only color pixel video data are involved and are captured by at least two cameras. The resulting video signals may undergo some processing steps like normalization, color correction, and rectiﬁcation but in contrast to other 3D video formats, no scene geometry information is involved. The video signals are meant in principle to be directly displayed using a 3D display system, though some video processing might also be involved before display
COURSE OF INVESTIGATION 23TABLE 1.4 ContinuedVideo TreatmentApproach DescriptionVideo plus depth The video plus depth (V + D) representation consists of a video (V + D) signal and a per pixel depth map. (This is also called 2D plus depth by some and color plus depth by others). Per pixel depth data is usually generated from calibrated stereo or multi-view video by depth estimation and can be regarded as a monochromatic, luminance-only video signal. The depth range is restricted to a range in between two extremes z near and z far indicating the minimum and maximum distance respectively, of the corresponding 3D point from the camera. Typically, the depth range is quantized with 8 bit, associating the closest point with the value 255 and the most distant point with the value 0. With that, the depth map is speciﬁed as a grayscale image that can be fed into the luminance channel of a video signal and then be processed by any state-of-the-art video codec. For displaying V + D at the decoder, a stereo pair can be rendered from the video and depth information by 3D warping with camera geometry informationMulti-view video Advanced 3D video applications are wide-ranging multi-view plus depth autostereoscopic displays and free viewpoint videos, where the user (MV + D) can choose an own viewpoint. They require a 3D video format that allows rendering a continuum of output views or a very large number of different output views at the decoder. Multi-view video by itself does not support a continuum and coding is increasingly inefﬁcient for a large number of views. V + D supports only a very limited continuum around the available original view since view synthesis artifact increase dramatically with the distance of the virtual viewpoint. Therefore, a MV + D representation is required for advanced 3D video applications. MV + D involves a number of complex and error-prone processing steps. Depth has to be estimated for the N views at the sender. N color with N depth videos have to be encoded and transmitted. At the receiver, the data have to be decoded and the virtual views have to be rendered. The multi-view video coding (MVC) standard–developed MPEG supports this format and is capable of exploiting the correlation between the multiple views that are required to represent 3D videoLayered depth Layered depth video is a derivative and alternative to MV + D. It video (LDV) uses one color video with associated depth map and a background layer with associated depth map. The background layer includes image content that is covered by foreground objects in the main layer. LDV might be more efﬁcient than MV + D because less data have to be transmitted. On the other hand, additional error-prone vision tasks are included that operate on partially unreliable depth data that may increase artifactsa Based on concepts from: 3DPHONE Document “All 3D Imaging Phone, 7th Framework Pro-gramme”.
24 INTRODUCTIONREFERENCES 1. Steinberg S. 3DTV: Is the World Really Ready to Upgrade? Digital Trends, Online Magazine. Jan 7, 2010. 2. Onural L. The 3DTV Toolbox—The Results of the 3DTV NoE. 3DTV NoE Coordi- nator, Bilkent University, Workshop on 3DTV Broadcasting, Geneva. Apr 30, 2009. 3. Dosch C, Wood D. Can we create the “holodeck”? The challenge of 3D television. ITU News Magazine: Article: Issue No 09. Nov 2008. 4. Wallop H. CES 2010: 3D TVs on sale in UK by April. Telegraph. Jan 7, 2010. telegraph.co.uk. 5. Tarr G. BDA Welcomes 3D into Blu-ray Fold. TWICE. Jan 8, 2010. 6. Shilov A. Blu-Ray Disc Association Finalizes Stereoscopic 3D Speciﬁcation: Blu- Ray 3D Spec Finalized: New Players Incoming. xbitslabs On line Magazine. Dec 18, 2009. http://www.xbitlabs.com. 7. 3D TV Round-Up. ITVT Online Magazine. Jan 6, 2010. 8. Aylsworth W. New SMPTE 3D Home Content Master Requirements Set Stage For New Market Growth. Las Vegas (NV): National Association of Broadcasters; 2009. 9. Otellini P. Intel Corporation President and CEO, Keynote Speech, Consumer Elec- tronics Show, Las Vegas (NV). Jan 7, 2010.10. 3-D Video Changes the Consumer Content Experience, CEA/ETC@USC SURVEY FINDS. Joint Consumer Study of the Consumer Electronics Association and the Entertainment and Technology Center at the University of Southern California. Feb 20, 2009.11. Digital Video Broadcasting Project (DVB), Online website material regarding the launch of the DVB 3D TV Kick-Off Workshop. Jan 2010.12. Dosch C. Toward Worldwide Standards for First and Second Generation 3d TV. Workshop on Three-Dimensional Television Broadcasting. Organized jointly by ITU- R, EBU and SMPTE, Geneva. April 30, 2009.13. Merritt R. Incomplete 3DTV Products in CES Spotlight HDMI Upgrade One of Latest Pieces in Stereo 3D Puzzle. EE Times. Dec 23, 2009.14. ITU-R Newsﬂash. ITU Journey To Worldwide “3D Television” System Begins, Geneva. Jun 3, 2008.15. Rosenhahn B, editor. D26.3 Technical Report # 3 on 3D: Time-varying Scene Cap- ture Technologies. Project Number: 511568, Project Acronym: 3DTV Initiative Title: Integrated Three-Dimensional Television—Capture, Transmission and Display. TC1 WP7 Technical Report 3. March 2008.16. Kauff P, M¨ ller M, et al. ICT- 215075 3D4YOU, Deliverable D2.1.2: Requirements u on Post-production and Formats Conversion. August 2008.17. Wood D. Adding value to 3D TV standards, Chair, ITU-R WP 6C. Apr 29, 2009. Comments found at International Telecommunication Union website. www.itu.int.18. Steenhuysen J. For Some, 3D Movies a Pain in the Head. Reuters. Jan 11, 2010.19. ITU-R Activities in 3D WP6C Rapporteurs for 3D TV. Apr 30, 2009.20. TVB, Television Broadcast. A 3DTV Update From the MPEG Industry Forum. Online Magazine. Jan 20, 2010. www.televisionbroadcast.com.
FURTHER READING 2521. IST–6th Framework Programme, 3DTV NoE, 2004. Project Coordinator: Prof. Levent Onural, EEE Department, Bilkent University, TR-06800 Ankara, Turkey.22. 3DPHONE. Project no. FP7-213349, Project title: All 3D Imaging Phone, 7th Framework Programme, Speciﬁc Programme “Cooperation”, FP7-ICT- 2007.1.5—Networked Media, D5.2- Report on ﬁrst study results for 3D video solu- tions. Dec 31, 2008.23. 3DPHONE, Project no. FP7-213349, Project title: All 3D Imaging Phone, 7th Framework Programme, Speciﬁc Programme “Cooperation”, FP7-ICT- 2007.1.5—Networked Media, D5.1- Requirements and speciﬁcations for 3D video. Aug 19, 2008.24. TVB Magazine. TVB’s 3DTV Timeline. Online Magazine. Jan 5, 2010. www. televisionbroadcast.com.FURTHER READINGOzaktas HM, Onural L, editors, Three-dimensional television: capture, transmission, display. New York: Springer Verlag; 2008. XVIII, 630, p. 316 illus. ISBN: 978-3- 540-72531-2.Bahram J, Fumio Okano, editors, Three-dimensional television, video and display tech- nology. New York: Springer Verlag; 2002.Schreer O, Kauff P, Sikora T, editors, 3D Videocommunication: Algorithms, concepts and real-time systems in human-centered communication (Hardcover), New York: Wiley, John & Sons; 2005, ISBN-13: 9780470022719.Minoli D, 3D Television (3DTV) Technology, Systems, and Deployment—Rolling out the Infrastructure for Next-Generation Entertainment. Francis and Taylor; 2010.
26 INTRODUCTIONAPPENDIX A1: SOME RECENT INDUSTRY EVENTS RELATED TO 3DTVThis appendix includes a listing of events during the year prior to the publicationof this text, so as to further document the activity in this arena. It is based in itsentirety on Ref. 24. Despite the economic difﬁculties of 2009, the year marked a turning point in the adoption of 3D a viable entertainment format. TVB presents a timeline of 3D video developments over the last year, from content to workﬂow initiatives to display tech- nologies: December 4, 2008 : The San Diego Chargers and the Oakland Raiders appeared in a 3D simulcast displayed at theaters in Boston, Hollywood, and New York. January 8, 2009 : A 3D version of the Gators–Sooners match-up was simulcast in Las Vegas at the Consumer Electronics Show. February 14, 2009 : The NBA’s All-Star Game was simulcast in 3D. February 24, 2009 : Toshiba announces the development of OLED Wallpaper TV, with a 3D version utilizing circularly polarized light in the works. March 2, 2009 : Avid Technology announced it was developing native support for the Sony XDCAM format, as well as adding 3D capabilities to its various editing software packages, Composer and Symphony. March 9, 2009 : BSkyB continued plowing toward 3DTV, with a goal to offer it by the end of the year. April 6, 2009 : BSkyB successfully transmitted live 3DTV across its HD infrastructure in the United Kingdom. April 20, 2009 : At the NAB show in Las Vegas, Panasonic announced work on a full 3D HD production system, encompassing everything from capture to Blu- ray distribution. The Panasonic gear list comprised authoring, a twin-lens P2 camera recorder and drives, 3D Blu-ray discs and players, and a 3D plasma display. Panasonic displayed its HD 3D Plasma Home Theater at the NAB convention. July 30, 2009 : BSkyB now plans to launch its 3D channel in 2010. August 24, 2009 : Panasonic joined James Cameron in a ﬂack blitz for “Avatar,” with a multipoint media and sales campaign and a nationwide tour with customized 18- wheelers outﬁtted with 103-inch Panasonic Viera plasma HDTVs and Blu-ray disc players. September 2, 2009 : Sony announced that it planned to introduce a consumer-ready 3D TV set in 2010, as well as build 3D capability into many of its consumer electronics, encompassing music, movies, and video games. September 10, 2009 : Mobile TV production specialist NEP has rolled out its ﬁrst 3D truck.