Your SlideShare is downloading. ×
0
×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

3DgraphicsAndAR

665

Published on

this is a presentation I did in Korea, 2012 at the Global 3D Technology Forum

this is a presentation I did in Korea, 2012 at the Global 3D Technology Forum

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
665
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
13
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. MPEG 3D GraphicsRecent achievements and future roadmap GLOBAL 3D Technology Forum, October 9-12, KINTEX, Korea Marius Preda, MPEG 3DG Chair Institut Mines TELECOM http://www.slideshare.net/MariusPreda/3DgraphicsAndAR
  • 2. 3D Graphics WorldTowards a standard representation: a double problem  No common representation – Heterogeneous kind of data • Different types of geometry, appearance and animation models – Always easier to specify a new data representation format than learning an existent one  Very different application domains – Different requirements and use cases
  • 3. 3D Graphics Chain : Authoring and Publishing Authoring Very heterogeneous production chain: -A 3D Asset is processed by several authoring tools, each one with dedicated functionalities -Each authoring tool has its own format -Artists and creators are investing time in learning how the tool works
  • 4. 3D Graphics Chain : Authoring and Publishing Publishing In BOX solutions - Games are containing all the data on DVD - Rendering engine friendly data format ONLINE solutions - ad hoc format, usually zipped version of an authoring format
  • 5. 3D Graphics Chain : Authoring and Publishing Gap between the authoring and publishing formats FAST TRANSPORT TROUGHT THE NETWORK Authoring Transmission Publishing Size is of main Size doesn’t matter Size doesn’t matter importance
  • 6. 3D Graphics Chain : Authoring and Publishing Solving the gap: encoding layer FAST TRANSPORT TROUGHT THE NETWORK Authoring Transmission Publishing Size doesn’t matter Size doesn’t matter MPEG-4 3DG Size is of main importance Encoder Decoder
  • 7. 3D Graphics Chain : Authoring and Publishing MPEG 3DG Vision To define a standard format for compressed 3D synthetic content. In other words to be for graphics what MP3 and AAC are for audio, MPEG-2 and MPEG-4 are for video and JPEG is for still images. Additionally "MPEG 3D Graphics" aims at providing mechanisms such as APIs to enable easy integration and development of applications using its standard representation tools.
  • 8. MPEG 3D GraphicsTopics of the dayWhat is MPEG?MPEG offer in the 3D Graphics fieldOngoing projects: integrating real and virtual
  • 9. MPEG 3D GraphicsTopics of the dayWhat is MPEG?MPEG offer in the 3D Graphics fieldOngoing projects: integrating real and virtual
  • 10. What is MPEG?A suite of ~130 ISO/IEC standardsCoding/compression of elementary media: – Audio (MPEG-1, 2 and 4) – Video (MPEG-1, 2 and 4) – 2D/3D graphics (MPEG-4)Storage and Transport – MPEG-2 Transport – File Format (MPEG-4) – Dynamic Adaptive Streaming over HTTP (DASH)Hybrid (natural & synthetic) scene description, user interaction (MPEG-4)Metadata (MPEG-7)Media management and protection (MPEG-21)Sensors and actuators, virtual worlds (MPEG-V)Advanced User interaction (MPEG-U)Media-oriented middleware (MPEG-M)More ISO/IEC standards under development for – 3D Video, 3D Audio – Coding and Delivery in Heterogeneous Environments – …
  • 11. What is MPEG?Involvement, approach, deploymentA standardization activity continuing for 24 years – Supported by several hundreds companies/organisations from ~25 countries – ~500 experts participating in quarterly meetings – More than 2300 active contributors – Many thousands experts working in companiesA proven manner to organize the work to deliver useful and used standards – Developing standards by integrating individual technologies – Well defined procedures – Subgroups with clear objectives – Ad hoc groups continuing coordinated work between meetingsMPEG standards are widely referenced by industry – 3GPP, ARIB, ATSC, DVB, DVD-Forum, BDA, EITSI, SCTE, TIA, DLNA, DECE, OIPF…Billions of software and hardware devices built on MPEG technologies – MP3 players, cameras, mobile handsets, PCs, DVD/Blue-Ray players, STBs, TVs, …
  • 12. MPEG 3D GraphicsTopics of the dayWhat is MPEG?MPEG offer in the 3D Graphics fieldOngoing projects: integrating real and virtual
  • 13. MPEG technologies related to 3D Graphics1992/4 1997 1998 1999MPEG-1/2(AV content) VRML MPEG-4 v.1 • Part 11 - BIFS: -Binarisation of VRML -Extensions for streaming -Extensions for server command -Extensions for 2D graphics - Real time augmentation with audio & video • Part 2 - Visual: - 3D Mesh compression MPEG-4 v.2 - Face animation • Part 2 – Visual - Body animation First form of broadcast signal augmentation
  • 14. MPEG technologies related to 3D Graphics2003 2005 2007 2011MPEG-4• Part 16 - AFX: - A rich set of 3D MPEG-4 graphics tools •AFX 2nd Edition: - Compression of - Animation by geometry, morphing appearance, - Multi-texturing animation MPEG-4 • AFX 3rd Edition - WSS for terrain and cities A rich set of 3D Graphics - Frame based MPEG-4 representation and animation • AFX 4th Edition compression tools - Scalable complexity mesh coding
  • 15. MPEG technologies related to 3D Graphics2003 2004 2005 2007 2009 2011MPEG-4• Part 16 - AFX: - A rich set of 3D MPEG-4 graphics tools •AFX 2nd Edition: - Compression of - Animation by geometry, morphing appearance, - Multi-texturing animation MPEG-4 • AFX 3rd Edition - WSS for terrain and cities MPEG-4 MPEG-4 - Frame based MPEG-4 • Part 16 • Part 25 animation • AFX 4th Edition - X3D - Compression of - Scalable complexity Interactive third-party XML mesh coding Profile (X3D, COLLADA)
  • 16. BBAMPEG 3DG Tools Compression Tool Type 3D Mesh Compression – 3DMC Geometry Scalable Complexity 3DMesh Coding – SC3DMC Geometry Wavelet Subdivision Surface – WSS Geometry Coordinate, Orientation and Position Interpolator – IC Animation Bone-based Animation – BBA Animation Frame-based Animated Mesh Compression - FAMC Animation Octree Compression for Depth Image-based Representation Appearance Point Texture AppearanceFAMC WSS 3DMC IC
  • 17. BBAMPEG 3DG Tools Compression Tool Type 3D Mesh Compression – 3DMC Geometry Scalable Complexity 3DMesh Coding – SC3DMC Geometry Wavelet Subdivision Surface – WSS Geometry Coordinate, Orientation and Position Interpolator – IC Animation Bone-based Animation – BBA Animation Frame-based Animated Mesh Compression - FAMC Animation Octree Compression for Depth Image-based Representation Appearance Point Texture AppearanceFAMC WSS 3DMC IC
  • 18. Reconfigurable Graphics Coding • Establish a standardized set of Functional Units (FUs) • Combine the FUs in a consistent decoder by describing it and transmit the description together with the encoded data
  • 19. Reconfigurable Graphics Coding Graphics Tool Library FU Name ArithmeticDecode_AdaptiveData [ AC_MaxLength ] [ N_DATA_MODELS ]Algo_ParserSC3DMC [ DM__LengthShift ]Algo_InverseQuantization1D dataSymbols_i dataOut_o lastSymbol_iAlgo_InverseQuantizationND tableBits_iAlgo_InversePrediction1D tableShift_i distribution_iAlgo_InversePredictionND useDecoderTable_i decoderTable_i dataModelFlag [ dataModelIndex ] = 0 Loop until the followingAlgo_ArithmeticCodec_StaticBit Loop until the following dataModelFlag [ dataModelIndex ] = 0 are available: are available: tableSize_i - decoderTableSize - decoderTable [ decoderTableSize ] - distribution [ distributionSize ]Algo_ArithmeticCodec_AdaptiveBit symbolCount_i Loop until the following is available: - distributionSize - lastSymbol dataModelIndex_i - tableShift [INPUT]:Algo_AdaptiveData_Model_GenerateAlphabet codeBytes_i [INPUT]: [INPUT]: decoderTableSize_i decoderTable_i [decoderTableSize] codeBuffer_i tableShift_i distributionSize_i distribution_i [distributionSize] lastSymbol_iAlgo_ArithmeticDecode_AdaptiveData [PROCESS]: [PROCESS]:Algo_ReOrderConnec1D dv s else t n elseAlgo_ReOrderConnecNDAlgo_ReOrderTab1D distribution[m] > dv [PROCESS]: n s != lastSymbol [PROCESS]: y useDecoderTable = 1 n>s+1 [PROCESS]: [PROCESS]:Algo_ReOrderTabND m [PROCESS]: x Loop until the following sAlgo_ReOrderPerm1D Loop until the following is available: is available: - codeBuffer Loop until the following is available: else else [ dataModelIndex ] = 0 - codeBytes - useDecoderTable elseAlgo_ReOrderPermND dataModelFlag else Loop until the following is available: [PROCESS]: - dataSymbols [INPUT]: [INPUT]: length [INPUT]:Algo_InverseBinarization codeBytes_i codeBuffer_i value n,s,x,y useDecoderTable_i [INPUT]: dataSymbols_i z > value [PROCESS]: else else [PROCESS]: n, yAlgo_Provider_1D codeBytes < 4 x s length [PROCESS]: z [PROCESS]: [PROCESS]: mAlgo_Provider_2D Loop until the following is available: else m else s,x m != sAlgo_GeometryDecoder_TFAN - dataModelIndexAlgo_Decode_Symbol [INPUT]: length < AC_MinLength dataModelIndex_iAlgo_Decode_I_Frame_Quantized [PROCESS]: value [PROCESS]: value ac_pointer [PROCESS]: symbolCount [ s ] [PROCESS]: dataModelFlag [dataModelIndex ] length length symbolsUntilUpdateAlgo_Decode_P_Frame Loop until the following length < AC_MinLength =1 is available:Algo_Huffman_Decode - reset else else symbolsUntilUpdate = 0 elseAlgo_DCT_Dequant START [INPUT]: reset_i [PROCESS]: reset = 1 bufferSizeAlgo_IDCT_Trans newBuffer codeBuffer [EMBED]: ac_pointer updateAlgo_InterpolateCoord dataModelFlag [] = 0Algo_QuatNormAlgo_QuatConj [OUTPUT]: dataOut_o = sAlgo_QuatScalar_NotNorm
  • 20. Reconfigurable Graphics CodingDecoding network example A sub-network of SC3DMC
  • 21. Applications Phone and tablet players Games Web databases (with and without plugins - WebGL)
  • 22. MPEG 3D GraphicsTopics of the dayWhat is MPEG?MPEG offer in the 3D Graphics fieldOngoing projects: integrating real and virtual
  • 23. MPEG technologies related to real world2011 2012 201x 201x 201xMPEG-V - MediaContext and Control MPEG-U –• 1st Edition Advanced - Sensors and User Interface MPEG-V actuators • 2nd Edition: - Interoperability - GPS between Virtual - Biosensors Worlds - 3D Camera MPEG-H - 3D Video • Compression A rich set of sensors and CDVS of video + actuators • Feature-point based depth - 3D Audio descriptors for image recognition
  • 24. MPEG-V: Sensors and Actuators
  • 25. Bridging Virtual and Real Worlds Virtual Objects and Virtual World Augmented Reality Real World User Real World (Sensor Device) (Actuators)
  • 26. Main features of MPEG AR technologies • All AR-related data is available from MPEG standards • Real time composition of synthetic and natural objects • Access to – Remotely/locally stored BIFS/compressed 2D/3D mesh objects – Streamed real-time BIFS/compressed 2D/3D mesh objects • Inherent object scalability (e.g. for streaming) • User interaction & server generated scene changes • Physical context – Captured by a broad range of standard sensors – Affected by a broad range of standard actuators
  • 27. MPEG vision on AR, the MPEG AR Browser Point to a URL – no need to download new applications for each context. The browser – Retrieves scenario from the internet – Starts video acquisition – Tracks objects – Recognizes objects from visual signatures – Recovers camera pose – Gets streamed 3D graphics – Composes new scenes – Gets inputs from various sensors – Offers optimal AR experience by constantly adapting interaction possibilities and objects from a remote server. Industry – Maximize number of customers through MPEG-compliant authoring tools and browsers – No need to develop a new application for each use case and device platform
  • 28. MPEG vision on AR Produce Download Compression Authoring Tool MPEG-4/MPEG-7/MPEG-21/ MPEG-U/MPEG-V MPEG Player
  • 29. Architecture Remote Remote Local Local Real World Sensors & Sensors & Real World Environment Actuators Actuators Environment AR Player User AR file or stream Media Service Servers Servers
  • 30. Augmented Reality Reference Model Enterprise viewpoint: global architecture and actors Local / Remote Context MCP ARTC TO Telecommunication Operator (TO) End-User (EU) DM MCP AREC AR EU TO AR Player User Document Device Manufacturer (DM) AC TO TO Middleware/Component Provider (MCP)AR Tools Creator (ARTC)AR Experience Creator (AREC) Online Middleware/ComponentAssets Creator (AC) Media Provider (OMCP) Service AR Service Provider (ARSP) Servers ServersAssets Aggregator (AA) AC AA OMCP ARSP
  • 31. Augmented Reality Reference ModelInformation viewpoint Local/Remote Context Scene/Real World Device Context • Raw image • Device capabilities • Sensed data Location of Device • Virtual Camera view • Location • Detected features • Orientation • Area of Interest/Anchors Presentation AR Player Spatial Models • • Coordinate Ref. Sys. Augmentation • (Geol)ocation User • Registration AR • Styling/complexity • Projections • • Coordinate conversion Document Spatial Filtering, e.g. User Input range Tracking objects • Query • Markers • Manipulation of • Marker-less Presentation • Topics of interest • Preferences Digital Assets • Presentation data Media Service • Trigger/Event rules Servers Servers • Accuracy based
  • 32. Augmented Reality Reference ModelComputational viewpoint Local / Remote Context 2 1 5 AR AR Player User Document 3 4 Media Service Servers Servers
  • 33. Augmented Reality Reference ModelComputational viewpoint Local / Remote Context 2 1 3 AR AR Player User Document 4 5 Media Service Servers Servers
  • 34. Augmented Reality Reference ModelEngineering viewpoint Local/Remote Context Accelero Camera Mic Compass GPS … meter AR Player User AR Rendering Display Application Document … Engine (A/V/H) Engine Media Service Servers Servers
  • 35. Augmented Reality Reference ModelGlossary
  • 36. Augmented Reality Reference ModelUse cases
  • 37. Augmented Reality Reference ModelHow to contribute? Use Trac! http://wg11.sc29.org/trac/augmentedreality/
  • 38. MPEG-A Part 13 ARAFThree components: scene, sensors/actuators, medias A set of scene graph nodes/protos as defined in MPEG-4 Part 11 – Existing nodes – Audio, image, video, graphics, programming, communication, user interactivity, animation – New standard PROTOs – Map, MapMarker, Overlay, ReferenceSignal, ReferenceSignalLocation, CameraCalibration, AugmentedRegion Connection to sensors as defined in MPEG-V – Orientation, Position, Angular Velocity, Acceleration, GPS, Geomagnetic, Altitude – Local camera sensor Compressed media
  • 39. MPEG-A Part 13 ARAFMPEG Augmented Reality examples AR Quiz Augmented Book
  • 40. MPEG-A Part 13 ARAFMPEG Augmented Reality examples AR Quiz Augmented Book http://youtu.be/la-Oez0aaHE http://youtu.be/LXZUbAFPP-Y
  • 41. Conclusions• MPEG standardized a rich set of 3D Graphics representations and theirs compressions• MPEG recently proposed an approach to address the complexity of 3DG domain: Reconfigurable Graphics Coding• Joint development of an AR Reference Model (with SC24/WG9, ARStandards, Khronos, OGC) – The community at large is invited to contribute – http://wg11.sc29.org/trac/augmentedreality• MPEG promoted at CD a first version of an integrated and consistent solution for representing content in AR applications and services – Continue synchronized/harmonized development of technical specifications with X3D, COLLADA, OGC content models
  • 42. Thank you ! • www.MyMultimediaWorld.com • www-artemis.it-sudparis.eu • marius.preda@it-sudparis.eu

×