Your SlideShare is downloading. ×
  • Like
Basics of Mpeg 4 3D Graphics Compression
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Basics of Mpeg 4 3D Graphics Compression

  • 1,089 views
Published

A presentation of MPEG-4 (Part 16) tools for video compression

A presentation of MPEG-4 (Part 16) tools for video compression

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • Open3DGC provides a cross platform C++ implementation (under MIT License) of patent free MPEG tools for 3D graphics compression

    https://github.com/amd/rest3d

    http://kmamou.blogspot.ca/2013/07/open-3d-graphics-compression-open3dgc.html
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
1,089
On SlideShare
0
From Embeds
0
Number of Embeds
3

Actions

Shares
Downloads
42
Comments
1
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Marius Preda, PhDChairman of MPEG 3DGInstitut MINES TELECOM
  • 2.   It’s a crazy multimedia world! –  Network is everywhere… but very heterogeneous –  Terminals are “same same”… but different! –  Content must be designed with a priori knowledge of its future use –  Applications are platform-centric instead of user-centric  Fragmented value chain Production Transmission Consumption Capturing HW/SW Playback Networks devices providers devices INTEROPERABILITY NEEDED!!! Authoring IPR Terminal Providers tools holders manufacturers Service End users scenarios
  • 3. MPEG has several names •  Common: MPEG = Moving Picture Experts Group •  Official: MPEG = ISO/IEC JTC1/SC29/WG11 ISO Over 224 technical committees JTC1 Joint Technical Committee with IEC SC24 SC29Computer graphics and image processing Coding of audio, picture, multimedia and hypermedia information GKS, PHIGS WG1 JPEG CGM, VRML, X3D Coding of still pictures WG11 MPEG Coding of moving pictures and audio
  • 4. MPEG has several subgroups Video ... and produced several Requirements Audio Systems successful standards 3D Graphics MPEG MPEG-1 MPEG-2 MPEG-4 MPEG-7 MPEG-21 ISO/IEC 11172 ’92 ISO/IEC 13818 ’94 ISO/IEC 14496 ’99 ISO/IEC 15938 ’01 ISO/IEC 21000 ’02 Video & audio Video & audio Multimedia Metadata Terminal (CD-ROM) (DVD & DVB) & interactive (description & network applications of content) specificationMPEG-A MPEG-B MPEG-C MPEG-D MPEG-E MPEG-M MPEG-U MPEG-V
  • 5. MPEG-4 FeaturesMPEG-4 ObjectsNaturals – still images, audio, 2D/3D videoSynthetic - audio, 2D/3D objects and scenes Compression CompressionMPEG-4 Objectives CompressionProvide technologies for efficientcompression and transmissionMPEG-4 TerminalComposition, at end-user side, of naturaland synthetic objects, into hybrid andinteractive scenes
  • 6. MPEG-4 FeaturesSystem architecture Interactive Scene Description Scene layer Object descriptor layerMPEG-4stream Media data layer
  • 7.   Video Compression –  MPEG-4 Visual and MPEG-4 AVC  3D Graphics Compression
  • 8.   No common representation –  Heterogeneous kind of data •  Different types of geometry, appearance and animation models –  Always easier to specify a new data representation format than learning an existent one  Very different application domains
  • 9. FAST TRANSPORT TROUGHT THE NETWORK Size is of main Size doesn’t matterSize doesn’t matter importance
  • 10. FAST TRANSPORT TROUGHT THE NETWORK Size doesn’t matterSize doesn’t matter Size is of main importance
  • 11. To define a standard format for compressed 3D synthetic content. In other words to be for graphics what MP3 and AAC are for audio, MPEG-2 and MPEG-4 are for video and JPEG is for still images.Additionally "MPEG 3D Graphics" aims at providing mechanisms such as APIs to enable easy integration and development of applications using its standard representation tools.
  • 12. MPEG 3D Graphics within the MPEG standards familyCore technologies in MPEG 3D GraphicsCompressing other standards (COLLADA, X3D, …) with MPEGVirtual Worlds Interoperability for avatars with MPEG
  • 13. MPEG 3D Graphics within the MPEG standards familyCore technologies in MPEG 3D GraphicsCompressing other standards (COLLADA, X3D, …) with MPEGVirtual Worlds Interoperability for avatars with MPEG
  • 14. MPEG-4 3D chronology
  • 15. MPEG-4 3D chronology
  • 16. MPEG-4 3D chronology
  • 17. MPEG 3D Graphics within the MPEG standards familyCore technologies in MPEG 3D GraphicsCompressing other standards (COLLADA, X3D, …) with MPEGVirtual Worlds Interoperability for avatars with MPEG
  • 18.   IFS surfaces
  • 19. Approximation of target surface•  Method: tesselation with planar facets•  Quality: first order (linear) ⇒ no smoothness (C0 continuity)Mesh definition: IFS (Indexed Face Set)•  Connectivity: list of faces {…, Pn = {in0, in1, in2, …}, …} ⇒ arbitrary topology (non-manifold, open, higher genus, etc.)•  Geometry: list of vertices {…, Vk = (xk, yk, zk), …}Mesh coding•  Connectivity (lossless): triangle strips, triangle+vertex trees, etc.•  Geometry (lossy): coordinate quantisation + prediction from conn.
  • 20. LOD concept•  1976: Clark introduced the idea•  Main interest: rendering efficiencyTaxonomy of LOD extraction techniques•  Static vs. dynamic•  Global vs. local•  Progressive vs. hierarchical LODsSuccessful simplification techniques•  1996: Hoppe’s edge collapses•  1997: Garland’s quadrics ⇒ qslimProgressive 3D mesh coding•  1996: Hoppe’s PM (Progressive Mesh)•  1998: IBM’s PFS (Progressive Forest Split)
  • 21. Triangles: 69451 vs. 29519
  • 22. Triangles: 69451 vs. 9575
  • 23. Triangles: 69451 vs. 1627
  • 24. Triangles: 69451 vs. 238
  • 25. MPEG-4 (1999): IFSs•  Based on VRML97•  Arbitrary topology meshes•  “Properties” (normals, colours and textures)MPEG-4 Amd.1 (2000): 3DMC (3D Mesh Coding)•  40-50:1 compression of IFSs by IBM’s TS (Topological Surgery)•  Incremental transmission and rendering•  Progressive coding by IBM’s PFS•  Error resilience by SAIT
  • 26. MPEG-4 (1999): IFSs•  Based on VRML97•  Arbitrary topology meshes•  “Properties” (normals, colours and textures)MPEG-4 Amd.1 (2000): 3DMC (3D Mesh Coding)•  40-50:1 compression of IFSs by IBM’s TS (Topological Surgery)•  Incremental transmission and rendering•  Progressive coding by IBM’s PFS•  Error resilience by SAIT
  • 27.   IFS surfaces
  • 28.   IFS surfaces  Patches
  • 29. Approximation of target surface•  Method: tesselation with predefined curved patches•  Quality: higher order (polynomic/rational) ⇒ Cn continuityMesh definition•  Connectivity: regular grid of quads. or triangles ⇒ planar topology•  Geometry: list of control points {…, Pk = (xk, yk, zk), …}Mesh coding•  Connectivity (lossless): implicit•  Geometry (lossy): coordinate quantisation + prediction from conn.
  • 30. Tensor product of cubic Bézier curves Single patch (4x4 control points) vs. two patches (4x7 control points) Compression  (3547 polygons; 1215 vertices) vs. (86 patches; 212 control points)
  • 31. MPEG-4 Part 16 (2003): NURBS•  Based on VRML97 Amd., originally proposed by blaxxun•  Support for NURBS curves and patches   Specific nodes for Bézier’s curves and patches (for increased efficiency)•  Support for free-form deformations
  • 32.   IFS surfaces  Patches  Subdivision surfaces
  • 33. SS = limit of recursive refinement of base control mesh NB: refinement affects both Geometry smoothing achieved withconnectivity (of abstract graph) and stencils particular to each scheme geometry (of 3D mapping) -1/16 -1/16 ½ -1/16 9/16 9/16 -1/16 1/8 1/8 Border/sharp vs. z interior edge stencils ½ of “butterfly” scheme x y -1/16 -1/16 SS inherently define hierarchically nested LODs
  • 34. Approximation of target surface•  Method: tesselation with curved patches•  Quality: higher order ⇒ Cn continuityMesh definition•  Connectivity: list of triangles/quads., e.g., {…, Tn = {in0, in1, in2}, …} ⇒ arbitrary (manifold) topology•  Geometry: list of control points {…, Pk = (xk, yk, zk), …}Mesh coding•  Connectivity (lossless): as for polygonal (manifold) mesh•  Geometry (lossy): as for polygonal (manifold) mesh
  • 35. Polygons+ Are the simplest approach (linear approximation)+ Can resolve fine details and handle arbitrary topologies– Lead to unstructured, huge meshesPatches+ Are a more powerful approach (higher order approximation)+ Are convenient for coarse and smooth models– Need cumbersome trimming and stitching mechanismsSSs+ Connect and unify the two extremes above++ Provide multi-resolution handles for hierarchical coding/editing
  • 36. Catmull-Clark’s (1978): quadrilateral, primal, approximating, C2 Loop’s (1987): triangular, primal, approximating, C2Dyn++’s “butterfly” (1990): triangular, primal, interpolating, C1
  • 37.   IFS surfaces  Patches  Subdivision surfaces  Wavelet subdivision surfaces
  • 38. Target surfaceBasemesh Subdivide Add details
  • 39. Target surfaceBasemesh Subdivide Add detailsPrice (requirements)•  Base mesh extraction•  Subdivision scheme = predictor•  Details (3D vectors) = prediction errors ⇒ remeshingPrize (advantages)•  Predictive coding ⇒ immediate (if smooth target mesh)
  • 40. Spatial partitioning Adding details in appearing parts Removing details in disappearing partsZT ZT ZT … ZT
  • 41. by courtesy of FranceTelecom
  • 42. MPEG-4 Part 16 (2003): “plain” + wavelet SSs•  “Plain” SSs for mesh smoothing   Considered schemes: Catmull-Clark, [extended] Loop, butterfly   No details are added but…   normal control achievable through edge/vertex tagging of initial control mesh•  Wavelet/detailed SSs for surface approximation   Possibly tagged base mesh   Details are added after each subdivision step, which are…   wavelet-transformed according to one of several possible schemes   Most suited for multi-resolution editing/animation   Most suited for view-dependent transmission
  • 43.   IFS surfaces  Patches  Subdivision surfaces  Wavelet subdivision surfaces  Mesh Grid
  • 44. CW (Connectivity-Wireframe)RG (Reference-Grid)
  • 45. Vertex offset is a relative value Update vertex position when grid is deformed or animated
  • 46.   IFS surfaces  Patches  Subdivision surfaces  Wavelet subdivision surfaces  Mesh Grid  Solids
  • 47. Solid primitivesSolid models: the “arithmetic of forms”
  • 48. Implicit equation:Quadrics (2nd order):Quartics (4th order):
  • 49. F1 * 0 1 2 0 0 0 0 F0 1 0 1 2 2 0 2 4A cube = multiplication of 3 degenerated quadrics Multiplication of two forms Examples of solid operations
  • 50. Architecture MechanicsBiotechnology Virtual models
  • 51. Exact geometryCompactness 21 Kb 37 Kb 407 Kb 1.1 Mb
  • 52.   IFS surfaces  Patches  Subdivision surf.  Wavelet SS  Mesh Grid  Solids  SC-3DMC
  • 53. MPEG 3DGC Scalable Complexity 3D Mesh Compression •  Not all 3DG applications have the same needs in compression •  Not all the 3DG applications can afford spending extra CPU/GPU for compression TFAN SVA SC3DMC QBCR Same Quantization and Binarization blocksTowards the continuum model: enlarge the application domain where MPEG-4 3DG can be used
  • 54. MPEG 3DGCScalable Complexity 3D Mesh Compression x, y, z (floats) + normals (floats) x, y, z (floats) + colors (floats) x, y, z (floats) + … (floats) i, j, k (integers) Attributes i, j, k (integers) i, j, k (integers) Connectivity
  • 55. SC-3DMC General SchemaConnectivity TFAN BP CABAC Delta (lossless) SVA FLB AC BAC Connectivity Prediction Binarization Entropy i, j, k Analysis Encoding Attributes Prediction Entropy Binarization (lossy) Encoding Parallelogram Delta BP CABAC x, y, z Quantization Barycentre FLB AC 5 different paths with different performances
  • 56. SC-3DMCConnectivity Analysis: EmptyDo Nothing 3 1 4 2 123234
  • 57. SC-3DMCConnectivity Analysis: SVA Previous Face4 modes for encoding how consecutive Current Face faces share vertices Shared Vertex 1 5 3 1, 2, 3, 1, 2, 3, 1 4 3 2, 3, 4 3, 4, 5 12304 2 2 4 123245 Mode 0 Mode 2 1 4 1 11, 2, 3, 1, 2, 3,4, 5, 6 1, 2, 3 2 3 2 3 2 3 5 6 OR 1231 5 Mode 1 Mode 3 1233
  • 58. SC-3DMCConnectivity Analysis: TFANSplit the mesh in set of triangle fans and encode each fan 1675 62937 2984 Transformed in local indices
  • 59. SC-3DMCWhat we measure?Encoding performances but also complexity for decoder (and encoder) TFANPerformances SVA-AC 3DMC SVA-BAC SVA-BP NCA BIFS Complexity
  • 60.   IFS surfaces  Patches  Subdivision surf.  Wavelet SS  Mesh Grid  Solids  SC-3DMC
  • 61. •  Visual Texture Coding Generic compression tool•  Synthesized Textures Very Compact representation for•  Procedural Textures specific applications•  Depth Image-based Representation Less calculation intensive
  • 62. •  MPEG-4 3DMC•  MPEG-4 MeshGrid•  MPEG-4 Subdivision Surfaces3D Mesh •  MPEG-4 VTC •  JPEG 2000 3D object 2D Texture
  • 63. Zero-TreeWT 3D mipmaps
  • 64. Multiple In Place mapping = multi-resolution textures Without mipmapping With mipmapping
  • 65. Zero-TreeWTComputational Graceful Degradation: Quality => processing power
  • 66. Region of Interest with resolution/quality selection Visual importance depends on viewing angle Low Resolution High Resolution
  • 67. Packet selection by using error-resilience markers
  • 68.   IFS surfaces   VTC  Patches   Synthesized texture  Subdivision surf.  Wavelet SS  Mesh Grid  Solids  SC-3DMC
  • 69. Vector Graphics Primitives Color profile Line Color Profile ( LC ) Area Color Point ( A C ) Line ( LN ), bounded by 2 Terminal Points ( TP ) The image cannot be displayed. Color Patch ( PA ) Line Segment ( LS ), bounded by 2 Line Points ( LP ) Sub -Texture ( ST ) Line marked as control line (Skeleton) The image cannot be displayed.
  • 70. 65x96 pixels10 seconds animation1.35 kB
  • 71.   IFS surfaces   VTC  Patches   Synthesized texture  Subdivision surf.   Procedural texture  Wavelet SS  Mesh Grid  Solids  SC-3DMC
  • 72. DEF Fabric ProceduralTexture { type 2 width 256 height 256 cellWidth 4 cellHeight 4 roughness 1 distortion 0.05 seed 114300 color [ 0.898 0.89418 0.95294, 0.34118 0.29418 0.70196, 0 0 0, 0 0 0 ] aWarpmap [ 0 0, 0.03 1, 0.88 1, 1 0 ] bWarpmap [ 0 0, 0.48 1, 1 0 ] aWeights [ 0, 0, 0, 0, 0, 0, 0.56, 0, 0, 0, 0, 0, 0, 0, 0.20, 0.24 ] bWeights [ 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0 ]} DEF Marble ProceduralTexture { width 256 height 256 roughness 1 seed 22209 color [ 0.8 0.7098 0.6902, 0.95686 0.8902 0.87451, 0.87451 0.37255 0.23529, 0.95686 0.8902 0.87451 ] aWarpmap [ 0 1, 0.33 0, 1 1 ] bWarpmap [ 0 0, 0.55 0, 0.6 1, 0.65 0, 1 0 ] bWeights [ 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0 ] }
  • 73.   IFS surfaces   VTC  Patches   Synthesized texture  Subdivision surf.   Procedural texture  Wavelet SS   DIBR  Mesh Grid  Solids  SC-3DMC
  • 74. Reconstruct 3D representation from projections Depth image Projection
  • 75.   IFS surfaces   VTC  Patches   Synthesized texture  Subdivision surf.   Procedural texture  Wavelet SS   DIBR  Mesh Grid   Point Texture  Solids  SC-3DMC
  • 76.   IFS surfaces   VTC   Interpolators  Patches   Synthesized texture  Subdivision surf.   Procedural texture  Wavelet SS   DIBR  Mesh Grid   Point Texture  Solids  SC-3DMC
  • 77. Piece-wise linear functions definedby a set of pairs time-value Standardized by MPEG-4 for: - position - orientation - scale - coordinate in IFS - normals in IFSStraightforward animation based on key-frames
  • 78. Large amount of data for high quality, smooth animation
  • 79. Re-sampling, sub-sampling: twomethods supported by MPEG-4 Path preserving mode Key preserving modeThree schemes supported byMPEG-4 Orientation Interpolators Coordinate Interpolators Position InterpolatorsExploit temporal redundancy for the most common interpolators
  • 80. Re-sampling, sub-sampling: twomethods supported by MPEG-4 Key value Key value Path preserving mode Key(time) Key(time) Key preserving mode N key values N-1 key valuesThree schemes supported byMPEG-4 Key value Key value Orientation Interpolators Coordinate Interpolators Key(time) Key(time) Position Interpolators N-2 key values N-3 key valuesSub-sampling or re-sampling based on minimal distortion
  • 81. A dedicated elementary stream for IC, multiplexed into an AFX stream
  • 82.   IFS surfaces   VTC   Interpolators  Patches   Synthesized texture   Bone-Based  Subdivision surf.   Procedural texture  Wavelet SS   DIBR  Mesh Grid   Point Texture  Solids  SC-3DMC
  • 83.   Face, MPEG-4 V1, 1999  Body, MPEG-4 Amd1, 2000  Skinned Model, MPEG-4 Part 16, 2003 Two frameworks: human-like (FBA) and generic skeleton (BBA)
  • 84.   Geometry: Seamless mesh: shapes sharing the same vertices list  Texture: Image Mapping on vertices sub-set  Hierarchy: Skeleton layer Muscle layer Seamless mesh affected by a hierarchical skeleton
  • 85.   1D controllers: bone & muscle  for each bone and each muscle - a list of affected vertices - a measure of affectednessare providedRight balance between control parameters and influence volume
  • 86. Uncompressed Uncompressed Segment #n Frame #n BAP/FAP/BBABAP/FAP/BBA Frame P DCT Prediction DC Coeff. AC Coeff. Frame I Segment P Segment Prediction I Quantization Quantization DC Q DC Q AC Q Arithmetic Arithmetic coding coding Huffman coding Huffman coding Binary file Binary fileBasic comp.: prediction, freq.transform, quantization and entropy encoding
  • 87.   Face, MP4 V1, 2kbps  Body, MP4 Amd1, 5-30 kbps  Skinned Model, MP4 Part 16, 5-30 kbps for a human like skeleton Very low bit-rate
  • 88.   IFS surfaces   VTC   Interpolators  Patches   Synthesized texture   Bone-Based  Subdivision surf.   Procedural texture   Morphing  Wavelet SS   DIBR  Mesh Grid   Point Texture  Solids  SC-3DMC
  • 89.   Defined as a base mesh and a collection of target meshes  Animation obtained by updating the weights of the target meshes  BBA stream updated to include morph data  Usable for any kind of 3D objectLocal and precise control for shape deformation
  • 90.   IFS surfaces   VTC   Interpolators  Patches   Synthesized texture   Bone-Based  Subdivision surf.   Procedural texture   Morphing  Wavelet SS   DIBR   FAMC  Mesh Grid   Point Texture  Solids  SC-3DMC
  • 91.   Cluster the vertices withrespect to their motion  Encode a cluster motion byan affine transform  Encode the residual error atvertex level by traditionalapproach (DCT/W,quantization, entropy encoding)
  • 92.   IFS surfaces   VTC   Interpolators  Patches   Synthesized texture   Bone-Based  Subdivision surf.   Procedural texture   Morphing  Wavelet SS   DIBR   FAMC   Remote and  Mesh Grid   Point Texture programmatic  Solids  SC-3DMC
  • 93. Bifs-Anim  Remote animation BIFS-Anim & BIFS Command Bifs-Command scene  Programmatic animation Script Node - ECMA Script scene part of the scene description - Java Code standardized API for accessing MPEG-J Stream the scene graph sceneComplex application scenario can be built
  • 94.   IFS surfaces   VTC   Interpolators  Patches   Synthesized texture   Bone-Based  Subdivision surf.   Procedural texture   Morphing  Wavelet SS   DIBR   FAMC  Mesh Grid   Remote and   Point Texture programmatic  Solids  SC-3DMC
  • 95. MPEG 3DGHow measuring compression performances?On-line benchmarking platform: www.MyMultimediaWorld.com•  Allows easy integration of proprietary algorithms by using an API in C++•  No need to disclaim the algorithm source code•  Benchmark automatically updated for new content•  Restricts and refines the benchmark by means of easy-to- control parameters.
  • 96. MPEG 3DGHow measuring compression performances?On-line benchmarking : www.MyMultimediaWorld.com 3D compression Web site benchmark Benchmark 2 Benchmark 3 Filter and Presentation Engine Benchmark manager 2 Benchmark manager 1 Indexed MDB 3D Compression benchmark manager API MP4 Extended MP7 Algorithm 1 Algorithm 2 … Algorithm n
  • 97. MPEG 3DGHow measuring compression performances?On-line benchmarking : www.MyMultimediaWorld.com GetNbBitstream n Vertex Buffer Encoder Proprietary Coder library DumpBitSream BitStream (1,n) DumpCompressedVB Proprietary Decoder Decoder library .Compressed_VB (1,n)
  • 98. MPEG 3DGHow measuring compression performances?Benchmarking platform, per-object visualization - object properties (number of vertices/ triangles, number of components and files size, - distortion graph (linear or logarithmic), - compression gain with respect to 3DMC. -  encoding time -  decoding times.
  • 99. MPEG 3DGHow measuring compression performances?Benchmarking platform, global visualization Filters - semantic category, - average distortion, - number of vertices, - number of connected components in the object, - database subset.
  • 100. MPEG 3DGHow measuring compression performances?Benchmarking platform, global visualization
  • 101. MPEG 3D Graphics within the MPEG standards familyCore technologies in MPEG 3D GraphicsCompressing other standards (COLLADA, X3D, …) with MPEGVirtual Worlds Interoperability for avatars with MPEG
  • 102. Geometry compression: -compression ratio 40:1Animation compression:-compression ratio 100:1
  • 103. Any XML Scene RepresentationXML XML (Binary) XMLBinarisation3D GraphicsCompression
  • 104. Layer 1:Any XML Scene RepresentationXML Textual XML representation of theXML scene (Binary) XMLBinarisation3D GraphicsCompression
  • 105. Layer 2:Any XML Scene RepresentationXML Binarized XML layerXML Contains the (Binary) XMLBinarisation unclassified elements of the scene graph3D GraphicsCompression
  • 106. Layer 3:Any XML Scene RepresentationXML Compressed layerXML Contains specific (Binary) XMLBinarisation types of media information3D Graphics (geometry,Compression animation, ...)
  • 107. Result:Any XML Scene RepresentationXML MultiplexedXML layers 2 and 3 (Binary) XMLBinarisation3D GraphicsCompression
  • 108. MPEG-4 P25 Encoder side: possible implementation (informative) Encoder Side MP4 XMT,COLLADA, X3DComp ratio: 40-70:1
  • 109. MPEG-4 P25 Decoder side (normative) Decoder Side MP4 XMT, COLLADA, X3DLossless for the data structure, lossless or lossy for graphics primitives
  • 110. MPEG 3D Graphics within the MPEG standards familyCore technologies in MPEG 3D GraphicsCompressing other standards (COLLADA, X3D, …) with MPEGVirtual Worlds Interoperability for avatars with MPEG
  • 111. Standards for Avatars as visualization support Standard Generic Features Avatar representationCOLLADA 3D objects/ scenes Generic ObjectVRML/ X3D 3D objects /scenes H- Anim Application behaviorMPEG-4 2D/3D objects/ scenes A set of dedicated nodes Application behavior A dedicated compressed stream Compression
  • 112. Standards for Avatars as interaction support Representation Features HumanML human physical description, emotion EmotionML emotion, facial expressions, gestures BML speech, gesture, gaze MPML speech VHML facial and body animation, emotional representation CML character attribute and animation definition
  • 113. Why none of the existing standards is solving the issue of avatar interoperability in Virtual Worlds? –  The Virtual Worlds are proprietary applications and the 3D assets including the avatars have economical value –  The Virtual Worlds and in general 3D applications have specific data format allowing rendering optimizationHowever, while maintaining a strict control for economic and technical reasons, Virtual Worlds allow users to personalize the avatars
  • 114. Avatar Template Attributes that can be modified by the userSpecify the set of Personalization Parameters (PP) that transforms atemplate of an arbitrary Virtual World into the user designed avatar
  • 115. What avatar feature can be personalized?Mainly the appearance Analysis of SecondLife, IMVU, Entropia Universe, SonyPlaystation and HumanML
  • 116. Very heterogeneous set of personalization parameters Entropia Nintendo Universe Wii HumanML Second Life PlayStation
  • 117. “Appearance” element <Appearance> <Body> <BodyHeight value=165/> <BodyFat value=15/> </Body> <Head> <HeadShape value="oval"/> <EggHead value="true"/> </Head> </Appearance>
  • 118. Interoperability at the Animation level <Animation> <Greeting> <Salute>salut</Salute> <Cheer>cheer</Cheer> </Greeting> <Fighting> <shoot>pousse</shoot> <throw>throw</throw> </Fighting> </Animation>
  • 119. Motion retargeting No “Walk” Avatar Avatar animation “Walk” animationTemplate in VW1 Template in VW2 defined in VW2 In VW1
  • 120. “Control” element Body Face Control Control <Control> <BodyFeaturesControl > <UpperBodyBones> <LCalvicle>my_LCalvicle</LCalvicle> Control <RClavicle>my_RCalvicle</RClavicle> </UpperBodyBones> </BodyFeaturesControl> <FaceFeaturesControl> <HeadOutline> <Left X=0.23 Y=1.25 Z=7.26/> <Right X=0.25 Y=1.25 Z=7.21/> </HeadOutline> </FaceFeaturesControl> </Control>
  • 121. MPEG-V (Personalization Parameters) MPEG-V + MPEG-4The Avatar in The Avatar in VW1 VW2 The Avatar in an external player
  • 122. MPEG 3DGCOngoing work
  • 123. MPEG 3DGCMulti-Resolution 3DMC Current 3DG content in VW© Samsung © Second Life Tomorrow’s 3DG content in VW (100 times denser) © OTOY © OTOY
  • 124. MPEG 3DGCReconfigurable Graphics Coding A framework that allows to set up the decoder at run time
  • 125. Thank you! www.MyMultimediaWorld.com www-artemis.it-sudparis.eu marius.preda@it-sudparis.eu
  • 126. MPEG 3DGC Multi-resolution 3DMCProgressive mesh [Hoppe’96] Progressive Forest Split (PFS) [Taubin’98] Patch coloring [Cohen-Or’99]Valence-based decimation approach [Alliez’01] Octree-based compression [Peng’05] Spectral coding [Karni’01]
  • 127. MPEG 3DGC Multi-resolution 3DMC : two proposed methods1. Progressive TFAN Original Interpolated with 1% of the vertices
  • 128. MPEG 3DGC Multi-resolution 3DMC : two proposed methods2. KLT encoder 2 2 2 1 1 1
  • 129. MPEG 3DGC Multi-resolution 3DMC : two proposed methods1. Work on -  document the use case scenarios (object examination, navigation, …) -  database -  comparison with other codecs -  comparison method (PSNR, Resolution) -  platform for testing2. You are invited to contribute