Basics of Mpeg 4 3D Graphics Compression

2,268 views
2,044 views

Published on

A presentation of MPEG-4 (Part 16) tools for video compression

1 Comment
1 Like
Statistics
Notes
  • Open3DGC provides a cross platform C++ implementation (under MIT License) of patent free MPEG tools for 3D graphics compression

    https://github.com/amd/rest3d

    http://kmamou.blogspot.ca/2013/07/open-3d-graphics-compression-open3dgc.html
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total views
2,268
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
77
Comments
1
Likes
1
Embeds 0
No embeds

No notes for slide

Basics of Mpeg 4 3D Graphics Compression

  1. 1. Marius Preda, PhDChairman of MPEG 3DGInstitut MINES TELECOM
  2. 2.   It’s a crazy multimedia world! –  Network is everywhere… but very heterogeneous –  Terminals are “same same”… but different! –  Content must be designed with a priori knowledge of its future use –  Applications are platform-centric instead of user-centric  Fragmented value chain Production Transmission Consumption Capturing HW/SW Playback Networks devices providers devices INTEROPERABILITY NEEDED!!! Authoring IPR Terminal Providers tools holders manufacturers Service End users scenarios
  3. 3. MPEG has several names •  Common: MPEG = Moving Picture Experts Group •  Official: MPEG = ISO/IEC JTC1/SC29/WG11 ISO Over 224 technical committees JTC1 Joint Technical Committee with IEC SC24 SC29Computer graphics and image processing Coding of audio, picture, multimedia and hypermedia information GKS, PHIGS WG1 JPEG CGM, VRML, X3D Coding of still pictures WG11 MPEG Coding of moving pictures and audio
  4. 4. MPEG has several subgroups Video ... and produced several Requirements Audio Systems successful standards 3D Graphics MPEG MPEG-1 MPEG-2 MPEG-4 MPEG-7 MPEG-21 ISO/IEC 11172 ’92 ISO/IEC 13818 ’94 ISO/IEC 14496 ’99 ISO/IEC 15938 ’01 ISO/IEC 21000 ’02 Video & audio Video & audio Multimedia Metadata Terminal (CD-ROM) (DVD & DVB) & interactive (description & network applications of content) specificationMPEG-A MPEG-B MPEG-C MPEG-D MPEG-E MPEG-M MPEG-U MPEG-V
  5. 5. MPEG-4 FeaturesMPEG-4 ObjectsNaturals – still images, audio, 2D/3D videoSynthetic - audio, 2D/3D objects and scenes Compression CompressionMPEG-4 Objectives CompressionProvide technologies for efficientcompression and transmissionMPEG-4 TerminalComposition, at end-user side, of naturaland synthetic objects, into hybrid andinteractive scenes
  6. 6. MPEG-4 FeaturesSystem architecture Interactive Scene Description Scene layer Object descriptor layerMPEG-4stream Media data layer
  7. 7.   Video Compression –  MPEG-4 Visual and MPEG-4 AVC  3D Graphics Compression
  8. 8.   No common representation –  Heterogeneous kind of data •  Different types of geometry, appearance and animation models –  Always easier to specify a new data representation format than learning an existent one  Very different application domains
  9. 9. FAST TRANSPORT TROUGHT THE NETWORK Size is of main Size doesn’t matterSize doesn’t matter importance
  10. 10. FAST TRANSPORT TROUGHT THE NETWORK Size doesn’t matterSize doesn’t matter Size is of main importance
  11. 11. To define a standard format for compressed 3D synthetic content. In other words to be for graphics what MP3 and AAC are for audio, MPEG-2 and MPEG-4 are for video and JPEG is for still images.Additionally "MPEG 3D Graphics" aims at providing mechanisms such as APIs to enable easy integration and development of applications using its standard representation tools.
  12. 12. MPEG 3D Graphics within the MPEG standards familyCore technologies in MPEG 3D GraphicsCompressing other standards (COLLADA, X3D, …) with MPEGVirtual Worlds Interoperability for avatars with MPEG
  13. 13. MPEG 3D Graphics within the MPEG standards familyCore technologies in MPEG 3D GraphicsCompressing other standards (COLLADA, X3D, …) with MPEGVirtual Worlds Interoperability for avatars with MPEG
  14. 14. MPEG-4 3D chronology
  15. 15. MPEG-4 3D chronology
  16. 16. MPEG-4 3D chronology
  17. 17. MPEG 3D Graphics within the MPEG standards familyCore technologies in MPEG 3D GraphicsCompressing other standards (COLLADA, X3D, …) with MPEGVirtual Worlds Interoperability for avatars with MPEG
  18. 18.   IFS surfaces
  19. 19. Approximation of target surface•  Method: tesselation with planar facets•  Quality: first order (linear) ⇒ no smoothness (C0 continuity)Mesh definition: IFS (Indexed Face Set)•  Connectivity: list of faces {…, Pn = {in0, in1, in2, …}, …} ⇒ arbitrary topology (non-manifold, open, higher genus, etc.)•  Geometry: list of vertices {…, Vk = (xk, yk, zk), …}Mesh coding•  Connectivity (lossless): triangle strips, triangle+vertex trees, etc.•  Geometry (lossy): coordinate quantisation + prediction from conn.
  20. 20. LOD concept•  1976: Clark introduced the idea•  Main interest: rendering efficiencyTaxonomy of LOD extraction techniques•  Static vs. dynamic•  Global vs. local•  Progressive vs. hierarchical LODsSuccessful simplification techniques•  1996: Hoppe’s edge collapses•  1997: Garland’s quadrics ⇒ qslimProgressive 3D mesh coding•  1996: Hoppe’s PM (Progressive Mesh)•  1998: IBM’s PFS (Progressive Forest Split)
  21. 21. Triangles: 69451 vs. 29519
  22. 22. Triangles: 69451 vs. 9575
  23. 23. Triangles: 69451 vs. 1627
  24. 24. Triangles: 69451 vs. 238
  25. 25. MPEG-4 (1999): IFSs•  Based on VRML97•  Arbitrary topology meshes•  “Properties” (normals, colours and textures)MPEG-4 Amd.1 (2000): 3DMC (3D Mesh Coding)•  40-50:1 compression of IFSs by IBM’s TS (Topological Surgery)•  Incremental transmission and rendering•  Progressive coding by IBM’s PFS•  Error resilience by SAIT
  26. 26. MPEG-4 (1999): IFSs•  Based on VRML97•  Arbitrary topology meshes•  “Properties” (normals, colours and textures)MPEG-4 Amd.1 (2000): 3DMC (3D Mesh Coding)•  40-50:1 compression of IFSs by IBM’s TS (Topological Surgery)•  Incremental transmission and rendering•  Progressive coding by IBM’s PFS•  Error resilience by SAIT
  27. 27.   IFS surfaces
  28. 28.   IFS surfaces  Patches
  29. 29. Approximation of target surface•  Method: tesselation with predefined curved patches•  Quality: higher order (polynomic/rational) ⇒ Cn continuityMesh definition•  Connectivity: regular grid of quads. or triangles ⇒ planar topology•  Geometry: list of control points {…, Pk = (xk, yk, zk), …}Mesh coding•  Connectivity (lossless): implicit•  Geometry (lossy): coordinate quantisation + prediction from conn.
  30. 30. Tensor product of cubic Bézier curves Single patch (4x4 control points) vs. two patches (4x7 control points) Compression  (3547 polygons; 1215 vertices) vs. (86 patches; 212 control points)
  31. 31. MPEG-4 Part 16 (2003): NURBS•  Based on VRML97 Amd., originally proposed by blaxxun•  Support for NURBS curves and patches   Specific nodes for Bézier’s curves and patches (for increased efficiency)•  Support for free-form deformations
  32. 32.   IFS surfaces  Patches  Subdivision surfaces
  33. 33. SS = limit of recursive refinement of base control mesh NB: refinement affects both Geometry smoothing achieved withconnectivity (of abstract graph) and stencils particular to each scheme geometry (of 3D mapping) -1/16 -1/16 ½ -1/16 9/16 9/16 -1/16 1/8 1/8 Border/sharp vs. z interior edge stencils ½ of “butterfly” scheme x y -1/16 -1/16 SS inherently define hierarchically nested LODs
  34. 34. Approximation of target surface•  Method: tesselation with curved patches•  Quality: higher order ⇒ Cn continuityMesh definition•  Connectivity: list of triangles/quads., e.g., {…, Tn = {in0, in1, in2}, …} ⇒ arbitrary (manifold) topology•  Geometry: list of control points {…, Pk = (xk, yk, zk), …}Mesh coding•  Connectivity (lossless): as for polygonal (manifold) mesh•  Geometry (lossy): as for polygonal (manifold) mesh
  35. 35. Polygons+ Are the simplest approach (linear approximation)+ Can resolve fine details and handle arbitrary topologies– Lead to unstructured, huge meshesPatches+ Are a more powerful approach (higher order approximation)+ Are convenient for coarse and smooth models– Need cumbersome trimming and stitching mechanismsSSs+ Connect and unify the two extremes above++ Provide multi-resolution handles for hierarchical coding/editing
  36. 36. Catmull-Clark’s (1978): quadrilateral, primal, approximating, C2 Loop’s (1987): triangular, primal, approximating, C2Dyn++’s “butterfly” (1990): triangular, primal, interpolating, C1
  37. 37.   IFS surfaces  Patches  Subdivision surfaces  Wavelet subdivision surfaces
  38. 38. Target surfaceBasemesh Subdivide Add details
  39. 39. Target surfaceBasemesh Subdivide Add detailsPrice (requirements)•  Base mesh extraction•  Subdivision scheme = predictor•  Details (3D vectors) = prediction errors ⇒ remeshingPrize (advantages)•  Predictive coding ⇒ immediate (if smooth target mesh)
  40. 40. Spatial partitioning Adding details in appearing parts Removing details in disappearing partsZT ZT ZT … ZT
  41. 41. by courtesy of FranceTelecom
  42. 42. MPEG-4 Part 16 (2003): “plain” + wavelet SSs•  “Plain” SSs for mesh smoothing   Considered schemes: Catmull-Clark, [extended] Loop, butterfly   No details are added but…   normal control achievable through edge/vertex tagging of initial control mesh•  Wavelet/detailed SSs for surface approximation   Possibly tagged base mesh   Details are added after each subdivision step, which are…   wavelet-transformed according to one of several possible schemes   Most suited for multi-resolution editing/animation   Most suited for view-dependent transmission
  43. 43.   IFS surfaces  Patches  Subdivision surfaces  Wavelet subdivision surfaces  Mesh Grid
  44. 44. CW (Connectivity-Wireframe)RG (Reference-Grid)
  45. 45. Vertex offset is a relative value Update vertex position when grid is deformed or animated
  46. 46.   IFS surfaces  Patches  Subdivision surfaces  Wavelet subdivision surfaces  Mesh Grid  Solids
  47. 47. Solid primitivesSolid models: the “arithmetic of forms”
  48. 48. Implicit equation:Quadrics (2nd order):Quartics (4th order):
  49. 49. F1 * 0 1 2 0 0 0 0 F0 1 0 1 2 2 0 2 4A cube = multiplication of 3 degenerated quadrics Multiplication of two forms Examples of solid operations
  50. 50. Architecture MechanicsBiotechnology Virtual models
  51. 51. Exact geometryCompactness 21 Kb 37 Kb 407 Kb 1.1 Mb
  52. 52.   IFS surfaces  Patches  Subdivision surf.  Wavelet SS  Mesh Grid  Solids  SC-3DMC
  53. 53. MPEG 3DGC Scalable Complexity 3D Mesh Compression •  Not all 3DG applications have the same needs in compression •  Not all the 3DG applications can afford spending extra CPU/GPU for compression TFAN SVA SC3DMC QBCR Same Quantization and Binarization blocksTowards the continuum model: enlarge the application domain where MPEG-4 3DG can be used
  54. 54. MPEG 3DGCScalable Complexity 3D Mesh Compression x, y, z (floats) + normals (floats) x, y, z (floats) + colors (floats) x, y, z (floats) + … (floats) i, j, k (integers) Attributes i, j, k (integers) i, j, k (integers) Connectivity
  55. 55. SC-3DMC General SchemaConnectivity TFAN BP CABAC Delta (lossless) SVA FLB AC BAC Connectivity Prediction Binarization Entropy i, j, k Analysis Encoding Attributes Prediction Entropy Binarization (lossy) Encoding Parallelogram Delta BP CABAC x, y, z Quantization Barycentre FLB AC 5 different paths with different performances
  56. 56. SC-3DMCConnectivity Analysis: EmptyDo Nothing 3 1 4 2 123234
  57. 57. SC-3DMCConnectivity Analysis: SVA Previous Face4 modes for encoding how consecutive Current Face faces share vertices Shared Vertex 1 5 3 1, 2, 3, 1, 2, 3, 1 4 3 2, 3, 4 3, 4, 5 12304 2 2 4 123245 Mode 0 Mode 2 1 4 1 11, 2, 3, 1, 2, 3,4, 5, 6 1, 2, 3 2 3 2 3 2 3 5 6 OR 1231 5 Mode 1 Mode 3 1233
  58. 58. SC-3DMCConnectivity Analysis: TFANSplit the mesh in set of triangle fans and encode each fan 1675 62937 2984 Transformed in local indices
  59. 59. SC-3DMCWhat we measure?Encoding performances but also complexity for decoder (and encoder) TFANPerformances SVA-AC 3DMC SVA-BAC SVA-BP NCA BIFS Complexity
  60. 60.   IFS surfaces  Patches  Subdivision surf.  Wavelet SS  Mesh Grid  Solids  SC-3DMC
  61. 61. •  Visual Texture Coding Generic compression tool•  Synthesized Textures Very Compact representation for•  Procedural Textures specific applications•  Depth Image-based Representation Less calculation intensive
  62. 62. •  MPEG-4 3DMC•  MPEG-4 MeshGrid•  MPEG-4 Subdivision Surfaces3D Mesh •  MPEG-4 VTC •  JPEG 2000 3D object 2D Texture
  63. 63. Zero-TreeWT 3D mipmaps
  64. 64. Multiple In Place mapping = multi-resolution textures Without mipmapping With mipmapping
  65. 65. Zero-TreeWTComputational Graceful Degradation: Quality => processing power
  66. 66. Region of Interest with resolution/quality selection Visual importance depends on viewing angle Low Resolution High Resolution
  67. 67. Packet selection by using error-resilience markers
  68. 68.   IFS surfaces   VTC  Patches   Synthesized texture  Subdivision surf.  Wavelet SS  Mesh Grid  Solids  SC-3DMC
  69. 69. Vector Graphics Primitives Color profile Line Color Profile ( LC ) Area Color Point ( A C ) Line ( LN ), bounded by 2 Terminal Points ( TP ) The image cannot be displayed. Color Patch ( PA ) Line Segment ( LS ), bounded by 2 Line Points ( LP ) Sub -Texture ( ST ) Line marked as control line (Skeleton) The image cannot be displayed.
  70. 70. 65x96 pixels10 seconds animation1.35 kB
  71. 71.   IFS surfaces   VTC  Patches   Synthesized texture  Subdivision surf.   Procedural texture  Wavelet SS  Mesh Grid  Solids  SC-3DMC
  72. 72. DEF Fabric ProceduralTexture { type 2 width 256 height 256 cellWidth 4 cellHeight 4 roughness 1 distortion 0.05 seed 114300 color [ 0.898 0.89418 0.95294, 0.34118 0.29418 0.70196, 0 0 0, 0 0 0 ] aWarpmap [ 0 0, 0.03 1, 0.88 1, 1 0 ] bWarpmap [ 0 0, 0.48 1, 1 0 ] aWeights [ 0, 0, 0, 0, 0, 0, 0.56, 0, 0, 0, 0, 0, 0, 0, 0.20, 0.24 ] bWeights [ 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0 ]} DEF Marble ProceduralTexture { width 256 height 256 roughness 1 seed 22209 color [ 0.8 0.7098 0.6902, 0.95686 0.8902 0.87451, 0.87451 0.37255 0.23529, 0.95686 0.8902 0.87451 ] aWarpmap [ 0 1, 0.33 0, 1 1 ] bWarpmap [ 0 0, 0.55 0, 0.6 1, 0.65 0, 1 0 ] bWeights [ 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0 ] }
  73. 73.   IFS surfaces   VTC  Patches   Synthesized texture  Subdivision surf.   Procedural texture  Wavelet SS   DIBR  Mesh Grid  Solids  SC-3DMC
  74. 74. Reconstruct 3D representation from projections Depth image Projection
  75. 75.   IFS surfaces   VTC  Patches   Synthesized texture  Subdivision surf.   Procedural texture  Wavelet SS   DIBR  Mesh Grid   Point Texture  Solids  SC-3DMC
  76. 76.   IFS surfaces   VTC   Interpolators  Patches   Synthesized texture  Subdivision surf.   Procedural texture  Wavelet SS   DIBR  Mesh Grid   Point Texture  Solids  SC-3DMC
  77. 77. Piece-wise linear functions definedby a set of pairs time-value Standardized by MPEG-4 for: - position - orientation - scale - coordinate in IFS - normals in IFSStraightforward animation based on key-frames
  78. 78. Large amount of data for high quality, smooth animation
  79. 79. Re-sampling, sub-sampling: twomethods supported by MPEG-4 Path preserving mode Key preserving modeThree schemes supported byMPEG-4 Orientation Interpolators Coordinate Interpolators Position InterpolatorsExploit temporal redundancy for the most common interpolators
  80. 80. Re-sampling, sub-sampling: twomethods supported by MPEG-4 Key value Key value Path preserving mode Key(time) Key(time) Key preserving mode N key values N-1 key valuesThree schemes supported byMPEG-4 Key value Key value Orientation Interpolators Coordinate Interpolators Key(time) Key(time) Position Interpolators N-2 key values N-3 key valuesSub-sampling or re-sampling based on minimal distortion
  81. 81. A dedicated elementary stream for IC, multiplexed into an AFX stream
  82. 82.   IFS surfaces   VTC   Interpolators  Patches   Synthesized texture   Bone-Based  Subdivision surf.   Procedural texture  Wavelet SS   DIBR  Mesh Grid   Point Texture  Solids  SC-3DMC
  83. 83.   Face, MPEG-4 V1, 1999  Body, MPEG-4 Amd1, 2000  Skinned Model, MPEG-4 Part 16, 2003 Two frameworks: human-like (FBA) and generic skeleton (BBA)
  84. 84.   Geometry: Seamless mesh: shapes sharing the same vertices list  Texture: Image Mapping on vertices sub-set  Hierarchy: Skeleton layer Muscle layer Seamless mesh affected by a hierarchical skeleton
  85. 85.   1D controllers: bone & muscle  for each bone and each muscle - a list of affected vertices - a measure of affectednessare providedRight balance between control parameters and influence volume
  86. 86. Uncompressed Uncompressed Segment #n Frame #n BAP/FAP/BBABAP/FAP/BBA Frame P DCT Prediction DC Coeff. AC Coeff. Frame I Segment P Segment Prediction I Quantization Quantization DC Q DC Q AC Q Arithmetic Arithmetic coding coding Huffman coding Huffman coding Binary file Binary fileBasic comp.: prediction, freq.transform, quantization and entropy encoding
  87. 87.   Face, MP4 V1, 2kbps  Body, MP4 Amd1, 5-30 kbps  Skinned Model, MP4 Part 16, 5-30 kbps for a human like skeleton Very low bit-rate
  88. 88.   IFS surfaces   VTC   Interpolators  Patches   Synthesized texture   Bone-Based  Subdivision surf.   Procedural texture   Morphing  Wavelet SS   DIBR  Mesh Grid   Point Texture  Solids  SC-3DMC
  89. 89.   Defined as a base mesh and a collection of target meshes  Animation obtained by updating the weights of the target meshes  BBA stream updated to include morph data  Usable for any kind of 3D objectLocal and precise control for shape deformation
  90. 90.   IFS surfaces   VTC   Interpolators  Patches   Synthesized texture   Bone-Based  Subdivision surf.   Procedural texture   Morphing  Wavelet SS   DIBR   FAMC  Mesh Grid   Point Texture  Solids  SC-3DMC
  91. 91.   Cluster the vertices withrespect to their motion  Encode a cluster motion byan affine transform  Encode the residual error atvertex level by traditionalapproach (DCT/W,quantization, entropy encoding)
  92. 92.   IFS surfaces   VTC   Interpolators  Patches   Synthesized texture   Bone-Based  Subdivision surf.   Procedural texture   Morphing  Wavelet SS   DIBR   FAMC   Remote and  Mesh Grid   Point Texture programmatic  Solids  SC-3DMC
  93. 93. Bifs-Anim  Remote animation BIFS-Anim & BIFS Command Bifs-Command scene  Programmatic animation Script Node - ECMA Script scene part of the scene description - Java Code standardized API for accessing MPEG-J Stream the scene graph sceneComplex application scenario can be built
  94. 94.   IFS surfaces   VTC   Interpolators  Patches   Synthesized texture   Bone-Based  Subdivision surf.   Procedural texture   Morphing  Wavelet SS   DIBR   FAMC  Mesh Grid   Remote and   Point Texture programmatic  Solids  SC-3DMC
  95. 95. MPEG 3DGHow measuring compression performances?On-line benchmarking platform: www.MyMultimediaWorld.com•  Allows easy integration of proprietary algorithms by using an API in C++•  No need to disclaim the algorithm source code•  Benchmark automatically updated for new content•  Restricts and refines the benchmark by means of easy-to- control parameters.
  96. 96. MPEG 3DGHow measuring compression performances?On-line benchmarking : www.MyMultimediaWorld.com 3D compression Web site benchmark Benchmark 2 Benchmark 3 Filter and Presentation Engine Benchmark manager 2 Benchmark manager 1 Indexed MDB 3D Compression benchmark manager API MP4 Extended MP7 Algorithm 1 Algorithm 2 … Algorithm n
  97. 97. MPEG 3DGHow measuring compression performances?On-line benchmarking : www.MyMultimediaWorld.com GetNbBitstream n Vertex Buffer Encoder Proprietary Coder library DumpBitSream BitStream (1,n) DumpCompressedVB Proprietary Decoder Decoder library .Compressed_VB (1,n)
  98. 98. MPEG 3DGHow measuring compression performances?Benchmarking platform, per-object visualization - object properties (number of vertices/ triangles, number of components and files size, - distortion graph (linear or logarithmic), - compression gain with respect to 3DMC. -  encoding time -  decoding times.
  99. 99. MPEG 3DGHow measuring compression performances?Benchmarking platform, global visualization Filters - semantic category, - average distortion, - number of vertices, - number of connected components in the object, - database subset.
  100. 100. MPEG 3DGHow measuring compression performances?Benchmarking platform, global visualization
  101. 101. MPEG 3D Graphics within the MPEG standards familyCore technologies in MPEG 3D GraphicsCompressing other standards (COLLADA, X3D, …) with MPEGVirtual Worlds Interoperability for avatars with MPEG
  102. 102. Geometry compression: -compression ratio 40:1Animation compression:-compression ratio 100:1
  103. 103. Any XML Scene RepresentationXML XML (Binary) XMLBinarisation3D GraphicsCompression
  104. 104. Layer 1:Any XML Scene RepresentationXML Textual XML representation of theXML scene (Binary) XMLBinarisation3D GraphicsCompression
  105. 105. Layer 2:Any XML Scene RepresentationXML Binarized XML layerXML Contains the (Binary) XMLBinarisation unclassified elements of the scene graph3D GraphicsCompression
  106. 106. Layer 3:Any XML Scene RepresentationXML Compressed layerXML Contains specific (Binary) XMLBinarisation types of media information3D Graphics (geometry,Compression animation, ...)
  107. 107. Result:Any XML Scene RepresentationXML MultiplexedXML layers 2 and 3 (Binary) XMLBinarisation3D GraphicsCompression
  108. 108. MPEG-4 P25 Encoder side: possible implementation (informative) Encoder Side MP4 XMT,COLLADA, X3DComp ratio: 40-70:1
  109. 109. MPEG-4 P25 Decoder side (normative) Decoder Side MP4 XMT, COLLADA, X3DLossless for the data structure, lossless or lossy for graphics primitives
  110. 110. MPEG 3D Graphics within the MPEG standards familyCore technologies in MPEG 3D GraphicsCompressing other standards (COLLADA, X3D, …) with MPEGVirtual Worlds Interoperability for avatars with MPEG
  111. 111. Standards for Avatars as visualization support Standard Generic Features Avatar representationCOLLADA 3D objects/ scenes Generic ObjectVRML/ X3D 3D objects /scenes H- Anim Application behaviorMPEG-4 2D/3D objects/ scenes A set of dedicated nodes Application behavior A dedicated compressed stream Compression
  112. 112. Standards for Avatars as interaction support Representation Features HumanML human physical description, emotion EmotionML emotion, facial expressions, gestures BML speech, gesture, gaze MPML speech VHML facial and body animation, emotional representation CML character attribute and animation definition
  113. 113. Why none of the existing standards is solving the issue of avatar interoperability in Virtual Worlds? –  The Virtual Worlds are proprietary applications and the 3D assets including the avatars have economical value –  The Virtual Worlds and in general 3D applications have specific data format allowing rendering optimizationHowever, while maintaining a strict control for economic and technical reasons, Virtual Worlds allow users to personalize the avatars
  114. 114. Avatar Template Attributes that can be modified by the userSpecify the set of Personalization Parameters (PP) that transforms atemplate of an arbitrary Virtual World into the user designed avatar
  115. 115. What avatar feature can be personalized?Mainly the appearance Analysis of SecondLife, IMVU, Entropia Universe, SonyPlaystation and HumanML
  116. 116. Very heterogeneous set of personalization parameters Entropia Nintendo Universe Wii HumanML Second Life PlayStation
  117. 117. “Appearance” element <Appearance> <Body> <BodyHeight value=165/> <BodyFat value=15/> </Body> <Head> <HeadShape value="oval"/> <EggHead value="true"/> </Head> </Appearance>
  118. 118. Interoperability at the Animation level <Animation> <Greeting> <Salute>salut</Salute> <Cheer>cheer</Cheer> </Greeting> <Fighting> <shoot>pousse</shoot> <throw>throw</throw> </Fighting> </Animation>
  119. 119. Motion retargeting No “Walk” Avatar Avatar animation “Walk” animationTemplate in VW1 Template in VW2 defined in VW2 In VW1
  120. 120. “Control” element Body Face Control Control <Control> <BodyFeaturesControl > <UpperBodyBones> <LCalvicle>my_LCalvicle</LCalvicle> Control <RClavicle>my_RCalvicle</RClavicle> </UpperBodyBones> </BodyFeaturesControl> <FaceFeaturesControl> <HeadOutline> <Left X=0.23 Y=1.25 Z=7.26/> <Right X=0.25 Y=1.25 Z=7.21/> </HeadOutline> </FaceFeaturesControl> </Control>
  121. 121. MPEG-V (Personalization Parameters) MPEG-V + MPEG-4The Avatar in The Avatar in VW1 VW2 The Avatar in an external player
  122. 122. MPEG 3DGCOngoing work
  123. 123. MPEG 3DGCMulti-Resolution 3DMC Current 3DG content in VW© Samsung © Second Life Tomorrow’s 3DG content in VW (100 times denser) © OTOY © OTOY
  124. 124. MPEG 3DGCReconfigurable Graphics Coding A framework that allows to set up the decoder at run time
  125. 125. Thank you! www.MyMultimediaWorld.com www-artemis.it-sudparis.eu marius.preda@it-sudparis.eu
  126. 126. MPEG 3DGC Multi-resolution 3DMCProgressive mesh [Hoppe’96] Progressive Forest Split (PFS) [Taubin’98] Patch coloring [Cohen-Or’99]Valence-based decimation approach [Alliez’01] Octree-based compression [Peng’05] Spectral coding [Karni’01]
  127. 127. MPEG 3DGC Multi-resolution 3DMC : two proposed methods1. Progressive TFAN Original Interpolated with 1% of the vertices
  128. 128. MPEG 3DGC Multi-resolution 3DMC : two proposed methods2. KLT encoder 2 2 2 1 1 1
  129. 129. MPEG 3DGC Multi-resolution 3DMC : two proposed methods1. Work on -  document the use case scenarios (object examination, navigation, …) -  database -  comparison with other codecs -  comparison method (PSNR, Resolution) -  platform for testing2. You are invited to contribute

×