Downloaded 24 times
![Graphics Tool Library
START
Loop until the following
is available:
- codeBytes
[INPUT]:
codeBytes_i
[PROCESS]:
length
value
n,s,x,y
Loop until the following
is available:
- codeBuffer
[INPUT]:
codeBuffer_i
codeBytes < 4
else
[INPUT]:
useDecoderTable_i
Loop until the following
is available:
- useDecoderTable
useDecoderTable
=
1
[INPUT]:
decoderTableSize_i
distributionSize_i
Loop until the following
are available:
- decoderTableSize
- distributionSize
[INPUT]:
decoderTable_i
[decoderTableSize]
distribution_i [distributionSize]
lastSymbol_i
Loop until the following
are available:
- decoderTable [ decoderTableSize ]
- distribution [ distributionSize ]
- lastSymbol
[PROCESS]:
s
n
n > s + 1 [PROCESS]:
m
[PROCESS]:
n
distribution[m] > dv
else
[PROCESS]:
s
[PROCESS]:
x
else
[PROCESS]:
y
s != lastSymbol
else
[PROCESS]:
x
s
length
m
[PROCESS]:
z
[PROCESS]:
n, y
[PROCESS]:
s,x
m != s
[PROCESS]:
m
z > value
else
[PROCESS]:
value
length
else
else
[PROCESS]:
value
ac_pointer
length
length < AC_MinLength
length < AC_MinLength
else
[PROCESS]:
symbolCount [ s ]
symbolsUntilUpdate
[EMBED]:
update
[OUTPUT]:
dataOut_o = s
symbolsUntilUpdate = 0 else
[INPUT]:
tableShift_i
Loop until the following
is available:
- tableShift
[INPUT]:
dataSymbols_i
Loop until the following is available:
- dataSymbols
[INPUT]:
reset_i
Loop until the following
is available:
- reset
[PROCESS]:
bufferSize
newBuffer
codeBuffer
ac_pointer
dataModelFlag [] = 0
reset
=
1
else
[INPUT]:
dataModelIndex_i
Loop until the following
is available:
- dataModelIndex
[PROCESS]:
dv
t
dataModelFlag
[
dataModelIndex
]
=
0
dataModelFlag
[
dataModelIndex
]
=
0
else
else
dataModelFlag
[
dataModelIndex
]
=
0
else
[PROCESS]:
dataModelFlag
[dataModelIndex ]
= 1
ArithmeticDecode_AdaptiveData
dataSymbols_i
lastSymbol_i
tableBits_i
tableShift_i
distribution_i
useDecoderTable_i
decoderTable_i
tableSize_i
symbolCount_i
dataModelIndex_i
codeBytes_i
codeBuffer_i
[ AC_MaxLength ]
[ DM__LengthShift ]
[ N_DATA_MODELS ]
dataOut_o
FU Name
Algo_ParserSC3DMC
Algo_InverseQuantization1D
Algo_InverseQuantizationND
Algo_InversePrediction1D
Algo_InversePredictionND
Algo_ArithmeticCodec_StaticBit
Algo_ArithmeticCodec_AdaptiveBit
Algo_AdaptiveData_Model_GenerateAlphabet
Algo_ArithmeticDecode_AdaptiveData
Algo_ReOrderConnec1D
Algo_ReOrderConnecND
Algo_ReOrderTab1D
Algo_ReOrderTabND
Algo_ReOrderPerm1D
Algo_ReOrderPermND
Algo_InverseBinarization
Algo_Provider_1D
Algo_Provider_2D
Algo_GeometryDecoder_TFAN
Algo_Decode_Symbol
Algo_Decode_I_Frame_Quantized
Algo_Decode_P_Frame
Algo_Huffman_Decode
Algo_DCT_Dequant
Algo_IDCT_Trans
Algo_InterpolateCoord
Algo_QuatNorm
Algo_QuatConj
Algo_QuatScalar_NotNorm
Reconfigurable Graphics Coding](https://image.slidesharecdn.com/mpeg3dgraphics3dforum2012-121008235257-phpapp01/75/3DgraphicsAndAR-19-2048.jpg)
Uploaded byMarius Preda PhD
3DgraphicsAndAR
The document discusses MPEG's work on 3D graphics standards. It provides an overview of MPEG's past and current 3D graphics standards including MPEG-4 Part 16 and Part 25. It outlines the various 3D graphics tools that have been standardized, such as 3D mesh compression and animation tools. It also discusses MPEG's work integrating real and virtual worlds through standards like MPEG-V for sensors and actuators and MPEG's work on 3D video and augmented reality. Finally, it provides an example of a reconfigurable graphics coding network using functional units from the MPEG 3D graphics toolbox.
3DgraphicsAndAR
- 1. MPEG 3D Graphics Recentachievements and future roadmap GLOBAL 3D Technology Forum, October 9-12, KINTEX, Korea Marius Preda, MPEG 3DG Chair Institut Mines TELECOM http://www.slideshare.net/MariusPreda/3DgraphicsAndAR
- 2. 3D Graphics World No common representation – Heterogeneous kind of data • Different types of geometry, appearance and animation models – Always easier to specify a new data representation format than learning an existent one Very different application domains – Different requirements and use cases Towards a standard representation: a double problem
- 3. 3D Graphics Chain: Authoring and Publishing Very heterogeneous production chain: -A 3D Asset is processed by several authoring tools, each one with dedicated functionalities -Each authoring tool has its own format -Artists and creators are investing time in learning how the tool works Authoring
- 4. In BOX solutions -Games are containing all the data on DVD - Rendering engine friendly data format ONLINE solutions - ad hoc format, usually zipped version of an authoring format 3D Graphics Chain : Authoring and Publishing Publishing
- 5. FAST TRANSPORT TROUGHT THE NETWORK Size doesn’tmatter Size is of main importance Authoring Size doesn’t matter Transmission Publishing 3D Graphics Chain : Authoring and Publishing Gap between the authoring and publishing formats
- 6. FAST TRANSPORT TROUGHT THE NETWORK Size doesn’tmatter Size is of main importance Authoring Size doesn’t matter Transmission Encoder Decoder Publishing MPEG-4 3DG 3D Graphics Chain : Authoring and Publishing Solving the gap: encoding layer
- 7. To define astandard format for compressed 3D synthetic content. In other words to be for graphics what MP3 and AAC are for audio, MPEG-2 and MPEG-4 are for video and JPEG is for still images. Additionally "MPEG 3D Graphics" aims at providing mechanisms such as APIs to enable easy integration and development of applications using its standard representation tools. 3D Graphics Chain : Authoring and Publishing MPEG 3DG Vision
- 8. Topics of theday Ongoing projects: integrating real and virtual MPEG offer in the 3D Graphics field What is MPEG? MPEG 3D Graphics
- 9. Topics of theday MPEG offer in the 3D Graphics field What is MPEG? MPEG 3D Graphics Ongoing projects: integrating real and virtual
- 10. What is MPEG? Coding/compressionof elementary media: – Audio (MPEG-1, 2 and 4) – Video (MPEG-1, 2 and 4) – 2D/3D graphics (MPEG-4) Storage and Transport – MPEG-2 Transport – File Format (MPEG-4) – Dynamic Adaptive Streaming over HTTP (DASH) Hybrid (natural & synthetic) scene description, user interaction (MPEG-4) Metadata (MPEG-7) Media management and protection (MPEG-21) Sensors and actuators, virtual worlds (MPEG-V) Advanced User interaction (MPEG-U) Media-oriented middleware (MPEG-M) More ISO/IEC standards under development for – 3D Video, 3D Audio – Coding and Delivery in Heterogeneous Environments – … A suite of ~130 ISO/IEC standards
- 11. A standardization activitycontinuing for 24 years – Supported by several hundreds companies/organisations from ~25 countries – ~500 experts participating in quarterly meetings – More than 2300 active contributors – Many thousands experts working in companies A proven manner to organize the work to deliver useful and used standards – Developing standards by integrating individual technologies – Well defined procedures – Subgroups with clear objectives – Ad hoc groups continuing coordinated work between meetings MPEG standards are widely referenced by industry – 3GPP, ARIB, ATSC, DVB, DVD-Forum, BDA, EITSI, SCTE, TIA, DLNA, DECE, OIPF… Billions of software and hardware devices built on MPEG technologies – MP3 players, cameras, mobile handsets, PCs, DVD/Blue-Ray players, STBs, TVs, … What is MPEG? Involvement, approach, deployment
- 12. Topics of theday MPEG offer in the 3D Graphics field What is MPEG? MPEG 3D Graphics Ongoing projects: integrating real and virtual
- 13. MPEG technologies relatedto 3D Graphics MPEG-1/2 (AV content) 1992/4 VRML 1997 • Part 11 - BIFS: -Binarisation of VRML -Extensions for streaming -Extensions for server command -Extensions for 2D graphics - Real time augmentation with audio & video • Part 2 - Visual: - 3D Mesh compression - Face animation 1998 • Part 2 – Visual - Body animation 1999 MPEG-4 v.1 MPEG-4 v.2 First form of broadcast signal augmentation
- 14. MPEG technologies relatedto 3D Graphics MPEG-4 2003 •AFX 2nd Edition: - Animation by morphing - Multi-texturing 2005 • AFX 3rd Edition - WSS for terrain and cities - Frame based animation 2007 MPEG-4 MPEG-4 • Part 16 - AFX: - A rich set of 3D graphics tools - Compression of geometry, appearance, animation • AFX 4th Edition - Scalable complexity mesh coding 2011 MPEG-4 A rich set of 3D Graphics representation and compression tools
- 15. MPEG technologies relatedto 3D Graphics MPEG-4 2003 •AFX 2nd Edition: - Animation by morphing - Multi-texturing 2005 • AFX 3rd Edition - WSS for terrain and cities - Frame based animation 2007 MPEG-4 MPEG-4 • Part 16 - AFX: - A rich set of 3D graphics tools - Compression of geometry, appearance, animation • AFX 4th Edition - Scalable complexity mesh coding 2011 MPEG-4 2009 • Part 25 - Compression of third-party XML (X3D, COLLADA) MPEG-4 2004 • Part 16 - X3D Interactive Profile MPEG-4
- 16. MPEG 3DG Tools CompressionTool Type 3D Mesh Compression – 3DMC Geometry Scalable Complexity 3DMesh Coding – SC3DMC Geometry Wavelet Subdivision Surface – WSS Geometry Coordinate, Orientation and Position Interpolator – IC Animation Bone-based Animation – BBA Animation Frame-based Animated Mesh Compression - FAMC Animation Octree Compression for Depth Image-based Representation Appearance Point Texture Appearance FAMC IC WSS 3DMC BBA
- 17. MPEG 3DG Tools CompressionTool Type 3D Mesh Compression – 3DMC Geometry Scalable Complexity 3DMesh Coding – SC3DMC Geometry Wavelet Subdivision Surface – WSS Geometry Coordinate, Orientation and Position Interpolator – IC Animation Bone-based Animation – BBA Animation Frame-based Animated Mesh Compression - FAMC Animation Octree Compression for Depth Image-based Representation Appearance Point Texture Appearance FAMC IC WSS 3DMC BBA
- 18. • Establish astandardized set of Functional Units (FUs) • Combine the FUs in a consistent decoder by describing it and transmit the description together with the encoded data Reconfigurable Graphics Coding
- 19. Graphics Tool Library START Loopuntil the following is available: - codeBytes [INPUT]: codeBytes_i [PROCESS]: length value n,s,x,y Loop until the following is available: - codeBuffer [INPUT]: codeBuffer_i codeBytes < 4 else [INPUT]: useDecoderTable_i Loop until the following is available: - useDecoderTable useDecoderTable = 1 [INPUT]: decoderTableSize_i distributionSize_i Loop until the following are available: - decoderTableSize - distributionSize [INPUT]: decoderTable_i [decoderTableSize] distribution_i [distributionSize] lastSymbol_i Loop until the following are available: - decoderTable [ decoderTableSize ] - distribution [ distributionSize ] - lastSymbol [PROCESS]: s n n > s + 1 [PROCESS]: m [PROCESS]: n distribution[m] > dv else [PROCESS]: s [PROCESS]: x else [PROCESS]: y s != lastSymbol else [PROCESS]: x s length m [PROCESS]: z [PROCESS]: n, y [PROCESS]: s,x m != s [PROCESS]: m z > value else [PROCESS]: value length else else [PROCESS]: value ac_pointer length length < AC_MinLength length < AC_MinLength else [PROCESS]: symbolCount [ s ] symbolsUntilUpdate [EMBED]: update [OUTPUT]: dataOut_o = s symbolsUntilUpdate = 0 else [INPUT]: tableShift_i Loop until the following is available: - tableShift [INPUT]: dataSymbols_i Loop until the following is available: - dataSymbols [INPUT]: reset_i Loop until the following is available: - reset [PROCESS]: bufferSize newBuffer codeBuffer ac_pointer dataModelFlag [] = 0 reset = 1 else [INPUT]: dataModelIndex_i Loop until the following is available: - dataModelIndex [PROCESS]: dv t dataModelFlag [ dataModelIndex ] = 0 dataModelFlag [ dataModelIndex ] = 0 else else dataModelFlag [ dataModelIndex ] = 0 else [PROCESS]: dataModelFlag [dataModelIndex ] = 1 ArithmeticDecode_AdaptiveData dataSymbols_i lastSymbol_i tableBits_i tableShift_i distribution_i useDecoderTable_i decoderTable_i tableSize_i symbolCount_i dataModelIndex_i codeBytes_i codeBuffer_i [ AC_MaxLength ] [ DM__LengthShift ] [ N_DATA_MODELS ] dataOut_o FU Name Algo_ParserSC3DMC Algo_InverseQuantization1D Algo_InverseQuantizationND Algo_InversePrediction1D Algo_InversePredictionND Algo_ArithmeticCodec_StaticBit Algo_ArithmeticCodec_AdaptiveBit Algo_AdaptiveData_Model_GenerateAlphabet Algo_ArithmeticDecode_AdaptiveData Algo_ReOrderConnec1D Algo_ReOrderConnecND Algo_ReOrderTab1D Algo_ReOrderTabND Algo_ReOrderPerm1D Algo_ReOrderPermND Algo_InverseBinarization Algo_Provider_1D Algo_Provider_2D Algo_GeometryDecoder_TFAN Algo_Decode_Symbol Algo_Decode_I_Frame_Quantized Algo_Decode_P_Frame Algo_Huffman_Decode Algo_DCT_Dequant Algo_IDCT_Trans Algo_InterpolateCoord Algo_QuatNorm Algo_QuatConj Algo_QuatScalar_NotNorm Reconfigurable Graphics Coding
- 20. Decoding network example ReconfigurableGraphics Coding A sub-network of SC3DMC
- 21. Applications Phone and tabletplayers Web databases (with and without plugins - WebGL) Games
- 22. Topics of theday Ongoing projects: integrating real and virtual MPEG offer in the 3D Graphics field What is MPEG? MPEG 3D Graphics
- 23. MPEG technologies relatedto real world MPEG-V - Media Context and Control 2011 • 2nd Edition: - GPS - Biosensors - 3D Camera 201x • Compression of video + depth 201x MPEG-V - 3D Video • 1st Edition - Sensors and actuators - Interoperability between Virtual Worlds • Feature-point based descriptors for image recognition 201x CDVS MPEG-U – Advanced User Interface 2012 A rich set of sensors and actuators - 3D Audio MPEG-H
- 24. MPEG-V: Sensors andActuators
- 25. Bridging Virtual andReal Worlds Real World (Sensor Device) Real World (Actuators) User Virtual Objects and Virtual World Augmented Reality
- 26. • All AR-relateddata is available from MPEG standards • Real time composition of synthetic and natural objects • Access to – Remotely/locally stored BIFS/compressed 2D/3D mesh objects – Streamed real-time BIFS/compressed 2D/3D mesh objects • Inherent object scalability (e.g. for streaming) • User interaction & server generated scene changes • Physical context – Captured by a broad range of standard sensors – Affected by a broad range of standard actuators Main features of MPEG AR technologies
- 27. MPEG vision onAR, the MPEG AR Browser Point to a URL – no need to download new applications for each context. The browser – Retrieves scenario from the internet – Starts video acquisition – Tracks objects – Recognizes objects from visual signatures – Recovers camera pose – Gets streamed 3D graphics – Composes new scenes – Gets inputs from various sensors – Offers optimal AR experience by constantly adapting interaction possibilities and objects from a remote server. Industry – Maximize number of customers through MPEG-compliant authoring tools and browsers – No need to develop a new application for each use case and device platform
- 28. MPEG vision onAR MPEG-4/MPEG-7/MPEG-21/ MPEG-U/MPEG-V MPEG Player Compression Authoring Tool Produce Download
- 29. Architecture AR Player Media Servers Service Servers User Local Sensors & Actuators Remote Sensors& Actuators AR file or stream Local Real World Environment Remote Real World Environment
- 30. AR Player Media Servers Service Servers User Local /Remote Context AR Document Enterprise viewpoint: global architecture and actors Augmented Reality Reference Model ARTC AREC AC AR Tools Creator (ARTC) AR Experience Creator (AREC) Assets Creator (AC) AC AA Assets Aggregator (AA) Device Manufacturer (DM) Middleware/Component Provider (MCP) DM MCP MCP End-User (EU) EU Telecommunication Operator (TO) TO TO TO TO Online Middleware/Component Provider (OMCP) AR Service Provider (ARSP) OMCP ARSP
- 31. Information viewpoint Augmented RealityReference Model AR Player Media Servers Service Servers User Local/Remote Context AR Document Scene/Real World • Raw image • Sensed data • Virtual Camera view • Detected features • Area of Interest/Anchors Tracking objects • Markers • Marker-less Device Context • Device capabilities Location of Device • Location • Orientation Spatial Models • Coordinate Ref. Sys. • (Geol)ocation • Projections • Coordinate conversion Presentation • Augmentation • Registration • Styling/complexity • Spatial Filtering, e.g. range User Input • Query • Manipulation of Presentation • Topics of interest • Preferences Digital Assets • Presentation data • Trigger/Event rules • Accuracy based
- 32. AR Player Media Servers Service Servers User Local /Remote Context AR Document Computational viewpoint Augmented Reality Reference Model 1 2 3 4 5
- 33. AR Player Media Servers Service Servers User Local /Remote Context AR Document Computational viewpoint Augmented Reality Reference Model 1 2 4 5 3
- 34. Engineering viewpoint Augmented RealityReference Model AR Player Media Servers Service Servers User Local/Remote Context AR Document Camera Mic Accelero meter Compass GPS … Rendering Engine Display (A/V/H) Application Engine …
- 35.
- 36. Use cases Augmented RealityReference Model
- 37. How to contribute? AugmentedReality Reference Model Use Trac! http://wg11.sc29.org/trac/augmentedreality/
- 38. A set ofscene graph nodes/protos as defined in MPEG-4 Part 11 – Existing nodes – Audio, image, video, graphics, programming, communication, user interactivity, animation – New standard PROTOs – Map, MapMarker, Overlay, ReferenceSignal, ReferenceSignalLocation, CameraCalibration, AugmentedRegion Connection to sensors as defined in MPEG-V – Orientation, Position, Angular Velocity, Acceleration, GPS, Geomagnetic, Altitude – Local camera sensor Compressed media Three components: scene, sensors/actuators, medias MPEG-A Part 13 ARAF
- 39. MPEG-A Part 13ARAF AR Quiz Augmented Book MPEG Augmented Reality examples
- 40. MPEG Augmented Realityexamples MPEG-A Part 13 ARAF AR Quiz Augmented Book http://youtu.be/LXZUbAFPP-Y http://youtu.be/la-Oez0aaHE
- 41. • MPEG standardizeda rich set of 3D Graphics representations and theirs compressions • MPEG recently proposed an approach to address the complexity of 3DG domain: Reconfigurable Graphics Coding • Joint development of an AR Reference Model (with SC24/WG9, ARStandards, Khronos, OGC) – The community at large is invited to contribute – http://wg11.sc29.org/trac/augmentedreality • MPEG promoted at CD a first version of an integrated and consistent solution for representing content in AR applications and services – Continue synchronized/harmonized development of technical specifications with X3D, COLLADA, OGC content models Conclusions
- 42. • www.MyMultimediaWorld.com • www-artemis.it-sudparis.eu •marius.preda@it-sudparis.eu Thank you !
Editor's Notes
- #4 The 3D Graphics chain consists of 2 main parts: authoring-the part where content is produced and publishing-the part where content is used like a final product. On authoring side, nowadays we have very heterogeneous production chains: -we have a lot of content creation tools and each of them have their own capabilities. Sometimes some effects can be created in one tool, but not in the other. -each of this tools has its own format. -usually artists and creators are common with one, not with all tools. They spent a lot of time to learn how the tool works, so it’s normal than they prefer to work with it.
- #5 On the other side, we have the publishing part. Here we have 2 solutions how to get the data: -in BOX solution: where all data is contained on DVD or other large medium. In this case, most of the time the companies that create this content have their own data format which is adapted to their rendering engine. -and we have ONLINE solutions: in this case the data is delivered by network. Here, we usually have an ad hoc format which in most cases is zipped version of an authoring format.
- #6 If we decide to use online data delivery, we can conclude the following: in a production line we have a different authoring tool that can produce the final product in different formats. Anyway, for production the size of the data is not important, because it is in the same place, there is no need of transport it over network etc.. What is important for the production part is than the file format should be complete == to be able to contain all of the data. On the other side we have end users where this content should be published. Here also we don’t care about the size but we just care to have all information. The problem raises when production and the end user are not on the same side; in such a case we must TRANSFER the content trough the network. This is where the file size is very important.
- #7 The solution is to have an encoding layer: due to the fact that the file size is not important in the production line, one can use a non compressed file format, but it should be sure that it contains all of the information; when transfer trough a Network is needed, ENCODE it and TRANSFER it in a compressed form. After, in the end users applications, DECODE it and use the information from the file.