This is a presentation I did during the 5th ATStandards meeting in Austin, 2012. It contains the MPEG Vision on AR as well as a very short overview of MPEG technologies related to AR

1. MPEG Technologies and
Roadmap for AR
5th ARStandards Meeting, March 19-20,
Austin, US
Marius Preda
Institut TELECOM

2. What is MPEG?
A family of standards published by ISO/IEC dealing with:
• Coding/compression of elementary media: audio/video (MPEG-1, 2 and 4), 2D/3D
graphics (MPEG-4)
• Hybrid (natural & synthetic) scene description, user interaction (MPEG-4)
• Metadata (MPEG-7)
• Media management and protection (MPEG-21)
• Sensors and actuators (MPEG-V)
• Virtual Worlds (MPEG-V)
• Advanced User interaction (MPEG-U)
• Media-oriented middleware (MPEG-M)
• Coding and Delivery in Heterogeneous Environments
• Dynamic Adaptive Streaming over HTTP
•…

3. What is MPEG?
An activity continuing for 24 years
Several hundreds of companies from all over the world
More than 2300 active users
A proven manner to organize the work in order to deliver useful and used standards
- 4 meetings per year, around the world, 500+ participants
- subgroups with clear objectives
- well defined procedure
Hundreds of millions of devices contain MPEG technologies (cameras, mobile phones,
DVD/Blue-Ray players, STB and TVs, …)

4. MPEG tools related to AR
1992/4 1997 1998 1999
MPEG-1/2
(AV content) VRML
MPEG-4
• Part 11 - BIFS:
-Binarisation of VRML
-Extensions for streaming
-Extensions for server command
-Extensions for 2D graphics
• Part 2 - Visual:
- 3D Mesh compression
- Face animation
MPEG-4
• Part 2 – Visual
- Body animation
First form of augmentation of broadcast signal

5. MPEG tools related to AR
2003 2005 2007 2011
MPEG-4
• Part 16 - AFX:
- A rich set of 3D
MPEG-4
graphics tools
•AFX 2nd Edition:
- Compression of
- Animation by
geometry, appearence,
morphing
animation
- Multi-texturing
MPEG-4
• AFX 3rd Edition
- WSS for terrain
and cities
A rich set of 3D Graphics - Frame based MPEG-4
representation and animation • AFX 4th Edition
compression tools - Scalable complexity
mesh coding

6. MPEG tools related to AR
2003 2004 2005 2007 2009 2011
MPEG-4
• Part 16 - AFX:
- A rich set of 3D
MPEG-4
graphics tools
•AFX 2nd Edition:
- Compression of
- Animation by
geometry, appearence,
morphing
animation
- Multi-texturing
MPEG-4
• AFX 3rd Edition
- WSS for terrain
and cities
MPEG-4 MPEG-4
- Frame based MPEG-4
• Part 16 • Part 25
animation • AFX 4th Edition
- X3D - Compression of
- Scalable complexity
Interactive third-party XML (X3D,
mesh coding
Profile COLLADA)

7. MPEG tools related to AR
2011 2012 201x 201x 201x
MPEG-V
• 1st Edition MPEG-U
- Sensors and • Advanced
actuators MPEG-V
User Interface
- Interoperability • 2nd Edition:
between Virtual - GPS
Worlds - Biosensors
- 3D Camera
3D Video
• Compression of
video + depth
A rich set of sensors, CDVS
actuators • Feature-points
based descriptors for
image recognition

8. MPEG tools related to AR
2011 2011 2012 201x 201x 201x
MPEG-V
• 1st Edition MPEG-U
- Sensors and • Advanced
actuators MPEG-V
User Interface
- Interoperability • 2nd Edition:
between Virtual - GPS
Worlds - Biosensors
- 3D Camera
3D Video
• Compression of
video + depth
Augmented Reality CDVS
• A study about the MPEG • Feature-points
offer in the field of AR based descriptors for
image recognition

9. MPEG vision on AR
Using MPEG technologies it is possible to envision having a generic AR
browser. Instead of downloading a new and heavy application for every
context, users could simply point to a URL. The browser would retrieve a
scenario from the internet, start the video acquisition, track objects,
recognize few of them from visual signatures, recover the camera pose,
download 3D graphics, compose a new scene, get inputs from various
sensors, and constantly adapt the interaction possibilities and objects
from a remote server to offer an optimal AR experience.
Instead of developing a new application for each use case and smart-phone
platform, the industry could rely on MPEG-compliant authoring tools, and
MPEG-compliant browsers to reach the maximum number of customers.

10. MPEG vision on AR
Produce
Download
Compression
Authoring Tool
MPEG-4/MPEG-7/MPEG-21/
MPEG-U/MPEG-V
MPEG Player

11. MPEG Plans for AR
• Document use-cases
• Extract requirements
• Profile existing tools to address the requirements
• Issue Call for Proposals for new technologies
• Develop an Application Format for AR as part of
MPEG-A

12. AR use-cases
• Broadcast AR

13. AR use-cases
• Realistic augmentation

14. AR use-cases
• AR/MR Games

15. AR Requirements
Requirements for Augmented Reality Application Format
(N12423)
– on scene description
– on media
– on metadata
– on remote access
– on compression
– on management and protection
– on integration and communication

16. AR Context

17. Main features of MPEG AR technologies
• Representation of synthetic and natural objects
• Real-time BIFS/3D mesh based streaming
• Objects from remote/local storage, stored/real-time
• Inherent object scalability
• User interaction/server generated scene changes
• Physical context captured/affected by a broad range
of sensors/actuators

18. ARAF CfP
Call for Proposals for Additional Augmented Reality
Technologies (N12588, February 2012, San Jose)
- ARAF metadata
- ARAF management and protection

19. ARAF
• WD2.0
• Some technical details and a first implementation of
ARAF on mobile phone will be presented tomorrow
in Session 8