Augmented Reality: Connecting physical and digital worlds


Published on

I made this presentation at the MPEG Multimedia Ecosystem 2013 in Incheon.

It includes a summary of MPEG technologies related to Augmented Reality and is focuses on the separation between the AR creation and AR consuming. A system architecture for AR is also presented.

Photo credits: Lisa Blum, Richard Wetzel, Veronica Scurtu
Note: many pictures used in this presentation are downloaded from the Internet; I'll be happy to add credits to the original authors if they let me know

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Augmented Reality: Connecting physical and digital worlds

  2. 2. From centuries, we are building two different types of worldsPhysicalInformationalImmersion, from a human user point of view, is making abstraction of the Physicalworld (remove all possible connection to it) and have stimuli only from theInformational oneImmersion, presence, stimuli, perception, …storiesmusicfilmsknowledge
  3. 3. In which world our users are?The stimuli from physical worldsare familiar or at very lowintensities
  4. 4. We intentionally block ourinterfaces with thephysical worldImproving immersion feeling
  5. 5. Strategies for blockingthe interfaces with thePhysical worldImproving immersion feeling
  6. 6. Strategies for extending the space ofthe Informational world stimuliImproving immersion feeling
  7. 7. How far we’ve gone withthis strategy?Relatively far …Improving immersion feeling
  8. 8. Probably too far …Improving immersion feelingHow far?
  9. 9. It’s the time to connect our worlds
  10. 10. Why now?• For the first time, smartphones enable“everyone” to access Informational worldwhile sensing also the Physical world• They change our view of the environment• Enhance the rich historyand meaningfulness ofthe real world withpowerful digitalelements
  11. 11. Physical & Informational Augment The RealityAn invitation: become architects of a newkind of bridges
  12. 12. In AR, almost everything remains to be inventedLas VegasThis place some years ago
  13. 13. In AR, almost everything remains to be invented
  14. 14. • Marketing• Tourism• Culture• Education• Serious and less serious games• …• Hardware manufacturers• Software vendorsIn AR, almost everything remains to be invented
  15. 15. We (and few others) already started and inviteyou to be part ofThe AR RevolutionIn AR, almost everything remains to be invented
  16. 16. Tidy City
  17. 17. Portal Hunt
  18. 18. Elements
  19. 19. ARQuiz
  20. 20. Augmented Books
  21. 21. Summer School games
  23. 23. MPEG technologies related to ARMPEG-1/2(AV content)1992/4VRML1997• Part 11 - BIFS:-Binarisation of VRML-Extensions for streaming-Extensions for server command-Extensions for 2D graphics- Real time augmentation withaudio & video• Part 2 - Visual:- 3D Mesh compression- Face animation1998• Part 2 – Visual- Body animation1999MPEG-4 v.1MPEG-4 v.2First form of broadcast signal augmentation
  24. 24. MPEG technologies related to ARMPEG-42003•AFX 2nd Edition:- Animation bymorphing- Multi-texturing2005• AFX 3rd Edition- WSS for terrainand cities- Frame basedanimation2007MPEG-4MPEG-4• Part 16 - AFX:- A rich set of 3Dgraphics tools- Compression ofgeometry,appearance,animation• AFX 4th Edition- Scalable complexitymesh coding2011MPEG-4A rich set of 3D Graphicsrepresentation andcompression tools
  25. 25. MPEG technologies related to ARMPEG-42003•AFX 2nd Edition:- Animation bymorphing- Multi-texturing2005• AFX 3rd Edition- WSS for terrainand cities- Frame basedanimation2007MPEG-4MPEG-4• Part 16 - AFX:- A rich set of 3Dgraphics tools- Compression ofgeometry,appearance,animation• AFX 4th Edition- Scalable complexitymesh coding2011MPEG-42009• Part 25- Compression ofthird-party XML(X3D, COLLADA)MPEG-42004• Part 16- X3DInteractiveProfileMPEG-4
  26. 26. MPEG technologies related to ARMPEG-V - MediaContext and Control2011• 2nd Edition:- GPS- Biosensors- 3D Camera201x• Compressionof video +depth201xMPEG-V- 3D Video• 1st Edition- Sensors andactuators- Interoperabilitybetween VirtualWorlds• Feature-point baseddescriptors for imagerecognition201xCDVSMPEG-U –AdvancedUser Interface2012A rich set of sensors andactuators- 3D AudioMPEG-H
  27. 27. MPEG technologies related to ARMPEG-V – Media Context and Control
  28. 28. ActuatorsLightFlashHeatingCoolingWindVibrationSprayerScentFogColor correctionInitialize color correction parameterRigid body motionTactileKinestheticGlobal position commandSensorsLightAmbient noiseTemperatureHumidityDistanceAtmospheric pressurePositionVelocityAccelerationOrientationAngular velocityAngular accelerationForceTorquePressureMotionIntelligent camera typeMulti Interaction pointGaze trackingWindDustBody heightBody weightBody temperatureBody fatBlood typeBlood pressureBlood sugarBlood oxygenHeart rateElectrographEEG , ECG, EMG, EOG , GSRWeatherFacial expressionFacial morphologyFacial expression characteristicsGeomagneticGlobal positionAltitudeBendGasMPEG technologies related to ARMPEG-V – Media Context and Control
  29. 29. • All AR-related data is available from MPEG standards• Real time composition of synthetic and natural objects• Access to• Remotely/locally stored BIFS/compressed 2D/3D mesh objects• Streamed real-time BIFS/compressed 2D/3D mesh objects• Inherent object scalability (e.g. for streaming)• User interaction & server generated scene changes• Physical context• Captured by a broad range of standard sensors• Affected by a broad range of standard actuatorsMain features of MPEG AR technologies
  30. 30. MPEG vision on AR: the MPEG AR BrowserPoint to a URL – no need to download new applications for each contextThe browser• Retrieves scenario from the internet• Starts video acquisition• Tracks objects• Recognizes objects from visual signatures• Recovers camera pose• Gets streamed 3D graphics• Composes new scenes• Gets inputs from various sensors• Offers optimal AR experience by constantly adapting interaction possibilities andobjects from a remote server.Industry• Maximize number of customers through MPEG-compliant authoring tools andbrowsers• No need to develop a new application for each use case and device platform
  31. 31. MPEG vision on ARMPEG-4 + MPEG-7 + MPEG-21 ++ MPEG-U+ MPEG-V= ARAFMPEG PlayerCompressionAuthoring ToolProduceDownload
  32. 32. Technology insightsModeling the content chain– Creation– Consumption– The pivot : a representation formalism
  33. 33. Technology insights: the chainARAFBrowserMediaServersServiceServersUserLocalSensors &ActuatorsRemoteSensors &ActuatorsMPEGARAFLocalReal WorldEnvironmentRemoteReal WorldEnvironmentAuth. Tools
  34. 34. Technology insights: InfrastructureARAF BrowserMediaServersServiceServersUserLocalSensors &ActuatorsRemoteSensors &ActuatorsMPEGARAFLocalReal WorldEnvironmentRemoteReal WorldEnvironmentAuth. Tools
  35. 35. The ARAF browserARAF Browser• Full scene graph representation (2D and 3D)• Static and dynamic objects and scenes• Programmable behavior (JS)• Access to sensors and cameras• Access to remote resources and services (httprequest and web sockets)• Support for user interaction• Support for compressed resources
  36. 36. The ARAF browserARAF BrowserARAF BrowserJavaScript EngineMPEG Totem ServicesWebSocketsServerCommunicationserviceWebSocketsClientARAF BrowserSC3DMCDecoderBBA DecoderARAF FileBIFS StreamSC3DMCStreamBBA StreamBIFS DecoderMPEG-4SceneIndexedFaceSetSBVCAnimationCameraImage dataReferenceImagesImage dataImage dataImage dataAFAR BrowserReferenceSignalDetectionDetectedImages
  37. 37.• Convert media assets in a common format• Describe, index, organize, search digital assets• Compressed assets, ready for transmissionA multimedia repository and processing platformsupporting both creation and consumption Media&ServiceServers
  38. 38. Technology insights: the formalismARAF PlayerMediaServersServiceServersUserLocalSensors &ActuatorsRemoteSensors &ActuatorsMPEGARAFLocalReal WorldEnvironmentRemoteReal WorldEnvironmentAuth. Tools
  39. 39. ARAF Description LanguageARAF : an XML representationPlayerCompressionAuthoring ToolProduceDownloadARAF
  40. 40. A set of scene graph nodes defined in MPEG-4• Audio, image, video, graphics, programming, communication, userinteractivity, animation• Map, MapMarker, Overlay, ReferenceSignal, ReferenceSignalLocation, CameraCalibration, AugmentedRegionConnection to sensors defined in MPEG-V• Orientation, Position, AngularVelocity, Acceleration, GPS, Geomagnetic, Altitude, Local camera(s)Compressed media• Image, sound, video, 2D/3D graphicsAugmented Reality Application FormatARAF
  41. 41. Technology insights: creationARAFBrowserMediaServersServiceServersUserLocalSensors &ActuatorsRemoteSensors &ActuatorsMPEGARAFLocalReal WorldEnvironmentRemoteReal WorldEnvironmentAuth. Tools
  42. 42. Authoring ARAF• Specific tools• Generic tools– TOTEM.Designer– TOTEM.ScoutAuth. Tools
  43. 43. Specific Authoring Tools• Scenario driven approach– Tidy City, Portal Hunt, ARQuiz, AR Book, TreasureHuntAuth. Tools
  44. 44. Towards a generic Designer• Create templates for app. objects• Fill structure with data• Create GUI• Create application logic• Export data
  45. 45. On site designer
  46. 46. Conclusions• AR – the next digital revolution• For making revolutions, one needs weapons:– MPEG technologies represent an efficient andpowerful set of weapons• Current MPEG activities– Improve and enrich the weapons– Package them in consistent and easy to use tools