Dieter Schmalstieg gave a presentation summarizing recent trends in augmented reality (AR) research presented at ISMAR 2016. Some of the key trends discussed included improved camera-based tracking using specialized hardware, new methods for modeling blur in optical see-through head-mounted displays, techniques for dynamic registration in environments with moving users and objects, real-time photorealistic lighting effects, and integrating real haptics and virtual objects. Examples of award-winning papers from ISMAR 2016 were presented covering areas such as high-speed 6DOF tracking, modeling display blur, dynamic occlusion handling, and remote 3D scanning exploration.

1. ISMAR
ISMAR 2016 Research Review
Dieter Schmalstieg
Graz University of Technology, Austria

2. ISMAR
About me
• Professor at Graz University of Technology
• 20 years of research experience in AR
• First collaborative AR system, 1996
• First self‐contained AR system on PDA, 2003
• Author of Augmented Reality – Principles and Practice 
• http://www.augmentedrealitybook.org
• Many industrial collaborations (Qualcomm, DAQRI, …)
• Long term supporter of the ISMAR academic conference
Graz through a smartphone, 2008
Dieter Schmalstieg ISMAR 2

3. ISMAR
About ISMAR
• The IEEE International Symposium on
Mixed and Augmented Reality
• Premier academic conference on MR/AR
• Established in 1998
• Partnership with IEEE journal
• Transactions on Visualization and Computer Graphics
• Accepts work from all areas of MR/AR
• Displays (head‐mounted, projector‐based, light field…)
• Computer vision (tracking, localization, SLAM…)
• Computer graphics (x‐ray vision, common illumination, diminished reality…)
• Human‐computer interaction (tele‐collaboration, tangible interfaces…)
• http://ismar.vgtc.org
Dieter Schmalstieg ISMAR 3

4. ISMAR
Trends in Tracking
• Two directions
• Better results with standard hardware
• SLAM in particular
• New methods using special hardware
• Cluster of scanline cameras
•  ISMAR2016 best paper award
Dieter Schmalstieg ISMAR 4

5. ISMARTowards KHz 6DOF Visual Tracking using an
Egocentric Cluster of Rolling Shutter Camera
• A. Bapat, E. Dunn, J.‐M. Frahm (UNC Chapel Hill)
• 5x2 GoPros stereo rigs
• Treated as line cameras
• Line matching at 50K lines/sec
Dieter Schmalstieg ISMAR 5

6. ISMAR
Trends in Displays
• Head‐mounted displays everywhere
• But many optical limitations not addressed by vendors
• Controlling blur and chromatic aberration
•  ISMAR2016 best paper runner‐up
Dieter Schmalstieg ISMAR 6

7. ISMAR
Gaussian Light Field: Estimation of Viewpoint‐Dependent
Blur for Optical See‐Through Head‐Mounted Display
• Y. Itoh, T. Amano, D. Iwai, G. Klinker (TU Munich, Wakayama U., Osaka U.)
• Model blur in see‐thru HMD
• Fit Gauss distribution to point spread function
• Index like light field (2D eye position, 2D direction)
Observed PSF Estimated PSF
Dieter Schmalstieg ISMAR 7

8. ISMAR
Trends in Registration
• Big remaining problem: registration in dynamic environments
• Moving user
• User in moving vehicle – novel driver assistance systems
• Motion capture of user and retargeting to avatar
• Moving objects
• Tracking dynamic objects for correct occlusion contours
Dieter Schmalstieg ISMAR 8

9. ISMAR
Real‐Time Augmented Reality System to See‐Through Cars
• F. Rameau, H. Ha, K. Joo, J. Choi, K. Park, I. S. Kweon (KAIST)
• Diminish front car by replacing with stereo video from front car
Dieter Schmalstieg ISMAR 9

10. ISMAR
Trends in Registration
• Big remaining problem: registration in dynamic environments
• Moving user
• User in moving vehicle – novel driver assistance systems
• Motion capture of user and retargeting to avatar
• Moving objects
• Tracking dynamic objects for correct occlusion contours
Dieter Schmalstieg ISMAR 10

11. ISMAR
Retargeting Human‐Object Interaction to Virtual Avatars
• Y‐ Kim, H. Park, S. Bang, S.‐H. Lee (KAIST)
• Extract motion and modify it so it
looks plausible on avatar
Dieter Schmalstieg ISMAR 11

12. ISMAR
Trends in Registration
• Big remaining problem: registration in dynamic environments
• Moving user
• User in moving vehicle – novel driver assistance systems
• Motion capture of user and retargeting to avatar
• Moving objects
• Tracking dynamic objects for correct occlusion contours
Dieter Schmalstieg ISMAR 12

13. ISMAR
Edge Snapping Based Depth Enhancement for
Dynamic Occlusion Handling in AR
• C. Du, Y.‐L. Chen, M. Ye, L. Ren (U. Kentucky, Bosch)
• Pixel‐precise registration between color and depth in RGB‐D video
Dieter Schmalstieg ISMAR 13

14. ISMAR
Trends in Graphics
• Photorealistic common (virtual+real) illumination
• In real time
• With minimal hardware (RGB‐D sensor)
• With minimal preparation and scene knowledge
• Instant high‐quality reflections
• Instant lighting and relighting
• Mediated reality
Dieter Schmalstieg ISMAR 14

15. ISMARSingle Camera Image Based Approach for Glossy Reflections
• T. Schwandt, W. Broll (TU Illmenau)
• Reflections from a single RGB‐D image
• Diffuse environment map from RGB
• Raytraced specular from depth map
Dieter Schmalstieg ISMAR 15

16. ISMAR
Trends in Graphics
• Photorealistic common (virtual+real) illumination
• In real time
• With minimal hardware (RGB‐D sensor)
• With minimal preparation and scene knowledge
• Instant high‐quality reflections
• Instant lighting and relighting
• Mediated reality
Dieter Schmalstieg ISMAR 16

17. ISMAR
Instant Mixed Reality Lighting from Casual Scanning
• T. Richter‐Trummer, D. Kalkofen, J. Park, D. Schmalstieg (TU Graz, KAIST)
• Extract geometry, incident light and materials from RGB‐D camera
• Real‐time scene relighting and mediated reality
Dieter Schmalstieg ISMAR 17

18. ISMAR
Trends in Graphics
• Photorealistic common (virtual+real) illumination
• In real time
• With minimal hardware (RGB‐D sensor)
• With minimal preparation and scene knowledge
• Instant high‐quality reflections
• Instant lighting and relighting
• Mediated reality
Dieter Schmalstieg ISMAR 18

19. ISMARRealitySkins: Creating Immersive and Tactile
Virtual Environments
• L. Shapira, D. Freedman (Google, Microsoft)
• Scan a room and generate
a “themed” room in VR
Dieter Schmalstieg ISMAR 19

20. ISMAR
Trends in Interaction
• Real haptics + immersive VR
• Selecting real objects in 3D scans
• Remote exploration of 3D scans
Dieter Schmalstieg ISMAR 20

21. ISMARTactileVR: Integrating Physical Toys to Learn
and Play Virtual Reality Experiences
• L. Shapira, J. Amores, X. Benavides (Google, MIT)
• Games for kids physically interacting with tracked objects
Dieter Schmalstieg ISMAR 21

22. ISMAR
Trends in Interaction
• Real haptics + immersive VR
• Selecting real objects in 3D scans
• Remote exploration of 3D scans
Dieter Schmalstieg ISMAR 22

23. ISMARPixel‐Point‐Volume Segmentation for Object
Referencing in Collaborative AR
• K.‐C. Lien, B. Nuernberger, T. Höllerer, M. Turk (UCSB)
• Identify which object the user points to in RGB‐D scan
Dieter Schmalstieg ISMAR 23

24. ISMAR
Trends in Interaction
• Real haptics + immersive VR
• Selecting real objects in 3D scans
• Remote exploration of 3D scans
Dieter Schmalstieg ISMAR 24

25. ISMARStreaming and Exploration of Dynamically Changing
Dense 3D Reconstruction in Immersive VR
• A. Mossel, M. Kroeter (TU Vienna)
• Full system for RGB‐D reconstruction streaming and viewing in VR
Dieter Schmalstieg ISMAR 25

26. ISMAR
Conclusions
• ISMAR will be held in Nantes, France, October 2017
• Academic paper submission deadline: 15 March 2017
• Job fair
• Or, get involved as sponsor,
exhibitor, workshop organizer…
• Many ways to get connected!
• Future will see a strong and
continued collaboration with AWE
• http://ismar.vgtc.org
Dieter Schmalstieg ISMAR 26