Hybrid User Interfaces for Augmented Reality

      Alex Olwal, Ph.D.
      olwal.com
      Human-Computer Interaction, KTH
Alex Olwal, Ph.D.

Display Systems




Sensing




Interaction Techniques




2003-2010           KTH                     ...
Augmented Reality (AR) on mobile phones
                                      Layar




                      WIKITUDE
X: The Man with the X-Ray Eyes (1963)
RoboCop (1987)
Terminator 3 (2003)
Augmented Reality (AR)

•   Empower our senses → enhance perception
•   Supplement real
•   Seamless
AR
Fuse virtual objects with real environment
•    View
•    Perspective
•    Real-time
Head-worn




 Ivan Sutherland 1968   5dt HMD   Nomad   ProView XL40/50 STm   AddVisor 150
Head-worn




Handheld, Situated & Hybrid



Everywhere Displays [Pinhanez et al. 2001], iLamp Projectors [Raskar et al. 2...
Ubiquitous computing
Weiser 1993


Ubiquitous
• Displays
• Sensing
• Connectivity




Devices
• Tabs, pads & boards
Interactive surfaces & tangible UIs

• Display 2D graphics
• Sense on & above surface
• Detect & augment objects




Digit...
Spatially aware displays
• Tracked display
• Context-sensitive
• Focus + context




Chameleon [ Fitzmaurice 1993 ]
NaviCa...
NaviCam
Rekimoto 1995




                14
Hybrid UIs for AR

1 Unobtrusive                         AR, UbiComp, Mobile, Interactive Surfaces, …

2 Sensing in enviro...
SurfaceFusion
Sensing & augmenting everyday objects




GI 2008 [Olwal & Wilson]
Proc. Graphics Interface
LightSense
Dynamic augmentations of printed media




ISMAR 2006 [Olwal]
Proc. IEEE & ACM International Symposium on Mixed...
LUMAR
Hybrid 2D + 3D handheld AR




ICAT 2007 [Olwal & Henrysson]
Proc. International Conference on Artifical Reality & T...
LUMAR
  Hybrid 2D + 3D handheld AR




Printed media          2D graphics   3D graphics
Floor plan             Photographs...
Spatially aware handhelds
Enhancing interaction with large displays




TEI 2009 [Olwal & Feiner]
Proc. International Conf...
mobile pixels
                                                  projected                             improved visual fide...
Resolution
Denser pixels & sensors on handhelds

px/mm2
180


160


140


120


100


 80


 60


 40


 20


  0
        ...
Augmenting surface interaction
Mobile devices ・ expanded capabilities




INTERACT 2009 [Olwal]
Proc. IFIP TC13 Conference...
Augmenting surface interaction
Collaboration for multiple users, devices & locations


    • Need HybridSurface slide




...
Hybrid UIs for AR

1 Unobtrusive                         AR, UbiComp, Mobile, Interactive Surfaces, …

2 Sensing in enviro...
ASTOR
Transparent window with autostereoscopic 3D overlays




SPIE 2008 [Olwal, Gustafsson & Lindfors]
Proc. SPIE Electro...
process data



               workspace
workspace &
process data
Unencumbered 3D interaction
Supporting walk-up-and-use for public displays




NordiCHI 2008 [Olwal]
Proc. Nordic Conferen...
transparent 3D display
( holographic optical element )



                                                touch overlay

 ...
Unencumbered 3D interaction
Manipulate 3D data using mobile, gestures & touch




NordiCHI 2008 [Olwal]
Proc. Nordic Confe...
Immaterial displays
 Face-to-face, reach-through & walk-through interaction




3DTV Book 2008 [DiVerdi, Olwal, Rakkolaine...
Hybrid UIs for AR

1 Unobtrusive                                            AR, UbiComp, Mobile, Interactive Surfaces, …

...
Nintendo Wiimote
Camera + accelerometers + gyros + “sensor” bar
PlayStation Move
PS Eye camera + Motion controller
Microsoft Kinect
Depth camera
Mobile projection
AR marketing
Rule Communication – www.rule.se




• Relevant
• Content consumption
• Interactivity
• Activity analysis



• projects@ru...
HybridSurface – www.harmonic.se

• Hybrid interaction: Touch surface + mobile devices
• Ultra-thin design
• Mobile World C...
Acknowledgements

Organizers                                  Funding
•   Disruptive Media                        •   KK f...
Hybrid User Interfaces for Augmented Reality

      Alex Olwal, Ph.D.              research: olwal.com
      olwal.com    ...
Hybrid User Interfaces for Augmented Reality

      Alex Olwal, Ph.D.              research: olwal.com
      olwal.com    ...
Hybrid User Interfaces for Augmented Reality
Hybrid User Interfaces for Augmented Reality
Hybrid User Interfaces for Augmented Reality
Hybrid User Interfaces for Augmented Reality
Hybrid User Interfaces for Augmented Reality
Upcoming SlideShare
Loading in …5
×

Hybrid User Interfaces for Augmented Reality

3,078 views

Published on

Presentation by Dr. Alex Olwal (KTH) for the Disruptive Code conference, Stockholm 2010.

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,078
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
96
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Hybrid User Interfaces for Augmented Reality

  1. 1. Hybrid User Interfaces for Augmented Reality Alex Olwal, Ph.D. olwal.com Human-Computer Interaction, KTH
  2. 2. Alex Olwal, Ph.D. Display Systems Sensing Interaction Techniques 2003-2010 KTH Stockholm 2006 Microsoft Research Redmond 2005 University of California Santa Barbara 2002-2003 Columbia University New York startup Rule Communication Stockholm www.rule.se startup Harmonic Solutions Stockholm www.harmonic.se
  3. 3. Augmented Reality (AR) on mobile phones Layar WIKITUDE
  4. 4. X: The Man with the X-Ray Eyes (1963)
  5. 5. RoboCop (1987)
  6. 6. Terminator 3 (2003)
  7. 7. Augmented Reality (AR) • Empower our senses → enhance perception • Supplement real • Seamless
  8. 8. AR Fuse virtual objects with real environment • View • Perspective • Real-time
  9. 9. Head-worn Ivan Sutherland 1968 5dt HMD Nomad ProView XL40/50 STm AddVisor 150
  10. 10. Head-worn Handheld, Situated & Hybrid Everywhere Displays [Pinhanez et al. 2001], iLamp Projectors [Raskar et al. 2003], On-board Mixed Reality Projector [Karitsuka & Sato 2003], Transflective Surfaces [Bimber et al. 2000], Cell phone [Möhring & Bimber 2004], NaviCam [Rekimoto 1997], PDA [Wagner & Schmalstieg 2003]
  11. 11. Ubiquitous computing Weiser 1993 Ubiquitous • Displays • Sensing • Connectivity Devices • Tabs, pads & boards
  12. 12. Interactive surfaces & tangible UIs • Display 2D graphics • Sense on & above surface • Detect & augment objects DigitalDesk [ Wellner 1993 ] Tangible Bits [ Ullmer & Ishii & Buxton 1997 ] Augmented Surfaces [ Rekimoto & Saitoh 1999 ] PlayAnywhere / PlayTogether [ Wilson 2005 / Wilson & Robbins 2006 ]
  13. 13. Spatially aware displays • Tracked display • Context-sensitive • Focus + context Chameleon [ Fitzmaurice 1993 ] NaviCam [ Rekimoto 1995 ] Augmented Notebook [ Mackay et al. 2002 ] Focus + Context displays [ Baudisch et al. 2002 ] VITA [ Benko et al. 2004 ] Ubiquitous graphics [ Sanneblad & Holmquist 2006 ]
  14. 14. NaviCam Rekimoto 1995 14
  15. 15. Hybrid UIs for AR 1 Unobtrusive AR, UbiComp, Mobile, Interactive Surfaces, … 2 Sensing in environment Shopping window Microsoft Surface
  16. 16. SurfaceFusion Sensing & augmenting everyday objects GI 2008 [Olwal & Wilson] Proc. Graphics Interface
  17. 17. LightSense Dynamic augmentations of printed media ISMAR 2006 [Olwal] Proc. IEEE & ACM International Symposium on Mixed & Augmented Reality Ericsson Demo Center Stockholm 2008
  18. 18. LUMAR Hybrid 2D + 3D handheld AR ICAT 2007 [Olwal & Henrysson] Proc. International Conference on Artifical Reality & Teleexistence Judge’s Choice Award in Nokia “Calling All Innovators” Challenge 2009 Mobile World Congress, Barcelona, 13-16 February, 2009
  19. 19. LUMAR Hybrid 2D + 3D handheld AR Printed media 2D graphics 3D graphics Floor plan Photographs 3D model
  20. 20. Spatially aware handhelds Enhancing interaction with large displays TEI 2009 [Olwal & Feiner] Proc. International Conference on Tangible & Embedded Interaction INTERACT 2009 [Olwal] Proc. IFIP TC13 Conference on Human-Computer Interaction Ericsson Trade Show Events in 2009 / 2010 Barcelona, Las Vegas, Boston, Galway, Stockholm, Paris, Amsterdam, San Francisco
  21. 21. mobile pixels projected improved visual fidelity & pixels touch input resolution 352 px camera view 240 px 217 px 408 mm device tracking region projection & 48 mm 71 px 320 px touch input 2021 mobile px 288 px 306 mm 163 px 95 px 600 px 36 mm 800 px 2704 mobile px mobile device
  22. 22. Resolution Denser pixels & sensors on handhelds px/mm2 180 160 140 120 100 80 60 40 20 0 Projector LCD Apple iPhone Sony Ericsson G900 Nokia 5800XM Google Nexus One Apple iPhone 4 32" 21" 3.5" 2.4" 3.2" 3.7" 3.5" 1024×768 1600×1200 320×480 240×320 360×640 480×800 640x960
  23. 23. Augmenting surface interaction Mobile devices ・ expanded capabilities INTERACT 2009 [Olwal] Proc. IFIP TC13 Conference on Human-Computer Interaction Ericsson Trade Show Events in 2009 / 2010 Barcelona, Las Vegas, Boston, Galway, Stockholm, Paris, Amsterdam, San Francisco
  24. 24. Augmenting surface interaction Collaboration for multiple users, devices & locations • Need HybridSurface slide TEI 2009 [Olwal & Feiner] Proc. International Conference on Tangible & Embedded Interaction INTERACT 2009 [Olwal] Proc. IFIP TC13 Conference on Human-Computer Interaction Ericsson Trade Show Events in 2009 / 2010 Barcelona, Las Vegas, Boston, Galway, Stockholm, Paris, Amsterdam, San Francisco
  25. 25. Hybrid UIs for AR 1 Unobtrusive AR, UbiComp, Mobile, Interactive Surfaces, … 2 Sensing in environment 3 Novel display technology Shopping window Microsoft Surface Microvision HUD Heliodisplay, Mid-air
  26. 26. ASTOR Transparent window with autostereoscopic 3D overlays SPIE 2008 [Olwal, Gustafsson & Lindfors] Proc. SPIE Electronic Imaging, Vol 6804 ISMAR 2005 [Olwal, Lindfors, Gustafsson, Kjellberg & Mattson] Proc. IEEE & ACM Symposium on Mixed & Augmented Reality SIGGRAPH 2004 Sketches [Olwal, Lindfors & Gustafsson] International Conference on Computer Graphics & Interactive Techniques
  27. 27. process data workspace
  28. 28. workspace & process data
  29. 29. Unencumbered 3D interaction Supporting walk-up-and-use for public displays NordiCHI 2008 [Olwal] Proc. Nordic Conference on Human Computer Interaction
  30. 30. transparent 3D display ( holographic optical element ) touch overlay 3D camera eye tracker
  31. 31. Unencumbered 3D interaction Manipulate 3D data using mobile, gestures & touch NordiCHI 2008 [Olwal] Proc. Nordic Conference on Human Computer Interaction
  32. 32. Immaterial displays Face-to-face, reach-through & walk-through interaction 3DTV Book 2008 [DiVerdi, Olwal, Rakkolainen & Höllerer] Three-Dimensional Television: Capture, Transmission & Display, Springer VR 2006 [Olwal, DiVerdi, Rakkolainen & Höllerer] Proc. IEEE Virtual Reality SIGGRAPH 2005 Emerging Technologies [Rakkolainen, DiVerdi, Olwal, Candussi & Höllerer] International Conference on Computer Graphics & Interactive Techniques
  33. 33. Hybrid UIs for AR 1 Unobtrusive AR, UbiComp, Mobile, Interactive Surfaces, … 2 Sensing in environment 3 Novel display technology 4 Hybrid approaches → sensing / displays / interaction BiDi: Flat depth-sensing displays SecondLight Interactions in the air Hirsch, Lanman, Holtzman & Izadi, Hodges, Taylor, Rosenfeld, Hilliges, Izadi, Wilson, Hodges, Raskar 2009 Villar, Butler & Westhues 2008 Mendoza & Butz 2009
  34. 34. Nintendo Wiimote Camera + accelerometers + gyros + “sensor” bar
  35. 35. PlayStation Move PS Eye camera + Motion controller
  36. 36. Microsoft Kinect Depth camera
  37. 37. Mobile projection
  38. 38. AR marketing
  39. 39. Rule Communication – www.rule.se • Relevant • Content consumption • Interactivity • Activity analysis • projects@rule.se
  40. 40. HybridSurface – www.harmonic.se • Hybrid interaction: Touch surface + mobile devices • Ultra-thin design • Mobile World Congress 2009 (Barcelona), CTIA Wireless 2009 (Las Vegas), Ericsson Volvo Ocean Race VIP Pavilion 2009 (Boston, Galway, Stockholm), Ericsson Experience Center • Next generation for new Maritime Museum Exhibit 2011
  41. 41. Acknowledgements Organizers Funding • Disruptive Media • KK foundation • Swedish Institute for Assistive Collaborators Technologies • Department of Production Engineering, • Engineer’s Science Academy / Innovation KTH Bridge • Center for Medicine, Health and • Blanceflor Foundation Technology, KTH / KI / KS • Swedish Research Council • FunkIS (HCI + radiology research) • Department of Mechatronics Equipment, donations & funding • VITA, Linköping University • Ericsson, Nokia, Microsoft Research, Mitsubishi, Doro, … • Computer Graphics & UI lab, Columbia University • ilab, USCB
  42. 42. Hybrid User Interfaces for Augmented Reality Alex Olwal, Ph.D. research: olwal.com olwal.com e-marketing: rule.se new user interfaces : harmonic.se
  43. 43. Hybrid User Interfaces for Augmented Reality Alex Olwal, Ph.D. research: olwal.com olwal.com e-marketing: rule.se new user interfaces: harmonic.se

×