Ramesh Raskar at AWE 2018. Lets go beyond our obession with HMDs and consider multi modal multiformat augmentation.

1. Raskar | MIT Media Lab
Ramesh Raskar
Assoc. Prof
MIT Media Lab
Unspoken Challenges in AR/VR:

2. Raskar | MIT Media Lab
Multi-ModeMulti-Format

3. Raskar | MIT Media Lab
Multi-ModeMulti-Format

4. Raskar | MIT Media Lab
© Andrei State
Augmented Human:
Beyond X-ray vision

5. Raskar | MIT Media Lab
Augmented Reality (AR)
World Input Process Output World
Sense beyond
human ability Abstract
Present within
human ability

6. Raskar | MIT Media Lab
Augmented Reality (AR)
World Input Process Output World
Camera,
Location
Sensors
Identify,
Track,
Abstract
Display,
Overlay
Interaction
Resolution,
FoV, FPS,
Optics
Bounds of
Physics + Perception

7. Raskar | MIT Media Lab
Multi-ModeMulti-Format

8. Raskar | MIT Media Lab
Multi-ModeMulti-Format
IoT
Perception
Ergo,
Hygiene
Ethics
Gestural Language

9. Raskar | MIT Media Lab
Challenges in AR
World Input Process Output World
Camera,
Location
Sensors
Identify,
Track,
Abstract
Display,
3D Overlay
Interaction
MultiModal MultiFormat
IoT
Training
DataSets
Perception
Ergo, Hygiene
Ethics
Gestural Language
EyeGlasses
Laziness
Authoring
IoT

10. Raskar | MIT Media Lab
World Input Process Output World
Camera,
Location
Sensors
Identify,
Track,
Abstract
Display,
3D Overlay
Interaction
MultiFormat
IoT
Training
DataSets
Ergo, Hygiene
Ethics
Gestural Language
EyeGlasses
Laziness
Authoring
IoT
Challenges in AR
MultiModal
Perception

11. Raskar | MIT Media Lab
South Park - Jurassic Park

12. Raskar | MIT Media Lab
Photoreal Functional Realism
NonPhotoRealistic Camera, Siggraph 2004
http://Raskar.info/NprCamera

13. Raskar | MIT Media Lab
Realism
Functional
Photoreal
Physical
Non-visual
Ferwerda 2003

14. Raskar | MIT Media Lab
Realism
Functional
Photoreal
Physical
Non-visual
Archaic neurosc +
perception
Few open datasets

15. Raskar | MIT Media Lab
Head-mounted Display Projectors
© Andrei State
Multi-Mode Augmented Reality

16. Raskar | MIT Media Lab
VienViewer
Near-IR camera locates subcutaneous veins
Project ‘veins’ onto skin.
Coaxial IR camera
+ Projector

17. Raskar | MIT Media Lab
hand-held
display
spatial
see-through
display
head-
mounted
display
retinal
display
Head-attached Body-attached spatial
projector
Spatial Augmented Reality
1998
Multi-Mode AR
Real
Object

18. Raskar | MIT Media Lab
Virtual Reflectance Virtual Illumination
InteractionVirtual Motion
ShaderLamps
1997 - 2000

19. Raskar | MIT Media Lab
Virtual Reflectance Virtual Illumination
Lamps for Colorfully Shading
the World
1997 - 2000

20. Raskar | MIT Media Lab
Shader Lamps on a Saturn

21. Raskar | MIT Media Lab
Guerilla Marketing
FHM Magazine

22. Raskar | MIT Media Lab
Augmented Reality (AR)
World Input Process Output World
Camera,
Location
Sensors
Identify,
Track,
Abstract
Display,
3D Overlay
Interaction
MultiModal MultiFormat
IoT
Training
DataSets
Perception
Ergo, Hygiene
Ethics
Gestural Language
EyeGlasses
Laziness
Authoring
IoT

26. Raskar | MIT Media Lab
“People are lazy”
Rob Lindeman, Benjamin Lok, Bill Baxter
Cognitive or Physical Exertion
Fatigue = Reduced Usage

27. Raskar | MIT Media Lab
Gesture Lexicon
Phone AR: No gesture
No Controllers
Natural Objects
Open Data Sets

28. Raskar | MIT Media Lab
Augmented Reality (AR)
World Input Process Output World
Camera,
Location
Sensors
Identify,
Track,
Abstract
Display,
3D Overlay
Interaction
MultiModal MultiFormat
IoT
Training
DataSets
Perception
Ergo, Hygiene
Ethics
Gestural Language
EyeGlasses
Laziness
Authoring
IoT

29. Phoropter

30. Pamplona, Mohan, Oliveira, Raskar ‘NETRA’
Eyeglasses Prescription on Headset

31. Raskar | MIT Media Lab
Mk Haley
Disney Imagineering
“My peers working with VR get frustrated with
women "ruining" the gear
because of mascara, foundation, or moisturizer on their face. Or
women being "difficult"
if they have ponytails or other hairstyles..

32. Raskar | MIT Media Lab
‘Glassholes’ Ethics
Info Asymmetry
Visual filter bubbles
‘Turn off sensors’
Eye tracking

33. Raskar | MIT Media Lab
Multi-ModeMulti-Format
IoT
Perception
Ergo,
Hygiene
Ethics
Gestural Language

34. Raskar | MIT Media Lab
Poor Man’s Palace

35. Raskar | MIT Media Lab
Poor Man’s Palace
Augment the world, projectors and proxy geometry, controlling light every millimeter at every millisecond.

36. Fake World

37. Raskar | MIT Media Lab
Acknowledgements
• MERL
• Amit Agrawal, Ashok Veeraraghavan,
Ankit Mohan, Jack Tumblin, Jeroen van
Baar, Paul Beardsley, Remo Ziegler,
Thomas Willwacher, Srinivas Rao, Cliff
Forlines, Paul Dietz, Joe Marks, Darren
Leigh
• Office of the Future grp UNC Chapel Hill
• Greg Welch, Kok-lim Low, Deepak
B’padhyay, Aditi Majumder,
Michael Brown, Ruigang Yang
• Henry Fuchs, Herman Towles
• Wei-chao Chen
• Camera Culture Group, MIT
Media Lab
• John Werner
• Marco Jacobs
• Mark Billinghurst, U S Australia
• Sameer Rawal, DISQ, India
• Tim Smith, Sevaleader
• Alvaro Cassinelli, U of Tokyo
• Benjamin Lok, U of Florida
• Bill Baxter
• Rob Lindenman,
• Brett Jones
• Mk Haley, Disney

38. Raskar | MIT Media Lab
Challenges in AR
World Input Process Output World
Camera,
Location
Sensors
Identify,
Track,
Abstract
Display,
3D Overlay
Interaction
MultiModal MultiFormat
IoT
Training
DataSets
Perception Ergo, Hygiene
Ethics
Gestural Language
EyeGlasses
Laziness
Authoring
IoT
Free Book: raskar.info/book
Slideshare.net/cameraculture