A talk from the Tools & Products Track at AWE USA 2017 - the largest conference for AR+VR in Santa Clara, California May 31- June 2, 2017. Chris Wren (WrenAR LLC): Beyond the Hands Commercial applications and games in AR and VR have primarily focused on interactions involving the hands. Whether using a controller like the HTC Vive, Oculus Touch to simulate hand positions and gestures, or using an optical gesture recognizer like LEAP Motion or USens, the majority of interactions we see are hand based. As we see more sophisticated peripherals arrive on the scene, what does the future look like for interactivity using other parts of your body for interactions. Tracked object peripherals, as well as utilizing more body parts including the feet and head to interact will become prominent this year. This moves the whole industry towards a deeper level of immersion and opens the possibility for many new types of interactions to take place. Imagine using your feet to juggle a soccer ball, bounce it off your head and then drill it into the back of the net, or pick up a hockey stick and actually do a slapshot complete with haptics and sounds to make it feel like the real thing. WrenAR has focused this past year on advanced human avatars which allow for this sophisticated and natural interaction with VR elements. This session will focus on the state of peripherals, the future of interactivity in VR space and lessons learned from working with these advanced avatars and developing interactions beyond the hands. http://AugmentedWorldExpo.com

1. Join the conversation @ARealityEvent & #AWE2017
Exploring the interactive boundaries of VR
Beyond the Hands
Chris Wren

2. WHAT’S THIS TALK ABOUT?
• Survey of Existing/Upcoming Sensory Tech for VR
• Showcase my work in this area / Lessons learned
• VR Avatars
• Full Body VR
• Mixed Reality
• Hand Tracking
• A little bit of VR Social
• Where are we going?

3. WHO AM I
Chris Wren
• 12 years professionally developing games (Microprose/EA/Namco)
• 9 years instructing Game Design at George Mason
• I do AR/VR research/consulting in Virginia: www.wrenar.com
• I work with an amazing AR/VR company: www.notiontheory.com

4. My First Job

5. “The Sims”
I shipped 7 Sims titles in 4 years.

6. Warhammer: Mark of Chaos

7. Inspirational signposts of what AR/VR interaction might be like

8. More real than real.
WHY ARE WE DOING THIS?

9. TAPPING THE SENSES

10. SIGHT
• Higher FOV
• Better Pixel Density
• Better Lenses
• Higher Framerate
• Beyond the Visual Spectrum
• Transferring to AR

11. FUTURE SIGHT
• Thermal
• IR
• Lightfield
• Retinal Projection
• Bypassing Optic Nerve?

12. SOUND
Ambisonics

13. TOUCH
Reach out, reach out and touch someone.
– 1980’s ad campaign from AT&T

14. HAPTIC GLOVES

15. NOTION THEORY GLOVES

16. VR TOUCH
Haptic Fingertips

17. RAINBOW JELLIES AND LEAP MOTION
Synesthesia

18. MONSTER SHOP AND USENS
Maybe we don’t need controllers…

19. TACTICAL HAPTICS
Strafe Haptics

20. CORNELL ORGANIC ROBOTICS
OmniPulse

21. SOFT ROBOTICS AND VR
Future Peripherals

22. UNLIMITEDHANDS
H2L Inc (China)

23. MUSCLE STIMULATION
Hasso-Plattner-Institut in Potsdam

24. WHERE IS THIS HEADED?

25. VIRTUAL HANDSHAKES

26. THE SUITS
The outside of the suit was covered with an
elaborate exoskeleton, a network of artificial tendons
and joints that could both sense and inhibit my
movements. Built into the inside of the suit was a
weblike network of miniature actuators that made
contact with my skin every few centimeters. These
could be activated in small or large groups for the
purpose of tactile simulation—to make my skin feel
things that weren’t really there. They could
convincingly simulate the sensation of a tap on the
shoulder, a kick to the shin, or a gunshot in the chest.
-Ernest Cline, Ready Player One

27. TESLA SUIT

28. SKINTERFACE
Student Project from London’s Royal College of Art

29. SUBPAC

30. HARDLIGHT VR
NullSpace VR

31. WHERE IS THIS HEADED?

32. VIRTUAL HUGS

33. SMELL

34. OLFACTORY BULB

35. FEEL REAL
Heat, Wind + Smells

36. NOSULUS RIFT
Not all smells are welcome in VR

37. BRAIN HACKED COOKIES
University of Tokyo

38. MOVEMENT
I spent the majority of my time in my Shaptic
Technologies HC5000 fully adjustable haptic chair. It
was suspended by two jointed robotic arms
anchored to my apartment’s walls and ceiling. These
arms could rotate the chair on all four axes, so when I
was strapped in to it, the unit could flip, spin, or
shake.
-Ernest Cline, Ready Player One

39. CABLE ROBOTS
FraunHofer IPA

40. GIMBALS

41. TREADMILLS

42. VIRZOOM

43. MARS BUS
• Made by Framestore
• Uses Unreal Engine
• Designed for real-world movement

44. RE-DIRECTED WALKING

45. VESTIBULAR STIMULATION

46. SCANNING

47. WALK AROUND 360 VIDEO
HYPE VR
Demoed by Intel at CES 2017

48. SHARED DATASETS
• Think: Google Maps
• Tracking Matrices
• Only updating that which has changed
• Confirming Data from multiple sources

49. BACKPACK VR
• Allowed Freedom of Movement
• Great with Redirected Walking
• NOT Good for Athletics: Jumping
• Made Layups, Running, Jumping Possible
• Wireless non-backpack solution determined

50. VR AVATAR
DEVELOPMENT

51. MY ORIGINAL VR AVATAR TEST
Kyle the Unity Robot(Just Head – No IK) - Oculus DK2 - May 2015

52. KYLE PLAYING BASKETBALL
Basketball Avatar Test – June 2016 (Steam VR IK Rigging)

53. KYLE JELLIES
Jelly Avatar Test – June 2016 (Unity IK Rigging)

54. THE UNCANNY VALLEY
Getting Away From The Robot

55. SCANNED-IN PEOPLE
Scanned Model from Sketchfab – Auto-rigged w/Mixamo (Unity IK Rigging)

56. HOVER JUNKERS IK
Using “Final IK” Rigging

57. SCANNED AVATAR + LEAP MOTION +
FOOT CONTROLLERS
1st Foot Rigging - August 2016

58. FULL BODY VR
• Running, Jumping in Real Environments
• Interacting with real tracked objects

59. ENTER THE FEET

60. STRAP/CLIP SHOE EXPERIMENTS

61. OVERSHOE PUCK SOLUTIONS

62. SOCCER DEMO

63. ANOTHER MIXED REALITY DEMO

64.
SOCIAL VR

65. MY VRML DAYS

66. ALTSPACE VR

67. BIGSCREEN VR
Hayden from (ConVRge)

68. VR CHAT

69. OCULUS SOCIAL

70. TENCENT VR HMD
Integration of WeChat – VR Social

71. V TIME

72. REC ROOM

73. PROJECT SANSAR
Linden Labs (Second Life)

74. HIGH FIDELITY
Philip Rosedale on stage Above

75. AR: EYE TRACKING

76. EYE TRACKING HMDS
SMI, Fove, Qualcomm

77. MAGIC LEAP
AR Lightfield

78. AVEGANT
First Standalone Headset to do Lightfield Synthesis

79. LOCATION BASED VR
Room Scale àHouse Scale à Warehouse Scale àArena Scale à
MR Ubiquity ~5 years

80. THE VOID
• Smell of Burnt Marshmallows
• Misting Effect for Sliming
• Vibrations in Haptic Vest and Gun
• Clever Use of Redirected Walking/Elevators
• Multiplayer

81. ZERO LATENCY
• LaserTag-Type Arena Focus
• System Limited To open warehouse setup
• PS3 Cameras and Move Controller Tracking

82. UNIVERSAL ORLANDO HAUNTED HOUSE
• Similar to Void with tracking
• Real Actors Involved
• Mixed Experience – Putting headset on as part of experience

83. IMAX VR LOS ANGELES
• Vive + Star VR Experiences
• Custom Star Trek, DC SuperHero, John Wick experiences
• Buy tickets to “rides”

84. END OF 2018 HARDWARE ANTE
BY 2018 VR/AR HMDs will need to have:
• Wireless (Tracking and Power)
• Mobile, not confined to a space
• Scanning in Real-Time & Sharing Scan Data
• Eye + Facial Gesture Tracking
• Predictive Interface
• 180+ Degrees FOV + 8K Resolution
• Small Form Factor (Thick Pair of Sunglasses)

85. THE STANDALONE HMDS OF 2018
HTC, Lenovo, Intel, Samsung, Asus, Qualcomm…
Apple?…Amazon?…Sony?...Avegant?...Magic Leap?

86. BEYOND THE BASELINE

87. THANK YOU
Questions?
Chris Wren
wwren@gmu.edu
chris@wrenar.com
chris@notiontheory.com
game.gmu.edu
www.wrenar.com
www.notiontheory.com

88. APPENDIX A
Sports Examples

89. BASKETBALL

90. SOCCER

91. APPENDIX B
Augmented Reality HMDS and USES

92. AUGMENTED REALITY
Recognition

93. WINDOWS HOLOGRAPHIC
MS not even talking about VR anymore - only Mixed Reality

94. INTEL PROJECY ALLOY
Intel tech is similar to Hololens - Real Sense

95. AR HMDS
Meta 2

96. AR HMDS
Hololens

97. AR HMDS
Daqri

98. AR HMDS
ODG

99. AR HMDS
Epson

100. AR HMDS
Atheer

101. AR HMDS
Occipital Bridge

102. AR HMDS
Vuzix

103. APPLE’S SECRET
EFFORT
“The next big thing, like the cellphone” - Tim Cook

104. SAMSUNG AR
Monitorless

105. AR: FEATURES
● Recognition
● Using Real World Data To Drive Graphical Experiences
● Integrating Graphics into Real World Environments

106. LYTRO
Capturing & Processing Ultra HD lightfield video

107. AR: HAND GESTURES
Ability to See and Distinguish Hands and Gestures

108. AR: LEGO
LEGO

109. AR: BOOKS &
MAGAZINES

110. AR: SHOPPING

111. ALIBABA BUY+