For the full video of this presentation, please visit: http://www.embedded-vision.com/platinum-members/amd/embedded-vision-training/videos/pages/may-2016-embedded-vision-summit For more information about embedded vision, please visit: http://www.embedded-vision.com Allen Rush, Fellow at AMD, presents the "How Computer Vision Is Accelerating the Future of Virtual Reality" tutorial at the May 2016 Embedded Vision Summit. Virtual reality (VR) is the new focus for a wide variety of applications including entertainment, gaming, medical, science, and many others. The technology driving the VR user experience has advanced rapidly in the past few years, and it is now poised to proliferate into these applications with solid products that offer a range of cost, performance and capabilities. The next question is: how does computer vision intersect this emerging modality? Already we are seeing examples of the integration of computer vision and VR, for example for simple eye tracking and gesture recognition. This talk explores how we can expect more complex computer vision capabilities to become part of the VR landscape and the business and technical challenges that must be overcome to realize these compelling capabilities.

1. Copyright © 2016 AMD 1
How Computer Vision is Accelerating
the Future of Virtual Reality
Allen Rush
May 2, 2016

2. Copyright © 2016 AMD 2
• Virtual Reality — current view
• Computer Vision support in VR
• Future of VR with Advanced CV
• Challenges — Technical, Business
Agenda

3. Copyright © 2016 AMD 3
VR Has Come a Long way
• Experience primarily thru close
display and optics
• Immersion thru control of local view
& positional info
• Factors: resolution, optics
accommodation, vergence, frame
rate, dynamic range…
• 2016: ~2M users, 200,000 developers
• >260 startups + key players

4. Copyright © 2016 AMD 4
VR Experiences — Imagination to Immersion
Training and Simulation
Gaming
Big Data Visualization
Entertainment Virtual Social World Remote Presence
MedicineEducation

5. Photorealism
in VR
Full Presence
Progression of REAL-TIME Visual Experiences
Basic 2D
Rendering
Physically
Based
Rendering
Basic 3D
Rendering
Immersive
2D Displays
Shaders VR
Level of
Presence
Today

6. Copyright © 2016 AMD 6
VR + Camera, CV Basic Functions
Viewport Display vs Human FOV

7. Copyright © 2016 AMD 7
Functions and processes to produce VR:
Latency challenges

8. Copyright © 2016 AMD 8
• Elementary: Feedback for eye tracking, positional information
(X,Y,Z, pitch, roll, yaw, x,y,z motion- 9 DOF), extremity detection
(hands, feet, etc.)
• Fundamental: IR tracking, positioning, calibration
• Complex: deep learning training for user’s eyes expressions,
motion prediction
• Sensory fusion: combining visual, tactile, accelerometers to
model dynamic environment
Computer Vision in VR

9. Copyright © 2016 AMD 9
• “Rendering of a virtual human that can purposefully interact with a real person — for
example, through speech recognition, the generation of meaningful sentences, facial
expression, emotion, skin color and tone, and muscle and joint movements — is still beyond
the capabilities of real-time computer graphics and artificial intelligence,” write
neuroscientist Maria V. Sanchez-Vives and computer scientist Mel Slater
• Key improvements in: latency, resolution, FOV, color, dynamic range, focus management
• Key additions: eye tracking, surround vision capture, position accuracy, object detection and
tracking, 3D view angle
• Key Assumption: Speeds and feeds will improve, optics will improve, additions such as eye
tracking and immersive capture will develop as natural components of VR platform
• Emergence of VR/CV development community — SDKs, app development, user groups,
metrics, reviewers, etc.
Virtual Reality — Quickly evolving and
improving, but has a way to go…

10. Copyright © 2016 AMD 10
Example: Leap Motion “Virtual Hands”
• Start of Visual Projection + Visual Feedback
• Next Step: Complex rendering, recognition, depth, tracking, etc.

11. Copyright © 2016 AMD 11
• Application developers: rich collection of topics to choose from
• Military (long history), medical, games (obvious), cinema (new), social media,
education, immersive work environment
• Content creation: CGI, cameras, video composite
VR Application Space — Diverse and Innovative
VR Touch UIHTC Vive Prototype Demo VR Hockey

12. Copyright © 2016 AMD 12
• “The seat at the 50-yard line in the first or second
row, that might go for around $1,000 a ticket, but
what you could also do is sell that same seat for
$10 a ticket to 100,000 people.”
– Max Cohen, Oculus
• Seeing vs. experiencing: immersive and sensory
• Collaborative: Immersive telepresence
• CV: eye tracking, face reconstruction,
expression projection, etc.
Advanced VR Applications: Extending Basic
Ideas to Make Compelling Experiences
Virtual conferencing

13. Copyright © 2016 AMD 13
What Does Advanced VR Look Like?
Gaze detection, learning and prediction
Automotive safety: obstacles,
cues, dictionary detection
Once baseline VR is acceptable,
many new apps, features, ideas,
and functions can be developed

14. Copyright © 2016 AMD 14
• New solutions to old ideas
• Low latency rendering: creative
management of pipeline rendering
(ex: LiquidVR)
• Foveated rendering
• Sync of sensor, detection, overlay
• New Tools
• Stitching, mixing content,
workflow management
What Does it Take?
Videostitch panoramic workflow
Mix in: CV detection, recognition,
learning, tracking and tagging

15. VR Vision
To achieve the vision of
full presence you need
Scalable
CPUs, GPUs,
Accelerators
Sound
Other Senses
Sight
Sensory
Integration

16. Copyright © 2016 AMD 16
Business Opportunities and Ecosystem

17. Copyright © 2016 AMD 17
• Content Developers
• Obvious: game developers. 3D games natural transition to 3D VR
• VR cinema. 360o capture, reformat, render, projection to VR display
• UI: mouse/keyboard replacement
• Captured content mixed with graphics content
• CV overlays: detection, object tagging, tracking
• Platform SDKs: standard methods for development
• Testing, metrics, distribution, support
• Optimization: compilers, libraries, porting to new platforms
Ecosystem

18. Copyright © 2016 AMD 18
• Game titles: sell vs. rent vs. service?
• Collaborative: time on station? Server access
• 1+1+1=4 (cardboard + smartphone + camera = VR/CV)
• Embedded: system integration + services
Business Models

19. Copyright © 2016 AMD 19
Challenges Going Forward

20. Copyright © 2016 AMD 20
• Improve render and display experience
• Limit of human vision, based on foveal
cone spacing
• 2x210degx150@ 1pix/arc-min-> ~226Mpix
• At 90FPS ->~20Gpix/sec rendering speed:
settle for much less
• Need good optics, lens shading, CA, etc.
• Render complexity
• 1k-10k+(!) Ops/Pix ->100’s to1000’s GOPs
on GPU
Technical and Design Challenges —
Display and Graphics

21. Copyright © 2016 AMD 21
Technical Challenges — Camera and CV
• Example: Hand and Eye Tracking
• Detail capture, frame rate,
tracking, position, prediction
• New research
• Deep learning to customize
and train

22. Copyright © 2016 AMD 22
• Content Development: titles, stories, 360/180 surround, volume audio…
• Distribution: Gb of content- how to efficiently transport
• How to test, evaluate, use feedback in content creation
• Who can make money?
Challenges — Ecosystem and Business
Environment

23. Copyright © 2016 AMD 23
• VR — picking up as a business!
• Improvements in historical limitations
• Cost effective solutions
• Improvements in performance
• CV — accelerates emergence of wide variety of applications
• Common attributes — detection, tracking
• Unique attributes — accuracy, categories (medical vs. games, etc.)
• Ecosystem — SDK, user groups, content creators, reviewers,
• Business — new paradigms in entertainment, efficiency, capabilities
Summary

24. Copyright © 2016 AMD 24
• Alger, Mike, “Visual Design Methods for Virtual Reality” Sept 2015
• MacVey, Matthew “Exploring Future Reality”, NYC Media Lab, 2016
References