Note: Update presentation available at: https://www.slideshare.net/MichaelHoffman47/object-theory-mixed-reality-use-cases-and-challenges-122165854 Object Theory presents 17 use cases identified as delivering quantifiable business value with mixed reality as well as some of the current challenges. Object Theory has already delivered 11 of these use cases to a variety of F1000 clients. Michael Hoffman, Object Theory co-founder, presented this at AWE 2018 in Santa Clara. Once the video of this presentation is available, I will post it to LinkedIn.

1. Mixed Reality
Use Cases and Challenges

2. As a trusted advisor,
Object Theory
helps F1000 clients
strategically identify, design and build
XR solutions
that provide real business value
utilizing today’s XR devices

3. Mixed Reality Benefits
• Hands-free – Increase information access efficiency while performing tasks
• Shared 3D context – Accurately convey focus & intent to reduce misunderstandings
• Stereoscopic vision – Improve comprehension of subject matter
• True scale – Accurately experience & evaluate spatial relationships and appearance
• Spatial mapping – Reduce mistakes by proximity of information to objects/environment
• 3D sound – Increase sense of presence, and accurately direct attention to focal points
• Data visualization – Improve comprehension of complex geospatial data sets
• High fidelity 3D objects –
• Reduce need for expensive 3D print & paint
• Reduce dependency on expensive equipment and supplies
• Safely train for dangerous tasks
• 3D environments – Leverage spatial memory to improve retention & navigation
Quantifiable business value from these unique features & benefits:
© 2018 object theory, llc

4. Mixed Reality Use Cases
Remote Collaboration
Product Design
Annotated Environment
Premium Selling
Soft Skills Training
Hard Skills Training
Space Planning
Situation Planning
Experience Invisible Data
Medical Imaging
Remote Expert
Infinite Screen
Education
Expert Coaching
Automated Guidance
Art and Creativity
Entertainment
© 2018 object theory, llc

5. Remote Collaboration
The Business Challenge
When dispersed teams collaborate on 3D problems, it’s
difficult to establish an accurate shared context and convey
intent with existing 2D tools such as screen sharing, which
can result in costly mistakes.
Mixed Reality Solution
Collaborators feel a sense of presence, plus virtual
geospatial annotations and markers can accurately convey
intent to establish a more accurate shared context resulting
in more informed decisions and fewer mistakes due to
miscommunication.
Why Mixed Reality?
• Gaze indicates focus and intent
• 3D Spatial sound provides cues and sense of presence
• Stereoscopic vision improves comprehension
• Geospatial annotations provide critical context
• Ability to see content while conversing/collaborating
Potential Challenges
• 3D audio chat requires reliable internet connection
• Geometry complexity
• Outdoor use / safety compliance requirements
• WiFi Security (for Defense use cases)
© 2018 object theory, llc

6. Remote Collaboration
The Business Challenge
When dispersed teams collaborate on 3D problems, it’s
difficult to establish an accurate shared context and convey
intent with existing 2D tools such as screen sharing, which
can result in costly mistakes.
Mixed Reality Solution
Collaborators feel a sense of presence, plus virtual
geospatial annotations and markers can accurately convey
intent to establish a more accurate shared context resulting
in more informed decisions and fewer mistakes due to
miscommunication.
Why Mixed Reality?
• Gaze indicates focus and intent
• 3D Spatial sound provides cues and sense of presence
• Stereoscopic vision improves comprehension
• Geospatial annotations provide critical context
• Ability to see content while conversing/collaborating
Potential Challenges
• 3D audio chat requires reliable internet connection
• Geometry complexity
• Outdoor use / safety compliance requirements
• WiFi Security (for Defense use cases)
© 2018 object theory, llc

7. Product Design
The Business Challenge
When designing products, it is difficult to understand how
all of its components interplay and how it will appear as a
final product. Furthermore, efficiently collaborating
between different design disciplines to find the best
outcomes is challenging using existing 2D means.
Mixed Reality Solution
Mixed Reality makes it possible for design collaborators
to establish an accurate shared context of a product
under design for efficient, accurate decision making and
faster iteration cycle times.
Why Mixed Reality?
• Stereoscopic vision
• Accurate, true scale visualization of product
• Ability to see content while conversing/collaborating
• Efficient ability to try alternative designs
Potential Challenges
• Color fidelity not as good as calibrated monitors
• Large model geometries
© 2018 object theory, llc

8. Product Design
The Business Challenge
When designing products, it is difficult to understand how
all of its components interplay and how it will appear as a
final product. Furthermore, efficiently collaborating
between different design disciplines to find the best
outcomes is challenging using existing 2D means.
Mixed Reality Solution
Mixed Reality makes it possible for design collaborators
to establish an accurate shared context of a product
under design for efficient, accurate decision making and
faster iteration cycle times.
Why Mixed Reality?
• Stereoscopic vision
• Accurate, true scale visualization of product
• Ability to see content while conversing/collaborating
• Efficient ability to try alternative designs
Potential Challenges
• Color fidelity not as good as calibrated monitors
• Large model geometries
© 2018 object theory, llc

9. Annotated Environment (BIM / IoT)
The Business Challenge
BIM data and environmental sensor data is collected by
the terabytes, yet most of that data is never utilized. Only
a fraction of the potential value is extracted.
Mixed Reality Solution
With ML/AI + MR, the collected data can be analyzed for
patterns an then the insights can be presented
geosnatially right where an issues need addressing.
Further, the relationships between the metadata and the
physical plant are much easier to glean from observation.
Why Mixed Reality?
• Virtual on Physical
• Ability to see data while performing tasks
• Stereoscopic vision for better insights
Potential Challenges
• Not recommended yet for outdoor use cases
• Need to use markers when navigating large facilities.
© 2018 object theory, llc

10. Annotated Environment (BIM / IoT)
The Business Challenge
BIM data and environmental sensor data is collected by
the terabytes, yet most of that data is never utilized. Only
a fraction of the potential value is extracted.
Mixed Reality Solution
With ML/AI + MR, the collected data can be analyzed for
patterns an then the insights can be presented
geosnatially right where an issues need addressing.
Further, the relationships between the metadata and the
physical plant are much easier to glean from observation.
Why Mixed Reality?
• Virtual on Physical
• Ability to see data while performing tasks
• Stereoscopic vision for better insights
Potential Challenges
• Not recommended yet for outdoor use cases
• Need to use markers when navigating large facilities.
© 2018 object theory, llc

11. Premium Selling
The Business Challenge
It is difficult to sell expensive products, services and
experiences that are hard to put in front of a client or
hard to explain in detail.
Mixed Reality Solution
Telling the rich story of an offering in 3D makes it more
personal, relatable, understandable and memorable
Why Mixed Reality?
• Stereoscopic vision to better visualize
• Alignment to physical to reveal hidden features
• Spatial story telling is more compelling and
memorable
Potential Challenges
• It takes 5 - 15 minutes to acclimate to mixed reality
© 2018 object theory, llc

12. Premium Selling
The Business Challenge
It is difficult to sell expensive products, services and
experiences that are hard to put in front of a client or
hard to explain in detail.
Mixed Reality Solution
Telling the rich story of an offering in 3D makes it more
personal, relatable, understandable and memorable
Why Mixed Reality?
• Stereoscopic vision to better visualize
• Alignment to physical to reveal hidden features
• Spatial story telling is more compelling and
memorable
Potential Challenges
• It takes 5 - 15 minutes to acclimate to mixed reality
© 2018 object theory, llc

13. Soft skills training
The Business Challenge
It is expensive to train soft skills that require
understanding nuance and adept reaction to varying
external stimuli and behaviors.
Mixed Reality Solution
The combination of AI+MR reduces training cost by
making it possible to recreate realistic interactive soft skill
scenarios, automate the AI/ML assessment of
performance and generation of precise feedback.
Why Mixed Reality?
• Stereoscopic vision for more realistic scenarios
• Alignment to physical further increases realism
• Gaze indicates trainee focus and intent
• Positional info, camera and mics feed data to cognitive
services
• Hands free enables using actual soft skill devices
Potential Challenges
• No significant challenges
© 2018 object theory, llc

14. Soft skills training
The Business Challenge
It is expensive to train soft skills that require
understanding nuance and adept reaction to varying
external stimuli and behaviors.
Mixed Reality Solution
The combination of AI+MR reduces training cost by
making it possible to recreate realistic interactive soft skill
scenarios, automate the AI/ML assessment of
performance and generation of precise feedback.
Why Mixed Reality?
• Stereoscopic vision for more realistic scenarios
• Alignment to physical further increases realism
• Gaze indicates trainee focus and intent
• Positional info, camera and mics feed data to cognitive
services
• Hands free enables using actual soft skill devices
Potential Challenges
• No significant challenges
© 2018 object theory, llc

15. Hard skills training
The Business Challenge
It is expensive to train skills to a large workforce that
require learning a wide variety of steps and operations,
particularly in high-turnover environments.
Mixed Reality Solution
Creation of self-guided, interactive skills training tutorials
substantially reduces the labor cost of repeatedly training
the same skill.
Why Mixed Reality?
• Hands free
• Stereoscopic vision for better, faster comprehension
• Alignment to physical enables practicing on real
equipment
• Information close to where you need it
Potential Challenges
• Creating 3D interactive content for each skill can be
expensive
© 2018 object theory, llc

16. Hard skills training
The Business Challenge
It is expensive to train skills to a large workforce that
require learning a wide variety of steps and operations,
particularly in high-turnover environments.
Mixed Reality Solution
Creation of self-guided, interactive skills training tutorials
substantially reduces the labor cost of repeatedly training
the same skill.
Why Mixed Reality?
• Hands free
• Stereoscopic vision for better, faster comprehension
• Alignment to physical enables practicing on real
equipment
• Information close to where you need it
Potential Challenges
• Creating 3D interactive content for each skill can be
expensive
© 2018 object theory, llc

17. Space Planning
The Business Challenge
Experiencing the feel and function of a space
under design is a critical step in the design
process to reduce mistakes, yet is so costly to
achieve that it’s rarely done.
Mixed Reality Solution
Mixed Reality makes it possible to efficiently
experience and even modify a non-existent
room design at true scale.
Why Mixed Reality?
• True size and scale
• Stereoscopic vision
• Ability to see content while conversing/collaborating
• Align virtual changes to physical environment
Potential Challenges
• True scale immersive scenarios can benefit from wider
FOV, particularly for new users
© 2018 object theory, llc

18. Space Planning
The Business Challenge
Experiencing the feel and function of a space
under design is a critical step in the design
process to reduce mistakes, yet is so costly to
achieve that it’s rarely done.
Mixed Reality Solution
Mixed Reality makes it possible to efficiently
experience and even modify a non-existent
room design at true scale.
Why Mixed Reality?
• True size and scale
• Stereoscopic vision
• Ability to see content while conversing/collaborating
• Align virtual changes to physical environment
Potential Challenges
• True scale immersive scenarios can benefit from wider
FOV, particularly for new users
© 2018 object theory, llc

19. Situation Planning
The Business Challenge
Planning and monitoring complex geospatial scenarios
demands comprehensive understanding of complex
information across a dispersed team of experts. It is
difficult to provide all relevant information to create a
comprehensive and accurate shared context.
Mixed Reality Solution
Mixed reality enables effective collaboration across
dispersed teams while interacting with a wide variety of
geospatially accurate 3D information and supplementary
2D information.
Why Mixed Reality?
• Geospatially accurate information
• Stereoscopic vision
• Ability to see content while conversing/collaborating
Potential Challenges
• Content creation can be expensive
• Geometry complexity
• WiFi security (for Defense use cases)
© 2018 object theory, llc

20. Situation Planning
The Business Challenge
Planning and monitoring complex geospatial scenarios
demands comprehensive understanding of complex
information across a dispersed team of experts. It is
difficult to provide all relevant information to create a
comprehensive and accurate shared context.
Mixed Reality Solution
Mixed reality enables effective collaboration across
dispersed teams while interacting with a wide variety of
geospatially accurate 3D information and supplementary
2D information.
Why Mixed Reality?
• Geospatially accurate information
• Stereoscopic vision
• Ability to see content while conversing/collaborating
Potential Challenges
• Content creation can be expensive
• Geometry complexity
• WiFi security (for Defense use cases)
© 2018 object theory, llc

21. Experience Invisible Data
The Business Challenge
Data analysis of 3D data sets and simulations of
geospatial information on 2D screens makes it difficult to
comprehend the full extent of insights that can be
gained.
Mixed Reality Solution
With Mixed Reality, simulations and visualizations of
collected sensor data or simulations can be played and
animated geospatially and can be aligned with physical
objects related to the data.
Why Mixed Reality?
• Stereoscopic vision
• 3D in 3D
• Align virtual to physical
• Ability to see content while conversing/collaborating
• Experience the inaccessible
Potential Challenges
• No major challenges
© 2018 object theory, llc

22. Experience Invisible Data
The Business Challenge
Data analysis of 3D data sets and simulations of
geospatial information on 2D screens makes it difficult to
comprehend the full extent of insights that can be
gained.
Mixed Reality Solution
With Mixed Reality, simulations and visualizations of
collected sensor data or simulations can be played and
animated geospatially and can be aligned with physical
objects related to the data.
Why Mixed Reality?
• Stereoscopic vision
• 3D in 3D
• Align virtual to physical
• Ability to see content while conversing/collaborating
• Experience the inaccessible
Potential Challenges
• No major challenges
© 2018 object theory, llc

23. Medical Imaging
The Business Challenge
Surgeons are constantly evaluating 3D problems on 2D
screens. Missing critical insights and longer assessment
times can lead to worse outcomes for patients.
Mixed Reality Solution
Hands-free access to all relevant 2D and 3D information
exactly where it’s needed enables more accurate and
efficient assessment for better outcomes.
Why Mixed Reality?
• Hands free
• Stereoscopic vision for better, faster insights
• Alignment to physical reduces errors
• Information proximity without physical interference
• Instant access to vast amounts of 2D & 3D information
Potential Challenges
• Accurate alignment to body below 1-2mm difficult
• Regulatory requirements for operating room
• Weight and comfort for long-term use
© 2018 object theory, llc

24. Medical Imaging
The Business Challenge
Surgeons are constantly evaluating 3D problems on 2D
screens. Missing critical insights and longer assessment
times can lead to worse outcomes for patients.
Mixed Reality Solution
Hands-free access to all relevant 2D and 3D information
exactly where it’s needed enables more accurate and
efficient assessment for better outcomes.
Why Mixed Reality?
• Hands free
• Stereoscopic vision for better, faster insights
• Alignment to physical reduces errors
• Information proximity without physical interference
• Instant access to vast amounts of 2D & 3D information
Potential Challenges
• Accurate alignment to body below 1-2mm difficult
• Regulatory requirements for operating room
• Weight and comfort for long-term use
© 2018 object theory, llc

25. Remote Expert
The Business Challenge
Flying subject matter experts onsite to support field
operations is expensive and requires too many experts on
staff for peak load.
Mixed Reality Solution
A smaller team of centralized experts can effectively assist
many field technicians as needed at lower travel and labor
cost.
Why Mixed Reality?
• Hands free enables performing tasks while being assisted
• Geospatial indicators/annotations increases accuracy
• Stereoscopic vision increases comprehension
Potential Challenges
• Requires reliable internet connection for audio & video
• Safety compliance requirements
• Outdoor use
• Safety compliance requirements
• WiFi security (for Defense use cases)
© 2018 object theory, llc

26. Remote Expert
The Business Challenge
Flying subject matter experts onsite to support field
operations is expensive and requires too many experts on
staff for peak load.
Mixed Reality Solution
A smaller team of centralized experts can effectively assist
many field technicians as needed at lower travel and labor
cost.
Why Mixed Reality?
• Hands free enables performing tasks while being assisted
• Geospatial indicators/annotations increases accuracy
• Stereoscopic vision increases comprehension
Potential Challenges
• Requires reliable internet connection for audio & video
• Safety compliance requirements
• Outdoor use
• Safety compliance requirements
• WiFi security (for Defense use cases)
© 2018 object theory, llc

27. Infinite Screen
The Business Challenge
Many tasks require analyzing huge amounts of 2D and 3D
information. We are constrained by the realistic number
of pixels we can surround ourselves with.
Mixed Reality Solution
Mixed Reality enables augmenting existing screen real
estate with a nearly infinite amount of additional 2D
panels as well as 3D content for a vast increase in
available information for analysis and decision making.
Why Mixed Reality?
• Can interact with physical and virtual world
simultaneously
• Not constrained by limits of 2D monitors
Potential Challenges
• Today’s devices don’t yet run the sophisticated apps
professionals depend on.
© 2018 object theory, llc

28. Infinite Screen
The Business Challenge
Many tasks require analyzing huge amounts of 2D and 3D
information. We are constrained by the realistic number
of pixels we can surround ourselves with.
Mixed Reality Solution
Mixed Reality enables augmenting existing screen real
estate with a nearly infinite amount of additional 2D
panels as well as 3D content for a vast increase in
available information for analysis and decision making.
Why Mixed Reality?
• Can interact with physical and virtual world
simultaneously
• Not constrained by limits of 2D monitors
Potential Challenges
• Today’s devices don’t yet run the sophisticated apps
professionals depend on.
© 2018 object theory, llc

29. Education
The Business Challenge
It is difficult and expensive to teach concepts and
techniques that consume expensive supplies, depend on
expensive equipment, or can be dangerous.
Mixed Reality Solution
For at least the early stages of the learning process,
simulating concepts and techniques is a less dangerous
and expensive alternative, and can reduce dependency
on access to expensive equipment.
Why Mixed Reality?
• Hands free
• Stereoscopic vision for better, faster comprehension
• Alignment to physical enables practicing with some or
all physical equipment, or even props.
• Information close to where you need it
• Instant access to vast amounts of 2D & 3D information
• Ability to see content while conversing/collaborating
Potential Challenges
• Creating a large variety of interactive 3D content can
be expensive.
© 2018 object theory, llc

30. Education
The Business Challenge
It is difficult and expensive to teach concepts and
techniques that consume expensive supplies, depend on
expensive equipment, or can be dangerous.
Mixed Reality Solution
For at least the early stages of the learning process,
simulating concepts and techniques is a less dangerous
and expensive alternative, and can reduce dependency
on access to expensive equipment.
Why Mixed Reality?
• Hands free
• Stereoscopic vision for better, faster comprehension
• Alignment to physical enables practicing with some or
all physical equipment, or even props.
• Information close to where you need it
• Instant access to vast amounts of 2D & 3D information
• Ability to see content while conversing/collaborating
Potential Challenges
• Creating a large variety of interactive 3D content can
be expensive.
© 2018 object theory, llc

31. Expert Coaching
The Business Challenge
Improving a fine motor skill such as the perfect golf swing
is extremely difficult and requires deep expertise.
Mixed Reality Solution
IoT sensors plus mixed reality enables accurate capture
and visualization of a wide variety of information for more
accurate assessment and improvement of technique.
Why Mixed Reality?
• Hands free
• Alignment to physical to see relationships and variance
• 3D visualization for accurate assessment and
comparison of 3D movements
Potential Challenges
• Outdoor use not recommended
© 2018 object theory, llc

32. Expert Coaching
The Business Challenge
Improving a fine motor skill such as the perfect golf swing
is extremely difficult and requires deep expertise.
Mixed Reality Solution
IoT sensors plus mixed reality enables accurate capture
and visualization of a wide variety of information for more
accurate assessment and improvement of technique.
Why Mixed Reality?
• Hands free
• Alignment to physical to see relationships and variance
• 3D visualization for accurate assessment and
comparison of 3D movements
Potential Challenges
• Outdoor use not recommended
© 2018 object theory, llc

33. Automated Guidance
The Business Challenge
Utilizing only physical props, providing all useful
information to guide an audience through an experience
creates traffic bottlenecks, unwanted visual clutter and
limits storytelling possibilities.
Mixed Reality Solution
Mixed reality increases expressiveness and efficiency
while eliminating bottlenecks. Any amount of interactive
2D, 3D and spatial audio content can be situated
geospatially around points of interest to convey useful
information in compelling ways that maximize
expressiveness, comprehension and memory.
Why Mixed Reality?
• Stereoscopic vision for better, faster comprehension
• Information proximity & alignment increases context
• Hands free
• Instant access to vast amounts of 2D & 3D information
Potential Challenges
• Outdoor use not recommended
• Spatial mapping limits may require ID targets
© 2018 object theory, llc

34. Automated Guidance
The Business Challenge
Utilizing only physical props, providing all useful
information to guide an audience through an experience
creates traffic bottlenecks, unwanted visual clutter and
limits storytelling possibilities.
Mixed Reality Solution
Mixed reality increases expressiveness and efficiency
while eliminating bottlenecks. Any amount of interactive
2D, 3D and spatial audio content can be situated
geospatially around points of interest to convey useful
information in compelling ways that maximize
expressiveness, comprehension and memory.
Why Mixed Reality?
• Stereoscopic vision for better, faster comprehension
• Information proximity & alignment increases context
• Hands free
• Instant access to vast amounts of 2D & 3D information
Potential Challenges
• Outdoor use not recommended
• Spatial mapping limits may require ID targets
© 2018 object theory, llc

35. Art and Creativity
The Business Challenge
Monetizing entertainment is becoming increasingly
difficult. Theater and sports arena attendance continues
to drop.
Mixed Reality Solution
Mixed Reality adds interactivity and extends the average
time spent consuming entertainment. This increases the
opportunities to monetize it.
Why Mixed Reality?
• Can augment existing means of consuming
entertainment.
• Can move and interact with environment while
experiencing content.
Potential Challenges
• MR not yet a ubiquitous consumer device.
© 2018 object theory, llc

36. Art and Creativity
The Business Challenge
Monetizing entertainment is becoming increasingly
difficult. Theater and sports arena attendance continues
to drop.
Mixed Reality Solution
Mixed Reality adds interactivity and extends the average
time spent consuming entertainment. This increases the
opportunities to monetize it.
Why Mixed Reality?
• Can augment existing means of consuming
entertainment.
• Can move and interact with environment while
experiencing content.
Potential Challenges
• MR not yet a ubiquitous consumer device.
© 2018 object theory, llc

37. Entertainment
The Business Challenge
Monetizing entertainment is becoming increasingly
difficult. Theater and sports arena attendance continues
to drop.
Mixed Reality Solution
Mixed Reality adds interactivity and extends the average
time spent consuming entertainment. This increases the
opportunities to monetize it.
Why Mixed Reality?
• Can augment existing means of consuming
entertainment.
• Can move and interact with environment while
experiencing content.
Potential Challenges
• MR not yet a ubiquitous consumer device.
© 2018 object theory, llc

38. Entertainment
The Business Challenge
Monetizing entertainment is becoming increasingly
difficult. Theater and sports arena attendance continues
to drop.
Mixed Reality Solution
Mixed Reality adds interactivity and extends the average
time spent consuming entertainment. This increases the
opportunities to monetize it.
Why Mixed Reality?
• Can augment existing means of consuming
entertainment.
• Can move and interact with environment while
experiencing content.
Potential Challenges
• MR not yet a ubiquitous consumer device.
© 2018 object theory, llc

39. Addressing Today’s Challenges
• Safety: 3rd Party HoloLens hard hats (Trimble)
• Typing: Device assisted authentication
• Placement/Location:
• Fiducial markers
• Manual logical alignment
• Global anchor sharing
• Attention: 3D audio, narrative and visual cues
• Static 3D Content:
• Expert manual optimization
• Level-of-detail (LODs)
• Smart use of geometry culling
• FOV:
• Start small for context, then enlarge
• Attention directors and audio cues
• Dynamic 3D Content: Simplygon automation
• Material fidelity: Hand optimized shaders
• Input:
• Hand controllers and clickers
• Smart phones as companion input devices
• All day:
• USB battery packs
• Device swap
• Spectator View for professional videos / photos
• Spatial mapping / tracking:
• Clear out old spaces
• Guide user through good room scan
• Setting late stage reproduction (LSR) plane
• Environment: Comprehensive client training
How Object Theory addresses today’s MR challenges:
© 2018 object theory, llc

40. With thoughtful experience design,
diligent optimization
and smart engineering,
today’s mixed reality devices
deliver real business value
for a wide variety of use cases.

41. Thank you!
Michael Hoffman
Co-founder / Partner
Object Theory
michael@objecttheory.com
linkedin.com/in/mthoffman
@objecttheory