It's high time for augmented reality to be brought to a wider audience. In ABB, we know that it is not just a gimmick any more. However, with every innovative technology comes new challenges. In these slides, we show how to overcome them and deliver valuable products with Hololens and Unity.
Speakers:
Maciej Włodarczyk - ABB
Rafał Kielar - ABB
Watch the session on YouTube: https://youtu.be/QFsj8Pi_3Ho
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
How ABB shapes the future of industry with Microsoft HoloLens and Unity - Unite Copenhagen 2019
1. GenerativeArt—MadewithUnity
How ABB shapes the
future of industry with
Microsoft and Unity
Nick Davis
Senior Content Marketing
Manager
Unity
Nick Klingensmith
Mixed Reality Academy
Lead Engineer
Microsoft
Maciej Włodarczyk
IS Technology Expert
Chief Specialist
ABB
Rafał Kielar
AR/VR Developer
ABB
2. Agenda
— Microsoft: The future of AR with HoloLens 2
— ABB: Best practices for AR field application development
— Q&A
21. Improved skilling
4x improvement to service time among
employees who receive mixed reality training.
80% of companies improve collaboration across cross-
functional teams using mixed reality.
MIXED REALITY DRIVES TRANSFORMATIONAL BENEFITS
Efficiency and quality gains
25% boost in productivity, plus errors reduced to
effectively zero, with directions and data given in
mixed reality
27. ABB IS Innovation & Digital Scouting
— Technical CoE
— Digital Scouting
— IS Innovation
– Founded in 2014
– 8 AR projects
– Minimum Awesome Product
28. Interaction between realities:
— Ability to see hidden
/inaccessible elements
— Adapting holograms to
real life
Everyone is an expert:
— Remote support
— Perform on-site
actions without
additional preparation
Real life HUD:
— Instant access to real
life data
— Augmented guidelines
to help in real world
— Hands-free navigation
Why does AR stand out?
30. Problems to solve
Time-consuming trainings for field
operators
Tasks dangerous for operators if
done improperly
Procedures written on paper
Difficult communication between
on-site and control room operators
Task assigning and tracking
Difficult access to device
parameters
32. — Integration with
live device data
— Documents panel
(animation, video,
PDF)
— Remote assist panel
Our solution
33. Challenges we faced
— Location/device recognition
— UI design (safety concerns)
— Lack of internet connection on industrial sites
— Ergonomy of interactions
— Storing hologram positions
34. Designing Augmented Reality UI
Disappearing windows
Menu obscuring user's view
Auto docking/scaling
Clear way of navigation
Easily visible windows
Context menu on request
Adjustable windows
35. Designing Augmented Reality UI
Too small interactables for gesture/gaze
No accounting for FOV limitations
Voice commands as the only way of
interaction
Bigger elements if far interaction
Buttons close to context
Eye tracking for scrolling
Multiple interaction possibilities
36. Ergonomy of interactions
— Voice commands
— "One click experience"
— Window spawning
— Automate as much as
you can
— Instinctive design
— Testing involves your
whole body
37. • QR scan provides its
position and rotation in
Unity coordinate
system
• Not using web camera
• QR codes allow for
easy data gathering
• ARUCO codes allow to
precisely determine
location
• Many similar devices
on-site make it difficult
to identify them
individually using
image/model
recognition
• Can’t download specific
data
Device & Location recognition
Currently
Improvments
38. Storing hologram
positions and data
Storing
— World anchors
— Azure spatial anchors
Data
— Connecting anchors with
backend data
— Storing coordinates on backend
43. — Cannot run HoloLens 1 apps on
HoloLens 2
• UWP on x86 platform vs UWP on ARM
platform
• Breaking changes in HoloToolkit →
MixedRealityToolkit
— Some effort is required to prepare app
to run on HoloLens 2
Migrating projects
January 29, 2020Slide 43
Analysis and basic questions
44. Migrating projects
January 29, 2020Slide 44
Analysis and basic questions
Question Consideration
Which Unity version is used for project development? Migration between Unity versions can be big task (ex.
TextMeshPro asset) (latest editor version recommended)
Which scripting backend is used? .Net → deprecated in Unity 2018.2
IL2CPP → the only option in latest Unity version
What native platform plugins are used in project? Which platform given plugin supports?
x86? x64? ARM?
How are user interactions designed? On HoloLens 2 device the user interface should be closer to
the user to be able to use hand (finger) manipulation.
45. — Step 1: Download tools
• Install the most recent version of Windows 10
• Visual Studio 2019 (16.1 or higher)
• Windows SDK, build 18362 or later
• (optional) HoloLens Emulator
— Step 2: Update Unity Project
• Migrate project to latest version of
Unity and Update scene/project settings in Unity
Migrating projects
January 29, 2020Slide 45
Steps
è
è
46. — Step 3: Migrate to MRTKv2
• New input model
• Replace scene prefabs
• Replace interfaces in scripts
• Replace components
Migrating projects
January 29, 2020Slide 46
— Step 4: Compile for ARM
• Dependencies/plugins for ARM procesor
• Developing for ARM64 applications is generally
preferred as there is a material difference in
performance.