Giving a brief overview on Mixed Reality app development at Tokyo iOS Meetup in September 2017

1. Interactions in
Mixed Reality
...or what is mixed reality and
how can we make applications for it?
Bektur Ryskeldiev
Tokyo, Fall 2017

2. Outline
• About me
• What is Mixed Reality?
• Recent improvements in tracking technologies
• Emerging modes of interaction
• Developing MR Apps
• Further reading

3. About Me
• PhD candidate (3rd year) in Computer
Science at University of Aizu
• Specialized in Mixed Reality,
Multimedia Streaming, and Mobile
Computing
• Research Intern at NTT Network
Technologies Laboratory
• Team Leader at ACM SIGGRAPH
International Resources Committee &
Student Volunteer Program

4. What is Mixed Reality?
• Mixed Reality (MR) is a concept most famously known
from Reality-Virtuality continuum paper by Paul Milgram
and Fumio Kishino
• Includes different "blends" of reality and virtuality, e.g.,
Augmented Reality and Augmented Virtuality

5. What is Mixed Reality?
• Later on it was extended into three
continua:
• Extent of World Knowledge:
measures how well the system
understands the world around it
• Extent of Presence Metaphor:
measures the level of user
immersion
• Reproduction Fidelity:
measures the display fidelity of
a mixed reality system

6. Improvements in tracking
• Recent developments in mobile tracking systems open up a lot
of new possibilities
• New mobile hardware and software can understand more about
the world around it
• As a result the possible spatial interactions are getting more
sophisticated
Google
Cardboard
Marker-based
Tracking
Markerless
Tracking
Project
Tango
Microsoft
HoloLens
Markerless
Tracking

7. Emerging Interactions
With such advanced tracking available on almost any modern
mobile device, out of the box, developers can easily create new
forms of spatial interactions in Mixed Reality
Video from Android ARCore: https://www.youtube.com/watch?v=ttdPqly4OF8

8. Emerging Interactions
which may not look practical...

9. Emerging Interactions
but could potentially offer interesting forms
of interaction in real and virtual space
@heyadam @zachlieberman

10. Developing MR apps
• Easiest way is to use Unity due to
abundance of sample code and
relative ease of integration
• Popular mobile & cross-platform
libraries:
• Marker-based: Vuforia, Kudan,
AR.js (browser)
• Markerless: ARKit (iOS),
ARCore (Android)
• Project Tango for Tango-enabled
phones (featuring IR-tracker)

11. Developing MR apps: ARKit
• Native solution. Well-documented and
beginner-friendly.
• Can be used either in a native app or as
a Unity plugin
• Native app: uses SceneKit and SpriteKit
to render content, provides Metal
support for custom rendering engines
• Unity plugin: install ARKit plugin, drag
and drop objects for camera control to
setup ARKit
• Unity plugin also supports realtime
remote debugging
Video from: https://blogs.unity3d.com/2017/08/03/introducing-the-unity-arkit-remote/

12. Developing MR apps: Vuforia
• Before ARKit used to be the next best
solution for mobile AR on iOS
• Provides marker-based tracking, but you
get to define your own custom markers
• Widely used both in research and
commercial projects (e.g., MergeVR)
• Cross-platform: supports iOS, Android,
Windows (both tablets and HoloLens)
• Can be integrated either as a native
plugin (not recommended – lots of low-
level code with lacking documentation),
or as a Unity plugin (much easier to use).
Video from: https://www.youtube.com/watch?v=YaSWX6hqnDk

13. Developing MR apps: AR.js
• Run Mixed Reality apps in
your browser!
• Supports both iOS (from iOS
11) and Android (from 4.4.2)
• Uses javascript + WebGL to
run the apps
• Good performance: 60 FPS
even on old Android devices
Try now!
https://goo.gl/6piHqg

14. Further reading
• https://webxr.io/ – WebXR (AR/VR) newsletter
• https://vrscout.com/ – Latest AR/VR news aggregator
• http://voicesofvr.com/ – Voices of VR Podcast with insider interviews
• M.A. Magnor, Digital Representations of The Real World (book)
• Upcoming expos and conferences:
• Digital Contents Expo - October 2017, Tokyo
• ACM SIGGRAPH Asia 2018 – November 2018, Tokyo

15. Thank you!