AR/VR meetup of Open Tech Week by Sigma Software, Kharkiv 2019

2. Reality vs Virtuality
ARKit
DMITRY SHCHERBINA
iOS, OS X Senior Developer, TeamLead
DataArt Dnepr

3. Origins of AR
http://augmentedreality.com.ua/zak
hvatyvayushhaya-istoriya-
dopolnennoy-r/
The Master Key: An Electrical Fairy Tale, Founded Upon
the Mysteries of Electricity and the Optimism of Its
Devotees, 1901 novel by L. Frank Baum
1962 the Sensorama created
by Morton Heilig, along with
five short films for it to display.

4. Origins of AR
The Sword of Damocles, first head mounted display,
created in 1968 by computer scientist Ivan Sutherland with
the help of his student Bob Sproull.
First AR game - ARQuake,
2000 year.

5. Milgram's Reality-Virtuality
Continuum
Paul Milgram and Fumio Kishino defined in 1994 MR as “Any space between
the ends of the virtual continuum“.
The main MR (mixed reality) characteristics are next:
● It allows to combine real and virtual environments;
● It is an interactive technology and real-time technology;
● Can be registered in three dimensions.

6. Milgram's Reality-Virtuality
Continuum

7. Milgram's Reality-Virtuality
Continuum

8. Extended realities

9. Mixed reality scale and existing
projects

10. Username, be ready for VR/AR

11. Mobile SDKs

12. Vuforia
Vuforia is one of the most popular platforms. Pricey - $499 mobile
licence, or 99$/month cloud.
Platforms: Android, iOS, UWP and Unity Editor.
Main functionalities:
- recognition of the different types of visual objects (a box, cylinder, plane);
text and environments recognition;
- VuMark (a combination of picture and QR-code);
- Vuforia Object Scanner, helps to create object targets;
- Unity support.

13. Vuforia

14. EasyAR
Is a free and easy to use alternative to Vuforia.
Supported platforms: Android, iOS, UWP, Windows, Mac and Unity Editor.
Version 2.0 will include the following features:
- 3D Object Recognition;
- Environment perception;
- Cloud Recognition;
- Smart Glass Solution;
- App Cloud Packaging.

15. EasyAR

16. Google’s ARCore
Supported platforms: Android, Smart Glasses.
ARCore uses three technologies to integrate virtual content with the real
world:
- Motion tracking, allows to understand and track device’s position relative to
the world;
- Environmental understanding, allows the phone to detect the size and
location of flat horizontal surfaces like coffee table;
- Light estimation allows the phone to estimate the environment's current
lighting conditions.

17. AR for iOS users with ARKit!

18. Real Life

19. AR submodules

20. ARKit 1.0

21. ARKit

22. ARKit v1.0

23. ARKit Plane detection

24. How do you get 3D from a single
lens?
One camera captures overlapping
images of a moving object. Based
on this, the 3D shape of the object
is determined.
Two or more cameras acquire images from
different points of view. Then, the disparity
map of these images is calculated. Based
on this, the 3D shape of the object is
determined.

25. Depth API (iOS 10)

26. ARKit v 1.0 (iOS 11)

27. ARKit Quick Start

28. ARKit Quick Start
Basic requirements:
● OS X or Hackintosh, v10.13
● XCode 9
● iOS 11 or 12 beta (recommended)
● Apple device with Apple A9, A10, and A11
processors (iPhone 6S, SE or newer)
https://mobile-ar.reality.news/collection/apple-arkit/

29. AR Shooter
https://github.com/farice/ARShooter

30. Core ML
Core ML lets you integrate a broad variety of machine learning model types
into your app.
You can run machine learning models on the device so data doesn't need
to leave the device to be analyzed.

31. Apple Vision framework
Supported features:
● face tracking
● face detection
● landmarks
● text detection
● rectangle detection
● barcode detection
● object tracking
● image registration

32. ML + Vision
TODO add demo video
Inception v3 model. Detects the dominant objects present
in an image from a set of 1000 categories such as trees,
animals, food, vehicles, people, and more.

33. ARKit + ML + Vision
https://github.com/nahttoter/CoreML-in-ARKit

34. Vertical surface detection
Your app can see and place virtual objects on vertical surfaces,
and more accurately map irregularly shaped surfaces.
Real world images, such as signs, posters, and artwork can be
integrated into the AR experience, so your app can fill a
museum with interactive content.

35. ● Create an app that runs an AR session and uses
plane detection to place 3D content using
SceneKit.
Building Your First AR Experience
● Discover supporting concepts, features, and best
practices for building great AR experiences.
Understanding World Tracking in ARKit
● Scanning an objects to create AR Objects
https://developer.apple.com/documentation/arkit/scanning_and_detecting_3d
_objects
● Tips for making your own multiplayer games
using ARKit, SceneKit, and Swift. Sources
available!
SwiftShot: Creating a Game for Augmented Reality
From where to start. ARKit 2.0

36. ARKit v2.0 and SwiftShot
iOS 12 +

37. ARKit v2.0 and SwiftShot
=

38. SwiftShot Internals
● Establishing a shared coordinate space
● Networking and state sharing
● Physics
● Asset import and management
● Flag simulation
● Dynamic audio
605th session of WWDC 2018

39. Establishing a shared coordinate
space
● Device scans area, captures features
● Asks ARSession for world map
● Serializes to disk
Share over network
● Peer-to-peer network connection
● Encrypt data in flight
● UI guidance to aid relocalization

40. Establishing a shared connection flow
● Capture and Send the AR World Map
● Device starting the game creates a session,
starts advertising
● Other devices see session listed in menu
● User selects game to join
● Device sends request
● Advertising device accepts or denies
Once session set up, devices are peers in the
network,.UDP for transport were used in this
specific app

41. SwiftShot Physics

42. ARKit v2.0 and SwiftShot

43. ARKit v2.0 Resources
AR File Format Backed by Apple, Pixar and Adobe

44. ARKit Scanning and Detecting 3D
Objects
In iOS 12, you can create such AR experiences by enabling object detection in ARKit
https://developer.apple.com/documentation/arkit/scanning_and_detecting_3d_objects

45. Scanning tips
● Detailed, textured objects work better for detection than plain or reflective
objects.
● Object scanning and detection is optimized for objects small to fit on table.
● An object to be detected must have the same shape as the scanned reference
object. Rigid objects better than soft bodies or items that bend, twist, fold, or
otherwise change shape.
● Lighting conditions same to original scanning.
● High-quality object scanning requires peak device performance. But may be
evaluated at any supported

46. Awesome ARKit
● Projects
● AppStore
● Tutorials
● Resources

47. So what does it all mean?

48. 7 key drivers to deliver AR/VR to
mass market
● Mobility: AR can be used anywhere because it doesn’t block out the real world.
● Vision: AR and VR are all about vision.
● Immersion: Truly immersive experiences are magical, lifting the spirit and
compelling users to want more.
● Usability: Processing power, user comfort, reliable input.
● Flexibility: Easy switch for user between VR and AR.
● Wearability: Wearing VR helmet outside is weird.
● Affordability: People don’t like to buy extra devices.

49. Why ARKit is good
Pros:
● Free to use
● Low entrance level, native iOS support
● Lots of features available from box (Visual Inertial Odometry, Scene
understanding, Surface detection, Light estimation)
● Awesome performance, compatibility with Metal, Unity, Unreal, Scene Kit
● Multiplayer, shared AR experience
Cons:
● Errors gathered over the time
● Battery consumption
● Small screen size
● Not so precise

50. AR/VR in future
VR/ARrevenue($B)

51. Current Advertising Technology

52. HoloLens 2

53. Questions are welcomed
FB: Dmitry Shcherbyna

54. What to do next? Useful Links
● SwiftShot https://developer.apple.com/documentation/arkit/swiftshot_creating_a_game_for_augmented_reality
● ARKit Sources/Tuts/Apps https://github.com/olucurious/Awesome-ARKit
● VR History https://www.youtube.com/watch?v=43mA_ypfwKg, http://augmentedreality.com.ua/zakhvatyvayushhaya-istoriya-
dopolnennoy-r/
● Introducing ARKit by Apple https://developer.apple.com/arkit/
● Best AR SDK https://thinkmobiles.com/blog/best-ar-sdk-review/
● Comparing current alternatives https://www.imore.com/apple-arkit-vs-google-tango-vs-microsoft-hololens-augment-my-reality
● Made with AR http://www.madewitharkit.com/
● ARKit WWDC https://developer.apple.com/videos/play/wwdc2017/602/
● Google VR/AR https://vr.google.com/ https://developers.google.com/ar/discover/
● CoreML+ Vision https://developer.apple.com/documentation/vision/classifying_images_with_vision_and_core_ml
● Depth API https://developer.apple.com/videos/play/wwdc2017/507/
● Recent news https://augmented.reality.news/
● https://venturebeat.com/2018/04/13/sony-is-considering-new-psvr-livestreaming-and-mixed-reality-features/
● New things of Arkit 2.0 https://medium.com/vladimir-novick/new-cool-things-you-can-do-with-arkit-2-0-and-react-native-
29cae163849f
● ARKIT 101 https://mobile-ar.reality.news/collection/apple-arkit/

55. Presentation
https://www.facebook.com/shcherbinadmitriy