Augmented reality in mobile applications

Scalable eCommerce Platform Solutions
Egor Zubkov
ezubkov@griddynamics.com
Vitalii Ruban
vruban@griddynamics.com
Augmented Reality in
Mobile Apps

PRIVILEGED AND CONFIDENTIALDecember 9, 2017
1. VR and AR Introduction
2. AR eCommerce Apps on Market
3. ARKit
● Introducing ARKit Framework
● Device Compatibility
● Features
● Framework Components and API
4. Hands-on Experience in ARKit
5. 3D Object Transformation
Agenda

PRIVILEGED AND CONFIDENTIAL
VR and AR Introduction
December 9, 2017

So, what is the difference between VR, AR and MR?
I’m confused…?!?!
December 9, 2017

Wikipedia says
Virtual Reality (VR)
It is a computer technology that uses virtual reality headsets or multi-projected environments,
sometimes in combination with physical environments or props, to generate realistic images,
sounds and other sensations that simulate a user's physical presence in a virtual or imaginary
environment.
December 9, 2017

Augmented reality (AR)
It is a live direct or indirect view of a physical, real-world environment whose elements are
"augmented" by computer-generated or extracted real-world sensory input such as sound,
video, graphics, haptics or GPS data.
Wikipedia says
December 9, 2017

Mixed reality (MR) or sometimes referred to as Hybrid reality (HR)
It is the merging of real and virtual worlds to produce new environments and visualizations
where physical and digital objects co-exist and interact in real time. Mixed reality takes place
not only in the physical world or the virtual world, but is a mix of reality and virtual reality,
encompassing both augmented reality and augmented virtuality via immersive technology.
Wikipedia says
December 9, 2017

Nowadays the term “Mixed reality” seems to be fading out in favour of “Augmented reality”.
This means they are currently being used interchangeably.
AR vs. MR
AR == MR
December 9, 2017

AR eCommerce Apps on Market
December 9, 2017

IKEA Place
IKEA Place lets you virtually 'place' IKEA products in your space.
AR eCommerce Apps in App Store
December 9, 2017
In the e-commerce sector that AR is already starting to dramatically change what is possible
for people to experience. There are a lot of AR e-commerce apps on the market already. So,
we are able to load and try them right now.

Amazon
Amazon integrated AR in the app to let customers visualize online products in their own living
space, using their smartphone camera.
December 9, 2017

Wayfair
Wayfair added an augmented reality feature to its mobile shopping app, allowing shoppers to
view 3D images of furniture and décor projected onto its device’s view of a space.
December 9, 2017

ARKit
December 9, 2017

It is a new framework that allows you to easily create unparalleled augmented
reality experiences for iPhone and iPad. By blending digital objects and information
with the environment around you, ARKit takes apps beyond the screen, freeing
them to interact with the real world in entirely new ways.
Introducing ARKit Framework
December 9, 2017

iOS
ARKit Framework is introduced in iOS 11
Device Compatibility
Hardware
ARKit requires an iOS device with an A9 or later
processor, namely:
● iPhone: SE, 6s/6s Plus, 7/7 Plus, 8/8 Plus, X
● iPad: 5th gen, all Pro variations
Face Tracking feature
A face tracking is available only on iOS devices
with a front-facing TrueDepth camera.
● iPhone X only
December 9, 2017

ARKit Features
December 9, 2017

TrueDepth Camera
ARKit enable a capability for robust face tracking in augmented reality apps. Using the TrueDepth
camera, your app can detect the position, topology, and expression of the user’s face, all with high
accuracy and in real time, making it easy to apply live selfie effects or use facial expressions to
drive a 3D character.
ARKit Features
December 9, 2017

ARKit Features
Tracking position and orientation
ARKit uses Visual Inertial Odometry (VIO) to accurately track the world around it. VIO fuses camera
sensor data with Core Motion data. These two inputs allow the device to sense how it moves
within a room with a high degree of accuracy, and without any additional calibration.
December 9, 2017

ARKit Features
Environment understanding
ARKit can analyze the scene presented by the camera view and find horizontal planes in the room like
tables and floors. Also, the framework can detect Feature Points of objects and contrast surface in an
environment to provide the ability to build virtual reality on the scene.
December 9, 2017

ARKit Features
Note! ARKit doesn’t render the detected planes from the box. It has been achieved by SCNPlane
object of SceneKit framework with appropriate grid texture.
December 9, 2017

ARKit Features
Lighting estimation
ARKit also makes use of the camera sensor to estimate the total amount of light available in a scene,
its direction, ambient intensity and applies the correct amount of lighting to virtual objects.
guard let currentFrame = sceneView.session.currentFrame else {
return
}
let ligntEstimate: ARLightEstimate? = currentFrame.lightEstimate
...
December 9, 2017

ARKit Components and API
December 9, 2017

Framework Components and API
ARSKView
ARSKView class to create augmented reality experiences that position 2D elements in 3D space
within a device camera view of the real world.
ARSKViewDelegate extends protocols <SCNSceneRendererDelegate, ARSessionObserver>
Provide the ability:
● to manage nodes on a scene: whether it was added, deleted or updated;
● to perform operations at various times during the rendering: objects or per-frame game logic,
animations handling;
● to respond to changes in ARKit session status.
func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {
If let planeAnchor = anchor as? ARPlaneAnchor,
let faceAnchor = anchor as? ARFaceAnchor else {
// a new Plane or Face was detected
} else {
// a Feature Point was added
}
}
December 9, 2017

ARSession
This object coordinates the major processes that ARKit performs on your behalf to create an
augmented reality experience. These processes include reading data from the device's motion sensing
hardware, controlling the device's built-in camera, and performing image analysis on captured camera
images.
3 session types are presented in ARKit:
● AROrientationTrackingConfiguration - to track just device orientation (NOT position)
● ARWorldTrackingConfiguration - to track device position and rotation in the world, and planes
also
● ARFaceTrackingConfiguration - a configuration for running face tracking
December 9, 2017

@IBOutlet weak var sceneView: ARSCNView!
let configuration = ARWorldTrackingConfiguration()
override func viewDidLoad() {
super.viewDidLoad()
configuration.planeDetection = .horizontal // only the one option is available at the moment
sceneView.debugOptions = [.showFeaturePoints, .showWorldOrigin]
}
override func viewWillAppear(_ animated: Bool) {
super.viewWillAppear(animated)
sceneView.session.run(configuration)
}
override func viewWillDisappear(_ animated: Bool) {
super.viewWillDisappear(animated)
sceneView.session.pause()
}
December 9, 2017

Hands-on Experience in ARKit
December 9, 2017

Meaning of the application
The program is actually an Art demo-application which provides the ability for user to hang pictures
(3D object with texture) on walls in the real environment (room) and transform them after that (change
position, orientation and frame texture). The world scanning and detection are achieved using ARKit
framework.
December 9, 2017

Implementation complexity
Vertical plane detection is not (yet) a feature that exists in ARKit framework. The only existed
.horizontal enum case suggests that this feature could be being worked on and might be added there
in the future.
Solution
At first glance, an implementation for this would be difficult. But, there are many solutions were found
to achieve the requirements and each of them has own advantages and disadvantages.
December 9, 2017

If a wall has a good contrasting view or
there are hung contrasting objects on
the wall, notable features detected on
scene while scanning. These features
are called Feature Points and ARKit
render them as yellow points in
camera image for debug mode.
As soon as Feature points were
detected, ARKit places anchors in the
world scene, so we are able to grab a
position of some point or even a
surface by a points cloud.
Wall Detection Approach using Feature Points
December 9, 2017

The photos were made with iPhone 6s.
The current example shows that wall
recognition works well with contrasting
wallpapers and doesn’t require planes
detection.
December 9, 2017

if let hitResult = sceneView.hitTest(sceneView.center, types: .featurePoint).first {
PictureHelper.addPicture(at: SCNVector3Make(
hitResult.worldTransform.columns.3.x,
hitResult.worldTransform.columns.3.y,
hitResult.worldTransform.columns.3.z
), to: sceneView)
}
Option 1: Handle ARSCNView class method to detect intersection with one of the points using
.featurePoint type
guard let currentFrame = sceneView.session.currentFrame else {
return
}
let featurePoints = currentFrame.rawFeaturePoints?.points
Option 2: Calculate wall position and orientation by found feature points cloud of the current frame
December 9, 2017

But, what about plain walls?
December 9, 2017

Any 2D point in the scene view’s coordinate space can refer to any point along a line segment in the 3D
coordinate space. The current approach assumes the process of finding a plane of the current frame in the
world located along this line segment to get a normal to this plane in the intersection point.
Wall Detection Approach through the search of a plane normal by point
of scene view
December 9, 2017

Wall Detection Approach through the search of a frame plane by point of
scene view
var point = sceneView.center
var hitResult: ARHitTestResult?
repeat {
guard let nextHitResult = sceneView.hitTest(point, types: .existingPlaneUsingExtent).first else {
point = CGPoint(x: point.x, y: point.y + 1)
continue
}
hitResult = nextHitResult
} while hitResult == nil && screenBounds > point.y
Handle ARSCNView class method to detect intersection with a plane using .existingPlaneUsingExtent
type
December 9, 2017

scene view
But, if there is no visible plane in the current frame?
December 9, 2017

Ray casting is the use of line
intersection tests to determine the first
object intersected by a ray in scene.
So, this allows to find a needed
normal to the plane to hang a picture.
Wall Detection Approach through the search of a plane normal using Ray
casting
December 9, 2017

Wall Detection Approach through the search of a plane using Ray casting
var currentAngle: Float = 0
var hitResult: SCNHitTestResult?
repeat {
...
guard let nextHitResult = sceneView.scene.rootNode.hitTestWithSegment(from: startPoint, to:
endPoint, options: options).first else {
currentAngle += raycastAngleStepDegrees
continue
}
hitResult = nextHitResult
} while intersectionPosition == nil && (cameraAngleY - currentAngle) > 0
Handle SCNNode class method to detect intersection with a plane object using in the scene. This
implementation is provided by SceneKit framework
December 9, 2017

3D Object Transformation
December 9, 2017

Each object in 3D graphics has Transformation matrix to represent its position, orientation and scale.
December 9, 2017
It is possible to achieve any object transformation by multiplying its original Transformation matrix with
Translation, Rotation or Scale matrix:
P’ = P * T * R * S

All transformation manipulation were done using SceneKit framework. For example, end point position of
Ray line was calculated in the way:
December 9, 2017
var translation = matrix_identity_float4x4
translation.columns.3.y = -rayDistance * sin(angle)
translation.columns.3.z = -rayDistance * cos(angle)
let transform = matrix_multiply(currentFrame.camera.transform, translation)
let endPoint = SCNVector3Make(transform.columns.3.x, transform.columns.3.y, transform.columns.3.z)

Conclusion
The ARKit framework doesn’t render any virtual objects, doesn’t recognize real objects or characters in
the world itself. It is the role of CoreML, Vision, SceneKit, SpriteKit frameworks.
The basic requirement for ARKit is the ability to create and track a correspondence between the
real-world space the user inhabits and a virtual space where you can model visual content. Its role is
to track the exact position and direction of the camera screen respective to objects recognized in
the physical world.
December 9, 2017

PRIVILEGED AND CONFIDENTIALDecember 9, 2017

ARCore provides SDKs for many of the most popular development environments.
These SDKs provide native APIs for all of the essential AR features like motion
tracking, environmental understanding, and light estimation. With these capabilities
you can build entirely new AR experiences or enhance existing apps with AR
features.
Introducing ARCore Framework
December 9, 2017

Android SDK
version 7.0 (API level 24) or higher.
Device Compatibility
Devices
● Google Pixel, Pixel XL, Pixel 2, Pixel 2 XL
● Samsung Galaxy S8
Prepare your device
● Enable developer options
● Enable USB debugging
● Install the ARCore Service on the device
December 9, 2017
Devices using Tango
● Lenovo Phab 2 Pro
● Asus ZenFone AR

ARCore Features
December 9, 2017

Fundamental Concepts
● Environmental mapping
● Virtual object rendering
● Motion tracking
● Light level detection
● User interaction
● Object anchoring
ARCore Features
December 9, 2017

ARCore Features
Motion tracking
The challenge with motion tracking is converging virtual scenes with the camera view. This
requires translating the movement and position of the smartphone. Inaccurate translation can
lead to virtual objects floating and swimming on the view of the physical world.
December 9, 2017

ARCore Features
Plane Finding
To understand the environment, ARCore looks for clusters of feature points that appear to form
horizontal surfaces. ARCore also defines the bounds of these surfaces as polygons. Because
ARCore uses feature points, though, it has difficulty detecting smooth or reflective surfaces.
December 9, 2017

ARCore Features
Like ARKit, ARCore takes into account the lighting of the real world and applies it to the virtual
scene so that the shading of the two worlds match. This is necessary to give the virtual objects
a more realistic appearance.
December 9, 2017

ARCore Components and API
December 9, 2017

ARCore api implementation. Classes
December 9, 2017

GLSurfaceView
GLSurfaceView is a new API class GLSurfaceView makes OpenGL ES applications easier to
write by:
● Providing the glue code to connect OpenGL ES to the View system.
● Providing the glue code to make OpenGL ES work with the Activity life-cycle.
● Making it easy to choose an appropriate frame buffer pixel format.
● Creating and managing a separate rendering thread to enable smooth animation.
● Providing easy-to-use debugging tools for tracing OpenGL ES API calls and checking for
errors.
December 9, 2017

class ClearRenderer implements GLSurfaceView.Renderer {
public void onSurfaceCreated(GL10 gl, EGLConfig config) {
// Create the texture and pass it to ARCore session to be filled during update().
}
public void onSurfaceChanged(GL10 gl, int w, int h) {
gl.glViewport(0, 0, w, h);
}
public void onDrawFrame(GL10 gl) {
// Clear screen to notify driver it should not load any pixels from previous frame.
gl.glClear(GL10.GL_COLOR_BUFFER_BIT | GL10.GL_DEPTH_BUFFER_BIT);
// Handle taps. Handling only one tap per frame, as taps are usually low frequency
// compared to frame rate.
// Visualize planes.
// Visualize anchors created by touch.
}
}
December 9, 2017
GLSurfaceView

Frame frame = mSession.update();
float[] projmtx = new float[16];
mSession.getProjectionMatrix(projmtx, 0, 0.1f, 100.0f); // Get projection matrix.
float[] viewmtx = new float[16];
frame.getViewMatrix(viewmtx, 0); // Get camera matrix and draw.
float lightIntensity = frame.getLightEstimate().getPixelIntensity(); // Compute lighting from average intensity of the image.
// Visualize tracked points.
mPointCloud.update(frame.getPointCloud());
mPointCloud.draw(frame.getPointCloudPose(), viewmtx, projmtx);
// Check if we detected at least one plane. If so, hide the loading message.
for (Plane plane : mSession.getAllPlanes()) {
if (plane.getType() == com.google.ar.core.Plane.Type.HORIZONTAL_UPWARD_FACING
|| plane.getType() == com.google.ar.core.Plane.Type.NON_HORIZONTAL
// Visualize planes.
mPlaneRenderer.drawPlanes(mSession.getAllPlanes(), frame.getPose(), projmtx);
planeAttachment.getPose().toMatrix(mAnchorMatrix, 0); //Get the current combined pose of an Anchor and Plane in world.
object.updateModelMatrix(mAnchorMatrix, scaleFactor);
object.draw(viewmtx, projmtx, lightIntensity);
December 9, 2017
onDrawFrame

Hands-on Experience in ARCore
December 9, 2017

Meaning of the application
The program is actually an Art demo-application which provides the ability for user to hang pictures
(3D object with texture) on walls in the real environment (room) and transform them after that (change
position, orientation and frame texture). The world scanning and detection are achieved using ARCore
framework.
December 9, 2017

Implementation complexity
Vertical plane detection is not (yet) a feature that exists in ARCore framework. The only existed
.horizontal enum case suggests that this feature could be being worked on and might be added there
in the future.
December 9, 2017

If a wall has a good contrasting view or there are hung contrasting objects on the wall, notable features
detected on scene while scanning. These features are called Feature Points and ARcore render them as
blue points in camera image for debug mode.
As soon as Feature points were detected, ARCore places anchors in the world scene, so we are able to
grab a position of some point or even a surface by a points cloud.
December 9, 2017

MotionEvent tap = mQueuedSingleTaps.poll();
for (HitResult hit : frame.hitTest(tap)) {
if (hit instanceof PlaneHitResult && ((PlaneHitResult) hit).isHitInPolygon()) {
addItemToPoligon(hit);
}}
float[] getTranslationPosition(Frame frame, float translationX, float translationY, float translationZ) {
float [3] mPoseTranslation, float[3] mPoseTranslation2, float [] result;
//Get zero point for world coordinate
Pose inversePose = frame.getPose().inverse();
float[] point1 = inversePose.transformPoint(mPoseTranslation);
mPoseTranslation2[0] = translationY;
mPoseTranslation2[1] = translationX;
mPoseTranslation2[2] = translationZ;
//Get translation point for world coordinate
float[] point2 = inversePose.transformPoint(mPoseTranslation2);
result[0] = getLength(point1[0], point2[0]);
Get new translation values in the world coordinates
December 9, 2017
Detect tap on polygon

Any 2D point in the scene view’s coordinate space can refer to any point along a line segment in the 3D
coordinate space. The current approach assumes the process of finding plane in the world located along
this line segment to get a normal to this plane in the intersection point.
scene view
December 9, 2017

Ray casting is the use of line intersection tests to determine the first object intersected by a ray in scene.
So, this allows to find a needed normal to the plane to hang a picture.
Wall Detection Approach through the search of a plane using Ray casting
December 9, 2017
float [] screenPointToWorldRay(float xPx, float yPx, Frame frame,
GLSurfaceView mSurfaceView, Session mSession) {
float[] points , float[] matrices , float[] out ;
// Set up the clip-space coordinates of our query point
points[0] = 2.0f * xPx / mSurfaceView.getMeasuredWidth() - 1.0f;
// +y is up (android UI Y is down):
points[1] = 1.0f - 2.0f * yPx / mSurfaceView.getMeasuredHeight();
points[2] = 1.0f; // +z is forwards (remember clip, not camera)
points[3] = 1.0f; // w (homogenous coordinates)
mSession.getProjectionMatrix(matrices, 0, 1.0f, 100.0f);
Matrix.invertM(matrices, 16, matrices, 0);
// Transform clip-space point to camera-space.
Matrix.multiplyMV(points, 4, matrices, 16, points, 0);
// points[4,5,6] is now a camera-space vector. Transform to world
space to get a point along the ray.
frame.getPose().transformPoint(points, 4, out, 3);
frame.getPose().transformPoint(points, 8, out, 0);
// normalize the direction vector:
float dx = out[3] - out[0]; float dy = out[4] - out[1];
float dz = out[5] - out[2];
float scale = 1.0f / (float) Math.sqrt(dx*dx + dy*dy + dz*dz);
out[3] = dx * scale; out[4] = dy * scale; out[5] = dz * scale;

Want to change the light? Just switch the light on or off in the physical world. The image on your
device will immediately reflect any changes in ambient illumination.
December 9, 2017
float lightIntensity = frame.getLightEstimate().getPixelIntensity();

Conclusion
The ARCore framework doesn’t render any virtual objects, doesn’t recognize real objects or
characters in the world itself.
A technical solution they are very very close in capability. Effectively indistinguishable to users
when it comes to the user experiences. ARKit has some tech advantages around hw/sw
integration and more reliable tracking. ARCore has some advantages around mapping and more
reliable recovery. Both of these advantages are mostly only noticeable by engineer.
ARCore provides SDKs for many of the most popular development environments. These SDKs
provide native APIs for all of the essential AR features like motion tracking, environmental
understanding, and light estimation.
ARCore has a nice advantage in the pipeline of Tango. Only Google Tango devices can detect
distances, sizes, and surfaces in the environment using depth sensing cameras. Apple has just
introduced iPhone X phone with a front TrueDepth camera which is used just for face recognition.
Both systems have their strengths & weaknesses, but what’s important is that both can enable a
good enough consumer experience that developers have wide open spaces to explore.
December 9, 2017

Links
1) A link to the project of Art demo-application:
https://drive.google.com/a/griddynamics.com/file/d/1yYjrkDiLhwVz3GMeJxGBiZhUqykcQZ_m/vi
ew?usp=sharing
2) Apple Documentation:
https://developer.apple.com/documentation/arkit
3) A list of awesome ARKit projects, App Store applications, tutorials and resources:
https://github.com/olucurious/Awesome-ARKit
4) Android documentation:
https://developers.google.com/ar/develop/java/getting-started
5) A list of awesome ARCore examples:
https://experiments.withgoogle.com/ar
December 9, 2017

Scalable eCommerce Platform Solutions
Egor Zubkov
ezubkov@griddynamics.com
www.griddynamics.com
Thank you!
December 9, 2017
Vitalii Ruban
vruban@griddynamics.com
www.griddynamics.com

Augmented reality in mobile applications

Recommended

Recommended

More Related Content

What's hot

What's hot (18)

Similar to Augmented reality in mobile applications

Similar to Augmented reality in mobile applications (20)

More from Grid Dynamics

More from Grid Dynamics (20)

Recently uploaded

Recently uploaded (20)

Augmented reality in mobile applications