I demonstrate how to build an iOS AR app (before ARKit) so that you can build AR games. I present the math needed and the code.

1. Programming
Augmented Reality:
Make your own sci-fi
Mobile App Dev
@praeclarum
Frank A. Krueger

2. Augmented Reality
a live direct or indirect view of a physical,
real-world environment whose elements
are augmented (or supplemented) by
computer-generated sensory input such
as sound, video, graphics or GPS data

5. Why Augmented Reality?
• Presents data in a way that we are
accustomed to
• Can present a lot of data in multiple
ways
Useful
• Remember the Terminator, he was fun
right?
• Extends our senses
• Physical games are more fun
Fun
• Takes advantage of distinguishing
platform elements
Mobile Only

6. Anatomy
World Simulation
Video Camera
Location & Orientation
Augmented View

7. Anatomy
World Simulation
Video Camera
Location & Orientation
Augmented View
This is where the magic
happens
AR library

8. Video Camera

9. Video Camera
■
Many resolutions (width and height) and
framerates
■
Image filtering controls
■
Field of View
■
High-performance code - be careful!
■
8 x 106 pixels * 30 Hz =
240,000,000 pixels / second
It all starts here

10. Directly Accessing the Camera (iOS)
session = new AVCaptureSession ();
session.SessionPreset = AVCaptureSession.PresetMedium;
device = AVCaptureDevice.DefaultDeviceWithMediaType (AVMediaType.Video);
device.LockForConfiguration(out error);
device.ActiveVideoMinFrameDuration = new CMTime (1, 30);
device.UnlockForConfiguration();
input = AVCaptureDeviceInput.FromDevice (device, out error);
session.AddInput (input);

11. Getting Images (iOS)
output = new AVCaptureVideoDataOutput ();
output.VideoSettings = new AVVideoSettings (CVPixelFormatType.CV32BGRA);
queue = new DispatchQueue ("VideoCameraQueue");
output.SetSampleBufferDelegateAndQueue (new VideoCameraDelegate { Camera = this } , queue);
session.AddOutput (output);
class VideoCameraDelegate : AVCaptureVideoDataOutputSampleBufferDelegate
{
public override void DidOutputSampleBuffer (AVCaptureOutput captureOutput,
CMSampleBuffer sampleBuffer,
AVCaptureConnection connection)
{
try {
var frame = ImageFromSampleBuffer (sampleBuffer);
Camera.OnFrameCaptured (frame);
sampleBuffer.Dispose ();
} catch (Exception ex) {
Debug.WriteLine (ex);
}
}
}

12. Field of View
public float FieldOfView {
get {
return device.ActiveFormat.VideoFieldOfView;
}
}
Field of View (FOV)
is the Visible Angle

13. Location & Orientation

14. Location Sensor
■
Our friends Latitude & Longitude
■
Altitude above sea level (sometimes)
■
10m resolution, on a good day
■
“Better” with Wi-fi
■
“Not great” indoors
■
(There are alternatives such as
triangulation from known points)
Global Spherical Coordinates

15. Receiving Location Updates
lman = new CLLocationManager {
DesiredAccuracy = CLLocation.AccuracyBest,
};
lman.LocationsUpdated += (sender, e) => {
var loc = e.Locations [0];
Timestamp = loc.Timestamp;
Location = new Location (loc.Coordinate.Latitude, loc.Coordinate.Longitude, loc.Altitude);
HorizontalAccuracy = loc.HorizontalAccuracy;
VerticalAccuracy = loc.VerticalAccuracy;
};
lman.StartUpdatingLocation ();

16. Orientation Sensor
■
Pitch, Roll, Yaw (3 numbers)
■
Quaternion (4 numbers)
■
4x4 Transformation Matrix (16
numbers)
A Variety of Formats
Pitch
Yaw
Roll
■
Need a “reference orientation” or known
orientation from which to measure
It’s relative…

17. Receiving Orientation Updates
mman = new CMMotionManager {
ShowsDeviceMovementDisplay = true,
};
mman.StartDeviceMotionUpdates (
// CMAttitudeReferenceFrame.XArbitraryZVertical,
// CMAttitudeReferenceFrame.XArbitraryCorrectedZVertical,
// CMAttitudeReferenceFrame.XMagneticNorthZVertical,
CMAttitudeReferenceFrame.XTrueNorthZVertical,
new NSOperationQueue (),
(motion, error) => {
Orientation = ToMatrix4d (motion.Attitude.RotationMatrix);
} );

18. Augmented View

19. How do you augment th
Video Camera images?

20. Augmentation Techniques
Overlay 2D Annotations
• Heads Up Display
• Terminator
• Use OpenGL, SceneKit, MonoGame, …
• Mixing virtual reality with augmented
Show 3D Virtual Objects
Distort Camera Image
• Enhance recognized and correlated objects
• Hot / Cold Maps
• Pixel perfect
• No lag

21. Aligning the Augmented View to the Video
Camera
A Pinhole Camera Model
• Simple
• Used by 3D renderers
• Only depends on Field of View
• “Close enough” to the video camera
In a 3D Cartesian World
• Use 3D X, Y, Z coordinates instead of
Latitude, Longitude, Altitude
• Orient the camera by applying the
orientation transform
Projection
Matrix
ModelView
Matrix

22. Pinhole Camera Model
Projection
Matrix
ModelView
Matrix
3D
Worl
d
Point
2D
View
Point
x x =
(after a
perspective
divide)
Video Camera
Location &
Orientation

23. Projection Matrix
public static Matrix4d GetProjectionMatrix (VideoCamera videoCamera, UIImage frame)
{
return Matrix4d.CreatePerspectiveFieldOfView (
fovy: videoCamera.FieldOfView,
aspect: frame.Size.Width / frame.Size.Height,
zNear: 0.01,
zFar: 4700);
}

24. ModelView Matrix - Two Techniques
• Very easy
• Not accurate
• High latency
• Best for outdoor unpredictable
environments
Location & Orientation
Sensors
• Very difficult
• Very accurate
• No latency
• Best for indoor or controlled
environments
Image Recognition

25. ModelView Matrix from Location & Orientation
public static Matrix4d GetModelView (Location location, Matrix4d orientation)
{
// 1. Calculate position in 3D cartesian world
// 2. Find “up"
// 3. Orient to face the north pole
// 4. Apply the device orientation
}

26. ModelView Matrix: Location to 3D Cartesian
• Center of earth is at (0, 0, 0)
• Does not rotate, the universe around it does
• Physically accurate
• X, Y, Z are generally unintelligible
Earth-centered, Earth-fixed
• Flatten the earth like a map
• X axis is East/West
• Y axis is North/South
• Z axis is Altitude
North, East, Up

27. ModelView Matrix: Location to 3D Cartesian (ECEF)
//
// 1. Calculate position in 3D cartesian world
//
var pos = location.Position;
public Vector3d Position {
get {
var omega = ToRad * Longitude;
var phi = ToRad * Latitude;
var r = RadiusOfEarth + Altitude;
return new Vector3d (
r * Math.Cos(phi) * Math.Cos(omega),
r * Math.Cos(phi) * Math.Sin(omega),
r * Math.Sin(phi));
}
}
• (X, Y, Z)

28. ModelView Matrix: Up
//
// 2. Find "up"
//
var up = location.Position;
up.Normalize ();
• (0, 0, 0)
• (X, Y, Z)
Up

29. ModelView Matrix: Look at the North Pole
//
// 3. Orient to face the north pole
//
var northPos = Location.NorthPole.Position;
var northZAxis = (pos - northPos);
northZAxis.Normalize ();
var northYAxis = up;
var northXAxis = Vector3d.Cross (northYAxis, northZAxis);
northXAxis.Normalize ();
northZAxis = Vector3d.Cross (northXAxis, northYAxis);
northZAxis.Normalize ();
var lookNorthI = new Matrix4d (
new Vector4d(northXAxis),
new Vector4d(northYAxis),
new Vector4d(northZAxis),
Vector4d.UnitW);
Location NorthPole = new Location (90, 0, 0);
True North
• (X, Y, Z)

30. ModelView Matrix: Orient
//
// 4. Apply the device orientation
//
var newOrient = new Matrix4d (
-orientation.Column1,
orientation.Column2,
-orientation.Column0,
Vector4d.UnitW);
var newOrientI = newOrient;
newOrientI.Transpose ();
var modelViewI = (newOrientI * lookNorthI);
modelViewI.Row3 = new Vector4d (pos.X, pos.Y, pos.Z, 1);
var modelView = modelViewI;
modelView.Invert ();
return modelView;

31. Location to 2D View
Projection
Matrix
ModelView
Matrix
3D
Worl
d
Point
2D
View
Point
x x =
(after a
perspective
divide)

32. Location to 2D View
public PointF LocationToView (Location location)
{
// Move location to 3D earth
var pos3d = new Vector4d (location.Position, 1);
// Camera model
var m = Matrix4d.Mult (modelViewMatrix, projectionMatrix);
// Project into homogeneous 2D point
var pos2h = Vector4d.Transform (pos3d, m);
// Perform the perspective divide
var pos2 = pos2h / pos2h.W;
// Stretch into our view
return new PointF (
(float)((pos2.X + 1) * 0.5) * viewSize.Width,
(float)((-pos2.Y + 1) * 0.5) * viewSize.Height
);
}

33. Demo

34. Same Math Can be Used to Make 3D
void SetOpenGLCamera ()
{
GL.MatrixMode (All.Projection);
GL.LoadMatrix (ref projectionMatrix.Row0.X);
GL.MatrixMode (All.Modelview);
GL.LoadMatrix (ref modelViewMatrix.Row0.X);
}

35. Demo

36. World Simulation

37. It’s up to you
The world’s full of geo tagged
data

38. Applications
Games
• Massively Multiplayer
Online Live Action Role
Playing Games
(MMOLARPGs)
• Treasure Hunt
• Intercontinental Worms
• Friend locator
• Friend holograms
• Tourist attractions
Social
• Physically secure digital
goods
• Architectural projections
• Airplane & satellite
tracking
• Boat tracking
Industrial

39. DroidAR, ARmedia, Vuforia, ARLab, Beyond Reality Face,
D'Fusion, instantreality, Metaio SDK, Viewdle, Xloudia, ARPA,
ALVAR, AndAR, AR23D, ARToolkit, Aurasma, Awila,
BeyondAR, Catchoom, Cortexia, Google Goggles, IN2AR,
Koozyt, layar, LibreGeoSocial, mixare, NyARToolkit, Obvious
Engine, PanicAR, PointCloud, popcode, PRAugmentedReality,
PTAM, Qoncept AR, Robocortex, SLARToolkit, snaptell, SSTT,
String, Studierstube Tracker, UART, Wikitude, xpose visual
search, yvision, Zenitum Feature Tracker
45+ Mobile SDKs to Choose From
http://socialcompare.com/en/comparison/augmented-reality-sdks

40. Thank you
Frank A. Krueger
@praeclarum
https://github.com/praeclarum/ARDemo