This document discusses several core technologies for augmented reality development including ARKit, SceneKit, SpriteKit, and Metal. It covers capturing video and motion data, world tracking to determine position and orientation, plane detection and scene understanding, and optimized rendering using hardware acceleration. Key features covered include frame updating, hit testing, light estimation, and model deployment for tasks like image classification, style transfer, and language processing.

9. AVFoundation CoreMotion
Capturing
ARKit
Rendering
Application
SceneKit SpriteKit Metal
Processing
ARKit
ARSessionConfiguration
ARFrame
ARFrame
ARFrame
AVCaptureSession CMMotionManager
run(_ configuration) currentFrame
ARSession
World Tracking
Position and orientation
Physical distances
Relative to starting position
3D-feature points
1m

10. Scene Understanding
Plane detection
Hit-testing
Light estimation
Plane Detection
Horizontal with respect to gravity
Runs over multiple frames
Aligned extent for surface
Plane merging
Existing plane using extent
Existing plane
Estimated plane
Hit-Test Types Light Estimation
Ambient intensity based on captured image
Defaults to 1000 lumen
Enabled by default
configuration.isLightEstimationEnabled = true
// Get ambient intensity value
let intensity = frame.lightEstimate?.ambientIntensity

20. Optimized for Hardware
Dog playing in a
soccer field
Context Switching
Runs on GPU Runs on CPU
Today’s lunch was
terrible and
disappointing
Word Counts
Use NLP to Process Text
{
"Core" : 1,
"ML" : 1,
"is" : 1,
"awesome" : 1,
"I" : 1,
"love" : 1,
"using" : 1,
"it" : 1,
}
Xcode Your App
Model as CodeGetting the Model
Model
Function learned from data
Observed inputs
Predicts outputs
?Model
photo: flowerType:
Underlying Function
Scene Classification Style TransferTranslation
Sentiment Analysis Handwriting Recognition
Music Tagging Predicting Text
!That was totally
awesome Leo!
I love you mom 사랑해 엄마
7
Do you know the way to San Jose
Beach
Rock