Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Accelerometers to Augmented Reality
1. From Accelerometers
to Augmented Reality
Jonathan Blocksom
@jblocksom
iOSDevCamp 2011
August 13, 2011
Jonathan Saggau
(@jonmarimba)
present in spirit
2. The Theme
• Mobile devices are not just for viewing
information on a screen; they can be a
gateway to interacting with the world
around us.
• What can we do with them and how do
we do it?
3. iOS and the World
Around Us
Purpose Sensor SDK
Device Movement Accelerometer, Gyros Core Motion
Geolocation GPS, Magnetometer Core Location
Video Camera AVFoundation
AVFoundation,
Audio Microphone
Core Jose
4. Core Motion
• High level interface
to orientation and
movement data
• iOS 4+
• Accelerometer and
Gyroscope
• Sensor Fusion and
Data Filtering
14. Building OpenCV on iOS
• Follow this 20 step recipe:
http://computer-vision-talks.com/
2010/12/building-opencv-for-ios/
• Or go here:
https://github.com/jonmarimba/
OpenCV-iOS
15. Face Detection
• Use Haar Wavelet classification
• Built in classifiers in OpenCV to find
front and side facing faces
• Not perfect, not too fast, but not bad
• Video: http://vimeo.com/12774628
18. Running the classifier
CvSeq*
faces
=
cvHaarDetectObjects(small_image,
cascade,
storage,
1.2f,
2,
CV_HAAR_DO_CANNY_PRUNING,
cvSize(20,
20));
• Image, Haar cascades, spare storage
• 1.2f: Size inc. for features per stage
• 2: Minimum rectangle neighbors
• Canny Pruning: Throw out areas with too
few / too many edges
19. What it’s Doing
• Windows
show where
wavelets being
checked
• Overlapping
rectangles are
a detection
20. Defeating Face
Detection
• cvDazzle project
• Can also just turn to
the side
22. Feature Matching
• Feature Matching is the
workhorse of modern
computer vision
• Panoramas
• Image stabilization
• Superresolution
• 3D reconstruction
23. Feature Matching
• SIFT, SURF, FLANN:
Salient points in an
image (a) (b)
...
Scale
(next
octave)
(c) (d)
Figure 5: This figure shows the stages of keypoint selection. (a) The 233x189 pixel original image.
(b) The initial 832 keypoints locations at maxima and minima of the difference-of-Gaussian function.
Keypoints are displayed as vectors indicating scale, orientation, and location. (c) After applying
a threshold on minimum contrast, 729 keypoints remain. (d) The final 536 keypoints that remain
following an additional threshold on ratio of principal curvatures.
Scale
(first
octave) As suggested by Brown, the Hessian and derivative of D are approximated by using dif-
ferences of neighboring sample points. The resulting 3x3 linear system can be solved with
minimal cost. If the offset x is larger than 0.5 in any dimension, then it means that the ex-
ˆ
tremum lies closer to a different sample point. In this case, the sample point is changed and
Difference of the interpolation performed instead about that point. The final offset x is added to the location
ˆ
Gaussian Gaussian (DOG) of its sample point to get the interpolated estimate for the location of the extremum.
The function value at the extremum, D(ˆ), is useful for rejecting unstable extrema with
x
Figure 1: For each octave of scale space, the initial image is repeatedly convolved with Gaussians to
produce the set of scale space images shown on the left. Adjacent Gaussian images are subtracted low contrast. This can be obtained by substituting equation (3) into (2), giving
to produce the difference-of-Gaussian images on the right. After each octave, the Gaussian image is
down-sampled by a factor of 2, and the process repeated. Image gradients 1 ∂D T Keypoint descriptor
D(ˆ) = D +
x x.
ˆ
2 ∂x
Figure 7: A keypoint descriptor is created by first computing the gradient magnitude and orientation
In addition, the difference-of-Gaussian function provides a close approximation to the at each image sample point in a region around the keypoint location,value of |D(ˆ)| less than 0.03 were
For the experiments in this paper, all extrema with a as shown onx left. These are
the
scale-normalized Laplacian of Gaussian, σ 2 2 G, as studied by Lindeberg (1994). Lindeberg weighted bydiscarded (as before,indicated by the overlaid circle. These samples are then accumulated
a Gaussian window, we assume image pixel values in the range [0,1]).
showed that the normalization of the Laplacian with the factor σ 2 is required for true scale into orientation histograms summarizing the contents over 4x4 subregions, as shown on the right, with too
Figure 5 shows the effects of keypoint selection on a natural image. In order to avoid
32. Marker Based AR
• Find a marker
• Figure out camera transform to it
• Render something on top of it
• String SDK
• Qualcomm AR SDK
33. Demo:
Marker Based AR
SDK License Notes
NYAR GPL Old
String Commercial $
http://poweredbystring.com
Qualcomm
http://developer.qualcomm.com/dev/ Commercial, No cost Still in beta
augmented-reality
34. Qualcomm SDK
• FLANN to find initial features
• FAST to update after marker is found
