From Accelerometersto Augmented Reality     Jonathan Blocksom        @jblocksom      iOSDevCamp 2011       August 13, 2011...
The Theme• Mobile devices are not just for viewing  information on a screen; they can be a  gateway to interacting with th...
iOS and the World         Around Us   Purpose              Sensor               SDKDevice Movement   Accelerometer, Gyros ...
Core Motion• High level interface  to orientation and  movement data• iOS 4+• Accelerometer and  Gyroscope• Sensor Fusion ...
Motion Sensors:      Accelerometer• Accelerometer • From iPhone 1 • Noisy Gravity   Detector          !"#$#%&%()*+,%-#,.#/...
Motion Sensors:       Gyroscopes• Gyroscope • iPhone 4, iPad 2 • Rotation Rate • 200/500/2500   dps
Demo:            Vibration• Visualize  Accelerometer Data• App Store, $4.99  http://  itunes.apple.com/  app/vibration/  i...
Demo 1:Core Motion Teapot
Using Core Motion• Poll for data or  block handlers for  updates• Data as  Yaw, Pitch, Roll  Quaternion  4x4 Transform• Ti...
Demo:   Core Motion Viewer• https://bitbucket.org/jblocksom/  coremotionviewer
Core Motion Resources• Core Motion Framework Reference• Event Handling Guide for iPhone OS • “Core Motion” under “Motion E...
Computer Vision
Computer Vision:         OpenCV• The SDK I love to hate
Building OpenCV on iOS• Follow this 20 step recipe:  http://computer-vision-talks.com/  2010/12/building-opencv-for-ios/• ...
Face Detection• Use Haar Wavelet classification• Built in classifiers in OpenCV to find  front and side facing faces• Not per...
Haar classification• “Cascade of  boosted classifiers  working with haar-  like features”
Loading the Classifier                  • Just call cvLoad              	  	  	  	  NSString	  *path	  =	  [[NSBundle	  mai...
Running the classifier	  CvSeq*	  faces	  =	  cvHaarDetectObjects(small_image,	  cascade,                                  ...
What it’s Doing• Windows  show where  wavelets being  checked• Overlapping  rectangles are  a detection
Defeating Face         Detection• cvDazzle project• Can also just turn to  the side
Demo:      Face Detection• OpenCV based
Feature Matching• Feature Matching is the  workhorse of modern  computer vision • Panoramas • Image stabilization • Superr...
Feature Matching               • SIFT, SURF, FLANN:                 Salient points in an                 image            ...
Application: Automatic Panoramas• Application: Panoramas
Tracking• Feature finding and  matching is slow• Lower quality  features can match  faster with same  results
Demo: FAST Tracking
Augmented Reality
Augmented Reality• Fuse live video with generated pixels  based on device sensors• Geolocated• Marker Based• Commercial SD...
Geolocated AR• AR Based on GPS Location• Fuse rendered objects with real world  locations
3DAR Toolkit• http://spotmetrix.com/• Drop in replacement for MKMapView• Shows AR view based on phone  orientation• Free i...
Demos:LAYAR, 3DAR
Marker Based AR• Find a marker• Figure out camera transform to it• Render something on top of it• String SDK• Qualcomm AR ...
Demo:              Marker Based AR            SDK                           License            Notes          NYAR        ...
Qualcomm SDK• FLANN to find initial features• FAST to update after marker is found
That’s It!• Qualcomm AR SDK:  http://developer.qualcomm.com/dev/  augmented-reality• String SDK:  http://poweredbystring.c...
Upcoming SlideShare
Loading in …5
×

Accelerometers to Augmented Reality

5,063 views

Published on

iOSDevCampDC 2011 talk by Jonathan Blocksom

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
5,063
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
72
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Accelerometers to Augmented Reality

  1. 1. From Accelerometersto Augmented Reality Jonathan Blocksom @jblocksom iOSDevCamp 2011 August 13, 2011 Jonathan Saggau (@jonmarimba) present in spirit
  2. 2. The Theme• Mobile devices are not just for viewing information on a screen; they can be a gateway to interacting with the world around us.• What can we do with them and how do we do it?
  3. 3. iOS and the World Around Us Purpose Sensor SDKDevice Movement Accelerometer, Gyros Core Motion Geolocation GPS, Magnetometer Core Location Video Camera AVFoundation AVFoundation, Audio Microphone Core Jose
  4. 4. Core Motion• High level interface to orientation and movement data• iOS 4+• Accelerometer and Gyroscope• Sensor Fusion and Data Filtering
  5. 5. Motion Sensors: Accelerometer• Accelerometer • From iPhone 1 • Noisy Gravity Detector !"#$#%&%()*+,%-#,.#/% "012334445+67+$58#93:;+,<3=9)><39<$)3=!?@A1BCDEE@4E&>% • 100Hz, +/- 2.3G
  6. 6. Motion Sensors: Gyroscopes• Gyroscope • iPhone 4, iPad 2 • Rotation Rate • 200/500/2500 dps
  7. 7. Demo: Vibration• Visualize Accelerometer Data• App Store, $4.99 http:// itunes.apple.com/ app/vibration/ id301097580
  8. 8. Demo 1:Core Motion Teapot
  9. 9. Using Core Motion• Poll for data or block handlers for updates• Data as Yaw, Pitch, Roll Quaternion 4x4 Transform• Timestamps included
  10. 10. Demo: Core Motion Viewer• https://bitbucket.org/jblocksom/ coremotionviewer
  11. 11. Core Motion Resources• Core Motion Framework Reference• Event Handling Guide for iPhone OS • “Core Motion” under “Motion Events”• WWDC ’10: CoreMotionTeapot• J. R. Powers Talk from VTM November ’10• O’Reilly Basic Sensors in iOS
  12. 12. Computer Vision
  13. 13. Computer Vision: OpenCV• The SDK I love to hate
  14. 14. Building OpenCV on iOS• Follow this 20 step recipe: http://computer-vision-talks.com/ 2010/12/building-opencv-for-ios/• Or go here: https://github.com/jonmarimba/ OpenCV-iOS
  15. 15. Face Detection• Use Haar Wavelet classification• Built in classifiers in OpenCV to find front and side facing faces• Not perfect, not too fast, but not bad• Video: http://vimeo.com/12774628
  16. 16. Haar classification• “Cascade of boosted classifiers working with haar- like features”
  17. 17. Loading the Classifier • Just call cvLoad        NSString  *path  =  [[NSBundle  mainBundle]                        pathForResource:@"haarcascade_frontalface_default"          ofType:@"xml"]; CvHaarClassifierCascade  *cascade  =                  (CvHaarClassifierCascade  *)cvLoad(                                [path  cStringUsingEncoding:NSASCIIStringEncoding],     NULL,  NULL,  NULL);
  18. 18. Running the classifier  CvSeq*  faces  =  cvHaarDetectObjects(small_image,  cascade, storage,  1.2f,  2,                  CV_HAAR_DO_CANNY_PRUNING, cvSize(20,  20)); • Image, Haar cascades, spare storage • 1.2f: Size inc. for features per stage • 2: Minimum rectangle neighbors • Canny Pruning: Throw out areas with too few / too many edges
  19. 19. What it’s Doing• Windows show where wavelets being checked• Overlapping rectangles are a detection
  20. 20. Defeating Face Detection• cvDazzle project• Can also just turn to the side
  21. 21. Demo: Face Detection• OpenCV based
  22. 22. Feature Matching• Feature Matching is the workhorse of modern computer vision • Panoramas • Image stabilization • Superresolution • 3D reconstruction
  23. 23. Feature Matching • SIFT, SURF, FLANN: Salient points in an image (a) (b) ... Scale (next octave) (c) (d) Figure 5: This figure shows the stages of keypoint selection. (a) The 233x189 pixel original image. (b) The initial 832 keypoints locations at maxima and minima of the difference-of-Gaussian function. Keypoints are displayed as vectors indicating scale, orientation, and location. (c) After applying a threshold on minimum contrast, 729 keypoints remain. (d) The final 536 keypoints that remain following an additional threshold on ratio of principal curvatures. Scale (first octave) As suggested by Brown, the Hessian and derivative of D are approximated by using dif- ferences of neighboring sample points. The resulting 3x3 linear system can be solved with minimal cost. If the offset x is larger than 0.5 in any dimension, then it means that the ex- ˆ tremum lies closer to a different sample point. In this case, the sample point is changed and Difference of the interpolation performed instead about that point. The final offset x is added to the location ˆ Gaussian Gaussian (DOG) of its sample point to get the interpolated estimate for the location of the extremum. The function value at the extremum, D(ˆ), is useful for rejecting unstable extrema with xFigure 1: For each octave of scale space, the initial image is repeatedly convolved with Gaussians toproduce the set of scale space images shown on the left. Adjacent Gaussian images are subtracted low contrast. This can be obtained by substituting equation (3) into (2), givingto produce the difference-of-Gaussian images on the right. After each octave, the Gaussian image isdown-sampled by a factor of 2, and the process repeated. Image gradients 1 ∂D T Keypoint descriptor D(ˆ) = D + x x. ˆ 2 ∂x Figure 7: A keypoint descriptor is created by first computing the gradient magnitude and orientation In addition, the difference-of-Gaussian function provides a close approximation to the at each image sample point in a region around the keypoint location,value of |D(ˆ)| less than 0.03 were For the experiments in this paper, all extrema with a as shown onx left. These are thescale-normalized Laplacian of Gaussian, σ 2 2 G, as studied by Lindeberg (1994). Lindeberg weighted bydiscarded (as before,indicated by the overlaid circle. These samples are then accumulated a Gaussian window, we assume image pixel values in the range [0,1]).showed that the normalization of the Laplacian with the factor σ 2 is required for true scale into orientation histograms summarizing the contents over 4x4 subregions, as shown on the right, with too Figure 5 shows the effects of keypoint selection on a natural image. In order to avoid
  24. 24. Application: Automatic Panoramas• Application: Panoramas
  25. 25. Tracking• Feature finding and matching is slow• Lower quality features can match faster with same results
  26. 26. Demo: FAST Tracking
  27. 27. Augmented Reality
  28. 28. Augmented Reality• Fuse live video with generated pixels based on device sensors• Geolocated• Marker Based• Commercial SDKs available
  29. 29. Geolocated AR• AR Based on GPS Location• Fuse rendered objects with real world locations
  30. 30. 3DAR Toolkit• http://spotmetrix.com/• Drop in replacement for MKMapView• Shows AR view based on phone orientation• Free if branded• $5K for unbranded
  31. 31. Demos:LAYAR, 3DAR
  32. 32. Marker Based AR• Find a marker• Figure out camera transform to it• Render something on top of it• String SDK• Qualcomm AR SDK
  33. 33. Demo: Marker Based AR SDK License Notes NYAR GPL Old String Commercial $ http://poweredbystring.com Qualcommhttp://developer.qualcomm.com/dev/ Commercial, No cost Still in beta augmented-reality
  34. 34. Qualcomm SDK• FLANN to find initial features• FAST to update after marker is found
  35. 35. That’s It!• Qualcomm AR SDK: http://developer.qualcomm.com/dev/ augmented-reality• String SDK: http://poweredbystring.com• Me: http://twitter.com/jblocksom/

×