Your SlideShare is downloading. ×
0
iPhone Augmented
            Reality
  Jonathan Blocksom          Jonathan Saggau
blocksom@gmail.com    jonathan@jonathans...
About Us


•   Jonathan Blocksom    •   Jonathan Saggau
    (No PhD)                 (Also No PhD)
VR / AR History


• 1965
  Sutherland
VR / AR History

• 1995 -
  UNC,
  Nintendo
• who is that? ------>
Stella!
Existing AR Apps:
        Theolodite
• Sort of like fancy binoculars
DishPointer
Existing AR Apps:
           Layar
• Overlay markers on the world
Augmented Surreality:
     Numen
AR Drone




• iPhone controlled remote flyer
• Sends video stream to phone over Wifi
• http://ardrone.parrot.com/parrot-ar-...
More Existing AR Apps

• http://mashable.com/2009/12/05/
  augmented-reality-iphone/
• http://www.youtube.com/watch?
  v=p...
From zero to hero: starting with the
 “View based application” template
 and ending with a tagged landmark

• Hello (Augme...
Baby steps to a better reality :
 Hello (Augmented) World

  • Goal: Overlay some text onto the live
    camera image
  • ...
UIImagePicker overlay
-(void)viewDidLoad
{
    [super viewDidLoad];
    picker = [[UIImagePickerController alloc] init];
 ...
Picture (er, shopping cart) Level

• Goal: Overlay a level bar on the center of
  the live camera view
• Requires: real ti...
-(void)viewDidAppear:(BOOL)animated
{
    [super viewDidAppear:animated];
    [self presentModalViewController:picker anim...
#pragma mark UIAccelerometerDelegate
- (void)accelerometer:(UIAccelerometer *)accelerometer
        didAccelerate:(UIAccel...
Artificial Horizon
• Goal: Keep the bar on the horizon (not just
  level)
• Compensate for rotation along x-axis
• Need to ...
iPhone View Angles
        53 Degrees Vertical*
        37.5 Degrees Horizontal




*Close to this, but not exactly on the...
#pragma mark UIAccelerometerDelegate
- (void)accelerometer:(UIAccelerometer *)accelerometer
        didAccelerate:(UIAccel...
Directional Horizon Marker

• Goal: Artificial horizon pegged to a specific
  cardinal direction (North)
• Adds compass dire...
- (void)updateUI
{
    [label setText: [NSString stringWithFormat:@"H-Angle:%5.1f V-Angle:%5.1f Heading:%5.1f",
          ...
GPS Hotel Locator

• Goal: Speakers must find Hotel in any (um)
  state, so we’ll tag it for them.
• Add GPS location with
...
In other words:

   Given the position of the hotel, the position of
the iPhone, and the direction the iPhone is pointing,...
- (void)updateUI
{
    CLLocation *marriot = [[LocationTools class] locationForMarriott];
    float directionToMarriott = ...
Gotchas
• GPS and compass haven’t enough accuracy
  to place advertisements or labels on near
  objects.
• On a “bad day” ...
“Best practices... ” or “stuff we noticed
       making a demo AR app”


• Don’t try to tag a near object, inaccuracies
  ...
Download the code


• http://code.google.com/p/ar360/
iPhone OS 3 Stack
                         :;;261*'&<=*$>")3$"77*$%
                              ?@&4*"%&)9#3A%


       ...
iOS 4 Ideal
                             0:;"#)46<")%
                             =>&4*"%&)9#3?%


                      ...
AV Foundation

• Allows processing of the frame before
  display
• Very complicated to setup
• Check out the WWDC samples ...
What’s next?
• Smoothing / Filtering Sensor Data (wobble
  wobble)
• Full-on 3D AR world -- Dancing hippos in
  swimming p...
How we might transition to
  a Full virtual 3D World
• Need useful local coordinate system to
  map onto OpenGL; we’re pre...
ARKit :
   Existing AR Toolkit for iPhone



• http://github.com/zac/iphonearkit/
• Marker based
• Simple CoreLocation - b...
What counts as AR?

• Video with... ?
 • Overlay?
 • Must be Interactive?
 • GPS?
 • Modify the virtual environment?
AR News Sources
• Bruce Sterling’s Beyond the Beyond blog
  http://www.wired.com/
  beyond_the_beyond/
• Tish Shute:
  htt...
203 Is It Real or Is It Virtual? Augmented Reality on the iPhone
Upcoming SlideShare
Loading in...5
×

203 Is It Real or Is It Virtual? Augmented Reality on the iPhone

3,237

Published on

203 Is It Real or Is It Virtual? Augmented Reality on the iPhone with Jonathan Blocksom
Tuesday, Sept. 28, 11:15 am — 12:30 pm
San Diego

2 Comments
1 Like
Statistics
Notes
No Downloads
Views
Total Views
3,237
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
108
Comments
2
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "203 Is It Real or Is It Virtual? Augmented Reality on the iPhone"

  1. 1. iPhone Augmented Reality Jonathan Blocksom Jonathan Saggau blocksom@gmail.com jonathan@jonathansaggau.com @jblocksom @jonmarimba
  2. 2. About Us • Jonathan Blocksom • Jonathan Saggau (No PhD) (Also No PhD)
  3. 3. VR / AR History • 1965 Sutherland
  4. 4. VR / AR History • 1995 - UNC, Nintendo • who is that? ------>
  5. 5. Stella!
  6. 6. Existing AR Apps: Theolodite • Sort of like fancy binoculars
  7. 7. DishPointer
  8. 8. Existing AR Apps: Layar • Overlay markers on the world
  9. 9. Augmented Surreality: Numen
  10. 10. AR Drone • iPhone controlled remote flyer • Sends video stream to phone over Wifi • http://ardrone.parrot.com/parrot-ar-drone/
  11. 11. More Existing AR Apps • http://mashable.com/2009/12/05/ augmented-reality-iphone/ • http://www.youtube.com/watch? v=ps49T0iJwVg (acrossair) • Augmentia • Simple Geo • smaato • Layar • Wikitude
  12. 12. From zero to hero: starting with the “View based application” template and ending with a tagged landmark • Hello (Augmented) World • Picture Level (rotation about 1 axis) • Artificial Horizon (rotation about 2 axes) • Artificial Horizon + Compass (add cardinal direction / translation) • GPS Locator: tagging a landmark (add location / translation)
  13. 13. Baby steps to a better reality : Hello (Augmented) World • Goal: Overlay some text onto the live camera image • UIImagePicker + overlay • Step 1: collect underwear r3
  14. 14. UIImagePicker overlay -(void)viewDidLoad { [super viewDidLoad]; picker = [[UIImagePickerController alloc] init]; picker.sourceType = UIImagePickerControllerSourceTypeCamera; picker.showsCameraControls = NO; picker.navigationBarHidden = YES; picker.wantsFullScreenLayout = YES; //1:1.124.. = aspect ratio of the screen picker.cameraViewTransform = CGAffineTransformScale(picker.cameraViewTransform, 1.0f, 1.12412f); } -(void)viewDidAppear:(BOOL)animated { [super viewDidAppear:animated]; [self presentModalViewController:picker animated:NO]; UILabel *aLabel = [[UILabel alloc] initWithFrame:CGRectMake(10., 50., 300., 30.)]; [aLabel setText:@"Hello augmented world!"]; [aLabel setBackgroundColor:[UIColor clearColor]]; [aLabel setTextColor:[UIColor whiteColor]]; [aLabel setFont:[UIFont boldSystemFontOfSize:12]]; picker.cameraOverlayView = aLabel; }
  15. 15. Picture (er, shopping cart) Level • Goal: Overlay a level bar on the center of the live camera view • Requires: real time info regarding the rotation of the camera about the Z-axis • Adds UIAccelerometer r7
  16. 16. -(void)viewDidAppear:(BOOL)animated { [super viewDidAppear:animated]; [self presentModalViewController:picker animated:NO]; label = [[UILabel alloc] initWithFrame:CGRectMake(10., 50., 300., 30.)]; [label setText:@"Hello augmented world!"]; [label setBackgroundColor:[UIColor clearColor]]; [label setTextColor:[UIColor whiteColor]]; [label setFont:[UIFont boldSystemFontOfSize:12]]; CGRect viewFrame = self.view.frame; viewFrame.origin.x = viewFrame.origin.y = 0.0; UIView *overlayView = [[UIView alloc] initWithFrame:viewFrame]; [overlayView addSubview:label]; [overlayView bringSubviewToFront:label]; // Load the image to show in the overlay: UIImage *overlayGraphic = [UIImage imageNamed:@"artificialHorizon.png"]; overlayGraphicView = [[UIImageView alloc] initWithImage:overlayGraphic]; overlayGraphicView.frame = CGRectMake(30, 100, 260, 200); [overlayView addSubview:overlayGraphicView]; picker.cameraOverlayView = overlayView; [overlayView release]; [[UIAccelerometer sharedAccelerometer] setDelegate:self]; }
  17. 17. #pragma mark UIAccelerometerDelegate - (void)accelerometer:(UIAccelerometer *)accelerometer didAccelerate:(UIAcceleration *)acceleration; { double angle = -atan2(acceleration.y, acceleration.x) - M_PI/2.0; double angleInDeg = angle * 180.0f/M_PI; [label setText: [NSString stringWithFormat:@"Angle: %f", angleInDeg]]; overlayGraphicView.transform = CGAffineTransformMakeRotation(angle); } Video will go here
  18. 18. Artificial Horizon • Goal: Keep the bar on the horizon (not just level) • Compensate for rotation along x-axis • Need to know field of view of camera to find horizon position • ~53 degrees vertical • ~37.5 degrees horizontal r8
  19. 19. iPhone View Angles 53 Degrees Vertical* 37.5 Degrees Horizontal *Close to this, but not exactly on the iPhone 4
  20. 20. #pragma mark UIAccelerometerDelegate - (void)accelerometer:(UIAccelerometer *)accelerometer didAccelerate:(UIAcceleration *)acceleration; { overlayGraphicView.transform = CGAffineTransformIdentity; double vertAngle = -atan2(acceleration.y, acceleration.z) - M_PI/2.0; double vertAngleInDeg = vertAngle * 180.0f/M_PI; [label setText: [NSString stringWithFormat:@"V-Angle: %f", vertAngleInDeg]]; CGRect overlayFrame = [overlayGraphicView frame]; overlayFrame.origin.y = 240.0 - 537.8 * sin(vertAngle); [overlayGraphicView setFrame:overlayFrame]; double angle = -atan2(acceleration.y, acceleration.x) - M_PI/2.0; //double angleInDeg = angle * 180.0f/M_PI; overlayGraphicView.transform = CGAffineTransformMakeRotation(angle); }
  21. 21. Directional Horizon Marker • Goal: Artificial horizon pegged to a specific cardinal direction (North) • Adds compass direction using CLLocationManager locationManager = [[CLLocationManager alloc] init]; [locationManager setDelegate:self]; [locationManager startUpdatingHeading]; r10
  22. 22. - (void)updateUI { [label setText: [NSString stringWithFormat:@"H-Angle:%5.1f V-Angle:%5.1f Heading:%5.1f", angleInDeg, vertAngleInDeg, magCompassHeadingInDeg]]; CGPoint overlayCenter = [overlayGraphicView center]; overlayCenter.y = 240.0 - 537.8 * sin(vertAngle); overlayCenter.x = 160.0 - 497.8 * sin((magCompassHeadingInDeg) * (M_PI / 180.0)); [overlayGraphicView setCenter:overlayCenter]; overlayGraphicView.transform = CGAffineTransformMakeRotation(angle); } #pragma mark UIAccelerometerDelegate - (void)accelerometer:(UIAccelerometer *)accelerometer didAccelerate:(UIAcceleration *)acceleration; { vertAngle = -atan2(acceleration.y, acceleration.z) - M_PI/2.0; vertAngleInDeg = vertAngle * 180.0f/M_PI; angle = -atan2(acceleration.y, acceleration.x) - M_PI/2.0; angleInDeg = angle * 180.0f/M_PI; [self updateUI]; } #pragma mark CLLocationManagerDelegate - (void)locationManager:(CLLocationManager *)manager didUpdateHeading:(CLHeading *) newHeading; { magCompassHeadingInDeg = [newHeading magneticHeading]; [self updateUI]; }
  23. 23. GPS Hotel Locator • Goal: Speakers must find Hotel in any (um) state, so we’ll tag it for them. • Add GPS location with CLLocationManager • Compensate for “the hotel is over there and I’m looking over here”
  24. 24. In other words: Given the position of the hotel, the position of the iPhone, and the direction the iPhone is pointing, we want to know the cardinal direction of the hotel relative to the iPhone’s pointing direction and the elevation of the hotel relative the iPhone’s pointing direction in order to overlay a label onto the hotel in real time.
  25. 25. - (void)updateUI { CLLocation *marriot = [[LocationTools class] locationForMarriott]; float directionToMarriott = [[LocationTools class] angleFromCoordinate:[self.currentLocation coordinate] toCoordinate:[marriot coordinate]]; CLLocationDistance dist = [self.currentLocation getDistanceFrom:marriot]; double deltaAlt = [marriot altitude] - [self.currentLocation altitude]; //verticalAccuracy of -1 == invalid if ([self.currentLocation verticalAccuracy] < 0) deltaAlt = 0; double vertAngleToMarriott = atan2(deltaAlt, dist); [marriotLabel setText:[NSString stringWithFormat:@"DirHotel:%5.1f Dist: %5f VertAngle: %f", directionToMarriott * (180.0 / M_PI), dist, vertAngleToMarriott * (180.0 / M_PI)]]; [anglesLabel setText: [NSString stringWithFormat:@"Horizon H-Angle:%5.1f V-Angle:%5.1f Heading:%5.1f", angleInDeg, vertAngleInDeg, magCompassHeadingInDeg]]; double relativeDirectionToMarriott = magCompassHeadingInDeg * (M_PI / 180.) - directionToMarriott; if (relativeDirectionToMarriott < (-M_PI / 2.0)) relativeDirectionToMarriott = (-M_PI / 2.0); if (relativeDirectionToMarriott > (M_PI / 2.0)) relativeDirectionToMarriott = (M_PI / 2.0); double relativeVertAngleToMarriott = vertAngleToMarriott - vertAngle; CGPoint overlayCenter = [overlayGraphicView center]; overlayCenter.y = 240.0 - 537.8 * sin(relativeVertAngleToMarriott); overlayCenter.x = 160.0 - 497.8 * sin(relativeDirectionToMarriott); [overlayGraphicView setCenter:overlayCenter]; overlayGraphicView.transform = CGAffineTransformMakeRotation(angle); }
  26. 26. Gotchas • GPS and compass haven’t enough accuracy to place advertisements or labels on near objects. • On a “bad day” your iPhone’s version of “here” might be a 1/2 kilometer circle around a cell tower... or worse. • Location updates will drain battery... Quickly. • But the dream is alive.
  27. 27. “Best practices... ” or “stuff we noticed making a demo AR app” • Don’t try to tag a near object, inaccuracies compound as you near the target. • Check altitude / location accuracy: Phone can’t get altitude w/o good GPS fix. See documentation for accuracy (check for -1) • The broad side of the barn is easier to hit. • Bottom line: “here” isn’t here, yet.
  28. 28. Download the code • http://code.google.com/p/ar360/
  29. 29. iPhone OS 3 Stack :;;261*'&<=*$>")3$"77*$% ?@&4*"%&)9#3A% :;0<<*7*$"2*3*$%4*7*163*% !"#$%&'(")*%+,%-./% ?0<<*7*$"2*3*$A% 0#12*)3*4%5*67&38%099% B"<6C")D6)61*$% ?E',F%>"296GGA%
  30. 30. iOS 4 Ideal 0:;"#)46<")% =>&4*"%&)9#3?% E"$*%B"<")% !"#$%&'(")*%+,%-./% =0AA*7*$"2*3*$D%C8$"F?% 0#12*)3*4%5*67&38%099% @"A6<")B6)61*$% =C',D%E"296FF?%
  31. 31. AV Foundation • Allows processing of the frame before display • Very complicated to setup • Check out the WWDC samples and video
  32. 32. What’s next? • Smoothing / Filtering Sensor Data (wobble wobble) • Full-on 3D AR world -- Dancing hippos in swimming pools with Dolphins... • Image tracking to mitigate the swings •? • Profit!
  33. 33. How we might transition to a Full virtual 3D World • Need useful local coordinate system to map onto OpenGL; we’re pretty close with the demo code, actually. • Match OpenGL transform matrices with current position relative to some scene we might want to add to the real world, camera parameters; GLULookAt... • Need to not mind that compositing could be very slow...
  34. 34. ARKit : Existing AR Toolkit for iPhone • http://github.com/zac/iphonearkit/ • Marker based • Simple CoreLocation - based data model • Hasn’t been updated for a while
  35. 35. What counts as AR? • Video with... ? • Overlay? • Must be Interactive? • GPS? • Modify the virtual environment?
  36. 36. AR News Sources • Bruce Sterling’s Beyond the Beyond blog http://www.wired.com/ beyond_the_beyond/ • Tish Shute: http://www.ugotrade.com/ • Twitter streams: @anselm, @bruces, @tishshute, @brady • O’Reilly Radar: http://radar.oreilly.com
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×