203 Is It Real or Is It Virtual? Augmented Reality on the iPhone

iPhone Augmented Reality Jonathan Blocksom Jonathan Saggau blocksom@gmail.com jonathan@jonathansaggau.com @jblocksom @jonmarimba

About Us • Jonathan Blocksom • Jonathan Saggau (No PhD) (Also No PhD)

VR / AR History • 1965 Sutherland

VR / AR History • 1995 - UNC, Nintendo • who is that? ------>

Stella!

Existing AR Apps: Theolodite • Sort of like fancy binoculars

DishPointer

Existing AR Apps: Layar • Overlay markers on the world

Augmented Surreality: Numen

AR Drone • iPhone controlled remote ﬂyer • Sends video stream to phone over Wiﬁ • http://ardrone.parrot.com/parrot-ar-drone/

More Existing AR Apps • http://mashable.com/2009/12/05/ augmented-reality-iphone/ • http://www.youtube.com/watch? v=ps49T0iJwVg (acrossair) • Augmentia • Simple Geo • smaato • Layar • Wikitude

From zero to hero: starting with the “View based application” template and ending with a tagged landmark • Hello (Augmented) World • Picture Level (rotation about 1 axis) • Artiﬁcial Horizon (rotation about 2 axes) • Artiﬁcial Horizon + Compass (add cardinal direction / translation) • GPS Locator: tagging a landmark (add location / translation)

Baby steps to a better reality : Hello (Augmented) World • Goal: Overlay some text onto the live camera image • UIImagePicker + overlay • Step 1: collect underwear r3

UIImagePicker overlay -(void)viewDidLoad { [super viewDidLoad]; picker = [[UIImagePickerController alloc] init]; picker.sourceType = UIImagePickerControllerSourceTypeCamera; picker.showsCameraControls = NO; picker.navigationBarHidden = YES; picker.wantsFullScreenLayout = YES; //1:1.124.. = aspect ratio of the screen picker.cameraViewTransform = CGAffineTransformScale(picker.cameraViewTransform, 1.0f, 1.12412f); } -(void)viewDidAppear:(BOOL)animated { [super viewDidAppear:animated]; [self presentModalViewController:picker animated:NO]; UILabel *aLabel = [[UILabel alloc] initWithFrame:CGRectMake(10., 50., 300., 30.)]; [aLabel setText:@"Hello augmented world!"]; [aLabel setBackgroundColor:[UIColor clearColor]]; [aLabel setTextColor:[UIColor whiteColor]]; [aLabel setFont:[UIFont boldSystemFontOfSize:12]]; picker.cameraOverlayView = aLabel; }

Picture (er, shopping cart) Level • Goal: Overlay a level bar on the center of the live camera view • Requires: real time info regarding the rotation of the camera about the Z-axis • Adds UIAccelerometer r7

-(void)viewDidAppear:(BOOL)animated { [super viewDidAppear:animated]; [self presentModalViewController:picker animated:NO]; label = [[UILabel alloc] initWithFrame:CGRectMake(10., 50., 300., 30.)]; [label setText:@"Hello augmented world!"]; [label setBackgroundColor:[UIColor clearColor]]; [label setTextColor:[UIColor whiteColor]]; [label setFont:[UIFont boldSystemFontOfSize:12]]; CGRect viewFrame = self.view.frame; viewFrame.origin.x = viewFrame.origin.y = 0.0; UIView *overlayView = [[UIView alloc] initWithFrame:viewFrame]; [overlayView addSubview:label]; [overlayView bringSubviewToFront:label]; // Load the image to show in the overlay: UIImage *overlayGraphic = [UIImage imageNamed:@"artificialHorizon.png"]; overlayGraphicView = [[UIImageView alloc] initWithImage:overlayGraphic]; overlayGraphicView.frame = CGRectMake(30, 100, 260, 200); [overlayView addSubview:overlayGraphicView]; picker.cameraOverlayView = overlayView; [overlayView release]; [[UIAccelerometer sharedAccelerometer] setDelegate:self]; }

#pragma mark UIAccelerometerDelegate - (void)accelerometer:(UIAccelerometer *)accelerometer didAccelerate:(UIAcceleration *)acceleration; { double angle = -atan2(acceleration.y, acceleration.x) - M_PI/2.0; double angleInDeg = angle * 180.0f/M_PI; [label setText: [NSString stringWithFormat:@"Angle: %f", angleInDeg]]; overlayGraphicView.transform = CGAffineTransformMakeRotation(angle); } Video will go here

Artificial Horizon • Goal: Keep the bar on the horizon (not just level) • Compensate for rotation along x-axis • Need to know field of view of camera to find horizon position • ~53 degrees vertical • ~37.5 degrees horizontal r8

iPhone View Angles 53 Degrees Vertical* 37.5 Degrees Horizontal *Close to this, but not exactly on the iPhone 4

#pragma mark UIAccelerometerDelegate - (void)accelerometer:(UIAccelerometer *)accelerometer didAccelerate:(UIAcceleration *)acceleration; { overlayGraphicView.transform = CGAffineTransformIdentity; double vertAngle = -atan2(acceleration.y, acceleration.z) - M_PI/2.0; double vertAngleInDeg = vertAngle * 180.0f/M_PI; [label setText: [NSString stringWithFormat:@"V-Angle: %f", vertAngleInDeg]]; CGRect overlayFrame = [overlayGraphicView frame]; overlayFrame.origin.y = 240.0 - 537.8 * sin(vertAngle); [overlayGraphicView setFrame:overlayFrame]; double angle = -atan2(acceleration.y, acceleration.x) - M_PI/2.0; //double angleInDeg = angle * 180.0f/M_PI; overlayGraphicView.transform = CGAffineTransformMakeRotation(angle); }

Directional Horizon Marker • Goal: Artiﬁcial horizon pegged to a speciﬁc cardinal direction (North) • Adds compass direction using CLLocationManager locationManager = [[CLLocationManager alloc] init]; [locationManager setDelegate:self]; [locationManager startUpdatingHeading]; r10

- (void)updateUI { [label setText: [NSString stringWithFormat:@"H-Angle:%5.1f V-Angle:%5.1f Heading:%5.1f", angleInDeg, vertAngleInDeg, magCompassHeadingInDeg]]; CGPoint overlayCenter = [overlayGraphicView center]; overlayCenter.y = 240.0 - 537.8 * sin(vertAngle); overlayCenter.x = 160.0 - 497.8 * sin((magCompassHeadingInDeg) * (M_PI / 180.0)); [overlayGraphicView setCenter:overlayCenter]; overlayGraphicView.transform = CGAffineTransformMakeRotation(angle); } #pragma mark UIAccelerometerDelegate - (void)accelerometer:(UIAccelerometer *)accelerometer didAccelerate:(UIAcceleration *)acceleration; { vertAngle = -atan2(acceleration.y, acceleration.z) - M_PI/2.0; vertAngleInDeg = vertAngle * 180.0f/M_PI; angle = -atan2(acceleration.y, acceleration.x) - M_PI/2.0; angleInDeg = angle * 180.0f/M_PI; [self updateUI]; } #pragma mark CLLocationManagerDelegate - (void)locationManager:(CLLocationManager *)manager didUpdateHeading:(CLHeading *) newHeading; { magCompassHeadingInDeg = [newHeading magneticHeading]; [self updateUI]; }

GPS Hotel Locator • Goal: Speakers must ﬁnd Hotel in any (um) state, so we’ll tag it for them. • Add GPS location with CLLocationManager • Compensate for “the hotel is over there and I’m looking over here”

In other words: Given the position of the hotel, the position of the iPhone, and the direction the iPhone is pointing, we want to know the cardinal direction of the hotel relative to the iPhone’s pointing direction and the elevation of the hotel relative the iPhone’s pointing direction in order to overlay a label onto the hotel in real time.

- (void)updateUI { CLLocation *marriot = [[LocationTools class] locationForMarriott]; float directionToMarriott = [[LocationTools class] angleFromCoordinate:[self.currentLocation coordinate] toCoordinate:[marriot coordinate]]; CLLocationDistance dist = [self.currentLocation getDistanceFrom:marriot]; double deltaAlt = [marriot altitude] - [self.currentLocation altitude]; //verticalAccuracy of -1 == invalid if ([self.currentLocation verticalAccuracy] < 0) deltaAlt = 0; double vertAngleToMarriott = atan2(deltaAlt, dist); [marriotLabel setText:[NSString stringWithFormat:@"DirHotel:%5.1f Dist: %5f VertAngle: %f", directionToMarriott * (180.0 / M_PI), dist, vertAngleToMarriott * (180.0 / M_PI)]]; [anglesLabel setText: [NSString stringWithFormat:@"Horizon H-Angle:%5.1f V-Angle:%5.1f Heading:%5.1f", angleInDeg, vertAngleInDeg, magCompassHeadingInDeg]]; double relativeDirectionToMarriott = magCompassHeadingInDeg * (M_PI / 180.) - directionToMarriott; if (relativeDirectionToMarriott < (-M_PI / 2.0)) relativeDirectionToMarriott = (-M_PI / 2.0); if (relativeDirectionToMarriott > (M_PI / 2.0)) relativeDirectionToMarriott = (M_PI / 2.0); double relativeVertAngleToMarriott = vertAngleToMarriott - vertAngle; CGPoint overlayCenter = [overlayGraphicView center]; overlayCenter.y = 240.0 - 537.8 * sin(relativeVertAngleToMarriott); overlayCenter.x = 160.0 - 497.8 * sin(relativeDirectionToMarriott); [overlayGraphicView setCenter:overlayCenter]; overlayGraphicView.transform = CGAffineTransformMakeRotation(angle); }

Gotchas • GPS and compass haven’t enough accuracy to place advertisements or labels on near objects. • On a “bad day” your iPhone’s version of “here” might be a 1/2 kilometer circle around a cell tower... or worse. • Location updates will drain battery... Quickly. • But the dream is alive.

“Best practices... ” or “stuff we noticed making a demo AR app” • Don’t try to tag a near object, inaccuracies compound as you near the target. • Check altitude / location accuracy: Phone can’t get altitude w/o good GPS ﬁx. See documentation for accuracy (check for -1) • The broad side of the barn is easier to hit. • Bottom line: “here” isn’t here, yet.

Download the code • http://code.google.com/p/ar360/

iPhone OS 3 Stack :;;261*'&<=*$>")3$"77*$% ?@&4*"%&)9#3A% :;0<<*7*$"2*3*$%4*7*163*% !"#$%&'(")*%+,%-./% ?0<<*7*$"2*3*$A% 0#12*)3*4%5*67&38%099% B"<6C")D6)61*$% ?E',F%>"296GGA%

iOS 4 Ideal 0:;"#)46<")% =>&4*"%&)9#3?% E"$*%B"<")% !"#$%&'(")*%+,%-./% =0AA*7*$"2*3*$D%C8$"F?% 0#12*)3*4%5*67&38%099% @"A6<")B6)61*$% =C',D%E"296FF?%

AV Foundation • Allows processing of the frame before display • Very complicated to setup • Check out the WWDC samples and video

What’s next? • Smoothing / Filtering Sensor Data (wobble wobble) • Full-on 3D AR world -- Dancing hippos in swimming pools with Dolphins... • Image tracking to mitigate the swings •? • Proﬁt!

How we might transition to a Full virtual 3D World • Need useful local coordinate system to map onto OpenGL; we’re pretty close with the demo code, actually. • Match OpenGL transform matrices with current position relative to some scene we might want to add to the real world, camera parameters; GLULookAt... • Need to not mind that compositing could be very slow...

ARKit : Existing AR Toolkit for iPhone • http://github.com/zac/iphonearkit/ • Marker based • Simple CoreLocation - based data model • Hasn’t been updated for a while

What counts as AR? • Video with... ? • Overlay? • Must be Interactive? • GPS? • Modify the virtual environment?

AR News Sources • Bruce Sterling’s Beyond the Beyond blog http://www.wired.com/ beyond_the_beyond/ • Tish Shute: http://www.ugotrade.com/ • Twitter streams: @anselm, @bruces, @tishshute, @brady • O’Reilly Radar: http://radar.oreilly.com

203 Is It Real or Is It Virtual? Augmented Reality on the iPhone

by jonmarimba on Sep 12, 2010

Accessibility

Tags

Upload Details

Usage Rights

Statistics

203 Is It Real or Is It Virtual? Augmented Reality on the iPhone — Presentation Transcript