Your SlideShare is downloading. ×

COSC 426 lect. 4: AR Interaction


Published on

Published in: Technology
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Lecture 4. AR Interaction Mark Billinghurst Gun Lee Aug 2011 COSC 426: Augmented Reality
  • 2. Building Compelling AR ExperiencesB ildi C lli E i experiences applications Interaction tools Authoring components Tracking, Display Sony CSL © 2004
  • 3. AR Interaction Designing AR System = Interface Design Using diff U i different input and output technologies i d h l i Objective is a high quality of user experience j g q y p Ease of use and learning Performance and satisfaction
  • 4. Interaction Tasks 2D (from [Foley]): Selection, Text Entry, Quantify, Position y y 3D (from [Bowman]): Navigation (Travel/Wayfinding) Selection Manipulation System Control/Data Input AR: 2D + 3D Tasks and.. more specific tasks?[Foley] The Human Factors of Computer Graphics Interaction Techniques Foley, J. D.,V. Wallace & P. Chan. IEEE ComputerGraphics and Applications(Nov.): 13-48. 1984.[Bowman]: 3D User Interfaces:Theory and Practice D. Bowman, E. Kruijff, J. Laviola, I. Poupyrev Addison Wesley 2005
  • 5. AR Interfaces as Data Browsers2D/3D virtual objects areregistered in 3D “VR in Real World”InteractionI t ti 2D/3D virtual viewpoint control lApplications Visualization, training
  • 6. AR Information BrowsersInformation is registered toreal-world context Hand held AR displaysInteraction Manipulation f M i l i of a windowi d into information spaceApplications Context-aware information displays Rekimoto, et al. 1997
  • 7. Architecture
  • 8. Current AR Information BrowsersMobile AR GPS + compassMany Applications Layar y Wikitude Acrossair PressLite Yelp AR Car Finder …
  • 9. JunaioAR Browser from Metaio p jAR browsing GPS + compass 2D/3D object placement Photos/live video Community viewing
  • 10. Advantages and DisadvantagesImportant class of AR interfaces Wearable computers AR simulation, trainingLimited interactivity Modification of virtual content is difficult Rekimoto, et al. 1997
  • 11. 3D AR InterfacesVirtual objects displayed in 3Dphysical space and manipulated HMDs and 6DOF head-tracking 6DOF hand trackers for input h d k f iInteraction Viewpoint control Traditional 3D user interface Kiyokawa, et al. 2000 interaction: manipulation, selection, etc.
  • 12. AR 3D Interaction
  • 13. AR G ffiti
  • 14. Advantages and DisadvantagesImportant class of AR interfaces Entertainment, design, trainingAdvantages User can interact with 3D virtual object everywhere i space bj t h in Natural, familiar interactionDisadvantages Usually no tactile feedback User has to use different devices for virtual and physical objects Oshima, et al. 2000
  • 15. Augmented S f Surfaces andTangible Interfaces Basic principles Virtual objects are projected on a surface Physical objects are used as controls f virtual l for l objects Support for collaboration
  • 16. Augmented SurfacesRekimoto, et al. 1998 Front projection F j i Marker-based tracking Multiple projection surfaces
  • 17. Tangible User Interfaces (Ishii 97)Create digital shadowsfor h i l bjf physical objectsForeground g graspable UIBackgroundB k d ambient interfaces
  • 18. Tangible Interfaces - Ambient Dangling String D l S Jeremijenko 1995 Ambient ethernet monitor A bi h i Relies on peripheral cues Ambient Fixtures Dahley, Wisneski, Ishii 1998 Use natural material qualities for information display
  • 19. Tangible Interface: ARgrooveCollaborative InstrumentExploring Physically Based Interaction Map physical actions to Midi output - Translation rotation Translation, - Tilt, shake
  • 20. ARgroove in Use
  • 21. Visual FeedbackContinuous Visual Feedback is KeySingle Virtual Image Provides: Rotation Tilt Tl Height
  • 22. i/O Brush (Ryokai, Marti, Ishii)
  • 23. Other ExamplesTriangles (Gorbert 1998) Triangular based story telling g y gActiveCube (Kitamura 2000-) Cubes C b with sensors h
  • 24. Lessons from Tangible InterfacesPhysical objects make us smart Norman’s “Things that Make Us Smart” g encode affordances, constraintsObjects aid collaboration establish shared meaningObjects increase understanding serve as cognitive artifacts
  • 25. TUI LimitationsDifficult to change object properties can’t tell state of digital dataLimited display capabilities projection screen = 2D dependent on physical display surfaceSeparation between object and display p j p y ARgroove
  • 26. Advantages and DisadvantagesAdvantages Natural - users hands are used for interacting with both virtual and real objects. - N need for special purpose input devices No df i l i td iDisadvantages g Spatial gap - Interaction is limited only to 2D surface • Full 3D interaction and manipulation is difficult - Separation between interaction object and display
  • 27. Orthogonal Nature of AR Interfaces
  • 28. Back to the Real WorldAR overcomes limitation of TUIs enhance display possibilities merge task/display space provide public and private viewsTUI + AR = Tangible AR g Apply TUI methods to AR interface design
  • 29. Space vs. Time - MultiplexedSpace-multiplexed Many devices each with one function y - Quicker to use, more intuitive, clutter - Real ToolboxTime multiplexedTime-multiplexed One device with many functions - Space efficient - mouse
  • 30. Tangible AR: Tiles (Space Multiplexed) Tiles T l semantics data tiles operation tiles i il Operation on tiles proximity spatial arrangements space-multiplexed lti l d
  • 31. Space-multiplexed Interface Data authoring in Tiles
  • 32. Proximity-based Interaction
  • 33. Object Based Interaction: MagicCup Intuitive Virtual Object Manipulation on a Table-Top Workspace Time multiplexed Multiple Markers M li l M k - Robust Tracking Tangible User Interface - Intuitive Manipulation Stereo Display - Good Presence
  • 34. MagicCup systemMain table, Menu table, Cup interface
  • 35. Tangible AR: Time-multiplexed Interaction Use of natural physical object manipulations to j control virtual objects VOMAR Demo Catalog book: - Turn over the page p g Paddle operation: - Push, shake, incline, hit, scoop
  • 36. VOMAR Interface
  • 37. Advantages and DisadvantagesAdvantages Natural interaction with virtual and physical tools - No need for special purpose input devices Spatial interaction with virtual objects - 3D manipulation with virtual objects anywhere in physical spaceDisadvantages Requires Head Mounted Display
  • 38. Wrap-upBrowsing Interfaces simple (conceptually!), unobtrusive (conceptually!)3D AR Interfaces expressive, creative, require attentionTangible Interfaces Embedded into conventional environmentsTangibleT ibl AR Combines TUI input + AR display p p y
  • 39. Designing ARApplications
  • 40. Interface Design Path1/ Prototype Demonstration2/ Adoption of Interaction Techniques from other interface metaphors p Augmented Reality3/ Development of new interface metaphors appropriate to the medium Virtual Reality4/ Development of formal theoretical models for predicting and modeling user actions Desktop WIMP
  • 41. AR Design Space Reality Virtual Reality Augmented R lit A t d RealityPhysical Design Virtual Design
  • 42. AR is mixture of physical affordance andvirtual affordancePhysical Tangible T ibl controllers and objects ll d bjVirtual Virtual graphics and audio
  • 43. AR Design PrinciplesInterface Components Physical components Display elements - Visual/audio Interaction metaphors Physical y Display p y Elements Interaction Elements Metaphor Input Output
  • 44. Tangible AR MetaphorAR overcomes limitation of TUIs enhance dis la possibilities display ssibilities merge task/display space provide public and private viewsTUI + AR = Tangible AR Apply TUI methods to AR interface design
  • 45. Tangible AR Design PrinciplesTangible AR Interfaces use TUI principles Physical controllers for moving virtual content Support for spatial 3D interaction techniques Support for multi-handed interaction Match object affordances to task requirements j q Support parallel activity with multiple objects Allow collaboration between multiple users
  • 46. Case Study 1: 3D AR LensGoal: Develop a lens based AR interface MagicLenses Developed at Xerox PARC in 1993 View Vi a region of the workspace differently to the rest i f h k diff l h Overlap MagicLenses to create composite effects
  • 47. 3D MagicLensesMagicLenses extended to 3D (Veiga et. al. 96) Volumetric and flat lenses
  • 48. AR Lens Design PrinciplesPhysical Components Lens handle - Virtual lens attached to real objectDisplay Elements Lens view - Reveal layers in datasetInteraction Metaphor Physically holding lens Ph i ll h ldi l
  • 49. 3D AR Lenses: Model ViewerDisplays models made up of multiple partsEach part can be shown or hidden through the lensAllows the user to peer inside the modelMaintains focus + context
  • 50. AR Lens Demo
  • 51. AR FlexiLensReal handles/controllers with flexible AR lens
  • 52. Techniques based on AR LensesObject Selection Select objects by targeting them with the lensInformation Filtering Show different representations through the lens Hide certain content to reduce clutter, look inside things
  • 53. Case Study 2 : LevelHeadBlock based game
  • 54. Case Study 2: LevelHeadPhysical Components Real blocksDisplay Elements Virtual person and roomsInteraction MI i Metaphor h Blocks are rooms
  • 55. Case Study 3: AR Chemistry (Fjeld 2002)Tangible AR chemistry education
  • 56. Goal: An AR application to test molecularG structure in chemistry y Physical Components Real book, R l b k rotation cube, scoop, tracking markers i b ki k Display Elements p y AR atoms and molecules Interaction M t h I t ti Metaphor Build your own molecule
  • 57. AR Chemistry Input Devices
  • 58. Case Study 4: Transitional InterfacesGoal: An AR interface supporting transitions from reality to virtual reality Physical Components Real book Display Elements AR and VR content Interaction Metaphor Book pages hold virtual scenes p g
  • 59. Milgram’s Continuum (1994) Mixed Reality (MR)Reality Augmented Augmented Virtuality y(Tangible Reality (AR) Virtuality (AV) (VirtualInterfaces) Reality) Central Hypothesis yp The next generation of interfaces will support transitions along the Reality-Virtuality continuum
  • 60. TransitionsInterfaces of the future will need to supporttransitions along the RV continuum ii l h iAugmented Reality is preferred for: co-located collaborationImmersive Virtual Reality is preferred for:I i Vi t l R lit i f df experiencing world immersively (egocentric) sharing views remote collaboration
  • 61. The MagicBookDesign Goals: Allows user to move smoothly between reality and virtual reality Support collaboration S ll b i
  • 62. MagicBook Metaphor
  • 63. FeaturesSeamless transition between Reality and Virtuality Reliance on real d R li l decreases as virtual increases i liSupports egocentric and exocentric views User can pick appropriate viewComputer becomes invisible Consistent interface metaphors Virtual content seems realSupports collaboration
  • 64. Design alternatives for common user tasks in Tangible AR User Tasks Interface Design Camera on HMDViewpoint Fixed camera – top view, front view, mirroringControl Handheld camera Statically paired virtual and physical objectsSelection Dynamically paired virtual and physical objects - paddle, pointer D i ll i d i t l d h i l bj t ddl i t Direct mapping of whole 6DOF3D Filtered/distorted mapping - snapping, non-linear mappingManipulation Multiplexed mapping - rotation from one, position from another Location and pose based - proximity, spatial configuration Gestures with props - tilt shake tilt,Event & Keyboard & mouseCommand Menu, 2D/3D GUI with tracking objects as pointers Buttons Occlusion based interaction Custom hardware devices
  • 65. Design Tips for Tangible ARUsing metaphors from the real worldTake advantage of parallel interactionsUse timers to prevent accidentsInteraction volume – user, tracking, whitespaceWhat happens when tracking gets lost or objectis out of view?Problems in visualization Field of view, occlusions ,
  • 66. OSGART:From Registration to Interaction
  • 67. Keyboard and Mouse InteractionTraditional input techniquesOSG provides a framework for handling keyboard p g yand mouse input events (osgGA) 1. Subclass osgGA::GUIEventHandler g 2. Handle events: • Mouse up / down / move / drag / scroll-wheel • Key up / down 3. Add instance of new handler to the viewer
  • 68. Keyboard and Mouse Interaction Create your own event handler classclass KeyboardMouseEventHandler : public osgGA::GUIEventHandler {public: KeyboardMouseEventHandler() : osgGA::GUIEventHandler() { } virtual bool handle(const osgGA::GUIEventAdapter& ea,osgGA::GUIActionAdapter& aa, osg::Object* obj, osg::NodeVisitor* nv) { switch (ea.getEventType()) { // Possible events we can handle case osgGA::GUIEventAdapter::PUSH: break; case osgGA::GUIEventAdapter::RELEASE: break; case osgGA::GUIEventAdapter::MOVE: break; case osgGA::GUIEventAdapter::DRAG: break; case osgGA::GUIEventAdapter::SCROLL: break; case osgGA::GUIEventAdapter::KEYUP: b d break; k case osgGA::GUIEventAdapter::KEYDOWN: break; } return false; }}; Add it to the viewer to receive eventsviewer.addEventHandler(new KeyboardMouseEventHandler());
  • 69. Keyboard Interaction Handle W,A,S,D keys to move an objectcase osgGA::GUIEventAdapter::KEYDOWN: { switch (ea.getKey()) { case w: // Move forward 5mm localTransform->preMult(osg::Matrix::translate(0, -5, 0)); return true; case s: // Move back 5mm localTransform->preMult(osg::Matrix::translate(0, 5, 0)); return true; case a: // Rotate 10 degrees left localTransform->preMult(osg::Matrix::rotate(osg::DegreesToRadians(10.0f), osg::Z_AXIS)); return true; case d: // Rotate 10 degrees right localTransform->preMult(osg::Matrix::rotate(osg::DegreesToRadians(-10.0f), osg::Z_AXIS)); return t t true; case : // Reset the transformation localTransform->setMatrix(osg::Matrix::identity()); return true; }break;localTransform = new osg::MatrixTransform();localTransform->addChild(osgDB::readNodeFile("media/car.ive"));arTransform->addChild(localTransform.get());arTransform >addChild(localTransform get());
  • 70. Keyboard Interaction Demo
  • 71. Mouse InteractionMouse is pointing device…Use mouse to select objects in an AR sceneOSG provides methods for ray-casting andintersection testing Return an osg::NodePath (the p g ( path from the hit node all the way back to the root) Projection Plane (screen) scene
  • 72. Mouse Interaction Compute the list of nodes under the clicked position Invoke an action on nodes that are hit, e.g. select, deletecase osgGA::GUIEventAdapter::PUSH: osgViewer::View* view = dynamic_cast<osgViewer::View*>(&aa); osgUtil::LineSegmentIntersector::Intersections intersections; // Clear previous selections for ( f (unsigned i t i = 0 i < t i d int 0; targets.size(); i++) { t i () targets[i]->setSelected(false); } // Find new selection based on click position if (view && view->computeIntersections(ea.getX(), ea.getY(), intersections)) { for (osgUtil::LineSegmentIntersector::Intersections::iterator iter = intersections.begin(); iter != intersections.end(); iter++) { if (Target* target = dynamic_cast<Target*>(iter->nodePath.back())) { std::cout << "HIT!" << std::endl; target >setSelected(true); target->setSelected(true); return true; } } } break;
  • 73. Mouse Interaction Demo
  • 74. Proximity TechniquesInteraction based on the distance between a marker and the camera the distance between multiple markers
  • 75. Single Marker Techniques: ProximityUse distance from camera to marker asinput parameter e.g. e g Lean in close to examineCan use the osg::LOD class to showdifferent content at different depthranges Image: OpenSG Consortium
  • 76. Single Marker Techniques: Proximity// Load some modelsosg::ref_ptr<osg::Node> farNode = osgDB::readNodeFile("media/far.osg");osg::ref_ptr<osg::Node> closerNode = osgDB::readNodeFile("media/closer.osg");osg::ref_ptr<osg::Node> nearNode = osgDB::readNodeFile("media/near.osg");// Use a Level-Of-Detail node to show each model at different distance ranges.osg::ref_ptr<osg::LOD> lod = new osg::LOD();lod->addChild(farNode.get(), 500.0f, 10000.0f); // Show the "far" node from 50cm to 10m awaylod->addChild(closerNode.get(), 200.0f, 500.0f); // Show the "closer" node from 20cm to 50cm awaylod->addChild(nearNode.get(), 0.0f, 200.0f); // Show the "near" node from 0cm to 2cm awayarTransform->addChild(lod.get()); Define depth ranges for each node Add as many as you want Ranges can overlap
  • 77. Single Marker Proximity Demo
  • 78. Multiple Marker ConceptsInteraction based on the relationship betweenmarkers e.g. When the distance between two markers decreases below threshold invoke an action Tangible User InterfaceApplications:A l Memory card g y games File operations
  • 79. Multiple Marker Proximity Virtual Camera Transform A Transform B Distance > Threshold Switch A Switch BModel Model Model Model A1 A2 B1 B2
  • 80. Multiple Marker Proximity Virtual Camera Transform A Transform B Distance <= Threshold Switch A Switch BModel Model Model Model A1 A2 B1 B2
  • 81. Multiple Marker Proximity Use a node callback to test for proximity and update the relevant nodesvirtual void operator()(osg::Node* node, osg::NodeVisitor* nv) { if (mMarkerA != NULL && mMarkerB != NULL && mSwitchA != NULL && mSwitchB != NULL) { if (mMarkerA->valid() && mMarkerB->valid()) { osg::Vec3 posA = mMarkerA->getTransform().getTrans(); osg::Vec3 posB = mMarkerB->getTransform().getTrans(); osg::Vec3 offset = posA - posB; float distance = offset.length(); if (distance <= mThreshold) { if (mSwitchA->getNumChildren() > 1) mSwitchA->setSingleChildOn(1); if (mSwitchB->getNumChildren() > 1) mSwitchB->setSingleChildOn(1); } else { if (mSwitchA->getNumChildren() > 0) mSwitchA->setSingleChildOn(0); if (mSwitchB->getNumChildren() > 0) mSwitchB->setSingleChildOn(0); } } } traverse(node,nv);}
  • 82. Multiple Marker Proximity
  • 83. Paddle Interaction Use U one marker as a tool for selecting and k lf l d manipulating objects (tangible user interface) Another A h marker provides a frame of reference k id f f f A grid of markers can alleviate problems with occlusionMagicCup (Kato et al) VOMAR (K t et al) (Kato t l)
  • 84. Paddle InteractionOften useful to adopt a local coordinate system Allows the camera to move without disrupting Tlocal Places the paddle in the th same coordinate di t system as the content on the grid Simplifies interactionosgosgART co putes Tlocal using the osgART::LocalTransformationCallback computes us g t e osg :: oca a s o at o Ca bac
  • 85. Tilt and Shake InteractionDetect types of paddle movement: yp p Tilt - gradual change in orientation Shake - short sudden chan es in translation short, s dden changes
  • 86. More Information• Mark Billinghurst Billingh rst –• Gun Lee – g @ g• Websites – www hitlabnz org
  • 87. Building Tangible AR Interfaces with ARToolKit
  • 88. Required CodeCalculating Camera PC l l C Position Range to markerLoading Multiple Patterns/Models /Interaction between objects Proximity Relative position/orientationOcclusion Stencil buffering Multi-marker tracking
  • 89. Tangible AR Coordinate Frames
  • 90. Local vs. Global InteractionsLocal Actions determined from single camera to marker transform - shaking, appearance, relative position, rangeGlobal Actions determined from two relationships - marker to camera, world to camera coords. - Marker transform determined in world coordinates • object tilt, absolute position, absolute rotation, hitting
  • 91. Range-based Interaction Sample Fil R S l File: RangeTest.c T t/* get the camera transformation */ h f iarGetTransMat(&marker_info[k], marker_center, marker_width, marker_trans); marker width marker trans);//* find the range */ /Xpos = marker_trans[0][3];Ypos = marker_trans[1][3]; p [ ][ ]Zpos = marker_trans[2][3];range = sqrt(Xpos*Xpos+Ypos*Ypos+Zpos*Zpos);
  • 92. Loading Multiple PatternsSample F l L dM lS l File: LoadMulti.c Uses object.c to loadObject Structuretypedef struct { char name[256]; 2 6 int id; int visible;; double marker_coord[4][2]; double trans[3][4]; double d bl marker_width; k idth double marker_center[2];} ObjectData_T; _
  • 93. Finding Multiple Transforms Create object list CObjectData_T *object; Read in objects - in init( )read_ObjData(read ObjData( char *name int *objectnum ); *name, Find Transform – in mainLoop( ) p(for( i = 0; i < objectnum; i++ ) { ..Check patterns p ..Find transforms for each marker }
  • 94. Drawing Multiple Objects Send the object list to the draw functiondraw( object objectnum ); object, Draw each object individuallyfor( i = 0; i < objectnum; i++ ) { if( object[i].visible == 0 ) continue; argConvGlpara(object[i].trans, gl_para); draw_object( object[i].id, gl_para);}
  • 95. Proximity Based InteractionSample File – CollideTest.cDetect distance between markerscheckCollisions(object[0],object[1], DIST)If distance < collide distanceThen change the model/perform interaction
  • 96. Multi-marker TrackingSample File – multiTest.cMultiple markers to establish asingle coordinate frame g Reading in a configuration file Tracking from sets of markers T ki f t f k Careful camera calibration
  • 97. MultiMarker Configuration FileSample File - Data/multi/marker dat Data/multi/marker.datContains list of all the patterns and their exactpositions ii #the number of patterns to be recognized 6 Pattern File #marker 1 Pattern Width + Data/multi/patt.a Coordinate Origin 40.0 0.0 0.0 Pattern Transform 1.0000 0.0000 0.0000 -100.0000 Relative to Global 0.0000 1.0000 0.0000 50.0000 Origin 0.0000 0.0000 1.0000 0.0000 0 0000 0 0000 1 0000 0 0000 …
  • 98. Camera Transform CalculationInclude <AR/arMulti.h>I l d <AR/ M lti h>Link to libARMulti.libIn mainLoop() Detect markers as usual arDetectMarkerLite(dataPtr, thresh, &marker_info, &marker_num) Use MultiMarker Function if( (err=arMultiGetTransMat(marker_info, marker_num, config)) < 0 ) { k fi )) argSwapBuffers(); return; }
  • 99. Paddle-based InteractionTracking single marker relative to multi-marker set - paddle contains single marker p g
  • 100. Paddle Interaction Code Sample File – PaddleDemo.c Get paddle marker location + draw paddle before drawing p p g background model paddleGetTrans(paddleInfo, marker_info, marker_flag, marker_num, marker flag marker num &cparam); /* draw the paddle */ if( paddleInfo->active ){ draw_paddle( draw paddle( paddleInfo); }draw_paddledraw paddle uses a Stencil Buffer to increase realism
  • 101. Paddle Interaction Code IISample File – paddleDrawDemo.cFinds the paddle position relative to global coordinate frame:setBlobTrans(Num,paddle_trans[3][4],base_trans[3][4])Sample File – paddleTouch.cFinds the paddle position:findPaddlePos(&curPadPos,paddleInfo->trans,config->trans);Checks for collisions:checkCollision(&curPaddlePos,myTarget[i].pos,20.0)checkCollision(&curPaddlePos myTarget[i] pos 20 0)
  • 102. General Tangible AR Librarycommand_sub.c, command_sub.h d b d bhContains functions for recognizing a range ofdifferent paddle motions:diff ddl iint check_shake( );inti t check_punch( ) h k h( );int check_incline( );int check pickup( ); check_pickup(int check_push( );Eg: to check angle between paddle and basecheck_incline(paddle->trans, base->trans, &ang)