COSC 426 Lect. 6: Collaborative AR


Published on

Lecture on Collaborative Augmented Reality given to the COSC 426 graduate class in AR. Taught by Mark Billinghurst from the HIT Lab NZ at the University of Canterbury.

  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

COSC 426 Lect. 6: Collaborative AR

  1. 1. Lecture 6: Collaborative AR Applications pp Mark Billinghurst g HIT Lab NZ University of Canterbury
  2. 2. Collaboration: What’s that ? Definitionf.n.f n a joint work ; between several peoples who generate the creation of a shared task
  3. 3. Collaborative activitiesCollaboration: Work, LeisureCSCW (Computer Supported Collaborative Work)Groupware
  4. 4. Collaboration: TaxonomyTime/Space [Ellis91] Time bu e boa d bulletin board pos , email, a post, e a , fax asynchronous Shared editor meeting, chat, brainstorming, teleconference synchronous seminar Space colocated distant
  5. 5. Today’s TechnologyVideo Conferencing lack of spatial cues limited participants 2D collaborationCollaborative Virtual Environments separation from real world reduced conversational cues
  6. 6. Beyond Video Conferencing2D Interface onto 3D VRMLProjection Screen CAVE, WorkBenchVolumetric Display p y scanning laserVirtual Reality natural spatial cues
  7. 7. Beyond Virtual RealityLessons from CSCW Seamless Enhance RealityImmersive Virtual Reality separates from real world reduces conversational cues
  8. 8. Collaboration in the Future ?Remote Conferencing Face to face Conferencing
  9. 9. Central ClaimAugmented Reality techniques can be used to providespatial cues that significantly enhance face-to-face andremote collaboration on three-dimensional tasks.
  10. 10. Construct3D [Kaufmann 2000]Collaborative geometry education tool g yDifferent learning modes (teacher, student, exam)Tangible interaction – personal interaction panel
  11. 11. Collaborative Augmented Reality Seamless Interaction Natural Communication Attributes: Att ib t Virtuality Augmentation Cooperation Independence Individuality
  12. 12. Seamless CSCWSeam (Ishii et. al.) spatial, temporal, functional discontinuityTypes of Seams Functional - between different functional workspaces Cognitive g - between different work practices
  13. 13. Functional Seams
  14. 14. Cognitive Seams
  15. 15. Effect of SeamsFunctional Seams: Mediated differs from F-to-F Conversation - Loss of Gaze Information - Degradation of Non-Verbal CuesCognitive Seams: Learning Curve Effects User Frustration
  16. 16. Unanswered QuestionsDoes seamlessness enhance performance?WhatWh AR cues can enhance collaboration ? h ll bHow does AR collaboration differ ?What technology is required ?...
  17. 17. Collaborative AR InterfacesFace to Face CollaborationWebSpace, Shared Space, Table Top Demo, Interface p , p , p , Comparison, AR Interface ComparisonRemote CollaborationSharedView, RTAS, Wearable Info Space, WearCom, AR Conferencing, BlockPartyTransitional InterfacesT lI fMagicBookHybrid InterfacesAR PRISM, GI2VIS
  18. 18. Face to Face Collaboration
  19. 19. Communication Cues A wide variety of communication cues used. sua Visual Audio Gaze GestureSpeech Face ExpressionParalinguistic Body PositionParaverbalsProsodicsIntonation Object Manipulation Writing/Drawing Environmental Spatial Relationship Object Presence
  20. 20. In computer supported collaboration, however, it is often hard for users to exchange non-verbal communication g cues, even when they are co-located.
  21. 21. Face-to-face collaboration People surround a table It is easy to see each other Communication S C i ti Space Task S T k SpaceComputer supportedcollaboration ll b ti People sit side by side It is hard to see each other i h dt h th Communication Space Task Space
  22. 22. Shared Space - Table Top DemoGoal create compelling collaborative AR p g interface usable by novicesExhibit content matching card game face to face collaboration physical interaction
  23. 23. Results2,500 3,0002 500 - 3 000 usersObservations no problems with the interface bl ih h i f - only needed basic instructions physical objects easy to manipulate spontaneous collaborationSubjective survey (157 people) Users felt they could easily play with other people and interact with objects jImprovements reduce lag, improve image quality, better HMD g, p g q y,
  24. 24. AR PadHandheld AR Display LCD screen SpaceOrb Camera Peripheral awareness P h l
  25. 25. Support for Collaboration Virtual Viewpoint Visualization
  26. 26. Face to Face CollaborationCompare two person collaboration in: Face to Face, AR, Projection DisplayTask Urban design logic puzzle - Arrange 9 building to satisfy 10 rules in 7 minutesSubjects Within subjects study (counter-balanced) 12 pairs of college students
  27. 27. Face to Face ConditionMoving Model Buildings
  28. 28. AR ConditionCards with AR ModelsSVGA AR Display (800x600)Video see-through AR g
  29. 29. Projection ConditionTracked Input Devices
  30. 30. Task Space Separation
  31. 31. Interface Conditions FtF AR ProjectionUser Viewpoint p Independent Private Public Easy to change Independent Common Easy to change Difficult to change Limited FOVInteraction Two handed Two handed Mouse-based Natural object Tangible AR One-handed manipulation techniques Time-multiplexed Space-multiplexed Space-multiplexed
  32. 32. HypothesisCollaboration with AR technology will produce behaviors that are more like natural face-to- face collaboration than from using a screen- screen based interface.
  33. 33. MetricsSubjective Evaluative survey after each condition Forced-choice survey after all conditions Post experiment interviewObjective j Communication measures - Video transcription p
  34. 34. Measured ResultsPerformance AR collaboration slower than FtF + Projection jCommunication Pointing/Picking gesture behaviors same in AR as FtF Deictic speech patterns same in AR as FtF - Both significantly different than Projection condition g y jSubjective FtF easier to work together and understand Interaction in AR easier than Proj. and same as FtF
  35. 35. Deictic Expressions 30% 25% 20% 15% 10% 5% 0% FtF Proj ARSignificant difference – ANOVA, F(2,33) = 5.77, P < 0.01No difference between FtF and AR
  36. 36. Ease of InteractionSignificant d ffS f difference Pick - F(2,69) = 37.8, P < 0.0001 Move - F(2,69) = 28.4, P < 0.0001
  37. 37. Interview Comments“AR’s biggest limit was lack of peripheral vision. The interaction was natural, it was just difficult to see. In the projection condition you could see everything but the interaction was tough” Face to Face Subjects focused on task space - gestures easy to see gaze difficult see, Projection display Interaction difficult (8/14) - not mouse-like, invasion of space AR display – “working solo together” Lack of peripheral cues = “tunnel vision (10/14 people) tunnel vision”
  38. 38. Face to Face Summary Collaboration is partly a Perceptual task AR reduces perceptual cues -> Impacts collaboration Tangible AR metaphor enhances ease of interaction Users felt that AR collaboration different from FtF But: measured speech and gesture behaviors in AR condition is more similar to FtF condition than in Projection displayThus we need to design AR interfaces that don’t reduce perceptual cues, while k h l keeping ease of interaction f
  39. 39. Collaborative ARAR Tennis Virtual tennis court Two user game Audio + haptic feedback Bluetooth messaging
  40. 40. Research QuestionsDoes having an AR interface enhance the faceto face gaming experience? g g p
  41. 41. AR ExperienceConditions A: Face to Face AR B: Face to Face non-AR C: Non Face to FaceMeasures Survey questions Ranking
  42. 42. Survey Questions1/ How easy was it to work with your partner?2/ How easily did your partner work with you? y y p y3/ How easy was it to be aware of what your partner was doing?4/ How enjoyable was the game?Answered on a Likert scale from 1 to 7 1 = not very easy, 7 = very easy
  43. 43. Results RankingsUser felt AR condition was much easier to see what theirppartner was doing (Q3) g( )Easier to work with partner in AR case (Q1)Easier for partner to work together in AR case (Q2)Face to Face AR ranked best
  44. 44. Observations and ConclusionsObservations Tangible input proved to be very intuitive Users preferred audio over haptic cues as it provided greater awareness Players adapted behavior to interfaceConclusions Collaborative AR enhances face to face awareness
  45. 45. Remote Collaboration
  46. 46. 2008 - CNN
  47. 47. AR ConferencingMoves conferencing from the desktop to the workspace
  48. 48. FeaturesHardware SGI O2 Virtual i-O HMD head mounted cameraSoftware live video shared whiteboard vision based registration/tracking
  49. 49. Pilot StudyHow does AR conferencing differ ? Task discussing images 12 pairs of subjects Conditions audio only ( ) y (AC) video conferencing (VC) mixed reality conferencing (MR)
  50. 50. Sample Transcript
  51. 51. Transcript AnalysisUsers speak most in Audio Only conditionMR fewest words/min and interruptions/minMore results needed
  52. 52. Presence and Communication Presence Rating (0-100)100 908070605040302010 Could tell when Partner was Concentrating 0 14 AC VC MR 12 10 8 6 4 2 0 AC VC MR
  53. 53. Subjective CommentsPaid more attention to picturesRemote video provided peripheral cuesIn AR condition Difficult to see everything Remote user distracting Communication asymmetries
  54. 54. A Wearable Conferencing SpaceFeatures mobile video conferencing full size images spatial audio/visual cues interaction with real world dozens of users body-stabilized data
  55. 55. Initial PrototypeInternet TelephonySpatial Audio/VisualsSee-through HMD gHead TrackingWireless InternetWearable ComputerStatic Images
  56. 56. Software ArchitectureMulticast GroupsPosition BroadcastingP B d 10 kb/s per personAudio Broadcasting 172 kb/s per personLocal sound spatialization DirectSound3DGraphics Interface DirectX/Direct3D
  57. 57. Pilot User StudyCan MR spatial cues aid comprehension?Task recognize words in spoken phrasesConditions Number of speakers - 1,3,5 simultaneous speakers Spatial/Non Spatial Audio Visual/Non visual cues
  58. 58. Spatial Sound Spatial vs. Non Spatial Performance 5 4Score 3 SS 2 NS 1 0 1 3 5 Num Speakers
  59. 59. Subjective Ratings Ease of Understanding 6 5 5) ating (1-5 4 SS 3 NS 2Ra 1 0 1 3 5 Num Speakers
  60. 60. 3D Live: Virtual Viewpoint Generation
  61. 61. Virtual Viewpoint GenerationInterpolate between real camera views togenerate a virtual camera view
  62. 62. AR ConferencingSuperimpose video of remote person over real world
  63. 63. System Architecture
  64. 64. Tangible ManipulationUsing real paddle to move virtual user
  65. 65. AR Remote ConferencingProgression 2D to Spatial Cues to 3D S i lC Increasing realism (visual/audio cues)
  66. 66. Multiscale Collaboration
  67. 67. The MagicBookGoal: A collaborative AR interface supporting transitions from reality to virtual reality Physical Components Real book Display Elements p y AR and VR content Interaction Metaphor Book pages hold virtual scenes
  68. 68. Milgram’s Reality-Virtuality continuum Mixed Reality Real Augmented Augmented VirtualEnvironment Reality (AR) Virtuality (AV) Environment Reality - Virtuality (RV) Continuum
  69. 69. Milgram’s Continuum (1994) Mixed Reality (MR)Reality Augmented Augmented Virtuality y(Tangible Reality (AR) Virtuality (AV) (VirtualInterfaces) Reality) Central Hypothesis yp The next generation of interfaces will support transitions along the Reality-Virtuality continuum
  70. 70. TransitionsInterfaces of the future will need to support transitionsalong the RV continuumAugmented Reality is preferred for: co-located collaborationImmersive Virtual Reality is preferred for: experiencing world immersively (egocentric) sharing views remote c llab rati n rem te collaboration
  71. 71. MagicBook Metaphor
  72. 72. FeaturesSeamless transition between Reality and Virtuality Reliance on real d R li l decreases as virtual increases i liSupports egocentric and exocentric views User can pick appropriate viewComputer becomes invisible Consistent interface metaphors Virtual content seems realSupports collaboration
  73. 73. CollaborationCollaboration on multiple levels: Physical Object y j AR Object Immersive Virtual SpaceEgocentric + exocentric collaboration multiple multi-scale users pIndependent Views Privacy, role division, scalability
  74. 74. TechnologyReality No technologyAugmented Reality Camera – tracking Switch – fly inVirtual Reality Compass – tracking Press pad – move Switch – fly out
  75. 75. Scientific Visualization
  76. 76. Education
  77. 77. Hybrid Interfaces
  78. 78. Hybrid User InerfacesGoal: ToG l T incorporate AR into normal meeting l environment Physical Components Real props Display El Di l Elements t 2D and 3D (AR) displays Interaction Metaphor Use multiple tools – each relevant for the task
  79. 79. Hybrid User Interfaces 1 2 3 4 PERSONAL TABLETOP WHITEBOARD MULTIGROUPPrivate Display Private Display Private Display Private Display Group Display Public Display Group Display Public Display
  80. 80. Bridging Space (1) Office of the Future [Raskar98] office environment augmented with embedded front projection 3D video conferencing
  81. 81. Bridging Space (2)Emmie [Butz99] Shared virtual “ether“ metaphor Incorporate existing standard applications
  82. 82. Bridging S B id i Space (3)Studierstube (V2.0) [Schmalstieg2000] Similar multi-display AR Mixed view applications Example: Storyboard design
  83. 83. MagicMeeting [Regenbrecht 2002]Hybrid Interface for Face to Face Collaboration AR view, Projection Screen, Desktop, PDA j pTangible Interaction Physical manipulators
  84. 84. Tangible Interactions Cake Platter Rotation Real Props – Cli i Pl R lP Clipping Plane, Obj t Li hti Object Lighting
  85. 85. Conclusions
  86. 86. Lessons LearnedFace to face collaboration AR preferred over immersive VR f d i i AR facilitates seamless/natural communicationRemote C ll b tiR t Collaboration AR spatial cues can enhance communication AR conferencing improves video conferencing f i i id f i Many possible confounding factors
  87. 87. Future WorkWearable collaborative AR system opportunistic collaboration just in time trainingCommunication Asymmetries interface, expertise, rolesUsability Studies y multi-user AR systems communication tasks