Downloaded 623 times
Uploaded byJoe Lamantia
Social Interaction Design For Augmented Reality: Patterns and Principles for Playing Well With Others
The document discusses the evolution and implications of augmented reality (AR) and its interaction with social dynamics, emphasizing that current AR technologies often lack social maturity. It highlights the need for AR experiences to enhance rather than replace social interactions, avoid creating critical points of failure, and adhere to local social norms. Finally, it presents design principles for successful social augmented reality, advocating for equal value among elements and responsiveness to user context.
More Related Content
Similar to Social Interaction Design For Augmented Reality: Patterns and Principles for Playing Well With Others
Social Interaction Design For Augmented Reality: Patterns and Principles for Playing Well With Others
- 1. Social Design For Augmented Reality The WebAnd Beyond Amsterdam | June 2010
- 2. Joe Lamantia Experience Design& Strategy @moJoe JoeLamantia.com Joe.Lamantia@gmail.com
- 3. The future... Beyond Findability:Frameworks IA Summit 2009 | Joe Lamantia | MediaCatalyst 3
- 4.
- 5.
- 6.
- 7.
- 8. Cloud Computing Augmente d
- 9. Augmented Reality Ubiquitous Computing
- 10. Engagement Spaces Environments Systems Architectures Interactions Experiences
- 11. Engagement Spaces Environments Systems Architectures Interactions Experiences Humanity
- 13.
- 14.
- 15. “Cyberspace has leaked intothe real world. …what was inside the box …is outside.” Vernor Vinge
- 16.
- 17. Mixed Reality Real Augmented Augmented Virtual Environment Reality (AR) Virtuality (AV) Environment
- 18. Augmented Reality “Blends reality& computer-generated elements”
- 19.
- 20. Social Experiences “Support andengage in social interactions”
- 21. Social Experiences between people synchronous sociallysituated active
- 22. Social Augmented Experiences “Blendreality & computer-generated elements to support and engage in social interactions”
- 23. Social Augmented Experiences real-world real-time between people generated synchronous tangible socially situated active
- 24.
- 26. AR Concept Face recognition Close-rangescanning of faces for automated recognition and profile lookup via mobile devices.
- 27. Interaction Design Tricorder Pattern Problematic interactions Social pattern is worse...
- 28. Social Pattern Like beingstopped by the police for ID. Or security scanned! “Show me your papers.”
- 29. Experience “Anyone pointing a devicein my direction to try to identify me better be prepared for either a law suit, or a punch in the face.” Anonymous Comment
- 30. AR interaction design 4patterns 4 anti-patterns
- 31.
- 33.
- 35. “People donʼt necessarily wantto walk around the world holding cell-phone screens in front of their faces.”
- 36.
- 38.
- 40.
- 41.
- 42.
- 43.
- 44. Pay No AttentionTo That Man Behind The Curtain
- 46.
- 48. Anti Patterns: Loner Second HandSmoke The Man Behind the Curtain Tunnel Vision
- 49. Anti Patterns: Loner Second HandSmoke The Man Behind the Curtain Tunnel Vision
- 50.
- 51. Anti-social behavior often indicateslow ʻsocial maturityʼ in people Social maturity = ability to • understand social dynamics • follow social norms • engage socially with others • adapt to social situations
- 52. Bad News Current ARexperiences show generally low social maturity. Good News Immaturity is common for emerging media & spaces. Technology ʻmaturesʼ rapidly,as the rise of the Social / 2.0 Web. AR is ʻyoung and malleableʼ.
- 53. “augmented reality is ...stalling,waiting for a breakthrough either technological or conceptual... (much like Foursquare was for location based services) ” Augmented Times
- 54. Social Maturation Paths Immature Social Individual Integrated w/ social contexts Optimized for individual contexts Supports social dynamics Supports a/semi-social dynamics Social interaction patterns Individual interaction patterns
- 55.
- 57. Be Human ByDefault Design experiences that echo human behavior and expectations.
- 59. Looks like a wayto talk to people far away by video.
- 60. Itʼs really just adoor.
- 61. Enhancement, Not Replacement Enhancesocial interactions, instead of replacing them with AR gimmicks. Anti-pattern: “AR for ARʼs Sake” lowers experience value in the present, though creating comedy for the future.
- 62.
- 63. Build Real Bridges Socialtechnologies succeed when they enable people to overcome genuine barriers to interaction. ʻBridgeʼ experiences are valid only when interaction is impossible.
- 65. Successful Interaction depends onmany details.
- 66. Avoid the CriticalPath AR elements should be ʻoptionalʼ for social interactions. Making AR elements essential creates a ʻsingle point of failureʼ.
- 70. Let People ManageSocial Complexity Simple designs allow people to manage social complexity and interactions as they need and choose.
- 71. Empathic response torobots
- 72.
- 73.
- 74. Interactions inspire feelings of creepiness.
- 75. Avoid The UncannyValley Empathic response to robots
- 76. Just made a baddeal.
- 77. Equal Value All augmented elementsmust be equally valuable to all participants.
- 79. Are We OnAiR? Indicate the presence & status of AR elements to all participants.
- 81. Context Is King Followlocal norms for interaction. Trumps all other design principles!
- 82. Be Human ByDefault Enhancement, Not Replacement Build Real Bridges Avoid the Critical Path Let People Manage Social Complexity Avoid the Uncanny Valley Equal Value Are We On AiR? Context Is King
- 83.
- 84. Social Maturation Paths Immature Social Individual Integrated w/ social contexts Optimized for individual contexts Supports social dynamics Supports a/semi-social dynamics Social interaction patterns Individual interaction patterns
- 85. AR Flashmob Amsterdam “...nerds looking through a phone.” “Its getting busy at the #arflashmob“
- 87. individual AR as ʻsocialobjectʼ
- 88. Social Social Augmented Experiences hybrids Individual AR Individual
- 89. Social Utility ubiquitous accessible public /open mass-market
- 90.
- 91. Additional info UXMatters: EverywareColumn - UXMatters.com Games Alfresco - http://gamesalfresco.com/ Augmented Times - http://artimes.rouli.net/ Augmented Planet - http://www.augmentedplanet.com/ Beyond the Beyond - http://www.wired.com/beyond_the_beyond/ Designing Social Interfaces & Designing For the Social Web SxDSalon - http://blog.sxdsalon.org/ Designing for Social Interaction - http://www.boxesandarrows.com/view/designing-for-social
- 92. Joe Lamantia Experience Design& Strategy @moJoe JoeLamantia.com Joe.Lamantia@gmail.com
Editor's Notes
- #4 http://www.flickr.com/photos/27032340@N00/428068243/
- #12 http://www.flickr.com/photos/lastgunslinger/3354205233/
- #13 http://www.flickr.com/photos/7670505@N02/2236560304/
- #15 http://www.flickr.com/photos/wikitude/3873717201/in/set-72157622291853583
- #16 In 1994 Paul Milgram and Fumio Kishino defined a mixed reality as "...anywhere between the extrema of the virtuality continuum."[4] (VC), where the Virtuality Continuum extends from the completely real through to the completely virtual environment with augmented reality and augmented virtuality ranging between.
- #17 Wikitude mobile AR browser
- #18 key aspects of augmented reality from design perspective
- #19 between people = not human to machine (actors with social norms / perspective in effect) social situated = imbued with meaning based on social context and perspective not simply transactional (e.g recommending a product to a friend vs. making purchase from seller)
- #20 between people = not human to machine (actors with social norms / perspective in effect) social situated = imbued with meaning based on social context and perspective not simply transactional (e.g recommending a product to a friend vs. making purchase from seller)
- #21 key aspects of social augmented experiences from design perspective
- #23 However, as a result of the inherently social nature of augmented reality, we can be sure the value and impact of many augmented experiences depends in large part on how effectively they integrate the social dimensions of real-world settings, in real time. the social maturity of current augmented experiences is similar to that of a young child who is learning the complex rules and norms that determine socially acceptable behavior. With unevenly developed abilities and understanding, fitting into social situations is very difficult.
- #28 The contextual model for this interaction assumes that interrupted or delayed conversations are socially acceptable—though asynchrony is one of the most frustrating aspects of poorly mediated interactions. (‘Can you hear me now...?’) This scenario also depends on directed, gaze-based, device-powered scanning of faces at close personal distances. For strangers who have no affiliation, I would wager this experience is too similar to having a policeman stop you and ask you for identification for most people to consider it an acceptable social interaction. Even among people with existing, but weaker ties like the indirect relationships that loosely link colleagues in a large organization, this feels like a depersonalizing and inherently suspicious way of greeting someone. Using a device in this way also creates a physical barrier between people and, in terms of attention, shifts focus away from the person to the device and the information it presents. This hardly feels like an improvement over the everyday act of introducing yourself or joining a conversation group at a party. Anyone pointing a cell phone or any other device in my direction to try to "identify" me better be prepared for a either a law suit or a punch in the face. Anonymous comment http://www.technologyreview.com/computing/24639/?a=f
- #29 most augmented experiences created using limited set of design patterns for interaction models and concepts http://www.flickr.com/photos/maggiesworld/2502144304/
- #30 The Head-Up Display interaction pattern echoes the targeting and navigation displays in military and other aircraft like that shown in Figure 2. This is the oldest of the AR interaction patterns. Augmented experiences using the Head-Up Display pattern add information about the real objects in view into a complete mixed-reality experience that built-in AR tools and devices generate. While many AR experiences rely on external devices—that is, external to the body—those using the Head-Up Display pattern commonly depend on hardware that is integral to a vehicle or cockpit-like physical setting. Another very important attribute of the Head-Up Display pattern is integration with a fixed point of view—typically the focus of the user’s visual field. This pattern also applies to wearable systems such as the helmet-mounted displays modern infantries use.
- #31 Augmented Driving app from imaGinyze
- #33 The Tricorder interaction pattern became familiar to us in the original Star Trek TV series. Mr. Spock, in a landing party on the surface of some new planet, would use his Tricorder scanning device to explore the local area, as shown in Figure 6. Most of the time, interactions involved waving the Tricorder around in the air in the general direction of interest, while peering at the display screen. This is the dominant pattern of physical behavior when using AR browsers such as Wikitude demand. The essence of the Tricorder interaction pattern is that it adds pieces of information to an existing real-world experience, representing them directly within the combined, augmented-reality, or mixed-reality experience.
- #34 A key characteristic of the Tricorder interaction pattern is that it requires an external device—typically a hand-held device, though this is not essential—to access the augmented aspects of the mixed-reality environment. The Tricorder interaction pattern takes up a lot of the user’s available interaction bandwidth. It requires an external device, active focus on the part of the user, exclusive use of at least one hand, abnormal—at least until social conventions change—movements, and greatly expanded personal space. AR experiences that rely on the Tricorder pattern are not good in crowds, small spaces, or in areas with fast-moving objects or people.
- #35 Sutter, John D. “New Phone Apps Seek to ’Augment’ Reality.” CNN. August 5, 2009. Retrieved August 1, 2009.
- #36 AR adds new and wholly virtual objects directly into the augmented experience, combining them with existing, real objects. The virtual items in Holochess interaction patterns often interact with one another—and sometimes with the real elements of the mixed-reality experience. Commonly marker-based AR.
- #38 the X-ray Vision interaction pattern simulates seeing beneath the surface of objects, people, or places, showing their internal structure or contents. AR experiences using the X-ray Vision pattern often use a combination of projection and rendering—frequently, a schematic or abstracted rendering—of the object of interest, as in Medical Augmented Reality (MAR). Tim O’Reilly Ultimate reference manual
- #41 To reach its potential and avoid dismissal as a novelty technology, augmented reality needs new interaction patterns and experience concepts that address the weaknesses and gaps of this limited set of existing patterns. Only in the early stages of its evolution, augmented reality has the opportunity to refine and expand its range of interaction patterns without disrupting familiar models or incurring substantial costs. http://www.flickr.com/photos/_dvd_/1488286379/
- #42 One of the weakest aspects of the existing interaction patterns for augmented reality is their reliance on single-person, socially disconnected user experiences. The interaction patterns for augmented experience must become adept at creating valuable experiences for both individuals in social settings and groups participating in mixed-reality experiences together. The micro/pico projectors emerging now offer potential for expanding the social envelope of augmented reality, by extending mixed realities directly into the real world. Compact projectors could allow group interactions and experiences without multiple hardware devices. Projecting mixed realities into public, common, or social spaces makes them social by default. http://www.flickr.com/photos/shimmergirl/3614660077/
- #43 Secondhand Smoke Another important social dimension is the indirect experience of augmented reality—that is, how other people around you experience your augmented or mixed-reality experience. The spread of mobile phones, pagers, and other portable communications devices has disrupted or complicated basic social norms like attention, awareness, presence, and conversation. Everyone within environmental range of the guy talking too loudly about his medication, the pager someone has left unattended on a nearby desk buzzing with incoming messages, or the crackberry addict who never makes eye contact during meetings experiences this kind of disruption. http://www.flickr.com/photos/threeperf/417644402/
- #44 Identity mismanagement With tools like augmented ID on the way, what happens if your environmentally aware AR device, service, or application recognizes me and broadcasts my identity locally—or globally—when I want to remain incognito? At least until the advent of effective privacy management solutions—including hardware, software, standards, and legal frameworks—AR experiences that identify people by face, marker, or RFID tag could severely challenge our ability to do ordinary things like get lost in a crowd, sit quietly at the back of a room, or attend a surprise party for a friend.
- #45 the “scramble suit” from Philip K. Dick’s ‘A Scanner Darkly’, as imagined in the Richard Linklater film
- #46 Most AR systems are limited to one visual input such as the camera on a mobile phone. In practical terms, this means they can monitor only a single field of view, in monocular vision. Lacking the depth perception and breadth of human fields of view gives them tunnel vision, limiting their ability to react to stimuli beyond their narrow, monocular view. Additionally, monocular vision makes it difficult to create mixed-reality experiences that mesh smoothly with the human viewpoint. “predatory gaze” of “terminator vision” YDreams offers depth perception now. http://www.flickr.com/photos/jmmpereda/2306035764/
- #47 “Terminator vision”
- #50 It’s easy to say that AR creates ‘anti-social’ experiences http://www.flickr.com/photos/31271394@N07/3754867811/
- #51 More constructive to think in terms of overall ‘social maturity’ as social science understands it. The Vineland Social Maturity Scale measures social competence, self-help skills, and adaptive behavior from infancy to adulthood. It is used in planning for therapy and/or individualized instruction for persons with mental retardation or emotional disorders. The Vineland scale, which can be used from birth up to the age of 30, consists of a 117-item interview with a parent or other primary caregiver. (There is also a classroom version for ages 3-12 that can be completed by a teacher.) Personal and social skills are evaluated in the following areas: daily living skills (general self-help, eating, dressing); communication (listening, speaking, writing); motor skills (fine and gross, including locomotion); socialization (interpersonal relationships, play and leisure, and coping skills); occupational skills; and self-direction. (An optional Maladaptive Behavior scale is also available.) The test is untimed and takes 20-30 minutes. Raw scores are converted to an age equivalent score (expressed as social age) and a social quotient.
- #52 Not suggesting a specific measure of social maturity for AR experiences (interesting exercise!), Rather social maturity isuseful perspective that allows designers to look forward and help mature this class of experiences.
- #53 If you'll ask me, augmented reality is not dying, but stalling, waiting for a breakthrough either technological or conceptual that will bring new type of applications to the market (much like Foursquare was for location based services). Would it come for one of the incumbents companies, a giant like Google, or maybe a stealthy startup? only time will tell. The breakthrough could / will be social integration http://artimes.rouli.net/2010/04/has-augmented-reality-peaked.html
- #54 Augmented reality could mature socially, properly understand social dynamics, and offer experiences that successfully integrate with the social sphere—providing truly social augmented experiences and accelerating its growth and relevance. Without substantial social integration, augmented reality might remain restricted to a class of specialized utilities that are better suited for focused, asocial or semisocial activities like technical reference—one of the primary applications of augmented reality from the beginning.
- #55 http://www.flickr.com/photos/sssdc/466537549/
- #56 Why? By their nature, people are well designed for social interaction: Our neuro-linguistic, cognitive, and sensory systems do a very good job of supporting synchronous, human-to-human interactions on both small and large scales. We already have amazing capabilities for face, voice, and name recognition; recognizing body-language cues; simultaneous verbal communication; and inferring people’s emotional states. Making human-to-human interaction demonstrably better by designing AR experiences that add new social modalities, using computerized versions of our existing social abilities is possible, but will require great delicacy. Sociality is one of the most fundamental indicators of humanness we use to understand ourselves and the world around us. Augmented reality has not yet taken us to the realm of the transhuman or posthuman. However, embedding nonhuman interactions in our social fabric would necessarily blur and stretch the markers for our common humanity.
- #57 Augmented reality and its bigger brother ubicomp, or everyware, make many new types of interactions and behaviors possible. The vast majority of these possibilities for augmented reality, however, simply would not be relevant to the way people socialize and interact, and some would be strange, unpleasant, or even harmful in certain contexts. When envisioning social augmented experiences, we should design interactions, behaviors, and situations that follow human norms and expectations by default. http://www.flickr.com/photos/jup3nep/2326524827/
- #61 Augmented reality should enhance real-world social interactions and situations rather than directly replacing them. Think of the classic sight gag in Airplane II. One character operates an elaborate, wall-sized video communications console to contact his commander at what seems like a far-away location. But in the middle of their conversation, the commander opens and walks through the video console, revealing it to be a simple door with an ordinary window, which they’ve been talking through as though it were a live video link.
- #63 Socially situated technologies succeed when they enable people to overcome real barriers to interactions and relationships. Augmented interactions can likewise succeed by bridging gaps between people and extending their social reach, as long as the augmented elements themselves are relevant and valuable. Bridge to nowhere in New Zealand But a bridge experience is valid only when social interaction is impossible for some reason. For example, people find each other in crowds using their mobile phones, then end the call as soon as they see one another.
- #64 Consider the simple act of exchanging business cards at the start of a meeting. Scene from American Psycho shows ‘face recognition’ scenario. Complexity of socially situation interactions.
- #65 Many formalized / stylized elements of exchange - greeting, gesture, flourish.
- #66 Augmentations should be optional for the social interactions and settings they aim to enhance. Introducing an indispensable augmentation into a social interaction potentially makes the augmentation the single point of failure for the entire interaction. Apparently simple interactions like exchanging business cards are often finely nuanced social rituals with many layers of meaning. Think of Patrick Bateman and his colleagues scrutinizing the designs of their minutely different cards. AR designers should note that dozens of technologies and products have tried and failed to augment the exchange of business cards over decades. (Do you Poken? I have two of the devices, but given Poken’s low adoption rate, all I can do is Poken with myself.)
- #67 Facebook Privacy settings just for profile information. Managing this in real-time, in the midst of social settings will be difficult.
- #69 When defining the interactions and elements of augmented social experiences, remember that simple designs can successfully integrate with the complexity of people’s decisions and behavior, without directly managing them. On many mobile devices, the convenient buttons that activate the silent-ringing mode and mute conversations—like the mute button Figure 6 shows—provide simple solutions that effectively address the complexities of managing presence, attention, and disruption in social settings and contexts.
- #70 In 1970, roboticist Masahiro Mori identified the “Uncanny Valley,” [http://en.wikipedia.org/wiki/Uncanny_valley] when he noticed that people interacting with robots that look and act like human beings respond with increasing empathy as the robots become more humanlike. However, when robots reach the point at which they seem very like real humans, but are still identifiably non-human, people’s empathic responses to them drop sharply, and people become repulsed by the robots. The name of this phenomenon echoes the shape of the data graph. The exact causes of the Uncanny Valley effect are unknown, but possible explanations include avoiding infection or recognizing genetic abnormalities when choosing a mate. Recent research with monkeys shows the same pattern of responses, so it is common across a broad range of senses for at least two members of the primate family. Very soon, it will be possible to create augmented experiences that incorporate realistic, but still ersatz human faces, voices, and movement that would invoke the Uncanny Valley effect.
- #71 Cameron the ‘Good’ Terminator http://www.flickr.com/photos/gifake/4040776678/
- #73 This is a the Hobo character from Polar Express - a ‘good’ character.
- #74 This is principle #6...
- #76 Securities traders use this expression to make it clear that all parties to a deal must receive something of real value for an exchange to be successful. For traders, this means they must receive something they can use as the basis of another trade rather than something they must hold—as an investment—to gain some uncertain, future benefit. For designers of augmented experiences, this means all of the elements and social interactions must be valuable to all of the people engaging with them. Otherwise, people will perceive the effort and costs of the augmentation as overhead or a burden of some sort. Further, the augmented elements must provide value within the context of a particular interaction rather than only within other contexts or for other purposes.
- #77 To paraphase William Gibson, augmented reality is here, but it is certainly not evenly distributed. Nor do people expect ordinary social experiences and interactions to be mixed realities that include significant augmented elements. Until consumers take mixed reality for granted as the norm, designers must indicate the presence and status of augmented elements in social AR experiences.
- #78 To paraphase William Gibson, augmented reality is here, but it is certainly not evenly distributed. Nor do people expect ordinary social experiences and interactions to be mixed realities that include significant augmented elements. Until consumers take mixed reality for granted as the norm, designers must indicate the presence and status of augmented elements in social AR experiences.
- #79 This simple guideline could trump all other AR design principles. Design AR experiences that follow the established norms for behavior and interaction in a social experience you are augmenting. In true inside-out fashion, this might mean that it is appropriate to take cues from those at the very bottom of the Uncanny Valley, as the characters in the zombie comedy Shaun of the Dead realize when they must find a way to temporarily blend in with a large crowd of shambling undead to reach shelter. Keen-eyed students of behavior and design would note how the group carefully rehearsed their zombie impersonations, coaching one another to properly emulate the inchoate moans, semi-random shuffling, and stilted, slow-motion urgency that are typical of zombies—all to avoid the fatal consequences of bungling their unheimliche, or uncanny, first impression.
- #80 Study local norms Practice / adapt behavior to match “prototype” “Field test” This simple guideline could trump all other AR design principles. Design AR experiences that follow the established norms for behavior and interaction in a social experience you are augmenting. In true inside-out fashion, this might mean that it is appropriate to take cues from those at the very bottom of the Uncanny Valley, as the characters in the zombie comedy Shaun of the Dead realize when they must find a way to temporarily blend in with a large crowd of shambling undead to reach shelter. Keen-eyed students of behavior and design would note how the group carefully rehearsed their zombie impersonations, coaching one another to properly emulate the inchoate moans, semi-random shuffling, and stilted, slow-motion urgency that are typical of zombies—all to avoid the fatal consequences of bungling their unheimliche, or uncanny, first impression.
- #83 Augmented reality could mature socially, properly understand social dynamics, and offer experiences that successfully integrate with the social sphere—providing truly social augmented experiences and accelerating its growth and relevance. Without substantial social integration, augmented reality might remain restricted to a class of specialized utilities that are better suited for focused, asocial or semisocial activities like technical reference—one of the primary applications of augmented reality from the beginning.
- #84 http://twitpic.com/1i1buy http://www.mobypicture.com/user/Dutchcowboy/view/6400035
- #85 http://www.berryplasman.nl/augmented-reality-flashmob/ http://zosociaal.files.wordpress.com/2010/04/201004241427151.png
- #86 indiviaul AR serving as a social object - http://zosociaal.files.wordpress.com/2010/04/2.jpg
- #88 we need AR to be a genuine social utility
- #89 In simple terms its a protocol for storing geolocated data on Wave servers that's currently being developed. We believe this will help lay the foundations for an open, universally accessible, and decentralised system for shared augmented reality overlays which various clients can connect to and use. Wave allows the advantages of both real-time communication, as well as the advantages of persistent hosting of data. It is both like IRC, and like a Wiki. It allows anyone to create a Wave, and share it with anyone else. It allows Waves to be edited at the same time by many people, or used as a private reference for just one person. These are all incredibly useful properties for any AR-experience, more so Wave is open. Anyone can make a server or client for Wave. Better yet, these servers will exchange data with each other, providing a seamless world for the user: a single login will let you browse the whole world of public waves, regardless of who’s providing or hosting the data. Wave is also quite scalable and secure: data is only exchanged when necessary, and will stay local to just one server if no one else needs to view it. Wave allows bots to run on it and thus allowing blips in a waves to be automatically updated, created or destroyed based on any criteria the coders choose. Wave even allows the playback of all edits since the wave was created. For all these reasons and a few more, Wave makes a great platform for AR. How? In basic terms, we will diverse a standard way to geolocate a bit of data and store it as a Blip within a wave. This data could be a 3d mesh, a bit of text, or even a piece of audio. Then various clients on various devices could logon, locate, interpret and display this data as they see fit.