Augmented reality blends the real world and the Internet in real time, making many new kinds of proximity, context, and location based experiences possible for individuals and groups. Despite these many possibilities, we know from history that the long term value and impact of augmented reality for most people will depend on how well these experiences integrate with ordinary social settings, and support everyday interactions. Yet the interaction patterns and behavior we see in current AR experiences seem almost ‘anti-social’ by design. This is an important gap that design must close in order to create successful AR offerings. In other words, much like children going to school for the first time, AR must to learn to ‘play well with others’ to be valuable and successful. This presentation reviews the interaction design patterns common to augmented reality, suggests tools to help understand and improve the ’social maturity’ of AR products and applications, and shares design principles for creating genuinely social augmented experiences that integrate well with human social settings and interactions.
29. Experience
“Anyone pointing a
device in my direction
to try to identify me
better be prepared for
either a law suit, or a
punch in the face.”
Anonymous Comment
51. Anti-social behavior often
indicates low ʻsocial
maturityʼ in people
Social maturity = ability to
• understand social dynamics
• follow social norms
• engage socially with others
• adapt to social situations
52. Bad News
Current AR experiences show
generally low social maturity.
Good News
Immaturity is common for
emerging media & spaces.
Technology ʻmaturesʼ rapidly,as
the rise of the Social / 2.0 Web.
AR is ʻyoung and malleableʼ.
53. “augmented reality
is ...stalling, waiting for a
breakthrough either
technological or
conceptual... (much like
Foursquare was for location
based services) ”
Augmented Times
54. Social Maturation Paths
Immature
Social Individual
Integrated w/ social contexts Optimized for individual contexts
Supports social dynamics Supports a/semi-social dynamics
Social interaction patterns Individual interaction patterns
61. Enhancement, Not Replacement
Enhance social
interactions, instead
of replacing them
with AR gimmicks.
Anti-pattern: “AR for ARʼs Sake”
lowers experience value in the
present, though creating comedy for
the future.
63. Build Real Bridges
Social technologies
succeed when they
enable people to
overcome genuine
barriers to interaction.
ʻBridgeʼ experiences are
valid only when interaction
is impossible.
66. Avoid the Critical Path
AR elements should
be ʻoptionalʼ for social
interactions.
Making AR elements
essential creates a ʻsingle
point of failureʼ.
67.
68.
69.
70. Let People Manage Social
Complexity
Simple designs allow
people to manage
social complexity and
interactions as they
need and choose.
82. Be Human By Default
Enhancement, Not Replacement
Build Real Bridges
Avoid the Critical Path
Let People Manage Social Complexity
Avoid the Uncanny Valley
Equal Value
Are We On AiR?
Context Is King
84. Social Maturation Paths
Immature
Social Individual
Integrated w/ social contexts Optimized for individual contexts
Supports social dynamics Supports a/semi-social dynamics
Social interaction patterns Individual interaction patterns
85. AR Flashmob Amsterdam
“...nerds
looking
through a
phone.”
“Its getting busy at the #arflashmob“
91. Additional info
UXMatters: Everyware Column - UXMatters.com
Games Alfresco - http://gamesalfresco.com/
Augmented Times - http://artimes.rouli.net/
Augmented Planet - http://www.augmentedplanet.com/
Beyond the Beyond - http://www.wired.com/beyond_the_beyond/
Designing Social Interfaces & Designing For the Social Web
SxDSalon - http://blog.sxdsalon.org/
Designing for Social Interaction - http://www.boxesandarrows.com/view/designing-for-social
In 1994 Paul Milgram and Fumio Kishino defined a mixed reality as "...anywhere between the extrema of the virtuality continuum."[4] (VC), where the Virtuality Continuum extends from the completely real through to the completely virtual environment with augmented reality and augmented virtuality ranging between.
Wikitude mobile AR browser
key aspects of augmented reality from design perspective
between people = not human to machine (actors with social norms / perspective in effect)
social situated = imbued with meaning based on social context and perspective not simply transactional (e.g recommending a product to a friend vs. making purchase from seller)
between people = not human to machine (actors with social norms / perspective in effect)
social situated = imbued with meaning based on social context and perspective not simply transactional (e.g recommending a product to a friend vs. making purchase from seller)
key aspects of social augmented experiences from design perspective
However, as a result of the inherently social nature of augmented reality, we can be sure the value and impact of many augmented experiences depends in large part on how effectively they integrate the social dimensions of real-world settings, in real time.
the social maturity of current augmented experiences is similar to that of a young child who is learning the complex rules and norms that determine socially acceptable behavior. With unevenly developed abilities and understanding, fitting into social situations is very difficult.
The contextual model for this interaction assumes that interrupted or delayed conversations are socially acceptable—though asynchrony is one of the most frustrating aspects of poorly mediated interactions. (‘Can you hear me now...?’) This scenario also depends on directed, gaze-based, device-powered scanning of faces at close personal distances. For strangers who have no affiliation, I would wager this experience is too similar to having a policeman stop you and ask you for identification for most people to consider it an acceptable social interaction. Even among people with existing, but weaker ties like the indirect relationships that loosely link colleagues in a large organization, this feels like a depersonalizing and inherently suspicious way of greeting someone. Using a device in this way also creates a physical barrier between people and, in terms of attention, shifts focus away from the person to the device and the information it presents. This hardly feels like an improvement over the everyday act of introducing yourself or joining a conversation group at a party.
Anyone pointing a cell phone or any other device in my direction to try to "identify" me better be prepared for a either a law suit or a punch in the face.
Anonymous comment
http://www.technologyreview.com/computing/24639/?a=f
most augmented experiences created using limited set of design patterns for interaction models and concepts
http://www.flickr.com/photos/maggiesworld/2502144304/
The Head-Up Display interaction pattern echoes the targeting and navigation displays in military and other aircraft like that shown in Figure 2. This is the oldest of the AR interaction patterns. Augmented experiences using the Head-Up Display pattern add information about the real objects in view into a complete mixed-reality experience that built-in AR tools and devices generate. While many AR experiences rely on external devices—that is, external to the body—those using the Head-Up Display pattern commonly depend on hardware that is integral to a vehicle or cockpit-like physical setting.
Another very important attribute of the Head-Up Display pattern is integration with a fixed point of view—typically the focus of the user’s visual field. This pattern also applies to wearable systems such as the helmet-mounted displays modern infantries use.
Augmented Driving app from imaGinyze
The Tricorder interaction pattern became familiar to us in the original Star Trek TV series. Mr. Spock, in a landing party on the surface of some new planet, would use his Tricorder scanning device to explore the local area, as shown in Figure 6. Most of the time, interactions involved waving the Tricorder around in the air in the general direction of interest, while peering at the display screen. This is the dominant pattern of physical behavior when using AR browsers such as Wikitude demand.
The essence of the Tricorder interaction pattern is that it adds pieces of information to an existing real-world experience, representing them directly within the combined, augmented-reality, or mixed-reality experience.
A key characteristic of the Tricorder interaction pattern is that it requires an external device—typically a hand-held device, though this is not essential—to access the augmented aspects of the mixed-reality environment. The Tricorder interaction pattern takes up a lot of the user’s available interaction bandwidth. It requires an external device, active focus on the part of the user, exclusive use of at least one hand, abnormal—at least until social conventions change—movements, and greatly expanded personal space. AR experiences that rely on the Tricorder pattern are not good in crowds, small spaces, or in areas with fast-moving objects or people.
Sutter, John D. “New Phone Apps Seek to ’Augment’ Reality.” CNN. August 5, 2009. Retrieved August 1, 2009.
AR adds new and wholly virtual objects directly into the augmented experience, combining them with existing, real objects. The virtual items in Holochess interaction patterns often interact with one another—and sometimes with the real elements of the mixed-reality experience. Commonly marker-based AR.
the X-ray Vision interaction pattern simulates seeing beneath the surface of objects, people, or places, showing their internal structure or contents. AR experiences using the X-ray Vision pattern often use a combination of projection and rendering—frequently, a schematic or abstracted rendering—of the object of interest, as in Medical Augmented Reality (MAR).
Tim O’Reilly Ultimate reference manual
To reach its potential and avoid dismissal as a novelty technology, augmented reality needs new interaction patterns and experience concepts that address the weaknesses and gaps of this limited set of existing patterns. Only in the early stages of its evolution, augmented reality has the opportunity to refine and expand its range of interaction patterns without disrupting familiar models or incurring substantial costs.
http://www.flickr.com/photos/_dvd_/1488286379/
One of the weakest aspects of the existing interaction patterns for augmented reality is their reliance on single-person, socially disconnected user experiences. The interaction patterns for augmented experience must become adept at creating valuable experiences for both individuals in social settings and groups participating in mixed-reality experiences together.
The micro/pico projectors emerging now offer potential for expanding the social envelope of augmented reality, by extending mixed realities directly into the real world. Compact projectors could allow group interactions and experiences without multiple hardware devices. Projecting mixed realities into public, common, or social spaces makes them social by default.
http://www.flickr.com/photos/shimmergirl/3614660077/
Secondhand Smoke
Another important social dimension is the indirect experience of augmented reality—that is, how other people around you experience your augmented or mixed-reality experience. The spread of mobile phones, pagers, and other portable communications devices has disrupted or complicated basic social norms like attention, awareness, presence, and conversation. Everyone within environmental range of the guy talking too loudly about his medication, the pager someone has left unattended on a nearby desk buzzing with incoming messages, or the crackberry addict who never makes eye contact during meetings experiences this kind of disruption.
http://www.flickr.com/photos/threeperf/417644402/
Identity mismanagement
With tools like augmented ID on the way, what happens if your environmentally aware AR device, service, or application recognizes me and broadcasts my identity locally—or globally—when I want to remain incognito? At least until the advent of effective privacy management solutions—including hardware, software, standards, and legal frameworks—AR experiences that identify people by face, marker, or RFID tag could severely challenge our ability to do ordinary things like get lost in a crowd, sit quietly at the back of a room, or attend a surprise party for a friend.
the “scramble suit” from Philip K. Dick’s ‘A Scanner Darkly’, as imagined in the Richard Linklater film
Most AR systems are limited to one visual input such as the camera on a mobile phone. In practical terms, this means they can monitor only a single field of view, in monocular vision. Lacking the depth perception and breadth of human fields of view gives them tunnel vision, limiting their ability to react to stimuli beyond their narrow, monocular view. Additionally, monocular vision makes it difficult to create mixed-reality experiences that mesh smoothly with the human viewpoint.
“predatory gaze” of “terminator vision”
YDreams offers depth perception now.
http://www.flickr.com/photos/jmmpereda/2306035764/
“Terminator vision”
It’s easy to say that AR creates ‘anti-social’ experiences
http://www.flickr.com/photos/31271394@N07/3754867811/
More constructive to think in terms of overall ‘social maturity’ as social science understands it.
The Vineland Social Maturity Scale measures social competence, self-help skills, and adaptive behavior from infancy to adulthood. It is used in planning for therapy and/or individualized instruction for persons with mental retardation or emotional disorders. The Vineland scale, which can be used from birth up to the age of 30, consists of a 117-item interview with a parent or other primary caregiver. (There is also a classroom version for ages 3-12 that can be completed by a teacher.) Personal and social skills are evaluated in the following areas: daily living skills (general self-help, eating, dressing); communication (listening, speaking, writing); motor skills (fine and gross, including locomotion); socialization (interpersonal relationships, play and leisure, and coping skills); occupational skills; and self-direction. (An optional Maladaptive Behavior scale is also available.) The test is untimed and takes 20-30 minutes. Raw scores are converted to an age equivalent score (expressed as social age) and a social quotient.
Not suggesting a specific measure of social maturity for AR experiences (interesting exercise!), Rather social maturity isuseful perspective that allows designers to look forward and help mature this class of experiences.
If you'll ask me, augmented reality is not dying, but stalling, waiting for a breakthrough either technological or conceptual that will bring new type of applications to the market (much like Foursquare was for location based services). Would it come for one of the incumbents companies, a giant like Google, or maybe a stealthy startup? only time will tell.
The breakthrough could / will be social integration
http://artimes.rouli.net/2010/04/has-augmented-reality-peaked.html
Augmented reality could mature socially, properly understand social dynamics, and offer experiences that successfully integrate with the social sphere—providing truly social augmented experiences and accelerating its growth and relevance.
Without substantial social integration, augmented reality might remain restricted to a class of specialized utilities that are better suited for focused, asocial or semisocial activities like technical reference—one of the primary applications of augmented reality from the beginning.
http://www.flickr.com/photos/sssdc/466537549/
Why? By their nature, people are well designed for social interaction: Our neuro-linguistic, cognitive, and sensory systems do a very good job of supporting synchronous, human-to-human interactions on both small and large scales. We already have amazing capabilities for face, voice, and name recognition; recognizing body-language cues; simultaneous verbal communication; and inferring people’s emotional states. Making human-to-human interaction demonstrably better by designing AR experiences that add new social modalities, using computerized versions of our existing social abilities is possible, but will require great delicacy. Sociality is one of the most fundamental indicators of humanness we use to understand ourselves and the world around us. Augmented reality has not yet taken us to the realm of the transhuman or posthuman. However, embedding nonhuman interactions in our social fabric would necessarily blur and stretch the markers for our common humanity.
Augmented reality and its bigger brother ubicomp, or everyware, make many new types of interactions and behaviors possible. The vast majority of these possibilities for augmented reality, however, simply would not be relevant to the way people socialize and interact, and some would be strange, unpleasant, or even harmful in certain contexts. When envisioning social augmented experiences, we should design interactions, behaviors, and situations that follow human norms and expectations by default.
http://www.flickr.com/photos/jup3nep/2326524827/
Augmented reality should enhance real-world social interactions and situations rather than directly replacing them. Think of the classic sight gag in Airplane II. One character operates an elaborate, wall-sized video communications console to contact his commander at what seems like a far-away location. But in the middle of their conversation, the commander opens and walks through the video console, revealing it to be a simple door with an ordinary window, which they’ve been talking through as though it were a live video link.
Socially situated technologies succeed when they enable people to overcome real barriers to interactions and relationships. Augmented interactions can likewise succeed by bridging gaps between people and extending their social reach, as long as the augmented elements themselves are relevant and valuable.
Bridge to nowhere in New Zealand
But a bridge experience is valid only when social interaction is impossible for some reason. For example, people find each other in crowds using their mobile phones, then end the call as soon as they see one another.
Consider the simple act of exchanging business cards at the start of a meeting.
Scene from American Psycho shows ‘face recognition’ scenario. Complexity of socially situation interactions.
Many formalized / stylized elements of exchange - greeting, gesture, flourish.
Augmentations should be optional for the social interactions and settings they aim to enhance. Introducing an indispensable augmentation into a social interaction potentially makes the augmentation the single point of failure for the entire interaction. Apparently simple interactions like exchanging business cards are often finely nuanced social rituals with many layers of meaning. Think of Patrick Bateman and his colleagues scrutinizing the designs of their minutely different cards. AR designers should note that dozens of technologies and products have tried and failed to augment the exchange of business cards over decades. (Do you Poken? I have two of the devices, but given Poken’s low adoption rate, all I can do is Poken with myself.)
Facebook Privacy settings just for profile information. Managing this in real-time, in the midst of social settings will be difficult.
When defining the interactions and elements of augmented social experiences, remember that simple designs can successfully integrate with the complexity of people’s decisions and behavior, without directly managing them. On many mobile devices, the convenient buttons that activate the silent-ringing mode and mute conversations—like the mute button Figure 6 shows—provide simple solutions that effectively address the complexities of managing presence, attention, and disruption in social settings and contexts.
In 1970, roboticist Masahiro Mori identified the “Uncanny Valley,” [http://en.wikipedia.org/wiki/Uncanny_valley] when he noticed that people interacting with robots that look and act like human beings respond with increasing empathy as the robots become more humanlike. However, when robots reach the point at which they seem very like real humans, but are still identifiably non-human, people’s empathic responses to them drop sharply, and people become repulsed by the robots. The name of this phenomenon echoes the shape of the data graph. The exact causes of the Uncanny Valley effect are unknown, but possible explanations include avoiding infection or recognizing genetic abnormalities when choosing a mate. Recent research with monkeys shows the same pattern of responses, so it is common across a broad range of senses for at least two members of the primate family. Very soon, it will be possible to create augmented experiences that incorporate realistic, but still ersatz human faces, voices, and movement that would invoke the Uncanny Valley effect.
Cameron the ‘Good’ Terminator
http://www.flickr.com/photos/gifake/4040776678/
This is a the Hobo character from Polar Express - a ‘good’ character.
This is principle #6...
Securities traders use this expression to make it clear that all parties to a deal must receive something of real value for an exchange to be successful. For traders, this means they must receive something they can use as the basis of another trade rather than something they must hold—as an investment—to gain some uncertain, future benefit.
For designers of augmented experiences, this means all of the elements and social interactions must be valuable to all of the people engaging with them. Otherwise, people will perceive the effort and costs of the augmentation as overhead or a burden of some sort. Further, the augmented elements must provide value within the context of a particular interaction rather than only within other contexts or for other purposes.
To paraphase William Gibson, augmented reality is here, but it is certainly not evenly distributed. Nor do people expect ordinary social experiences and interactions to be mixed realities that include significant augmented elements. Until consumers take mixed reality for granted as the norm, designers must indicate the presence and status of augmented elements in social AR experiences.
To paraphase William Gibson, augmented reality is here, but it is certainly not evenly distributed. Nor do people expect ordinary social experiences and interactions to be mixed realities that include significant augmented elements. Until consumers take mixed reality for granted as the norm, designers must indicate the presence and status of augmented elements in social AR experiences.
This simple guideline could trump all other AR design principles. Design AR experiences that follow the established norms for behavior and interaction in a social experience you are augmenting. In true inside-out fashion, this might mean that it is appropriate to take cues from those at the very bottom of the Uncanny Valley, as the characters in the zombie comedy Shaun of the Dead realize when they must find a way to temporarily blend in with a large crowd of shambling undead to reach shelter. Keen-eyed students of behavior and design would note how the group carefully rehearsed their zombie impersonations, coaching one another to properly emulate the inchoate moans, semi-random shuffling, and stilted, slow-motion urgency that are typical of zombies—all to avoid the fatal consequences of bungling their unheimliche, or uncanny, first impression.
Study local norms
Practice / adapt behavior to match
“prototype”
“Field test”
This simple guideline could trump all other AR design principles. Design AR experiences that follow the established norms for behavior and interaction in a social experience you are augmenting. In true inside-out fashion, this might mean that it is appropriate to take cues from those at the very bottom of the Uncanny Valley, as the characters in the zombie comedy Shaun of the Dead realize when they must find a way to temporarily blend in with a large crowd of shambling undead to reach shelter. Keen-eyed students of behavior and design would note how the group carefully rehearsed their zombie impersonations, coaching one another to properly emulate the inchoate moans, semi-random shuffling, and stilted, slow-motion urgency that are typical of zombies—all to avoid the fatal consequences of bungling their unheimliche, or uncanny, first impression.
Augmented reality could mature socially, properly understand social dynamics, and offer experiences that successfully integrate with the social sphere—providing truly social augmented experiences and accelerating its growth and relevance.
Without substantial social integration, augmented reality might remain restricted to a class of specialized utilities that are better suited for focused, asocial or semisocial activities like technical reference—one of the primary applications of augmented reality from the beginning.
indiviaul AR serving as a social object -
http://zosociaal.files.wordpress.com/2010/04/2.jpg
we need AR to be a genuine social utility
In simple terms its a protocol for storing geolocated data on Wave servers that's currently being developed.
We believe this will help lay the foundations for an open, universally accessible, and decentralised system for shared augmented reality overlays which various clients can connect to and use.
Wave allows the advantages of both real-time communication, as well as the advantages of persistent hosting of data. It is both like IRC, and like a Wiki. It allows anyone to create a Wave, and share it with anyone else. It allows Waves to be edited at the same time by many people, or used as a private reference for just one person.
These are all incredibly useful properties for any AR-experience, more so Wave is open. Anyone can make a server or client for Wave. Better yet, these servers will exchange data with each other, providing a seamless world for the user: a single login will let you browse the whole world of public waves, regardless of who’s providing or hosting the data. Wave is also quite scalable and secure: data is only exchanged when necessary, and will stay local to just one server if no one else needs to view it.
Wave allows bots to run on it and thus allowing blips in a waves to be automatically updated, created or destroyed based on any criteria the coders choose. Wave even allows the playback of all edits since the wave was created.
For all these reasons and a few more, Wave makes a great platform for AR.
How?
In basic terms, we will diverse a standard way to geolocate a bit of data and store it as a Blip within a wave.
This data could be a 3d mesh, a bit of text, or even a piece of audio.
Then various clients on various devices could logon, locate, interpret and display this data as they see fit.