This poster, presented at the Information Architecture Summit in Vancouver, March 2017 with Lisa Nguyen, presents an interaction model for mixed reality, including gesture, gaze, sound, spatial mapping, and emotion. When personal computing software and the Web were young, user interfaces and navigation controls borrowed heavily from physical world metaphors, from the trash can icon to Web "pages". Since then, we have come to rely on native digital UI frameworks and pattern libraries. Now we face new design challenges for mixed reality, which allows us to collaborate over digital holograms in physical space. Early mixed reality applications rely on familiar Web controls, porting the digital affordances we are familiar with - buttons, menus, and two-dimensional displays of data - into our physical environment. These familiar controls allow us to easily transition from 2D interactions to 3D. But will this approach be successful or will it prove too limiting? Recent research in embodied cognition hypothesizes that knowledge is created through how our bodies interact with objects in the world. To fully embrace the knowledge creation and sharing opportunities that mixed reality offers, we designers need to expand our vocabulary of interactions to engage with digital objects and data in the real world. This poster presents a conceptual framework that will continue to evolve as augmented reality and mixed reality experiences become part of our daily lives.

1. WHAT WILL THE FUTURE BRING?
1725-1970
1960 -Present
1963 -Present
2007-Present
2010 -Present
The evolution of human interaction with computing systems – from punched cards through graphical and natural
interfaces – is marked by a gap, layers of abstraction, that gets smaller and smaller as the interface evolves. This
reduced abstraction in the interface allows us to engage more closely and directly with the information we seek. As
we design for Extended Reality (XR), the level of abstraction between real and virtual is reduced further, until we
ourselves become the operating system.
Over the next five years, the border between our real and virtual worlds will blur as we experience digital and
physical objects together in mixed reality environments. Problem-solving, learning, and collaboration are changing
as holographic technologies like Microsoft’s HoloLens allow us to create knowledge using our whole bodies to
manipulate digital information. A new vocabulary of interactions is emerging that includes spatial mapping, sound,
gaze, emotion, and gesture. As information architects, our challenge is to let go of our 2D affordances and think
beyond the patterns we use today, toward universal interactions for a human-centered mixed reality of the future.
GESTURE
· Reaching into a pocket
· Looking at your wrist/watch
· Nodding or shaking your head
· Twisting a doorknob
· Taking a bite of food
SPATIAL MAPPING
· Proximity
· Physical environment
· Pinning digital items so they travel with you
· Breadcrumbs - virtual markers
GAZE
· Selecting objects
· Trigger options on the periphery
SOUND
· Voice commands
· Interactive sonification
INPUT
· Choice of input modes
· Speech to text
· Keyboard
· Handwriting
· Fingerspelling
EMOTION
· Recognition of universal emotions
Reality 2.0 - Evolution of Interaction
Punched cards
Originally for weaving,
paper cards were used for
data processing in the
20th century.
Command line UI
Text commands are
entered one line at a time
in a dialog with the
computer system.
Graphical UI
Pointing devices allow users
to control an application by
clicking images of widgets
and icons on a screen.
Natural UI
Multi-touch gestures and
voice commands enable
direct interaction with
content, with less
dependence on visual
metaphors like buttons.
XR (Extended Reality)
Common intuitive gestures
combined with contextual
attributes allow us to
engage directly with virtual
and real worlds
simultaneously.