The Glass Class
Designing Wearable Interfaces
May 1, CHI 2014
Mark Billinghurst
HIT Lab NZ
University of Canterbury
mark.b...
1: Introduction
Mark Billinghurst
▪ Director of HIT Lab NZ, University
of Canterbury
▪ PhD Univ. Washington
▪ Research on AR, mobile HCI,
...
Hayes Raffle
▪ Interaction Research Lead,
Google Glass
▪ PhD MIT Media Lab
▪ Ran a couple of companies
▪ Launched a few pr...
Major changes in computing
How do you Design for this?
Course Goals
In this course you will learn
▪ Introduction to head mounted wearable computers
▪ Understanding of current we...
What You Won’t Learn
▪ Low level programming
▪ Glass Mirror API, GDK, Vuzix SDK, etc
▪ Designing for non-HMD based interfa...
Schedule
9:00 Introduction (Mark + Hayes)
9:05 Overview/History (Mark)
9:20 Evolution and Design Principles (Hayes)
9:45 P...
Design Group (33 People)
If your name is on this list you are in the Design Group
Yang Wang
Konstantino Kapetaneas
Preethi...
Display Demos You Can Try
Google Glass Display
Glass UI, AR demos, Games, multimedia capture
Vuzix M-100 Display
Monocular...
CHI Wearables Exhibit
■
Online at http://wcc.gatech.edu/exhibition
2: Overview/History
A Brief History of Time
▪ Trend
▪ smaller, cheaper, more functions, more intimate
▪ Time pieces moved from public space on...
A Brief History of Computing
Trend
▪ Smaller, cheaper, faster, more intimate
▪ Moving from fixed to handheld and onto body...
Room Desk Lap Hand Head
What is a Wearable Computer ?
▪ A computer that is:
▪ Portable while operational
▪ Enables hands-free/hands-limited use
▪ ...
In Other Words ..
▪ A computer that is ..
▪ Eudaemonic: User considers it part of him/herself
▪ Existential: User has comp...
Wearable Computing
▪ Computer on the body that is:
▪ Always on
▪ Always accessible
▪ Always connected
▪ Other attributes
▪...
Augmented Interaction
Rekimoto, J., & Nagao, K. (1995, December). The world through the computer:
Computer augmented inter...
The Ideal Wearable
▪ Persists and Provides Constant Access: Designed for
everyday and continuous user over a lifetime.
▪ S...
Wearable Attributes
▪ fafds
History of Wearables
▪ 1960-90: Early Exploration
▪ Custom build devices
▪ 1990 - 2000: Academic, Military Research
▪ MIT,...
Thorp and Shannon (1961)
▪ Wearable timing device for roulette prediction
▪ Audio feedback, four button input
Ed Thorp
Tho...
Keith Taft (1972)
▪ Wearable computer for blackjack card counting
▪ Toe input, LED in Glasses for feedback
Belt computer S...
Steve Mann (1980s - )
http://wearcomp.org/
MIT Wearable Computing (1993-)
http://www.media.mit.edu/wearables/
Enabling Technologies (1989+)
▪ Private Eye Display (Reflection Technologies)
▪ 720 x 280 dipslay
▪ Red LED
▪ Vibrating mi...
MIT Tin Lizzy (1993)
▪ General Purpose Wearable
▪ Doug Platt, Thad Starner
▪ 150 MHz Pentium CPU
▪ 32-64 Mb RAM
▪ 6 Gb har...
Thad Starner 1998
Early Wearable Computing
Early Technology
▪ Computing
▪ Belt or Backpack
▪ Displays
▪ Head Mounted, LCD Panel, Audio
▪ Input Devices
▪ Chording Key...
US Military Wearables (1989- )
▪ Early experimentation
▪ 386 computer, VGA display
▪ GPS, mapping software
▪ Land Warrior ...
Wearables at CMU (1991–2000)
▪ Industry focused wearables
▪ Maintenance, repair
▪ Custom designed interface
▪ Dial/button ...
Early Commercial Systems
▪ Xybernaut (1996 - 2007)
▪ Belt worn, HMD, 200 MHz
▪ ViA (1996 – 2001)
▪ Belt worn, Audio Interf...
Prototype Applications
▪ Remembrance Agent
▪ Rhodes (97)
▪ Augmented Reality
▪ Feiner (97), Thomas (98)
▪ Remote Collabora...
Mobile AR: Touring Machine (1997)
▪ University of Columbia
▪ Feiner, MacIntyre, Höllerer, Webster
▪ Combines
▪ See through...
MARS View
▪ Virtual tags overlaid on the real world
▪ “Information in place”
Backpack/Wearable Systems
1997 Backpack Wearables
▪ Feiner’s Touring Machine
▪ AR Quake (Thomas)
▪ Tinmith (Piekarski)
▪ M...
PCI 3D Graphics Board
Hard Drive
Serial
Ports
CPU
PC104 Sound Card
PC104 PCMCIA
GPS
Antenna
RTK correction Antenna
HMD
Con...
HIT Lab NZ Wearable AR (2004)
▪ Highly accurate outdoor AR
tracking system
▪ GPS, Inertial, RTK system
▪ HMD
▪ First proto...
2008: Location Aware Phones
Nokia NavigatorMotorola Droid
2009 - Layar (www.layar.com)
• Location based data
– GPS + compass location
– Map + camera view
• AR Layers on real world
...
Wearable Evolution
Backpack+HMD:
…10+ kg
Handheld + HMD
… Separate sensors
.... UMPC 1.1GHz
…1.5kg
…still >$5K
Scale it do...
Google Glass (2011 - )
▪ Hardware
▪ CPU TI OMAP 4430 – 1 Ghz
▪ 16 GB SanDisk Flash,1 GB Ram
▪ 570mAh Battery
▪ Input
▪ 5 mp camera, 720p recordin...
Other Wearables
▪ Vuzix M-100
▪ $999, professional
▪ Recon Jet
▪ $600, more sensors, sports
▪ Opinvent
▪ 500 Euro, multi-v...
Ex: Recon Instruments Snow
Ski display/computer
▪ Location, speed, altitude, phone headset
http://www.reconinstruments.com/
Projected Market
dsfh
Summary
Wearables are a new class of computing
Intimate, persistent, aware, accessible, connected
Evolution over 50 year h...
Evolution + Design Principles
Last year Last week NowForever
The Now machine
Focus on location, contextual
and timely information, and
communication.
Why Glass?
Leadership vision
"Computing
should just be
more comfortable"
"Google should do the hard
work, and you should have a
chance to live, have a ...
As technology becomes more
personal and immediate, it can
start to disappear.
Distant Intimate
Early prototyping
Early prototyping
Early prototyping
Early prototyping
Early prototyping
Early prototyping
Early prototyping
Early prototyping
Early prototyping
Early prototyping
Early prototyping
Early prototyping
Early prototyping
Early prototyping
Concept video
Reality
What we learned
Transparent displays are tricky
Colors are funny and inconsistent.
You can only add light to a scene, not cover anything u...
Reading
Some things don’t work
Immersion
Some things don’t work
Details
Some things don’t work
Design principles
The world is the experience
Get the interface and interactions out of the way.
Micro
Interactions
The position of the display and
limited input ability makes
longer interactions less
comfortable.
Using...
A rear view mirror
Don't overload the user. Stick to the
absolutely essential, avoid long
interactions. Be explicit.
As personal as
it gets
Recognize and adapt to the
user… not the other way
around.
Glass is the most personal device you ow...
For the closest people
and most important
moments
Glass should be for prioritizing your
closest people and creating value
...
Examples
Search Picture Messaging PhoneNavigation Video CallVideo
Platform
How are people using Glass
for creative expression?
A world of stories
In Viewpoint of Billions by David
Datuna, Glass allows viewers to
unlock images and video with
interact...
Social action
First-person journalist Tim Pool
broadcasts an intimate view
of Istanbul protests.
'I want to show you what ...
Sharing and connecting
Conductor Cynthia Johnston Turner
shares a 1st
person experience with her
orchestra.
Personal expression
Alexander Chen’s Viola through Glass.
Our tools are becoming more
intimate and immediate.
We can craft a future of learning, creative
expression and empathy.
Di...
4: Prototyping Tools
How can we quickly
prototype Wearable
experiences with little
or no coding?
Why Prototype?
▪ Quick visual design
▪ Capture key interactions
▪ Focus on user experience
▪ Communicate design ideas
▪ “L...
Prototyping Tools
▪ Static/Low fidelity
▪ Sketching
▪ User interface templates
▪ Storyboards/Application flows
▪ Interacti...
Important Note
▪ Most current wearables run Android OS
▪ eg Glass, Vuzix, Atheer, Epson, etc
▪ So many tools for prototypi...
Typical Development Steps
▪ Sketching
▪ Storyboards
▪ UI Mockups
▪ Interaction Flows
▪ Video Prototypes
▪ Interactive Prot...
Sketched Interfaces
▪ Sketch + Powerpoint/Photoshop/Illustrator
GlassSim – http://glasssim.com/
▪ Simulate the view through Google Glass
▪ Multiple card templates
GlassSim Card Builder
▪ Use HTML for card details
▪ Multiple templates
▪ Change background
▪ Own image
▪ Camera view
GlassSim Samples
Glass UI Templates
▪ Google Glass Photoshop Templates
▪ http://glass-ui.com/
▪ http://dsky9.com/glassfaq/the-google-glass-...
Sample Slides From Templates
Application Storyboard
▪ http://dsky9.com/glassfaq/google-glass-
storyboard-template-download/
ToolKit for Designers
▪ Vectoform Google Glass Toolkit for Designers
▪ http://blog.vectorform.com/2013/09/16/google-glass-...
Application Flow
Limitations
▪ Positives
▪ Good for documenting screens
▪ Can show application flow
▪ Negatives
▪ No interactivity/transiti...
Transitions
▪Series of still photos in a movie format.
▪Demonstrates the experience of the product
▪Discover where concept needs flesh...
See https://vine.co/v/bgIaLHIpFTB
Example: Video Sketch of Vine UI
UI Concept Movies
Interactive Wireframes
Interactive Wireframing
▪ Developing interactive interfaces/wireframes
▪ Transitions, user feedback, interface design
▪ We...
UXpin - www.uxpin.com
▪ Web based wireframing tool
▪ Mobile/Desktop applications
▪ Glass templates, run in browser
https:/...
Proto.io - http://www.proto.io/
▪ Web based mobile prototyping tool
▪ Features
▪ Prototype for multiple devices
▪ Gesture ...
Proto.io - Interface
Demo: Building a Simple Flow
Gesture Flow
Scr1
Scr2 Scr3
Scr4 Scr5 Scr6
Ta
p
Swipe
Start Transitions
Demo
Justinmind - http://www.justinmind.com/
▪ Native wireframing tool
▪ Build mobile apps without programming
▪ drag and drop,...
User Interface - Glass Templates
Web Simulation Tool
Comparing Wireframe Tools
Tool Web Native Wearable
Template
Interaction
Uxpin X X
Proto.io X X
Justinmind X X X
Axure X X X
Wireframe Limitations
▪ Can’t deploy on Glass
▪ No access to sensor data
▪ Camera, orientation sensor
▪ No multimedia play...
Processing for Wearables
Processing
▪ Programming tool for Artists/Designers
▪ http://processing.org
▪ Easy to code, Free, Open source, Java based
...
Processing - Motivation
▪ Language of Interaction
▪ Sketching with code
▪ Support for rich interaction
▪ Large developer c...
http://processing.org/
http://openprocessing.org/
Development Enviroment
Basic Parts of a Processing Sketch
/* Notes comment */
//set up global variables
float moveX = 50;
//Initialize the Sketch...
Importing Libraries
▪ Can add functionality by Importing Libraries
▪ java archives - .jar files
▪ Include import code
impo...
http://toxiclibs.org/
Processing and Glass
▪ One of the easiest ways to build rich
interactive wearable applications
▪ focus on interactivity, n...
Example: Hello World
//called initially at the start of the Processing sketch
void setup() {
size(640, 360);
background(0)...
Demo
Hello World Image
PImage img; // Create an image variable
void setup() {
size(640, 360);
//load the ok glass home screen i...
Demo
Touch Pad Input
▪ Tap recognized as DPAD input
void keyPressed() {
if (key == CODED){
if (keyCode == DPAD) {
// Do somethi...
Motion Event
//Glass Touch Events - reads from touch pad
public boolean dispatchGenericMotionEvent(MotionEvent event) {
fl...
Demo
Sensors
▪ Ketai Library for Processing
▪ https://code.google.com/p/ketai/
▪ Support all phone sensors
▪ GPS, Compass, Ligh...
Using Sensors
▪ Setup in Setup( ) function
▪ sensor = new KetaiSensor(this);
▪ sensor.start();
▪ sensor.list();
▪ Event ba...
Sensor Demo
Using the Camera
▪ Import camera library
▪ import ketai.camera.*;
▪ KetaiCamera cam;
▪ Setup in Setup( ) function
▪ cam = ...
Camera Demo
Timeline Demo
▪ Create Card Class
▪ load image, card number, children/parent cards
▪ Timeline Demo
▪ Load cards in order
▪...
Native Coding
Overview
▪ For best performance need native coding
▪ Low level algorithms etc
▪ Most current wearables based on Android OS...
Mirror API + Glass GDK
Glassware and Timeline
Glassware and Timeline
▪ Static Cards
▪ Static content with text, HTML, images, and video.
▪ e.g. notification messages, n...
Glassware Development
▪ Mirror API
▪ Server programming, online/web application
▪ Static cards / timeline management
▪ GDK...
▪ REST API
▪ Java servlet, PHP, Go,
Python, Ruby, .NET
▪ Timeline based apps
▪ Static cards
- Text, HTML, media attachment...
GDK
▪ Glass Development Kit
▪ Android 4.0.3 ICS + Glass specific APIs
▪ Use standard Android Development Tools
▪ GDK add-on features
▪ Timeline and cards
▪ Menu and UI
▪ Touch pad and gesture
▪ Media (sound, camera and voice input)
G...
Glass Summary
▪ Use Mirror API if you need ...
▪ Use GDK if you need ...
▪ Or use both
Hardware Prototyping
Fake Display
3D print Thingiverse model
see http://www.thingiverse.com/thing:65706
Have the social impact of Google Glass ...
Build Your Own Wearable
▪ MyVu display + phone + sensors
Beady-i
▪ http://www.instructables.com/id/DIY-
Google-Glasses-AKA-the-Beady-i/
Rasberry Pi Glasses
▪ Modify video glasses, connect to Rasberry Pi
▪ $200 - $300 in parts, simple assembly
▪ https://learn...
Physical Input Devices
▪ Can we develop unobtrusive input devices ?
▪ Reduce need for speech, touch pad input
▪ Socially m...
Prototyping Platform
Arduino Kit Bluetooth Shield Google Glass
Example: Glove Input
▪ Buttons on fingertips
▪ Map touches to commands
Example: Ring Input
▪ Touch strip, button, accelerometer
▪ Tap, swipe, flick actions
How it works
Bracelet
Armband
Gloves
1,2,
3,4
Values/
output
Summary
▪ Prototyping for wearables is similar to mobiles
▪ Tools for UI design, storyboarding, wireframing
▪ Android tool...
Other Tools
▪ Wireframing
▪ pidoco
▪ FluidUI
▪ Rapid Development
▪ Phone Gap
▪ AppMachine
▪ Interactive
▪ App Inventor
▪ W...
App Inventor - http://appinventor.mit.edu/
▪ Visual Programming for Android Apps
▪ Features
▪ Access to Android Sensors
▪ ...
App Inventor Designer View
App Inventor Blocks View
Orientation Demo
▪ Use wearable orientation sensor
WearScript
▪ JavaScript development for Glass
▪ http://www.wearscript.com/en/
▪ Script directory
▪ http://weariverse.com/
Best Practices (Dos + don’ts)
✓ Don’t design an app
Glass OS is time-based model, not an app model.
X
✓ Know what makes Glass different than a phone
Glass has certain superpowers. Remember what these
superpowers are and use ...
X
Don’t just port your mobile
experience over to Glass
It won’t work. It will be too busy. It will be hard for users to
qu...
✓ Design for the (hyper)now
When is
my next
meeting?
How many
calories have I
eaten today? Can I
get a burger for
lunch?
S...
✓ Do one thing at a time
✓ Design for emotion
thumbs up viewsnap - running
✓ Make it glanceable
Seek to rigorously reduce information density. Successful designs
afford for recognition, not reading...
✓ Reduce the number of info chunksX✓
You are designing for recognition, not reading. Reducing the total # of
information c...
✓ Design single interactions to be faster than 4 s
Eye movements
For 1: 1 230ms
For 2: 1 230ms
For 3: 1 230ms
For 4: 3 (52...
Test the glanceability of your design✓
✓ Test your design indoors + outdoors
✓ White is your new black
✓ On the device, black is not blackX
Your mock Device simulation
✓ Establish hierarchy with color - not font size
White is your <h1> and grey is your <h2> or <h3>. Footer text -
establish...
✓ If you have brand-specific typography - use it
✓ Use relative information display
calendar card - in 10 minutes, Fri - viewsnap
✓
Remember, people have an ever-growing
ecosystem of wearablesX
Each device should be used when it’s most relevant and whe...
✓ The Glass screen is just one part of the
experience
✓ Do view your design on the device
1. Download Android Design Preview
2. Plug in Glass
3. Terminal command: java -jar And...
5: Concept Design Exercise
Design Group (33 People)
▪ If your name is on this list you are in the Design Group
Yang Wang
Konstantino Kapetaneas
Preet...
6: Wearable Technologies
Wearable System
Some Key Aspects
▪ Display Technologies
▪ Input Devices
▪ Interaction Metaphors
▪ Perceptual Factors
▪ Ergonomics
▪ Cognit...
Display Technologies
Key Properties of HMD
▪ Field of View
▪ Human eye 95 deg. H, 60/70 deg. V
▪ Resolution
▪ > 320x240 pixel
▪ Refresh Rate
▪ ...
Types of Head Mounted Displays
Occluded
See-thru
Multiplexed
Optical see-through HMD
Virtual images
from monitors
Real
World
Optical
Combiners
Optical See-Through HMD
Epson Moverio BT-200
▪ Stereo see-through display ($700)
▪ 960 x 540 pixels, 23 degree FOV, 60Hz, 88g
▪ Android Powered, s...
Strengths of optical see-through
▪ Simpler (cheaper)
▪ Direct view of real world
▪ Full resolution, no time delay (for rea...
Video see-through HMD
Video
cameras
Monitors
Graphics
Combiner
Video
Video See-Through HMD
Vuzix Wrap 1200DXAR
▪ Stereo video see-through display ($1500)
▪ Twin 852 x 480 LCD displays, 35 deg. FOV
▪ Stereo VGA cam...
Strengths of Video See-Through
▪ True occlusion
▪ Block image of real world
▪ Digitized image of real world
▪ Flexibility ...
Multiplexed Displays
▪ Above or below line of sight
▪ Strengths
▪ User has unobstructed view of real world
▪ Simple optics...
Vuzix M-100
▪ Monocular multiplexed display ($1000)
▪ 852 x 480 LCD display, 15 deg. FOV
▪ 5 MP camera, HD video
▪ GPS, gy...
Display Types
▪ Curved Mirror
▪ off-axis projection
▪ curved mirrors in front of eye
▪ high distortion, small eye-box
▪ Wa...
See-through thin displays
▪ Waveguide techniques for thin see-through displays
▪ Wider FOV, enable AR applications
▪ Socia...
Waveguide Methods
See: http://optinvent.com/HUD-HMD-benchmark#benchmarkTable
Holographic
Hologram diffracts light
Limited ...
Waveguide Methods
See: http://optinvent.com/HUD-HMD-benchmark#benchmarkTable
Clear-Vu Reflective
Several reflective elemen...
Comparison Chart
Input Technologies
Input Options
▪ Physical Devices
▪ Keyboard
▪ Pointer
▪ Stylus
▪ Natural Input
▪ Speech
▪ Gesture
▪ Other
▪ Physiological
Twiddler Input
▪ Chording or multi-tap input
▪ Possible to achieve 40 - 60 wpm after 30+ hours
▪ Chording input about 50% ...
Virtual Keyboards
▪ In air text input
▪ Virtual QWERTY keyboard up to 20 wpm
▪ On real keyboard around 45-60+ wpm
▪ Word G...
Unobtrusive Input Devices
▪ GestureWrist
▪ Capacitive sensing
▪ Change signal depending on hand shape
Rekimoto, J. (2001)....
Unobtrusive Input Devices
▪ GesturePad
▪ Capacitive multilayered touchpads
▪ Supports interactive clothing
Skinput
Using EMG to detect muscle activity
Tan, D., Morris, D., & Saponas, T. S. (2010). Interfaces on the go. XRDS:
Cros...
Issues to Consider
▪ Fatigue
▪ “Gorrilla” Arm from free-hand input
▪ Comfort
▪ People want to do small gestures by waist
▪...
Interaction on the Go
▪ Fitt’s law still applies while interacting on the go
▪ Eg: Tapping while walking reduces speed by ...
Interface Metaphors
Information Display
Head Stabilized
Fixed View
Body Stabilized
3 DOF Tracking
World Stabilized
6 DOF Tracking
Spatial Cues for Wearable Info
Billinghurst, M., Bowskill, J., Dyer, N., & Morphett, J. (1998, March). An evaluation of we...
Spatial Conferencing
3+ attendees can be distinguished with spatialized audio
but could not without spatialized audio
Bill...
Organizing Tools in a Halo Display
Biocca, F., Tang, A., Lamas, D., Gregg, J., Brady, R., & Gai, P. (2001). How do users o...
User Attention Metaphors
Cognitive continuums (a) Input, (b) Output
Increase cognitive load from left to right
Notification Interruptions
▪ Gradually increase engagement
▪ Reduce attention load
Receiving SMS on Glass
“Bing”
Tap
Swipe...
Nomadic Radio (2000)
▪ Spatial audio wearable interface
Sawhney, N., & Schmandt, C. (2000). Nomadic radio: speech and audi...
Spatial Audio Metaphor
Messages/Events arranged depending on time of day
Notification Interruptions
▪ Dynamic scaling of incoming message
based on interruptibility of the user
▪ Busy = silence
▪ ...
Layered Audio Notifications
Background ambient audio
Notification scale depending on priority
Perception Issues
AR Perceptual Issues
❖ Environment: Issues related to the environment itself.
❖ Capturing: Issues related to digitizing th...
Cognitive Issues in Mobile AR
▪ Three categories of issues
▪ Information Presentation – displaying virtual
information on ...
Depth Cues
❖ Pictorial: visual cues
• Occlusion, texture, relative brightness
❖ Kinetic: motion cues
• Relative motion par...
Use the Following Depth Cues
▪ Movement parallax.
▪ Icon/Object size (for close objects)
▪ Linear perspective
▪ To add sid...
Information Presentation
• Amount of information
• Clutter, complexity
• Representation of information
• Navigation cues, ...
Twitter 360
▪ www.twitter-360.com
▪ iPhone application
▪ See geo-located tweets in real world
▪ Twitter.com supports geo t...
Wikitude – www.mobilizy.com
Bl
ah
Bl
ah
Bl
ah
Bl
ah Bl
ah
Bl
ah
Bl
ah
Bl
ah
Bl
ah
Bl
ah
Bl
ah
Bl
ah
Bl
ah
Bl
ah
Bl
ah
Bl
a...
Information Filtering
Information Filtering
•Use context to support information filtering
•Show less information on AR mode (uncluttered screen)
Text Representation
▪ Object space
▪ In general, does not fit wearable applications
▪ Billboards
▪ Screen space
▪ Annotati...
Outdoor background textures
Changing outdoor illuminanation, and text drawing styles.
Billboard and green text drawing sty...
Screen Space Annotation
AR Navigation
▪ Problem – how to AR to assist with navigation to POI
Multi-View Navigation
▪ Use multiple views (ego vs exo-centric)
▪ Map/top down view – long distance navigation
▪ AR View –...
Wearable Display: Limited FOV
Transitioning Between Views
▪ Seamless spatial awareness
Cognitive Models
Resource Competition Framework
▪ Mobility tasks compete for cognitive resources
with other tasks
▪ the most important give...
RCF Key Assumptions
Four Key Assumptions
1. Functional Modularity: cognitive system divided into
functionally separate sys...
Cognitive Interference
▪ Structural interference
▪ Two or more tasks compete for limited
resources of a peripheral system
...
Cognitive Resources & Limitations
asdfasdf
Using RCF
1. Map cognitive faculty to task
2. Look for conflicts/overloads
3. Analyse for competition for attention
4. Loo...
Example: Going to work ..
Which is the most cognitively demanding?
Application of RCF
Busy street > Escalator > Café > Laboratory.
But if you made Wayfinding, Path Planning, Estimating Time...
Ergonomics
Where to put Wearables?
▪ Places for unobtrusive wearable technology
Gemperle, F., Kasabach, C., Stivoric, J., Bauer, M., ...
Where to Place Trackpad?
▪ User study 25 people different postures
▪ Front of thigh most preferred, torso/upper arm worst
...
Where do
users want
Wearables?
29% on clothing
28% on wrist
12% on Glasses
Tool/Task Design Paradigm
Lin, R., & Kreifeldt, J. G. (2001). Ergonomics in wearable computer design.
International Journa...
Social Perception
How is the User Perceived?
TAT Augmented ID
7: Design Exercise
Building on Glass
Design Exercise
Design for Glass
Guidelines
Design for Glass
Don’t get in the way
Guidelines
Design for Glass
Don’t get in the way
Keep it timely
Guidelines
Design for Glass
Don’t get in the way
Keep it timely
Avoid the unexpected
Guidelines
Design for Glass
Don’t get in the way
Keep it timely
Avoid the unexpected
Design for people
Guidelines
Platform
Platform
Platform - Mirror API
Platform
If you could build anything on Glass,
what would it be?
Sprint - Ideas
Person (mom, family, astronomer)
60 seconds
Sprint - Ideas
Place (in the kitchen, car, hike)
60 seconds
Sprint - Ideas
Function (dance, stargaze, sequence dna)
60 seconds
Sprint - Final deliverable
Make a poster - (Hayes will demo)
Title
One liner
Picture
Pitch
What is it?
What does it do?
Wh...
Sprint - Focus
10 min - Ideas to create a product to solve
need
3 min - Share with your group
Sprint - Focus
Sprint - Focus
3 min - Pick a concept
Sprint - Focus
15 min - Dive deeper, develop, poster
Sprint - Focus
15 min - Poster
Sprint - Lightning
2 minute pitch
7: Design Presentations
Design Concepts
8: Research Directions
Challenges for the Future (2001)
▪ Privacy
▪ Power use
▪ Networking
▪ Collaboration
▪ Heat dissipation
▪ Interface design
...
Interface Design
Gesture Interaction
Capturing Behaviours
▪ 3 Gear Systems
▪ Kinect/Primesense Sensor
▪ Two hand tracking
▪ http://www.threegear.com
Gesture Interaction With Glass
▪ 3 Gear Systems
▪ Hand tracking
▪ Hand data sent to glass
▪ Wifi networking
▪ Hand joint p...
Performance
▪ Full 3d hand model input
▪ 10 - 15 fps tracking, 1 cm fingertip resolution
User Study
▪ Gesture vs. Touch pad vs. Combined input
▪ Gesture 3x faster, no difference in accuracy
●Meta Gesture Interaction
▪Depth sensor + Stereo see-through
▪https://www.spaceglasses.com/
Collaboration
Social Panoramas
Ego-Vision Collaboration
▪ Wearable computer
▪ camera + processing + display + connectivity
Current Collaboration
▪ First person remote conferencing/hangouts
▪ Limitations
▪ Single POV, no spatial cues, no annotati...
Sharing Space: Social Panoramas
▪ Capture and share social spaces in real time
▪ Enable remote people to feel like they’re...
Key Technology
▪ Google Glass
▪Capture live panorama (compass + camera)
▪Capture spatial audio, live video
▪ Remote device...
Awareness Cues
▪ Where is my partner looking?
▪Enhanced radar display, Context compass
Interaction
▪ Glass Touchpad Input/Tablet Input
▪Shared pointers, Shared drawing
User Evaluation
▪ Key Results
Visual cues significantly increase awareness
Pointing cues preferred for collaboration
Drawi...
Intellectual Tools
Context Sensing
▪ Using context to manage information
▪ progressive information display as user shows
interest
▪ Context f...
Gaze Interaction
AR View
More Information Over Time
OpenSource Eyetracker
▪ Could use open source eyetracking
▪ Open Shades Eye Tracker
▪ http://www.wearscript.com/en/latest/...
9: Resources
Glass Resources
▪ Main Developer Website
▪ https://developers.google.com/glass/
▪ Glass Apps Developer Site
▪ http://glass...
Other Resources
▪ AR for Glass Website
▪ http://www.arforglass.org/
▪ Vandrico Database of wearable devices
▪ http://vandr...
Books
▪ Programming Google Glass
▪ Eric Redmond
▪ Rapid Android Development:
Build Rich, Sensor-Based
Applications with Pr...
Contact Details
Mark Billinghurst
▪ email: mark.billinghurst@hitlabnz.org
▪ twitter: @marknb00
Hayes Raffle
▪ email: hraff...
The Glass Class: Designing Wearable Interfaces
The Glass Class: Designing Wearable Interfaces
The Glass Class: Designing Wearable Interfaces
The Glass Class: Designing Wearable Interfaces
The Glass Class: Designing Wearable Interfaces
The Glass Class: Designing Wearable Interfaces
The Glass Class: Designing Wearable Interfaces
The Glass Class: Designing Wearable Interfaces
The Glass Class: Designing Wearable Interfaces
The Glass Class: Designing Wearable Interfaces
The Glass Class: Designing Wearable Interfaces
The Glass Class: Designing Wearable Interfaces
The Glass Class: Designing Wearable Interfaces
The Glass Class: Designing Wearable Interfaces
The Glass Class: Designing Wearable Interfaces
The Glass Class: Designing Wearable Interfaces
The Glass Class: Designing Wearable Interfaces
The Glass Class: Designing Wearable Interfaces
The Glass Class: Designing Wearable Interfaces
The Glass Class: Designing Wearable Interfaces
The Glass Class: Designing Wearable Interfaces
The Glass Class: Designing Wearable Interfaces
The Glass Class: Designing Wearable Interfaces
The Glass Class: Designing Wearable Interfaces
The Glass Class: Designing Wearable Interfaces
The Glass Class: Designing Wearable Interfaces
The Glass Class: Designing Wearable Interfaces
Upcoming SlideShare
Loading in...5
×

The Glass Class: Designing Wearable Interfaces

6,469

Published on

This was a course taught at the CHI 2014 conference on May 1st by Mark Billinghurst (HIT Lab NZ) and Hayes Raffle (Google). It teaches the fundamentals of designing wearable interfaces.

Published in: Technology
2 Comments
24 Likes
Statistics
Notes
  • Thanks Philip - we're happy to help..
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Hi Mark and Hayes

    I really appreciate you making your extensive - indeed authoritative - research available here on slideshare (rather than on some worthy, but dull and unread, academic journal).

    It has really helped me understand how wearables are a completely different beast to earlier computing platforms, and has also enabled me to jump immediately to the bleeding edge of this exciting new technology!

    Many, many thanks.

    Philip Hart
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
6,469
On Slideshare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
299
Comments
2
Likes
24
Embeds 0
No embeds

No notes for slide

Transcript of "The Glass Class: Designing Wearable Interfaces"

  1. 1. The Glass Class Designing Wearable Interfaces May 1, CHI 2014 Mark Billinghurst HIT Lab NZ University of Canterbury mark.billinghurst@canterbury.ac.nz Hayes Raffle Glass team Google [x] hraffle@google.com
  2. 2. 1: Introduction
  3. 3. Mark Billinghurst ▪ Director of HIT Lab NZ, University of Canterbury ▪ PhD Univ. Washington ▪ Research on AR, mobile HCI, Collaborative Interfaces ▪ More than 250 papers in AR, VR, interface design ▪ Sabbatical in Glass team at Google [x] in 2013
  4. 4. Hayes Raffle ▪ Interaction Research Lead, Google Glass ▪ PhD MIT Media Lab ▪ Ran a couple of companies ▪ Launched a few products ▪ Won a few awards ▪ Published many papers in HCI
  5. 5. Major changes in computing
  6. 6. How do you Design for this?
  7. 7. Course Goals In this course you will learn ▪ Introduction to head mounted wearable computers ▪ Understanding of current wearable technology ▪ Key design principles/interface metaphors ▪ Relevant human perceptual principles ▪ Rapid prototyping tools ▪ Areas for future research ▪ Hands on experience with the technology
  8. 8. What You Won’t Learn ▪ Low level programming ▪ Glass Mirror API, GDK, Vuzix SDK, etc ▪ Designing for non-HMD based interfaces ▪ Watches, fitness bands, etc ▪ How to develop wearable hardware ▪ optics, sensor assembly, etc ▪ Evaluation methods ▪ Experimental design, statistics, etc
  9. 9. Schedule 9:00 Introduction (Mark + Hayes) 9:05 Overview/History (Mark) 9:20 Evolution and Design Principles (Hayes) 9:45 Prototyping Tools + Best Practices (Mark / Hayes) 10:20 Break/Demo 10:30 Concept Design Exercise (Design group / Hayes) 10:50 Wearable Technologies (Lecture group / Mark) 11:20 Design Presentations (Design group / Hayes) 11:50 Research Directions (Mark + Hayes) 12:20 Finish
  10. 10. Design Group (33 People) If your name is on this list you are in the Design Group Yang Wang Konstantino Kapetaneas Preethi Srinivas Tony James Kate Vogt Aneesh Tarun Josh Andres Maria Maimó Bram Reurings Luke Mill Tuck-Voon How M Gill Janaki Kumar Melinda Knight M Calkins Mike Tissenbaum Samantha Tse Kal McDowd Adora Tam Oscar Meruvia Mike Chen Anita Hoechtl Merlin Stone Ashoomi Dohlakia Icy Zhu Zdenek Mikovec Cristina Manresa-Yee Christian Winkler Angela Noh Amyris Fernandez Deborah Ptak Arne Renkema-Padmos Thomas Fritz
  11. 11. Display Demos You Can Try Google Glass Display Glass UI, AR demos, Games, multimedia capture Vuzix M-100 Display Monocular display Epson BT-100, Epson BT-200 See through displays, Junaio markerless tracking Brother AirScouter display Projected see-through image Recon Snow Micro-display integrated into ski goggles
  12. 12. CHI Wearables Exhibit ■ Online at http://wcc.gatech.edu/exhibition
  13. 13. 2: Overview/History
  14. 14. A Brief History of Time ▪ Trend ▪ smaller, cheaper, more functions, more intimate ▪ Time pieces moved from public space onto the body 18th Century 20th Century 13th Century
  15. 15. A Brief History of Computing Trend ▪ Smaller, cheaper, faster, more intimate ▪ Moving from fixed to handheld and onto body 1950’s 1980’s 1990’s
  16. 16. Room Desk Lap Hand Head
  17. 17. What is a Wearable Computer ? ▪ A computer that is: ▪ Portable while operational ▪ Enables hands-free/hands-limited use ▪ Able to get the user’s attention ▪ Is always on, acting on behalf of the user ▪ Able to sense the user’s current context Rhodes, B. J. (1997). The wearable remembrance agent: A system for augmented memory. Personal Technologies, 1(4), 218-224.
  18. 18. In Other Words .. ▪ A computer that is .. ▪ Eudaemonic: User considers it part of him/herself ▪ Existential: User has complete control of the system ▪ Ephemeral: System always operating at some level Mann, S. (1997). Wearable computing: A first step toward personal imaging. Computer, 30(2), 25-32.
  19. 19. Wearable Computing ▪ Computer on the body that is: ▪ Always on ▪ Always accessible ▪ Always connected ▪ Other attributes ▪ Augmenting user actions ▪ Aware of user and surroundings
  20. 20. Augmented Interaction Rekimoto, J., & Nagao, K. (1995, December). The world through the computer: Computer augmented interaction with real world environments. In Proceedings of the 8th annual ACM symposium on User interface and software technology (pp. 29-36).
  21. 21. The Ideal Wearable ▪ Persists and Provides Constant Access: Designed for everyday and continuous user over a lifetime. ▪ Senses and Models Context: Observes and models the users environment, mental state, it’s own state. ▪ Augments and Mediates: Information support for the user in both the physical and virtual realities. ▪ Interacts Seamlessly: Adapts its input and output modalities to those most appropriate at the time. Starner, T. E. (1999). Wearable computing and contextual awareness (Doctoral dissertation, Massachusetts Institute of Technology).
  22. 22. Wearable Attributes ▪ fafds
  23. 23. History of Wearables ▪ 1960-90: Early Exploration ▪ Custom build devices ▪ 1990 - 2000: Academic, Military Research ▪ MIT, CMU, Georgia Tech, EPFL, etc ▪ 1997: ISWC conference starts ▪ 1995 – 2005+: First Commercial Uses ▪ Niche industry applications, Military ▪ 2010 - : Second Wave of Wearables ▪ Consumer applications, Head Worn
  24. 24. Thorp and Shannon (1961) ▪ Wearable timing device for roulette prediction ▪ Audio feedback, four button input Ed Thorp Thorp, E. O. (1998, October). The invention of the first wearable computer. In Wearable Computers, 1998. Second International Symposium on (pp. 4-8). IEEE.
  25. 25. Keith Taft (1972) ▪ Wearable computer for blackjack card counting ▪ Toe input, LED in Glasses for feedback Belt computer Shoe Input Glasses Display
  26. 26. Steve Mann (1980s - ) http://wearcomp.org/
  27. 27. MIT Wearable Computing (1993-) http://www.media.mit.edu/wearables/
  28. 28. Enabling Technologies (1989+) ▪ Private Eye Display (Reflection Technologies) ▪ 720 x 280 dipslay ▪ Red LED ▪ Vibrating mirror ▪ Twiddler (Handykey) ▪ Chording keypad ▪ Mouse emulation
  29. 29. MIT Tin Lizzy (1993) ▪ General Purpose Wearable ▪ Doug Platt, Thad Starner ▪ 150 MHz Pentium CPU ▪ 32-64 Mb RAM ▪ 6 Gb hard disk ▪ VGA display ▪ 2 PCMCIA slots ▪ Cellular modem http://www.media.mit.edu/wearables/lizzy/lizzy/index.html
  30. 30. Thad Starner 1998
  31. 31. Early Wearable Computing
  32. 32. Early Technology ▪ Computing ▪ Belt or Backpack ▪ Displays ▪ Head Mounted, LCD Panel, Audio ▪ Input Devices ▪ Chording Keyboard, Speech, Camera ▪ Networking ▪ Wireless LAN, Infra-Red, Cellular
  33. 33. US Military Wearables (1989- ) ▪ Early experimentation ▪ 386 computer, VGA display ▪ GPS, mapping software ▪ Land Warrior (1991-) ▪ Integrated wearable system ▪ Camera, colour display, radio ▪ Navigation, reports, photos Zieniewicz, M. J., Johnson, D. C., Wong, C., & Flatt, J. D. (2002). The evolution of army wearable computers. IEEE Pervasive Computing, 1(4), 30-40.
  34. 34. Wearables at CMU (1991–2000) ▪ Industry focused wearables ▪ Maintenance, repair ▪ Custom designed interface ▪ Dial/button input ▪ Rapid prototyping approach ▪ Industrial designed, ergonomic http://www.cs.cmu.edu/afs/cs/project/vuman/www/frontpage.html
  35. 35. Early Commercial Systems ▪ Xybernaut (1996 - 2007) ▪ Belt worn, HMD, 200 MHz ▪ ViA (1996 – 2001) ▪ Belt worn, Audio Interface ▪ 700 MHz Crusoe ■ Symbol (1998 – 2006) ■ Wrist worn computer ■ Finger scanner
  36. 36. Prototype Applications ▪ Remembrance Agent ▪ Rhodes (97) ▪ Augmented Reality ▪ Feiner (97), Thomas (98) ▪ Remote Collaboration ▪ Garner (97), Kraut (96) ■ Maintenance ■ Feiner (93), Caudell (92) ▪ Factory Work ▪ Thompson (97)
  37. 37. Mobile AR: Touring Machine (1997) ▪ University of Columbia ▪ Feiner, MacIntyre, Höllerer, Webster ▪ Combines ▪ See through head mounted display ▪ GPS tracking ▪ Orientation sensor ▪ Backpack PC (custom) ▪ Tablet input Feiner, S., MacIntyre, B., Höllerer, T., & Webster, A. (1997). A touring machine: Prototyping 3D mobile augmented reality systems for exploring the urban environment. Personal Technologies, 1(4), 208-217.
  38. 38. MARS View ▪ Virtual tags overlaid on the real world ▪ “Information in place”
  39. 39. Backpack/Wearable Systems 1997 Backpack Wearables ▪ Feiner’s Touring Machine ▪ AR Quake (Thomas) ▪ Tinmith (Piekarski) ▪ MCAR (Reitmayr) ▪ Bulky, HMD based Piekarski, W., & Thomas, B. (2002). ARQuake: the outdoor augmented reality gaming system. Communications of the ACM, 45(1), 36-38.
  40. 40. PCI 3D Graphics Board Hard Drive Serial Ports CPU PC104 Sound Card PC104 PCMCIA GPS Antenna RTK correction Antenna HMD Controller Tracker Controller DC to DC Converter Battery Wearable Computer GPS RTK correction Radio Example self-built working solution with PCI-based 3D graphics Columbia Touring Machine Mobile AR - Hardware
  41. 41. HIT Lab NZ Wearable AR (2004) ▪ Highly accurate outdoor AR tracking system ▪ GPS, Inertial, RTK system ▪ HMD ▪ First prototype ▪ Laptop based ▪ Video see-through HMD ▪ 2-3 cm tracking accuracy
  42. 42. 2008: Location Aware Phones Nokia NavigatorMotorola Droid
  43. 43. 2009 - Layar (www.layar.com) • Location based data – GPS + compass location – Map + camera view • AR Layers on real world – Customized data – Audio, 3D, 2D content • Easy authoring • Android, iPhone
  44. 44. Wearable Evolution Backpack+HMD: …10+ kg Handheld + HMD … Separate sensors .... UMPC 1.1GHz …1.5kg …still >$5K Scale it down more: Smartphone…$500 …Integrated …0.1kg …billions of units 1997 2003 2007
  45. 45. Google Glass (2011 - )
  46. 46. ▪ Hardware ▪ CPU TI OMAP 4430 – 1 Ghz ▪ 16 GB SanDisk Flash,1 GB Ram ▪ 570mAh Battery ▪ Input ▪ 5 mp camera, 720p recording, microphone ▪ GPS, InvenSense MPU-9150 inertial sensor ▪ Output ▪ Bone conducting speaker ▪ 640x360 micro-projector display Google Glass Specs
  47. 47. Other Wearables ▪ Vuzix M-100 ▪ $999, professional ▪ Recon Jet ▪ $600, more sensors, sports ▪ Opinvent ▪ 500 Euro, multi-view mode ▪ Motorola Golden-i ▪ Rugged, remote assistance
  48. 48. Ex: Recon Instruments Snow Ski display/computer ▪ Location, speed, altitude, phone headset http://www.reconinstruments.com/
  49. 49. Projected Market
  50. 50. dsfh
  51. 51. Summary Wearables are a new class of computing Intimate, persistent, aware, accessible, connected Evolution over 50 year history Backpack to head worn Custom developed to consumer ready device Enables new applications Collaboration, memory, AR, industry, etc Many head worn wearables are coming Android based, sensor package, micro-display
  52. 52. Evolution + Design Principles
  53. 53. Last year Last week NowForever The Now machine Focus on location, contextual and timely information, and communication.
  54. 54. Why Glass? Leadership vision
  55. 55. "Computing should just be more comfortable" "Google should do the hard work, and you should have a chance to live, have a good life, and get on with it."
  56. 56. As technology becomes more personal and immediate, it can start to disappear. Distant Intimate
  57. 57. Early prototyping
  58. 58. Early prototyping
  59. 59. Early prototyping
  60. 60. Early prototyping
  61. 61. Early prototyping
  62. 62. Early prototyping
  63. 63. Early prototyping
  64. 64. Early prototyping
  65. 65. Early prototyping
  66. 66. Early prototyping
  67. 67. Early prototyping Early prototyping
  68. 68. Early prototyping Early prototyping
  69. 69. Concept video
  70. 70. Reality
  71. 71. What we learned
  72. 72. Transparent displays are tricky Colors are funny and inconsistent. You can only add light to a scene, not cover anything up. Motion can be disorienting. Clarity, contrast, brightness, visual field and attention are important.
  73. 73. Reading Some things don’t work
  74. 74. Immersion Some things don’t work
  75. 75. Details Some things don’t work
  76. 76. Design principles
  77. 77. The world is the experience Get the interface and interactions out of the way.
  78. 78. Micro Interactions The position of the display and limited input ability makes longer interactions less comfortable. Using it shouldn’t take longer than taking out your phone.
  79. 79. A rear view mirror Don't overload the user. Stick to the absolutely essential, avoid long interactions. Be explicit.
  80. 80. As personal as it gets Recognize and adapt to the user… not the other way around. Glass is the most personal device you own. It operates closer to your most valuable senses and your environment, and it knows more about both. Glass should adapt, be personal and recognize the wearer; be aware of what consists his identity (physiology, memory, preferences, environment) and connect to it. As personal as it gets Glass is the most personal device you own. It should recognize and adapt to you… not the other way around.
  81. 81. For the closest people and most important moments Glass should be for prioritizing your closest people and creating value for the whole group, not just the wearer.
  82. 82. Examples
  83. 83. Search Picture Messaging PhoneNavigation Video CallVideo
  84. 84. Platform
  85. 85. How are people using Glass for creative expression?
  86. 86. A world of stories In Viewpoint of Billions by David Datuna, Glass allows viewers to unlock images and video with interactive experiences.
  87. 87. Social action First-person journalist Tim Pool broadcasts an intimate view of Istanbul protests. 'I want to show you what it's like to be there as best I can, even if that ends with me running full-speed into a cafe and rubbing lemons all over my face after being tear-gassed'
  88. 88. Sharing and connecting Conductor Cynthia Johnston Turner shares a 1st person experience with her orchestra.
  89. 89. Personal expression Alexander Chen’s Viola through Glass.
  90. 90. Our tools are becoming more intimate and immediate. We can craft a future of learning, creative expression and empathy. Distant Intimate
  91. 91. 4: Prototyping Tools
  92. 92. How can we quickly prototype Wearable experiences with little or no coding?
  93. 93. Why Prototype? ▪ Quick visual design ▪ Capture key interactions ▪ Focus on user experience ▪ Communicate design ideas ▪ “Learn by doing/experiencing”
  94. 94. Prototyping Tools ▪ Static/Low fidelity ▪ Sketching ▪ User interface templates ▪ Storyboards/Application flows ▪ Interactive/High fidelity ▪ Wireframing tools ▪ Mobile prototyping ▪ Native Coding
  95. 95. Important Note ▪ Most current wearables run Android OS ▪ eg Glass, Vuzix, Atheer, Epson, etc ▪ So many tools for prototyping on Android mobile devices will work for wearables ▪ If you want to learn to code, learn ▪ Java, Android, Javascript/PHP
  96. 96. Typical Development Steps ▪ Sketching ▪ Storyboards ▪ UI Mockups ▪ Interaction Flows ▪ Video Prototypes ▪ Interactive Prototypes ▪ Final Native Application Increased Fidelity & Interactivity
  97. 97. Sketched Interfaces ▪ Sketch + Powerpoint/Photoshop/Illustrator
  98. 98. GlassSim – http://glasssim.com/ ▪ Simulate the view through Google Glass ▪ Multiple card templates
  99. 99. GlassSim Card Builder ▪ Use HTML for card details ▪ Multiple templates ▪ Change background ▪ Own image ▪ Camera view
  100. 100. GlassSim Samples
  101. 101. Glass UI Templates ▪ Google Glass Photoshop Templates ▪ http://glass-ui.com/ ▪ http://dsky9.com/glassfaq/the-google-glass-psd-template/
  102. 102. Sample Slides From Templates
  103. 103. Application Storyboard ▪ http://dsky9.com/glassfaq/google-glass- storyboard-template-download/
  104. 104. ToolKit for Designers ▪ Vectoform Google Glass Toolkit for Designers ▪ http://blog.vectorform.com/2013/09/16/google-glass- toolkit-for-designers-2/ ▪ Sample cards, app flows, icons, etc
  105. 105. Application Flow
  106. 106. Limitations ▪ Positives ▪ Good for documenting screens ▪ Can show application flow ▪ Negatives ▪ No interactivity/transitions ▪ Can’t be used for testing ▪ Can’t deploy on wearable ▪ Can be time consuming to create
  107. 107. Transitions
  108. 108. ▪Series of still photos in a movie format. ▪Demonstrates the experience of the product ▪Discover where concept needs fleshing out. ▪Communicate experience and interface ▪You can use whatever tools, from Flash to iMovie. Video Sketching
  109. 109. See https://vine.co/v/bgIaLHIpFTB Example: Video Sketch of Vine UI
  110. 110. UI Concept Movies
  111. 111. Interactive Wireframes
  112. 112. Interactive Wireframing ▪ Developing interactive interfaces/wireframes ▪ Transitions, user feedback, interface design ▪ Web based tools ▪ UXpin - http://www.uxpin.com/ ▪ proto.io - http://www.proto.io/ ▪ Native tools ▪ Justinmind - http://www.justinmind.com/ ▪ Axure - http://www.axure.com/
  113. 113. UXpin - www.uxpin.com ▪ Web based wireframing tool ▪ Mobile/Desktop applications ▪ Glass templates, run in browser https://www.youtube.com/watch?v=0XtS5YP8HcM
  114. 114. Proto.io - http://www.proto.io/ ▪ Web based mobile prototyping tool ▪ Features ▪ Prototype for multiple devices ▪ Gesture input, touch events, animations ▪ Share with collaborators ▪ Test on device
  115. 115. Proto.io - Interface
  116. 116. Demo: Building a Simple Flow
  117. 117. Gesture Flow Scr1 Scr2 Scr3 Scr4 Scr5 Scr6 Ta p Swipe
  118. 118. Start Transitions
  119. 119. Demo
  120. 120. Justinmind - http://www.justinmind.com/ ▪ Native wireframing tool ▪ Build mobile apps without programming ▪ drag and drop, interface templates ▪ web based simulation ▪ test on mobile devices ▪ collaborative project sharing ▪ Templates for Glass, custom templates
  121. 121. User Interface - Glass Templates
  122. 122. Web Simulation Tool
  123. 123. Comparing Wireframe Tools Tool Web Native Wearable Template Interaction Uxpin X X Proto.io X X Justinmind X X X Axure X X X
  124. 124. Wireframe Limitations ▪ Can’t deploy on Glass ▪ No access to sensor data ▪ Camera, orientation sensor ▪ No multimedia playback ▪ Audio, video ▪ Simple transitions ▪ No conditional logic ▪ No networking
  125. 125. Processing for Wearables
  126. 126. Processing ▪ Programming tool for Artists/Designers ▪ http://processing.org ▪ Easy to code, Free, Open source, Java based ▪ 2D, 3D, audio/video support ▪ Processing For Android ▪ http://wiki.processing.org/w/Android ▪ Strong Android support ▪ Generates Android .apk file
  127. 127. Processing - Motivation ▪ Language of Interaction ▪ Sketching with code ▪ Support for rich interaction ▪ Large developer community ▪ Active help forums ▪ Dozens of plug-in libraries ▪ Strong Android support ▪ Easy to run on wearables
  128. 128. http://processing.org/
  129. 129. http://openprocessing.org/
  130. 130. Development Enviroment
  131. 131. Basic Parts of a Processing Sketch /* Notes comment */ //set up global variables float moveX = 50; //Initialize the Sketch void setup (){ } //draw every frame void draw(){ }
  132. 132. Importing Libraries ▪ Can add functionality by Importing Libraries ▪ java archives - .jar files ▪ Include import code import processing.opengl.*; ▪ Popular Libraries ▪ Minim - audio library ▪ OCD - 3D camera views ▪ Physics - physics engine ▪ bluetoothDesktop - bluetooth networking
  133. 133. http://toxiclibs.org/
  134. 134. Processing and Glass ▪ One of the easiest ways to build rich interactive wearable applications ▪ focus on interactivity, not coding ▪ Collects all sensor input ▪ camera, accelerometer, touch ▪ Can build native Android .apk files ▪ Side load onto Glass
  135. 135. Example: Hello World //called initially at the start of the Processing sketch void setup() { size(640, 360); background(0); } //called every frame to draw output void draw() { background(0); //draw a white text string showing Hello World fill(255); text("Hello World", 50, 50); }
  136. 136. Demo
  137. 137. Hello World Image PImage img; // Create an image variable void setup() { size(640, 360); //load the ok glass home screen image img = loadImage("okGlass.jpg"); // Load the image into the program } void draw() { // Displays the image at its actual size at point (0,0) image(img, 0, 0); }
  138. 138. Demo
  139. 139. Touch Pad Input ▪ Tap recognized as DPAD input void keyPressed() { if (key == CODED){ if (keyCode == DPAD) { // Do something .. ▪ Java code to capture rich motion events ▪ import android.view.MotionEvent;
  140. 140. Motion Event //Glass Touch Events - reads from touch pad public boolean dispatchGenericMotionEvent(MotionEvent event) { float x = event.getX(); // get x/y coords float y = event.getY(); int action = event.getActionMasked(); // get code for action switch (action) { // let us know which action code shows up case MotionEvent.ACTION_DOWN: touchEvent = "DOWN"; fingerTouch = 1; break; case MotionEvent.ACTION_MOVE: touchEvent = "MOVE"; xpos = myScreenWidth-x*touchPadScaleX; ypos = y*touchPadScaleY; break;
  141. 141. Demo
  142. 142. Sensors ▪ Ketai Library for Processing ▪ https://code.google.com/p/ketai/ ▪ Support all phone sensors ▪ GPS, Compass, Light, Camera, etc ▪ Include Ketai Library ▪ import ketai.sensors.*; ▪ KetaiSensor sensor;
  143. 143. Using Sensors ▪ Setup in Setup( ) function ▪ sensor = new KetaiSensor(this); ▪ sensor.start(); ▪ sensor.list(); ▪ Event based sensor reading void onAccelerometerEvent(…) { accelerometer.set(x, y, z); }
  144. 144. Sensor Demo
  145. 145. Using the Camera ▪ Import camera library ▪ import ketai.camera.*; ▪ KetaiCamera cam; ▪ Setup in Setup( ) function ▪ cam = new KetaiCamera(this, 640, 480, 15); ▪ Draw camera image void draw() { //draw the camera image image(cam, width/2, height/2); }
  146. 146. Camera Demo
  147. 147. Timeline Demo ▪ Create Card Class ▪ load image, card number, children/parent cards ▪ Timeline Demo ▪ Load cards in order ▪ Translate cards with finger motion ▪ Swipe cards in both directions ▪ Snap cards into position
  148. 148. Native Coding
  149. 149. Overview ▪ For best performance need native coding ▪ Low level algorithms etc ▪ Most current wearables based on Android OS ▪ Need Java/Android skills ▪ Many devices have custom API/SDK ▪ Vusix M-100: Vusix SDK ▪ Glass: Mirror API, Glass Developer Kit (GDK)
  150. 150. Mirror API + Glass GDK
  151. 151. Glassware and Timeline
  152. 152. Glassware and Timeline ▪ Static Cards ▪ Static content with text, HTML, images, and video. ▪ e.g. notification messages, news clip ▪ Live Cards ▪ Dynamic content updated frequently. ▪ e.g. compass, timer ▪ Immersions ▪ Takes over the whole control, out from timeline. ▪ e.g. interactive game
  153. 153. Glassware Development ▪ Mirror API ▪ Server programming, online/web application ▪ Static cards / timeline management ▪ GDK ▪ Android programming, Java (+ C/C++) ▪ Live cards & Immersions ▪ See: https://developers.google.com/glass/
  154. 154. ▪ REST API ▪ Java servlet, PHP, Go, Python, Ruby, .NET ▪ Timeline based apps ▪ Static cards - Text, HTML, media attachment (image & video) - Standard and custom menu items ▪ Manage timeline - Subscribe to timeline notifications - Sharing with contacts - Location based services Mirror API
  155. 155. GDK ▪ Glass Development Kit ▪ Android 4.0.3 ICS + Glass specific APIs ▪ Use standard Android Development Tools
  156. 156. ▪ GDK add-on features ▪ Timeline and cards ▪ Menu and UI ▪ Touch pad and gesture ▪ Media (sound, camera and voice input) GDK
  157. 157. Glass Summary ▪ Use Mirror API if you need ... ▪ Use GDK if you need ... ▪ Or use both
  158. 158. Hardware Prototyping
  159. 159. Fake Display 3D print Thingiverse model see http://www.thingiverse.com/thing:65706 Have the social impact of Google Glass without the cost
  160. 160. Build Your Own Wearable ▪ MyVu display + phone + sensors
  161. 161. Beady-i ▪ http://www.instructables.com/id/DIY- Google-Glasses-AKA-the-Beady-i/
  162. 162. Rasberry Pi Glasses ▪ Modify video glasses, connect to Rasberry Pi ▪ $200 - $300 in parts, simple assembly ▪ https://learn.adafruit.com/diy-wearable-pi-near-eye-kopin-video- glasses
  163. 163. Physical Input Devices ▪ Can we develop unobtrusive input devices ? ▪ Reduce need for speech, touch pad input ▪ Socially more acceptable ▪ Examples ▪ Ring, ▪ pendant, ▪ bracelet, ▪ gloves, etc
  164. 164. Prototyping Platform Arduino Kit Bluetooth Shield Google Glass
  165. 165. Example: Glove Input ▪ Buttons on fingertips ▪ Map touches to commands
  166. 166. Example: Ring Input ▪ Touch strip, button, accelerometer ▪ Tap, swipe, flick actions
  167. 167. How it works Bracelet Armband Gloves 1,2, 3,4 Values/ output
  168. 168. Summary ▪ Prototyping for wearables is similar to mobiles ▪ Tools for UI design, storyboarding, wireframing ▪ Android tools to create interactive prototypes ▪ App Inventor, Processing, etc ▪ Arduino can be used for hardware prototypes ▪ Once prototyped Native Apps can be built ▪ Android + SDK for each platform
  169. 169. Other Tools ▪ Wireframing ▪ pidoco ▪ FluidUI ▪ Rapid Development ▪ Phone Gap ▪ AppMachine ▪ Interactive ▪ App Inventor ▪ WearScript
  170. 170. App Inventor - http://appinventor.mit.edu/ ▪ Visual Programming for Android Apps ▪ Features ▪ Access to Android Sensors ▪ Multimedia output ▪ Drag and drop web based interface ▪ Designer view – app layout ▪ Blocks view – program logic/control
  171. 171. App Inventor Designer View
  172. 172. App Inventor Blocks View
  173. 173. Orientation Demo ▪ Use wearable orientation sensor
  174. 174. WearScript ▪ JavaScript development for Glass ▪ http://www.wearscript.com/en/ ▪ Script directory ▪ http://weariverse.com/
  175. 175. Best Practices (Dos + don’ts)
  176. 176. ✓ Don’t design an app Glass OS is time-based model, not an app model. X
  177. 177. ✓ Know what makes Glass different than a phone Glass has certain superpowers. Remember what these superpowers are and use them to augment the experience you’re designing.
  178. 178. X Don’t just port your mobile experience over to Glass It won’t work. It will be too busy. It will be hard for users to quickly understand and navigate your content. The constraints on Glass are simply too strict. Bad idea. Even inverted, aside from the need to properly format each screen, the layout simply contains too much information for Glass. Good idea!
  179. 179. ✓ Design for the (hyper)now When is my next meeting? How many calories have I eaten today? Can I get a burger for lunch? Spend 90% of your time thinking about what people want to know (in sport or elsewhere) at any given moment. The more you know about what info people need and currently don’t have - the more compelling your design will be.
  180. 180. ✓ Do one thing at a time
  181. 181. ✓ Design for emotion thumbs up viewsnap - running
  182. 182. ✓ Make it glanceable Seek to rigorously reduce information density. Successful designs afford for recognition, not reading. Bad Good
  183. 183. ✓ Reduce the number of info chunksX✓ You are designing for recognition, not reading. Reducing the total # of information chunks will greatly increase the glanceability of your design. 1 2 3 1 2 3 4 5 (6) Test done by Morten Just using a watch
  184. 184. ✓ Design single interactions to be faster than 4 s Eye movements For 1: 1 230ms For 2: 1 230ms For 3: 1 230ms For 4: 3 (52/17) 690ms For 5: 2 460ms ~1,840ms Eye movements For 1: 1-2 460ms For 2: 1 230ms For 3: 1 230ms ~920ms 1 2 3 1 2 3 4 5 (6) Test done by Morten Just using a watch
  185. 185. Test the glanceability of your design✓
  186. 186. ✓ Test your design indoors + outdoors
  187. 187. ✓ White is your new black
  188. 188. ✓ On the device, black is not blackX Your mock Device simulation
  189. 189. ✓ Establish hierarchy with color - not font size White is your <h1> and grey is your <h2> or <h3>. Footer text - establishing time, attribution, or distance - is the only place where you see a smaller font size used.
  190. 190. ✓ If you have brand-specific typography - use it
  191. 191. ✓ Use relative information display calendar card - in 10 minutes, Fri - viewsnap
  192. 192. ✓ Remember, people have an ever-growing ecosystem of wearablesX Each device should be used when it’s most relevant and when it’s the easiest interaction available.
  193. 193. ✓ The Glass screen is just one part of the experience
  194. 194. ✓ Do view your design on the device 1. Download Android Design Preview 2. Plug in Glass 3. Terminal command: java -jar AndroidDesignPreview-0.3.2.jar 4. Drag the red rectangle over your work to take a look 5. Check out the developer site for more in-depth guidelines
  195. 195. 5: Concept Design Exercise
  196. 196. Design Group (33 People) ▪ If your name is on this list you are in the Design Group Yang Wang Konstantino Kapetaneas Preethi Srinivas Tony James Kate Vogt Aneesh Tarun Josh Andres Maria Maimó Bram Reurings Luke Mill Tuck-Voon How M Gill Janaki Kumar Melinda Knight M Calkins Mike Tissenbaum Samantha Tse Kal McDowd Adora Tam Oscar Meruvia Mike Chen Anita Hoechtl Merlin Stone Ashoomi Dohlakia Icy Zhu Zdenek Mikovec Cristina Manresa-Yee Christian Winkler Angela Noh Amyris Fernandez Deborah Ptak Arne Renkema-Padmos Thomas Fritz
  197. 197. 6: Wearable Technologies
  198. 198. Wearable System
  199. 199. Some Key Aspects ▪ Display Technologies ▪ Input Devices ▪ Interaction Metaphors ▪ Perceptual Factors ▪ Ergonomics ▪ Cognitive Aspects
  200. 200. Display Technologies
  201. 201. Key Properties of HMD ▪ Field of View ▪ Human eye 95 deg. H, 60/70 deg. V ▪ Resolution ▪ > 320x240 pixel ▪ Refresh Rate ▪ Focus ▪ Fixed/manual ▪ Size, Weight ▪ < 350g for long term ▪ Power
  202. 202. Types of Head Mounted Displays Occluded See-thru Multiplexed
  203. 203. Optical see-through HMD Virtual images from monitors Real World Optical Combiners
  204. 204. Optical See-Through HMD
  205. 205. Epson Moverio BT-200 ▪ Stereo see-through display ($700) ▪ 960 x 540 pixels, 23 degree FOV, 60Hz, 88g ▪ Android Powered, separate controller ▪ VGA camera, GPS, gyro, accelerometer
  206. 206. Strengths of optical see-through ▪ Simpler (cheaper) ▪ Direct view of real world ▪ Full resolution, no time delay (for real world) ▪ Safety ▪ Lower distortion ▪ No eye displacement ▪ see directly through display
  207. 207. Video see-through HMD Video cameras Monitors Graphics Combiner Video
  208. 208. Video See-Through HMD
  209. 209. Vuzix Wrap 1200DXAR ▪ Stereo video see-through display ($1500) ▪ Twin 852 x 480 LCD displays, 35 deg. FOV ▪ Stereo VGA cameras ▪ 3 DOF head tracking
  210. 210. Strengths of Video See-Through ▪ True occlusion ▪ Block image of real world ▪ Digitized image of real world ▪ Flexibility in composition ▪ Matchable time delays ▪ More registration, calibration strategies ▪ Wide FOV is easier to support ▪ wide FOV camera
  211. 211. Multiplexed Displays ▪ Above or below line of sight ▪ Strengths ▪ User has unobstructed view of real world ▪ Simple optics/cheap ▪ Weaknesses ▪ Direct information overlay difficult ▪ Display/camera offset from eyeline ▪ Wide FOV difficult
  212. 212. Vuzix M-100 ▪ Monocular multiplexed display ($1000) ▪ 852 x 480 LCD display, 15 deg. FOV ▪ 5 MP camera, HD video ▪ GPS, gyro, accelerometer
  213. 213. Display Types ▪ Curved Mirror ▪ off-axis projection ▪ curved mirrors in front of eye ▪ high distortion, small eye-box ▪ Waveguide ▪ use internal reflection ▪ unobstructed view of world ▪ large eye-box
  214. 214. See-through thin displays ▪ Waveguide techniques for thin see-through displays ▪ Wider FOV, enable AR applications ▪ Social acceptability Opinvent Ora
  215. 215. Waveguide Methods See: http://optinvent.com/HUD-HMD-benchmark#benchmarkTable Holographic Hologram diffracts light Limited FOV Colour bleeding Diffractive Slanted gratings Total internal reflection Costly, small FOV
  216. 216. Waveguide Methods See: http://optinvent.com/HUD-HMD-benchmark#benchmarkTable Clear-Vu Reflective Several reflective elements Thinner light guide Large FOV, eye-box Reflective Simple reflective elements Lower cost Size is function of FOV
  217. 217. Comparison Chart
  218. 218. Input Technologies
  219. 219. Input Options ▪ Physical Devices ▪ Keyboard ▪ Pointer ▪ Stylus ▪ Natural Input ▪ Speech ▪ Gesture ▪ Other ▪ Physiological
  220. 220. Twiddler Input ▪ Chording or multi-tap input ▪ Possible to achieve 40 - 60 wpm after 30+ hours ▪ Chording input about 50% faster than multi-tap ▪ cf 20 wpm on T9, or 60+ wpm for QWERTY Lyons, K., Starner, T., Plaisted, D., Fusia, J., Lyons, A., Drew, A., & Looney, E. W. (2004, April). Twiddler typing: One-handed chording text entry for mobile phones. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 671-678). ACM.
  221. 221. Virtual Keyboards ▪ In air text input ▪ Virtual QWERTY keyboard up to 20 wpm ▪ On real keyboard around 45-60+ wpm ▪ Word Gesture up to 28 wpm ▪ On tablet/phone Word Gesture up to 47 wpm ▪ Handwriting around 20-30 wpm A. Markussen, et. al. Vulture: A Mid-Air Word-Gesture Keyboard (CHI 2014)
  222. 222. Unobtrusive Input Devices ▪ GestureWrist ▪ Capacitive sensing ▪ Change signal depending on hand shape Rekimoto, J. (2001). Gesturewrist and gesturepad: Unobtrusive wearable interaction devices. In Wearable Computers, 2001. Proceedings. Fifth International Symposium on (pp. 21-27). IEEE.
  223. 223. Unobtrusive Input Devices ▪ GesturePad ▪ Capacitive multilayered touchpads ▪ Supports interactive clothing
  224. 224. Skinput Using EMG to detect muscle activity Tan, D., Morris, D., & Saponas, T. S. (2010). Interfaces on the go. XRDS: Crossroads, The ACM Magazine for Students, 16(4), 30-34.
  225. 225. Issues to Consider ▪ Fatigue ▪ “Gorrilla” Arm from free-hand input ▪ Comfort ▪ People want to do small gestures by waist ▪ Interaction on the go ▪ Can input be done while moving?
  226. 226. Interaction on the Go ▪ Fitt’s law still applies while interacting on the go ▪ Eg: Tapping while walking reduces speed by > 35% ▪ Increased errors while walking Lin, M., Goldman, R., Price, K. J., Sears, A., & Jacko, J. (2007). How do people tap when walking? An empirical investigation of nomadic data entry.International Journal of Human-Computer Studies, 65(9), 759-769.
  227. 227. Interface Metaphors
  228. 228. Information Display Head Stabilized Fixed View Body Stabilized 3 DOF Tracking World Stabilized 6 DOF Tracking
  229. 229. Spatial Cues for Wearable Info Billinghurst, M., Bowskill, J., Dyer, N., & Morphett, J. (1998, March). An evaluation of wearable information spaces. In Virtual Reality Annual International Symposium, 1998. Proceedings., IEEE 1998 (pp. 20-27). IEEE. ▪ Spatial cues sign. improve performance ▪ No difference between audio and visual cues
  230. 230. Spatial Conferencing 3+ attendees can be distinguished with spatialized audio but could not without spatialized audio Billinghurst, M., Bowskill, J., Jessop, M., & Morphett, J. (1998, October). A wearable spatial conferencing space. In Proceedings of ISWC, 1998. (pp. 76-83). IEEE.
  231. 231. Organizing Tools in a Halo Display Biocca, F., Tang, A., Lamas, D., Gregg, J., Brady, R., & Gai, P. (2001). How do users organize virtual tools around their body in immersive virtual and augmented environment?: An exploratory study of egocentric spatial mapping of virtual tools in the mobile infosphere. Media Interface and Network Design Labs, Michigan State University, East Lansing, MI.
  232. 232. User Attention Metaphors Cognitive continuums (a) Input, (b) Output Increase cognitive load from left to right
  233. 233. Notification Interruptions ▪ Gradually increase engagement ▪ Reduce attention load Receiving SMS on Glass “Bing” Tap Swipe Glass Show Message Start Reply User Look Up Say Reply
  234. 234. Nomadic Radio (2000) ▪ Spatial audio wearable interface Sawhney, N., & Schmandt, C. (2000). Nomadic radio: speech and audio interaction for contextual messaging in nomadic environments. ACM transactions on Computer-Human interaction (TOCHI), 7 (3), 353-383.
  235. 235. Spatial Audio Metaphor Messages/Events arranged depending on time of day
  236. 236. Notification Interruptions ▪ Dynamic scaling of incoming message based on interruptibility of the user ▪ Busy = silence ▪ Availble = preview
  237. 237. Layered Audio Notifications Background ambient audio Notification scale depending on priority
  238. 238. Perception Issues
  239. 239. AR Perceptual Issues ❖ Environment: Issues related to the environment itself. ❖ Capturing: Issues related to digitizing the environment ❖ Augmentation: Issues related to the design, layout, and registration or AR content ❖ Display device: Technical issues associated with the display device. ❖ User: Issues associated with user perceiving content. E. Kruijff, J. E. Swan, and S. Feiner. Perceptual issues in augmented reality revisited. 9th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2010, pp. 3--12.
  240. 240. Cognitive Issues in Mobile AR ▪ Three categories of issues ▪ Information Presentation – displaying virtual information on the real world ▪ Physical Interaction – content creation, manipulation and navigation ▪ Shared Experience – collaboration and supporting common experiences Li, Nai, and Henry Been-Lirn Duh. "Cognitive Issues in Mobile Augmented Reality: An Embodied Perspective." Human Factors in Augmented Reality Environments. Springer New York, 2013. 109-135.
  241. 241. Depth Cues ❖ Pictorial: visual cues • Occlusion, texture, relative brightness ❖ Kinetic: motion cues • Relative motion parallax, motion perspective ❖ Physiological: motion cues • Convergence, accommodation ❖ Binocular disparity • Two different eye images
  242. 242. Use the Following Depth Cues ▪ Movement parallax. ▪ Icon/Object size (for close objects) ▪ Linear perspective ▪ To add side perspective bar. ▪ Overlapping ▪ Works if the objects are big enough ▪ Shades and shadows. ▪ Depends on the available computation
  243. 243. Information Presentation • Amount of information • Clutter, complexity • Representation of information • Navigation cues, POI representation • Placement of information • Head, body, world stabilized • View combination • Multiple views
  244. 244. Twitter 360 ▪ www.twitter-360.com ▪ iPhone application ▪ See geo-located tweets in real world ▪ Twitter.com supports geo tagging
  245. 245. Wikitude – www.mobilizy.com Bl ah Bl ah Bl ah Bl ah Bl ah Bl ah Bl ah Bl ah Bl ah Bl ah Bl ah Bl ah Bl ah Bl ah Bl ah Bl ah Bl ah Bl ah Bl ah Bl ah Bl ah Bl ah
  246. 246. Information Filtering
  247. 247. Information Filtering •Use context to support information filtering •Show less information on AR mode (uncluttered screen)
  248. 248. Text Representation ▪ Object space ▪ In general, does not fit wearable applications ▪ Billboards ▪ Screen space ▪ Annotations placed on a 2D plane, usually parallel to the projection plane. ▪ Better legibility of text.
  249. 249. Outdoor background textures Changing outdoor illuminanation, and text drawing styles. Billboard and green text drawing styles are recommended. Active text drawing styles did not perform better relative to static styles.
  250. 250. Screen Space Annotation
  251. 251. AR Navigation ▪ Problem – how to AR to assist with navigation to POI
  252. 252. Multi-View Navigation ▪ Use multiple views (ego vs exo-centric) ▪ Map/top down view – long distance navigation ▪ AR View – nearby navigation, situational awareness ▪ Radar view in AR view
  253. 253. Wearable Display: Limited FOV
  254. 254. Transitioning Between Views ▪ Seamless spatial awareness
  255. 255. Cognitive Models
  256. 256. Resource Competition Framework ▪ Mobility tasks compete for cognitive resources with other tasks ▪ the most important given higher priority ▪ RCF is a method for analyzing this, based on: ▪ task analysis ▪ modelling cognitive resources ▪ a resource approach to attention Oulasvirta, A., Tamminen, S., Roto, V., & Kuorelahti, J. (2005, April). Interaction in 4-second bursts: the fragmented nature of attentional resources in mobile HCI. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 919-928). ACM.
  257. 257. RCF Key Assumptions Four Key Assumptions 1. Functional Modularity: cognitive system divided into functionally separate systems with diff. representations 2. Parallel Module Operation: cognitive modules operate in parallel, independent of each other 3. Limited Capacity: cognitive modules are limited in capacity with respect to time or content 4. Serial Central Operation: central coordination of modules (eg monitoring) is serial
  258. 258. Cognitive Interference ▪ Structural interference ▪ Two or more tasks compete for limited resources of a peripheral system ▪ eg two cognitive processes needing vision ▪ Capacity interference ▪ Total available central processing overwhelmed by multiple concurrent tasks ▪ eg trying to add and count at same time
  259. 259. Cognitive Resources & Limitations asdfasdf
  260. 260. Using RCF 1. Map cognitive faculty to task 2. Look for conflicts/overloads 3. Analyse for competition for attention 4. Look for opportunities for technology to reduce conflicts/competition
  261. 261. Example: Going to work .. Which is the most cognitively demanding?
  262. 262. Application of RCF Busy street > Escalator > Café > Laboratory. But if you made Wayfinding, Path Planning, Estimating Time to Target, Collision Avoidance easier?
  263. 263. Ergonomics
  264. 264. Where to put Wearables? ▪ Places for unobtrusive wearable technology Gemperle, F., Kasabach, C., Stivoric, J., Bauer, M., & Martin, R. (1998, October). Design for wearability. In Wearable Computers, 1998. Digest of Papers. Second International Symposium on (pp. 116-122). IEEE.
  265. 265. Where to Place Trackpad? ▪ User study 25 people different postures ▪ Front of thigh most preferred, torso/upper arm worst Thomas, Bruce, et al. "Determination of placement of a body-attached mouse as a pointing input device for wearable computers." 2012 16th International Symposium on Wearable Computers. IEEE Computer Society, 1999.
  266. 266. Where do users want Wearables? 29% on clothing 28% on wrist 12% on Glasses
  267. 267. Tool/Task Design Paradigm Lin, R., & Kreifeldt, J. G. (2001). Ergonomics in wearable computer design. International Journal of Industrial Ergonomics, 27(4), 259-269.
  268. 268. Social Perception
  269. 269. How is the User Perceived?
  270. 270. TAT Augmented ID
  271. 271. 7: Design Exercise
  272. 272. Building on Glass Design Exercise
  273. 273. Design for Glass Guidelines
  274. 274. Design for Glass Don’t get in the way Guidelines
  275. 275. Design for Glass Don’t get in the way Keep it timely Guidelines
  276. 276. Design for Glass Don’t get in the way Keep it timely Avoid the unexpected Guidelines
  277. 277. Design for Glass Don’t get in the way Keep it timely Avoid the unexpected Design for people Guidelines
  278. 278. Platform
  279. 279. Platform
  280. 280. Platform - Mirror API
  281. 281. Platform
  282. 282. If you could build anything on Glass, what would it be?
  283. 283. Sprint - Ideas Person (mom, family, astronomer) 60 seconds
  284. 284. Sprint - Ideas Place (in the kitchen, car, hike) 60 seconds
  285. 285. Sprint - Ideas Function (dance, stargaze, sequence dna) 60 seconds
  286. 286. Sprint - Final deliverable Make a poster - (Hayes will demo) Title One liner Picture Pitch What is it? What does it do? Why is it a good idea?
  287. 287. Sprint - Focus 10 min - Ideas to create a product to solve need
  288. 288. 3 min - Share with your group Sprint - Focus
  289. 289. Sprint - Focus 3 min - Pick a concept
  290. 290. Sprint - Focus 15 min - Dive deeper, develop, poster
  291. 291. Sprint - Focus 15 min - Poster
  292. 292. Sprint - Lightning 2 minute pitch
  293. 293. 7: Design Presentations
  294. 294. Design Concepts
  295. 295. 8: Research Directions
  296. 296. Challenges for the Future (2001) ▪ Privacy ▪ Power use ▪ Networking ▪ Collaboration ▪ Heat dissipation ▪ Interface design ▪ Intellectual tools ▪ Augmented Reality systems Starner, T. (2001). The challenges of wearable computing: Part 1. IEEE Micro,21(4), 44-52. Starner, T. (2001). The challenges of wearable computing: Part 2. IEEE Micro,21(4), 54-67.
  297. 297. Interface Design
  298. 298. Gesture Interaction
  299. 299. Capturing Behaviours ▪ 3 Gear Systems ▪ Kinect/Primesense Sensor ▪ Two hand tracking ▪ http://www.threegear.com
  300. 300. Gesture Interaction With Glass ▪ 3 Gear Systems ▪ Hand tracking ▪ Hand data sent to glass ▪ Wifi networking ▪ Hand joint position ▪ AR application rendering ▪ Vuforia tracking
  301. 301. Performance ▪ Full 3d hand model input ▪ 10 - 15 fps tracking, 1 cm fingertip resolution
  302. 302. User Study ▪ Gesture vs. Touch pad vs. Combined input ▪ Gesture 3x faster, no difference in accuracy
  303. 303. ●Meta Gesture Interaction ▪Depth sensor + Stereo see-through ▪https://www.spaceglasses.com/
  304. 304. Collaboration
  305. 305. Social Panoramas
  306. 306. Ego-Vision Collaboration ▪ Wearable computer ▪ camera + processing + display + connectivity
  307. 307. Current Collaboration ▪ First person remote conferencing/hangouts ▪ Limitations ▪ Single POV, no spatial cues, no annotations, etc
  308. 308. Sharing Space: Social Panoramas ▪ Capture and share social spaces in real time ▪ Enable remote people to feel like they’re with you
  309. 309. Key Technology ▪ Google Glass ▪Capture live panorama (compass + camera) ▪Capture spatial audio, live video ▪ Remote device (desktop, tablet) ▪Immersive viewing, live annotation
  310. 310. Awareness Cues ▪ Where is my partner looking? ▪Enhanced radar display, Context compass
  311. 311. Interaction ▪ Glass Touchpad Input/Tablet Input ▪Shared pointers, Shared drawing
  312. 312. User Evaluation ▪ Key Results Visual cues significantly increase awareness Pointing cues preferred for collaboration Drawing on Glass difficult, ranked low in usability
  313. 313. Intellectual Tools
  314. 314. Context Sensing ▪ Using context to manage information ▪ progressive information display as user shows interest ▪ Context from ▪ Speech ▪ Gaze ▪ Real world ▪ Wearable AR Display Ajanki, A., Billinghurst, M., Gamper, H., Järvenpää, T., Kandemir, M., Kaski, S., ... & Tossavainen, T. (2011). An augmented reality interface to contextual information. Virtual reality, 15(2-3), 161-173.
  315. 315. Gaze Interaction
  316. 316. AR View
  317. 317. More Information Over Time
  318. 318. OpenSource Eyetracker ▪ Could use open source eyetracking ▪ Open Shades Eye Tracker ▪ http://www.wearscript.com/en/latest/eyetracking.html
  319. 319. 9: Resources
  320. 320. Glass Resources ▪ Main Developer Website ▪ https://developers.google.com/glass/ ▪ Glass Apps Developer Site ▪ http://glass-apps.org/glass-developer ▪ Google Design Guidelines Site ▪ https://developers.google. com/glass/design/index?utm_source=tuicool ▪ Google Glass Emulator ▪ http://glass-apps.org/google-glass-emulator
  321. 321. Other Resources ▪ AR for Glass Website ▪ http://www.arforglass.org/ ▪ Vandrico Database of wearable devices ▪ http://vandrico.com/database
  322. 322. Books ▪ Programming Google Glass ▪ Eric Redmond ▪ Rapid Android Development: Build Rich, Sensor-Based Applications with Processing ▪ Daniel Sauter
  323. 323. Contact Details Mark Billinghurst ▪ email: mark.billinghurst@hitlabnz.org ▪ twitter: @marknb00 Hayes Raffle ▪ email: hraffle@google.com Feedback + followup form ▪ goo.gl/6SdgzA
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×