1. Rapid Prototyping for Wearables
January 15th 2015
TEI 2015 Studio
Mark Billinghurst
HIT Lab NZ
University of Canterbury
mark.billinghurst@canterbury.ac.nz
Daniela Busse
Director (UX)
Citi Ventures
daniela.busse@citi.com
3. Mark Billinghurst
▪ Director of HIT Lab NZ, University
of Canterbury
▪ PhD Univ. Washington
▪ Research on AR, mobile HCI,
Collaborative Interfaces
▪ More than 250 papers in AR, VR,
interface design
▪ Sabbatical in Glass team at
Google [x] in 2013
4. ● Daniela Busse
▪ UX Director
▪ Citi Ventures Global Innovation Network
▪ Previous
▪ Design Futurist, Samsung Research
America
▪ UX Director/Chief UX Architect, SAP
▪ PhD, Computer Science
▪ Glasgow University
5.
6.
7. ● What You Will Learn
▪ An introduction to wearable computers
▪ Key interface design guidelines
▪ Examples of good wearable design
▪ How to use a variety of rapid prototyping tools
▪ Hands on with Google Glass hardware
▪ Active areas of wearable computing research
8. ● Schedule
▪ 9:00: Introduction
▪ 9:15: Introduction to Wearables
▪ 9:35: Introduction to Design
▪ 10:00: Tools for Low Fidelity Prototyping
▪ 10:30 Break/Design Session
▪ 11:00 UX Design Guidelines
▪ 11:30: Tools for High Fidelity prototyping
▪ 12:00 Presentation of Designs
▪ 12:15 Research Directions
12. What is a Wearable Computer?
▪ Computer on the body that is:
▪ Always on
▪ Always accessible
▪ Always connected
▪ Other attributes
▪ Augmenting user actions
▪ Aware of user and surroundings
Rhodes, B. J. (1997). The wearable remembrance agent: A system for
augmented memory. Personal Technologies, 1(4), 218-224.
Mann, S. (1997). Wearable computing: A first step toward personal
imaging. Computer, 30(2), 25-32.
13. The Ideal Wearable
▪ Persists and Provides Constant Access: Designed
for everyday and continuous user over a lifetime.
▪ Senses and Models Context: Observes and
models the users and environment.
▪ Augments and Mediates: Information support for
the user in both the physical and virtual realities.
▪ Interacts Seamlessly: Adapts its input and output
modalities to those most appropriate at the time.
Starner, T. E. (1999). Wearable computing and contextual awareness
(Doctoral dissertation, Massachusetts Institute of Technology).
15. History of Wearables
▪ 1960-90: Early Exploration
▪ Custom build devices
▪ 1990 - 2000: Academic, Military Research
▪ MIT, CMU, Georgia Tech, EPFL, etc
▪ 1997: ISWC conference starts
▪ 1995 – 2005+: First Commercial Uses
▪ Niche industry applications, Military
▪ 2010 - : Second Wave of Wearables
▪ Consumer applications, Head & Wrist Worn
16. The Gamblers
▪ Timing device for roulette prediction
▪ Card counting hardware (toe input)
Ed Thorp (1961)
Thorp, E. O. (1998, October). The invention of the first wearable computer. In Wearable
Computers, 1998. Second International Symposium on (pp. 4-8). IEEE.
Belt computer
Shoe Input
Glasses
Display
Keith Taft (1972)
17. ● The Academics (1980’s - )
▪ MIT Media Lab – Wearable Computing (1993)
▪ CMU – Industrial wearables
http://www.media.mit.edu/wearables/
http://www.cs.cmu.edu/afs/cs/project/vuman/www/frontpage.html
19. Mobile AR: Touring Machine (1997)
▪ University of Columbia
▪ Feiner, MacIntyre, Höllerer, Webster
▪ Combines
▪ See through head mounted display
▪ GPS tracking
▪ Orientation sensor
▪ Backpack PC (custom)
▪ Tablet input
Feiner, S., MacIntyre, B., Höllerer, T., & Webster, A. (1997). A touring machine: Prototyping 3D mobile
augmented reality systems for exploring the urban environment. Personal Technologies, 1(4), 208-217.
20. Early Commercial Systems
▪ Xybernaut (1996 - 2007)
▪ Belt worn, HMD, 200 MHz
▪ ViA (1996 – 2001)
▪ Belt worn, Audio Interface
▪ 700 MHz Crusoe
▪ Symbol (1998 – )
▪ Wrist worn computer
▪ Finger scanner
21. ● Symbol WWC 1000 (1998 - )
▪ Wrist worn wearable + finger barcode scanner
▪ $3500 USD, current price $1000
▪ Over 30K sold in first 2 years, still selling (>100k units?)
▪ First widely deployed wearable computer
22. ● Reasons For Success
▪ Well defined large market niche
▪ Stock pickers with holster scanners
▪ Significant usability/ergonomics effort
▪ Over 40,000 hours user testing
▪ Provided significant performance improvement
▪ Met user needs, solved existing problems
▪ Addressed social factors
▪ Company with substantial R+D resources
Stein, R., Ferrero, S., Hetfield, M., Quinn, A., & Krichever, M. (1998, October). Development
of a commercially successful wearable data collection system. In Wearable Computers,
1998. Digest of Papers. Second International Symposium on (pp. 18-24). IEEE.
23. ● Second Gen. Systems
▪ Recon (2010 - )
▪ Head worn displays for sports
▪ Ski goggle display
▪ Investment from Intel (2013)
▪ Google (2011 - )
▪ Google Glass
▪ Consumer focus
31. ● User Experience
▪ Truly Wearable Computing
▪ Less than 46 ounces
▪ Hands-free Information Access
▪ Voice interaction, Ego-vision camera
▪ Intuitive User Interface
▪ Touch, Gesture, Speech, Head Motion
▪ Access to all Google Services
▪ Map, Search, Location, Messaging, Email, etc
36. ● Business Evolution
▪ First wearable companies
▪ Targeting niche markets
▪ Expensive/poorly designed solutions
▪ Mostly low sales (< 10,000)
▪ Current generation
▪ First general consumer wearable (Glass, others)
▪ Bigger niche markets (skiing, sports) – > 50K+ sales
▪ Many diversified devices
▪ Lower costs/better design
37. Summary
Wearables are a new class of computing
Intimate, persistent, aware, accessible, connected
Evolution over 50 year history
Backpack to head worn
Custom developed to consumer ready device
Enables new applications
Collaboration, memory, AR, industry, etc
Many types of wearables are coming
Android based, sensor package, micro-display
41. ● Process
▪ Empathize: Understand the user needs
▪ Define: Define the problem to be solved
▪ Ideate: Brainstorm solutions
▪ Prototype: Develop sample solutions
▪ Test: User evaluation/validation of solutions
43. ● Understanding
▪ Understand the Problem Space
▪ What types of problems are we trying to solve?
▪ What’s our bigger goal?
▪ Understand the User
▪ Who are we solving the problem for
▪ Creating a Persona (typical end user)
▪ Define the Design Challenge
▪ Express the problem you are addressing
▪ Identify problem Insight
45. Celine needs
& wants
because
_to capture and
share what she is
doing now_
_the experience to
be shared with
friends_______
she values being
able to take friends
with her on trips___
Persona - Celine
46. ● The Design Challenge
▪ How can a wearable computer be used to capture
and share a user’s experience with remote people?
▪ Key Insight: A wearable computer allows a person to
see and hear with the eyes and ears of another
47. ● Creating
▪ Brain storming
▪ Rapid idea generation
▪ “How might we?” questions
▪ “Yes and..” responses
▪ Body storming
▪ Act out ideas
▪ Organize ideas
▪ Idea map
48. It is easier to tone
down a wild idea than
to think up a new one.
Alex
Osborn
#androidwear
71. ● Glassware Flow Designer
▪ Visual tool for designing application flows
▪ Drag and drop interface
developers.google.com/glass/tools-downloads/glassware-flow-designer
72. Limitations
▪ Positives
▪ Good for documenting screens
▪ Can show application flow
▪ Negatives
▪ No interactivity/transitions
▪ Can’t be used for testing
▪ Can’t deploy on wearable
▪ Can be time consuming to create
73. ● Task #2
1. Sketch out a storyboard showing how the user
interacts with the application
2. Sketch out some sample user interfaces and
application screens
3. Create an interaction flow for the application
using Google Glassware Designer
75. ● Consider Your User
▪ Wearable User
▪ Probably Mobile
▪ One/no hand interaction
▪ Short application use
▪ Need to be able to multitask
▪ Use in outdoor or indoor environment
▪ Want to enhance interaction with real world
79. Micro
Interac7ons
The
posi*on
of
the
display
and
limited
input
ability
makes
longer
interac*ons
less
comfortable.
Using
it
shouldn’t
take
longer
than
taking
out
your
phone.
81. ● Time Looking at Screen
Oulasvirta, A. (2005). The fragmentation of attention in mobile
interaction, and what to do with it. interactions, 12(6), 16-18.
82. Design for MicroInteractions
▪ Design interaction less than a few seconds
▪ Tiny bursts of interaction
▪ One task per interaction
▪ One input per interaction
▪ Benefits
▪ Use limited input
▪ Minimize interruptions
▪ Reduce attention fragmentation
87. ● Don’t Get in the Way
▪ Enhance, not replace, real world interaction
88. ● Design for Interruptions
▪ Gradually increase engagement and attention load
▪ Respond to user engagement
Receiving SMS on Glass
“Bing”
Tap
Swipe
Glass
Show Message Start Reply
User
Look
Up
Say
Reply
91. Interface Guidelines
▪ Design for device
▪ Use Micro Interaction
▪ Make it glanceable
▪ Do one thing at a time
▪ Reduce number of information chunks
▪ Design for indoor and outdoor use
93. ● Ideal Applications
▪ Use cases that require:
▪ Hands-free interaction
▪ Mobile information access
▪ Constant access to information
▪ Access to computing/communication
▪ Supporting activity in real world
▪ Low likelihood of social issues
▪ Enhanced view of reality
97. Social
ac7on
First-‐person
journalist
Tim
Pool
broadcasts
an
in*mate
view
of
Istanbul
protests.
'I
want
to
show
you
what
it's
like
to
be
there
as
best
I
can,
even
if
that
ends
with
me
running
full-‐speed
into
a
cafe
and
rubbing
lemons
all
over
my
face
a<er
being
tear-‐gassed'
98.
99. ● CityViewAR
▪ Using AR to visualize Christchurch city buildings
▪ 3D models of buildings, 2D images, text, panoramas
▪ AR View, Map view, List view
▪ Available on Android market
102. ● Example –Telemedicine
▪ Vipaar + UAB - http://www.vipaar.com
▪ Endoscopic view streamed remotely
▪ Remote expert adds hands – viewed in Glass
103. ● Example: Virgin Atlantic
▪ Virgin Atlantic trialing Glass for customer check in
▪ Features
▪ Agent greets customer curb-side, start check-in process
▪ Provide customer details, personalized service
▪ Document verification – camera scanning of boarding pass
104. ▪ Advantages
▪ Focus attention on customer
▪ Moves agent to customer
- Earlier engagement
▪ Reduces technology barrier
between agent and customer
- Hide behind computer/desk
▪ Provides personalized service
- Name, flight details, weather,
diet, translation services, etc
105. “The trial was a huge success with positive feedback
from both our staff and customers on the usage of
wearable technology”
▪ Key findings
▪ Google Glass permitted the agent to maintain eye contact
showing they were engaged and interesting in helping.
▪ Some passengers were taken aback initially by Glass
wearing concierges, but, passengers responded well.
▪ Some technical challenges to overcome
- Short battery life, camera resolution, wifi issues
109. ▪ Series of still photos in a movie format.
▪ Demonstrates the experience of the product
▪ Discover where concept needs fleshing out.
▪ Communicate experience and interface
▪ You can use many tools, from Flash to iMovie.
● Video Sketching
115. Proto.io - http://www.proto.io/
▪ Web based mobile prototyping tool
▪ Features
▪ Prototype for multiple devices
▪ Gesture input, touch events, animations
▪ Share with collaborators
▪ Test on device
120. ● Android Design Preview
▪ Mirror portion of desktop to Android devices
▪ Works with Google Glass and other Android wearables
▪ Using any desktop application for prototyping
https://github.com/romannurik/AndroidDesignPreview
121. Wireframe Limitations
▪ Can’t deploy on Device
▪ No access to sensor data
▪ Camera, orientation sensor
▪ No multimedia playback
▪ Audio, video
▪ Simple transitions
▪ No conditional logic
▪ No networking
122. ● Task #3
1. Create a Pop interactive sketch project
showing the interface transitions
2. Develop a proto.io project showing the
interface interactivity
123. Processing
▪ Programming tool for Artists/Designers
▪ http://processing.org
▪ Easy to code, Free, Open source, Java based
▪ 2D, 3D, audio/video support
▪ Processing For Android
▪ http://wiki.processing.org/w/Android
▪ Strong Android support
▪ Generates Android .apk file
124. Processing - Motivation
▪ Language of Interaction
▪ Sketching with code
▪ Support for rich interaction
▪ Large developer community
▪ Active help forums
▪ Dozens of plug-in libraries
▪ Strong Android support
▪ Easy to run on wearables
127. Basic Parts of a Processing
Sketch/* Notes comment */
//set up global variables
float moveX = 50;
//Initialize the Sketch
void setup (){
}
//draw every frame
void draw(){
}
128. Processing and Glass
▪ One of the easiest ways to build rich
interactive wearable applications
▪ focus on interactivity, not coding
▪ Collects all sensor input
▪ camera, accelerometer, touch
▪ Can build native Android .apk files
▪ Side load onto Glass
129. Example: Hello World
//called initially at the start of the Processing sketch
void setup() {
size(640, 360);
background(0);
}
//called every frame to draw output
void draw() {
background(0);
//draw a white text string showing Hello World
fill(255);
text("Hello World", 50, 50);
}
131. Hello World Image
PImage img; // Create an image variable
void setup() {
size(640, 360);
//load the ok glass home screen image
img = loadImage("okGlass.jpg"); // Load the image into
the program
}
void draw() {
// Displays the image at its actual size at point (0,0)
image(img, 0, 0);
}
133. Touch Pad Input
▪ Tap recognized as DPAD input
void keyPressed() {
if (key == CODED){
if (keyCode == DPAD) {
// Do something ..
▪ Java code to capture rich motion events
▪ import android.view.MotionEvent;
134. Motion Event
//Glass Touch Events - reads from touch pad
public boolean dispatchGenericMotionEvent(MotionEvent event) {
float x = event.getX(); // get x/y coords
float y = event.getY();
int action = event.getActionMasked(); // get code for action
switch (action) { // let us know which action code shows up
case MotionEvent.ACTION_DOWN:
touchEvent = "DOWN";
fingerTouch = 1;
break;
case MotionEvent.ACTION_MOVE:
touchEvent = "MOVE";
xpos = myScreenWidth-x*touchPadScaleX;
ypos = y*touchPadScaleY;
break;
136. Sensors
▪ Ketai Library for Processing
▪ https://code.google.com/p/ketai/
▪ Support all phone sensors
▪ GPS, Compass, Light, Camera, etc
▪ Include Ketai Library
▪ import ketai.sensors.*;
▪ KetaiSensor sensor;
137. Using Sensors
▪ Setup in Setup( ) function
▪ sensor = new KetaiSensor(this);
▪ sensor.start();
▪ sensor.list();
▪ Event based sensor reading
void onAccelerometerEvent(…)
{
accelerometer.set(x, y, z);
}
139. Using the Camera
▪ Import camera library
▪ import ketai.camera.*;
▪ KetaiCamera cam;
▪ Setup in Setup( ) function
▪ cam = new KetaiCamera(this, 640, 480, 15);
▪ Draw camera image
void draw() {
//draw the camera image
image(cam, width/2, height/2);
}
142. ● WearScript Features
▪ Community of Developers
▪ Easy development of Glass Applications
▪ GDK card format
▪ Support for all sensor input
▪ Support for advanced features
▪ Augmented Reality
▪ Eye tracking
▪ Arduino input
146. Challenges for the Future (2001)
▪ Privacy
▪ Power use
▪ Networking
▪ Collaboration
▪ Heat dissipation
▪ Interface design
▪ Intellectual tools
▪ Augmented Reality systems
Starner, T. (2001). The challenges of wearable computing: Part 1. IEEE Micro,21(4), 44-52.
Starner, T. (2001). The challenges of wearable computing: Part 2. IEEE Micro,21(4), 54-67.
147. Gesture Interaction With Glass
▪ 3 Gear Systems
▪ Hand tracking
▪ Hand data sent to glass
▪ Wifi networking
▪ Hand joint position
▪ AR application rendering
▪ Vuforia tracking
148. Performance
▪ Full 3d hand model input
▪ 10 - 15 fps tracking, 1 cm fingertip resolution
150. Current Collaboration
▪ First person remote conferencing/hangouts
▪ Limitations
- Single POV, no spatial cues, no annotations, etc
151. Sharing Space: Social Panoramas
▪ Capture and share social spaces in real time
▪ Enable remote people to feel like they’re with you
152. Context Sensing
▪ Using context to manage information
▪ progressive information display as user shows
interest
▪ Context from
▪ Speech
▪ Gaze
▪ Real world
▪ Wearable AR Display
Ajanki, A., Billinghurst, M., Gamper, H., Järvenpää, T., Kandemir, M., Kaski, S., ... & Tossavainen, T.
(2011). An augmented reality interface to contextual information. Virtual reality, 15(2-3), 161-173.
163. ● Conclusions
▪ Wearable computing is a 4th generation of computing devices
▪ A range of wearables will appear in 2014
▪ Ecosystem of devices
▪ There are many existing tools for prototyping
▪ Low fidelity, high fidelity
▪ Significant research opportunities exist
▪ User interaction, displays, social impact
164. More Information
• Mark Billinghurst
– Email: mark.billinghurst@hitlabnz.org
– Twitter: @marknb00
• Daniela Busse
– Email: daniela.busse@gmail.com
• Website
– www.hitlabnz.org