COMP 4026 Lecture 6 on Wearable Computing and methods for rapid prototyping for Google Glass. Taught by Mark Billinghurst from the University of South Australian on September 1st 2016.

1. LECTURE 6: WEARABLE
COMPUTING
COMP 4026 – Advanced HCI
Semester 5 - 2016
Mark Billinghurst
University of South Australia
September 1st 2016

5. Major changes in computing

8. Living Heads Up vs.Heads Down

9. What is a Wearable Computer ?
▪ A computer that is:
▪ Portable while operational
▪ Enables hands-free/hands-limited use
▪ Able to get the user’s attention
▪ Is always on, acting on behalf of the user
▪ Able to sense the user’s current context
Rhodes, B. J. (1997). The wearable remembrance agent: A system for
augmented memory. Personal Technologies, 1(4), 218-224.

10. Wearable Computing
▪ Computer on the body that is:
▪ Always on
▪ Always accessible
▪ Always connected
▪ Other attributes
▪ Augmenting user actions
▪ Aware of user and surroundings

11. The Ideal Wearable
▪ Persists and Provides Constant Access:
Designed for everyday and continuous user over
a lifetime.
▪ Senses and Models Context: Models the users
environment, mental state, it’s own state.
▪ Augments and Mediates: Information support for
the user in both the physical and virtual realities.
▪ Interacts Seamlessly: Adapts its input and output
modalities to those most appropriate at the time.
Starner, T. E. (1999). Wearable computing and contextual awareness
(Doctoral dissertation, Massachusetts Institute of Technology).

12. History of Wearables
▪ 1960-90: Early Exploration
▪ Custom build devices
▪ 1990 - 2000: Academic, Military Research
▪ MIT, CMU, Georgia Tech, EPFL, etc
▪ 1997: ISWC conference starts
▪ 1995 – 2005+: First Commercial Uses
▪ Niche industry applications, Military
▪ 2010 - : Second Wave of Wearables
▪ Consumer applications, Head Worn

13. Thorp and Shannon (1961)
• Wearable timing device for roulette prediction
• Audio feedback, four button input
Ed Thorp

14. KeithTaft (1972)
• Wearable computer for blackjack card counting
• Toe input, LED in Glasses for feedback
Belt computer
Shoe Input
Glasses Display

15. Steve Mann (1980s - )
http://wearcomp.org/

16. MITWearable Computing (1996)

17. Enabling Technologies (1989+)
▪ Private Eye Display (Reflection Technologies)
▪ 720 x 280 dipslay
▪ Red LED
▪ Vibrating mirror
▪ Twiddler (Handykey)
▪ Chording keypad
▪ Mouse emulation

18. MIT Tin Lizzy (1993)
▪ General Purpose Wearable
▪ Doug Platt, Thad Starner
▪ 150 MHz Pentium CPU
▪ 32-64 Mb RAM
▪ 6 Gb hard disk
▪ VGA display
▪ 2 PCMCIA slots
▪ Cellular modem
http://www.media.mit.edu/wearables/lizzy/lizzy/index.html

19. Early Wearable Computing

20. US Military Wearables (1989- )
▪ Early experimentation
▪ 386 computer, VGA display
▪ GPS, mapping software
▪ Land Warrior (1991-)
▪ Integrated wearable system
▪ Camera, colour display, radio
▪ Navigation, reports, photos
Zieniewicz, M. J., Johnson, D. C., Wong, C., & Flatt, J. D. (2002). The evolution of
army wearable computers. IEEE Pervasive Computing, 1(4), 30-40.

21. Wearables at CMU (1991–2000)
▪ Industry focused wearables
▪ Maintenance, repair
▪ Custom designed interface
▪ Dial/button input
▪ Rapid prototyping approach
▪ Industrial designed, ergonomic
http://www.cs.cmu.edu/afs/cs/project/vuman/www/frontpage.html

23. Early Commercial Systems
▪ Xybernaut (1996 - 2007)
▪ Belt worn, HMD, 200 MHz
▪ ViA (1996 – 2001)
▪ Belt worn, Audio Interface
▪ 700 MHz Crusoe
• Symbol (1998 – 2006)
• Wrist worn computer
• Finger scanner

24. Prototype Applications
▪ Remembrance Agent
▪ Rhodes (97)
▪ Augmented Reality
▪ Feiner (97), Thomas (98)
▪ Remote Collaboration
▪ Garner (97), Kraut (96)
• Maintenance
• Feiner (93), Caudell (92)
▪ Factory Work
▪ Thompson (97)

25. Mobile AR: Touring Machine (1997)
▪ University of Columbia
▪ Feiner, MacIntyre, Höllerer, Webster
▪ Combines
▪ See through head mounted display
▪ GPS tracking
▪ Orientation sensor
▪ Backpack PC (custom)
▪ Tablet input
Feiner, S., MacIntyre, B., Höllerer, T., & Webster, A. (1997). A touring machine:
Prototyping 3D mobile augmented reality systems for exploring the urban
environment. Personal Technologies, 1(4), 208-217.

26. MARS View
▪ Virtual tags overlaid on the real world
▪ “Information in place”

27. Backpack/Wearable Systems
1997 Backpack Wearables
▪ Feiner’s Touring Machine
▪ AR Quake (Thomas)
▪ Tinmith (Piekarski)
▪ MCAR (Reitmayr)
▪ Bulky, HMD based
Piekarski, W., & Thomas, B. (2002). ARQuake: the outdoor
augmented reality gaming system. Communications of the
ACM, 45(1), 36-38.

28. 2008: Location Aware Phones
Nokia NavigatorMotorola Droid

29. 2009 - Layar (www.layar.com)
• Location based data
– GPS + compass location
– Map + camera view
• AR Layers on real world
– Customized data
– Audio, 3D, 2D content
• Easy authoring
• Android, iPhone

30. Watches

31. Second Gen. Systems
• Recon (2010 - )
• Head worn displays for sports
• Ski goggle display
• Investment from Intel (2013)
• Google (2011 - )
• Google Glass
• Consumer focus

32. Demo Video
• https://www.youtube.com/watch?v=u24cbjqiVfE

33. Google Glass (2011 - )

35. • Hardware
• CPU TI OMAP 4430 – 1 Ghz
• 16 GB SanDisk Flash,1 GB Ram
• 570mAh Battery
• Input
• 5 mp camera, 720p recording, microphone
• GPS, InvenSense MPU-9150 inertial sensor
• Output
• Bone conducting speaker
• 640x360 micro-projector display

36. ViewThrough Google Glass
Always available peripheral information display
Combining computing, communications and content capture

38. dsfh

39. Smart Watches
• Eg Motorola Moto 360, Apple Watch
• Measure heart rate, movement
• Provide notifications, runs apps, messages, etc

40. Moto 360 Smart Watch
https://www.youtube.com/watch?v=Q0prQniEZTA

41. Activity Monitor
• Fitbit, Microsoft Band
• Measures footsteps, heart rate, calories, sleep

42. Device Ecosystem

43. Usage Matterns

44. Number of Devices Shipped

45. Summary
Wearables are a new class of computing
Intimate, persistent, aware, accessible, connected
Evolution over 50 year history
Backpack to head worn
Custom developed to consumer ready device
Enables new applications
Collaboration, memory, AR, industry, etc
Many head worn wearables are coming
Android based, sensor package, micro-display

46. TECHNOLOGY

47. Wearable Attributes
▪ fafds

48. Some Key Aspects
▪ Display Technologies
▪ Input Devices
▪ Interaction Metaphors
▪ Perceptual Factors
▪ Ergonomics
▪ Cognitive Aspects

49. Types of Head Mounted Displays
Occluded
See-thru
Multiplexed

50. Multiplexed Displays
▪ Above or below line of sight
▪ Strengths
▪ User has unobstructed view of real world
▪ Simple optics/cheap
▪ Weaknesses
▪ Direct information overlay difficult
• Display/camera offset from eyeline
▪ Wide FOV difficult

51. Vuzix M-100
▪ Monocular multiplexed display ($1000)
■ 852 x 480 LCD display, 15 deg. FOV
■ 5 MP camera, HD video
■ GPS, gyro, accelerometer

52. Display Types
▪ Curved Mirror
▪ off-axis projection
▪ curved mirrors in front of eye
▪ high distortion, small eye-box
▪ Waveguide
▪ use internal reflection
▪ unobstructed view of world
▪ large eye-box

53. See-through thin displays
▪ Waveguide techniques for thin see-through displays
▪ Wider FOV, enable AR applications
▪ Social acceptability
Opinvent Ora

54. Waveguide Methods
See: http://optinvent.com/HUD-HMD-benchmark#benchmarkTable
Holographic
Hologram diffracts light
Limited FOV
Colour bleeding
Diffractive
Slanted gratings
Total internal reflection
Costly, small FOV

55. Waveguide Methods
See: http://optinvent.com/HUD-HMD-benchmark#benchmarkTable
Clear-Vu Reflective
Several reflective elements
Thinner light guide
Large FOV, eye-box
Reflective
Simple reflective elements
Lower cost
Size is function of FOV

56. Input Options
▪ Physical Devices
▪ Keyboard
▪ Pointer
▪ Stylus
▪ Natural Input
▪ Speech
▪ Gesture
▪ Other
▪ Physiological

57. Twiddler Input
▪ Chording or multi-tap input
▪ Possible to achieve 40 - 60 wpm after 30+ hours
▪ Chording input about 50% faster than multi-tap
▪ cf 20 wpm on T9, or 60+ wpm for QWERTY
Lyons, K., Starner, T., Plaisted, D., Fusia, J., Lyons, A., Drew, A., & Looney, E.
W. (2004, April). Twiddler typing: One-handed chording text entry for mobile
phones. In Proceedings of the SIGCHI conference on Human factors in
computing systems (pp. 671-678). ACM.

58. Virtual Keyboards
▪ In air text input
▪ Virtual QWERTY keyboard up to 20 wpm
- On real keyboard around 45-60+ wpm
▪ Word Gesture up to 28 wpm
- On tablet/phone Word Gesture up to 47 wpm
▪ Handwriting around 20-30 wpm
A. Markussen, et. al. Vulture: A Mid-Air Word-Gesture Keyboard (CHI 2014)

59. Unobtrusive Input Devices
▪ GestureWrist
▪ Capacitive sensing
▪ Change signal depending on hand shape
Rekimoto, J. (2001). Gesturewrist and gesturepad: Unobtrusive wearable
interaction devices. In Wearable Computers, 2001. Proceedings. Fifth
International Symposium on (pp. 21-27). IEEE.

60. Unobtrusive Input Devices
▪ GesturePad
▪ Capacitive multilayered touchpads
▪ Supports interactive clothing

61. Interaction on the Go
▪ Fitt’s law still applies while interacting on the go
▪ Eg: Tapping while walking reduces speed by > 35%
▪ Increased errors while walking
Lin, M., Goldman, R., Price, K. J., Sears, A., & Jacko, J. (2007). How do people
tap when walking? An empirical investigation of nomadic data entry.International
Journal of Human-Computer Studies, 65(9), 759-769.

62. Where to put Wearables?
▪ Places for unobtrusive wearable technology
Gemperle, F., Kasabach, C., Stivoric, J., Bauer, M., & Martin, R. (1998, October).
Design for wearability. In Wearable Computers, 1998. Digest of Papers. Second
International Symposium on (pp. 116-122). IEEE.

63. Where to Place Trackpad?
▪ User study 25 people different postures
▪ Front of thigh most preferred, torso/upper arm worst
Thomas, Bruce, et al. "Determination of placement of a body-attached mouse
as a pointing input device for wearable computers." 2012 16th International
Symposium on Wearable Computers. IEEE Computer Society, 1999.

64. Where do
users want
Wearables?
29% on clothing
28% on wrist
12% on Glasses

65. RAPID PROTOTYPING
FOR GLASS

66. How do you Design for this?

67. Typical Development Steps
▪ Sketching
▪ Storyboards
▪ UI Mockups
▪ Interaction Flows
▪ Video Prototypes
▪ Interactive Prototypes
▪ Final Native Application
Increased
Fidelity &
Interactivity

68. Sketched Interfaces
▪ Sketch + Powerpoint/Photoshop/Illustrator

69. GlassSim – http://glasssim.com/
▪ Simulate the view through Google Glass
▪ Multiple card templates

70. GlassSim Card Builder
▪ Use HTML for card details
▪ Multiple templates
▪ Change background
▪ Own image
▪ Camera view

71. GlassSim Samples

72. Glass UI Templates
▪ Google Glass Photoshop Templates
▪ http://glass-ui.com/
▪ http://dsky9.com/glassfaq/the-google-glass-psd-template/

73. GlassApplication Storyboard
• http://dsky9.com/glassfaq/google-glass-storyboard-
template-download/

74. ToolKit for Designers
▪ Vectoform Google Glass Toolkit for
Designers
▪ http://blog.vectorform.com/2013/09/16/google-glass-
toolkit-for-designers-2/
▪ Sample cards, app flows, icons, etc

75. Glassware Flow Designer
• Visual design tool for Glass interaction flows
• Collaboratively design apps using common patterns and layouts.
• https://glassware-flow-designer.appspot.com/

76. GlassApplication Flow

77. ▪ Series of still photos in a movie format.
▪ Demonstrates the experience of the product
▪ Discover where concept needs fleshing out.
▪ Communicate experience and interface
▪ You can use whatever tools, from Flash to iMovie.
Video Sketching

78. See https://vine.co/v/bgIaLHIpFTB
Example: Video Sketch of Vine UI

79. Interactive Wireframing
▪ Developing interactive interfaces/wireframes
▪ Transitions, user feedback, interface design
▪ Web based tools
▪ UXpin - http://www.uxpin.com/
▪ proto.io - http://www.proto.io/
▪ Native tools
▪ Justinmind - http://www.justinmind.com/
▪ Axure - http://www.axure.com/

80. UXpin - www.uxpin.com
▪ Web based wireframing tool
▪ Mobile/Desktop applications
▪ Glass templates, run in browser
https://www.youtube.com/watch?v=0XtS5YP8HcM

81. UXpin Demo
http://www.youtube.com/watch?v=0XtS5YP8HcM

82. Viewing on Glass
• Use Android Design Preview
• Push desktop onto Android screen
• https://github.com/romannurik/AndroidDesignPreview

83. Android Design Preview Demo
• https://www.youtube.com/watch?v=WvQrP1szEzg

84. Processing and Glass
▪ One of the easiest ways to build rich
interactive wearable applications
▪ focus on interactivity, not coding
▪ Collects all sensor input
▪ camera, accelerometer, touch
▪ Can build native Android .apk files
▪ Side load onto Glass

85. HelloWorld
//called initially at the start of the Processing sketch!
void setup() {!
size(640, 360);!
background(0);!
} !
!
//called every frame to draw output!
void draw() {!
background(0);!
//draw a white text string showing Hello World!
fill(255);!
text("Hello World", 50, 50);!
}!

86. Demo

87. HelloWorld Image
PImage img; // Create an image variable!
!
void setup() {!
size(640, 360);!
//load the ok glass home screen image!
img = loadImage("okGlass.jpg"); // Load the image into
the program !
}!
!
void draw() {!
// Displays the image at its actual size at point (0,0)!
image(img, 0, 0);!
}!

88. Demo

89. Touch Pad Input
• Tap recognized as DPAD input
!void keyPressed() {!
!if (key == CODED){!
! !if (keyCode == DPAD) {!
!// Do something ..!
• Java code to capture rich motion events
• import android.view.MotionEvent;!

90. Motion Event
//Glass Touch Events - reads from touch pad!
public boolean dispatchGenericMotionEvent(MotionEvent event) {!
float x = event.getX(); // get x/y coords !
float y = event.getY();!
int action = event.getActionMasked(); // get code for action!
!
switch (action) { // let us know which action code shows up!
!case MotionEvent.ACTION_DOWN:!
! !touchEvent = "DOWN";!
! !fingerTouch = 1;!
!break; !
!case MotionEvent.ACTION_MOVE:!
! !touchEvent = "MOVE";!
! !xpos = myScreenWidth-x*touchPadScaleX;!
! !ypos = y*touchPadScaleY;!
!break;!

91. Demo

92. Sensors
• Ketai Library for Processing
• https://code.google.com/p/ketai/
• Support all phone sensors
• GPS, Compass, Light, Camera, etc
• Include Ketai Library
• import ketai.sensors.*;!
• KetaiSensor sensor;!

93. Using Sensors
• Setup in Setup( ) function
• sensor = new KetaiSensor(this);!
• sensor.start();!
• sensor.list();
• Event based sensor reading
void onAccelerometerEvent(…)!
{!
accelerometer.set(x, y, z);!
}!

94. Sensor Demo

95. Sensors
▪ Ketai Library for Processing
▪ https://code.google.com/p/ketai/
▪ Support all phone sensors
▪ GPS, Compass, Light, Camera, etc
▪ Include Ketai Library
▪ import ketai.sensors.*;
▪ KetaiSensor sensor;

96. Using the Camera
▪ Import camera library
▪ import ketai.camera.*;
▪ KetaiCamera cam;
▪ Setup in Setup( ) function
▪ cam = new KetaiCamera(this, 640, 480, 15);
▪ Draw camera image
void draw() {
//draw the camera image
image(cam, width/2, height/2);
}

97. Camera Demo

98. RESOURCES

99. Online Wearables Exhibit
Online at http://wcc.gatech.edu/exhibition

100. Glass Resources
▪ Main Developer Website
▪ https://developers.google.com/glass/
▪ Glass Apps Developer Site
▪ http://glass-apps.org/glass-developer
▪ Google Design Guidelines Site
▪ https://developers.google.com/glass/design/
index?utm_source=tuicool
▪ Google Glass Emulator
▪ http://glass-apps.org/google-glass-emulator

101. Other Resources
▪ AR for Glass Website
▪ http://www.arforglass.org/
▪ Vandrico Database of wearable devices
▪ http://vandrico.com/database

102. Books
▪ Programming Google Glass
▪ Eric Redmond
▪ Rapid Android
Development: Build Rich,
Sensor-Based Applications
with Processing
▪ Daniel Sauter

103. www.empathiccomputing.org
@marknb00
mark.billinghurst@unisa.edu.au