Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Hollywood has a lot to answer for uxpa2015

989 views

Published on

In science fiction and action films, gestural interfaces are everywhere, and new gestural input technologies generate a lot of anticipation with their kickstarter videos. And yet in the real world, gestural input (with the exception of multitouch) has gained little traction. Why is that?

We've been working with a variety of gestural technologies, trying to incorporate them into professional products, and we have learned where the problems are, and what we need to do to cross the divide between hype and practice.

This presentation looks at the current state of gestural input technologies, analyses the strengths and failings of each, and charts a course to using them successfully. We present a case study showing how and why gestures need to be curated across different form-factors, and give some tips on how to test.

Published in: Design
  • Be the first to comment

  • Be the first to like this

Hollywood has a lot to answer for uxpa2015

  1. 1. © 2014 Autodesk Hollywood Has a Lot to Answer For: Making Gestural Input Work for the Real World Ian Hooper Principal UX Designer UXPA 2015 John Schrag UX Architect, Media & Entertainment Division
  2. 2. © 2014 Autodesk
  3. 3. © 2014 Autodesk Part One: A Brief History of Gestural Interaction
  4. 4. © 2014 Autodesk What do we mean by gestures? A gesture is a motion of the body that contains information. Waving goodbye is a gesture. Pressing a key on a keyboard is not a gesture because the motion of a finger on its way to hitting a key is neither observed nor significant. All that matters is which key was pressed. “ ” Kurtenbach and Hulteen (1990):
  5. 5. © 2014 Autodesk §  Human to Human Interaction: §  Music Conducting §  Dance, Theatre & Magic §  Sign Language & informal salutations and signaling §  Early Electronic Systems §  Musical instruments (Theremin, Sackbut) §  Sensors (The Clapper, Photodetectors) Historical Context
  6. 6. © 2014 Autodesk A long road from theory to practice Direct Manipulation of Graphical Objects 1990197019601950 1980 The Mouse Windows Research Commercialization 2000 2010 Gesture Recognition derived from: Brad A. Myers (1998). A brief history of human-computer interaction technology. Interactions, vol 5(2), pp. 44-54
  7. 7. © 2014 Autodesk 1981: Xerox Star 8010 §  Original Price: $16,595 §  Professional workstation was the inspiration for the Macintosh and all Windows / Icons / Mouse / Pointer (WIMP) interfaces Image: digibarn.com
  8. 8. © 2014 Autodesk 1983: HP-150 §  Original Price: $2795 §  Early personal computer had infrared touch-screen capability Image: columbia.edu
  9. 9. © 2014 Autodesk 1984: Apple Macintosh §  Original Price: $2,495 §  Popularized WIMP §  Introduced the ‘drag’ gesture in place of a dedicated mouse button Image: oldcomputers.net/
  10. 10. © 2014 Autodesk §  Games Outside the mainstream Dataglove (Fifth Dimension Technologies) §  Niche Phantom Omni (SensAble Technologies) Xbox Kinect (Microsoft) Playstation Move (Sony) Wii Remote (Nintendo)
  11. 11. © 2014 Autodesk 1998: Dance Dance Revolution (Konami) 2001: Black & White (Lionhead Studios) 2003: Eye Toy (Sony) §  Games Outside the mainstream
  12. 12. © 2014 Autodesk §  Art Outside the mainstream 1985: Video Place (Myron Krueger) Projector Camera Projector Screen Back-lit screen 2000: Body Language (Nathaniel Stern)
  13. 13. © 2014 Autodesk The Renaissance Image: Apple §  2007: iPhone §  2010: iPad §  2012: Cintiq Image: Wacom
  14. 14. © 2014 Autodesk §  The four best-known players in mid-air gestural input devices right now are: §  Built-in cameras §  Kinect camera §  LEAP Motion device §  Thalmic Labs’ Myo device A Look at the Hardware
  15. 15. © 2014 Autodesk Built-in Camera §  Cheap and ubiquitous §  Open-source libraries for extracting gesture §  Slower (image processing in software) §  Bad depth sense §  Error prone Image: Dell
  16. 16. © 2014 Autodesk Kinect Camera §  Reads full body or upper body and creates a 3d skeleton §  Close up: includes gaze direction, head position, fingers §  Further back: no fingers Image: Microsoft
  17. 17. © 2014 Autodesk LEAP Motion Device §  Provides full hand skeletal model §  Has a limited volume – hard to see its limits §  Occlusion is a problem Image: LEAP
  18. 18. © 2014 Autodesk Thalmic Labs Myo §  Senses hand pose using myoelectrics §  Tracks rotation & accelleration §  No occlusion problem §  Usable anywhere §  Only handles 5 poses §  Requires training Image: Thalmic Labs
  19. 19. © 2014 Autodesk Case Study: Curating Multi-touch Interaction Across Form-Factors
  20. 20. © 2014 Autodesk §  Started in January 2010. §  Wanted to create a common, good standard for using multi-touch for 3d Navigation. §  Wanted to avoid everyone doing it differently The History
  21. 21. © 2014 Autodesk §  The interactions should be easy to learn and retain. §  They should fit in naturally with existing standard 2d interactions. §  They should be suitable for common and key multitouch hardware setups. §  Blah blah other stuff Our research goals
  22. 22. © 2014 Autodesk First Test (July 2010) §  Mac Laptop running Mudbox §  Wacom Intuos 3 tablet §  iPhone providing a multitouch surface §  Also some “soft buttons” on the phone
  23. 23. © 2014 Autodesk The Gestures
  24. 24. © 2014 Autodesk §  Navigation Gestures intuitable? - almost §  Learnable? – Yes! §  Comfortable? – Hard to gesture on that tiny surface §  Soft buttons? – Bad idea; users are looking at the screen, and must shift focus to use buttons. This breaks their flow. First Test Results
  25. 25. © 2014 Autodesk Second Test (Nov 2010) §  Introducing the iPad! §  Tested gesture retention §  Compared navigation speed to mouse+hotkey
  26. 26. © 2014 Autodesk §  Gestures learnable? – Yes! §  Retained after 7-14 days? – Yes! §  Comfortable? – Yes, on the iPad §  Preferred? – Yes… except for fixable technical problems (lag, etc) §  Faster? – Yes Second Test Results
  27. 27. © 2014 Autodesk Third Test (April 2012) §  Goal was to try the new engine with many different kinds of people §  Try it on new form factor – 27” PPI multi-touch monitor with stylus §  First test with on-screen touch
  28. 28. © 2014 Autodesk §  Experience was surprisingly consistent for people of all sizes/shapes/genders/etc. §  Users kept trying to touch interface buttons, even when they knew it didn’t work. §  Camera tilt and zoom interactions felt wrong, unlike in earlier tests – Huh? Third test results
  29. 29. © 2014 Autodesk This feels right
  30. 30. © 2014 Autodesk Same interaction on a touch screen feels wrong
  31. 31. © 2014 Autodesk Obvious finger/screen mapping
  32. 32. © 2014 Autodesk §  Try separate behaviours for touchscreens and touch- tablets. §  Used 27” PPI touch/stylus monitor, vs large monitor with Wacom Intuos 5 tablet. §  Test new 3, 4, and 5-finger gestures (Mudbox specific). Fourth Test (July 2012)
  33. 33. © 2014 Autodesk §  Users perceived the different zoom/roll behaviours on different devices to be the same §  4 and 5 finger gestures were awkward on the small tablet, but fine on the big screen. Fourth Test Results
  34. 34. © 2014 Autodesk §  How do you map tablet touches to multiple 3d views? Related Considerations
  35. 35. © 2014 Autodesk §  Just another non-display multitouch tablet like the Intuos, so we used the same behaviour. §  WRONG. Adding Magic Trackpad Support Image: Apple
  36. 36. © 2014 Autodesk §  Even though this is not a screen device, it controls a persistent pointer on the screen. §  The entire surface is mounted over a big button, so that you can tap it (lightly) or click it (harder) like a mouse. How is the Magic Trackpad different?
  37. 37. © 2014 Autodesk Magic Mouse §  All the problems of the magic trackpad, and more! §  Tiny touch area §  Cursor moved by touch OR by moving mouse §  Frequent false touch detection
  38. 38. © 2014 Autodesk §  Testing with users is critical. §  Obvious device classification is not enough; dig for the salient differences. §  Sometimes you have to do things differently to make things feel the same across form-factors. Conclusions
  39. 39. © 2014 Autodesk Part Three: What’s Wrong with Mid-Air Gestures?
  40. 40. © 2014 Autodesk Mid-Air Gesture Control is Really Cool §  It’s been around for ages §  It’s completely natural §  The demos look awesome §  Why don’t we use it everywhere? Image: Minority Report (2002) Twentieth Centurey Fox
  41. 41. © 2014 Autodesk §  Cost of the equipment? §  Availability of hardware? §  To hard to install? §  Too computationally intensive? §  No killer application? §  The user experience sucks? Why we don’t use mid-air gestures everywhere
  42. 42. © 2014 Autodesk §  Cost of the equipment? §  Availability of hardware? §  To hard to install? §  Too computationally intensive? §  No killer application? §  The user experience sucks? Why we don’t use mid-air gestures everywhere
  43. 43. © 2014 Autodesk §  It looks awesome in demos. §  BUT in practice, it’s often: §  Disappointing §  Hard to control §  No better (and often worse) than existing interactions Primary Reason
  44. 44. © 2014 Autodesk §  Awesome YouTube Demo! Google Earth + Leap Motion
  45. 45. © 2014 Autodesk “A lot of potential, but it needs much more development” “It’s much more intuitive than clicking with a mouse… but it’s not actually doing what I want.” Maya + Leap Motion Image: Digital Arts Online
  46. 46. © 2014 Autodesk §  Gorilla arm §  Gesture delineation & intentionality §  No tactile feedback §  Context and mapping §  Internationalization Why Is Mid-Air Gestural Interaction Hard?
  47. 47. © 2014 Autodesk Gorilla Arm Photo by Richard Ruggiero, public domain §  Holding your arms up to interact is quickly tiring. Diagram from Austin, Gilbert (1806): Chiromania; or a Treatise on Rhetorical Delivery Seriously?
  48. 48. © 2014 Autodesk Gesture Delineation & Intentionality Photo by: Biswarup Ganguly, used under Creative Commons license (CC)
  49. 49. © 2014 Autodesk Context and Mapping Image: Bolt, Richard A. “Put That There”: Voice and Gesture at the graphics interface. SIGGRAPH 1980.
  50. 50. © 2014 Autodesk No tactile feedback Image: Syntact ultrasonic mid-air haptic feedback device by Ultrasonic-Audio.com
  51. 51. © 2014 Autodesk Internationalization Frames from the short film Hand Gestures, by Jeff Werner, Vancouver, Canada
  52. 52. © 2014 Autodesk Doomed? No.
  53. 53. © 2014 Autodesk How do we know?
  54. 54. © 2014 Autodesk 1. Because Kinect Games are pretty Awesome Frame from Kinect Star Wars
  55. 55. © 2014 Autodesk 2. Because music conductors don’t suffer from Gorilla Arm Photo courtesy of John Schrag
  56. 56. © 2014 Autodesk “Consumed Endurance” – metric for Gorilla arm Diagram from Consumed Endurance: A Metric to Quantify Arm Fatigue of Mid-Air Interactions. Juan David Hincapié-Ramos, Xiang Guo, Paymahn Moghadasian, Pourang Irani,University of Manitoba
  57. 57. © 2014 Autodesk §  We do it with voice, eye contact, facial expression, repetition. §  This is an area of active research, both in HCI and linguistics (ASL) 3. Because we can all delineate gestures
  58. 58. © 2014 Autodesk §  The question isn’t: “is mid-air gestural interaction good” §  The question is: “What is it good for?” 4. Because we understand this:
  59. 59. © 2014 Autodesk §  Is there a killer app for mid-air gestural input? (aside from Games) The Killer App?
  60. 60. © 2014 Autodesk The Killer App: VR and AR Image: technologyreview.com Video: ign.com
  61. 61. © 2014 Autodesk Case Study: Mid-Air Gestures with the Myo Device
  62. 62. © 2014 Autodesk §  Test the suitability of the Myo device for VR/AR navigation applications. §  Come up with useful design guidelines and mental models to help us design better gestural interactions with this class of device §  Improve our own shared gesture-interpretation library for other Autodesk developers. Our research goals
  63. 63. © 2014 Autodesk §  “Gorilla Arm”… no use of upper arm §  To delineate… use hand poses §  Occlusion… not a problem with this device §  Context… not absolute mapping needed §  Tactile feedback… not an issue for this task Designing the Gestural Interaction
  64. 64. © 2014 Autodesk First Test §  PC running Stingray game engine §  Thalmic Labs Myo arm band §  Calibrated Myo §  Users received training
  65. 65. © 2014 Autodesk §  Continuous pose detection is problematic on the Myo §  Holding poses continuously strains the hand §  Non-mnemonic gestures were poorly recalled §  Instinctive movement often overrode training §  Gesture success depends on personal physiology First Test Results
  66. 66. © 2014 Autodesk Second Test (informal) §  Changed steering to more natural mapping §  Removed need to hold hand poses §  Added speed control §  Various small fixes
  67. 67. © 2014 Autodesk §  Gestures are deep-rooted: adapt to your user’s movement patterns where possible. §  User testing is still crucial – test with different physiologies. §  Remember to measure physical comfort when testing – people may not mention it unless asked. §  Simplify your problem domain if necessary §  Don’t give up right away Conclusions
  68. 68. © 2014 Autodesk The Real World is Not Magic Video: https://www.youtube.com/channel/UC0bM0PxqixYjYY7ypt5mqGw §  Gestural control can seem like magic… IF it is designed well §  When you want to get something done… cool is irrelevant
  69. 69. © 2014 Autodesk John Schrag @jvschrag Ian Hooper +IanHooperUX
  70. 70. Autodesk is a registered trademark of Autodesk, Inc., and/or its subsidiaries and/or affiliates in the USA and/or other countries. All other brand names, product names, or trademarks belong to their respective holders. Autodesk reserves the right to alter product and services offerings, and specifications and pricing at any time without notice, and is not responsible for typographical or graphical errors that may appear in this document. © 2014 Autodesk. All rights reserved.

×