Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Prototyping GNOME UI for Gestural Input


Published on

This slide describes our proposition of concept regarding an OS wide user interface for free gestures input devices such as Leap Motion. This was presented in GNOME Asia 2015 on 9th May 2015. We discusses how we tackle the design issue and our prototype of the concept implemented in GNOME and browser.

Published in: Technology

Prototyping GNOME UI for Gestural Input

  1. 1. PROTOTYPING GNOME UI FOR GESTURAL INPUT Presented by Adityo Pratomo ( ) / Surya University Hidayat Febiansyah / Surya University GNOME.Asia 2015 @kotakmakan (@havban)
  2. 2. ABOUT US We are lecturers at the Informatics Department at Surya University Adityo Pratomo got a Master Degree in Interaction Design from University of Sydney, previously worked at an advertising agency Hidayat Febiansyah an IT wizard... jk went with yellow Jaket, obtained PhD from Sun Moon University, Republic of Korea
  3. 3. USER INPUT DEVICES Devices act as a medium for user to give input to computer
  5. 5. Keyboard
  6. 6. Mouse
  7. 7. Scanner
  8. 8. Graphic Tablet
  9. 9. Touchscreen
  10. 10. Voice Input
  11. 11. Tangible User Interface
  12. 12. Air Gesture Input
  13. 13. TODAY CONDITION Huge variety of input device
  14. 14. (Still) Follows the WIMP (Windows, Icon, Menu, Pointer) style Term coined in 1980 by Merzouga Wilberts Popularized by Apple Macintosh in 1984
  15. 15. Windows Runs a self contained program, isolated from other program, run at the same time in other windows
  16. 16. Icon Shortcut for an action that the computer performs
  17. 17. Menu Text or icon-based selection system that selects and executes programs or tasks
  18. 18. Pointer An onscreen symbol that represents movement of a physical device that the user controls to select icons, objects, data elements, etc
  19. 19. GESTURAL INPUT Provides gestures (mostly fingers) input for users
  20. 20. Touchpad
  21. 21. Implemented in many OS
  22. 22. NEW GESTURAL INPUT DEVICES Introduces free form gestures as means to interact
  23. 23. Microsoft Kinect Full body input, including voice No finger detection
  24. 24. Leap Motion Fingers input only Have built-in finger gestures detection
  25. 25. Intel RealSense Essentially a much smaller Kinect Windows only SDK
  26. 26. Generally speaking, hardware wise, these devices do their job well (can be improved though)
  27. 27. PROBLEMS Come from the software implementation of it Especially, for productive, non-entertainment use
  28. 28. Metro UI Supposed to be designed for touch Falls flat for these kind of interactions
  29. 29. Kinect Guidelines Still involves choosing and confirming action Chooses a big icon Confirms by hanging there for a period of time
  30. 30. Direct Mouse to Leap Motion Barely usable since it involves high accuracy in picking object Lost in translation between 2D (where the feedback happens) and 3D (where the action happens)
  31. 31. OS Wise Information are structured inside tree, explored by moving down branches, presented one node at a time (nothing wrong with this) Interaction wise, requires moving between directories, picking up objects, confirming and working on that (this requires specific interaction technique)
  32. 32. GOALS AND MOTIVATIONS To suggest an interaction model for an OS-wide application of free gestures input We view this interaction model as something that's human and natural, appreciating human's inherent motoric ability To push further research of this area, without waiting for big corp to do it (hooray Open Source)
  33. 33. INFLUENCES AND PREVIOUS WORKS This project takes cues from other projects that people already worked on
  34. 34. GNOME Shell Minimal use of icon, users are forced to memorize keyboard shortcuts
  35. 35. Controlling GNOME with Leap gnome-with-leap/
  36. 36. Leap Motion DBus
  37. 37. 9 to 5 a Leap Motion puzzle game
  38. 38. Kids Note Training System A tool for kids to learn about musical notes and test their hearing
  39. 39. DESIGN CONSIDERATION Applied in Leap Motion Complementary to mouse and keyboard Can be used with only one hand Focus on gesture use with minimal icon involved Main use case for productivity, non-entertainment functionality
  40. 40. ISOLATING THE PROBLEM Bigest problem is telling people how to choose an object using gestures
  41. 41. This is NOT a replacement for mouse Mouse is precise, gestures are not.
  42. 42. Giving a bigger icon doesn't necessarily means it'll be easier to pick It still requires a dexterity to accurately choose the object Plus, how to confirm that choice? Inaccurately pressing the air?
  43. 43. OUR SOLUTION To translate imprecise gestures into an act of choosing an object and confirming it
  44. 44. OUR PROTOTYPE Leap Motion-controlled windowing system UI-model prototype for desktop Both will demonstrate how choosing object is done with gestures
  45. 45. ARCHITECTURE Application OS WIndowing System System Event Leap Motion API OS
  46. 46. FEATURES Alternative live tile windowing system prototype All contents are live and directly editable Activated using gestures Mapping of various gestures into keyboard shortcut Translation of keyboard shortcuts into various events Usage of GNOME windowing capability Keyboard shortcuts receiver inside application
  47. 47. GESTURES CircleGesture() – A circular movement by a finger. SwipeGesture() – A straight line movement by the hand with fingers extended. ScreenTapGesture() – A forward tapping movement by a finger. KeyTapGesture() – A downward tapping movement by a finger.
  48. 48. MAPPING SHORTCUTS Connecting the element using keyboard shortcuts e.g. ctrl+alt+= No need to access lower level component Portable deployment Leap motion control is only complementary input, not primary Other input can be proceded with existing keyboard or mouse
  49. 49. COMPONENTS Leapjs Controller NWJS RobotJs With added functionality to accept multiple flags shortcut Packery.js Other: node-open, jquery, scrollTo, Knobjs, Togglejs, node-gyp
  50. 50. CHARACTERISTICS Open Sourceness Platform Agnostic as long as the device supports HTML5, specifically node- webkit Easy to configure Manage shortcuts Easy to develop Common technology platforms
  51. 51. SYSTEM MODE Running with existing gnome shortcut mode Move among workspaces ctrl+alt+- => Swipe down ctrl+alt+= => Swipe up Move among windows alt+tab => Swipe left alt+shift+tab => Swipe right Inner applications windows movement alt+` => keytap
  52. 52. DEMO MODE Move among widgets one step ctrl+alt+i => Swipe left ctrl+alt+y => Swipe right Move among widgets with rolling ctrl+alt+i * => Rotate finger right ctrl+alt+y * => Rotate finger left Focused widget → Full size widget ctrl+alt+space => keytap
  53. 53. FUTURE GOALS User test Integrate with other technologies (VR, haptic sensors) Possibility into integration with file system that behaves like a graph database Formalize the system
  54. 54. CONCLUSION Air gesture-based interaction will come along in the next few years (or even decades) Software and hardware capabilities are there, they just need to be unified by a proper design system Requires further development to integrate with other technologies and to user test it rigorously