• Like
Human Interaction Library
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Human Interaction Library


This report presents the design and the architecture of a java library that allows the final user to interact with applications through non conventional devices, like the Nintendo Wii® Controller …

This report presents the design and the architecture of a java library that allows the final user to interact with applications through non conventional devices, like the Nintendo Wii® Controller (WiiMote), the microphone and the webcam. It has been developed as a java API, that can be easily integrated inside any kind of software that is suitable to be controlled by devices different from a keyboard or a mouse. In particular in this report we will show how it has been used inside
WorldWind sdk, the open source NASA world exploring tool.

Published in Education , Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads


Total Views
On SlideShare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. Human Interaction Library In partial fulfillments of the requirements for the course of laboratory of computer graphics 2008 Magliocchetti Daniele - 125162
  • 2. Objectives
    • Build a new java library that allows the user to interact with applications through non conventional devices, like:
      • Nintendo Wii® Controller (WiiMote)
      • Microphone
      • Web Cam
      • Build the library as an API that can be easily reused
      • Integrate the library inside the WorldWind SDK
  • 3. Capabilities Description: Finger Tracking
      • Gesture recognition with the WiiMote IR camera and a set of IR emitters or reflective markers. The system should distinguish between two modes:
      • Navigation Mode (up to 2 fingers) :
        • Pan Action (1 finger) : default DRAG operation
        • Zoom Action (2 fingers) : ZOOM IN ZOOM OUT
        • Rotation Action (2 fingers) : CLOCKWISE COUNTERCLOCKWISE
        • Look Action (2 fingers) : LOOK UP LOOK DOWN
      • Edit Mode (3 or more fingers) :
        • Move Cursor (1 or 3 fingers) :
        • Single Click (1 or 3 fingers) :
        • Double Click (1 or 3 fingers) :
        • Drag Action (1 or 3 fingers) :
        • Right Click (1 or 3 fingers) :
  • 4.
    • Speech Recognition of a small set of navigation and editing commands through the system microphone:
    Speech Recognition Head Tracking Head Tracking with a webcam that allow the user to modify the perspective of the application, like it is watching through a window: Command Description Command Description Left Moves to the West direction Stop Stop all the movements Right Moves to the East direction Edit Mode Switch to the edit mode Look Up Moves to the North direction Navigation Mode Switch to the navigation mode Look Down Moves to the South direction Single Click Notifies a single left click to WWJ Turn Right Rotates the globe clockwise Double Click Notifies a double left click to WWJ Turn Left Rotates the globe counterclockwise Right Click Notifies a single right click to WWJ Zoom In Zoom in Mouse Drag Notifies a drag to WWJ Zoom Out Zoom out Command Description Head Right Moves the globe to the East direction Head Left Moves the globe to the West direction Head Up Moves the globe to the North direction Head Down Moves the globe to the South direction Head Near the Screen Zoom in Head Far from the Screen Zoom out
  • 5. Use Case User
  • 6. Requirements Classification
  • 7. System Architecture
    • Based on the following patterns:
      • Singleton:
      • ensures that a class has only one instance and provides a global point of access to it
      • Event-Listener:
      • the system calls all the defined external listeners each time an event occurs. Allows a complete separation from third party applications, the ideal solution for an API.
      • Organized in the following packages:
    • humanInteraction : with the main class of the system
    • humanInteraction.connection : WiiMote and network connection classes
    • humanInteraction.core.events : the event wrapper classes notified when an event occurs
    • humanInteraction.core.listeners.ext : the listener interfaces that should be implemented by the third party application to handle the incoming events from the system
    • humanInteraction.core.listeners.mote : the listener classes to handle the MoteJ library
    • humanInteraction.core.queue : a set of utility classes for the concurrency
    • humanInteraction.core.fingerTracking : the classes for the gesture recognition process
    • humanInteraction.core.headTracking : the classes for the head tracking process
    • humanInteraction.core.speechRecognition : the classes for the speech recognition process
  • 8. Bluetooth Dongle Nintendo Wii ® Device ConnectionManager HumanInteraction Manager HumanInteraction Library Human Interaction Library Architecture Sphinx4 java Library Microphone Device Speech Recognition Library OpenCV Library Webcam Device Face Recognition Library Face Recognition Client Class / Component Thread Library Set Device Reference Queue Connection Motej Library BlueCove Bluetooth Stack Wii Connection Libraries Socket Event Notification
  • 9. WorldWind Integration
    • Extension of AWTInputHandler and OrbitViewInputBroker into the following two classes:
      • WiiAWTInputHandler : implements all the event listeners of the human interaction library in addition to the mouse and key event listeners;
      • WiiOrbitViewInputBroker : extended with some concurrency workarounds
      • Each library event is mapped into WorldWind mouse or key events
      • Addition two listeners classes and a layer to the WorldWind SDK ( WiiMonitorLayer ) to show on the screen the position of the fingers and the detected voice commands.
      • Changes into the WWJ configuration file to load the layer and the WiiAWTInputHandler class
      • Changes to the CGFrame class to load the library
  • 10. Bugs and limitations
    • WiiMote discovery only short after the application start and if they are in connection mode (button 1 and 2 pressed)
    • The library can handle only one WiiMote at a time
    • While dragging, sometimes the sdk generates some inconsistencies that lead to an incorrect positioning of the globe (flipping and turning). We were not able to solve this problem, neither to detect it, even after a discussion on the WorldWind forum with the developers of the library.
    • The Speech Recognition accuracy depends from the quality of the microphone and the noise of the environment. Each machine, microrphone or environment change, requires the reconfiguration of the hi.config.xml file;
    • As a general rule for a good speech recognition, it’s preferable to speak slowly especially with multiwords (like “ turn left ”), and to pronounce the command “ stop ” with a long “ o” .
    • The library does not recognize combinations of buttons
    • The visibility of some WorldWind classes has been changed from protected to public
    • The head tracking client sets always the camera to the maximal resolution. With webcams with a resolution greater than 640x480 pixel and slow cpus, the detection can be really slow. In this case, it’s preferable to set higher values for cvSizeHorizontal and cvSizeVertical inside the htSettings.cfg file.
    • If there are multiple webcams, only the first installed will be used
  • 11. References
    • http://motej.sourceforge.net : The WiiMote java api official site
    • http://bluecove.sourceforge.net/ : The java bluetooth stack official site
    • http://www.wiili.org/index.php/Wiimote : A WiiMote hardware description
    • http://www.youtube.com/watch?v=0awjPUkBXOU : “ Tracking Your Fingers with the Wiimote” by Johnny Chung Lee
    • http://sourceforge.net/projects/opencv/ : The OpenCV official site
    • http://www.youtube.com/watch?v=Jd3-eiid-Uw : “Head tracking with the Wiimote” by Johnny Chung Lee
    • http://cmusphinx.sourceforge.net/sphinx4/ : The Sphinx official site: “A speech recognizer written entirely in the Java TM  programming language”
    • http://worldwind.arc.nasa.gov/java/index.html : The WorldWind official site
    • WWJ : WorldWind Java SDK
    • BT : Bluetooth
    • IR : Infrared
    • JNI : The Java Native Interface
    • BNF : Backus-Naur Form