Successfully reported this slideshow.

Multi-Touch technologies: Software, tracking and functions


Published on

Lecture series. Topic covers: Introduction to software programming, Gesture interaction, MT engines and applications

Published in: Education, Technology
  • Be the first to comment

Multi-Touch technologies: Software, tracking and functions

  1. 1. Lecture 3 : Multi-touch technologies: Software, tracking and functions. By Jazmi Jamal
  2. 2. Twitter <ul><li>Have you twit today? </li></ul><ul><li>Twitter : toaBmd </li></ul>
  3. 3. Outline <ul><li>Introduction to software programming for multi-touch </li></ul><ul><li>Gesture interaction </li></ul><ul><li>MT engines or applications </li></ul>
  4. 4. Re-cap: Multi-touch hardware
  5. 5. Re-cap: Multi-touch hardware
  6. 6. Software Programming MT <ul><li>Same programming with other language. </li></ul><ul><li>Certain protocols, methods and standards in the MT programming. </li></ul><ul><li>Read & translate raw blob as input, relay this info through pre-defined protocols and interact with an application. </li></ul>programming language Eg: AS3 Python C C++ Java
  7. 7. Tracking <ul><li>Tracking re-identify object in series of video frames. </li></ul><ul><li>Background-subtraction technique. </li></ul><ul><li>Foreground identify as centroids (points). Tracking algorithm predicts the centroids movement. </li></ul>
  8. 8. Computer Vision <ul><li>Touch pattern detection </li></ul><ul><li>Pattern detection for tangible objects with orientation and position </li></ul><ul><li>Calibration between camera and projector </li></ul><ul><li>Background and noise subtraction </li></ul>
  9. 9. CV framework outline
  10. 10. … Gesture Interaction
  11. 11. Gesture Interaction <ul><li>“ Communication through gestures has been one of the oldest form of interaction in human civilization owing to various psychological reasons.” </li></ul><ul><li>“ Thus a gesture based interface is a perfect candidate for social and collaborative tasks as well for applications involving an artistic touch . The interface is physical, more visible and with direct manipulation.” </li></ul>
  12. 12. Gesture Interaction <ul><li>Goal of gesture interaction: higher immersion with easy and familiar movement. </li></ul><ul><li>Eg: scaling, rotating and translating images with 2 fingers. (Natural gestures) </li></ul>
  13. 13. Gesture Interaction (Apple MT)
  14. 14. Gesture Interaction (Activity) <ul><li>My MT gesture </li></ul><ul><li>Establish a group of 3, create and discuss 2 inventive MT gestures. Some criterions must be applied </li></ul>Criterions What is it? How does it function? Where does this apply ? Why? Why you prefer that gesture ?
  15. 15. Save (example) <ul><li>What is it? </li></ul><ul><li>Shortcut button to save file. (In PC Ctrl+S, MAC Cmd+S) </li></ul>
  16. 16. Save (example) <ul><li>How does it function? </li></ul><ul><li>Narrow finger 1,2,3,4. (like grasping object movement) </li></ul><ul><li>(1: thumb, 2: index finger, 3: middle finger, 4: ring finger, and 5: pinky) </li></ul>
  17. 17. Save (example) <ul><li>Where does this apply ? </li></ul><ul><li>Applicable only on table-top multi-touch. </li></ul><ul><li>Why? </li></ul><ul><li>The gesture activity take space. Not suitable for small devices with space constraint. </li></ul>
  18. 18. Save (example) <ul><li>Why you prefer that gesture ? </li></ul><ul><li>The movement of grasp shows to own or hold something. </li></ul><ul><li>The movement suitable for save function. </li></ul>
  19. 19. 35 minutes ~ MT Application
  20. 20. MT apps & engines <ul><li>Ready-made multi-touch engines / software to control the camera, process, blob detection and communication with other client applications. </li></ul>
  21. 21. MT apps & engines <ul><li>Several open sources MT engines. </li></ul>Criterions Touchlib CCV (Community Core Vision) ReacTIVision OpenTouch Touche and etc…
  22. 22. CCV <ul><li>CCV or tbeta is an open source/cross-platform solution for computer vision and machine sensing. </li></ul><ul><li>Suitable for optical techniques approach. </li></ul><ul><li>Developed by NUI Group Community. </li></ul><ul><li>Platform: Mac, Windows, Linux </li></ul><ul><li> </li></ul>
  23. 23. CCV
  24. 24. CCV (configuration) <ul><li>You may Download the tbeta. & launch it. </li></ul><ul><li>Click Next/Previous camera to choose desirable input. </li></ul><ul><li>Then, press to background remove background to eliminate bg. Dynamic substract when the lighting is often changes </li></ul>
  25. 25. CCV (configuration) <ul><li>For DI users only: </li></ul><ul><li>Turn off the smooth and amplify filters </li></ul><ul><li>Turn on Highpass filter </li></ul><ul><li>Adjust highpass blur and noise sliders until see fingers are clear </li></ul><ul><li>If blobs are weak, turn on the amplify filter to brighten it. </li></ul>
  26. 26. CCV (configuration) <ul><li>Adjust threshold until only fingers are tracked and no false blobs </li></ul><ul><li>Press Save settings </li></ul><ul><li>Continue to Calibration guide to perform calibration </li></ul>
  27. 27. CCV (calibration) <ul><li>Calibrating allows touch points to line up with elements on screen. </li></ul><ul><li> </li></ul>
  28. 28. CCV (running demos) <ul><li>Download FLASH demos specifically for the tbeta </li></ul><ul><li>In order to run demo swf files, security settings first need to be set. Go to Flash Global Security Settings . Click on Edit Locations... and then Add Location. </li></ul><ul><li>Browse the demo folder and click confirm </li></ul><ul><li>Launch the flash demo (.swf) </li></ul>
  29. 29. CCV (overview) <ul><li>Configure blob tracking </li></ul><ul><li>Calibrate with screen </li></ul><ul><li>Send signal to XML Flash </li></ul><ul><li>Allow global security settings in Flash </li></ul><ul><li>Run Flash demo. </li></ul>
  30. 30. Try this out (MiniMT)
  31. 31. Need a break ? Group activity
  32. 32. Group activity <ul><li>Please twit and share 2 links regarding setup or process between CCV and Flash communication. </li></ul><ul><li>Don’t forget to add this guy: toaBmd </li></ul><ul><li>Each group require to present and discuss the link that have been twitted. </li></ul>
  33. 33. 35 minutes
  34. 34. Finish! Q&A session