Your SlideShare is downloading. ×
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Gesture Based Interaction
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Gesture Based Interaction

11,752

Published on

An overview of gesture sensing technologies, open source applications and how to build your own Tangible User Interface with Reactivision & Flash

An overview of gesture sensing technologies, open source applications and how to build your own Tangible User Interface with Reactivision & Flash

4 Comments
40 Likes
Statistics
Notes
  • i too want this ppt plz mail this to jeevajohn555@gmail.com
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • i too want this ppt plz send this... jeevajohn555@gmail.com
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • hey dude i requrid this ppt pls send me at my mail id phonevsmobile@gmail.com
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • :)
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
11,752
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
0
Comments
4
Likes
40
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Gesture-Based Interaction Design Lane@LaneKuhlman.com
  • 2. Gesture-Based Interaction Design Overview  Overview State of the Art State of the Industry  Overview gesture sensing technologies currently in the market Gesture Open Source  Possibilities and diIemmas within the industry Usability  Elements of gestural communication Demo Great Literature  Free applications and advice for building TUIs and touch tables  Usability study comparing single-user to multi-user interaction  Demonstration a DIY TUI Using Flash, FLOSC & Reactivition  Some great sources for more in depth information
  • 3. Gesture-Based Interaction Design Touch Surfaces -Video Based Overview  State of the Art Microsoft Surface State of the Industry Gesture • Runs on Vista OS Open Source • Develop using .Net Usability • Bluetooth Capabilities Demo • $15,000 for table Great Literature • $15,000 for development kit
  • 4. Gesture-Based Interaction Design Touch Surfaces -Video Based Overview  State of the Art Ion Touchlight State of the Industry Gesture • Tranlucent projection surface Open Source • Uses 3 Cameras & includes calibration Usability • Can copy an image held on the surface Demo • Vertical Display Great Literature • Forerunner to Surface
  • 5. Gesture-Based Interaction Design Touch Surfaces -Video Based Overview  State of the Art Perceptive Pixel State of the Industry Gesture • FTIR - Internal Reflection Open Source • Jeff Han - perceptivepixel.com Usability • Pressure sensing Demo • Patents Pending Great Literature • Not disclosing price
  • 6. Gesture-Based Interaction Design Touch Surfaces -Video Based? Overview  State of the Art Multitouch, Finland - Cellular State of the Industry Gesture • Expandable Open Source • Not disclosing price Usability • 60 FPS Demo • Differentiates Multiple Great Literature Fingers & Hands
  • 7. Gesture-Based Interaction Design Touch Surfaces -Capacitive Overview  State of the Art I-Phone State of the Industry Gesture • Use the capacitance of skin - won't work with gloves Open Source • An accelerometer enables flipping from portrait to landscape view Usability • An ambient light sensor adjust screen brightness automatically Demo • Infrared sensor disables touch screen when near the head Great Literature
  • 8. Gesture-Based Interaction Design IR Camera & Accelerometers Overview  State of the Art Wii State of the Industry Gesture • Low Cost Accelerometer Open Source • Less expensive processor than competitors Usability • Small Infrared Camera Demo • Encouraging new groups to start gaming Great Literature • Uses pantomime gestures with baton for things like sword fighting, archery, and other gestures
  • 9. Gesture-Based Interaction Design Accelerometers Overview  State of the Art Think/Thing OUiP! State of the Industry Gesture • Baby friendly motion sensing Open Source • Visualize your daily jog Usability Demo Great Literature
  • 10. Gesture-Based Interaction Design High Res. Mocap Overview  State of the Art Polhemus FaskTrack State of the Industry Gesture • Electromagnetic Open Source • 6 Degrees of Freedom Usability • Small senors provide high resolution tracking Demo • No Occlusion Great Literature Vicon Motion Systems - Cameras • Can expand to include tons of cameras • High Resolution • Occlusion can occur, but is less likey with more cameras
  • 11. Gesture-Based Interaction Design Data Gloves Overview  State of the Art 4D Dataglove State of the Industry Gesture Gestural Character Control Open Source This work is investigating how gestures can Usability be used to build and conduct virtual stories. Demo It is comprised of two simulations. In the first, Great Literature you can direct a character using programmed gesture signals, some of which are loosely based on modified American Sign Language. Using hand gestures you can tell Eva, a 3-D Virtual Character, when to walk, run, sit, stand, jump and open doors.The second simulation involves an interface that allows you to use the glove to navigate through the joint hierarchy of a virtual mannequin and manipulate its posture by selecting and reorienting it’s joints. The project was displayed on Good Day Columbus in May
  • 12. Gesture-Based Interaction Design RFID Technology Overview  State of the Art Digital Seeds State of the Industry Gesture Open Source Usability Demo Great Literature
  • 13. Gesture-Based Interaction Design TUI Table Overview  State of the Art Geometry Drawing Table State of the Industry Gesture Open Source Usability Demo Great Literature
  • 14. Gesture-Based Interaction Design State of the Industry Overview State of the Art  State of the Industry Difficulties Gesture Open Source Over pricing limits the audience and Usability the amount of people who can Demo develop for new platforms Great Literature Companies are quot;over protectivequot;, which slows development and innovation The quot;WOWquot; factors gets in the way of the quot;WHYquot; While most of these technologies are facinating, sometimes the tech- nologies fail to demonstrate the ability to fulfill a need that cheaper technologies cannot
  • 15. Gesture-Based Interaction Design State of the Industry Overview State of the Art  State of the Industry Patent Frenzy Gesture Open Source quot;Multitouch technology itself Usability is far from being an Apple Demo invention. Jeff Han wowed Great Literature the crowd at TED in 2006 with a demo of his multitouch research, and Bill Gates demo'd Microsoft's Surface Table in early 2007, months before the iPhone was released. Even Apple CEO Steve Jobs reportedly used Microsoft's work with tablet PCs as the jumping-off point for iPhone development.quot; - Wired Magazine NOTE: Myron Krueger was using pinching as a gesture by 1982
  • 16. Gesture-Based Interaction Design State of the Industry Overview State of the Art  State of the Industry Open Source Gesture Open Source • Lots of Sharing is going on Usability Demo • People seeking new quot;gesture Great Literature librariesquot; and openly sharing about new standards arising through practice www.interactivegestures.com • Artist and Developers have the opportunity to take advantage of computer vision frameworks developed by others, so they can focus on visual design, UI, UX and usability issues
  • 17. Gesture-Based Interaction Design Gesture Classifications Overview State of the Art State of the Industry  Gesture Open Source Usability Demo Great Literature Diectic & Pictographic Kinetographic Beat Gestures Spatial Gestures Gestures depict the shape depict bodily action Movement used to beat time of objects, people, that may be Gestures and create rhythm depict spatial boundaries or performed with or in speech position and regions. without reference relationships to an object at denote divisions in between objects includes drawing in hand timing and pace & people (real or the air or on a emphasis on imagined) surface speech Path & Location Shape Motion Pace
  • 18. Gesture-Based Interaction Design How it works Overview State of the Art State of the Industry Scientific American  Gesture How it Works: Open Source Multi-touch Surfaces Usability Explained Demo June 18, 2008 Great Literature Hands-On Computing: How Multi-Touch Screens Could Change the Way We Interact with Computers and Each Other: The iPhone and even wilder interfaces could improve collaboration without a mouse or keyboard By Stuart F. Brown
  • 19. Gesture-Based Interaction Design How it works Overview State of the Art State of the Industry Scientific American  Gesture How it Works: Open Source Multi-touch Surfaces Usability Explained Demo June 18, 2008 Great Literature Hands-On Computing: How Multi-Touch Screens Could Change the Way We Interact with Computers and Each Other: The iPhone and even wilder interfaces could improve collaboration without a mouse or keyboard By Stuart F. Brown
  • 20. Gesture-Based Interaction Design How it works Overview State of the Art State of the Industry Scientific American  Gesture How it Works: Open Source Multi-touch Surfaces Usability Explained Demo June 18, 2008 Great Literature Hands-On Computing: How Multi-Touch Screens Could Change the Way We Interact with Computers and Each Other: The iPhone and even wilder interfaces could improve collaboration without a mouse or keyboard By Stuart F. Brown
  • 21. Gesture-Based Interaction Design Open Source - Finger Tracking Overview State of the Art Touchlib State of the Industry Gesture  Open Source • Touchlib is a library for creating multi-touch Usability interaction surfaces. Demo Great Literature • Track IR Blobs and has built recognition of events like FLOSC - quot;a standalone 'finger down', 'finger moved', and 'finger released' application written in Java that sends and • Sends messages to several programs that recieve receives OSC packets OSC protocol ( sent via the UDP port ) via UDP, translates bidirectionally between NOTE: You can use Touchlib only on WIndows and binary OSC packets and it requires you to download a application called an XML encoding of OSC FLOSC before you can use it with FLASH packets, and sends and receives XML entities Visit Ben Chun's website for a free download: via TCP in a way that's http://www.benchun.net/flosc/ compatible with Flash's http://www.transmote.com/flosc/ XMLSocket feature.quot;
  • 22. Gesture-Based Interaction Design Open Source - Symbol Tracking Overview State of the Art Reactivison State of the Industry Gesture  Open Source • A computer vision engine that tracks the position Usability and rotation of specialized markers on a table top Demo surface Great Literature • Software, markers, and a calibration template are free to download. Calibration is simple and built into the software • Now also includes finger tracking • Calibration is simple and built into the software NOTE: This also requires you to download FLOSC Visit http://mtg.upf.edu/reactable/?software
  • 23. Gesture-Based Interaction Design Open Source - Symbol Tracking Overview State of the Art State of the Industry Gesture  Open Source Usability Demo Great Literature
  • 24. Gesture-Based Interaction Design Open Source - Symbol Tracking Overview State of the Art State of the Industry Gesture  Open Source Usability Demo Great Literature
  • 25. Gesture-Based Interaction Design Open Source - Computer Vision Overview State of the Art Processing State of the Industry Gesture  Open Source • Includes Code Libraries for computer Usability vision techniques such as shape tracking, Demo blob tracking, & face detection Great Literature • Includes libraries for Reactivision and the JMyron ( formerly known as WebcamXtra in Director ) Visit http://processing.org/reference/ libraries/ scroll down to the computer vision section
  • 26. Gesture-Based Interaction Design Usability Research Overview State of the Art Purpose State of the Industry Gesture Open Source • The study was conducted using  Usability the DNA Workbench, and interface Demo designed to teach about DNA base Great Literature pairing • QUESTION: Do people learn more when multiple people are interacting with the table simultaneously than they do when they are interacting alone? • The study was conducted to determine the benefits single-user vs The interface was created by: multi-user interaction and to explore http://accad.osu.edu/~aprice unique design considerations related to and was temporarily on display multi-user design and the multi-user at COSI experience.
  • 27. Gesture-Based Interaction Design Usability Research Overview State of the Art Methodology State of the Industry Gesture Open Source • 6 test groups total - 3 multi-user, 3 single-user  Usability • Each group was told to interact until they felt like they understood the Demo rules of the system and were able to explain it, groups had to come to Great Literature consensus • Afterwards each person was individual asked to explain the rules of the system and they were measured on the detail and accuracy of their answers Did they understand: • Nucleotide pairing • Mutation vs non-mutation • The role of the editase • Each participant took a survey upon completing the study in which they compared their experience learning with the DNA to previous experiences where they might have learned about DNA
  • 28. Gesture-Based Interaction Design Usability Research Overview State of the Art Results State of the Industry Gesture • When measured against five learning objectives, individuals within Open Source groups performed above average while individuals acting alone performed  Usability either at or below the average. Demo Great Literature • The time each individual required to feel they had adequate understanding of the system increased by an average of 3 minutes for individuals acting alone • The DNA Workbench has sounds that occur when nucleotides and editase randomly collide. At times it was diffifult to tell whether the sounds they heard were related to their actions or action on the other sides of the table. • After several errors the interface went into quot;Auto-Completequot; Mode. Individual all went into auto at some point, while member of groups did not • Members of groups gave for thourough and accurate answers than did participants who acted alone. They also gave descriptions with more confidence.
  • 29. Gesture-Based Interaction Design Usability Research Overview State of the Art Findings State of the Industry Gesture  People who learn with multi-user interfaces are likely to learn more Open Source  Usability information at a faster rate than individual interacting alone Demo Great Literature  People are more aware of issues, problems and questions and when these matters arise there is more opportunity to debate and share information related to the topic  Members of a group are more confident in the information they learn and are likely to take action with the knowledge they have more quickly.  Multi-User interfaces cannot use simple sound errors unless they associate a visual cue with the source of the error. Within gaming scenarios it is difficult to tell who or what may have caused the error.
  • 30. Gesture-Based Interaction Design DIY TUI Overview State of the Art Step 1 - Download & Install State of the Industry Gesture Open Source • Visit: http://mtg.upf.edu/reactable/?software Usability • print Fiducial Markers and calibration grid  Demo Great Literature
  • 31. Gesture-Based Interaction Design DIY TUI Overview State of the Art Step 1 - Download & Install State of the Industry Gesture Open Source • Visit: http://www.benchun.net/flosc/ Usability • Also look at this tutorial: http://www.transmote.com/flosc/  Demo Great Literature
  • 32. Gesture-Based Interaction Design DIY TUI Overview State of the Art Step 2 - Get a camera State of the Industry Gesture Open Source • Camera need to have a firewire connection Usability • It helps to have wide angle so you can get  Demo close to the table Great Literature • Some camera have detachable lenses • Some nice board cameras with detachable lenses can be found at: http://www.unibrain.com • To create an infrared camera, cover the lens 2-3 pieces of exposed film • It is better to have a black and white camera • Once you camera is positioned, you can calibrate it using the grid.
  • 33. Gesture-Based Interaction Design DIY TUI Overview State of the Art Step 3 - Build your table State of the Industry Gesture • Before going full tilt, stack some plexi on books and use clamps to Open Source positions lights. Test out several scenarios until you get the best lighting Usability and vision from your camera.  Demo Great Literature • Wait until you have everything working well before you build anything to spec. It's good to have some wiggle room. Consider the lighting scenario of the location where the table will eventually live. Is it going to have a lot of natural light? You may need to invest in some shades. • For projection underneath the table try using a mirror tilted at a 45 degree angle to increase the throw. Note that tilting projectors can cause the bulbs to burn out much faster. Some nice projectors exist that allow you to throw from short distances. These are ideal for table top interfaces where there isn't a lot of room for a projector.
  • 34. Gesture-Based Interaction Design DIY TUI Overview State of the Art Step 4 - Lighting State of the Industry Gesture Open Source • Lighting is the most difficult element of creating a table. In order for the Usability camera to see finger tips or markers while not recieving interference from  Demo a projector beam, the camera needs to look at a different spectrum of light, Great Literature thus the need for IR. • It is important to have very smooth even lighting in order to get quality sensitivity and optimal performance. Hot spots will create glitches for most open source computer vision engines. Try to use ambient or diffused lighting. Getting the lighting right is not a simple task. • Sanding the tips of LED lights can make light more diffused
  • 35. Gesture-Based Interaction Design Great Gesture Books Overview State of the Art State of the Industry Gesture Open Source Usability Demo  Great Literature Saffer, Dan Buxton, Bill. Designing Gestural Inter- Sketching User Ex- faces. perince: Getting the COMING SOON! design right and the First Chapter and Intro- right design. Boston, duction available at: MA: Elsevier, 2007. http://www.designingges- turalinterfaces.com/
  • 36. Gesture-Based Interaction Design Great Gesture Books Overview State of the Art State of the Industry Gesture Open Source Usability Demo  Great Literature Kendon, Adam. Goldin-Meadow, Gesture-Based Gesture:Visible Susan , Hearing Communication in Action as Utterance. Gesture: How our Human-Computer (Boston: MIT Press, Hands help us Interaction. 5th In- 2007) 2006. Think, Cambridge, ternational Gesture MA: Belknap Press Workshop, GW of Harvard Univer- 2003, Genova, Italy, sity Press, 2003; April 15-17, 2003
  • 37. Thanks You! Lane@LaneKuhlman.com

×