Web Motion: Motion Detection on the Web


Published on

Slides from a talk originally given on April 1, 2013 at the HTML5 Developer Conference in San Francisco
1. Overview of competitive landscape.
2. Up close with the LeapJS API
3. How will this fit into web standards?

  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Web Motion: Motion Detection on the Web

  1. 1. Web MotionMotion Detection on the WebBill Fisher | @fisherwebdev | #webmotionMonday, April 1, 13
  2. 2. People Get Ready• Slides: http://bill.hotstudiodev.com/webmotion.pdf• Code: https://github.com/fisherwebdev/webmotion• Me: @fisherwebdev• Us: #webmotionMonday, April 1, 13
  3. 3. Post-TouchMonday, April 1, 13
  4. 4. Monday, April 1, 13
  5. 5. Monday, April 1, 13
  6. 6. Monday, April 1, 13
  7. 7. The Leap Motion DeviceMonday, April 1, 13
  8. 8. • Very sensitive and high resolution motion detection• Somewhat prone to noisy and inconsistent data• 10,000 devices sent to developers• APIs for many platforms, including the WebLeap MotionMonday, April 1, 13
  9. 9. • Current price is at $79.99• App store called Airspace will include web apps• Shipping May 13th• Sold at BestBuy on May 19thLeap MotionMonday, April 1, 13
  10. 10. ~ 10 - 20 mm~ 400 - 600 mmMonday, April 1, 13
  11. 11. Competitive LandscapeMonday, April 1, 13
  12. 12. Monday, April 1, 13
  13. 13. • XBox only. IE10 may be controlled with a XBox LIVEGold account. Windows integration is coming.• Current integration with IE10 is focused on voicecontrol and is janky at best.• Most use cases involve being far away from thesensor, and are based on gross motor movements.• Complete loss of data within 16 cm• Price has now dropped to $99Kinect by MicrosoftMonday, April 1, 13
  14. 14. Monday, April 1, 13
  15. 15. • Leap Motion clone, but DIY friendly• Cameras are PS3 Eyes, all parts are off-the-shelf• Company history shows dedication to open source• Kickstarter project - $140 for a fully-assembled device• Drivers and SDK not yet released, Windows will be first• June 2013: estimated shipping date• http://duo3d.com/DUO by Code LaboratoriesMonday, April 1, 13
  16. 16. Monday, April 1, 13
  17. 17. • Embedded tech, not currently a consumer device• Uses time-of-flight instead of a camera-based solution• Native APIs are provided by third party partners, butno Web API yet.• http://www.pmdtec.com/CamBoard Pico by PMDMonday, April 1, 13
  18. 18. Monday, April 1, 13
  19. 19. Monday, April 1, 13
  20. 20. • PrimeSense technology is at the heart of the Kinect• Relationship with Microsoft is apparently dissolving• Now offering the tech as a standalone consumerdevice and as an embedded solution• No apparent interest in directly supporting the Web.OpenNI is the only API.• http://www.primesense.com/Capri by PrimeSenseMonday, April 1, 13
  21. 21. Monday, April 1, 13
  22. 22. Monday, April 1, 13
  23. 23. • Worn as an armband, connected via Blue Tooth• Detects electrical impulses of muscle movement• Thalmic claims MYO will “just work” on with existingoperating systems and the Web (mouse events???)• Backed by Y Combinator - $149 consumer price• Summer 2013: developer API• Late 2013: shipping first 25,000 units• https://getmyo.com/MYO by Thalmic LabsMonday, April 1, 13
  24. 24. Consistent ThemesOften Positioned as EmbeddedTiny or Discreet Form FactorOften Capable of DepthGames and EntertainmentIgnore the WebMonday, April 1, 13
  25. 25. leapjs APIMonday, April 1, 13
  26. 26. How it works• Leap.app starts a local Node server on port 6437, andleapjs connects to it via Web Sockets.• leapjs continuous loop uses requestAnimationFramefor 60fps by default. You may opt to use a faster loop.• Each cycle of the loop has a reference to the currentframe, which contains most of the useful data.Monday, April 1, 13
  27. 27. leapjs objects• Many objects can express the concept of validity.• The high level objects of the API are Controllers,Frames, Pointables / Hands, and Gestures.• Source code is highly readable and well commentedwith JSdoc annotations.• Modular code, so you can trim it down.Monday, April 1, 13
  28. 28. leapjs controller• Most central component of the API, but typically youdon’t use it much.• Manages the loop. “this” inside the loop is thecontroller.• Has references to other parts of the system, and this isthe first place to look if you are looking for somethingnot in the Frame.Monday, April 1, 13
  29. 29. leapjs frames• Contains all pointables, fingers, tools, hands, gesturesfor the current loop / rAF cycle.• ID and timestamp• Also contains global translation data, like scaleFactorand rotationMatrix• CircularBuffer is a collection of the previous 200framesMonday, April 1, 13
  30. 30. leapjs pointables• Generic object that may be fingers or tools• ID, validity, direction, length, width, velocity• Tracking multiple pointables can be inconsistent.What happens when your pointable vanishes?Monday, April 1, 13
  31. 31. leapjs hands• In some ways great, but even more inconsistent thanpointables. Multiple hands are particularlyproblematic.• ID, validity, position, direction, palm vector, collectionof pointables.• sphereCenter and sphereRadius based on the cup ofthe hand.• translations and matrix dataMonday, April 1, 13
  32. 32. leapjs handsMonday, April 1, 13
  33. 33. leapjs gestures• The most high-level part of the API. This is a goodthing.• Screen Tap, Key Tap, Swipe and Circle.• Screen Tap and Key Tap feel silly, Swipe takes practice,but Circle is reliable.• More gestures might be coming soon.Monday, April 1, 13
  34. 34. Demos and CodeMonday, April 1, 13
  35. 35. Where are We Going?Monday, April 1, 13
  36. 36. Pointer EventsMonday, April 1, 13
  37. 37. Web Standards• Pointer Events are a W3C submission from Microsoft.Already in IE10, and they will be in WebKit soon.• A replacement and abstraction of Mouse and Touch,with attention to stylus input.• Independent events for each pointer, unlike Touch.• Pointer Events Working Group has reached the “LastCall Working Draft”• Could they be used for 3D input also?Monday, April 1, 13
  38. 38. “Its unclear what the units wouldbe for all this stuff. ... As soon asyoure dealing with tracking realworld stuff, its in some differentcoordinate system. If you canbring it back to screen space,youre doing it with some weirdtransform.”-- Boris Smus at EdgeConf 2013Monday, April 1, 13
  39. 39. “Considering this for the V1spec would pose a risk to itscurrent trajectory.”“We should let a thousandflowers bloom.”Monday, April 1, 13
  40. 40. Will They Bloom?Monday, April 1, 13
  41. 41. Motion vs. Touch• Touch is extremely intuitive.• Physical contact does matter.• Motion tracking suffers even more for avatar issuesthan Mouse. We will need a lot of extra feedback.Monday, April 1, 13
  42. 42. Use Cases• Games• Medical• Gloved Interaction• Kitchens, or other cases of wet or dirty hands• Extremely casual or blind interaction: car stereos• Jet Packs!Monday, April 1, 13
  43. 43. ConclusionsMonday, April 1, 13
  44. 44. Leap Motion FTW• The data coming from the Leap Motion device is stillquite unstable in the leapjs API.• A single pointable or hand is more stable than multipleones.• High level abstractions are a good idea.• Consider using the leap as a controller for interactionengines, rather than directly mapping the data to theUI.• It’s a lot of fun!Monday, April 1, 13
  45. 45. Monday, April 1, 13
  46. 46. LinksMonday, April 1, 13
  47. 47. Demos and Documentation• https://github.com/leapmotion/leapjs• http://leapmotion.github.com/leapjs/index.html• https://developer.leapmotion.com/• http://dewmap.com/experiment/gilt-leap/index.html• https://github.com/rtilton1/leapjs/tree/master/examples/ironman2• http://leapgamer.com/Monday, April 1, 13
  48. 48. Don’t forget...GE Party6:30pm111 Minna Streetbeer!Monday, April 1, 13
  49. 49. Thanks!Bill Fisher | @fisherwebdev | #mowebappsMonday, April 1, 13