SlideShare a Scribd company logo
1 of 57
Download to read offline
The Death of a Mouse 
Geert Bevin - XRebel Product Manager
Who am I? 
> Geert Bevin 
> XRebel Product Manager at ZeroTurnaround 
> Java Champion 
> Creator of RIFE framework, native Java continuations 
> Many other open-source projects
The decline of cheese 
2008 2013
100 
75 
50 
25 
0 
100 
75 
50 
25 
0 
Death toll Cheese quality 
2008 2009 2010 2011 2012 2013
What qualifies me? 
> 3 of the most popular applications in the LeapMotion Store 
> GameWAVE : play existing keyboard-mouse games 
> GECO : make music with hand gestures 
> HandWAVE : simple computer control 
> Total of more than 40000 users 
> Alpha developer for Thalmic Myo, iOS music app on the way
Leap Motion Controller
Leap Motion Controller
Tracking demo
Technical overview
Coordinate system 
right-handed Cartesian 
25 to 600 millimeters above device
Motion tracking data 
> Unique ID for each visible entity 
> Frames 
> Hands 
> Fingers 
> Bones 
> Gestures 
> Tools 
> Images
Very precise 
> 10μm of real resolution (1/100th of a mm) 
> 200x more accurate than Kinect 
> Adaptive sample rates: 60-300 hz, default 120 hz 
> Data rate is preserved from end to end
Many supported languages 
> C++ 
> Objective-C 
> C# 
> Java 
> Python 
> JavaScript
Architecture
Application Interface
WebSocket Interface
Basic Java example
Java example : part 1 
import java.io.IOException; 
import com.leapmotion.leap.*; 
! 
class Sample { 
public static void main(String[] args) { 
SampleListener listener = new SampleListener(); 
Controller controller = new Controller(); 
controller.addListener(listener); 
System.out.println("Press Enter to quit..."); 
try { System.in.read(); } 
catch (IOException e) { e.printStackTrace(); } 
! 
controller.removeListener(listener); 
} 
}
class SampleListener extends Listener { 
public void onInit(Controller c) {} 
public void onDisconnect(Controller c) {} 
public void onExit(Controller c) {} 
public void onConnect(Controller controller) { 
controller.enableGesture(Gesture.Type.TYPE_SWIPE); 
} 
public void onFrame(Controller controller) { 
Frame frame = controller.frame(); 
// Your processing logic 
System.out.println("Frame id: " + frame.id() 
+ ", timestamp: " + frame.timestamp() 
+ ", hands: " + frame.hands().count() 
+ ", gestures: " + frame.gestures().count()); 
} 
} 
Java example : part 2
Compile and Run 
javac -classpath <LeapSDK>/lib/LeapJava.jar Leap.java 
! 
java -classpath <LeapSDK>/lib/LeapJava.jar:. 
-Djava.library.path=<LeapSDK>/lib 
Leap
GameWAVE demo
Leap data models
Frame Motion Analysis 
> Compare the current frame with a specified earlier frame 
> rotationAxis: direction vector expressing the axis of rotation 
> rotationAngle: angle of rotation clockwise around the rotation 
axis (using the right-hand rule) 
> rotationMatrix: transform matrix expressing the rotation 
> scaleFactor: factor expressing expansion or contraction 
> translation: vector expressing the linear movement
Hand Attributes 1/3 
> palmPosition: center of the palm from Leap origin 
> palmVelocity: speed of palm in millimeters per second 
> palmNormal: vector perpendicular to the palm plane 
> direction: vector pointing from center toward the fingers
Hand Attributes 2/3 
> grabStrength: conformity with grab posture 
> pinchStrength: conformity with pinch posture
Hand Attributes 3/3 
> sphereCenter: center of a sphere fit to curvature of hand 
> sphereRadius: radius of a sphere fit to curvature of hand
Hand Motion Analysis 
> Compare hand in current frame with a specified earlier frame 
> rotationAxis 
> rotationAngle 
> rotationMatrix 
> scaleFactor 
> translation
Hand Finger and Tool lists 
> pointables: both fingers and tools 
> fingers: just the fingers 
> tools: just the tools
Finger and Tool models 
> length: length of the visible portion of the object 
> width: average width of the visible portion 
> direction: direction vector pointing same as object 
> tipPosition: position of the tip 
> tipVelocity: speed of the tip
Finger bone model
Gestures
Circle gesture
Swipe gesture
Key tap gesture
Screen tap gesture
Basic JavaScript example
<html> 
<head> 
<title>Leap JavaScript Sample</title> 
<script src="//js.leapmotion.com/leap-0.6.2.js"></script> 
<script> 
var controllerOptions = {enableGestures: true}; 
Leap.loop(controllerOptions, function(frame) { 
var frameOutput = document.getElementById("frameData"); 
frameOutput.innerHTML = "Frame ID: " + frame.id + "<br />" 
+ "Timestamp: " + frame.timestamp + " &micro;s<br />" 
+ "Hands: " + frame.hands.length + "<br />" 
+ "Fingers: " + frame.fingers.length + "<br />" 
+ "Tools: " + frame.tools.length + "<br />" 
+ "Gestures: " + frame.gestures.length + "<br />"; 
}) 
</script> 
</head> 
<body> 
<h3>Frame data:</h3> 
<div id="frameData"></div> 
</body> 
</html>
GECO showcase
VR applications
Myo
Measures electrical 
activity in muscles
Myo 
> wireless over Bluetooth 4.0 LE 
> wearable 
> vibration feedback 
> status LED
Myo data models 
> orientation data stream 
> acceleration data stream 
> gyroscope data stream 
> poses : wave in, wave out, spread fingers, fist, thumb to pinky
Myo architecture 
> per-application HUB to access Myo instances 
> listener interface 
> limited language bindings atm: 
C++, Objective-C, Android Java, Lua
private DeviceListener listener = new AbstractDeviceListener() { 
public void onConnect(Myo myo, long t) { } 
public void onDisconnect(Myo myo, long t) { } 
public void onArmRecognized(Myo myo, long t, Arm arm, XDirection xdir) { } 
public void onArmLost(Myo myo, long timestamp) { } 
public void onOrientationData(Myo myo, long timestamp, Quaternion rotation) { 
float roll = (float) Math.toDegrees(Quaternion.roll(rotation)); 
float pitch = (float) Math.toDegrees(Quaternion.pitch(rotation)); 
float yaw = (float) Math.toDegrees(Quaternion.yaw(rotation)); 
// use roll, pitch and yaw 
} 
public void onPose(Myo myo, long timestamp, Pose pose) { 
if (pose == FIST) { // use fist pose } 
} 
}; 
! 
// initialize the Hub singleton with an application identifier. 
Hub hub = Hub.getInstance(); 
if (!hub.init(this, getPackageName())) { // handle init error } 
// register for DeviceListener callbacks. 
hub.addListener(listener); 
// pair with nearby Myo 
hub.pairWithAdjacentMyo();
MyoMIDI prototype demo
Some personal findings
Embrace precision 
> filter only when really necessary 
> avoid machine learning 
> enable users to learn how to best use your interactions 
> self-rewarding UI is very powerful
Carefully pick your gestures 
> tempting to do cool and unique interactions 
> users only occasionally use your application 
> they don’t want to learn and re-learn
Consider screen-less 
> keyboard-mouse has us always staring at the screen 
> gesture control turns ourselves into our own reference point 
> auditory and vibration cues can be good alternatives
Latency matters 
> quickly hand off operations to independent processing queue 
> make sure you test in sub-optimal environments 
> consider battery and cpu usage 
> sometimes less ‘correct’ algorithms might actually feel better
Leap Motion Controller 
http://www.leapmotion.com 
Thanks to 
! 
Thalmic Myo 
http://www.thalmic.com

More Related Content

What's hot

Dirty Durham: Dry cleaning solvents leaked into part of Trinity Park | News
Dirty Durham: Dry cleaning solvents leaked into part of Trinity Park | NewsDirty Durham: Dry cleaning solvents leaked into part of Trinity Park | News
Dirty Durham: Dry cleaning solvents leaked into part of Trinity Park | Newsdizzyspiral5631
 
Interactive Music II ProcessingとSuperColliderの連携 -2
Interactive Music II ProcessingとSuperColliderの連携 -2Interactive Music II ProcessingとSuperColliderの連携 -2
Interactive Music II ProcessingとSuperColliderの連携 -2Atsushi Tadokoro
 
About java
About javaAbout java
About javaJay Xu
 
FITC Web Unleashed 2017 - Introduction to the World of Testing for Front-End ...
FITC Web Unleashed 2017 - Introduction to the World of Testing for Front-End ...FITC Web Unleashed 2017 - Introduction to the World of Testing for Front-End ...
FITC Web Unleashed 2017 - Introduction to the World of Testing for Front-End ...Haris Mahmood
 
An Introduction to the World of Testing for Front-End Developers
An Introduction to the World of Testing for Front-End DevelopersAn Introduction to the World of Testing for Front-End Developers
An Introduction to the World of Testing for Front-End DevelopersFITC
 
Welcome to Modern C++
Welcome to Modern C++Welcome to Modern C++
Welcome to Modern C++Seok-joon Yun
 
Letswift18 워크숍#1 스위프트 클린코드와 코드리뷰
Letswift18 워크숍#1 스위프트 클린코드와 코드리뷰Letswift18 워크숍#1 스위프트 클린코드와 코드리뷰
Letswift18 워크숍#1 스위프트 클린코드와 코드리뷰Jung Kim
 
The Ring programming language version 1.8 book - Part 65 of 202
The Ring programming language version 1.8 book - Part 65 of 202The Ring programming language version 1.8 book - Part 65 of 202
The Ring programming language version 1.8 book - Part 65 of 202Mahmoud Samir Fayed
 
Pro typescript.ch03.Object Orientation in TypeScript
Pro typescript.ch03.Object Orientation in TypeScriptPro typescript.ch03.Object Orientation in TypeScript
Pro typescript.ch03.Object Orientation in TypeScriptSeok-joon Yun
 
Standford 2015 week6
Standford 2015 week6Standford 2015 week6
Standford 2015 week6彼得潘 Pan
 
Самые вкусные баги из игрового кода: как ошибаются наши коллеги-программисты ...
Самые вкусные баги из игрового кода: как ошибаются наши коллеги-программисты ...Самые вкусные баги из игрового кода: как ошибаются наши коллеги-программисты ...
Самые вкусные баги из игрового кода: как ошибаются наши коллеги-программисты ...DevGAMM Conference
 
start_printf: dev/ic/com.c comstart()
start_printf: dev/ic/com.c comstart()start_printf: dev/ic/com.c comstart()
start_printf: dev/ic/com.c comstart()Kiwamu Okabe
 
Capture and replay hardware behaviour for regression testing and bug reporting
Capture and replay hardware behaviour for regression testing and bug reportingCapture and replay hardware behaviour for regression testing and bug reporting
Capture and replay hardware behaviour for regression testing and bug reportingmartin-pitt
 
Moony li pacsec-1.8
Moony li pacsec-1.8Moony li pacsec-1.8
Moony li pacsec-1.8PacSecJP
 
2010 bb dev con
2010 bb dev con 2010 bb dev con
2010 bb dev con Eing Ong
 
2009 Hackday Taiwan Yui
2009 Hackday Taiwan Yui2009 Hackday Taiwan Yui
2009 Hackday Taiwan YuiJH Lee
 

What's hot (20)

WebXR if X = how?
WebXR if X = how?WebXR if X = how?
WebXR if X = how?
 
Dirty Durham: Dry cleaning solvents leaked into part of Trinity Park | News
Dirty Durham: Dry cleaning solvents leaked into part of Trinity Park | NewsDirty Durham: Dry cleaning solvents leaked into part of Trinity Park | News
Dirty Durham: Dry cleaning solvents leaked into part of Trinity Park | News
 
Interactive Music II ProcessingとSuperColliderの連携 -2
Interactive Music II ProcessingとSuperColliderの連携 -2Interactive Music II ProcessingとSuperColliderの連携 -2
Interactive Music II ProcessingとSuperColliderの連携 -2
 
About java
About javaAbout java
About java
 
ES6, WTF?
ES6, WTF?ES6, WTF?
ES6, WTF?
 
FITC Web Unleashed 2017 - Introduction to the World of Testing for Front-End ...
FITC Web Unleashed 2017 - Introduction to the World of Testing for Front-End ...FITC Web Unleashed 2017 - Introduction to the World of Testing for Front-End ...
FITC Web Unleashed 2017 - Introduction to the World of Testing for Front-End ...
 
An Introduction to the World of Testing for Front-End Developers
An Introduction to the World of Testing for Front-End DevelopersAn Introduction to the World of Testing for Front-End Developers
An Introduction to the World of Testing for Front-End Developers
 
Assignment
AssignmentAssignment
Assignment
 
Welcome to Modern C++
Welcome to Modern C++Welcome to Modern C++
Welcome to Modern C++
 
Letswift18 워크숍#1 스위프트 클린코드와 코드리뷰
Letswift18 워크숍#1 스위프트 클린코드와 코드리뷰Letswift18 워크숍#1 스위프트 클린코드와 코드리뷰
Letswift18 워크숍#1 스위프트 클린코드와 코드리뷰
 
The Ring programming language version 1.8 book - Part 65 of 202
The Ring programming language version 1.8 book - Part 65 of 202The Ring programming language version 1.8 book - Part 65 of 202
The Ring programming language version 1.8 book - Part 65 of 202
 
Pro typescript.ch03.Object Orientation in TypeScript
Pro typescript.ch03.Object Orientation in TypeScriptPro typescript.ch03.Object Orientation in TypeScript
Pro typescript.ch03.Object Orientation in TypeScript
 
Standford 2015 week6
Standford 2015 week6Standford 2015 week6
Standford 2015 week6
 
Самые вкусные баги из игрового кода: как ошибаются наши коллеги-программисты ...
Самые вкусные баги из игрового кода: как ошибаются наши коллеги-программисты ...Самые вкусные баги из игрового кода: как ошибаются наши коллеги-программисты ...
Самые вкусные баги из игрового кода: как ошибаются наши коллеги-программисты ...
 
start_printf: dev/ic/com.c comstart()
start_printf: dev/ic/com.c comstart()start_printf: dev/ic/com.c comstart()
start_printf: dev/ic/com.c comstart()
 
Capture and replay hardware behaviour for regression testing and bug reporting
Capture and replay hardware behaviour for regression testing and bug reportingCapture and replay hardware behaviour for regression testing and bug reporting
Capture and replay hardware behaviour for regression testing and bug reporting
 
Moony li pacsec-1.8
Moony li pacsec-1.8Moony li pacsec-1.8
Moony li pacsec-1.8
 
20120822 joxa
20120822 joxa20120822 joxa
20120822 joxa
 
2010 bb dev con
2010 bb dev con 2010 bb dev con
2010 bb dev con
 
2009 Hackday Taiwan Yui
2009 Hackday Taiwan Yui2009 Hackday Taiwan Yui
2009 Hackday Taiwan Yui
 

Viewers also liked

Gesture recognition adi
Gesture recognition adiGesture recognition adi
Gesture recognition adiaditya verma
 
android controlled robot
android controlled robotandroid controlled robot
android controlled robotsunny080593
 
Wireless robot ppt
Wireless robot pptWireless robot ppt
Wireless robot pptVarun B P
 
WIRELESS ROBOT PPT
WIRELESS ROBOT PPTWIRELESS ROBOT PPT
WIRELESS ROBOT PPTAIRTEL
 

Viewers also liked (9)

hand gestures
hand gestureshand gestures
hand gestures
 
Virtual Mouse
Virtual MouseVirtual Mouse
Virtual Mouse
 
Hand gesture recognition
Hand gesture recognitionHand gesture recognition
Hand gesture recognition
 
Gesture recognition adi
Gesture recognition adiGesture recognition adi
Gesture recognition adi
 
android controlled robot
android controlled robotandroid controlled robot
android controlled robot
 
Gesture Recognition
Gesture RecognitionGesture Recognition
Gesture Recognition
 
Wireless robot ppt
Wireless robot pptWireless robot ppt
Wireless robot ppt
 
Gesture recognition
Gesture recognitionGesture recognition
Gesture recognition
 
WIRELESS ROBOT PPT
WIRELESS ROBOT PPTWIRELESS ROBOT PPT
WIRELESS ROBOT PPT
 

Similar to The Death of a Mouse: Geert Bevin on Leap Motion, Myo, and Gesture Control

Mobile HTML, CSS, and JavaScript
Mobile HTML, CSS, and JavaScriptMobile HTML, CSS, and JavaScript
Mobile HTML, CSS, and JavaScriptfranksvalli
 
soft-shake.ch - Hands on Node.js
soft-shake.ch - Hands on Node.jssoft-shake.ch - Hands on Node.js
soft-shake.ch - Hands on Node.jssoft-shake.ch
 
mobl presentation @ IHomer
mobl presentation @ IHomermobl presentation @ IHomer
mobl presentation @ IHomerzefhemel
 
Asynchronous Programming at Netflix
Asynchronous Programming at NetflixAsynchronous Programming at Netflix
Asynchronous Programming at NetflixC4Media
 
Developing for Leap Motion
Developing for Leap MotionDeveloping for Leap Motion
Developing for Leap MotionIris Classon
 
Event-driven IO server-side JavaScript environment based on V8 Engine
Event-driven IO server-side JavaScript environment based on V8 EngineEvent-driven IO server-side JavaScript environment based on V8 Engine
Event-driven IO server-side JavaScript environment based on V8 EngineRicardo Silva
 
mobl - model-driven engineering lecture
mobl - model-driven engineering lecturemobl - model-driven engineering lecture
mobl - model-driven engineering lecturezefhemel
 
Is html5-ready-workshop-110727181512-phpapp02
Is html5-ready-workshop-110727181512-phpapp02Is html5-ready-workshop-110727181512-phpapp02
Is html5-ready-workshop-110727181512-phpapp02PL dream
 
Is HTML5 Ready? (workshop)
Is HTML5 Ready? (workshop)Is HTML5 Ready? (workshop)
Is HTML5 Ready? (workshop)Remy Sharp
 
2008 - TechDays PT: WCF, JSON and AJAX for performance and manageability
2008 - TechDays PT: WCF, JSON and AJAX for performance and manageability2008 - TechDays PT: WCF, JSON and AJAX for performance and manageability
2008 - TechDays PT: WCF, JSON and AJAX for performance and manageabilityDaniel Fisher
 
Bonnes pratiques de développement avec Node js
Bonnes pratiques de développement avec Node jsBonnes pratiques de développement avec Node js
Bonnes pratiques de développement avec Node jsFrancois Zaninotto
 
The Ring programming language version 1.7 book - Part 85 of 196
The Ring programming language version 1.7 book - Part 85 of 196The Ring programming language version 1.7 book - Part 85 of 196
The Ring programming language version 1.7 book - Part 85 of 196Mahmoud Samir Fayed
 
HTML5 - Daha Flash bir web?
HTML5 - Daha Flash bir web?HTML5 - Daha Flash bir web?
HTML5 - Daha Flash bir web?Ankara JUG
 
The Ring programming language version 1.8 book - Part 88 of 202
The Ring programming language version 1.8 book - Part 88 of 202The Ring programming language version 1.8 book - Part 88 of 202
The Ring programming language version 1.8 book - Part 88 of 202Mahmoud Samir Fayed
 
Gdc09 Minigames
Gdc09 MinigamesGdc09 Minigames
Gdc09 MinigamesSusan Gold
 

Similar to The Death of a Mouse: Geert Bevin on Leap Motion, Myo, and Gesture Control (20)

Mobile HTML, CSS, and JavaScript
Mobile HTML, CSS, and JavaScriptMobile HTML, CSS, and JavaScript
Mobile HTML, CSS, and JavaScript
 
mobl
moblmobl
mobl
 
soft-shake.ch - Hands on Node.js
soft-shake.ch - Hands on Node.jssoft-shake.ch - Hands on Node.js
soft-shake.ch - Hands on Node.js
 
mobl presentation @ IHomer
mobl presentation @ IHomermobl presentation @ IHomer
mobl presentation @ IHomer
 
Asynchronous Programming at Netflix
Asynchronous Programming at NetflixAsynchronous Programming at Netflix
Asynchronous Programming at Netflix
 
Developing for Leap Motion
Developing for Leap MotionDeveloping for Leap Motion
Developing for Leap Motion
 
A More Flash Like Web?
A More Flash Like Web?A More Flash Like Web?
A More Flash Like Web?
 
Event-driven IO server-side JavaScript environment based on V8 Engine
Event-driven IO server-side JavaScript environment based on V8 EngineEvent-driven IO server-side JavaScript environment based on V8 Engine
Event-driven IO server-side JavaScript environment based on V8 Engine
 
mobl - model-driven engineering lecture
mobl - model-driven engineering lecturemobl - model-driven engineering lecture
mobl - model-driven engineering lecture
 
Is html5-ready-workshop-110727181512-phpapp02
Is html5-ready-workshop-110727181512-phpapp02Is html5-ready-workshop-110727181512-phpapp02
Is html5-ready-workshop-110727181512-phpapp02
 
Is HTML5 Ready? (workshop)
Is HTML5 Ready? (workshop)Is HTML5 Ready? (workshop)
Is HTML5 Ready? (workshop)
 
2008 - TechDays PT: WCF, JSON and AJAX for performance and manageability
2008 - TechDays PT: WCF, JSON and AJAX for performance and manageability2008 - TechDays PT: WCF, JSON and AJAX for performance and manageability
2008 - TechDays PT: WCF, JSON and AJAX for performance and manageability
 
OpenCVの基礎
OpenCVの基礎OpenCVの基礎
OpenCVの基礎
 
Minicurso Android
Minicurso AndroidMinicurso Android
Minicurso Android
 
Bonnes pratiques de développement avec Node js
Bonnes pratiques de développement avec Node jsBonnes pratiques de développement avec Node js
Bonnes pratiques de développement avec Node js
 
The Ring programming language version 1.7 book - Part 85 of 196
The Ring programming language version 1.7 book - Part 85 of 196The Ring programming language version 1.7 book - Part 85 of 196
The Ring programming language version 1.7 book - Part 85 of 196
 
HTML5 - Daha Flash bir web?
HTML5 - Daha Flash bir web?HTML5 - Daha Flash bir web?
HTML5 - Daha Flash bir web?
 
The Ring programming language version 1.8 book - Part 88 of 202
The Ring programming language version 1.8 book - Part 88 of 202The Ring programming language version 1.8 book - Part 88 of 202
The Ring programming language version 1.8 book - Part 88 of 202
 
Gdc09 Minigames
Gdc09 MinigamesGdc09 Minigames
Gdc09 Minigames
 
Monitoring for the masses
Monitoring for the massesMonitoring for the masses
Monitoring for the masses
 

More from Geert Bevin

Intuitive User Interface Design for Modern Synthesizers
Intuitive User Interface Design for Modern SynthesizersIntuitive User Interface Design for Modern Synthesizers
Intuitive User Interface Design for Modern SynthesizersGeert Bevin
 
Mobile Music Making with iOS
Mobile Music Making with iOSMobile Music Making with iOS
Mobile Music Making with iOSGeert Bevin
 
Designing and implementing embedded synthesizer UIs with JUCE (Geert Bevin, A...
Designing and implementing embedded synthesizer UIs with JUCE (Geert Bevin, A...Designing and implementing embedded synthesizer UIs with JUCE (Geert Bevin, A...
Designing and implementing embedded synthesizer UIs with JUCE (Geert Bevin, A...Geert Bevin
 
From Arduino to LinnStrument
From Arduino to LinnStrumentFrom Arduino to LinnStrument
From Arduino to LinnStrumentGeert Bevin
 
LinnStrument : the ultimate open-source hacker instrument
LinnStrument : the ultimate open-source hacker instrumentLinnStrument : the ultimate open-source hacker instrument
LinnStrument : the ultimate open-source hacker instrumentGeert Bevin
 
10 Reasons Why Java Now Rocks More Than Ever
10 Reasons Why Java Now Rocks More Than Ever10 Reasons Why Java Now Rocks More Than Ever
10 Reasons Why Java Now Rocks More Than EverGeert Bevin
 

More from Geert Bevin (6)

Intuitive User Interface Design for Modern Synthesizers
Intuitive User Interface Design for Modern SynthesizersIntuitive User Interface Design for Modern Synthesizers
Intuitive User Interface Design for Modern Synthesizers
 
Mobile Music Making with iOS
Mobile Music Making with iOSMobile Music Making with iOS
Mobile Music Making with iOS
 
Designing and implementing embedded synthesizer UIs with JUCE (Geert Bevin, A...
Designing and implementing embedded synthesizer UIs with JUCE (Geert Bevin, A...Designing and implementing embedded synthesizer UIs with JUCE (Geert Bevin, A...
Designing and implementing embedded synthesizer UIs with JUCE (Geert Bevin, A...
 
From Arduino to LinnStrument
From Arduino to LinnStrumentFrom Arduino to LinnStrument
From Arduino to LinnStrument
 
LinnStrument : the ultimate open-source hacker instrument
LinnStrument : the ultimate open-source hacker instrumentLinnStrument : the ultimate open-source hacker instrument
LinnStrument : the ultimate open-source hacker instrument
 
10 Reasons Why Java Now Rocks More Than Ever
10 Reasons Why Java Now Rocks More Than Ever10 Reasons Why Java Now Rocks More Than Ever
10 Reasons Why Java Now Rocks More Than Ever
 

Recently uploaded

Odoo 14 - eLearning Module In Odoo 14 Enterprise
Odoo 14 - eLearning Module In Odoo 14 EnterpriseOdoo 14 - eLearning Module In Odoo 14 Enterprise
Odoo 14 - eLearning Module In Odoo 14 Enterprisepreethippts
 
A healthy diet for your Java application Devoxx France.pdf
A healthy diet for your Java application Devoxx France.pdfA healthy diet for your Java application Devoxx France.pdf
A healthy diet for your Java application Devoxx France.pdfMarharyta Nedzelska
 
SpotFlow: Tracking Method Calls and States at Runtime
SpotFlow: Tracking Method Calls and States at RuntimeSpotFlow: Tracking Method Calls and States at Runtime
SpotFlow: Tracking Method Calls and States at Runtimeandrehoraa
 
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...OnePlan Solutions
 
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...Angel Borroy López
 
Comparing Linux OS Image Update Models - EOSS 2024.pdf
Comparing Linux OS Image Update Models - EOSS 2024.pdfComparing Linux OS Image Update Models - EOSS 2024.pdf
Comparing Linux OS Image Update Models - EOSS 2024.pdfDrew Moseley
 
Powering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsPowering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsSafe Software
 
How to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion ApplicationHow to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion ApplicationBradBedford3
 
Introduction to Firebase Workshop Slides
Introduction to Firebase Workshop SlidesIntroduction to Firebase Workshop Slides
Introduction to Firebase Workshop Slidesvaideheekore1
 
Simplifying Microservices & Apps - The art of effortless development - Meetup...
Simplifying Microservices & Apps - The art of effortless development - Meetup...Simplifying Microservices & Apps - The art of effortless development - Meetup...
Simplifying Microservices & Apps - The art of effortless development - Meetup...Rob Geurden
 
SensoDat: Simulation-based Sensor Dataset of Self-driving Cars
SensoDat: Simulation-based Sensor Dataset of Self-driving CarsSensoDat: Simulation-based Sensor Dataset of Self-driving Cars
SensoDat: Simulation-based Sensor Dataset of Self-driving CarsChristian Birchler
 
Understanding Flamingo - DeepMind's VLM Architecture
Understanding Flamingo - DeepMind's VLM ArchitectureUnderstanding Flamingo - DeepMind's VLM Architecture
Understanding Flamingo - DeepMind's VLM Architecturerahul_net
 
Large Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and RepairLarge Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and RepairLionel Briand
 
UI5ers live - Custom Controls wrapping 3rd-party libs.pptx
UI5ers live - Custom Controls wrapping 3rd-party libs.pptxUI5ers live - Custom Controls wrapping 3rd-party libs.pptx
UI5ers live - Custom Controls wrapping 3rd-party libs.pptxAndreas Kunz
 
VictoriaMetrics Anomaly Detection Updates: Q1 2024
VictoriaMetrics Anomaly Detection Updates: Q1 2024VictoriaMetrics Anomaly Detection Updates: Q1 2024
VictoriaMetrics Anomaly Detection Updates: Q1 2024VictoriaMetrics
 
Precise and Complete Requirements? An Elusive Goal
Precise and Complete Requirements? An Elusive GoalPrecise and Complete Requirements? An Elusive Goal
Precise and Complete Requirements? An Elusive GoalLionel Briand
 
Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...
Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...
Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...Cizo Technology Services
 
2024 DevNexus Patterns for Resiliency: Shuffle shards
2024 DevNexus Patterns for Resiliency: Shuffle shards2024 DevNexus Patterns for Resiliency: Shuffle shards
2024 DevNexus Patterns for Resiliency: Shuffle shardsChristopher Curtin
 
The Role of IoT and Sensor Technology in Cargo Cloud Solutions.pptx
The Role of IoT and Sensor Technology in Cargo Cloud Solutions.pptxThe Role of IoT and Sensor Technology in Cargo Cloud Solutions.pptx
The Role of IoT and Sensor Technology in Cargo Cloud Solutions.pptxRTS corp
 
eSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration toolseSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration toolsosttopstonverter
 

Recently uploaded (20)

Odoo 14 - eLearning Module In Odoo 14 Enterprise
Odoo 14 - eLearning Module In Odoo 14 EnterpriseOdoo 14 - eLearning Module In Odoo 14 Enterprise
Odoo 14 - eLearning Module In Odoo 14 Enterprise
 
A healthy diet for your Java application Devoxx France.pdf
A healthy diet for your Java application Devoxx France.pdfA healthy diet for your Java application Devoxx France.pdf
A healthy diet for your Java application Devoxx France.pdf
 
SpotFlow: Tracking Method Calls and States at Runtime
SpotFlow: Tracking Method Calls and States at RuntimeSpotFlow: Tracking Method Calls and States at Runtime
SpotFlow: Tracking Method Calls and States at Runtime
 
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
 
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
 
Comparing Linux OS Image Update Models - EOSS 2024.pdf
Comparing Linux OS Image Update Models - EOSS 2024.pdfComparing Linux OS Image Update Models - EOSS 2024.pdf
Comparing Linux OS Image Update Models - EOSS 2024.pdf
 
Powering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsPowering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data Streams
 
How to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion ApplicationHow to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion Application
 
Introduction to Firebase Workshop Slides
Introduction to Firebase Workshop SlidesIntroduction to Firebase Workshop Slides
Introduction to Firebase Workshop Slides
 
Simplifying Microservices & Apps - The art of effortless development - Meetup...
Simplifying Microservices & Apps - The art of effortless development - Meetup...Simplifying Microservices & Apps - The art of effortless development - Meetup...
Simplifying Microservices & Apps - The art of effortless development - Meetup...
 
SensoDat: Simulation-based Sensor Dataset of Self-driving Cars
SensoDat: Simulation-based Sensor Dataset of Self-driving CarsSensoDat: Simulation-based Sensor Dataset of Self-driving Cars
SensoDat: Simulation-based Sensor Dataset of Self-driving Cars
 
Understanding Flamingo - DeepMind's VLM Architecture
Understanding Flamingo - DeepMind's VLM ArchitectureUnderstanding Flamingo - DeepMind's VLM Architecture
Understanding Flamingo - DeepMind's VLM Architecture
 
Large Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and RepairLarge Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and Repair
 
UI5ers live - Custom Controls wrapping 3rd-party libs.pptx
UI5ers live - Custom Controls wrapping 3rd-party libs.pptxUI5ers live - Custom Controls wrapping 3rd-party libs.pptx
UI5ers live - Custom Controls wrapping 3rd-party libs.pptx
 
VictoriaMetrics Anomaly Detection Updates: Q1 2024
VictoriaMetrics Anomaly Detection Updates: Q1 2024VictoriaMetrics Anomaly Detection Updates: Q1 2024
VictoriaMetrics Anomaly Detection Updates: Q1 2024
 
Precise and Complete Requirements? An Elusive Goal
Precise and Complete Requirements? An Elusive GoalPrecise and Complete Requirements? An Elusive Goal
Precise and Complete Requirements? An Elusive Goal
 
Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...
Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...
Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...
 
2024 DevNexus Patterns for Resiliency: Shuffle shards
2024 DevNexus Patterns for Resiliency: Shuffle shards2024 DevNexus Patterns for Resiliency: Shuffle shards
2024 DevNexus Patterns for Resiliency: Shuffle shards
 
The Role of IoT and Sensor Technology in Cargo Cloud Solutions.pptx
The Role of IoT and Sensor Technology in Cargo Cloud Solutions.pptxThe Role of IoT and Sensor Technology in Cargo Cloud Solutions.pptx
The Role of IoT and Sensor Technology in Cargo Cloud Solutions.pptx
 
eSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration toolseSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration tools
 

The Death of a Mouse: Geert Bevin on Leap Motion, Myo, and Gesture Control

  • 1. The Death of a Mouse Geert Bevin - XRebel Product Manager
  • 2. Who am I? > Geert Bevin > XRebel Product Manager at ZeroTurnaround > Java Champion > Creator of RIFE framework, native Java continuations > Many other open-source projects
  • 3. The decline of cheese 2008 2013
  • 4. 100 75 50 25 0 100 75 50 25 0 Death toll Cheese quality 2008 2009 2010 2011 2012 2013
  • 5. What qualifies me? > 3 of the most popular applications in the LeapMotion Store > GameWAVE : play existing keyboard-mouse games > GECO : make music with hand gestures > HandWAVE : simple computer control > Total of more than 40000 users > Alpha developer for Thalmic Myo, iOS music app on the way
  • 10. Coordinate system right-handed Cartesian 25 to 600 millimeters above device
  • 11. Motion tracking data > Unique ID for each visible entity > Frames > Hands > Fingers > Bones > Gestures > Tools > Images
  • 12. Very precise > 10μm of real resolution (1/100th of a mm) > 200x more accurate than Kinect > Adaptive sample rates: 60-300 hz, default 120 hz > Data rate is preserved from end to end
  • 13. Many supported languages > C++ > Objective-C > C# > Java > Python > JavaScript
  • 18. Java example : part 1 import java.io.IOException; import com.leapmotion.leap.*; ! class Sample { public static void main(String[] args) { SampleListener listener = new SampleListener(); Controller controller = new Controller(); controller.addListener(listener); System.out.println("Press Enter to quit..."); try { System.in.read(); } catch (IOException e) { e.printStackTrace(); } ! controller.removeListener(listener); } }
  • 19. class SampleListener extends Listener { public void onInit(Controller c) {} public void onDisconnect(Controller c) {} public void onExit(Controller c) {} public void onConnect(Controller controller) { controller.enableGesture(Gesture.Type.TYPE_SWIPE); } public void onFrame(Controller controller) { Frame frame = controller.frame(); // Your processing logic System.out.println("Frame id: " + frame.id() + ", timestamp: " + frame.timestamp() + ", hands: " + frame.hands().count() + ", gestures: " + frame.gestures().count()); } } Java example : part 2
  • 20. Compile and Run javac -classpath <LeapSDK>/lib/LeapJava.jar Leap.java ! java -classpath <LeapSDK>/lib/LeapJava.jar:. -Djava.library.path=<LeapSDK>/lib Leap
  • 22.
  • 24.
  • 25. Frame Motion Analysis > Compare the current frame with a specified earlier frame > rotationAxis: direction vector expressing the axis of rotation > rotationAngle: angle of rotation clockwise around the rotation axis (using the right-hand rule) > rotationMatrix: transform matrix expressing the rotation > scaleFactor: factor expressing expansion or contraction > translation: vector expressing the linear movement
  • 26. Hand Attributes 1/3 > palmPosition: center of the palm from Leap origin > palmVelocity: speed of palm in millimeters per second > palmNormal: vector perpendicular to the palm plane > direction: vector pointing from center toward the fingers
  • 27. Hand Attributes 2/3 > grabStrength: conformity with grab posture > pinchStrength: conformity with pinch posture
  • 28. Hand Attributes 3/3 > sphereCenter: center of a sphere fit to curvature of hand > sphereRadius: radius of a sphere fit to curvature of hand
  • 29. Hand Motion Analysis > Compare hand in current frame with a specified earlier frame > rotationAxis > rotationAngle > rotationMatrix > scaleFactor > translation
  • 30. Hand Finger and Tool lists > pointables: both fingers and tools > fingers: just the fingers > tools: just the tools
  • 31. Finger and Tool models > length: length of the visible portion of the object > width: average width of the visible portion > direction: direction vector pointing same as object > tipPosition: position of the tip > tipVelocity: speed of the tip
  • 33.
  • 40. <html> <head> <title>Leap JavaScript Sample</title> <script src="//js.leapmotion.com/leap-0.6.2.js"></script> <script> var controllerOptions = {enableGestures: true}; Leap.loop(controllerOptions, function(frame) { var frameOutput = document.getElementById("frameData"); frameOutput.innerHTML = "Frame ID: " + frame.id + "<br />" + "Timestamp: " + frame.timestamp + " &micro;s<br />" + "Hands: " + frame.hands.length + "<br />" + "Fingers: " + frame.fingers.length + "<br />" + "Tools: " + frame.tools.length + "<br />" + "Gestures: " + frame.gestures.length + "<br />"; }) </script> </head> <body> <h3>Frame data:</h3> <div id="frameData"></div> </body> </html>
  • 42.
  • 44.
  • 45. Myo
  • 47. Myo > wireless over Bluetooth 4.0 LE > wearable > vibration feedback > status LED
  • 48. Myo data models > orientation data stream > acceleration data stream > gyroscope data stream > poses : wave in, wave out, spread fingers, fist, thumb to pinky
  • 49. Myo architecture > per-application HUB to access Myo instances > listener interface > limited language bindings atm: C++, Objective-C, Android Java, Lua
  • 50. private DeviceListener listener = new AbstractDeviceListener() { public void onConnect(Myo myo, long t) { } public void onDisconnect(Myo myo, long t) { } public void onArmRecognized(Myo myo, long t, Arm arm, XDirection xdir) { } public void onArmLost(Myo myo, long timestamp) { } public void onOrientationData(Myo myo, long timestamp, Quaternion rotation) { float roll = (float) Math.toDegrees(Quaternion.roll(rotation)); float pitch = (float) Math.toDegrees(Quaternion.pitch(rotation)); float yaw = (float) Math.toDegrees(Quaternion.yaw(rotation)); // use roll, pitch and yaw } public void onPose(Myo myo, long timestamp, Pose pose) { if (pose == FIST) { // use fist pose } } }; ! // initialize the Hub singleton with an application identifier. Hub hub = Hub.getInstance(); if (!hub.init(this, getPackageName())) { // handle init error } // register for DeviceListener callbacks. hub.addListener(listener); // pair with nearby Myo hub.pairWithAdjacentMyo();
  • 53. Embrace precision > filter only when really necessary > avoid machine learning > enable users to learn how to best use your interactions > self-rewarding UI is very powerful
  • 54. Carefully pick your gestures > tempting to do cool and unique interactions > users only occasionally use your application > they don’t want to learn and re-learn
  • 55. Consider screen-less > keyboard-mouse has us always staring at the screen > gesture control turns ourselves into our own reference point > auditory and vibration cues can be good alternatives
  • 56. Latency matters > quickly hand off operations to independent processing queue > make sure you test in sub-optimal environments > consider battery and cpu usage > sometimes less ‘correct’ algorithms might actually feel better
  • 57. Leap Motion Controller http://www.leapmotion.com Thanks to ! Thalmic Myo http://www.thalmic.com