2. M assimo Bonanni
Paranormal Developer, with the head in the Cloud and all the
REST in microservices!
massimo.bonanni@tiscali.it
@massimobonanni
Project Prague & RealSense:
gestures in a simple way!!
4. “… the idea of gesture
input will become
mainstream –
both when working far
away from the screen
(TV, big meeting room)
and when working at your
desk (PC).”
Microsoft’s Focused Dreams to Survive and Thrive,
Bill Gates, January 2016
5.
6. What is Project Prague?
Is an easy-to-use SDK that helps
developers to incorporate gesture-
based controls into their apps.
Use a machine learning algorithm
called DFE (Discriminative Ferns
Ensemble) to obtain the hand
skeleton.
The gesture detection is implemented
using a straight forward heuristic,
with very few manually tuned
parameters.
Gesture
7. What do I need?
Intel® RealSense™ SR300
https://aka.ms/IntelRS300
Intel® RealSense™ F200
8. Requirements
Category Recommended Minimal
CPU Intel® Core™ i7 series, 8 logical cores Intel® Core™ i5 series, 4 logical cores
Free RAM 2GB or more 1GB
Operating System Windows 10 with Creator Update installed Windows 10
Brand Gesture Detection Range Experience
Intel® RealSense™ SR300 camera 20-60 [cm] Best
Intel® RealSense™ F200 camera 20-60 [cm] Good
Kinect for Windows v2 60-110 [cm] Good
http://aka.ms/gestures/download
9.
10. Palm Pose
Finger Poses
Fingertip Placement
Relations
+
+
Fingertip Distance
Relations+
HandPose
=
Direction: {left, right, up, down,
backward, forward}
Flexion: {opened, openedStretched,
folded, foldedTucked}
RelativePlacement: {left, right,
above, below, behind, inFront}
RelativeDistance: {notTouching, touching}
Direction: {left, right, up, down,
backward, forward}
Orientation: {left, right, up, down,
backward, forward}
Hand Pose
A hand pose refers to a
snapshot of the hand at
a given moment and
contains a complete
description of the state of
the palm and the fingers
in that snapshot
11. Pinch pose
Hand Pose
palm facing {forward}, orientation {up}
{index, thumb} are {touching}
{middle, ring, pinky} are {above}
{index, thumb}
{index} is {above} the {thumb}
{left} hand
{thumb, index, middle, ring, pinky}
are {opened}
12. <svg>
<path d="M 100 350 q 50 -50 100 0
q 50 -50 100 0 q 50 -50 100"/>
</svg>
HandMotion
= MotionScript
Upward,
Downward,
Right,Left,
ClockwiseArcRightUpward, ClockwiseArcRightDownward,
ClockwiseArcLeftUpward, ClockwiseArcLeftDownward,
CounterClockwiseArcLeftDownward, CounterClockwiseArcLeftUpward,
CounterClockwiseArcRightDownward, CounterClockwiseArcRightUpward,
For example, the SVG curve above can be simply described by:
ClockwiseArcRightUpward,
ClockwiseArcRightDownward,
ClockwiseArcRightUpward,
ClockwiseArcRightDownward,
ClockwiseArcRightUpward,
ClockwiseArcRightDownward,
DiagonalLeftUpward,
DiagonalRightDownward,
DiagonalRightUpward,
DiagonalLeftDownward,
Hand Motion
As you move your hand,
your fingertips trace a
curve through space: this
curve is a "hand motion"
13. A Gesture is a finite state machine that its states are either Pose objects, Motion objects
or other Gesture object
Basic templates:
A gesture should be Un/Registered with respect to the application context. Only
registered gestures trigger ( ).
Gesture
Pose Motion
Pose
Pose Pose
Gesture =
Gesture Pose
Gesture Motion
14. hand facing {forward}, orientation {up}
{index, thumb} are {touching}
{index} is {above} {thumb}
{left, right} hand
{thumb, index} are {opened}
Pinch Retract NotPinchingSlingshot =
{index, thumb} are {opened}
{left, right} hand
{index, thumb} are {notTouching}
{index, thumb} are {opened}
{left, right} hand
{index, thumb} are {notTouching} move {backward}
{left, right} hand
NotPinching
Slingshot Example
15.
16. Gestures Service
The Gestures Service provides gesture
detection as a local service for client
applications.
After you install Project Prague, the
Gestures Service will be launched
every time your machine starts.
17.
18. Using Gesture Service
The detection is performed in the
Microsoft.Gestures.Service.exe process that acts as a
service for gesture detection.
You have to use the GesturesServiceEndpoint class in order to
communicate with the service.
20. Detect a gesture
You need to hook up the event indicating gesture detection to the appropriate handler:
You can detect when the single pose is triggered by the user:
22. Skeleton tracking
Project Gesture also provides API for the skeleton tracking.
For each hand:
Position of the fingertips
Fingers direction
Palm position
Palm orientation
23.
24. V1 Released July ‘17
Runtime
◦ Supported sensors: Intel RealSense SR300 & F200, Kinect v2
◦ Gestures support injected into
◦ Windows Shell, PowerPoint, Skype, YouTube and Photos App
◦ Wolfly Unity Game
◦ Camera Gesture-triggered overlays (can be used in skype)
Docs
◦ Great Overview + tutorials
Samples @Github
◦ .NET
◦ Unity
26. Key takeaways
Human Interaction is the new challenge.
Project Gesture may be one of the solution.
In the roadmap will be different kind of camera (e.g. IR
camera)
Keep in mind: Project Gesture is an experimental project.
If we’ll use it, it will become a supported project.