Sessione tenuta il 16/03/2018 in occasione dell'evento .NET Day 2018 presso l'università degli studi di Perugia

1. Sviluppo di
applicazioni per
Mobile, Web ed IoT
Cross-Platform
&
Open Source
#dotnetday2018

2. M assimo Bonanni
Paranormal Developer, with the head in the Cloud and all the
REST in microservices!
massimo.bonanni@tiscali.it
@massimobonanni
Project Prague & RealSense:
gestures in a simple way!!

3. Gesture scenarios.....

4. “… the idea of gesture
input will become
mainstream –
both when working far
away from the screen
(TV, big meeting room)
and when working at your
desk (PC).”
Microsoft’s Focused Dreams to Survive and Thrive,
Bill Gates, January 2016

6. What is Project Prague?
Is an easy-to-use SDK that helps
developers to incorporate gesture-
based controls into their apps.
Use a machine learning algorithm
called DFE (Discriminative Ferns
Ensemble) to obtain the hand
skeleton.
The gesture detection is implemented
using a straight forward heuristic,
with very few manually tuned
parameters.
Gesture

7. What do I need?
Intel® RealSense™ SR300
https://aka.ms/IntelRS300
Intel® RealSense™ F200

8. Requirements
Category Recommended Minimal
CPU Intel® Core™ i7 series, 8 logical cores Intel® Core™ i5 series, 4 logical cores
Free RAM 2GB or more 1GB
Operating System Windows 10 with Creator Update installed Windows 10
Brand Gesture Detection Range Experience
Intel® RealSense™ SR300 camera 20-60 [cm] Best
Intel® RealSense™ F200 camera 20-60 [cm] Good
Kinect for Windows v2 60-110 [cm] Good
http://aka.ms/gestures/download

10. Palm Pose
Finger Poses
Fingertip Placement
Relations
+
+
Fingertip Distance
Relations+
HandPose
=
Direction: {left, right, up, down,
backward, forward}
Flexion: {opened, openedStretched,
folded, foldedTucked}
RelativePlacement: {left, right,
above, below, behind, inFront}
RelativeDistance: {notTouching, touching}
Direction: {left, right, up, down,
backward, forward}
Orientation: {left, right, up, down,
backward, forward}
Hand Pose
A hand pose refers to a
snapshot of the hand at
a given moment and
contains a complete
description of the state of
the palm and the fingers
in that snapshot

11. Pinch pose
Hand Pose
palm facing {forward}, orientation {up}
{index, thumb} are {touching}
{middle, ring, pinky} are {above}
{index, thumb}
{index} is {above} the {thumb}
{left} hand
{thumb, index, middle, ring, pinky}
are {opened}

12. <svg>
<path d="M 100 350 q 50 -50 100 0
q 50 -50 100 0 q 50 -50 100"/>
</svg>
HandMotion
= MotionScript
Upward,
Downward,
Right,Left,
ClockwiseArcRightUpward, ClockwiseArcRightDownward,
ClockwiseArcLeftUpward, ClockwiseArcLeftDownward,
CounterClockwiseArcLeftDownward, CounterClockwiseArcLeftUpward,
CounterClockwiseArcRightDownward, CounterClockwiseArcRightUpward,
For example, the SVG curve above can be simply described by:
ClockwiseArcRightUpward,
ClockwiseArcRightDownward,
ClockwiseArcRightUpward,
ClockwiseArcRightDownward,
ClockwiseArcRightUpward,
ClockwiseArcRightDownward,
DiagonalLeftUpward,
DiagonalRightDownward,
DiagonalRightUpward,
DiagonalLeftDownward,
Hand Motion
As you move your hand,
your fingertips trace a
curve through space: this
curve is a "hand motion"

13. A Gesture is a finite state machine that its states are either Pose objects, Motion objects
or other Gesture object
Basic templates:
A gesture should be Un/Registered with respect to the application context. Only
registered gestures trigger ( ).
Gesture
Pose Motion
Pose
Pose Pose
Gesture =
Gesture Pose
Gesture Motion

14. hand facing {forward}, orientation {up}
{index, thumb} are {touching}
{index} is {above} {thumb}
{left, right} hand
{thumb, index} are {opened}
Pinch Retract NotPinchingSlingshot =
{index, thumb} are {opened}
{left, right} hand
{index, thumb} are {notTouching}
{index, thumb} are {opened}
{left, right} hand
{index, thumb} are {notTouching} move {backward}
{left, right} hand
NotPinching
Slingshot Example

16. Gestures Service
The Gestures Service provides gesture
detection as a local service for client
applications.
After you install Project Prague, the
Gestures Service will be launched
every time your machine starts.

18. Using Gesture Service
The detection is performed in the
Microsoft.Gestures.Service.exe process that acts as a
service for gesture detection.
You have to use the GesturesServiceEndpoint class in order to
communicate with the service.

19. Create a gesture

20. Detect a gesture
You need to hook up the event indicating gesture detection to the appropriate handler:
You can detect when the single pose is triggered by the user:

21. Declare a Gesture as XAML

22. Skeleton tracking
Project Gesture also provides API for the skeleton tracking.
For each hand:
Position of the fingertips
Fingers direction
Palm position
Palm orientation

24. V1 Released July ‘17
Runtime
◦ Supported sensors: Intel RealSense SR300 & F200, Kinect v2
◦ Gestures support injected into
◦ Windows Shell, PowerPoint, Skype, YouTube and Photos App
◦ Wolfly Unity Game
◦ Camera Gesture-triggered overlays (can be used in skype)
Docs
◦ Great Overview + tutorials
Samples @Github
◦ .NET
◦ Unity

25. https://labs.cognitive.microsoft.com/

26. Key takeaways
Human Interaction is the new challenge.
Project Gesture may be one of the solution.
In the roadmap will be different kind of camera (e.g. IR
camera)
Keep in mind: Project Gesture is an experimental project.
If we’ll use it, it will become a supported project.

27. #DOTNETDAY2018 31
Grazie!!!!!
Massimo Bonanni
massimo.bonanni@tiscali.it
@massimobonanni
https://www.linkedin.com/in/massimobonanni/
#ProjectPraguePG

28. Project Gesture Web Site
https://labs.cognitive.microsoft.com/en-us/project-gesture
Project Gesture documentation
https://docs.microsoft.com/en-us/gestures/
Twitter/mail account
@ProjectPrague / gestures@microsoft.com
Project Gesture GitHub
https://github.com/Microsoft/Gestures-Samples
Toward Realistic Hands Gesture Interface: Keeping it Simple for
Developers and Machines [PDF]
https://aka.ms/TRHGI
Sample GitHub
https://github.com/massimobonanni/ProjectPragueSamples
Project Prague & RealSense: il potere nelle mani!!! (dotNet{podcast}
meet{cast})
http://bit.ly/ProjectPragueMeetCast
Reference

29. GRAZIE A TUTTI VOI …
#DOTNETDAY2018 33