Sessione su Project Gesture (ex Project Prague) tenuta all'università di Isernia in occasione della tappa di Isernia dell'IoTDay 2018.

2. Project Prague & RealSense:
il potere nelle mani!!
Massimo Bonanni
Paranormal Developer, with the head in the Cloud and all the REST in
microservices!
massimo.bonanni@microsoft.com
@massimobonanni

3. Gesture scenarios.....

4. “… the idea of gesture input
will become mainstream –
both when working far
away from the screen
(TV, big meeting room)
and when working at your
desk (PC).”
Microsoft’s Focused Dreams to Survive and Thrive,
Bill Gates, January 2016

6. What is Project Gesture?
Is an easy-to-use SDK that helps
developers to incorporate gesture-
based controls into their apps.
Use a machine learning algorithm
called DFE (Discriminative Ferns
Ensemble) to obtain the hand
skeleton.
The gesture detection is implemented
using a straight forward heuristic, with
very few manually tuned parameters.

7. what do I need?
Intel® RealSense™ SR300
https://aka.ms/IntelRS300
Intel® RealSense™ F200

8. Requirements
Category Recommended Minimal
CPU Intel® Core™ i7 series, 8 logical cores Intel® Core™ i5 series, 4 logical cores
Free RAM 2GB or more 1GB
Operating System Windows 10 with Creator Update installed Windows 10
Brand Gesture Detection Range Experience
Intel® RealSense™ SR300 camera 20-60 [cm] Best
Intel® RealSense™ F200 camera 20-60 [cm] Good
Kinect for Windows v2 60-110 [cm] Good
http://aka.ms/gestures/download

9. Hand pose, Hand Motion & Gestures

10. Palm Pose
Finger Poses
Fingertip Placement
Relations
+
+
Fingertip Distance
Relations+
HandPose
=
Direction: {left, right, up, down,
backward, forward}
Flexion: {opened, openedStretched,
folded, foldedTucked}
RelativePlacement: {left, right,
above, below, behind, inFront}
RelativeDistance: {notTouching, touching}
Direction: {left, right, up, down,
backward, forward}
Orientation: {left, right, up, down,
backward, forward}
Hand Pose
A hand pose refers to a snapshot of the hand at a given moment and contains
a complete description of the state of the palm and the fingers in that
snapshot

11. palm facing {forward}, orientation {up}
{index, thumb} are {touching}
{middle, ring, pinky} are {above}
{index, thumb}
{index} is {above} the {thumb}
{left} hand
{thumb, index, middle, ring, pinky}
are {opened}
Pinch pose
Hand Pose

12. <svg>
<path d="M 100 350 q 50 -50 100 0
q 50 -50 100 0 q 50 -50 100"/>
</svg>
HandMotion
= MotionScript
Upward,
Downward,
Right,Left,
ClockwiseArcRightUpward, ClockwiseArcRightDownward,
ClockwiseArcLeftUpward, ClockwiseArcLeftDownward,
CounterClockwiseArcLeftDownward, CounterClockwiseArcLeftUpward,
CounterClockwiseArcRightDownward, CounterClockwiseArcRightUpward,
For example, the SVG curve above can be simply described by:
ClockwiseArcRightUpward,
ClockwiseArcRightDownward,
ClockwiseArcRightUpward,
ClockwiseArcRightDownward,
ClockwiseArcRightUpward,
ClockwiseArcRightDownward,
DiagonalLeftUpward,
DiagonalRightDownward,
DiagonalRightUpward,
DiagonalLeftDownward,
Hand Motion
As you move your hand, your fingertips trace a curve through space: this
curve is a "hand motion"

13. A Gesture is a finite state machine that its states are either Pose objects,
Motion objects or other Gesture object
Basic templates:
A gesture should be Un/Registered with respect to the application context.
Only registered gestures trigger ( ).
Pose Motion
Pose
Pose Pose
Gesture =
Gesture
Gesture
Pose
Motion
Gesture

14. hand facing {forward}, orientation {up}
{index, thumb} are {touching}
{index} is {above} {thumb}
{left, right} hand
{thumb, index} are {opened}
Pinch Retract NotPinchingSlingshot =
{index, thumb} are {opened}
{left, right} hand
{index, thumb} are {notTouching}
{index, thumb} are {opened}
{left, right} hand
{index, thumb} are {notTouching} move {backward}
{left, right} hand
NotPinching
Slingshot Example

15. Developers, developers, developers….

16. Gestures Service
The Gestures Service provides gesture
detection as a local service for client
applications.
After you install Project Gesture, the
Gestures Service will be launched every
time your machine starts.

17. Demo
Project Gesture SDK and Tools

18. Using Gesture Service
The detection is performed in the
Microsoft.Gestures.Service.exe process that acts
as a service for gesture detection.
You have to use the GesturesServiceEndpoint class in
order to communicate with the service.

19. Create a gesture

20. Detect a gesture
You need to hook up the event indicating gesture detection to the appropriate
handler:
You can detect when the single pose is triggered by the user:

21. Declare a Gesture as XAML

22. Skeleton tracking
Project Gesture also provides API for the skeleton tracking.
For each hand:
Position of the fingertips
Fingers direction
Palm position
Palm orientation

23. V1 Released July ‘17
Runtime
• Supported sensors: Intel RealSense SR300 & F200, Kinect v2
• Gestures support injected into
• Windows Shell, PowerPoint, Skype, YouTube and Photos App
• Wolfly Unity Game
• Camera Gesture-triggered overlays (can be used in skype)
Docs
• Great Overview + tutorials
Samples @Github
• .NET
• Unity

24. Demo
Using Project Gesture SDK

25. Cognitive Service Labs
https://labs.cognitive.microsoft.com/
Trials have both rate
limit, in terms of
transactions per
second or minute, and
a monthly usage cap.

26. Key takeaways
Human Interaction is the new challenge.
Project Gesture may be one of the solution.
In the roadmap will be different kind of camera (e.g. IR
camera)
Keep in mind: Project Gesture is an experimental project.
If we’ll use it, it will become a supported project.

27. Feedback & Questions
Grazie!!!!!
Massimo Bonanni
massimo.bonanni@microsoft.com
@massimobonanni
https://www.linkedin.com/in/massimobonanni/
#ProjectGestureIS

28. Project Gesture Reference
Project Gesture Web Site
https://labs.cognitive.microsoft.com/en-us/project-gesture
Project Gesture documentation
https://docs.microsoft.com/en-us/gestures/
Twitter/mail account
@ProjectPrague / gestures@Microsoft.com
Project Gesture GitHub
https://github.com/Microsoft/Gestures-Samples
Toward Realistic Hands Gesture Interface: Keeping it Simple for Developers and Machines [PDF]
https://aka.ms/TRHGI
Sample GitHub
https://github.com/massimobonanni/ProjectPragueSamples