The document provides an overview of Intel's Perceptual Computing SDK capabilities including: 1) The SDK allows for perceptual computing including speech recognition, facial analysis, depth tracking, and augmented reality. It provides APIs for C++, C#, Unity and other frameworks. 2) The SDK enables capabilities like close-range finger tracking, facial tracking and analysis, voice recognition and synthesis, and augmented reality. It can retrieve data like images, geonodes, and gestures. 3) The document provides information on initializing and using the SDK within Unity including selecting capabilities, acquiring frames, retrieving image and gesture data, and cleaning up. It directs developers to additional resources for learning more.

1. Intel Confidential1
Perceptual Computing
– Pour l’instant destiné aux développeurs
– Camera Creative Senz3D disponible au prochain
trimestre
– Intégration dans des machines Intel prévue
pour le second semestre 2014 !
• Perceptual Computing SDK
– C++
– Passerelles C#, Unity, Processing, open
frameworks…

2. Intel Confidential2
Speech Recognition:
Voice command and control, short sentence
dictation, and text to speech synthesis
SDK Usage Modes Today
1
1 New usage modes may be added in the future
Close-range Depth Tracking (6 in. to 3 ft.):
Recognize the positions of each of the user’s
hands, fingers, static hand poses and moving hand
gestures.
Facial Analysis:
Face detection and recognition (six and seven point landmark and
attribution detection, including smiles, blinks, and age groups)
Augmented Reality:
Combine real-time images from the camera and close-range
tracking from the depth sensor with 2D or 3D graphical images.

3. Intel Confidential3
Close-Range Finger Tracking
• Geometric Node Tracking
– 7-point tracking: fingertips, palm center and elbow.
– Special usability points: fingertip and grabbing
points.
– Estimate positions, volumes, openness and
handedness.
• Standardized Poses/Gestures
– Poses: thumb up/down and peace
– Gestures: swipe left/right/up/down, circle, and
wave
• Label map
– Hand image and its parameters.

4. Intel Confidential4
Face Tracking and Analysis
• Multiple face detection/tracking
• Landmark detection
– 6/7-point detection including eyes, nose, and
mouth
• Facial attribute detection
– Age-group including baby/youth/adult/senior
– Gender detection
– Smile/blink detection
• Face recognition
– Similarity among a set of faces

5. Intel Confidential5
• Nuance* Voice Command and Control
– Recognize within a list of predefined
commands.
• Nuance Voice Dictation
– Recognize short sentences (<30 seconds).
• Nuance Voice Synthesis
– Text to speech for short sentences
Voice Recognition and Synthesis

6. Intel Confidential6
• Track any 2D planer surfaces
– Report position, orientation and
other parameters
• Track limited 3D objects
– Based on 3D models
• Track faces orientation
Augmented Reality

7. Intel Confidential7
SDK Usage H/W Requirements
SDK Usage
Mode
Speech
Certified Dual-
Array
Microphones
RGB
Webcam
Creative*
Camera
Close-range
Depth
tracking
X
Speech
Recognition
X X
Facial
Tracking
X X
Augmented
Reality
X X

8. Intel Confidential8
PXCSession, PXCImage, PXCAudio, PXCCapture, PXCGesture, PXCFac
eAnalysis, PXCVoice
UtilCapture, UtilPipeline
C#
PXCMSession
PXCMImage
PXCMAudio
PXCMCapture
PXCMGesture
PXCMFaceAn
alysis
PXCMVoice
UtilMCapture
UtilMPipeline
pxcupipeline
Unity*
Pro
Processing
openFrame
works*
Applications
Core Functionalities
Module Interaction
Additional Language and
Framework Support
SDK API Hierarchy
*Other brands and trademarks may be claimed as the property of their respective owners

9. Intel Confidential9
• Color streams
 RGB24 640x480 25fps, 30fps
 RGB24 640x360 25fps, 30fps
 RGB24 1280x720 25fps, 30fps
• Depth streams (16-bit integer, 0-32000)
 320x240 25fps, 30fps, 50fps, 60fps
 UVMAP (Depth  Color)
 Confidence Map (16-bit integer)
• Vertices streams (real world coordinates in 3D
fixed-point integers)
• Audio streams (At least 2-array MIC)
 44.1KHz mono/stereo
 48KHz mono/stereo
Camera Streams

10. Intel Confidential10
PXCImage: Depth & Color
IMAGE_TYPE_COLOR
(of any color format)
• planes[0]: color
data
• Planes[1]: empty
• planes[2]: empty
• planes[3]: empty
IMAGE_FORMAT_DEPTH
•planes[0]: depthmap
•Planes[1]: confidencemap
•planes[2]: uvmap
•planes[3]: empty
2-byte per
pixel
8-byte per
pixel
IMAGE_FORMAT_VERTICES
•planes[0]: vertex (x,y,z)
•Planes[1]: confidencemap
•planes[2]: uvmap
•planes[3]: empty
12-byte
per pixel
One real depth stream, two virtual

11. Intel Confidential11
Programming Language and
Framework Support
• C++, C#, Java
• Supported Frameworks
– processing
– openFrameworks
– Unity
– Havok
– Total Immersion AR

12. Intel Confidential12
Declare The SDK Object
• Unity
private PXCUPipeline pipeline;
pipeline = new PXCUPipeline();

13. Intel Confidential13
Select Features and Initialize
• Select Features using PXCUPipeline.Mode
enum
• Use Bitwise OR (|) for Multiple Features
• Use Init() To Set Features and Enable SDK
Access
pipeline.Init(PXCUPipeline.Mode.COLOR_VGA|
PXCUPipeline.Mode.DEPTH_QVGA|
PXCUPipeline.Mode.GESTURE);

14. Intel Confidential14
Capture A Frame
• Poll For A Frame Using AcquireFrame(bool);
– Can be blocking or non-blocking
– AcquireFrame(true) is blocking,
AcquireFrame(false) is non-blocking
• Returns true If A Frame Is Available
if(pipeline.AcquireFrame(false))
{
}

15. Intel Confidential15
Retrieve Image Data
• Data Is Retrieved via Query*(<T>)
– QueryRGB(), QueryLabelMapAsImage(), etc…
• Unity
Texture2D rgbTexture = new
Texture2D(640,480,TextureFormat.ARGB32, false);
pipeline.QueryRGB(rgbTexture);

16. Intel Confidential16
Retrieve Geonodes Data
• Data Is Retrieved via QueryGeoNode()
– Use PXCGesture::GeoNode::Label and
PXCGesture::GeoNode to retrieve data
• Unity
PXCMGesture.GeoNode node;
pipeline.QueryGeoNode(PXCMGesture.GeoNode.LABEL_BODY
_HAND_PRIMARY, out node);

17. Intel Confidential17
Retrieve Gesture Data
• Data Is Retrieved via QueryGesture()
– Use
PXCGesture::GeoNode::Label, PXCGesture::Gest
ure::Label and PXCGesture::Gesture to retrieve
data
• Unity
PXCMGesture.Gesture gest;
pipeline.QueryGesture(PXCMGesture.GeoNode.LABEL_BODY
_HAND_PRIMARY, out gest);
if(gest.label==PXCMGesture.Gesture.Label.LABEL_HA
ND_WAVE)…

18. Intel Confidential18
Clean Up
• Use ReleaseFrame() To “Free Up The
Pipeline”
pipeline.ReleaseFrame();

19. Intel Confidential19
What next ?
http://perchackparis.eventbrite.fr/
https://perceptualchallenge.intel.com/