Perceptual Computing SDK
Sulamita Garcia, Technical
Marketing Engineer, Intel
sulamita.garcia@intel.com
@sulagarcia
Intel Confidential
/me
@sulagarcia
Intel Confidential
Intel®
Perceptual Computing SDK 2013
Next Generation Interactivity for
Intel® Core™ Processor-Based
App...
Intel Confidential
What is Perceptual Computing?
Interactivity Beyond Touch, Mouse and Keyboard …
Facial
Tracking
Speech
R...
Intel Confidential
Creative* Interactive Gesture Camera
For use with the Intel® Perceptual Computing SDK
 Small, light-we...
Intel Confidential
SDK Usage H/W Requirements
SDK Usage
Mode
Speech
Certified Dual-
Array
Microphones
RGB
Webcam
Creative*...
Intel Confidential
Programming Language and Framework
Support
• C++, C#, Java
• Supported Frameworks
– processing
– openFr...
Intel Confidential
Programming Language and Framework
Support
• Unsupported but Verified
– Cinder
– OGRE
– XNA / Monogame
...
Intel Confidential
PXCSession, PXCImage, PXCAudio, PXCCapture, PXCGesture,
PXCFaceAnalysis, PXCVoice
UtilCapture, UtilPipe...
Intel Confidential
SDK Features
• Video Capture
– RGB (VGA and HD)
– Depth
– Blobs
– IR/Confidence
Intel Confidential
SDK Features
• User Tracking
– Hand Detection
– Finger Detection
– Static Pose Detection
– Dynamic Gest...
Intel Confidential
X
Z
Y
Hands and fingers tracking
GeoNode:
– PXCPoint3DF32 positionWorld;
– PXCPoint3DF32 positionImage;...
Intel Confidential
SDK Features
• User Tracking
– Face Detection
– Face Location
Detection
– Face Feature Detection
– Face...
Intel Confidential
User Experience considerations
• Reality inspired, not cloning
• Consistency!!!
• Extensible – prepare ...
Intel Confidential
Intel Confidential
Resources
• Perceptual Computing Forums
– http://software.intel.com/en-us/forums/intel-perceptual-
comp...
Intel Confidential
And what about HTML5?
• Not that simple…
• No plans for HTML/JS SDK yet
• But there is always a workaro...
Intel Confidential
Questions?
Demos available!
Backup
Intel Confidential
Process Overview
• Declare The SDK Object
• Select Features and Initialize
• Capture A Frame
• Retrieve...
Intel Confidential
Declare The SDK Object – C++/C#
• C++
UtilPipeline pipeline;
• C#
UtilMPipeline pipeline;
pipeline = ne...
Intel Confidential
Declare The SDK Object - Frameworks
• Unity
private PXCUPipeline pipeline;
pipeline = new PXCUPipeline(...
Intel Confidential
Select Features and Initialize – C++/C#
• Select Features Using .Enable*() Methods
• Use Init() To Set ...
Intel Confidential
Select Features and Initialize - Frameworks
• Select Features using PXCUPipeline.Mode enum
• Use Bitwis...
Intel Confidential
Capture A Frame
• Poll For A Frame Using AcquireFrame(bool);
– Can be blocking or non-blocking
– Acquir...
Intel Confidential
Retrieve The Data
• Data Is Retrieved via Query*(<T>)
– QueryRGB(), QueryLabelMapAsImage(), etc…
• Unit...
Intel Confidential
Clean Up
• Use ReleaseFrame() To “Free Up The Pipeline”
pipeline.ReleaseFrame();
Intel Confidential
Hello World – C++
class GesturePipeline: public UtilPipeline {
public:
GesturePipeline(void):UtilPipeli...
Intel Confidential
Gesture recognition
~30cm
Blob Intermediate images, help separating:
• Background
• Hands
GeoNode Skele...
Intel Confidential
Select Features and Initialize – C++
• Only Need To Enable Gestures, Images Optional for
Feedback or Vi...
Intel Confidential
Select Features and Initialize - Frameworks
• Only Need „GESTURE‟, Images Optional for Feedback
or Visu...
Intel Confidential
Key Concepts
• PXCGesture
– Gesture and node tracking interface
• PXCGesture::Gesture
– Gesture, struct...
Intel Confidential
class PXCGesture: public PXCBase
{
struct GeoNode {} // geometric node data structure
struct Gesture {}...
Intel Confidential
Retrieve The Data - C++
• Data Is Retrieved via The PXCGesture Object/Interface
PXCGesture *gesture = p...
Intel Confidential
Retrieve The Data - C#
PXCMGesture gesture = pipeline.QueryGesture();
if(gesture)
{
PXCMGesture.Gesture...
Intel Confidential
Retrieve The Data - Frameworks
• Unity
PXCMGesture.Gesture gest;
pipeline.QueryGesture(PXCMGesture.GeoN...
Intel Confidential
Accessing Gesture Data
• C++
if(gest->active)
{
if(gest->label==PXCGesture::Gesture::LABEL_HAND_WAVE)
{...
Intel Confidential
In This Section
• Tracking Hands
• Tracking Fingers
Intel Confidential
Key Concepts
• PXCGesture
– Gesture and node tracking interface
• PXCGesture::GeoNode
– Geometric Node,...
Intel Confidential
Retrieve The Geonodes data – C++
PXCGesture gesture = pipeline->QueryGesture();
if(gesture)
{
PXCGestur...
Intel Confidential
Retrieve The Geonodes data - C#
PXCMGesture gesture = pipeline.QueryGesture();
if(gesture)
{
PXCMGestur...
Intel Confidential
Retrieve The Geonodes Data - Frameworks
• Unity
PXCMGesture.GeoNode node;
pipeline.QueryGeoNode(PXCMGes...
Intel Confidential
Available Video Streams
• RGB
– VGA (640x480)
– HD (1280x720)
– 30 FPS
Intel Confidential
Available Video Streams
• Labels/Blobs
– QVGA (320x240)
– 30/60 FPS
Intel Confidential
Available Video Streams
• Depth
– QVGA (320x240)
– Must Convert to Color
Space
– 30/60 FPS
Intel Confidential
Available Video Streams
• IR
– QVGA (320x240)
– Must Convert to Color
Space
– 30/60 FPS
– 16 bits
Intel Confidential
Retrieve The Data (C++)
• Data Is Retrieved via QueryImage() And Accessing The
Data Buffers
• Image Is ...
Intel Confidential
Retrieve The Data (C#)
• Image Data Can Be Retrieved Via QueryBitmap()
System.Drawing.Bitmap rgb;
PXCMI...
Intel Confidential
Retrieve The Data - Frameworks
• Image Data Can Be Retrieved Via QueryBitmap()
System.Drawing.Bitmap rg...
Intel Confidential
• Track any 2D planar surfaces
– Position, orientation and other
parameters
• Track limited 3D objects
...
Intel Confidential
Algorithm Modules: PXCDVTracker
D’Fusion Studio Computer Vision – PerC SDK version
Intel Confidential
Algorithm Modules: PXCDVTracker
D’Fusion Studio Computer Vision – PerC SDK version
Intel Confidential
class PXCDVTracker: public PXCBase
{
enum TargetType
{
TARGET_UNDEFINED,
TARGET_PLANE,
TARGET_OBJECT3D,...
Intel Confidential
Face Detection/Tracking
•Locate and track
multiple faces
•Unique identifier
for each face
Algorithm Mod...
Intel Confidential
class PXCFaceAnalysis: public PXCBase
{
class Detection {
QueryProfile(…);
SetProfile(…);
QueryData(…);...
Intel Confidential
• Nuance* Voice Command and Control
– Recognize from a list of predefined commands
• Nuance Voice Dicta...
Intel Confidential
class PXCVoiceRecognition: public PXCBase
{
struct Recognition {} // Recognized data structure
struct A...
Intel Confidential
class MyHandler: public PXCVoiceRecognition::Recognition::Handler, public PXCVoiceRecognition::Alert::H...
Intel Confidential
class PXCVoiceSynthesis: public PXCBase
{
public:
PXC_CUID_OVERWRITE(PXC_UID('V','I','T','S'));
struct ...
Intel Confidential
// Queue the sentence to the speech synthesis module
pxcUID tuid = 0;
sts = vtts->QueueSentence(cmdl.m_...
Intel Confidential
class aPXCInterface: public PXCBase {
public:
PXC_CUID_OVERWRITE(PXC_UID(‘M’,’Y’,’I’,’F’));
// configur...
Intel Confidential
• Users are notified when SDK
accesses Personally Identifiable
Information (PII)
• Can also launch a vi...
Intel Confidential
• Image Capture:
– 8-bit RGB in RGBA/RGB24/NV12/YUY2
– Creative* camera supports up to 1280x720@30p.
– ...
Intel Confidential
1. Enumerate and create capture device
QueryDevice Query capture device names
CreateDevice Create a cap...
Intel Confidential
SetDeviceProperty/QueryDevice
Property
• Color stream properties
– CONTRAST, BRIGHTNESS, HUE, SATURATIO...
Intel Confidential
• Alert and callback interface used for low-
frequency events and notifications
• Subscribe to events
P...
Intel Confidential
class MyPipeline: public UtilPipeline {
public:
MyPipeline(void):UtilPipeline() {
EnableGesture();
}
vi...
Intel Confidential
• Multiple processing modules on single
input device
– Live streaming or file-based
recording/playback
...
Intel Confidential
Speech Recognition:
Voice command and control, short sentence dictation,
and text to speech synthesis
S...
Intel Confidential
Your SDK „One-Stop-Shop”
intel.com/software/perceptual
@PerceptualSDK (Twitter)
CHALLENGE INFO
DOWNLOAD...
Intel Confidential
Key Upcoming Items
 Creative* Senz3D – Q3 2013
 Integration in Intel devices – H2 2014
*Other brands ...
Intel Confidential
Intel® Perceptual Computing Challenge
The $1Million 2013 Application Development Contest*
Enter Phase 2...
Intel Confidential
Speech Recognition & Dragon Assistant*
Perceptual
Computing SDK
Runtime
Speech
Recognition
Application
...
Intel Confidential
Sample Snippet (processing)
• Declarations
import intel.pcsdk.*;
PXCUPipeline pxc;
int[] cm = new int[2...
Intel Confidential
Sample Snippet (processing)
• Initialization
void setup()
{
pxc = new PXCUPipeline(this);
if(!pxc.Init(...
Intel Confidential
Sample Snippet (processing)
• Main Loop
void draw()
{
if(pxc.AcquireFrame(false))
{
pxc.QueryRGB(rgb);
...
Intel Confidential
• Color streams
 RGB24 640x480 25fps, 30fps
 RGB24 640x360 25fps, 30fps
 RGB24 1280x720 25fps, 30fps...
Intel Confidential
Image Conversion
Visual Computing
Products7
RGB24 RGB32 NV12 YUY2 GRAY
RGB24 Y Y Y
RGB32 Y Y Y
NV12 Y Y...
Upcoming SlideShare
Loading in …5
×

Ultracode Berlin #2 : Introduction to Perceptual Computing by Sulamita Garcia

2,432 views
2,181 views

Published on

Sulamita Garcia (@sulagarcia) introducing Perceptual Computing to the Ultracode Berlin meetup @ MobileSuite

http://www.meetup.com/Ultracode-Berlin/

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,432
On SlideShare
0
From Embeds
0
Number of Embeds
32
Actions
Shares
0
Downloads
38
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • QueryProfile to enumerate all supported configurationsQueryProfile(WORKING_PROFILE, [out] pinfo) to retrieve the current working parameters.SetProfile to set the desired configuration.
  • TYZX: Zero as low-confidence. No Saturation.AR: Xintian to send out the definitions in the Creative camera.AR: Gaile/Tzachi please send TYZX/IVCAM details.
  • Ultracode Berlin #2 : Introduction to Perceptual Computing by Sulamita Garcia

    1. 1. Perceptual Computing SDK Sulamita Garcia, Technical Marketing Engineer, Intel sulamita.garcia@intel.com @sulagarcia
    2. 2. Intel Confidential /me @sulagarcia
    3. 3. Intel Confidential Intel® Perceptual Computing SDK 2013 Next Generation Interactivity for Intel® Core™ Processor-Based Applications
    4. 4. Intel Confidential What is Perceptual Computing? Interactivity Beyond Touch, Mouse and Keyboard … Facial Tracking Speech Recognition Close-range Finger Tracking Augmented Reality Close-range Gesture Tracking Facilitates Application Developers Implementation of:  Games  Entertainment  Productivity  Accessibility  Immersive Teleconferencing  Education  Medical / Health  Enterprises  Retail  Industrial
    5. 5. Intel Confidential Creative* Interactive Gesture Camera For use with the Intel® Perceptual Computing SDK  Small, light-weight, low-power  Tuned for close-range interactivity  Designed with ease of setup and portability  Includes:  HD web camera  Depth sensor  Dual-array microphone Sign up to purchase a camera at intel.com/software/perceptual*Other brands and trademarks may be claimed as the property of their respective owners
    6. 6. Intel Confidential SDK Usage H/W Requirements SDK Usage Mode Speech Certified Dual- Array Microphones RGB Webcam Creative* Camera Close-range Depth tracking X Speech Recognition X X Facial Tracking X X Augmented Reality X X  Close-range depth tracking requires Creative camera  Speech requires dual-array microphones OR Creative* camera  2H’13 4th Gen Ultrabook devices are required to have speech certified microphones  Dell XPS 13* has speech-certified microphones  Facial tracking requires RGB Webcam OR Creative* Camera  Augmented Reality requires RGB Webcam OR Creative* Camera *Other brands and trademarks may be claimed as the property of their respective owners
    7. 7. Intel Confidential Programming Language and Framework Support • C++, C#, Java • Supported Frameworks – processing – openFrameworks – Unity – Havok – Total Immersion AR
    8. 8. Intel Confidential Programming Language and Framework Support • Unsupported but Verified – Cinder – OGRE – XNA / Monogame – Bullet Physics
    9. 9. Intel Confidential PXCSession, PXCImage, PXCAudio, PXCCapture, PXCGesture, PXCFaceAnalysis, PXCVoice UtilCapture, UtilPipeline C# PXCMSession PXCMImage PXCMAudio PXCMCapture PXCMGesture PXCMFaceAn alysis PXCMVoice UtilMCapture UtilMPipeline pxcupipeline Unity* Pro Processing openFrame works* Applications Core Functionalities Module Interaction Additional Language and Framework Support SDK API Hierarchy *Other brands and trademarks may be claimed as the property of their respective owners
    10. 10. Intel Confidential SDK Features • Video Capture – RGB (VGA and HD) – Depth – Blobs – IR/Confidence
    11. 11. Intel Confidential SDK Features • User Tracking – Hand Detection – Finger Detection – Static Pose Detection – Dynamic Gesture Detection
    12. 12. Intel Confidential X Z Y Hands and fingers tracking GeoNode: – PXCPoint3DF32 positionWorld; – PXCPoint3DF32 positionImage; – pxcU64 timeStamp; – pxcU32 confidence; – pxcF32 radius; – Label body; – PXCPoint3DF32 normal; – pxcU32 openness; o HAND_FINGERTIP PXCGesture::GeoNode::LABEL_BODY_HAND_* PXCGesture::GeoNode::LABEL_FINGER_* o HAND_UPPER o HAND_MIDDLE o HAND_LOWER
    13. 13. Intel Confidential SDK Features • User Tracking – Face Detection – Face Location Detection – Face Feature Detection – Face Recognition
    14. 14. Intel Confidential User Experience considerations • Reality inspired, not cloning • Consistency!!! • Extensible – prepare for future improvements • Manage persistence • Prevent Occlusion!!! • Give instant feedback acknowledging command • Show what‟s actionable
    15. 15. Intel Confidential
    16. 16. Intel Confidential Resources • Perceptual Computing Forums – http://software.intel.com/en-us/forums/intel-perceptual- computing-sdk • Perceptual Computing IDZ Portal – http://intel.com/software/perceptual • Github – http://github.com/IntelPerceptual
    17. 17. Intel Confidential And what about HTML5? • Not that simple… • No plans for HTML/JS SDK yet • But there is always a workaround 
    18. 18. Intel Confidential Questions? Demos available!
    19. 19. Backup
    20. 20. Intel Confidential Process Overview • Declare The SDK Object • Select Features and Initialize • Capture A Frame • Retrieve The Data – Convert if necessary • Cleanup
    21. 21. Intel Confidential Declare The SDK Object – C++/C# • C++ UtilPipeline pipeline; • C# UtilMPipeline pipeline; pipeline = new UtilMPipeline();
    22. 22. Intel Confidential Declare The SDK Object - Frameworks • Unity private PXCUPipeline pipeline; pipeline = new PXCUPipeline(); • Processing private PXCUPipeline pipeline; pipeline = new PXCUPipeline(this);
    23. 23. Intel Confidential Select Features and Initialize – C++/C# • Select Features Using .Enable*() Methods • Use Init() To Set Features and Enable SDK Access pipeline.EnableGesture(); pipeline.EnableImage(PXCImage::COLOR_FORMAT_RGB24); //C# PXCImage.ColorFormat.COLOR_FORMAT_RGB24 pipeline.Init();
    24. 24. Intel Confidential Select Features and Initialize - Frameworks • Select Features using PXCUPipeline.Mode enum • Use Bitwise OR (|) for Multiple Features • Use Init() To Set Features and Enable SDK Access pipeline.Init(PXCUPipeline.Mode.COLOR_VGA| PXCUPipeline.Mode.DEPTH_QVGA| PXCUPipeline.Mode.GESTURE);
    25. 25. Intel Confidential Capture A Frame • Poll For A Frame Using AcquireFrame(bool); – Can be blocking or non-blocking – AcquireFrame(true) is blocking, AcquireFrame(false) is non- blocking • Returns true If A Frame Is Available if(pipeline.AcquireFrame(false)) { }
    26. 26. Intel Confidential Retrieve The Data • Data Is Retrieved via Query*(<T>) – QueryRGB(), QueryLabelMapAsImage(), etc… • Unity Texture2D rgbTexture = new Texture2D(640,480,TextureFormat.ARGB32, false); Pipeline.QueryRGB(rgbTexture); • processing PImage rgbTexture = createImage(640,480,RGB); pipeline.QueryRGB(rgbTexture);
    27. 27. Intel Confidential Clean Up • Use ReleaseFrame() To “Free Up The Pipeline” pipeline.ReleaseFrame();
    28. 28. Intel Confidential Hello World – C++ class GesturePipeline: public UtilPipeline { public: GesturePipeline(void):UtilPipeline(), m_render(L"Gesture Viewer") { EnableGesture(); } virtual void PXCAPI OnGesture(PXCGesture::Gesture *data) { if (data->active) m_gdata = (*data); switch (data->label) { case PXCGesture::Gesture::LABEL_NAV_SWIPE_LEFT: break; //do something case PXCGesture::Gesture::LABEL_NAV_SWIPE_RIGHT: break; //do something default: break; } } virtual bool OnNewFrame(void) { return m_render.RenderFrame(QueryImage(PXCImage::IMAGE_TYPE_DEPTH), QueryGesture(), &m_gdata); } protected: GestureRender m_render; PXCGesture::Gesture m_gdata; };
    29. 29. Intel Confidential Gesture recognition ~30cm Blob Intermediate images, help separating: • Background • Hands GeoNode Skeleton nodes • Hand openness • Open Hand: Fingertips, middle, elbows • Closed Hand: up, middle, bottom Gesture • THUMB UP/DOWN, PEACE, BIG5 • WAVE, CIRCLE, SWIPE LEFT/RIGHT/UP/DOWN Alert • FOV_LEFT/_RIGHT/_TOP/_BOTTOM • FOV_BLOCKED/_OK • GEONODE_ACTIVE/INACTIVE
    30. 30. Intel Confidential Select Features and Initialize – C++ • Only Need To Enable Gestures, Images Optional for Feedback or Visualization pipeline.EnableGesture(); pipeline.Init();
    31. 31. Intel Confidential Select Features and Initialize - Frameworks • Only Need „GESTURE‟, Images Optional for Feedback or Visualization pipeline.Init(PXCUPipeline.Mode.GESTURE);
    32. 32. Intel Confidential Key Concepts • PXCGesture – Gesture and node tracking interface • PXCGesture::Gesture – Gesture, struct for detected gestures • PXCGesture::Gesture::Label – Enum indicating detected gesture
    33. 33. Intel Confidential class PXCGesture: public PXCBase { struct GeoNode {} // geometric node data structure struct Gesture {} // pose/gesture data structure struct Blob {} // label map data structure struct Alert {} // event data structure QueryProfile(…); // Retrieve configuration(s) SetProfile(…); // Set active configuration SubscribeGesture(Gesture::Handler); // Event setup SubscribeAlert(Alert::Handler); // Event setup ProcessImageAsync(images, …); // Data processing QueryGestureData(…);// Retrieve pose/gesture data QueryNodeData(…); // Retrieve geometric node data QueryBlobData(…); // Retrieve label map data }; Algorithm Modules: PXCGesture Module interface
    34. 34. Intel Confidential Retrieve The Data - C++ • Data Is Retrieved via The PXCGesture Object/Interface PXCGesture *gesture = pipeline.QueryGesture(); if(gesture) { PXCGesture::GeoNode *node; gesture->QueryNodeData(0, PXCGesture::GeoNode::LABEL_BODY_HAND_PRIMARY, node); }
    35. 35. Intel Confidential Retrieve The Data - C# PXCMGesture gesture = pipeline.QueryGesture(); if(gesture) { PXCMGesture.Gesture gest; gesture.QueryGestureData(0, PXCMGesture.GeoNode.Label.LABEL_BODY_HAND_PRIMARY,0, out gest); }
    36. 36. Intel Confidential Retrieve The Data - Frameworks • Unity PXCMGesture.Gesture gest; pipeline.QueryGesture(PXCMGesture.GeoNode.LABEL_BODY _HAND_PRIMARY, out gest); • processing PXCMGesture.Gesture gest = new PXCMGesture.Gesture(); pipeline.QueryGesture(PXCMGesture.GeoNode.LABEL_BODY _HAND_PRIMARY, gest);
    37. 37. Intel Confidential Accessing Gesture Data • C++ if(gest->active) { if(gest->label==PXCGesture::Gesture::LABEL_HAND_WAVE) { //Do stuff! } } • C#/Frameworks: if(gest.active) { if(gest.label==PXCMGesture.Gesture.Label.LABEL_HAND_WAVE) { //Do stuff! } }
    38. 38. Intel Confidential In This Section • Tracking Hands • Tracking Fingers
    39. 39. Intel Confidential Key Concepts • PXCGesture – Gesture and node tracking interface • PXCGesture::GeoNode – Geometric Node, struct for tracking data • PXCGesture::GeoNode::Label – Enum indicating tracked node
    40. 40. Intel Confidential Retrieve The Geonodes data – C++ PXCGesture gesture = pipeline->QueryGesture(); if(gesture) { PXCGesture::GeoNode node; gesture->QueryNodeData(0, PXCGesture::GeoNode::LABEL_BODY_HAND_PRIMARY, &node); }
    41. 41. Intel Confidential Retrieve The Geonodes data - C# PXCMGesture gesture = pipeline.QueryGesture(); if(gesture) { PXCMGesture.GeoNode node; gesture.QueryNodeData(0, PXCMGesture.GeoNode.Label.LABEL_BODY_HAND_PRIMARY, out node); }
    42. 42. Intel Confidential Retrieve The Geonodes Data - Frameworks • Unity PXCMGesture.GeoNode node; pipeline.QueryGeoNode(PXCMGesture.GeoNode.LABEL_BODY _HAND_PRIMARY, out node); • processing PXCMGesture.GeoNode node = new PXCMGesture.GeoNode(); pipeline.QueryGeoNode(PXCMGesture.GeoNode.LABEL_BODY _HAND_PRIMARY, node);
    43. 43. Intel Confidential Available Video Streams • RGB – VGA (640x480) – HD (1280x720) – 30 FPS
    44. 44. Intel Confidential Available Video Streams • Labels/Blobs – QVGA (320x240) – 30/60 FPS
    45. 45. Intel Confidential Available Video Streams • Depth – QVGA (320x240) – Must Convert to Color Space – 30/60 FPS
    46. 46. Intel Confidential Available Video Streams • IR – QVGA (320x240) – Must Convert to Color Space – 30/60 FPS – 16 bits
    47. 47. Intel Confidential Retrieve The Data (C++) • Data Is Retrieved via QueryImage() And Accessing The Data Buffers • Image Is Retrieved as PXCImage, Data Is Accessed Via PXCImage::ImageData.planes PXCImage rgb = pipeline.QueryImage(PXCImage::IMAGE_TYPE_COLOR); PXCImage::ImageData rgbData; rgb->AcquireAccess(PXCImage::ACCESS_READ, &rgbData); //Data can be loaded from rgbData.planes[0] rgb->ReleaseAccess(&rgbData);
    48. 48. Intel Confidential Retrieve The Data (C#) • Image Data Can Be Retrieved Via QueryBitmap() System.Drawing.Bitmap rgb; PXCMImage rgbImage = pipeline.QueryImage(PXCMImage.ImageType.IMAGE_TYPE_COLOR); rgbImage.QueryBitmap(pipeline.QuerySession(), out rgb);
    49. 49. Intel Confidential Retrieve The Data - Frameworks • Image Data Can Be Retrieved Via QueryBitmap() System.Drawing.Bitmap rgb; PXCMImage rgbImage = pipeline.QueryImage(PXCMImage.ImageType.IMAGE_TYPE_COLOR); rgbImage.QueryBitmap(pipeline.QuerySession(), out rgb);
    50. 50. Intel Confidential • Track any 2D planar surfaces – Position, orientation and other parameters • Track limited 3D objects – Based on 3D models • Track face orientation SDK Features
    51. 51. Intel Confidential Algorithm Modules: PXCDVTracker D’Fusion Studio Computer Vision – PerC SDK version
    52. 52. Intel Confidential Algorithm Modules: PXCDVTracker D’Fusion Studio Computer Vision – PerC SDK version
    53. 53. Intel Confidential class PXCDVTracker: public PXCBase { enum TargetType { TARGET_UNDEFINED, TARGET_PLANE, TARGET_OBJECT3D, TARGET_FACE, TARGET_PLANEBLACKBOX, TARGET_MARKER }; typedef struct { TrackingStatus status; // (-1) not initialized, 0 not tracking (recognition in process), 1 tracking pxcF64 position[3]; // Resulting pose (X,Y, Z) pxcF64 orientation[4]; // Quaternion to express the orientation int index; // Recognized keyFrame index (-1 none) } TargetData; QueryProfile(…); // Retrieve configuration(s) SetProfile(…); // Set active configuration GetTargetCount(…); // ActivateTarget(…); // Retrieve object tracking data GetTargetData(…); // ProcessImageAsync(…); // Data processing }; Algorithm Modules: PXCDVTracker Module Interface
    54. 54. Intel Confidential Face Detection/Tracking •Locate and track multiple faces •Unique identifier for each face Algorithm Modules: PXCFaceAnalysis Face tracking and analysis Landmark Detection •6/7-point detection including eyes, nose, and mouth Facial Attribute Detection •Age-group including baby/youth/adult/senior •Gender detection •Smile/blink detection Face Recognition •Similarity among a set of faces
    55. 55. Intel Confidential class PXCFaceAnalysis: public PXCBase { class Detection { QueryProfile(…); SetProfile(…); QueryData(…); }; class Landmark { QueryProfile(…); SetProfile(…); QueryLandmarkData(…); QueryPoseData(…); }; class Recognition { QueryProfile(…); SetProfile(…); CreateModel(…); }; class Attribute { QueryProfile(…); SetProfile(…); QueryData(…); }; QueryProfile(…); SetProfile(…); ProcessImageAsync(…); } Algorithm Modules: PXCFaceAnalysis Module interface Face location detection/tracking configuration and retrieve data Face landmark detection configuration and data retrieval Face attribute detection configuration and data retrieval Face recognition confirmation and data retrieval Face analysis overall configuration and data processing
    56. 56. Intel Confidential • Nuance* Voice Command and Control – Recognize from a list of predefined commands • Nuance Voice Dictation – Recognize short sentences (<30 seconds) • Nuance Voice Synthesis – Text to speech for short sentences SDK Features
    57. 57. Intel Confidential class PXCVoiceRecognition: public PXCBase { struct Recognition {} // Recognized data structure struct Alert {} // Event data structure QueryProfile(…); // Retrieve configuration(s) SetProfile(…); // Set active configuration SubscribeRecognition(…);// Recognition event setup SubscribeAlert(…); // Alert event setup CreateGrammar(…); // AddGrammar(…); // Command list construction SetGrammar(…); // DeleteGrammar(…); // ProcessAudioAsync(…); // Data processing }; Algorithm Modules: PXCVoiceRecognition Module Interface
    58. 58. Intel Confidential class MyHandler: public PXCVoiceRecognition::Recognition::Handler, public PXCVoiceRecognition::Alert::Handler { public: MyHandler(std::vector<pxcCHAR*> &commands) { this->commands = commands; } virtual void PXCAPI OnRecognized(PXCVoiceRecognition::Recognition *cmd) { wprintf_s(L"nRecognized: <%s>n", (cmd->label>=0)?commands[cmd->label]:cmd->dictation); } virtual void PXCAPI OnAlert(PXCVoiceRecognition::Alert *alert) { switch (alert->label) { case PXCVoiceRecognition::Alert::LABEL_SNR_LOW: wprintf_s(L"nAlert: <Low SNR>n"); break; case PXCVoiceRecognition::Alert::LABEL_VOLUME_LOW: wprintf_s(L"nAlert: <Low Volume>n"); break; case PXCVoiceRecognition::Alert::LABEL_VOLUME_HIGH: wprintf_s(L"nAlert: <High Volume>n"); break; default: wprintf_s(L"nAlert: <0x%x>n",alert->label); break; } Algorithm Modules: PXCVoiceSynthesis Voice recognition example – callback handlers
    59. 59. Intel Confidential class PXCVoiceSynthesis: public PXCBase { public: PXC_CUID_OVERWRITE(PXC_UID('V','I','T','S')); struct ProfileInfo { enum Language { LANGUAGE_US_ENGLISH=1, }; enum Voice { VOICE_MALE, VOICE_FEMALE, }; PXCCapture::AudioStream::DataDesc outputs; // output format, need bufferSize to limit the latency. Language language; Voice voice; pxcU32 reserved[6]; }; virtual pxcStatus PXCAPI QueryProfile(pxcU32 pidx, ProfileInfo *pinfo)=0; pxcStatus __inline QueryProfile(ProfileInfo *pinfo) { return QueryProfile(WORKING_PROFILE,pinfo); } virtual pxcStatus PXCAPI SetProfile(ProfileInfo *pinfo)=0; virtual pxcStatus PXCAPI QueueSentense(pxcCHAR *sentence, pxcU32 nchars, pxcUID *id)=0; virtual pxcStatus PXCAPI ProcessAudioAsync(pxcUID id, PXCAudio **audio, PXCScheduler::SyncPoint **sp)=0; }; Algorithm Modules: PXCVoiceSynthesis Module Interface
    60. 60. Intel Confidential // Queue the sentence to the speech synthesis module pxcUID tuid = 0; sts = vtts->QueueSentence(cmdl.m_ttstext, wcslen(cmdl.m_ttstext), &tuid); … while (1) { PXCSmartPtr<PXCAudio> audio; PXCSmartSP sp; // Read audio frame sts = vtts->ProcessAudioAsync(tuid, &audio, &sp); if (sts < PXC_STATUS_NO_ERROR) break; sts = sp->Synchronize(); if (sts < PXC_STATUS_NO_ERROR) { if ((sts == PXC_STATUS_PARAM_UNSUPPORTED) || (sts == PXC_STATUS_EXEC_TIMEOUT)) wprintf_s(L"Error in ProcessAudion"); if (sts == PXC_STATUS_ITEM_UNAVAILABLE) wprintf_s(L"Voice synthesis completed successfullyn"); break; } return 0; Algorithm Modules: PXCVoiceSynthesis Speech synthesis example - generation
    61. 61. Intel Confidential class aPXCInterface: public PXCBase { public: PXC_CUID_OVERWRITE(PXC_UID(‘M’,’Y’,’I’,’F’)); // configurations & inquiries struct ProfileInfo { … }; virtual pxcStatus PXCAPI QueryProfile(pxcU32 idx, ProfileInfo *pinfo)=0; virtual pxcStatus PXCAPI SetProfile(ProfileInfo *pinfo)=0; // data processing virtual pxcStatus PXCAPI ProcessDataAsync(…, PXCScheduler::SyncPoint **sp)=0; }; Each interface has a unique ID used by PXCBase::QueryInterface Consistent way of querying and setting configurations Asynchronous execution returns SP for later synchronization Core: PXCSession Module interface conventions PXC interfaces derive from the PXCBase class SDK interfaces contain only pure virtual functions No exception handling or dynamic_cast (replaced with PXCBase::DynamicCast)
    62. 62. Intel Confidential • Users are notified when SDK accesses Personally Identifiable Information (PII) • Can also launch a viewer from the taskbar icon that shows any apps currently accessing the sensor and what, in particular, they are accessing Core: Privacy Notification Keeping users informed
    63. 63. Intel Confidential • Image Capture: – 8-bit RGB in RGBA/RGB24/NV12/YUY2 – Creative* camera supports up to 1280x720@30p. – 16-bit depthmap, confidence map and vertices. – Creative camera supports up to QVGA@60p – Depthmap smoothing by default • Audio capture: – 1-2 channel PCM/IEEE-Float audio streams – Creative camera supports 44.1kHz and 48KHz • Device properties: – Standard camera properties such as brightness and exposure. – Depth-related properties such as confidence threshold, depthmap value range etc. I/O Modules Audio and video capture
    64. 64. Intel Confidential 1. Enumerate and create capture device QueryDevice Query capture device names CreateDevice Create a capture device instance 2. Enumerate and select streams QueryStream Query stream type CreateVideoStream Select a video stream CreateAudioStream Select an audio stream 3. Perform stream operations QueryProfile Query stream configurations SetProfile Set a stream configuration ReadStreamAsync Read samples from the stream I/O Modules: PXCCapture PXCCapture interface hierarchy
    65. 65. Intel Confidential SetDeviceProperty/QueryDevice Property • Color stream properties – CONTRAST, BRIGHTNESS, HUE, SATURATION … • Depth stream properties – DEPTH_SMOOTHING, SATURATION_VALUE, … • Audio stream properties – MIX_LEVEL • Misc. properties – ACCELEROMETER_READING I/O Modules: PXCCapture Device properties
    66. 66. Intel Confidential • Alert and callback interface used for low- frequency events and notifications • Subscribe to events PXCGesture::SubscribeAlert PXCGesture::SubscribeGesture PXCVoiceCommand::SubscribeAlert PXCVoiceCommand::SubscribeCommand • Implement the callback handler Algorithm Modules: PXCGesture Alerts and callback notifications class Handler: public PXCBaseImpl<PXCGesture::Gesture::Handler> { public: virtual pxcStatus PXCAPI OnGesture(Gesture *gesture) { … } };
    67. 67. Intel Confidential class MyPipeline: public UtilPipeline { public: MyPipeline(void):UtilPipeline() { EnableGesture(); } virtual void PXCAPI OnGesture (PXCGesture::Gesture *data) { printf_s(‚%dn‛,data->label); } }; int wmain(int argc, WCHAR* argv[]) { MyPipeline pipeline; pipeline.LoopFrames(); return 0; } class MyPipeline: UtilMPipeline { public MyPipeline():base() { EnableGesture(); } public override void OnGesture (ref PXCMGesture.Gesture data) { Console.WriteLn(data.label); } }; class Program { static void Main(string[] args) { MyPipeline pipeline=new MyPipeline(); pipeline.LoopFrames(); pipeline.Dispose(); } } C++ C# Enable Finger Tracking Gesture Callback Data Flow Loops UtilPipeline Class Gesture Recognition “Hello World”
    68. 68. Intel Confidential • Multiple processing modules on single input device – Live streaming or file-based recording/playback – Synchronized image (or audio) processing UtilPipeline pp; pp.EnableImage(PXCImage::COLOR_FORMAT_RGB32); pp.EnableImage(PXCImage::COLOR_FORMAT_DEPTH); for (;;) { if (!pp.AcquireFrame(true)) break; PXCImage *color, *depth; color=pp.QueryImage(PXCImage::IMAGE_TYPE_COLOR); depth=pp.QueryImage(PXCImage::IMAGE_TYPE_DEPTH); pp.ReleaseFrame(); } pp.Close(); UtilPipeline Class UtilPipeline-based application Color and depth are synchronized
    69. 69. Intel Confidential Speech Recognition: Voice command and control, short sentence dictation, and text to speech synthesis SDK Usage Modes Today 1 1 New usage modes may be added in the future Close-range Depth Tracking (6 in. to 3 ft.): Recognize the positions of each of the user‟s hands, fingers, static hand poses and moving hand gestures. Facial Analysis: Face detection and recognition (six and seven point landmark and attribution detection, including smiles, blinks, and age groups) Augmented Reality: Combine real-time images from the camera and close-range tracking from the depth sensor with 2D or 3D graphical images.
    70. 70. Intel Confidential Your SDK „One-Stop-Shop” intel.com/software/perceptual @PerceptualSDK (Twitter) CHALLENGE INFO DOWNLOAD SDK ORDER CAMERA DOCUMENTS DEMO APPS SUPPORT
    71. 71. Intel Confidential Key Upcoming Items  Creative* Senz3D – Q3 2013  Integration in Intel devices – H2 2014 *Other brands and trademarks may be claimed as the property of their respective owners
    72. 72. Intel Confidential Intel® Perceptual Computing Challenge The $1Million 2013 Application Development Contest* Enter Phase 2: perceptualchallenge.intel.com/  Focus: Games, Productivity, Creative UI & Multi-modal  Process: Developers submit working prototypes, panel judged  Two Phases:  Phase 1 (CLOSED): See Winner Showcase at http://goo.gl/EnNHv  Phase 2: March (GDC) to September - $800,000+ in prizes  Categories: Perceptual Gaming, Productivity, Creative User Interface and Open Innovation  Available in 16 countries *Terms and Conditions Apply
    73. 73. Intel Confidential Speech Recognition & Dragon Assistant* Perceptual Computing SDK Runtime Speech Recognition Application Drivers & Hardware Dragon Assistant* Dragon Assistant* Engine and Language Pack • Perceptual Computing speech recognition applications require Dragon Assistant* Engine and Language Packs to be installed on target platform • For app developers, Engine and Language Packs are available on SDK download site (THESE ARE FOR DEVELOPER INTERNAL USE ONLY AND NOT TO BE DISTRIBUTED). • For consumers, Dragon Assistant* (with Engine) is expected to available as follows: • Expected to be bundled with Creative* Camera (when available) • Expected to be pre-installed on speech-certified 4th Gen Core Ultrabook devices in late 2013 SDK Speech APIs use the Dragon Assistant* Engine and Language Packs
    74. 74. Intel Confidential Sample Snippet (processing) • Declarations import intel.pcsdk.*; PXCUPipeline pxc; int[] cm = new int[2]; //color map dimensions int[] dm = new int[2]; //depth map dimensions short[] buffer; Pimage rgb, depth;
    75. 75. Intel Confidential Sample Snippet (processing) • Initialization void setup() { pxc = new PXCUPipeline(this); if(!pxc.Init(PXCUPipeline.Mode.COLOR_VGA|PXCUPipeline.Mode.GESTURE)) println(“Error initializing PerC SDK”); if(pxc.QueryRGBSize(cm)) rgb = createImate(cm[0], cm[1], RGB); if(pxc.QueryDepthMapSize(dm)) { buffer = new short[dm[0]*dm[1]]; depth = createImage(dm[0], dm[1], RGB); } size(640,480); }
    76. 76. Intel Confidential Sample Snippet (processing) • Main Loop void draw() { if(pxc.AcquireFrame(false)) { pxc.QueryRGB(rgb); pxc.QueryDepthMap(buffer); pxc.ReleaseFrame(); } RemapDepth(); image(rgb,0,0,320,240); image(depth,320,0); }
    77. 77. Intel Confidential • Color streams  RGB24 640x480 25fps, 30fps  RGB24 640x360 25fps, 30fps  RGB24 1280x720 25fps, 30fps • Depth streams (16-bit integer, 0-32000)  320x240 25fps, 30fps, 50fps, 60fps  UVMAP (Depth  Color)  Confidence Map (16-bit integer) • Vertices streams (real world coordinates in 3D fixed-point integers) • Audio streams (At least 2-array MIC)  44.1KHz mono/stereo Camera Streams Visual Computing Products7
    78. 78. Intel Confidential Image Conversion Visual Computing Products7 RGB24 RGB32 NV12 YUY2 GRAY RGB24 Y Y Y RGB32 Y Y Y NV12 Y Y Y YUY2 Y Y Y Y Y GRAY Y Y Y Y DEPTH Y VERTICES Y For instance, raw DepthSense color image format is RGB24, with AcquireAccess(PXCImage::ACCESS_READ, PXCImage::COLOR_FORMAT_RGB32, &data) SDK framework will convert color image data from RGB24 to RGB32

    ×