Perceptual Computing WorkshopXavier Hallade, Technical MarketingEngineer@ph0b
Intel ConfidentialIntel®Perceptual Computing SDK 2013Next Generation Interactivity for Intel®Core™ Processor-Based Applica...
Intel ConfidentialCreative* Interactive Gesture CameraFor use with the Intel® Perceptual Computing SDK Small, light-weigh...
Intel ConfidentialKey Upcoming Items Creative* Senz3D – Q3 2013 Integration in Intel devices – H2 2014*Other brands and ...
Intel ConfidentialWhat is Perceptual Computing?Interactivity Beyond Touch, Mouse and Keyboard …FacialTrackingSpeechRecogni...
Intel ConfidentialSDK Usage H/W RequirementsSDK Usage Mode Speech CertifiedDual-ArrayMicrophonesRGB Webcam Creative*Camera...
Intel ConfidentialProgramming Language and FrameworkSupport• C++, C#, Java• Supported Frameworks– processing– openFramewor...
Intel ConfidentialProgramming Language and FrameworkSupport• Unsupported but Verified– Cinder– OGRE– XNA / Monogame– Bulle...
Intel ConfidentialPXCSession, PXCImage, PXCAudio, PXCCapture, PXCGesture,PXCFaceAnalysis, PXCVoiceUtilCapture, UtilPipelin...
Intel ConfidentialSDK files• Samples are located in:– $(PCSDK_DIR)bin (executables)– $(PCSDK_DIR)samples (source code)– $(...
Intel ConfidentialResources• Perceptual Computing Forums– http://software.intel.com/en-us/forums/intel-perceptual-computin...
Intel ConfidentialUser Experience considerations• Reality inspired, not cloning– Wrapping fingers around objects• Literal,...
Intel ConfidentialProcess Overview – prop 1• Subclass the SDK Pipeline Object• Enable features in the constructor• Reimple...
Intel ConfidentialHello World – C++class GesturePipeline: public UtilPipeline {public:GesturePipeline(void):UtilPipeline()...
Intel ConfidentialProcess Overview – prop 2• Declare The SDK Pipeline Object• Select Features and Initialize• Acquire A Fr...
Intel ConfidentialDeclare The SDK Object – C++/C#• C++UtilPipeline pipeline;• C#UtilMPipeline pipeline;pipeline = new Util...
Intel ConfidentialSelect Features and Initialize – C++/C#• Select Features Using .Enable*() Methods• Use Init() To Set Fea...
Intel ConfidentialDeclare The SDK Object - Frameworks• Unityprivate PXCUPipeline pipeline;pipeline = new PXCUPipeline();• ...
Intel ConfidentialSelect Features and Initialize - Frameworks• Select Features using PXCUPipeline.Mode enum• Use Bitwise O...
Intel ConfidentialCapture A Frame• Poll For A Frame Using AcquireFrame(bool);– Can be blocking or non-blocking– AcquireFra...
Intel ConfidentialRetrieve The Data• Data Is Retrieved via Query*(<T>)– QueryRGB(), QueryLabelMapAsImage(), etc…• UnityTex...
Intel ConfidentialClean Up• Use ReleaseFrame() To “Free Up The Pipeline”pipeline.ReleaseFrame();
Intel ConfidentialSDK Features• User Tracking– Hand Detection– Finger Detection– Static Pose Detection– Dynamic GestureDet...
Intel ConfidentialUser experience on Gestures• Innate vs. learned gestures– Aim for lower cognitive load• Actionable gestu...
Intel ConfidentialIn This Section• Tracking Hands• Tracking Fingers
Intel ConfidentialKey Concepts• PXCGesture– Gesture and node tracking interface• PXCGesture::GeoNode– Geometric Node, stru...
Intel ConfidentialXZYHands and fingers trackingGeoNode:– PXCPoint3DF32 positionWorld;– PXCPoint3DF32 positionImage;– pxcU6...
Intel ConfidentialRetrieve The Geonodes data – C++PXCGesture gesture = pipeline->QueryGesture();if(gesture){PXCGesture::Ge...
Intel ConfidentialRetrieve The Geonodes data - C#PXCMGesture gesture = pipeline.QueryGesture();if(gesture){PXCMGesture.Geo...
Intel ConfidentialRetrieve The Geonodes Data - Frameworks• UnityPXCMGesture.GeoNode node;pipeline.QueryGeoNode(PXCMGesture...
Intel ConfidentialGesture recognition~30cmBlob Intermediate images, help separating:• Background• HandsGeoNode Skeleton no...
Intel ConfidentialSelect Features and Initialize – C++• Only Need To Enable Gestures, Images Optional forFeedback or Visua...
Intel ConfidentialSelect Features and Initialize - Frameworks• Only Need ‘GESTURE’, Images Optional for Feedback orVisuali...
Intel ConfidentialKey Concepts• PXCGesture– Gesture and node tracking interface• PXCGesture::Gesture– Gesture, struct for ...
Intel Confidentialclass PXCGesture: public PXCBase{struct GeoNode {} // geometric node data structurestruct Gesture {} // ...
Intel ConfidentialRetrieve The Data - C#PXCMGesture gesture = pipeline.QueryGesture();if(gesture){PXCMGesture.Gesture gest...
Intel ConfidentialRetrieve The Data - Frameworks• UnityPXCMGesture.Gesture gest;pipeline.QueryGesture(PXCMGesture.GeoNode....
Intel ConfidentialAccessing Gesture Data• C++if(gest->active){if(gest->label==PXCGesture::Gesture::LABEL_HAND_WAVE){//Do s...
Intel ConfidentialOther Resourceshttp://bit.ly/11PquG1 - sources Flyhttp://intel.ly/ZK14uR - Hand skeletal tracking
Intel ConfidentialSDK Features• Video Capture– RGB (VGA and HD)– Depth– Blobs– IR/Confidence
Intel ConfidentialAvailable Video Streams• RGB– VGA (640x480)– HD (1280x720)– 30 FPS
Intel ConfidentialAvailable Video Streams• Labels/Blobs– QVGA (320x240)– 30/60 FPS
Intel ConfidentialAvailable Video Streams• Depth– QVGA (320x240)– Must Convert to Color Space– 30/60 FPS
Intel ConfidentialAvailable Video Streams• IR– QVGA (320x240)– Must Convert to Color Space– 30/60 FPS– 16 bits
Intel ConfidentialRetrieve The Data (C++)• Data Is Retrieved via QueryImage() And Accessing TheData Buffers• Image Is Retr...
Intel ConfidentialRetrieve The Data (C#)• Image Data Can Be Retrieved Via QueryBitmap()System.Drawing.Bitmap rgb;PXCMImage...
Intel ConfidentialRetrieve The Data - Frameworks• Image Data Can Be Retrieved Via QueryBitmap()System.Drawing.Bitmap rgb;P...
Intel Confidential• Track any 2D planar surfaces– Position, orientation and otherparameters• Track limited 3D objects– Bas...
Intel ConfidentialAlgorithm Modules: PXCDVTrackerD’Fusion Studio Computer Vision – PerC SDK version
Intel ConfidentialAlgorithm Modules: PXCDVTrackerD’Fusion Studio Computer Vision – PerC SDK version
Intel Confidentialclass PXCDVTracker: public PXCBase{enum TargetType{TARGET_UNDEFINED,TARGET_PLANE,TARGET_OBJECT3D,TARGET_...
Intel ConfidentialSDK Features• User Tracking– Face Detection– Face Location Detection– Face Feature Detection– Face Recog...
Intel ConfidentialFace Detection/Tracking•Locate and trackmultiple faces•Unique identifierfor each faceAlgorithm Modules: ...
Intel Confidentialclass PXCFaceAnalysis: public PXCBase{class Detection {QueryProfile(…);SetProfile(…);QueryData(…);};clas...
Intel Confidential• Nuance* Voice Command and Control– Recognize from a list of predefined commands• Nuance Voice Dictatio...
Intel Confidentialclass PXCVoiceRecognition: public PXCBase{struct Recognition {} // Recognized data structurestruct Alert...
Intel Confidentialclass MyHandler: public PXCVoiceRecognition::Recognition::Handler, public PXCVoiceRecognition::Alert::Ha...
Intel Confidentialclass PXCVoiceSynthesis: public PXCBase{public:PXC_CUID_OVERWRITE(PXC_UID(V,I,T,S));struct ProfileInfo {...
Intel Confidential// Queue the sentence to the speech synthesis modulepxcUID tuid = 0;sts = vtts->QueueSentence(cmdl.m_tts...
Intel ConfidentialUser Experience on Face Recognition• More expressions will be available in the future SDK• Give feedback...
Intel ConfidentialOverall Visual feedback• Give instant feedback acknowledging command– Gesture, voice, or anything else• ...
Intel Confidentialhttps://perceptualchallenge.intel.com
Backup
Intel Confidentialclass aPXCInterface: public PXCBase {public:PXC_CUID_OVERWRITE(PXC_UID(‘M’,’Y’,’I’,’F’));// configuratio...
Intel Confidential• Users are notified when SDKaccesses Personally IdentifiableInformation (PII)• Can also launch a viewer...
Intel Confidential• Image Capture:– 8-bit RGB in RGBA/RGB24/NV12/YUY2– Creative* camera supports up to 1280x720@30p.– 16-b...
Intel Confidential1. Enumerate and create capture deviceQueryDevice Query capture device namesCreateDevice Create a captur...
Intel ConfidentialSetDeviceProperty/QueryDeviceProperty• Color stream properties– CONTRAST, BRIGHTNESS, HUE, SATURATION …•...
Intel Confidential• Alert and callback interface used for low-frequency events and notifications• Subscribe to eventsPXCGe...
Intel Confidentialclass MyPipeline: public UtilPipeline {public:MyPipeline(void):UtilPipeline() {EnableGesture();}virtual ...
Intel Confidential• Multiple processing modules on singleinput device– Live streaming or file-basedrecording/playback– Syn...
Intel ConfidentialSpeech Recognition:Voice command and control, short sentence dictation, andtext to speech synthesisSDK U...
Intel ConfidentialYour SDK ‘One-Stop-Shop”intel.com/software/perceptual@PerceptualSDK (Twitter)CHALLENGE INFODOWNLOAD SDKO...
Intel ConfidentialIntel® Perceptual Computing ChallengeThe $1Million 2013 Application Development Contest*Enter Phase 2: p...
Intel ConfidentialSpeech Recognition & Dragon Assistant*PerceptualComputing SDKRuntimeSpeech RecognitionApplicationDrivers...
Intel ConfidentialSample Snippet (processing)• Declarationsimport intel.pcsdk.*;PXCUPipeline pxc;int[] cm = new int[2]; //...
Intel ConfidentialSample Snippet (processing)• Initializationvoid setup(){pxc = new PXCUPipeline(this);if(!pxc.Init(PXCUPi...
Intel ConfidentialSample Snippet (processing)• Main Loopvoid draw(){if(pxc.AcquireFrame(false)){pxc.QueryRGB(rgb);pxc.Quer...
Intel Confidential• Color streams RGB24 640x480 25fps, 30fps RGB24 640x360 25fps, 30fps RGB24 1280x720 25fps, 30fps• De...
Intel ConfidentialImage ConversionVisual ComputingProductsRGB24 RGB32 NV12 YUY2 GRAYRGB24 Y Y YRGB32 Y Y YNV12 Y Y YYUY2 Y...
Upcoming SlideShare
Loading in …5
×

Perceptual Computing Workshop à Paris

719 views
625 views

Published on

By Xavier Halade (@ph0b)

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
719
On SlideShare
0
From Embeds
0
Number of Embeds
122
Actions
Shares
0
Downloads
0
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Perceptual Computing Workshop à Paris

  1. 1. Perceptual Computing WorkshopXavier Hallade, Technical MarketingEngineer@ph0b
  2. 2. Intel ConfidentialIntel®Perceptual Computing SDK 2013Next Generation Interactivity for Intel®Core™ Processor-Based Applications
  3. 3. Intel ConfidentialCreative* Interactive Gesture CameraFor use with the Intel® Perceptual Computing SDK Small, light-weight, low-power Tuned for close-range interactivity Designed with ease of setup andportability Includes: HD web camera Depth sensor Dual-array microphoneSign up to purchase a camera at intel.com/software/perceptual*Other brands and trademarks may be claimed as the property of their respective owners
  4. 4. Intel ConfidentialKey Upcoming Items Creative* Senz3D – Q3 2013 Integration in Intel devices – H2 2014*Other brands and trademarks may be claimed as the property of their respective owners
  5. 5. Intel ConfidentialWhat is Perceptual Computing?Interactivity Beyond Touch, Mouse and Keyboard …FacialTrackingSpeechRecognitionClose-rangeFingerTrackingAugmentedRealityClose-rangeGestureTrackingFacilitates Application Developers Implementation of: Games Entertainment Productivity Accessibility Immersive Teleconferencing Education Medical / Health Enterprises Retail Industrial
  6. 6. Intel ConfidentialSDK Usage H/W RequirementsSDK Usage Mode Speech CertifiedDual-ArrayMicrophonesRGB Webcam Creative*CameraClose-rangeDepth trackingXSpeechRecognitionX XFace Recognition X XAugmentedRealityX X Close-range depth tracking requires Creative camera Speech requires dual-array microphones OR Creative* camera 2H’13 4th Gen Ultrabook devices are required to have speechcertified microphones Dell XPS 13* has speech-certified microphones Facial tracking requires RGB Webcam OR Creative* Camera Augmented Reality requires RGB Webcam OR Creative* Camera*Other brands and trademarks may be claimed as the property of their respective owners
  7. 7. Intel ConfidentialProgramming Language and FrameworkSupport• C++, C#, Java• Supported Frameworks– processing– openFrameworks– Unity Pro– Havok
  8. 8. Intel ConfidentialProgramming Language and FrameworkSupport• Unsupported but Verified– Cinder– OGRE– XNA / Monogame– Bullet Physics
  9. 9. Intel ConfidentialPXCSession, PXCImage, PXCAudio, PXCCapture, PXCGesture,PXCFaceAnalysis, PXCVoiceUtilCapture, UtilPipelineC#PXCMSessionPXCMImagePXCMAudioPXCMCapturePXCMGesturePXCMFaceAnalysisPXCMVoiceUtilMCaptureUtilMPipelinepxcupipelineUnity*ProProcessingopenFrameworks*ApplicationsCore FunctionalitiesModule InteractionAdditional Language andFramework SupportSDK API Hierarchy*Other brands and trademarks may be claimed as the property of their respective owners
  10. 10. Intel ConfidentialSDK files• Samples are located in:– $(PCSDK_DIR)bin (executables)– $(PCSDK_DIR)samples (source code)– $(PCSDK_DIR)framework (framework and game engines)– $(PCSDK_DIR)demo
  11. 11. Intel ConfidentialResources• Perceptual Computing Forums– http://software.intel.com/en-us/forums/intel-perceptual-computing-sdk• Perceptual Computing IDZ Portal– http://intel.com/software/perceptual• Github– http://github.com/IntelPerceptual
  12. 12. Intel ConfidentialUser Experience considerations• Reality inspired, not cloning– Wrapping fingers around objects• Literal, not abstract– Universal visual cues – switches and nobs• Consistency!!!• Extensible – prepare for future improvements• Manage persistence– Sometimes the hand will go out of the view
  13. 13. Intel ConfidentialProcess Overview – prop 1• Subclass the SDK Pipeline Object• Enable features in the constructor• Reimplement OnAlert|Gesture|NewFrame…() methods• Instantiate the object and call LoopFrames() on it.
  14. 14. Intel ConfidentialHello World – C++class GesturePipeline: public UtilPipeline {public:GesturePipeline(void):UtilPipeline(), m_render(L"Gesture Viewer") {EnableGesture();}virtual void PXCAPI OnGesture(PXCGesture::Gesture *data) {if (data->active) m_gdata = (*data);switch (data->label) {case PXCGesture::Gesture::LABEL_NAV_SWIPE_LEFT: break; //do somethingcase PXCGesture::Gesture::LABEL_NAV_SWIPE_RIGHT: break; //do somethingdefault: break;}}virtual bool OnNewFrame(void) {return m_render.RenderFrame(QueryImage(PXCImage::IMAGE_TYPE_DEPTH),QueryGesture(), &m_gdata);}protected:GestureRender m_render;PXCGesture::Gesture m_gdata;};
  15. 15. Intel ConfidentialProcess Overview – prop 2• Declare The SDK Pipeline Object• Select Features and Initialize• Acquire A Frame• Query The Data• CleanupYou can also subclass the pipeline object and reimplement OnXXX methods
  16. 16. Intel ConfidentialDeclare The SDK Object – C++/C#• C++UtilPipeline pipeline;• C#UtilMPipeline pipeline;pipeline = new UtilMPipeline();
  17. 17. Intel ConfidentialSelect Features and Initialize – C++/C#• Select Features Using .Enable*() Methods• Use Init() To Set Features and Enable SDK Accesspipeline.EnableGesture();pipeline.EnableImage(PXCImage::COLOR_FORMAT_RGB24);//C# PXCImage.ColorFormat.COLOR_FORMAT_RGB24pipeline.Init();
  18. 18. Intel ConfidentialDeclare The SDK Object - Frameworks• Unityprivate PXCUPipeline pipeline;pipeline = new PXCUPipeline();• Processingprivate PXCUPipeline pipeline;pipeline = new PXCUPipeline(this);
  19. 19. Intel ConfidentialSelect Features and Initialize - Frameworks• Select Features using PXCUPipeline.Mode enum• Use Bitwise OR (|) for Multiple Features• Use Init() To Set Features and Enable SDK Accesspipeline.Init(PXCUPipeline.Mode.COLOR_VGA|PXCUPipeline.Mode.DEPTH_QVGA|PXCUPipeline.Mode.GESTURE);
  20. 20. Intel ConfidentialCapture A Frame• Poll For A Frame Using AcquireFrame(bool);– Can be blocking or non-blocking– AcquireFrame(true) is blocking, AcquireFrame(false) is non-blocking• Returns true If A Frame Is Availableif(pipeline.AcquireFrame(false)){}
  21. 21. Intel ConfidentialRetrieve The Data• Data Is Retrieved via Query*(<T>)– QueryRGB(), QueryLabelMapAsImage(), etc…• UnityTexture2D rgbTexture = newTexture2D(640,480,TextureFormat.ARGB32, false);Pipeline.QueryRGB(rgbTexture);• processingPImage rgbTexture = createImage(640,480,RGB);pipeline.QueryRGB(rgbTexture);
  22. 22. Intel ConfidentialClean Up• Use ReleaseFrame() To “Free Up The Pipeline”pipeline.ReleaseFrame();
  23. 23. Intel ConfidentialSDK Features• User Tracking– Hand Detection– Finger Detection– Static Pose Detection– Dynamic GestureDetection
  24. 24. Intel ConfidentialUser experience on Gestures• Innate vs. learned gestures– Aim for lower cognitive load• Actionable gestures– What if the user turns the head, drinks coffee…• Prevent Occlusion!!!• Left-right gestures easier than up-down• Grabbed objects– Make it obvious where objects can be dropped
  25. 25. Intel ConfidentialIn This Section• Tracking Hands• Tracking Fingers
  26. 26. Intel ConfidentialKey Concepts• PXCGesture– Gesture and node tracking interface• PXCGesture::GeoNode– Geometric Node, struct for tracking data• PXCGesture::GeoNode::Label– Enum indicating tracked node
  27. 27. Intel ConfidentialXZYHands and fingers trackingGeoNode:– PXCPoint3DF32 positionWorld;– PXCPoint3DF32 positionImage;– pxcU64 timeStamp;– pxcU32 confidence;– pxcF32 radius;– Label body;– PXCPoint3DF32 normal;– pxcU32 openness;o HAND_FINGERTIPPXCGesture::GeoNode::LABEL_BODY_HAND_*PXCGesture::GeoNode::LABEL_FINGER_*o HAND_UPPERo HAND_MIDDLEo HAND_LOWER
  28. 28. Intel ConfidentialRetrieve The Geonodes data – C++PXCGesture gesture = pipeline->QueryGesture();if(gesture){PXCGesture::GeoNode node;gesture->QueryNodeData(0,PXCGesture::GeoNode::LABEL_BODY_HAND_PRIMARY,&node);}
  29. 29. Intel ConfidentialRetrieve The Geonodes data - C#PXCMGesture gesture = pipeline.QueryGesture();if(gesture){PXCMGesture.GeoNode node;gesture.QueryNodeData(0,PXCMGesture.GeoNode.Label.LABEL_BODY_HAND_PRIMARY,out node);}
  30. 30. Intel ConfidentialRetrieve The Geonodes Data - Frameworks• UnityPXCMGesture.GeoNode node;pipeline.QueryGeoNode(PXCMGesture.GeoNode.LABEL_BODY_HAND_PRIMARY, out node);• processingPXCMGesture.GeoNode node = newPXCMGesture.GeoNode();pipeline.QueryGeoNode(PXCMGesture.GeoNode.LABEL_BODY_HAND_PRIMARY, node);
  31. 31. Intel ConfidentialGesture recognition~30cmBlob Intermediate images, help separating:• Background• HandsGeoNode Skeleton nodes• Hand openness• Open Hand: Fingertips, middle, elbows• Closed Hand: up, middle, bottomGesture • THUMB UP/DOWN, PEACE, BIG5• WAVE, CIRCLE, SWIPE LEFT/RIGHT/UP/DOWNAlert • FOV_LEFT/_RIGHT/_TOP/_BOTTOM• FOV_BLOCKED/_OK• GEONODE_ACTIVE/INACTIVE
  32. 32. Intel ConfidentialSelect Features and Initialize – C++• Only Need To Enable Gestures, Images Optional forFeedback or Visualizationpipeline.EnableGesture();pipeline.Init();
  33. 33. Intel ConfidentialSelect Features and Initialize - Frameworks• Only Need ‘GESTURE’, Images Optional for Feedback orVisualizationpipeline.Init(PXCUPipeline.Mode.GESTURE);
  34. 34. Intel ConfidentialKey Concepts• PXCGesture– Gesture and node tracking interface• PXCGesture::Gesture– Gesture, struct for detected gestures• PXCGesture::Gesture::Label– Enum indicating detected gesture
  35. 35. Intel Confidentialclass PXCGesture: public PXCBase{struct GeoNode {} // geometric node data structurestruct Gesture {} // pose/gesture data structurestruct Blob {} // label map data structurestruct Alert {} // event data structureQueryProfile(…); // Retrieve configuration(s)SetProfile(…); // Set active configurationSubscribeGesture(Gesture::Handler); // Event setupSubscribeAlert(Alert::Handler); // Event setupProcessImageAsync(images, …); // Data processingQueryGestureData(…);// Retrieve pose/gesture dataQueryNodeData(…); // Retrieve geometric node dataQueryBlobData(…); // Retrieve label map data};Algorithm Modules: PXCGestureModule interface
  36. 36. Intel ConfidentialRetrieve The Data - C#PXCMGesture gesture = pipeline.QueryGesture();if(gesture){PXCMGesture.Gesture gest;gesture.QueryGestureData(0,PXCMGesture.GeoNode.Label.LABEL_BODY_HAND_PRIMARY,0,out gest);}
  37. 37. Intel ConfidentialRetrieve The Data - Frameworks• UnityPXCMGesture.Gesture gest;pipeline.QueryGesture(PXCMGesture.GeoNode.LABEL_BODY_HAND_PRIMARY, out gest);• processingPXCMGesture.Gesture gest = newPXCMGesture.Gesture();pipeline.QueryGesture(PXCMGesture.GeoNode.LABEL_BODY_HAND_PRIMARY, gest);
  38. 38. Intel ConfidentialAccessing Gesture Data• C++if(gest->active){if(gest->label==PXCGesture::Gesture::LABEL_HAND_WAVE){//Do stuff!}}• C#/Frameworks:if(gest.active){if(gest.label==PXCMGesture.Gesture.Label.LABEL_HAND_WAVE){//Do stuff!}}
  39. 39. Intel ConfidentialOther Resourceshttp://bit.ly/11PquG1 - sources Flyhttp://intel.ly/ZK14uR - Hand skeletal tracking
  40. 40. Intel ConfidentialSDK Features• Video Capture– RGB (VGA and HD)– Depth– Blobs– IR/Confidence
  41. 41. Intel ConfidentialAvailable Video Streams• RGB– VGA (640x480)– HD (1280x720)– 30 FPS
  42. 42. Intel ConfidentialAvailable Video Streams• Labels/Blobs– QVGA (320x240)– 30/60 FPS
  43. 43. Intel ConfidentialAvailable Video Streams• Depth– QVGA (320x240)– Must Convert to Color Space– 30/60 FPS
  44. 44. Intel ConfidentialAvailable Video Streams• IR– QVGA (320x240)– Must Convert to Color Space– 30/60 FPS– 16 bits
  45. 45. Intel ConfidentialRetrieve The Data (C++)• Data Is Retrieved via QueryImage() And Accessing TheData Buffers• Image Is Retrieved as PXCImage, Data Is Accessed ViaPXCImage::ImageData.planesPXCImage rgb = pipeline.QueryImage(PXCImage::IMAGE_TYPE_COLOR);PXCImage::ImageData rgbData;rgb->AcquireAccess(PXCImage::ACCESS_READ, &rgbData);//Data can be loaded from rgbData.planes[0]rgb->ReleaseAccess(&rgbData);
  46. 46. Intel ConfidentialRetrieve The Data (C#)• Image Data Can Be Retrieved Via QueryBitmap()System.Drawing.Bitmap rgb;PXCMImage rgbImage =pipeline.QueryImage(PXCMImage.ImageType.IMAGE_TYPE_COLOR);rgbImage.QueryBitmap(pipeline.QuerySession(), out rgb);
  47. 47. Intel ConfidentialRetrieve The Data - Frameworks• Image Data Can Be Retrieved Via QueryBitmap()System.Drawing.Bitmap rgb;PXCMImage rgbImage =pipeline.QueryImage(PXCMImage.ImageType.IMAGE_TYPE_COLOR);rgbImage.QueryBitmap(pipeline.QuerySession(), out rgb);
  48. 48. Intel Confidential• Track any 2D planar surfaces– Position, orientation and otherparameters• Track limited 3D objects– Based on 3D models• Track face orientationSDK Features
  49. 49. Intel ConfidentialAlgorithm Modules: PXCDVTrackerD’Fusion Studio Computer Vision – PerC SDK version
  50. 50. Intel ConfidentialAlgorithm Modules: PXCDVTrackerD’Fusion Studio Computer Vision – PerC SDK version
  51. 51. Intel Confidentialclass PXCDVTracker: public PXCBase{enum TargetType{TARGET_UNDEFINED,TARGET_PLANE,TARGET_OBJECT3D,TARGET_FACE,TARGET_PLANEBLACKBOX,TARGET_MARKER};typedef struct{TrackingStatus status; // (-1) not initialized, 0 not tracking (recognition in process), 1 trackingpxcF64 position[3]; // Resulting pose (X,Y, Z)pxcF64 orientation[4]; // Quaternion to express the orientationint index; // Recognized keyFrame index (-1 none)} TargetData;QueryProfile(…); // Retrieve configuration(s)SetProfile(…); // Set active configurationGetTargetCount(…); //ActivateTarget(…); // Retrieve object tracking dataGetTargetData(…); //ProcessImageAsync(…); // Data processing};Algorithm Modules: PXCDVTrackerModule Interface
  52. 52. Intel ConfidentialSDK Features• User Tracking– Face Detection– Face Location Detection– Face Feature Detection– Face Recognition
  53. 53. Intel ConfidentialFace Detection/Tracking•Locate and trackmultiple faces•Unique identifierfor each faceAlgorithm Modules: PXCFaceAnalysisFace tracking and analysisLandmark Detection•6/7-point detectionincluding eyes,nose, and mouthFacial Attribute Detection•Age-group includingbaby/youth/adult/senior•Gender detection•Smile/blink detectionFace Recognition•Similarity among a set offaces
  54. 54. Intel Confidentialclass PXCFaceAnalysis: public PXCBase{class Detection {QueryProfile(…);SetProfile(…);QueryData(…);};class Landmark {QueryProfile(…);SetProfile(…);QueryLandmarkData(…);QueryPoseData(…);};class Recognition {QueryProfile(…);SetProfile(…);CreateModel(…);};class Attribute {QueryProfile(…);SetProfile(…);QueryData(…);};QueryProfile(…);SetProfile(…);ProcessImageAsync(…);}Algorithm Modules: PXCFaceAnalysisModule interfaceFace location detection/trackingconfiguration and retrieve dataFace landmark detection configurationand data retrievalFace attribute detection configuration anddata retrievalFace recognition confirmation and dataretrievalFace analysis overall configuration anddata processing
  55. 55. Intel Confidential• Nuance* Voice Command and Control– Recognize from a list of predefined commands• Nuance Voice Dictation– Recognize short sentences (<30 seconds)• Nuance Voice Synthesis– Text to speech for short sentencesSDK Features
  56. 56. Intel Confidentialclass PXCVoiceRecognition: public PXCBase{struct Recognition {} // Recognized data structurestruct Alert {} // Event data structureQueryProfile(…); // Retrieve configuration(s)SetProfile(…); // Set active configurationSubscribeRecognition(…);// Recognition event setupSubscribeAlert(…); // Alert event setupCreateGrammar(…); //AddGrammar(…); // Command list constructionSetGrammar(…); //DeleteGrammar(…); //ProcessAudioAsync(…); // Data processing};Algorithm Modules: PXCVoiceRecognitionModule Interface
  57. 57. Intel Confidentialclass MyHandler: public PXCVoiceRecognition::Recognition::Handler, public PXCVoiceRecognition::Alert::Handler{public:MyHandler(std::vector<pxcCHAR*> &commands) { this->commands = commands; }virtual void PXCAPI OnRecognized(PXCVoiceRecognition::Recognition *cmd){wprintf_s(L"nRecognized: <%s>n", (cmd->label>=0)?commands[cmd->label]:cmd->dictation);}virtual void PXCAPI OnAlert(PXCVoiceRecognition::Alert *alert){switch (alert->label){case PXCVoiceRecognition::Alert::LABEL_SNR_LOW:wprintf_s(L"nAlert: <Low SNR>n");break;case PXCVoiceRecognition::Alert::LABEL_VOLUME_LOW:wprintf_s(L"nAlert: <Low Volume>n");break;case PXCVoiceRecognition::Alert::LABEL_VOLUME_HIGH:wprintf_s(L"nAlert: <High Volume>n");break;default:wprintf_s(L"nAlert: <0x%x>n",alert->label);break;}Algorithm Modules: PXCVoiceSynthesisVoice recognition example – callback handlers
  58. 58. Intel Confidentialclass PXCVoiceSynthesis: public PXCBase{public:PXC_CUID_OVERWRITE(PXC_UID(V,I,T,S));struct ProfileInfo {enum Language {LANGUAGE_US_ENGLISH=1,};enum Voice {VOICE_MALE,VOICE_FEMALE,};PXCCapture::AudioStream::DataDesc outputs; // output format, need bufferSize to limit the latency.Language language;Voice voice;pxcU32 reserved[6];};virtual pxcStatus PXCAPI QueryProfile(pxcU32 pidx, ProfileInfo *pinfo)=0;pxcStatus __inline QueryProfile(ProfileInfo *pinfo) { return QueryProfile(WORKING_PROFILE,pinfo); }virtual pxcStatus PXCAPI SetProfile(ProfileInfo *pinfo)=0;virtual pxcStatus PXCAPI QueueSentense(pxcCHAR *sentence, pxcU32 nchars, pxcUID *id)=0;virtual pxcStatus PXCAPI ProcessAudioAsync(pxcUID id, PXCAudio **audio, PXCScheduler::SyncPoint **sp)=0;};Algorithm Modules: PXCVoiceSynthesisModule Interface
  59. 59. Intel Confidential// Queue the sentence to the speech synthesis modulepxcUID tuid = 0;sts = vtts->QueueSentence(cmdl.m_ttstext, wcslen(cmdl.m_ttstext), &tuid);…while (1){PXCSmartPtr<PXCAudio> audio;PXCSmartSP sp;// Read audio framests = vtts->ProcessAudioAsync(tuid, &audio, &sp);if (sts < PXC_STATUS_NO_ERROR)break;sts = sp->Synchronize();if (sts < PXC_STATUS_NO_ERROR){if ((sts == PXC_STATUS_PARAM_UNSUPPORTED) || (sts == PXC_STATUS_EXEC_TIMEOUT))wprintf_s(L"Error in ProcessAudion");if (sts == PXC_STATUS_ITEM_UNAVAILABLE)wprintf_s(L"Voice synthesis completed successfullyn");break;}return 0;Algorithm Modules: PXCVoiceSynthesisSpeech synthesis example - generation
  60. 60. Intel ConfidentialUser Experience on Face Recognition• More expressions will be available in the future SDK• Give feedback on distance• Give feedback on lightening– So user avoid shadows• Notify the user if moving too fast to be tracked
  61. 61. Intel ConfidentialOverall Visual feedback• Give instant feedback acknowledging command– Gesture, voice, or anything else• Show what’s actionable• Show the current state• Consider physics
  62. 62. Intel Confidentialhttps://perceptualchallenge.intel.com
  63. 63. Backup
  64. 64. Intel Confidentialclass aPXCInterface: public PXCBase {public:PXC_CUID_OVERWRITE(PXC_UID(‘M’,’Y’,’I’,’F’));// configurations & inquiriesstruct ProfileInfo {…};virtual pxcStatus PXCAPI QueryProfile(pxcU32 idx, ProfileInfo *pinfo)=0;virtual pxcStatus PXCAPI SetProfile(ProfileInfo *pinfo)=0;// data processingvirtual pxcStatus PXCAPI ProcessDataAsync(…, PXCScheduler::SyncPoint **sp)=0;};Each interface has a unique ID used byPXCBase::QueryInterfaceConsistent way of querying andsetting configurationsAsynchronous execution returns SP forlater synchronizationCore: PXCSessionModule interface conventionsPXC interfaces derive from the PXCBaseclassSDK interfaces contain only pure virtualfunctionsNo exception handling or dynamic_cast(replaced with PXCBase::DynamicCast)
  65. 65. Intel Confidential• Users are notified when SDKaccesses Personally IdentifiableInformation (PII)• Can also launch a viewer fromthe taskbar icon that shows anyapps currently accessing thesensor and what, in particular,they are accessingCore: Privacy NotificationKeeping users informed
  66. 66. Intel Confidential• Image Capture:– 8-bit RGB in RGBA/RGB24/NV12/YUY2– Creative* camera supports up to 1280x720@30p.– 16-bit depthmap, confidence map and vertices.– Creative camera supports up to QVGA@60p– Depthmap smoothing by default• Audio capture:– 1-2 channel PCM/IEEE-Float audio streams– Creative camera supports 44.1kHz and 48KHz• Device properties:– Standard camera properties such as brightness and exposure.– Depth-related properties such as confidence threshold, depthmap value range etc.I/O ModulesAudio and video capture
  67. 67. Intel Confidential1. Enumerate and create capture deviceQueryDevice Query capture device namesCreateDevice Create a capture device instance2. Enumerate and select streamsQueryStream Query stream typeCreateVideoStream Select a video streamCreateAudioStream Select an audio stream3. Perform stream operationsQueryProfile Query stream configurationsSetProfile Set a stream configurationReadStreamAsync Read samples from the streamI/O Modules: PXCCapturePXCCapture interface hierarchy
  68. 68. Intel ConfidentialSetDeviceProperty/QueryDeviceProperty• Color stream properties– CONTRAST, BRIGHTNESS, HUE, SATURATION …• Depth stream properties– DEPTH_SMOOTHING, SATURATION_VALUE, …• Audio stream properties– MIX_LEVEL• Misc. properties– ACCELEROMETER_READINGI/O Modules: PXCCaptureDevice properties
  69. 69. Intel Confidential• Alert and callback interface used for low-frequency events and notifications• Subscribe to eventsPXCGesture::SubscribeAlertPXCGesture::SubscribeGesturePXCVoiceCommand::SubscribeAlertPXCVoiceCommand::SubscribeCommand• Implement the callback handlerAlgorithm Modules: PXCGestureAlerts and callback notificationsclass Handler: public PXCBaseImpl<PXCGesture::Gesture::Handler>{public:virtual pxcStatus PXCAPI OnGesture(Gesture *gesture) {…}};
  70. 70. Intel Confidentialclass MyPipeline: public UtilPipeline {public:MyPipeline(void):UtilPipeline() {EnableGesture();}virtual void PXCAPI OnGesture(PXCGesture::Gesture *data) {printf_s(“%dn”,data->label);}};int wmain(int argc, WCHAR* argv[]) {MyPipeline pipeline;pipeline.LoopFrames();return 0;}class MyPipeline: UtilMPipeline {public MyPipeline():base() {EnableGesture();}public override void OnGesture(ref PXCMGesture.Gesture data) {Console.WriteLn(data.label);}};class Program {static void Main(string[] args) {MyPipeline pipeline=new MyPipeline();pipeline.LoopFrames();pipeline.Dispose();}}C++ C#Enable Finger TrackingGesture CallbackData Flow LoopsUtilPipeline ClassGesture Recognition “Hello World”
  71. 71. Intel Confidential• Multiple processing modules on singleinput device– Live streaming or file-basedrecording/playback– Synchronized image (or audio) processingUtilPipeline pp;pp.EnableImage(PXCImage::COLOR_FORMAT_RGB32);pp.EnableImage(PXCImage::COLOR_FORMAT_DEPTH);for (;;) {if (!pp.AcquireFrame(true)) break;PXCImage *color, *depth;color=pp.QueryImage(PXCImage::IMAGE_TYPE_COLOR);depth=pp.QueryImage(PXCImage::IMAGE_TYPE_DEPTH);pp.ReleaseFrame();}pp.Close();UtilPipeline ClassUtilPipeline-based applicationColor and depth are synchronized
  72. 72. Intel ConfidentialSpeech Recognition:Voice command and control, short sentence dictation, andtext to speech synthesisSDK Usage Modes Today11 New usage modes may be added in the futureClose-range Depth Tracking (6 in. to 3 ft.):Recognize the positions of each of the user’s hands, fingers,static hand poses and moving hand gestures.Facial Analysis:Face detection and recognition (six and seven point landmark andattribution detection, including smiles, blinks, and age groups)Augmented Reality:Combine real-time images from the camera and close-range trackingfrom the depth sensor with 2D or 3D graphical images.
  73. 73. Intel ConfidentialYour SDK ‘One-Stop-Shop”intel.com/software/perceptual@PerceptualSDK (Twitter)CHALLENGE INFODOWNLOAD SDKORDER CAMERADOCUMENTSDEMO APPSSUPPORT
  74. 74. Intel ConfidentialIntel® Perceptual Computing ChallengeThe $1Million 2013 Application Development Contest*Enter Phase 2: perceptualchallenge.intel.com/ Focus: Games, Productivity, Creative UI& Multi-modal Process: Developers submit workingprototypes, panel judged Two Phases: Phase 1 (CLOSED): See WinnerShowcase at http://goo.gl/EnNHv Phase 2: March (GDC) to September -$800,000+ in prizes Categories: Perceptual Gaming,Productivity, Creative User Interfaceand Open Innovation Available in 16 countries*Terms and Conditions Apply
  75. 75. Intel ConfidentialSpeech Recognition & Dragon Assistant*PerceptualComputing SDKRuntimeSpeech RecognitionApplicationDrivers & HardwareDragon Assistant*Dragon Assistant*Engine andLanguage Pack• Perceptual Computing speech recognition applications require Dragon Assistant* Engine andLanguage Packs to be installed on target platform• For app developers, Engine and Language Packs are available on SDK download site (THESEARE FOR DEVELOPER INTERNAL USE ONLY AND NOT TO BE DISTRIBUTED).• For consumers, Dragon Assistant* (with Engine) is expected to available as follows:• Expected to be bundled with Creative* Camera (when available)• Expected to be pre-installed on speech-certified 4th Gen Core Ultrabook devices in late2013SDK Speech APIsuse the DragonAssistant* Engineand LanguagePacks
  76. 76. Intel ConfidentialSample Snippet (processing)• Declarationsimport intel.pcsdk.*;PXCUPipeline pxc;int[] cm = new int[2]; //color map dimensionsint[] dm = new int[2]; //depth map dimensionsshort[] buffer;Pimage rgb, depth;
  77. 77. Intel ConfidentialSample Snippet (processing)• Initializationvoid setup(){pxc = new PXCUPipeline(this);if(!pxc.Init(PXCUPipeline.Mode.COLOR_VGA|PXCUPipeline.Mode.GESTURE))println(“Error initializing PerC SDK”);if(pxc.QueryRGBSize(cm))rgb = createImate(cm[0], cm[1], RGB);if(pxc.QueryDepthMapSize(dm)){buffer = new short[dm[0]*dm[1]];depth = createImage(dm[0], dm[1], RGB);}size(640,480);}
  78. 78. Intel ConfidentialSample Snippet (processing)• Main Loopvoid draw(){if(pxc.AcquireFrame(false)){pxc.QueryRGB(rgb);pxc.QueryDepthMap(buffer);pxc.ReleaseFrame();}RemapDepth();image(rgb,0,0,320,240);image(depth,320,0);}
  79. 79. Intel Confidential• Color streams RGB24 640x480 25fps, 30fps RGB24 640x360 25fps, 30fps RGB24 1280x720 25fps, 30fps• Depth streams (16-bit integer, 0-32000) 320x240 25fps, 30fps, 50fps, 60fps UVMAP (Depth  Color) Confidence Map (16-bit integer)• Vertices streams (real world coordinates in3D fixed-point integers)• Audio streams (At least 2-array MIC) 44.1KHz mono/stereoCamera StreamsVisual ComputingProducts
  80. 80. Intel ConfidentialImage ConversionVisual ComputingProductsRGB24 RGB32 NV12 YUY2 GRAYRGB24 Y Y YRGB32 Y Y YNV12 Y Y YYUY2 Y Y Y Y YGRAY Y Y Y YDEPTH YVERTICES YFor instance, raw DepthSense color image format is RGB24, withAcquireAccess(PXCImage::ACCESS_READ, PXCImage::COLOR_FORMAT_RGB32, &data)SDK framework will convert color image data from RGB24 to RGB32

×