Your SlideShare is downloading. ×
Perceptual Computing Workshop in Munich
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Perceptual Computing Workshop in Munich


Published on

By Xavier Hallade @ Werk1. …

By Xavier Hallade @ Werk1.

Feel free to join us to the perceptual computing Hacknight on June 22-23th!

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Perceptual Computing WorkshopXavier Hallade, Technical MarketingEngineerSulamita Garcia, TechnicalMarketing Engineer
  • 2. Intel Confidential/me@sulagarcia
  • 3. Intel Confidential/me@ph0b
  • 4. Intel Confidential
  • 5. Intel ConfidentialIntel®Perceptual Computing SDK 2013Next Generation Interactivity for Intel®Core™ Processor-Based Applications
  • 6. Intel ConfidentialCreative* Interactive Gesture CameraFor use with the Intel® Perceptual Computing SDK Small, light-weight, low-power Tuned for close-range interactivity Designed with ease of setup andportability Includes: HD web camera Depth sensor Dual-array microphoneSign up to purchase a camera at*Other brands and trademarks may be claimed as the property of their respective owners
  • 7. Intel ConfidentialUser Experience considerations
  • 8. Intel ConfidentialWhat is Perceptual Computing?Interactivity Beyond Touch, Mouse and Keyboard …FacialTrackingSpeechRecognitionClose-rangeFingerTrackingAugmentedRealityClose-rangeGestureTrackingFacilitates Application Developers Implementation of: Games Entertainment Productivity Accessibility Immersive Teleconferencing Education Medical / Health Enterprises Retail Industrial
  • 9. Intel ConfidentialSDK Usage H/W RequirementsSDK Usage Mode Speech CertifiedDual-ArrayMicrophonesRGB Webcam Creative*CameraClose-rangeDepth trackingXSpeechRecognitionX XFace Recognition X XAugmentedRealityX X Close-range depth tracking requires Creative camera Speech requires dual-array microphones OR Creative* camera 2H’13 4th Gen Ultrabook devices are required to have speechcertified microphones Dell XPS 13* has speech-certified microphones Facial tracking requires RGB Webcam OR Creative* Camera Augmented Reality requires RGB Webcam OR Creative* Camera*Other brands and trademarks may be claimed as the property of their respective owners
  • 10. Intel ConfidentialProgramming Language and FrameworkSupport• C++, C#, Java• Supported Frameworks– processing– openFrameworks– Unity– Havok– Total Immersion AR
  • 11. Intel ConfidentialProgramming Language and FrameworkSupport• Unsupported but Verified– Cinder– OGRE– XNA / Monogame– Bullet Physics
  • 12. Intel ConfidentialPXCSession, PXCImage, PXCAudio, PXCCapture, PXCGesture,PXCFaceAnalysis, PXCVoiceUtilCapture, UtilPipelineC#PXCMSessionPXCMImagePXCMAudioPXCMCapturePXCMGesturePXCMFaceAnalysisPXCMVoiceUtilMCaptureUtilMPipelinepxcupipelineUnity*ProProcessingopenFrameworks*ApplicationsCore FunctionalitiesModule InteractionAdditional Language andFramework SupportSDK API Hierarchy*Other brands and trademarks may be claimed as the property of their respective owners
  • 13. Intel ConfidentialSDK files• Samples are located in:– $(PCSDK_DIR)bin (executables)– $(PCSDK_DIR)samples (source code)– $(PCSDK_DIR)framework (framework and game engines)– $(PCSDK_DIR)demo
  • 14. Intel ConfidentialResources• Perceptual Computing Forums–• Perceptual Computing IDZ Portal–• Github–
  • 15. Intel ConfidentialUser Experience considerations• Reality inspired, not cloning– Wrapping fingers around objects• Literal, not abstract– Universal visual cues – switches and nobs• Consistency!!!• Extensible – prepare for future improvements• Manage persistence– Sometimes the hand will go out of the view
  • 16. Intel ConfidentialProcess Overview – prop 1• Subclass the SDK Pipeline Object• Enable features in the constructor• Reimplement OnAlert|Gesture|NewFrame…() methods• Instantiate the object and call LoopFrames() on it.
  • 17. Intel ConfidentialHello World – C++class GesturePipeline: public UtilPipeline {public:GesturePipeline(void):UtilPipeline(), m_render(L"Gesture Viewer") {EnableGesture();}virtual void PXCAPI OnGesture(PXCGesture::Gesture *data) {if (data->active) m_gdata = (*data);switch (data->label) {case PXCGesture::Gesture::LABEL_NAV_SWIPE_LEFT: break; //do somethingcase PXCGesture::Gesture::LABEL_NAV_SWIPE_RIGHT: break; //do somethingdefault: break;}}virtual bool OnNewFrame(void) {return m_render.RenderFrame(QueryImage(PXCImage::IMAGE_TYPE_DEPTH),QueryGesture(), &m_gdata);}protected:GestureRender m_render;PXCGesture::Gesture m_gdata;};
  • 18. Intel ConfidentialProcess Overview – prop 2• Declare The SDK Pipeline Object• Select Features and Initialize• Acquire A Frame• Query The Data• CleanupYou can also subclass the pipeline object and reimplement OnXXX methods
  • 19. Intel ConfidentialDeclare The SDK Object – C++/C#• C++UtilPipeline pipeline;• C#UtilMPipeline pipeline;pipeline = new UtilMPipeline();
  • 20. Intel ConfidentialSelect Features and Initialize – C++/C#• Select Features Using .Enable*() Methods• Use Init() To Set Features and Enable SDK Accesspipeline.EnableGesture();pipeline.EnableImage(PXCImage::COLOR_FORMAT_RGB24);//C# PXCImage.ColorFormat.COLOR_FORMAT_RGB24pipeline.Init();
  • 21. Intel ConfidentialDeclare The SDK Object - Frameworks• Unityprivate PXCUPipeline pipeline;pipeline = new PXCUPipeline();• Processingprivate PXCUPipeline pipeline;pipeline = new PXCUPipeline(this);
  • 22. Intel ConfidentialSelect Features and Initialize - Frameworks• Select Features using PXCUPipeline.Mode enum• Use Bitwise OR (|) for Multiple Features• Use Init() To Set Features and Enable SDK Accesspipeline.Init(PXCUPipeline.Mode.COLOR_VGA|PXCUPipeline.Mode.DEPTH_QVGA|PXCUPipeline.Mode.GESTURE);
  • 23. Intel ConfidentialCapture A Frame• Poll For A Frame Using AcquireFrame(bool);– Can be blocking or non-blocking– AcquireFrame(true) is blocking, AcquireFrame(false) is non-blocking• Returns true If A Frame Is Availableif(pipeline.AcquireFrame(false)){}
  • 24. Intel ConfidentialRetrieve The Data• Data Is Retrieved via Query*(<T>)– QueryRGB(), QueryLabelMapAsImage(), etc…• UnityTexture2D rgbTexture = newTexture2D(640,480,TextureFormat.ARGB32, false);Pipeline.QueryRGB(rgbTexture);• processingPImage rgbTexture = createImage(640,480,RGB);pipeline.QueryRGB(rgbTexture);
  • 25. Intel ConfidentialClean Up• Use ReleaseFrame() To “Free Up The Pipeline”pipeline.ReleaseFrame();
  • 26. Intel ConfidentialSDK Features• User Tracking– Hand Detection– Finger Detection– Static Pose Detection– Dynamic GestureDetection
  • 27. Intel ConfidentialUser experience on Gestures• Innate vs. learned gestures– Aim for lower cognitive load• Actionable gestures– What if the user turns the head, drinks coffee…• Prevent Occlusion!!!• Left-right gestures easier than up-down• Grabbed objects– Make it obvious where objects can be dropped
  • 28. Intel ConfidentialIn This Section• Tracking Hands• Tracking Fingers
  • 29. Intel ConfidentialKey Concepts• PXCGesture– Gesture and node tracking interface• PXCGesture::GeoNode– Geometric Node, struct for tracking data• PXCGesture::GeoNode::Label– Enum indicating tracked node
  • 30. Intel ConfidentialXZYHands and fingers trackingGeoNode:– PXCPoint3DF32 positionWorld;– PXCPoint3DF32 positionImage;– pxcU64 timeStamp;– pxcU32 confidence;– pxcF32 radius;– Label body;– PXCPoint3DF32 normal;– pxcU32 openness;o HAND_FINGERTIPPXCGesture::GeoNode::LABEL_BODY_HAND_*PXCGesture::GeoNode::LABEL_FINGER_*o HAND_UPPERo HAND_MIDDLEo HAND_LOWER
  • 31. Intel ConfidentialRetrieve The Geonodes data – C++PXCGesture gesture = pipeline->QueryGesture();if(gesture){PXCGesture::GeoNode node;gesture->QueryNodeData(0,PXCGesture::GeoNode::LABEL_BODY_HAND_PRIMARY,&node);}
  • 32. Intel ConfidentialRetrieve The Geonodes data - C#PXCMGesture gesture = pipeline.QueryGesture();if(gesture){PXCMGesture.GeoNode node;gesture.QueryNodeData(0,PXCMGesture.GeoNode.Label.LABEL_BODY_HAND_PRIMARY,out node);}
  • 33. Intel ConfidentialRetrieve The Geonodes Data - Frameworks• UnityPXCMGesture.GeoNode node;pipeline.QueryGeoNode(PXCMGesture.GeoNode.LABEL_BODY_HAND_PRIMARY, out node);• processingPXCMGesture.GeoNode node = newPXCMGesture.GeoNode();pipeline.QueryGeoNode(PXCMGesture.GeoNode.LABEL_BODY_HAND_PRIMARY, node);
  • 34. Intel ConfidentialGesture recognition~30cmBlob Intermediate images, help separating:• Background• HandsGeoNode Skeleton nodes• Hand openness• Open Hand: Fingertips, middle, elbows• Closed Hand: up, middle, bottomGesture • THUMB UP/DOWN, PEACE, BIG5• WAVE, CIRCLE, SWIPE LEFT/RIGHT/UP/DOWNAlert • FOV_LEFT/_RIGHT/_TOP/_BOTTOM• FOV_BLOCKED/_OK• GEONODE_ACTIVE/INACTIVE
  • 35. Intel ConfidentialSelect Features and Initialize – C++• Only Need To Enable Gestures, Images Optional forFeedback or Visualizationpipeline.EnableGesture();pipeline.Init();
  • 36. Intel ConfidentialSelect Features and Initialize - Frameworks• Only Need ‘GESTURE’, Images Optional for Feedback orVisualizationpipeline.Init(PXCUPipeline.Mode.GESTURE);
  • 37. Intel ConfidentialKey Concepts• PXCGesture– Gesture and node tracking interface• PXCGesture::Gesture– Gesture, struct for detected gestures• PXCGesture::Gesture::Label– Enum indicating detected gesture
  • 38. Intel Confidentialclass PXCGesture: public PXCBase{struct GeoNode {} // geometric node data structurestruct Gesture {} // pose/gesture data structurestruct Blob {} // label map data structurestruct Alert {} // event data structureQueryProfile(…); // Retrieve configuration(s)SetProfile(…); // Set active configurationSubscribeGesture(Gesture::Handler); // Event setupSubscribeAlert(Alert::Handler); // Event setupProcessImageAsync(images, …); // Data processingQueryGestureData(…);// Retrieve pose/gesture dataQueryNodeData(…); // Retrieve geometric node dataQueryBlobData(…); // Retrieve label map data};Algorithm Modules: PXCGestureModule interface
  • 39. Intel ConfidentialRetrieve The Data - C#PXCMGesture gesture = pipeline.QueryGesture();if(gesture){PXCMGesture.Gesture gest;gesture.QueryGestureData(0,PXCMGesture.GeoNode.Label.LABEL_BODY_HAND_PRIMARY,0,out gest);}
  • 40. Intel ConfidentialRetrieve The Data - Frameworks• UnityPXCMGesture.Gesture gest;pipeline.QueryGesture(PXCMGesture.GeoNode.LABEL_BODY_HAND_PRIMARY, out gest);• processingPXCMGesture.Gesture gest = newPXCMGesture.Gesture();pipeline.QueryGesture(PXCMGesture.GeoNode.LABEL_BODY_HAND_PRIMARY, gest);
  • 41. Intel ConfidentialAccessing Gesture Data• C++if(gest->active){if(gest->label==PXCGesture::Gesture::LABEL_HAND_WAVE){//Do stuff!}}• C#/Frameworks:if({if(gest.label==PXCMGesture.Gesture.Label.LABEL_HAND_WAVE){//Do stuff!}}
  • 42. Intel ConfidentialSDK Features• Video Capture– RGB (VGA and HD)– Depth– Blobs– IR/Confidence
  • 43. Intel ConfidentialAvailable Video Streams• RGB– VGA (640x480)– HD (1280x720)– 30 FPS
  • 44. Intel ConfidentialAvailable Video Streams• Labels/Blobs– QVGA (320x240)– 30/60 FPS
  • 45. Intel ConfidentialAvailable Video Streams• Depth– QVGA (320x240)– Must Convert to Color Space– 30/60 FPS
  • 46. Intel ConfidentialAvailable Video Streams• IR– QVGA (320x240)– Must Convert to Color Space– 30/60 FPS– 16 bits
  • 47. Intel ConfidentialRetrieve The Data (C++)• Data Is Retrieved via QueryImage() And Accessing TheData Buffers• Image Is Retrieved as PXCImage, Data Is Accessed ViaPXCImage::ImageData.planesPXCImage rgb = pipeline.QueryImage(PXCImage::IMAGE_TYPE_COLOR);PXCImage::ImageData rgbData;rgb->AcquireAccess(PXCImage::ACCESS_READ, &rgbData);//Data can be loaded from rgbData.planes[0]rgb->ReleaseAccess(&rgbData);
  • 48. Intel ConfidentialRetrieve The Data (C#)• Image Data Can Be Retrieved Via QueryBitmap()System.Drawing.Bitmap rgb;PXCMImage rgbImage =pipeline.QueryImage(PXCMImage.ImageType.IMAGE_TYPE_COLOR);rgbImage.QueryBitmap(pipeline.QuerySession(), out rgb);
  • 49. Intel ConfidentialRetrieve The Data - Frameworks• Image Data Can Be Retrieved Via QueryBitmap()System.Drawing.Bitmap rgb;PXCMImage rgbImage =pipeline.QueryImage(PXCMImage.ImageType.IMAGE_TYPE_COLOR);rgbImage.QueryBitmap(pipeline.QuerySession(), out rgb);
  • 50. Intel Confidential• Track any 2D planar surfaces– Position, orientation and otherparameters• Track limited 3D objects– Based on 3D models• Track face orientationSDK Features
  • 51. Intel ConfidentialAlgorithm Modules: PXCDVTrackerD’Fusion Studio Computer Vision – PerC SDK version
  • 52. Intel ConfidentialAlgorithm Modules: PXCDVTrackerD’Fusion Studio Computer Vision – PerC SDK version
  • 53. Intel Confidentialclass PXCDVTracker: public PXCBase{enum TargetType{TARGET_UNDEFINED,TARGET_PLANE,TARGET_OBJECT3D,TARGET_FACE,TARGET_PLANEBLACKBOX,TARGET_MARKER};typedef struct{TrackingStatus status; // (-1) not initialized, 0 not tracking (recognition in process), 1 trackingpxcF64 position[3]; // Resulting pose (X,Y, Z)pxcF64 orientation[4]; // Quaternion to express the orientationint index; // Recognized keyFrame index (-1 none)} TargetData;QueryProfile(…); // Retrieve configuration(s)SetProfile(…); // Set active configurationGetTargetCount(…); //ActivateTarget(…); // Retrieve object tracking dataGetTargetData(…); //ProcessImageAsync(…); // Data processing};Algorithm Modules: PXCDVTrackerModule Interface
  • 54. Intel ConfidentialSDK Features• User Tracking– Face Detection– Face Location Detection– Face Feature Detection– Face Recognition
  • 55. Intel ConfidentialFace Detection/Tracking•Locate and trackmultiple faces•Unique identifierfor each faceAlgorithm Modules: PXCFaceAnalysisFace tracking and analysisLandmark Detection•6/7-point detectionincluding eyes,nose, and mouthFacial Attribute Detection•Age-group includingbaby/youth/adult/senior•Gender detection•Smile/blink detectionFace Recognition•Similarity among a set offaces
  • 56. Intel Confidentialclass PXCFaceAnalysis: public PXCBase{class Detection {QueryProfile(…);SetProfile(…);QueryData(…);};class Landmark {QueryProfile(…);SetProfile(…);QueryLandmarkData(…);QueryPoseData(…);};class Recognition {QueryProfile(…);SetProfile(…);CreateModel(…);};class Attribute {QueryProfile(…);SetProfile(…);QueryData(…);};QueryProfile(…);SetProfile(…);ProcessImageAsync(…);}Algorithm Modules: PXCFaceAnalysisModule interfaceFace location detection/trackingconfiguration and retrieve dataFace landmark detection configurationand data retrievalFace attribute detection configuration anddata retrievalFace recognition confirmation and dataretrievalFace analysis overall configuration anddata processing
  • 57. Intel Confidential• Nuance* Voice Command and Control– Recognize from a list of predefined commands• Nuance Voice Dictation– Recognize short sentences (<30 seconds)• Nuance Voice Synthesis– Text to speech for short sentencesSDK Features
  • 58. Intel Confidentialclass PXCVoiceRecognition: public PXCBase{struct Recognition {} // Recognized data structurestruct Alert {} // Event data structureQueryProfile(…); // Retrieve configuration(s)SetProfile(…); // Set active configurationSubscribeRecognition(…);// Recognition event setupSubscribeAlert(…); // Alert event setupCreateGrammar(…); //AddGrammar(…); // Command list constructionSetGrammar(…); //DeleteGrammar(…); //ProcessAudioAsync(…); // Data processing};Algorithm Modules: PXCVoiceRecognitionModule Interface
  • 59. Intel Confidentialclass MyHandler: public PXCVoiceRecognition::Recognition::Handler, public PXCVoiceRecognition::Alert::Handler{public:MyHandler(std::vector<pxcCHAR*> &commands) { this->commands = commands; }virtual void PXCAPI OnRecognized(PXCVoiceRecognition::Recognition *cmd){wprintf_s(L"nRecognized: <%s>n", (cmd->label>=0)?commands[cmd->label]:cmd->dictation);}virtual void PXCAPI OnAlert(PXCVoiceRecognition::Alert *alert){switch (alert->label){case PXCVoiceRecognition::Alert::LABEL_SNR_LOW:wprintf_s(L"nAlert: <Low SNR>n");break;case PXCVoiceRecognition::Alert::LABEL_VOLUME_LOW:wprintf_s(L"nAlert: <Low Volume>n");break;case PXCVoiceRecognition::Alert::LABEL_VOLUME_HIGH:wprintf_s(L"nAlert: <High Volume>n");break;default:wprintf_s(L"nAlert: <0x%x>n",alert->label);break;}Algorithm Modules: PXCVoiceSynthesisVoice recognition example – callback handlers
  • 60. Intel Confidentialclass PXCVoiceSynthesis: public PXCBase{public:PXC_CUID_OVERWRITE(PXC_UID(V,I,T,S));struct ProfileInfo {enum Language {LANGUAGE_US_ENGLISH=1,};enum Voice {VOICE_MALE,VOICE_FEMALE,};PXCCapture::AudioStream::DataDesc outputs; // output format, need bufferSize to limit the latency.Language language;Voice voice;pxcU32 reserved[6];};virtual pxcStatus PXCAPI QueryProfile(pxcU32 pidx, ProfileInfo *pinfo)=0;pxcStatus __inline QueryProfile(ProfileInfo *pinfo) { return QueryProfile(WORKING_PROFILE,pinfo); }virtual pxcStatus PXCAPI SetProfile(ProfileInfo *pinfo)=0;virtual pxcStatus PXCAPI QueueSentense(pxcCHAR *sentence, pxcU32 nchars, pxcUID *id)=0;virtual pxcStatus PXCAPI ProcessAudioAsync(pxcUID id, PXCAudio **audio, PXCScheduler::SyncPoint **sp)=0;};Algorithm Modules: PXCVoiceSynthesisModule Interface
  • 61. Intel Confidential// Queue the sentence to the speech synthesis modulepxcUID tuid = 0;sts = vtts->QueueSentence(cmdl.m_ttstext, wcslen(cmdl.m_ttstext), &tuid);…while (1){PXCSmartPtr<PXCAudio> audio;PXCSmartSP sp;// Read audio framests = vtts->ProcessAudioAsync(tuid, &audio, &sp);if (sts < PXC_STATUS_NO_ERROR)break;sts = sp->Synchronize();if (sts < PXC_STATUS_NO_ERROR){if ((sts == PXC_STATUS_PARAM_UNSUPPORTED) || (sts == PXC_STATUS_EXEC_TIMEOUT))wprintf_s(L"Error in ProcessAudion");if (sts == PXC_STATUS_ITEM_UNAVAILABLE)wprintf_s(L"Voice synthesis completed successfullyn");break;}return 0;Algorithm Modules: PXCVoiceSynthesisSpeech synthesis example - generation
  • 62. Intel ConfidentialUser Experience on Face Recognition• More expressions will be available in the future SDK• Give feedback on distance• Give feedback on lightening– So user avoid shadows• Notify the user if moving too fast to be tracked
  • 63. Intel ConfidentialOverall Visual feedback• Give instant feedback acknowledging command– Gesture, voice, or anything else• Show what’s actionable• Show the current state• Consider physics
  • 64. Intel Confidential
  • 65. Backup
  • 66. Intel Confidentialclass aPXCInterface: public PXCBase {public:PXC_CUID_OVERWRITE(PXC_UID(‘M’,’Y’,’I’,’F’));// configurations & inquiriesstruct ProfileInfo {…};virtual pxcStatus PXCAPI QueryProfile(pxcU32 idx, ProfileInfo *pinfo)=0;virtual pxcStatus PXCAPI SetProfile(ProfileInfo *pinfo)=0;// data processingvirtual pxcStatus PXCAPI ProcessDataAsync(…, PXCScheduler::SyncPoint **sp)=0;};Each interface has a unique ID used byPXCBase::QueryInterfaceConsistent way of querying andsetting configurationsAsynchronous execution returns SP forlater synchronizationCore: PXCSessionModule interface conventionsPXC interfaces derive from the PXCBaseclassSDK interfaces contain only pure virtualfunctionsNo exception handling or dynamic_cast(replaced with PXCBase::DynamicCast)
  • 67. Intel Confidential• Users are notified when SDKaccesses Personally IdentifiableInformation (PII)• Can also launch a viewer fromthe taskbar icon that shows anyapps currently accessing thesensor and what, in particular,they are accessingCore: Privacy NotificationKeeping users informed
  • 68. Intel Confidential• Image Capture:– 8-bit RGB in RGBA/RGB24/NV12/YUY2– Creative* camera supports up to 1280x720@30p.– 16-bit depthmap, confidence map and vertices.– Creative camera supports up to QVGA@60p– Depthmap smoothing by default• Audio capture:– 1-2 channel PCM/IEEE-Float audio streams– Creative camera supports 44.1kHz and 48KHz• Device properties:– Standard camera properties such as brightness and exposure.– Depth-related properties such as confidence threshold, depthmap value range etc.I/O ModulesAudio and video capture
  • 69. Intel Confidential1. Enumerate and create capture deviceQueryDevice Query capture device namesCreateDevice Create a capture device instance2. Enumerate and select streamsQueryStream Query stream typeCreateVideoStream Select a video streamCreateAudioStream Select an audio stream3. Perform stream operationsQueryProfile Query stream configurationsSetProfile Set a stream configurationReadStreamAsync Read samples from the streamI/O Modules: PXCCapturePXCCapture interface hierarchy
  • 70. Intel ConfidentialSetDeviceProperty/QueryDeviceProperty• Color stream properties– CONTRAST, BRIGHTNESS, HUE, SATURATION …• Depth stream properties– DEPTH_SMOOTHING, SATURATION_VALUE, …• Audio stream properties– MIX_LEVEL• Misc. properties– ACCELEROMETER_READINGI/O Modules: PXCCaptureDevice properties
  • 71. Intel Confidential• Alert and callback interface used for low-frequency events and notifications• Subscribe to eventsPXCGesture::SubscribeAlertPXCGesture::SubscribeGesturePXCVoiceCommand::SubscribeAlertPXCVoiceCommand::SubscribeCommand• Implement the callback handlerAlgorithm Modules: PXCGestureAlerts and callback notificationsclass Handler: public PXCBaseImpl<PXCGesture::Gesture::Handler>{public:virtual pxcStatus PXCAPI OnGesture(Gesture *gesture) {…}};
  • 72. Intel Confidentialclass MyPipeline: public UtilPipeline {public:MyPipeline(void):UtilPipeline() {EnableGesture();}virtual void PXCAPI OnGesture(PXCGesture::Gesture *data) {printf_s(“%dn”,data->label);}};int wmain(int argc, WCHAR* argv[]) {MyPipeline pipeline;pipeline.LoopFrames();return 0;}class MyPipeline: UtilMPipeline {public MyPipeline():base() {EnableGesture();}public override void OnGesture(ref PXCMGesture.Gesture data) {Console.WriteLn(data.label);}};class Program {static void Main(string[] args) {MyPipeline pipeline=new MyPipeline();pipeline.LoopFrames();pipeline.Dispose();}}C++ C#Enable Finger TrackingGesture CallbackData Flow LoopsUtilPipeline ClassGesture Recognition “Hello World”
  • 73. Intel Confidential• Multiple processing modules on singleinput device– Live streaming or file-basedrecording/playback– Synchronized image (or audio) processingUtilPipeline pp;pp.EnableImage(PXCImage::COLOR_FORMAT_RGB32);pp.EnableImage(PXCImage::COLOR_FORMAT_DEPTH);for (;;) {if (!pp.AcquireFrame(true)) break;PXCImage *color, *depth;color=pp.QueryImage(PXCImage::IMAGE_TYPE_COLOR);depth=pp.QueryImage(PXCImage::IMAGE_TYPE_DEPTH);pp.ReleaseFrame();}pp.Close();UtilPipeline ClassUtilPipeline-based applicationColor and depth are synchronized
  • 74. Intel ConfidentialSpeech Recognition:Voice command and control, short sentence dictation, andtext to speech synthesisSDK Usage Modes Today11 New usage modes may be added in the futureClose-range Depth Tracking (6 in. to 3 ft.):Recognize the positions of each of the user’s hands, fingers,static hand poses and moving hand gestures.Facial Analysis:Face detection and recognition (six and seven point landmark andattribution detection, including smiles, blinks, and age groups)Augmented Reality:Combine real-time images from the camera and close-range trackingfrom the depth sensor with 2D or 3D graphical images.
  • 76. Intel ConfidentialKey Upcoming Items Creative* Senz3D – Q3 2013 Integration in Intel devices – H2 2014*Other brands and trademarks may be claimed as the property of their respective owners
  • 77. Intel ConfidentialIntel® Perceptual Computing ChallengeThe $1Million 2013 Application Development Contest*Enter Phase 2: Focus: Games, Productivity, Creative UI& Multi-modal Process: Developers submit workingprototypes, panel judged Two Phases: Phase 1 (CLOSED): See WinnerShowcase at Phase 2: March (GDC) to September -$800,000+ in prizes Categories: Perceptual Gaming,Productivity, Creative User Interfaceand Open Innovation Available in 16 countries*Terms and Conditions Apply
  • 78. Intel ConfidentialSpeech Recognition & Dragon Assistant*PerceptualComputing SDKRuntimeSpeech RecognitionApplicationDrivers & HardwareDragon Assistant*Dragon Assistant*Engine andLanguage Pack• Perceptual Computing speech recognition applications require Dragon Assistant* Engine andLanguage Packs to be installed on target platform• For app developers, Engine and Language Packs are available on SDK download site (THESEARE FOR DEVELOPER INTERNAL USE ONLY AND NOT TO BE DISTRIBUTED).• For consumers, Dragon Assistant* (with Engine) is expected to available as follows:• Expected to be bundled with Creative* Camera (when available)• Expected to be pre-installed on speech-certified 4th Gen Core Ultrabook devices in late2013SDK Speech APIsuse the DragonAssistant* Engineand LanguagePacks
  • 79. Intel ConfidentialSample Snippet (processing)• Declarationsimport intel.pcsdk.*;PXCUPipeline pxc;int[] cm = new int[2]; //color map dimensionsint[] dm = new int[2]; //depth map dimensionsshort[] buffer;Pimage rgb, depth;
  • 80. Intel ConfidentialSample Snippet (processing)• Initializationvoid setup(){pxc = new PXCUPipeline(this);if(!pxc.Init(PXCUPipeline.Mode.COLOR_VGA|PXCUPipeline.Mode.GESTURE))println(“Error initializing PerC SDK”);if(pxc.QueryRGBSize(cm))rgb = createImate(cm[0], cm[1], RGB);if(pxc.QueryDepthMapSize(dm)){buffer = new short[dm[0]*dm[1]];depth = createImage(dm[0], dm[1], RGB);}size(640,480);}
  • 81. Intel ConfidentialSample Snippet (processing)• Main Loopvoid draw(){if(pxc.AcquireFrame(false)){pxc.QueryRGB(rgb);pxc.QueryDepthMap(buffer);pxc.ReleaseFrame();}RemapDepth();image(rgb,0,0,320,240);image(depth,320,0);}
  • 82. Intel Confidential• Color streams RGB24 640x480 25fps, 30fps RGB24 640x360 25fps, 30fps RGB24 1280x720 25fps, 30fps• Depth streams (16-bit integer, 0-32000) 320x240 25fps, 30fps, 50fps, 60fps UVMAP (Depth  Color) Confidence Map (16-bit integer)• Vertices streams (real world coordinates in3D fixed-point integers)• Audio streams (At least 2-array MIC) 44.1KHz mono/stereoCamera StreamsVisual ComputingProducts
  • 83. Intel ConfidentialImage ConversionVisual ComputingProductsRGB24 RGB32 NV12 YUY2 GRAYRGB24 Y Y YRGB32 Y Y YNV12 Y Y YYUY2 Y Y Y Y YGRAY Y Y Y YDEPTH YVERTICES YFor instance, raw DepthSense color image format is RGB24, withAcquireAccess(PXCImage::ACCESS_READ, PXCImage::COLOR_FORMAT_RGB32, &data)SDK framework will convert color image data from RGB24 to RGB32