Successfully reported this slideshow.
Introduction to Kinect v2
Tsukasa Sugiura
@UnaNancyOwen
Self-Introduction
Tsukasa Sugiura
Microsoft MVP for Kinect for Windows
 @UnaNancyOwen
 http://UnaNancyOwen.com
 t.sugiu...
Agenda
 What is Kinect v2
 Specifications
 New Features
 Demo
 Tutorial
Disclaimer
“This is preliminary software and/or hardware and APIs
are preliminary and subject to change.”
(Kinect for Wind...
Kinect for Windows v1 Sensor
MULTI-ARRAY MIC MOTORIZED TILT
3D DEPTH SENSORS
RGB CAMERA
Kinect for Windows v2 Sensor
MULTI-ARRAY MIC
3D DEPTH SENSOR
( IR Camera + IR Emitters )
RGB CAMERA
Kinect for Windows v2 Sensor
Image by iFixit
IR EMITTERS
IR CAMERA
Specifications
Kinect for Windows v1 Kinect for Windows v2
Color 640×480 @ 30fps 1920×1080 @ 30fps
Depth 320×240 @ 30fps 5...
Specifications
Kinect for Windows v1 Kinect for Windows v2
BodyIndex 6 people 6 people
Body 2 people 6 people
Joint 20 joi...
Basic Features
 Color
 1920×1080@30fps / 15fps (Lighting Condition)
 RGBA, YUV, BGRA, Bayer, YUY2
Basic Features
 Depth
 512×424@30fps
 500~4500[mm]
 ToF (Time of Flight)
Basic Features
 Infrared / LongExposureInfrared
 512×424@30fps
 16bit (higher 8 bits)
Basic Features
 BodyIndex
 512×424@30fps
 6 people
 Body Area : 0~5, Other Area : 255 (5 < Index)
255
0 1
Basic Features
 Body
 6 people
 25 joint / people (Add Tip, Thumb, Neck)
 Orientation (Quaternion)
 Hand Type (Right,...
Basic Features
 Audio
 Beamforming (+/-50 degree)
 Speaker Estimation
 Speech Recognition
Application Features
 Gesture
 Gesture Recognition using Machine Learning
 Discrete (detected true/false), Continuous (...
Application Features
 Face
 Bounding Box, Rotation, Points (Eye, Nose, Mouth Corner)
 Activity, Appearance, Expression
...
Application Features
 HDFace
 For Creating 3D Face Model
 Points (1347), Triangles (2340), Hair Color, Skin Color
 Fit...
Application Features
 Other
 Kinect Fusion (3D Shape Reconstruction)
 Controls (Assist in implementation of NUI)
Demo
System / Software Requirements
OS * Windows 8, 8.1, Embedded 8, Embedded 8.1 (x64)
CPU Intel Core i7 3.1GHz (or higher)
RA...
Connection
Kinect for Windows v1 Kinect for Windows v2
PCPC
Example Multiple Connection
PCPC PC
Hub
Server
System
Kinect
Driver
Kinect SDK
Application
Kinect
Driver
Kinect SDK
Application
Kinect
Service
Kinect SDK
Application
Kin...
Tutorial
 Basic Flow of Programming (C++)
Sensor Stream Frame Data
Sensor Source Reader Frame Data
Kinect for Windows SDK...
Tutorial
 Sensor
Sensor Source Reader Frame Data
// Sensor
IKinectSensor* pSensor;
HRESULT hResult = S_OK;
hResult = GetD...
Tutorial
 Source
Sensor Source Reader Frame Data
// Source
I***FrameSource* pSource;
hResult = pSensor->get_***FrameSourc...
Tutorial
 Reader
Sensor Source Reader Frame Data
// Reader
I***FrameReader* pReader;
hResult = pSource->OpenReader( &pRea...
Tutorial
 Frame
Sensor Source Reader Frame Data
while( 1 ){
// Frame
I***Frame* pFrame = nullptr;
hResult = pReader->Acqu...
Tutorial
 Data (Depth, Infrared, BodyIndex)
Sensor Source Reader Frame Data
while( 1 ){
// Frame
IDepthFrame* pDepthFrame...
Tutorial
 Data (Color)
Sensor Source Reader Frame Data
while( 1 ){
// Frame
IColorFrame* pColorFrame = nullptr;
hResult =...
Tutorial
 Data (Color with Converted Format)
Sensor Source Reader Frame Data
unsigned int buffeerSize = width * height * ...
Tutorial
Sensor Source Reader Frame Data
while( 1 ){
IBodyFrame* pFrame = nullptr;
hResult = pBodyReader->AcquireLatestFra...
Tutorial
Sensor Source Reader Frame Data
for( int count = 0; count < BODY_COUNT; count++ ){
BOOLEAN bTracked = false;
hRes...
Tutorial
 Coordinate System
 ColorSpace (Coordinate System of the Color Image)
… Color
 DepthSpace (Coordinate System o...
Tutorial
 Coordinate Mapper
// Coordinate Mapper
ICoordinateMapper* pCoordinateMapper;
hResult = pSensor->get_CoordinateM...
Reference
 Sample Program
 Kinect2Sample | GitHub
https://github.com/UnaNancyOwen/Kinect2Sample
 Article Series
 Intro...
Upcoming SlideShare
Loading in …5
×

Kinect v2 Introduction and Tutorial

14,546 views

Published on

2014/10/13 中部大学 ロボット理工学科 機械知覚&ロボティクスグループ

Published in: Technology

Kinect v2 Introduction and Tutorial

  1. 1. Introduction to Kinect v2 Tsukasa Sugiura @UnaNancyOwen
  2. 2. Self-Introduction Tsukasa Sugiura Microsoft MVP for Kinect for Windows  @UnaNancyOwen  http://UnaNancyOwen.com  t.sugiura0204@gmail.com (July 2014 - June 2015)
  3. 3. Agenda  What is Kinect v2  Specifications  New Features  Demo  Tutorial
  4. 4. Disclaimer “This is preliminary software and/or hardware and APIs are preliminary and subject to change.” (Kinect for Windows v2は暫定的なものであり、ソフトウェア、ハード ウェア、およびAPIは製品版で変更される可能性があります。)
  5. 5. Kinect for Windows v1 Sensor MULTI-ARRAY MIC MOTORIZED TILT 3D DEPTH SENSORS RGB CAMERA
  6. 6. Kinect for Windows v2 Sensor MULTI-ARRAY MIC 3D DEPTH SENSOR ( IR Camera + IR Emitters ) RGB CAMERA
  7. 7. Kinect for Windows v2 Sensor Image by iFixit IR EMITTERS IR CAMERA
  8. 8. Specifications Kinect for Windows v1 Kinect for Windows v2 Color 640×480 @ 30fps 1920×1080 @ 30fps Depth 320×240 @ 30fps 512×424 @ 30fps Sensor Structured Light (PrimeSense Light Coding) Time of Flight (ToF) Range 0.8~4.0 m 0.5~4.5 m Angle of View Horizontal / Vertical 57 / 43 degree 70 / 60 degree Microphone Array ◯ ◯
  9. 9. Specifications Kinect for Windows v1 Kinect for Windows v2 BodyIndex 6 people 6 people Body 2 people 6 people Joint 20 joint/people 25 joint/people Hand State Open / Closed Open / Closed / Lasso Gesture ☓ ◯ Face ◯ Speech / Beamforming ◯ ◯
  10. 10. Basic Features  Color  1920×1080@30fps / 15fps (Lighting Condition)  RGBA, YUV, BGRA, Bayer, YUY2
  11. 11. Basic Features  Depth  512×424@30fps  500~4500[mm]  ToF (Time of Flight)
  12. 12. Basic Features  Infrared / LongExposureInfrared  512×424@30fps  16bit (higher 8 bits)
  13. 13. Basic Features  BodyIndex  512×424@30fps  6 people  Body Area : 0~5, Other Area : 255 (5 < Index) 255 0 1
  14. 14. Basic Features  Body  6 people  25 joint / people (Add Tip, Thumb, Neck)  Orientation (Quaternion)  Hand Type (Right, Left),Hand State (Open, Closed, Lasso), Lean (-1.0f~1.0f)
  15. 15. Basic Features  Audio  Beamforming (+/-50 degree)  Speaker Estimation  Speech Recognition
  16. 16. Application Features  Gesture  Gesture Recognition using Machine Learning  Discrete (detected true/false), Continuous (progress 0.0f~1.0f)  Learning Classifier Tool “Visual Gesture Builder” Video by http://youtu.be/-XYoblrnDpg
  17. 17. Application Features  Face  Bounding Box, Rotation, Points (Eye, Nose, Mouth Corner)  Activity, Appearance, Expression  Activity … Eye Closed, Mouth Open / Moved, Looking Away  Appearance … Wearing Glasses  Expression … Happy
  18. 18. Application Features  HDFace  For Creating 3D Face Model  Points (1347), Triangles (2340), Hair Color, Skin Color  Fitting Face Model
  19. 19. Application Features  Other  Kinect Fusion (3D Shape Reconstruction)  Controls (Assist in implementation of NUI)
  20. 20. Demo
  21. 21. System / Software Requirements OS * Windows 8, 8.1, Embedded 8, Embedded 8.1 (x64) CPU Intel Core i7 3.1GHz (or higher) RAM 4GB (or more) GPU * DirectX 11 supported USB * USB 3.0 (Intel or Renesas Host Controller) Compiler * Visual Studio 2012, 2013 (Supported Express) Language Native (C++), Managed (C#,VB.NET), WinRT (C#,HTML) Other Unity Pro (Add-in), Cinder, openFrameworks (wrapper)
  22. 22. Connection Kinect for Windows v1 Kinect for Windows v2 PCPC
  23. 23. Example Multiple Connection PCPC PC Hub Server
  24. 24. System Kinect Driver Kinect SDK Application Kinect Driver Kinect SDK Application Kinect Service Kinect SDK Application Kinect SDK Application Kinect for Windows v1 Kinect for Windows v2
  25. 25. Tutorial  Basic Flow of Programming (C++) Sensor Stream Frame Data Sensor Source Reader Frame Data Kinect for Windows SDK v1 Kinect for Windows SDK v2  Source independent to each Data (e.g. ColorSource, DepthSource, InfraredSource, BodyIndexSource, BodySource, …)  Doesn’t depend on each other Source (e.g. Doesn't need to Depth Source when retrieve Body Data)
  26. 26. Tutorial  Sensor Sensor Source Reader Frame Data // Sensor IKinectSensor* pSensor; HRESULT hResult = S_OK; hResult = GetDefaultKinectSensor( &pSensor ); if( FAILED( hResult ) ){ std::cerr << "Error : GetDefaultKinectSensor" << std::endl; return -1; } hResult = pSensor->Open(); if( FAILED( hResult ) ){ std::cerr << "Error : IKinectSensor::Open()" << std::endl; return -1; }  IKinectSensor
  27. 27. Tutorial  Source Sensor Source Reader Frame Data // Source I***FrameSource* pSource; hResult = pSensor->get_***FrameSource( &pSource ); if( FAILED( hResult ) ){ std::cerr << "Error : IKinectSensor::get_FrameSource()" << std::endl; return -1; }  IColorFrameSource  IDepthFrameSource  IInfraredFrameSource  IBodyIndexFrameSource  IBodyFrameSource
  28. 28. Tutorial  Reader Sensor Source Reader Frame Data // Reader I***FrameReader* pReader; hResult = pSource->OpenReader( &pReader ); if( FAILED( hResult ) ){ std::cerr << "Error : IFrameSource::OpenReader()" << std::endl; return -1; }  IColorFrameReader  IDepthFrameReader  IInfraredFrameReader  IBodyIndexFrameReader  IBodyFrameReader
  29. 29. Tutorial  Frame Sensor Source Reader Frame Data while( 1 ){ // Frame I***Frame* pFrame = nullptr; hResult = pReader->AcquireLatestFrame( &pFrame ); if( SUCCEEDED( hResult ) ){ /* Data */ } SafeRelease( pFrame ); }  IColorFrame  IDepthFrame  IInfraredFrame  IBodyIndexFrame  IBodyFrame
  30. 30. Tutorial  Data (Depth, Infrared, BodyIndex) Sensor Source Reader Frame Data while( 1 ){ // Frame IDepthFrame* pDepthFrame = nullptr; hResult = pDepthReader->AcquireLatestFrame( &pDepthFrame ); if( SUCCEEDED( hResult ) ){ unsigned int bufferSize = 0; unsigned short* pBuffer = nullptr; hResult = pDepthFrame->AccessUnderlyingBuffer( &bufferSize, &pBuffer ); if( SUCCEEDED( hResult ) ){ /* Processing*/ } } SafeRelease( pDepthFrame ); }  Depth, Infrared … unsigned short  BodyIndex … unsigned char
  31. 31. Tutorial  Data (Color) Sensor Source Reader Frame Data while( 1 ){ // Frame IColorFrame* pColorFrame = nullptr; hResult = pColorReader->AcquireLatestFrame( &pColorFrame ); if( SUCCEEDED( hResult ) ){ unsigned int bufferSize = 0; unsigned char* pBuffer = nullptr; hResult = pColorFrame->AccessUnderlyingBuffer( &bufferSize, &pBuffer ); if( SUCCEEDED( hResult ) ){ /* Processing*/ } } SafeRelease( pColorFrame ); }  Color … unsigned char  Default Color Image Format … YUY2
  32. 32. Tutorial  Data (Color with Converted Format) Sensor Source Reader Frame Data unsigned int buffeerSize = width * height * channel * sizeof(unsigned char); unsigned char* pBuffer = new unsigned char[bufferSize]; while( 1 ){ // Frame IColorFrame* pColorFrame = nullptr; hResult = pReader->AcquireLatestFrame( &pColorFrame ); if( SUCCEEDED( hResult ) ){ hResult = pColorFrame->CopyConvertedFrameDataToArray( bufferSize, pBuffer, ColorImageFormat::ColorImageFormat_Bgra ); if( SUCCEEDED( hResult ) ){ /* Processing */ } } SafeRelease( pColorFrame ); } delete[] pBuffer;
  33. 33. Tutorial Sensor Source Reader Frame Data while( 1 ){ IBodyFrame* pFrame = nullptr; hResult = pBodyReader->AcquireLatestFrame( &pBodyFrame ); if( SUCCEEDED( hResult ) ){ IBody* pBody[BODY_COUNT] = { 0 }; hResult = pBodyFrame->GetAndRefreshBodyData( BODY_COUNT, pBody ); if( SUCCEEDED( hResult ) ){ /* Processing */ } for( int count = 0; count < BODY_COUNT; count++ ){ SafeRelease( pBody[count] ); } } SafeRelease( pBodyFrame ); }  Data (Body)
  34. 34. Tutorial Sensor Source Reader Frame Data for( int count = 0; count < BODY_COUNT; count++ ){ BOOLEAN bTracked = false; hResult = pBody[count]->get_IsTracked( &bTracked ); if( SUCCEEDED( hResult ) && bTracked ){ Joint joint[JointType::JointType_Count]; hResult = pBody[count]->GetJoints( JointType::JointType_Count, joint ); if( SUCCEEDED( hResult ) ){ for( int type = 0; type < JointType::JointType_Count; type++ ){ if( joint[type].TrackingState != TrackingState::TrackingState_NotTracked ){ CameraSpacePoint cameraSpacePoint = joint[type].Position; cameraSpacePoint.x; // x (+/- 1.0f) cameraSpacePoint.y; // y (+/- 1.0f) cameraSpacePoint.z; // z (500~4500[mm]) } } } } }  Data (Joint)
  35. 35. Tutorial  Coordinate System  ColorSpace (Coordinate System of the Color Image) … Color  DepthSpace (Coordinate System of the Depth Data) … Depth, Infrared, BodyIndex  CameraSpace (Coordinate System with the origin located the Depth Sensor) … Body (Joint)
  36. 36. Tutorial  Coordinate Mapper // Coordinate Mapper ICoordinateMapper* pCoordinateMapper; hResult = pSensor->get_CoordinateMapper( &pCoordinateMapper ); if( FAILED( hResult ) ){ std::cerr << "Error : IKinectSensor::get_CoordinateMapper()" << std::endl; return -1; }  ICoordinateMapper::Map○○○FrameTo△△△Space() … Mapping Coordinate System in the Frame  ICoordinateMapper::Map○○○PointsTo△△△Space() … Mapping Coordinate System in the Array of Points  ICoordinateMapper::Map○○○PointTo△△△Space() … Mapping Coordinate System in the Point
  37. 37. Reference  Sample Program  Kinect2Sample | GitHub https://github.com/UnaNancyOwen/Kinect2Sample  Article Series  Introduction to Kinect for Windows v2 | Build Insider http://www.buildinsider.net/small/kinectv2cpp  Blog  Kinect | Summary?Blog http://unanancyowen.com/?cat=3

×