The document provides an introduction to the Kinect sensor. It discusses Kinect hardware, including its RGB camera, IR depth sensor, and IR projector. It explains the Kinect's time-of-flight depth sensing technology. The document also discusses the OpenNI and NITE middleware for accessing Kinect sensor data. It provides code examples for retrieving depth maps and skeletal tracking data from the Kinect. Overall, the document aims to educate readers on developing applications using the Kinect by exploring its technical capabilities and available development platforms.

1. T he N ot So Short
I ntroduction to Kinect
一份不太简短的 Kinect介绍
Hang xie
sonicisdreaming@gmail.com
http://Xhbang.com

2. About M e
Chongqing University bachelor （单身汉
）
A boy couldn't stop dreaming
Bike travelling

3. About Him
•Dr.
•Sensor related
•Poet （魔幻现实主义）

4. About the Lab
•50% Idle
•25% Computer
•20% Programmer
•5% Student/Paper

5. About You?
Programming dummy?
Linux || Windows?
Understand 3D?
Work with the Kinect?

6. T he Contents
wha u will lea toda
t rn y
体感操作， Kinect ， OPENNI 介绍
基本程序设计
Working with Depth Image
Working with Point Clouds （ * ）
Working with Skeleton Data & Tracking
Kinect Hacks

7. 昨夜西风凋碧树 , 独上高楼 , 望尽天涯
路

10. Buy it!

11. I nside Kinect
Teardown
Principle
Hardware
Performace

12. Hardware & teardown

13. http://www.ifixit.com/Teardown/Microsoft-Kinect-Teardown

16. Kinect Ha re
rdwa
•RGB camera
•IR depth sensor
•IR projector
•Audio device

22. T a a closer look
ke
Noise at edges(shoulders)
Depth shadows
Misalignment between the color and depth images
Frame rate(30 vs 60)

23. Behind Kinect

24. T OF
•Time of Flight
•以色列公司 3DV （ 09 年微软收购
）
•光飞行时间，飞秒 (1e−15) 级快门
•砷化镓，使其时钟频率提高到上百
•高成本，难以适应消费电子品

25. Light Coding （光编码）
•连续的照明（而非脉冲）
•普通的 CMOS 感光芯片（低成本）
•结构光技术（三维纵深的“体编码”
）
•激光散斑（ laser speckle ）
http://www.freepatentsonline.com/7433024.pdf
http://www.wipo.int/pctdb/en/wo.jsp?WO=2007043036
http://www.javaforge.com/wiki/103639

27. Exemplar
•机器学习系统
•TB 计的数据训练集
4.寻找移动部位（背景分离，分割遮罩）
5.辨认身体部位可能性
6.模型匹配：生成骨架系统

29. Audio
•Something we forget
•Supported by OpenNI, not by
SensorKinect

30. M ore About Audio （题外话）
•“Sound Wave”
•微软研究院和华盛顿大学
•利用多普勒效应，使用笔记本内置的扬声器和麦
克风开发出类似Kinect的姿势感知系统
•低成本
•识别精度能达到 90% 以上
•用手势玩俄罗斯方块游戏（视频）

31. Concepts

33. http://en.wikipedia.org/wiki/Natural_user_interface

34. Next generation interface

35. Wii Remote

36. ASUS Xtion

37. Kinect

38. Project Glass

39. Development

41. http://www.openni.org/

42. http://code.google.com/p/simple-openni/

43. https://github.com/OpenKinect/libfreenect
http://openkinect.org/wiki/Main_Page

44. Architecture
Quick view(*)

45. OpenN I
•Short for “Open Natural Interface”
•Multi-langue
•Cross-platform
•Standard & Infrastructure

46. 向后兼容
使用 openNI 开发的程序在以后的
版本中都可以继续使用 , 因为开发
人员可以毫无顾虑的更新 openNI
到最新的版本。

47. OpenNI Framework(1)

48. 抽象分层视图
•Top: 代表软件，在 OpenNI 顶部，实现了自然交互的
应用程序。
•Middle: 代表 OpenNI ，提供通信接口，同时连接传感
器和中间件组件，后者分析来自传感器的数据。
•Bottom: 硬件设备，捕捉现场的视频和音频内容。

49. OpenNI Framework(2)

50. 通用的框架工具集
•一个 USB 抽象访问层
•一些基本数据类型的实现（包括列表，哈希，等等）
•系统日志和转储
•内存和性能分析
•活动（授权要注册的回调函数到一个特定的事件）
•任务调度
•不是标准 OpenNI 的一部分（ * ）

51. Ca ck func
llba
•函数指针
•注册
•事件响应

52. OpenN I の构成
 Generator
 Metadata
 Capability

53. Generator
注：绿色已经有，红色计划中，后略

54. 生产节点
•单位集
•生产性角色
•使用较低水平的生产节点
•“ 有意义的三位数据”

55. Sensor Related
•Device
•Depth Generator
•Image Generator
•IR Generator
•Audio Generator

56. M iddlewa Rela
re ted
•Gesture Generator
•Scene Analyzer
•Hands Generator
•User Generator
•//provided by NITE

57. Record & Play
•Recoder
•Player
•Codec

58. Matadata

59. Capability

60. N I T E の构成
Session
Detector

61. M iddleware
•Full body analysis
•Hand point analysis
•Gesture detection
•Scene Analyzer

62. Session

63. Detector

66. 衣带渐宽终不悔，为伊消得人憔悴。

67. Hello world!
•Preparation
•The program

68. Softwa Prepa tion
re ra
•OpenNI+NITE
•Avin’s SensorKinect
•Processing(*)
•SimpleOpenNI(*)
https://github.com/avin2/SensorKinect

69. Here’s the code,
too young too simple

70. import SimpleOpenNI.*;
SimpleOpenNI kinect;
void setup()
{
size(640*2, 480);
kinect = new SimpleOpenNI(this);
kinect.enableDepth();
kinect.enableRGB();
}
void draw()
{
kinect.update();
image(kinect.depthImage(), 0, 0);
image(kinect.rgbImage(), 640, 0);
}

71. Understand the code

72. import SimpleOpenNI.*;
SimpleOpenNI kinect;
•Import the library
•Declare the SimpleOpenNI object

73. void setup()
{
size(640*2, 480);
kinect = new SimpleOpenNI(this);
kinect.enableDepth();
kinect.enableRGB();
}
•Declare the size of app
•Instantiate the object
•Call two method

74. void draw()
{
kinect.update();
image(kinect.depthImage(), 0, 0);
image(kinect.rgbImage(), 640, 0);
}
•Get fresh data from kinect
•Place the image

75. Working with depth map
•R G B
•纯黑代表无穷远，纯白代表无穷近
•Demo:ex03_basic_depth_plus_rgb

76. M ore about depth map
•Projective
•Realworld
•Z 值相同，代表深度，单位 mm
•ConvertProjectiveToRealWorld()

77. The Kinect coordinate system

78. Demo
chPC_ex12_obj_in_point_cloud

79. 范围以及精确性
•最近距离： about 0.5(?)
•最远距离： 10m
•GetDeviceMaxDepth()
•官方建议： 1.2 – 3.6 米
•近远精确性比较

81. Working with Point Clouds(* )
Welcome to the Third Dimension
3d reconstruction
E.g:KinectFusion PCL
http://pointclouds.org/

82. DEMO
chPC_ex05_rotating_point_cloud

83. Working with Skeleton D ata
•User Detection
•Joints & Skeleton
•Distance in 3D
•Scene Map, Com
•Tracking

84. User D etection
•Calibration or not
•Stages in Process

86. Joints & Skeleton
•Joints
•Distance between two joints in 3D

87. WindowsSDK: 20 joints(all)

88. OpenNI: 24 joints

89. DEMO
chSK_ex02_skel_anatomy

90. Scene M ap
•Background remove
•User pixels
•Center of Mass

91. D emo
•chSK_scene_map
•chSK_scene_image
•chSK_center_of_mass

92. How
•kinect.enableScene();
•sceneMap = kinect.sceneMap();
•if(sceneMap[i] == n)

93. T racking
Demo:PointViewer

94. 众里寻他千百度，蓦然回首，
那人却在，灯火阑珊处。

95. Kinect Hacks
Apps out of box
Create your own cool stuffs
Living coding with VS & Cpp

96. M edia Art
Demo: chDE_face_changing

97. How
•Collision detection
•kinect.getJointPositionSkeleton()
•kinect.convertRealWorldToProjective()
•Picture scala

98. Computer I nterface
Demo:Minority Report(*)
Demo:2D TrackPad
Demo:Hotpoint 3D

99. FAAST
•Key event mapping
•Game playing
http://projects.ict.usc.edu/mxr/faast/

100. Sex Ga Controller
me
•DEMO: 宅男妄想地带
•Microsoft: NO!!

101. Sorry, Bill

102. Control ema )
cs(*
•Map(Joint position + gesture, key event)

103. Kinect Visua Ca (* )
l mera
http://www.kinectdevs.com/forums/kinect-virtual-camera-kinect-webcam-f17

104. The following demo need a
better computer for higher GPU
support

105. kinect-ultra )
(*
一秒钟凹凸曼合体！完成你童年
时候的梦想
Video
http://code.google.com/p/kinect-ultra/

106. kinect-ka meha )
meha (*

107. M ikuM ikuD ance(* )
初音妹子
Video

108. What do you want to make?
http://www.kinect-hacks.com/openni

109. reference
•http://viml.nchc.org.tw
•Visualization and Interactive Media Laboratory of NCHC
•国家科學視算與互動媒體實驗室
•http://www.javaforge.com/wiki/103532
•Kinect for PC Wiki
•中文

110. T hanks!
You a the controller.
re
Q &A
Hang xie
sonicisdreaming@gmail.com
http://www.xhbang.com

111. T his is ba templa
sic te
Do not delete this page