This document describes a project to develop a gesture-controlled shadow robot. The robot will use a Kinect sensor to track human gestures and mimic the movements. The goals are to develop code to read sensor data, design a robot frame, implement control actuators and drives, use Kinect data for upper body movement tracking, and create basic forward walking. The system will use the Kinect SDK, calculate joint angles from skeleton data, and generate servo signals with an Arduino. Future work may include dynamic walking, finger tracking, and developing an intelligent humanoid.

1. GESTURE BASED WIRELESS
SHADOW ROBOT
Group# 40
A Project Submitted
By
1. Ratul, Ahasan Ulla ID: 13-23070-1
2. Sumnoon, Sharif Md. ID: 11-19449-2
3. Kabir, Sharif Raihan ID: 12-21365-2
4. Islam, Md. Nazrul ID: 12-21730-2
Under the Supervision of
Dr. M. Tanseer Ali
Assistant Professor
American International University -
Bangladesh

2. INTRODUCTION
 In the field of robotics, gesture controlling is becoming
popular lately. People need more user friendly and interactive
robots for their work. Gesture detection makes thing easy for
the users.
 The project proposes a robotic body which mimics the action
of human body

3. PROJECT GOALS
 Develop the shadow code to read the sensor readings.
 Design and develop a skeleton structure for robot frame
 Implementation of control mechanism or Actuators and Drive
Systems.
 Using of Kinect Sensor data for the real time movement of
upper portion of body.
 Basic non-feedback walking sequence for the forward walking
of the Shadow Robot.
 Create save and replay mode the shadow commands .
 Implement a feedback IP camera at robot head.

4. PROJECT BLOCK DIAGRAM

5. THE KINECT SENSOR
1. Infrared (IR) projector.
2. IR depth sensor.
3. RGB camera.
4. Group of tilt motors.
5. Three-axis accelerometer.
6. Microphone array.

6. KINECT SDK
• The Kinect SDK is synthesized set of libraries of software’s
and the tools that is helpful for us for using the Kinect-based
natural input.

7. COLOR STREAM
Color image
format
Resolution FPS Data
InfraredResol
uzion640x480
Fps30
640 x 480 30 Pixel format
is gray16
RawBayerRes
oluzion1280x9
60Fps12
1280 x 960 12 Bayer data
RawBayerRes
oluzion640x48
0Fps30
640 x 480 30 Bayer data
RawYuvResol
uzion640x480
Fps15
640 x 480 15 Raw YUV
RgbResoluzio
n1280x960Fps
12
1280 x 960 12 RGB
(X8R8G8B8)
RgbResoluzio
n640x480Fps1
5
640 x 480 15 Raw YUV

8. DEPTH STREAM
Depth image format Resolution Frame rate
Resoluzion640x480Fps30 640 x 480 30 FPS
Resoluzion320x240Fps30 320 x 240 30 FPS
Resolution80x60Fps 80 x 60 30 FPS

9. DEPTH IMAGE

10. THE BODY POSITIONING
 The body positioning system incorporates depth mapping and
decision tree analysis.

11. SKELETON IMAGE

12. JOINT ANGLES CALCULATION FROM
SKELETON IMAGE
 Skeleton joints tracking from Skeleton Image.
 Skeleton frame data to skeleton Arrays.
 Angle Calculation (ex: Inverse Kinematic
approach) using X,Y,Z co-ordinates of the
joints that is saved in the arrays.

13. INVERSE KINEMATIC ANGLE
CALCULATION

14. ARDUINO AND SERVO MOTORS
The Arduino Mega is used in our project for two
purposes.
 To generate servo signals from the received
strings.
 To calibrate the servo motors for initial
position making and for creating walking
sequence

15. GENERATING SERVO SIGNAL
USING ARDUINO MEGA
END
START
If string Available
Read HC-05
If the slave String is entirely
Complete
Break the string until “&” sign
Convert the rest of the piece into servo angle
Compare servo Id if it matches with
saved Ids
Break it further until “:” for servo ID
NO
YES
YES
NO
NO YES

16. CALIBRATION OF JOINT SERVOS
USING ARDUINO
Arduino can be used to control up to 12 servos with minimum jitter. Two
major parts are there in the application
 The first part is the firmware, that needs to upload in the arduino.
 The second part is to install a simple software named “Serial servo
controller”.

17. COMMUNICATION BETWEEN
MASTER AND SLAVE BLUETOOTH
To enable Wireless connectivity that enables windows
pc bluetooth , C# language is used. Bluetooth can be
programmed by C# in two ways.
 Using Windows Sockets. That is programmed under
windows sockets class.
 Using Computer’s Serial Comport.

18. SENDING STRING DATA FROM
PC BLUETOOTH
if (serial.IsOpen)
{
try
{
String hexstring = ("1" + ":" + LShoulder + "&" + "2" + ":" + LElbow +
"&" + "3" + ":" + RShoulder + "&" + "4" + ":" + RElbow);;
serial.Writeln(hexstring);
Thread.Sleep(1);
}
catch (Exception ex)
{
para.Inlines.Add("Failed to SEND" + data + "n" + ex + "n");
}
}

19. NON-FEEDBACK WALKING OF
BIPED
The walking of the Shadow Robot follows 8 repetitive steps.
I. Stand still
II. Lean Left
III. Right Step forward
IV. Stand still
V. Lean Right
VI. Left step forward

20. DESIGN PROTOTYPE AND
SIMULATION

21. DESIGN PROTOTYPE AND
SIMULATION

22. DESIGN PROTOTYPE AND
SIMULATION

23. APPLICATIONS OF THE PROJECT
 Artificial Intelligence.
 Industrial use.
 Simulation.
 Training & Education.
 Assistive living.
Entertainment.

24. FUTURE WORK
 Design A Dynamic Walk sequence with
feedback from Gyro and accelerometer.
 Finger and Face tracking.
 Making an Trained and Intelligent
Humanoid .

25. REFERENCES
[1] Jungong Han; Ling Shao; Dong Xu; Shotton, J., "Enhanced Computer Vision With Microsoft Kinect
Sensor: A Review," in Cybernetics, IEEE Transactions on, vol.43, no.5, pp.1318-1334, Oct. 2013 doi:
10.1109/TCYB.2013.2265378.
[2] P. Baerlocher and R. Boilic, Inverse Kinematics Techniques for the Interactive Posture Control of
Articulated Figures, PhD thesis, EcolePolytechniqueFederale de Lausanne, 2001.
[3] Machida, E.; Meifen Cao; Murao, T.; Hashimoto, H., "Human motion tracking of mobile robot with
Kinect 3D sensor," in SICE Annual Conference (SICE), 2012 Proceedings of , vol., no.,pp.2207-
2211,20-23Aug.2012.
[4] Xu, D. and Acosta, A. (2005) An Analysis of the Inverse Kinematics for a 5-DOF Manipulator.
International Journal of Automation and Computing, 2, 114-124. http://dx.doi.org/10.1007/s11633-
005-0114-1.
[5] Stephan Waldherr; Roseli Romero and Sebastian Thrun, A Gesture Based Interface for Human-Robot
Interaction. , Autonomous Robots, September 2000, Volume 9, Issue 2, pp 151-173.
[6] W. Xu and E. J. Lee, “Continuous gesture recognition system using improved HMM algorithm based
on 2D and 3D space”, International Journal of Multimedia and Ubiquitous Engineering, vol. 7, no. 2,
(2012), pp. 335-340.
[7] Bhuiyan , M. & Picking, R. (2009). Gesture-controlled user interfaces, what have we done and what's
next?, Proceedings of the Fifth Collaborative Research Symposium on Security, E-Learning, Internet
and Networking (SEIN 2009), Darmstadt, Germany, 26-27 November 2009, pp59-60.

26. ANY QUARIES
For further information email: join2sharif@gmail.com

27. Thank You !