This document discusses a project to use computer vision and a Microsoft Kinect sensor to enable real-time gesture control of a welding robot. The project aims to detect and track a user's hand gestures to control robot movement, and to define the weld seam region of interest to allow for seam detection. The plan involves accessing Kinect data, detecting and tracking the hand in 3D space, recognizing gestures for robot movement commands, extracting color values from the hand for skin detection, and using the hand position to define the seam region of interest. The work so far has successfully defined the hand and fingers, tracked hand motion, and extracted the seam region. Further work is needed to finalize the gesture commands and integrate control of the robot.

1. COMPLEX WELD SEAM DETECTION
USING COMPUTER VISION
Glenn Silvers, 16115327
A presentation submitted for the partial fulfilment for the degree of:
Bachelor of Engineering (Mechatronic and Robotic) (Honours)
Supervisor
Dr Gu Fang

2. THE NEED FOR RESEARCH
 To allow for the effective and simple communication
between humans and machines.
 Simplification of complex setup tasks.
 Reduce time needed to complete tasks.
 Improved repeatability.

3. KEY OBJECTIVES
 To use computer vision to define the users hand to
enable movement via gestures of a welding robot.
 The definition and therefore tracking of the hand
must be real time to allow for adequate control over
the robots motion.
 To define the region of interest where the weld
seam lies to allow for seam detection.

4. CURRENT PROJECTS
 Imirok: Real-time imitative robotic arm control for
home robot applications (Heng-Tze
et al.,2011).
 Recognising Hand Gestures with Microsoft’s Kinect
(Tang, 2011).
 Recognition of Arm Movements (Duric
et al., 2002).

5. MICROSOFT KINECT
http://www.itsagadget.com/2010/11/microsofts-kinect-sensor-is-
now-officially-released.html

6. KINECT EXPLODED
http://hackedgadgets.com/wp-content/uploads/2010/11/inside-the-
microsoft-kinect_2.jpg

7. THE KINECT’S SENSORS
 The Kinect makes use of three different types of
sensors:
 A Depth Sensor
 A Colour Camera
 A Four Element Microphone Array
 Each sensor allows the programmer endless
possibilities in terms of application invention.

8. PLAN OF ATTACK
1.
Accessing
Kinect Data
2.
Detecting and
Tracking the
Hand
3.2 Extracting
3.1 Gesture
HSV Values
Recognition
from Hand
3.1.1
3.2.1
Commands for
Seam ROI
Movement of
Detection
the Robot

9. 1. ACCESSING KINECT’S DATA
 To access the Kinect’s data streams
it is first necessary to understand its data
structures.
 Once the data streams have
been opened, it is then a matter of
converting the data into a usable
format.

10. 2. DETECTING AND TRACKING THE HAND
 OpenCV libraries
 Created by Intel in 1999.
 Initially designed to provide optimised computer
vision code so that programmers would not
need to start projects from scratch.
 Has its own inbuilt data structures.
 Over 500 functions that span many different
areas of computer vision.

11. 2. DETECTING AND TRACKING THE HAND
CONTINUED
 Detecting the hand:
 Utilised the Kinect’s depth stream.
 Applied a threshold to the depth values to create
a depth region of interest where only the hand
would be visible.

12. 2. DETECTING AND TRACKING THE HAND
CONTINUED
 Detecting the hand:
 Once the hand is within the frame it is necessary
for the program to identify it as a hand.
 First step is to find the contours (Suzuki and
be, 1985) of the hand.

13. 2. DETECTING AND TRACKING THE HAND
CONTINUED
 Detecting the hand:
 The next step is to enclose the contour with the
use of the minimum perimeter polygon algorithm
(Sklansky, 1972).

14. 2. DETECTING AND TRACKING THE HAND
CONTINUED
 Detecting the hand:
 Looking at points where the contour and the minimum
perimeter polygon meet shows the position of the
fingers.

15. 2. DETECTING AND TRACKING THE HAND
CONTINUED
 Detecting the hand:
 To identify the fingers only a weighted average is
performed.
 This rejects all points lower than the thumb identifying
the four fingers and thumb.

16. 3.1 GESTURE RECOGNITION
 The robot being used in this research is a
Fanuc 100iC with a Lincoln Electric
welding attachment.
 As this robot has six degrees of freedom
there will need to be six different gestures
to allow for complete control.
 At this point in time there have been three
gestures coded and tested.
 The remaining gestures need to be coded
into the program.

17. 3.1.1 COMMANDS FOR MOVEMENT OF THE
ROBOT
 Once the gestures have been recognised by
the program, the robot movement speed will
be limited to 10%.
 The reason for this is simply to ensure the
health and safety of the users.
 Whilst the operator continues to make a
gesture to the Kinect the robot will continually
move in accordance with that gesture.
 As a safety precaution, for all gestures to be
recognised five fingers need to be identified.

18. 3.2 EXTRACTING HSV VALUES FROM THE
HAND
 Obtain colour image from Kinect.
 The colour image is converted to the HSV colour
space.
 An XY depth point from the hand is transformed
into the colour image.
 A HSV value is extracted for the hand.
 Makes use a Kinect function to transform
point positions between the depth image and
colour image.

19. 3.2 EXTRACTING HSV VALUES FROM THE
HAND CONTINUED
 The below images show the HSV value
extraction.

20. 3.2.1 SEAM REGION OF INTEREST
DETECTION
 A gesture causes an image to be taken
from an on-board camera.
 The user then moves their hand into the
image and after five seconds, video from
the on-board camera starts recording.
 The tip of the finger is then run along
the joint or seam that is to be welded.
 The same gesture is used to stop the
recording.

21. 3.2.1 SEAM REGION OF INTEREST
DETECTION
 The reason for the five second delay:
 So that the user has enough time to move their
finger to the starting point of the seam before
the recording starts.
 This ensures that the correct starting point
of the seam is identified.

22. 3.2.1 SEAM REGION OF INTEREST
DETECTION CONTINUED
 To identify the tip of the finger:
 The image from the on-board camera and the first
image from the video are converted into the HSV
colour space.
 The images are thresholded using the HSV values
extracted earlier.
 The first image of the video is the subtracted
from the original image without the finger in it.
 All that is left is the hand and some background
noise.

23. 3.2.1 SEAM REGION OF INTEREST
DETECTION CONTINUED
 The below images show the image
subtraction:

24. 3.2.1 SEAM REGION OF INTEREST
DETECTION CONTINUED
 To ensure that the hand as a whole is
identified the image is dilated to give a
stronger response:

25. 3.2.1 SEAM REGION OF INTEREST
DETECTION CONTINUED
 From this image the contours and
minimum perimeter polygon of the hand
are calculated:

26. 3.2.1 SEAM REGION OF INTEREST
DETECTION CONTINUED
 Once again where the contour and
minimum perimeter meet is the region of
interest (white circles).

27. WHAT HAS BEEN ACHIEVED
 Definition of the hand and fingers.
 Tracking of the hand and fingers in real time.
 Successfully defined the region of interest of the
seam to allow seam detection.

28. WHAT STILL NEEDS TO BE ACHIEVED
 Hand gesture coding needs to be finalised – 80%
 Communication with the robot.
 Complete an in-depth analysis of how repeatable
the methods set out are with many different skin
types.

29. THE MAJOR STEP FORWARD
 The major step forward from this research has been
the way in which the hand is identified within the
colour image.
 By defining the hand in the depth image and then
extracting HSV values from the colour image, a
hybrid skin detector is formed.
 No matter what race or skin colour the user has,
this method will be able to segment their hand to
allow for seam ROI definition.

30. REFERENCES
 HENG-TZE, C., ZHENG, S. & PEI, Z. Imirok: Real-time imitative robotic arm control for home
robot applications. Pervasive Computing and Communications Workshops (PERCOM
Workshops), 2011 IEEE International Conference on, 21-25 March 2011 2011. 360-363.
 TANG, M. 2011. Recognising Hand Gestures with Microsoft’s Kinect. BEng (Electrical), Stanford
University.
 DURIC, Z., LI, F. & WECHSLER, H. Recognition of arm movements. Automatic Face and
Gesture Recognition, 2002. Proceedings. Fifth IEEE International Conference on, 21-21 May
2002 2002. 348-353.
 http://www.itsagadget.com/2010/11/microsofts-kinect-sensor-is-now-officially-released.html.
Accessed 16/4/2012.
 http://hackedgadgets.com/wp-content/uploads/2010/11/inside-the-microsoft-kinect_2.jpg.
Accessed 16/4/2012.
 SUZUKI, S. & BE, K. 1985. Topological structural analysis of digitized binary images by border
following. Computer Vision, Graphics, and Image Processing, 30, 32-46.
 SKLANSKY, J. 1972. Measuring Concavity on a Rectangular Mosaic. Computers, IEEE
Transactions on, C-21, 1355-1364.