Presentation2

Khaled Hossam Emam
Supervisor : Dipl.-Ing. Jens Friedrich

Faculty of Engineering and Material
Science
Mechatronics Department
The German University in Cairo
Khaled H. Emam 2
 CNC
 Computer Numerical Control
 Applications and usage
 Collision avoidance
 Types of workshop accidents
 Consequence of accidents and misusage
 Multiple sensors
 Cameras
 Kinect

Science
Khaled H. Emam 3
 Introduction
 Kinect sensor
 Image processing techniques
 Methodology and code hierarchy
 Results and conclusion
 Future work and recommendations
 Germany and personal experience (2 min)

Science
Khaled H. Emam 4
Introduction
Kinect
sensor
Image
processing
Methodology &
matlab code
Results and
conclusion
Future work &
recommendations
 Motivation
 Collision avoidance using multiple cameras. Using the theory of
sensor fusion and matlab image processing toolbox.
 Assigned goals
 Detection of collision between cutting tool and machine table with
the workpiece attached
 Detection of collision between workpiece (clamp) and the machine
walls
 Project deliverables
 Building a matlab code for collision detection using three cameras.

Science
Khaled H. Emam 5
Introduction
Kinect
sensor
Image
processing
Methodology &
matlab code
Results and
conclusion
Future work &
recommendations

Science
Khaled H. Emam 6
Introduction
Kinect
sensor
Image
processing
Methodology &
matlab code
Results and
conclusion
Future work &
recommendations
March April May June July
Documentat-
ion and
literature
review
Extracting
information
from Kinect
Object
detection
using Kinect
Fuse webcams
information
with Kinect

Science
Khaled H. Emam 7
Introduction
Kinect
sensor
Image
processing
Methodology &
matlab code
Results and
conclusion
Future work &
recommendations
 Electing a suitable detection algorithm using Kinect for tracking the spindle
and machine table including the workpiece.
Documentat-
ion and
literature
review
Extracting
information
from Kinect
Object
detection
using Kinect

Science
Khaled H. Emam 8
Introduction
Kinect
sensor
Image
processing
Methodology &
matlab code
Results and
conclusion
Future work &
recommendations
 Placement of the two webcams to assist Kinect
 Detection algorithm for the moving objects using webcams
 Fuse the webcams information with Kinect sensor
Fuse webcams
information
with Kinect

Science
Khaled H. Emam 9
Introduction
Kinect
sensor
Image
processing
Methodology &
matlab code
Results and
conclusion
Future work &
recommendations
 Kinect or Kinect for Xbox 360 is a
Controller-free gaming experience
launched by Microsoft for Xbox 360.
 Kinect was one of the used hardware
sensors used in the thesis for the interest
of generating a depth map of the captured
image.
 Linking the device to matlab and windows
Operating system using OpenNI and
prime sense drivers
 Official Microsoft SDK for Kinect
Reference :Wikipedia Foundation
Inc. (nd.).Retrieved from
http://en.wikipedia.org/wiki/Kinect

Science
Khaled H. Emam 10
Introduction
Kinect
sensor
Image
processing
Methodology &
matlab code
Results and
conclusion
Future work &
recommendations
 Converting from u[pixels],v[pixels],Z[mm] coordinates to
X[mm],Y[mm],Z[mm] Cartesian coordinates
 Segmentation of objects using depth map

Science
Khaled H. Emam 11
Introduction
Kinect
sensor
Image
processing
Methodology &
matlab code
Results and
conclusion
Future work &
recommendations
 Digital image
 Gray level image
 RGB images
 Binary image
 Indexed image
 Image processing
 Low level
 Middle level
 High level
 Video processing
 Object detection
 Object tracking

Science
Khaled H. Emam 12
Introduction
Kinect
sensor
Image
processing
Methodology &
matlab code
Results and
conclusion
Future work &
recommendations

Science
Khaled H. Emam 13
Introduction
Kinect
sensor
Image
processing
Methodology &
matlab code
Results and
conclusion
Future work &
recommendations
 Proved efficiency over conventional frame
differencing , median frame difference and
mixture of Gaussians (MOG) background
subtraction techniques.
 No trails are left after a moving object.
 No noise like moving slightly moving
background object
 Keep information about foreground objects
,when they are not in motion for a convenient
time, according to a preset learning rate

Science
Khaled H. Emam 14
Introduction
Kinect
sensor
Image
processing
Methodology &
matlab code
Results and
conclusion
Future work &
recommendations
 Detection stages for SIFT
features are:
1. Scale-space extrema
detection
2. Keypoint localization
3. Orientation assignment
4. Generation of keypoint
descriptors.
 Matching algorithm using dot
product instead of Euclidean
distance

Science
Khaled H. Emam 15
Introduction
Kinect
sensor
Image
processing
Methodology &
matlab code
Results and
conclusion
Future work &
recommendations
Reference: David G. Lowe, "Object recognition from local scale-invariant features," International
Conference on Computer Vision, Corfu, Greece (September 1999), pp. 1150-1157.

Science
Khaled H. Emam 16
Introduction
Kinect
sensor
Image
processing
Methodology &
matlab code
Results and
conclusion
Future work &
recommendations
 Consider a template of an object and a
frame (image) containing this object,
normalized cross correlation is based on
padding a template image on the frame.
 Multiplying both the template and the
frame till reaching the maximum value
which is the location of the object in the
template on the frame.
 The peak in the figure shows the
location of matching between the
template and frame.
Reference: Mathworks www.mathworks.com

Science
Khaled H. Emam 17
Introduction
Kinect
sensor
Image
processing
Methodology &
matlab code
Results and
conclusion
Future work &
recommendations
Webcam 2 Webcam 1

Science
Khaled H. Emam 18
Introduction
Kinect
sensor
Image
processing
Methodology &
matlab code
Results and
conclusion
Future work &
recommendations
Webcam 2 Kinect & Webcam 1

Science
Khaled H. Emam 19
Introduction
Kinect
sensor
Image
processing
Methodology &
matlab code
Results and
conclusion
Future work &
recommendations
1. A saved copy of the
black circle on the
spindle is used to locate
the spindle using
normalized cross
correlation
2. Using matlab boundary
detection function, the
boundary of the spindle
is located so that lowest
part is the cutting tool
3. A copy of the tool is
saved to be used in
multi-sensor fusion

Science
Khaled H. Emam 20
Introduction
Kinect
sensor
Image
processing
Methodology &
matlab code
Results and
conclusion
Future work &
recommendations
 Webcam detects the cutting tool using SIFT matching between a template of
the tool shape captured by Kinect and the frame captured by each webcam.
 I made of the property of the tool shape that it have similar shape from
different viewpoints
Webcam 1 - SIFT Webcam 2 - SIFT

Science
Khaled H. Emam 21
Introduction
Kinect
sensor
Image
processing
Methodology &
matlab code
Results and
conclusion
Future work &
recommendations
 Workpiece and machine table are considered to be a single moving object.
 This object is detected using ABS (Adaptive Background Subtraction) using
the Kinect and the two usb-webcams
 Initial background is already saved for the three cameras with the table is
outside the vision of each camera. Thus a background model is initialized
without the required object.
Background model Current frame Foreground without
filtration ‘noise’

Science
Khaled H. Emam 22
Introduction
Kinect
sensor
Image
processing
Methodology &
matlab code
Results and
conclusion
Future work &
recommendations
 Collision between workpiece ‘clamp’ and the machine wall
 This type of collision is detected using Kinect sensor without the
assistance of the webcams.
 Using the option of camera calibration and converting from pixels
coordinates to the Cartesian coordinates with respect to Kinect, location of
the workpiece is detected after being detected as a foreground abject with
ABS
 Collision between workpiece and cutting tool
 A counter is initialized to count for the collision detected by the three
cameras. Whenever two cameras detect collision, a warning is given to the
operator
 Collision Algorithm will be discussed with the results and conclusion

Science
Khaled H. Emam 23
Introduction
Kinect
sensor
Image
processing
Methodology &
matlab code
Results and
conclusion
Future work &
recommendations
Capture rgb and depth
frames using Kinect
Capture frame
using webcam1
Capture frame
using webcam2
SIFT matching:
webcam1 and tool
template
SIFT matching:
webcam2 and tool
template
Collision Warning
between tool and
table
ABS to detect
table - workpiece
Correlation to locate
spindle and tool shape
Collision warning
between
workpiece and
machine wall
Capture rgb and
depth frames
using Kinect

Science
Khaled H. Emam 24
Introduction
Kinect
sensor
Image
processing
Methodology &
matlab code
Results and
conclusion
Future work &
recommendations
 Detecting this type of
collision showed
success using Kinect
 The figure shows the
foreground object at
collision with the wall
 Collision is given
when the left most
point of bounding box
of the foreground
object is at -500 mm
with respect to the
Kinect sensor

Science
Khaled H. Emam 25
Introduction
Kinect
sensor
Image
processing
Methodology &
matlab code
Results and
conclusion
Future work &
recommendations
 Workpiece-tool collision
is detected using Kinect
using the spindle
boundary shape
 During collision the
lowest part of the spindle
is changed from being the
tooltip to be the lowest
part of the workpiece.
 At tool-table collision,
the lowest boundary
point is the table

Science
Khaled H. Emam 26
Introduction
Kinect
sensor
Image
processing
Methodology &
matlab code
Results and
conclusion
Future work &
recommendations
 The collision is detected by the
interference of the bounding
boxes of the spindle and
foreground objects
 At collision, the collision
counter is incremented
Foreground objects by webcam - error
due to occlusion and background change

Science
Khaled H. Emam 27
Introduction
Kinect
sensor
Image
processing
Methodology &
matlab code
Results and
conclusion
Future work &
recommendations
 Full knowledge of the target objects (workpiece and cutting tool) locations
and shapes.
 Data fusion between multi-sensors in order to have a 3D sense of the scene
within the CNC workspace in order to detect collision
 The average processing time of the whole code is 2~3 seconds which is half
or one third frames per second due to the limited capabilities of the used
computer and matlab is not the optimum program for live image processing .

Science
Khaled H. Emam 28
Introduction
Kinect
sensor
Image
processing
Methodology &
matlab code
Results and
conclusion
Future work &
recommendations
 Link between matlab and the CAD/CAM system in order to know the exact
coordinates of the moving objects for easier segmentation and more accurate
results.
 Using OpenCV software (Open Source Computer Vision)
 Exploring Microsoft Kinect SDK, official Kinect driver.
 More pipelining of the code and using the matlab parallel computation
toolbox, in addition to using a faster computer.

Science
Khaled H. Emam 29
• Stuttgart, Germany
• Student life
• Academic atmosphere
• Cultural exchange

Science
Khaled H. Emam 30
Thanks for listening
Your questions or feedback are highly appreciated
Khaled Hossam Emam
khaled.emam@student.guc.edu.eg

Presentation2

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (20)

Similar to Presentation2

Similar to Presentation2 (20)

Presentation2