This document is the final report for an automated optical inspection system project. The system was designed to detect errors in printing up to 0.5mm at a rate of 1 image per 8 seconds while covering an area of 32" x 41". It uses an array of two cameras mounted on an adjustable arch to capture images which are processed by software on a laptop to detect errors and notify workers. The project was tested and modifications are suggested for future improvements.

1.
IGEN 330 Final Report
Submitted April 7, 2015
Zefan Sramek, Jasmin Liang, Ryan Corkery, Gurjot Bal, Michael Harvey, Rahim Moosa,
Darshan Soni

2.
Executive Summary
Optical quality assessment, more commonly known as ​Automated Optical Inspection
(AOI) ​[Glossary], is used in a variety of industries. The printing industry faces considerable
losses due to errors in the printing process itself. The aim of this project is to develop a cost
effective system to detect errors in the printing stage and allow for corrections to be made
sooner. The primary specifications of our project are that it must be able to detect errors up to 0.5
mm small and be able to cover an area of 32” X 41”. A system comprising of an array of two
cameras, a three piece mount and a laptop running on linux were chosen to meet the design
goals.
With the various specifications in mind, the project was then split into three design
subgroups: camera array, camera mount, and software. The camera array is responsible for the
detection, capturing and uploading of images of the moving labels with potential flaws. These
design goals are met by using a development kit associated with the cameras, and uses built in
commands to accomplish these functions.
The camera mount is responsible for designing an arch that is able to safely hold and
stabilize the cameras, and be resizeable for various sizes of printed operations. The chosen
design is dimensioned in SolidWorks and is built by hand using ​ABS​ [Glossary] piping.

3. The software is responsible for aligning and analyzing the captured photos for any errors.
If any errors are found in the photo, a notification is sent to the worker. All of this is
accomplished through scripting in C++ and using the library ​OpenCV ​[Glossary].
A prototype was assembled with these sub­groups. The testing of the prototype, and its
results are documented below. Future improvements to the prototype are investigated.

4. Table of Contents
1.0 ​Project Design
1.1 ​Needs Statement
1.2 ​Problem Specifications
1.3 ​Initial Design
1.4 ​Final Design
1.4.1 Camera Array
1.4.2 Software
1.4.3 Camera Mount
2.0 ​Assembly and Commissioning
2.1 ​Prototype Assembly
2.2 ​Production Model Assembly
3.0 Project Testing and Modifications
3.1 ​Individual Testing
3.2 ​Overall System Testing
3.3 ​Results
3.4 Modifications
4.0 ​Future Work
5.0 ​Conclusion
6.0 ​Appendices
7.0 ​Glossary
8.0 ​References

5. Table of Figures
Figure 1: Flowchart of the process
Figure 2: Representation of two different camera arrays
Figure 3: Geometric model of the mount done using SolidWorks
Figure 4: Camera motion detection grid, 6X6
Figure 5: Camera motion detection grid, 1X6
Figure 6: With error on right leg of Koala
Figure 7: Identical photos with different brightness and associated error
Figure 8: Images in grayscale outlining edges and errors
Figure 9: Different levels of Pixel brightness are converted to completely black or white
Figure 10: All errors are detected and are present in white
Figure 11: Typical Input Image
Figure 12: Corners identified including inner corner
Figure 13: Result After Alignment
Figure 14: Slight Variations Detected
Figure 15: Label with error detected
Figure 16: Error detection of tested label
Figure 17: Initial stand drawings
Figure 18: Dimensions of the frame
Figure 19: Tested label placed on a black poster to detect edges and corners
Figure 20: Testing done on tripod
Figure 21: Black poster board background with white labels being used for testing
Figure 22: Black cloth background with white labels being used for testing

6. 1.0 Project Design
1.1 Needs Statement
The aim of this project is to develop a cost effective system to detect errors in the printing
stage for silk screen printers and allow for corrections to be made sooner. The label creation
process includes several stages after the printing stage, most of which are more complicated and
more costly than printing. Implementing an automated optical inspection (AOI) [Refer to
Glossary] system into the screen printing process will allow flaws to be detected before these
later stages. Production time is reduced if prints with errors are removed from the production line
immediately and feedback is given so that the errors are not replicated. The product will cut
production and operating costs by replacing manual error detection at the end of the line, with
automated optical inspection at the beginning of the line.
1.2 Problem Specifications
To gain more insight on the specifications of the project, a tour was conducted at Ampco.
Ampco is currently Metro­Vancouver’s largest printing company, and provided relevant
specifications relating to printed labels. The majority of products printed by Ampco are to their
internal A standard, as described in Visual Quality Inspection for Overlays, Panels and Graphics
[1]​
. The specifications for error detection come from this standard. No more than a single error is
permitted per item. The only permitted errors are dots with a diameter of less than 0.5mm and
dust particles with a diameter of less than 0.5mm [Table 1]. The most common type of error
seen, and the hardest to detect by eye are dots, pin holes, and dust particles. As a result the
primary error detection specs are:

7. 1. Detect the presence of multiple errors on a single label on a multi­label sheet
2. Detect the presence of dots and pin holes of 0.5mm diameter or greater
3. Detect the presence of dust particles or their results of 0.5mm diameter or greater
Beyond these error detection specifications, other specifications are necessary to be met
in order to create a system that will operate efficiently with Ampco’s current printing method.
The majority of errors occur during the screen­printing stage of label production. The error
detection system must, therefore, interface with the printer used by Ampco. The primary
integration specifications are:
1. Process images at the same rate as prints are produced (approximately 1 image per 8 seconds)
2. Self­support in such a way as to not interfere with the printing process
3. Process prints up to a maximum size of approximately 32” x 41”
Once the system detects an error, it is necessary to remove that print. Since most errors
are caused by a dirty screen with residue ink​[2]​
, screen cleaning is required. The cleaning is to be
done by a worker.

8. 1.3 Initial Design
Upon examining the needs and specifications, the problem was broken down into 5
categories or stages:
1. Acquire Ideal Label for Comparison (Optional)
2. Capture Label for Processing
3. Mount Capture Technology
4. Process Image to Detect Presence of Errors
5. Notify Worker of Error
Figure 1: Flowchart of the process
Multiple methods to detect labels, mount the system, and to analyze the labels were
investigated. These included which types of cameras to use, the shape and material needed for
the mount, and what system was required to run the code for analysis.
By comparing various aspects of three areas of interest (camera type and array, mounting
hardware, computer hardware) using design matrices, it was finalized to use a system comprised
of two Canon PowerShot ELPH130IS cameras to capture the image, a three piece arch to mount
the system, and a laptop running on Linux to process the image​[3]​
.

9.
Figure 2: Representation of two different camera arrays
Figure 3: Geometric model of the mount done using SolidWorks
1.4 Final Design
1.4.1 Camera Array
The category of the camera array relates to the two Canon PowerShot ELPH130IS
cameras and their functionality. With the cameras in mind, three design goals were set out:
1) Be able to consistently detect a moving label, and notify the system when the entire label
is in range of the camera.
2) Trigger both cameras at the same time, and stitch the photos together to create an image
capturing the entire label.
3) Upload the image to a computer, and make it accessible to analyze.

10.
Initially, there were some issues, as the means to program the cameras, Canon’s source
development kit (SDK), was not available for the point and shoot models. A possible solution
was to change the scope of the project, and use a single ​digital single­lens reflex (DSLR)
[Glossary]​ ​camera which would be supported by Canon’s SDK. These cameras however were
priced at a minimum of $450, and were not deemed suitable for the project and its limited
budget.
Through further research, an unofficial development kit, the ​Canon Hack Development
Kit (CHDK) [Glossary]​, was found that accomplishes the same tasks as the official SDK, but
supports point & shoot type cameras. The CHDK is loaded onto an SD card, and inserted into the
camera, enabling the user to upload scripts to the camera. An extension application, the ​Picture
Transfer Protocol Cam (PTPCAM) ​[Glossary], is also used to access CHDK’s functionalities
remotely with a USB cable and computer.
In the first week, work was focused in triggering both cameras remotely. A script was
written in Bash that incorporated both PTPCAM and the CHDK commands to loop the cameras,
and made them shoot in synchronization. The following week, the script was edited to upload
and delete photos off the camera to a given directory by using the PTPCAM extension.
To detect moving labels, research was done involving different types of sensors. It was
found that the cameras had built in Complementary metal–oxide–semiconductor (CMOS)

11. sensors, that work as colour sensors, by capturing light and converting it into electrical signals.
The CMOS sensors were chosen, as they accomplished the main goals and fit within the budget.
A script was written in bash that incorporates the sensors.
The camera’s image range is split up into a array (6X6 for chosen design) of squares
[Figure 4].
Figure 4: Camera motion detection grid, 6X6
Each square represents a portion of the entire image with an associated threshold
sensitivity value. Since the camera can’t actually detect motion, it instead detects the change in
pixelation. When an object moves past the camera, the colour and thus pixelation will change.
The CMOS sensor picks up this change of colour and assigns a threshold sensitivity value to the
rate of which the colour is changing.
With these aspects in mind, a singular column of the 6X6 array is chosen to be the
‘activation zone’. In the ‘activation zone’, a parameter is set to the threshold sensitivity value,
and will trigger the cameras if this parameter exceeds this value [Figure 5]. After testing, and

12. finding a numeric value suitable to detect labels entering the ‘activation zone’ [Glossary], the
script was finalized and incorporated with the software.
Figure 5: Camera motion detection grid, 1X6
1.4.2 Software
The specific goals of the software side of the project were to be able to detect dots and
pinholes, the presence of dust particles, and to be essentially unaffected by the layout and design
of the labels being printed. In essence, the goal was to detect the presence of deviations from a
set design, known at the onset of printing. Seeing as the initial label design was to be set from the
beginning of the printing, it was determined that the most efficient method by which to detect
errors would be to compare between a “perfect” image and the photograph of a newly printed
label.
The first step in processing a label for defects is to check that a photo of the label was
taken and saved onto the computer. The system does this by looking into the specified computer
directory and checking if a file exists. After analysis, the image is deleted from the directory
before the next photo taken is saved into the same directory.

13. It was necessary to determine a method for processing the images that would include the
required tools and would execute in a limited time. The C++ library OpenCV was chosen
because it was designed precisely for the type of task required for this project and works with the
fast­executing C++ language.
The first goal in the development of the error­detection software was to detect a
difference between two otherwise identical images. In OpenCV, and more generally in digital
systems, images are represented as matrices of colour data. For a standard colour image, the
matrix contains vectors with three values, one for the red, the blue, and the green values that
make up the colour of each pixel. Thus, the preliminary algorithm simply iterated through the
matrices of the “perfect” image and the test image and marked any pixels with different colour
values in bright green [Figure 6].
Figure 6: With error on right leg of Koala

14. This method was effective, but only for images that were mathematically identical in
every way except for the defect. This meant that the same image with different brightness levels
would result in an error being detected at every pixel [Figure 7].
Figure 7: Identical photos with different brightness and associated error
The solution was to simplify the image and remove unnecessary data. To detect a defect, it
was necessary to detect a sudden shift in colour. The colour itself, however, was unimportant. The
types of labels being scanned were likely to be simple and blocky, with little intricate detail. These
facts allowed a new, more robust algorithm to be developed. Since the colour of the images was
irrelevant and sudden changes in colour needed to be detected, a method making use of edge
detection algorithms became apparent as a possible solution.
The first step of the algorithm is to apply a​ Gaussian blur ​[Glossary] to remove noise and
convert the images to ​grayscale ​[Glossary]. This is achieved using OpenCV functions. Next the
gradients of each image are taken in the X and Y directions and stored into matrices using the
Sobel operator ​[Glossary]. When these matrices are overlayed, they generate an image in which
the edges of colour regions are white while the rest of the image is black [Figure 8].

15.
Figure 8: Images in grayscale outlining edges and errors
Once this operation is complete, the errors become particularly apparent to the human eye.
However, the simple comparison algorithm still has difficulty identifying only the errors. The
brightness of the pixels tracks the actual gradient of the image, and is still affected by the original
images’ brightness differences. As a result, further processing is necessary.
To deal with the different levels of pixel brightness, a threshold is applied to the images,
colouring the pixels either completely white or completely black [Figure 9].
Figure 9: Different levels of Pixel brightness are converted to completely black or white
As visible above, the images still contain a fair bit of noise at this stage, and thus still result in
imperfect error detection when compared. This final stage is to remove the noise present in the

16. image resulting from the comparison by subtracting the completely black and white “perfect”
image from the result and making any negative values black. This results in an image where only
the errors are visible in white [Figure 10].
Figure 10: All errors are detected and are present in white
This method of error detection is still affected by deviations in alignment of the test
image. Naturally, unavoidable shifts in alignment with the camera will throw the algorithm off.
A typical input image to the system from the camera includes the label itself and a large portion
of background [Figure 11].
Figure 11: Typical Input Image

17. In order to allow the comparison algorithm to work, the background must be removed
and the label must be aligned. This perspective shift and cropping can be achieved using another
function of OpenCV. This function takes a set of four x­y coordinates from one image and maps
them to another set of four x­y coordinates on another image. The more complex step involved in
the alignment process is the locating of the corners of the label in the input image. The corner
detection is achieved using the OpenCV implementation of the ​Harris corner detection
algorithm​ [Glossary]. This algorithm works by calculating the change in pixel values over an
area and identifies a corner when there is a significant change of colour in multiple directions.
Running this algorithm often identifies multiple coordinates for each corner and thus the next
step is to remove duplicate points. This is achieved by identifying points within five pixels of
each other and removing all but one.
Figure 12: Corners identified including inner corner
Once only unique points are left, it is necessary to identify the points that are the outer
corners of the label. Often other corners are identified within the image on the label and these
must be disregarded [Figure 12]. This is done by calculating the distance from the centre of the

18. image to each corner and taking the 4 points with the largest distances as the four outer corners.
This techniques assumes that the background is smooth and free of any sudden changes in colour
or shade. Once the outer corners are identified, they are sorted based on their position relative to
the quadrants of the original image and sent to the function to apply the alignment, as described
above [Figure 13]. Once the alignment is complete, the images are sent to the comparison
algorithm.
Figure 13: Result After Alignmen​t
Although the alignment technique makes it possible to compare images with skewed
labels, the comparison algorithm as described above still results in the detection of non erroneous
label features due to slight variations in the exact pixel locations of the corners [Figure 14]. Thus
the output of the original comparison algorithm does not accurately identify only errors.

19.
Figure 14: Slight Variations Detected
In order to identify the areas that constitute real errors, a blob detection algorithm is run
on the output image [Figure 15]. This OpenCV function identifies large regions of neighbouring
identical pixels and outputs their coordinates. Applying this algorithm with a threshold blob area
of fifty pixels and blob colours of black or white allow only true errors (blobs of large enough
area) to be detected and identified with red circles on the original image [Figure 16].
Figure 15: Label with error detected

20.
​Figure 16: Error detection of tested label
1.4.3 Camera Mount
The camera mounting hardware went through several design iterations, as the
requirements of the mount changed. Through the use of a design matrix, the first design was a
large steel frame that offered a large degree of adjustability [Figure 17]

21.
Figure 17: Initial stand drawings
As it became clear that the product would not be delivered to Ampco, it was decided that
the frame no longer needed such a large investment. A cheaper design that could be built by hand
was investigated. The first component considered was the material of the frame. ABS piping
replaced steel as the material for the frame, due to ABS piping’s lower cost and ease of
manufacturing. The camera housing was initially one large box intended to hold both cameras
and the components of a desktop PC. The desktop PC was removed from the design and replaced
with a laptop, reducing the components in the box to only the two cameras. The best option was
then to use two small boxes rather than one large box.

22.
Figure 18: Dimensions of the frame
2.0 Assembly and Commissioning
2.1 Prototype Assembly
The prototype was manufactured manually. This is the most effective method because of
the integration between hardware and software components. The hardware assembly uses simple
ABS piping for the stands as it is readily available in different sizes and is easy to process. The
piping is cut to stand measurements. The software is assembled by simply integrating the two
parts of the code (camera & picture analysis scripts) into one executable file on the laptop.
2.2 Production Model Assembly
The mass production model will be very similar to the prototype but the ABS piping will
be replaced by a simple steel framework with adjustable slots. Steel will give the frame the
durability required to last in an industrial environment.

23. The 3 piece arch will be held together with large pins. They are capable of holding the
frame in place for long periods of time, but are also easily adjustable.
With each aspect of the project covered, each part was assembled into the initial
prototype, and testing was then done.
3.0 Project Testing and Modifications
3.1 Individual Testing
Initial testing was done to determine the reliability of the software, consisting of only the
camera triggering code and error analysis/alignment. To test the image analysis and alignment
code, photos of various labels were taken, some with errors, on top of a black poster board. A
black poster board was used as a background to allow easier detection of the edges and
corners [Figure 18].
Figure 19: Tested label placed on a black poster board to detect edges and corners

24. However, there were problems that were encountered when the code was executed. The
corner detection script would find less than 4 corners, and therefore was not able to re­align the
image correctly. It was thought that these issues were caused because:
a) The background was not dark enough and so the contrast of the white label to the black
background was not sufficient for edge detection.
b) Images were taken by hand, throwing off the focus of the camera and making the
image blurry.
To get around this, testing was then done using a tripod to hold the camera, setting an
exact range for the focus through CHDK, and by spray painting the poster black to emphasize
the contrast [Figure 20].
With the updated testing conditions, the 4 corners of the label could be detected, however
not consistently. This is believed to be the result of unevenness in the spray painting, leading to
different shades on the poster board. Albeit the inconsistency, the testing showed that the image
realignment script worked as intended.
Testing involving the label detection was also done to determine the reliability of the
code. The camera was set on the tripod facing a black background. A piece of paper held by hand
was then passed by the camera to simulate a label moving down the production line. As
expected, once the paper passed by the ​activation range​ [Glossary] of the camera, the camera
triggered, took a photo, and uploaded it to a given directory on the computer system.

25.
Figure 20: Testing done on tripod
3.2 Overall System Testing
To test the overall system, the physical and software components of the project were
combined into one system. All components of the software including camera triggering, image
analysis, and image alignment were combined into an executable file to be run on a laptop. The
camera was mounted on the three piece arch a vertical distance away from a table [Figure 21].
The camera was then triggered remotely for error analysis and alignment. Some problems were
encountered in the testing of the system:
Camera focus: Focus was an issue as the alignment script was very sensitive, and would
be thrown off if the image was not perfectly focused. The camera’s auto­focus was not deemed
suitable and instead a manual focus command was used. By measuring the distance between the
table and camera, the exact focal distance could be set in the code. This allowed the camera to
take a clear, sharp image of the label that was suitable for error detection.

26.
Corner detection: As in the individual component testing, a recurring problem was that
less than 4 corners were detected. In attempt to improve the results from individual testing, a
black plastic table cover was used to lay the labels on. This was not ideal as parts of the black
cover displayed glare which possibly interfered with error computation. A black cloth cover was
then used instead, and as expected, the errors in the label were consistently detected. The
extreme contrast of the black cloth and the white label was the solution to the problem.
Figure 21: Black poster board background with white labels being used for testing ­ corners not
reliably detected

27.
Figure 22: Black cloth background with white labels being used for testing ­ corners detected
reliably
3.3 Results
Through testing, it was shown that the system was able to consistently pick up most
errors in simple images to a max size of 18” by 18”. The label detection and error analysis code
were able to work together to provide consistent results, and the mount was able to safely house
the entire system while stabilizing the cameras to provide clear photos.

28. In the error analysis, only the approximate area of the errors are labeled [Figure 16].
Tests were done using a variety of different labels with sample errors drawn on, and proved to
consistently detect and label said errors.
There were some issues relating with the PTPCAM extension and file uploading. When
the photo is uploaded, the cameras must reboot before they can start detecting again. These
reboot times would take only approximately 8 seconds, so would still meet the specifications of
the project.
3.4 Modifications
Modifications and additions are required in the software. The initial specifications of
detecting labels 32” by 41” were not met. To meet this range, additional coding would be
required to stitch two photos together. Other modifications to the code would be to work on the
error detection, as the current code shows only the approximate area of faulty pixels.
4.0 Future Work
Due to time constraints and complications with image analysis, the prototype was not
able to meet the specification of detecting labels 32” by 41”. Future work would involve creating
an image stitching script that utilizes two cameras to capture a bigger area. Work would also be
put aside to develop a more reliable code to detect errors.
Using CHDK, tethered shooting could be supplemented with live view on a computer
monitor. This would allow the user to see the image of the label and to adjust the focus before

29. the camera is triggered. This would increase the reliability of the error detection by reducing
unfocused label images.
5.0 Conclusion
The overall scope of the project changed during design. Various design aspects, such as
the number of cameras used, what computer system to use, and the material of the mount
changed during the design. These changes were necessary to meet deadlines and budgetary
constraints. Overcoming these design challenges helped develop problem solving skills.
Project design skills were also developed throughout the course of the project. Relevant
skills learned include the ability to study a market, determine whether the market is suitable to
enter, and how to develop a project to meet specifications effectively.
Other skills involve teamwork skills. These include being able to effectively
communicate individual progress to the group, splitting up work evenly and efficiently, and
having a general scope of each role.
Overall, the project goals of the project were accomplished by creating a device that
detects errors in printed labels. Although the system did not turn out exactly as envisioned, the
final label inspection system is able to take a photo of a label, align the label properly for error
detection, and detect errors in the label.

30. 6.0​​Appendices
Cost Breakdown
Bill of Materials
ABS piping ­ $59.76
ABS joints ­ $28.52
Nuts and Bolts ­ $11.04
ABS cement ­ $6.20
Cameras ­ $252.97
2 SD Cards ­ $22.01
Black table cloth ­ $10.02
Black paint ­ $16.78
A fully functioning AOI system ­ Priceless
Total ­ $407.30

31.
Table 1: ​Ampco’s Class A Visual Quality Standard for labels

32.
Table 2:​ Program Flow Chart

33. 7.0 Glossary
ABS (​Acrylonitrile butadiene styrene)​ ­ Amorphous thermoplastic polymer that is low cost and
resistant to heat, chemical, and impact. Often used for appliance housing, luggage, camera bodies, and
various furniture components.
Activation Range ­ ​During motion detection, the camera’s image is separated into a grid system.
The activation range is the selected squares off the grid that when triggered will cause the camera
to shoot. This is to provide more control over the process and trigger the camera only when the
entire image is in range.
AOI (automated optical inspection)​ ­ A system using a camera to scan an image for
manufacturing errors and quality defects.
CDHK (Canon Hack Development Kit)​ ­ An unofficial development kit, similar to Canon’s
SDK but aimed at point and shoot cameras. The downloaded CHDK file is loaded onto a SD
card and inserted into the camera, enabling the user to upload scripts and extensions to the
camera.
DSLR (Digital Single Lens Reflex)​ ­ A digital camera with a digital imaging sensor and
interchangeable lens. Images are typically of higher resolution and quality than images taken
with a point and shoot camera.

34.
Gaussian Blur​ ­ Blurring of an image using a Gaussian function to reduce image noise and
detail. Most edge­detection algorithms are sensitive to noise, so this pre­processing stage
enhances image structures at different stages and improves the result of the algorithm.
Grayscale​ ­ A range of colourless shades in gray.
GUI (Graphical User Interface)​ ­ A program interface that allows users to interact with
electronic devices though graphical icons and visual components. Instead of triggering camera
from terminal, GUI would allow worker to click on one icon to take a photo of the label.
Harris Corner Detector​ ­ A function used in image processing that detects corners by detecting
variations in pixel values in an image matrix. Corners are detected when there is significant
change in multiple directions in a small region of pixels.
OpenCV​ ­ An open source library of C/C++ computer programming functions aimed at
real­time computer vision (methods for acquiring, processing, analysing, and understanding
images).
PTPCAM ­ ​An extension that allows the computer to send commands to the camera via CHDK.
Sobel Operator​ ­ A function used in image processing and computer vision within edge
detection algorithms to create an image with defined edges and transitions. ​By computing an

35. approximation of the gradient of the image intensity function, the result of the Sobel operator is a
2­dimensional map of the gradient at each point.

36. 8.0 References
1. Ampco, Visual Quality Inspection for Overlays, Panels and Graphics
2. Tour with Brian Fewtrell
Sr. Manager of Engineering & Quality
October 6th 2014
3. Interim Report 1
4. OpenCV Online Documentation : http://docs.opencv.org/