"Embedded Vision Made Smart: Introduction to the HALCON Embedded Machine Vision Library," a Presentation from MVTec

MVTec Software GmbH
MVTec is a leading manufacturer of standard software for machine vision and in
business since over 20 years. MVTec products are used in all demanding areas
of imaging: Semi-conductor industry, web inspection, quality control and
inspection applications in general, medicine, 3D vision, and surveillance.
Dr. Olaf Munkelt is one of the co-founders of MVTec Software GmbH and, since its
founding in 1996, also one of the company’s managing directors. Among others, his
main areas of work include supervising the sales department and representing the
company.
Since 2009 Dr. Olaf Munkelt is also serving as the chairman of the board of directors of
the machine vision group within the VDMA (German Engineering Federation).
1

HALCON is on the market since over 20 years!
HALCON is the comprehensive standard software for machine vision with an integrated
development environment (HDevelop) that is used worldwide. HALCON’s flexible
architecture facilitates rapid development of any kind of machine vision application.
MVTec HALCON provides outstanding performance and a comprehensive support of
multi-core platforms, special instruction sets like AVX2 and NEON, as well as GPU
acceleration. It serves all industries, with a library used in hundreds of thousands of
installations in all areas of imaging like blob analysis, morphology, matching, measuring,
identification, and 3D vision.
The software secures your investment by supporting a wide range of operating systems
and providing interfaces to hundreds of industrial cameras and frame grabbers, in
particular by supporting standards like GenICam, GigE Vision, and USB3 Vision.
HALCON Embedded is the comprehensive standard software for machine vision running
on your special platform. With this, MVTec HALCON can be ported to various
microprocessors/DSPs, operating systems, and compilers.
HALCON Embedded allows software engineers to develop the machine vision part of
applications on a standard platform and thereby greatly eases the programming of an
embedded system.
Simply put: Develop on a PC, and the application runs on an embedded system. HALCON
Embedded is available for various smart cameras and other embedded platforms.
2

HALCON 9.0 was chosen to integrate the various sensor data types and perform all the
complex computations in a single development environment. The stereo-vision pair
consists of two Prosilica GC2450 cameras.
These cameras use the GigEVision communication standard. HALCON supports
GigEVision and allows for a quick setup of each camera in software via its automatic
code generation feature. The Time-of-Flight sensor is a Swiss Ranger SR4000.
The tactile force and finger position sensors are custom designed and fabricated. The
Swiss Ranger, tactile, and finger position sensors use custom C/C++ code to perform
depth measurements and tactile object recognition. HALCON’s Extension Package
Programming feature allows to import all custom code into the integrated development
environment, HDevelop which is used for rapid prototyping of our applications.
The stereo camera calibration methods inside HALCON are used to calibrate the stereo
pair and will be used eventually to calibrate the Swiss Ranger.
3

More than 150 million tons of tomatoes are produced every year around the world. With seedlings costing €0.25, plant growers
such as Westland Plantenkwekerij (WPK; Rotterdam, the Netherlands) must ensure that the seeds they receive from their suppliers
will germinate as expected.
Within their contract research organization based at the Wageningen University & Research Centre, the Netherlands, Rick van de
Zedde and his colleagues have been developing vision applications in the agrifood industry for more than 20 years. In this project,
they studied how numerous growers evaluate their tomato seedlings. Although multiple characteristics such as shape and color
were evaluated by the growers, it became clear that the seedling sorting process could be robustly automated by measuring the
mass of each seedling.
FBR designed the machine vision system and Flier Systems, Barendrecht, the Netherlands, currently builds the machine.
Now installed at WPK's facility in Made, the Netherlands, the machine is capable of sorting tomato seedlings at a rate of
18,000/hr. According to Erik van der Arend, owner and director of WPK, the current version of the system sorts seedlings on the
basis of the plant's biomass; it will eventually be upgraded to sort plants based on multiple characteristics such as the plant's
shape, size of the leaves, and defects in shape or color.
As each pot enters the vision station, an optical switch triggers the presence of the plant and 10 individual images of the plant are
captured from numerous angles. The plants are classified according to volume after a high-speed calculation.
"Because of the relatively complex and nonuniform nature of the seedling plants," says van de Zedde, "ten cameras were
required to properly recreate a three-dimensional model of the plants. And still our software is able to generate a 3-D model and
calculate the biomass within 25 msec per seedling."
To calibrate these cameras, a flat checkerboard pattern is used in conjunction with a modified stereo-vision calibration algorithm
available in the HALCON software package from MVTec Software, Munich, Germany.
Because plants are illuminated by a high-frequency fluorescent backlight, the 10 captured images provide multiple views of the
seedlings under inspection from different viewpoints. By subtracting the background from each of these images, a silhouette of
the plant at different viewpoints can be created.
Then by using a technique known as Space Carving, a 3-D rendering of the plant can be created (see Fig. 3). Using this method,
the biomass of each plant can be computed and the data used to classify each plant.
At present, FBR's van de Zedde is installing the next machine at a breeder and is upgrading the vision system to sort seedlings
based on other plant characteristics, including color. "But the productivity gain in the quality of sorting is already huge," says
WPK's van der Arend, "since our investment in this machine will be recovered within four years."
Author: Andy Wilson
Article kindly provided by Vision Systems Design.
4

Whatever programming language you are using in your project, HALCON is likely to have
an appropriate interface.
Through the interface, programmers have access to more than 2500 HALCON operators,
and therefore to easy-to-use fast and robust cutting-edge machine vision algorithms.

Don‘t ask what HALCON can do. The more interesting question is, what HALCON can‘t
do ;-)

Reading of data codes
MVTec software products read ECC 200, QR, Micro QR, Aztec, and PDF417 codes of any size with modules smaller than 2x2
pixels. They can also read data codes with a distorted finder pattern. In addition to printed codes, the software robustly reads
"Direct Part Mark" (DPM) codes and etched codes on different surfaces and under varying illumination conditions.
Reading of bar codes
All common bar codes can be read regardless of orientation - even with an element width of only 1.5 pixels, or, if the code is partly
occluded. The error-free identification of bar codes with our software products is constantly improved.
It bears mentioning that our software reads bar codes particularly in the case of defocus as well as significantly overexposed
images in which the code bars are displayed extremely narrow. It is able to read individual bars that are only five percent of their
original width due to overexposure. In addition, bar codes with up to 95% of "print growth" can be reliably identified as well. Print
growth happens when the bars become much too wide during printing because too much ink was used.
2D metrology
With 2D metrology, you can measure the dimensions of objects that can be represented by specific geometric primitives. The
geometric shapes that can be measured comprise circles, ellipses, rectangles, and lines.
Calibration
Using a calibration plate you can easily calibrate your cameras using HALCON.
Surface-based 3D matching
Recognition and 3D pose determination of arbitrary 3D objects: the cutting-edge 3D matching of MVTec software determines the
position and orientation of 3D objects represented by their CAD model.
Shape-based matching
The software’s superior subpixel-accurate matching technology finds objects robustly and accurately in real-time. It does so even if
they are rotated, scaled, perspectively distorted, locally deformed, partially occluded or located outside of the image, or undergo
nonlinear illumination changes. It can process images with 8 or 16 bits and also handles color or multi-channel images. Objects can
be trained from images or from CAD-like data. Moreover, MVTec’s unique component-based matching is able to locate objects
that are composed of multiple parts that can move with respect to each other. Our local deformable matching finds objects with

deformed or wrinkled surfaces, and our perspective deformable matching robustly localizes objects with perspective
distortions.
8

Besides other technologies HALCON offers a wide range of highly sophisticated matching
technologies:
HALCON allows to locate objects with arbitrary orientation in 3D (3D alignment), it
provides the well established shape-based matching – working even with color images,
the unique component based matching and the well proven normalized cross
correlation.
Since 2008 HALCON features two more matching technologies which can be used for 3D
alignment:
- Descriptor-based matching. This revolutionary new matching technology is able to find
perspectively distorted objects. It is based on the detection of interest points where gray
values are clearly differentiated from neighboring areas (brightness, curvature, corners,
spots).
- Perspective, deformable matching. This new matching technology is also able to match
perspectively distorted objects. In contrast to the descriptor-based matching, the
perspective, deformable matching is edge-based (like HALCON’s shape-based matching)
and thus can best be used with objects with clearly distinguishable edges.

Whatever programming language you are using in your project, HALCON is likely to have
an appropriate interface.

The following AOP concepts were already available in earlier HALCON versions and are
continuously further enhanced:
AOP automatically detects the number of available CPUs and splits the image.
The programmer is able to choose an ROI in an image that even can have an arbitrary
shape (that means HALCON is not restricted to shapes like rectangles which are
common in most other libraries). AOP only processes the ROI of the image. That means
significant saving of processing time, because the runtime only depends on the ROI size,
but not on the image size (as you might find in various libraries).
HALCON processes multi-channel images (e.g., color images) with an unlimited number
of channels. AOP automatically processes every channel in parallel.
Moreover, HALCON is able to process image sequences in parallel.
If HALCON has to process a tuple of regions as an output of segmentation (e.g., OCR,
blob analysis) - which can be thousands of single regions – they also are processed by
AOP.
If XLD (HALCON’s term for subpixel-contours) must have to be extracted, the separate
groups of contours also will be split by AOP. This is also true if contours are further
processed or features are extracted.

Many HALCON customers already build multi-threading applications. E.g., they use
multiple threads for different tasks, e.g., image acquisition, processing, and visualization.
HALCON’s integrated development environment (IDE), HDevelop, has supports parallel
programming and thus allows concurrency. Also, HALCON supports event-based
processing. The call stack or program line view has been extended by a thread view,
which is helpful for debugging multi-threaded programs.
In the thread view, an overview of all currently active threads is shown. By selecting one
of the threads, it is possible to step through single threads for debugging.

In the „traditional“ approach you create the machine vision part of your application in
HDevelop. Once you are satisfied with your program code, you use HDevelop to export
it as C++, C, C#, or Visual basic source code. This code is then inserted into your
application‘s source code, where other essential parts of the application (e.g., user
interface, process integration) are realized. Once done, the final application is compiled.
This approach comes with a number of drawbacks, e.g.,:
• If alterations to the machine vision part of the application are necessary, the code
needs to be exported again and the whole application then needs to be compiled
once more. This makes prototyping quite tedious and slow.
• If changes to the machine vision part of the application are being made, the whole
application also needs to be recertified, to for example keep a certain safety status.
14

With HDevEngine, it is also possible to implement HDevelop code into an application in
a smart way, because using HDevEngine allows you to directly load and execute
HDevelop programs and procedures from within your C++ or C# application.
This brings a lot of advantages compared to the „traditional“ approach, e.g.:
• Alterations to the vision part of your applications can be done „on the fly“ without
needing to recompile the whole application. This allows you to quickly create
different prototypes (rapid prototyping) and thus results in a shorter time to market.
• Modularizing different parts of your program (e.g., having a separate „vision module“)
complies with state-of-the-art software development standards, and allows to assign
different modules to dedicated machine vision, UX, or process integration experts.
15

You can see the HDevEngine code running on a Raspberry Pi – this does not differ from
the code you would be using on a standard pc; seeing the code which is being executed
live on the machine used to be quite difficult. But with HALCON Embedded, based on
HALCON 13, this is easy! Consequently, the maintenance of your embedded vision
application gets much easier as well.
Furthermore, all the advantages of HDevEngine apply to HALCON Embedded as well.
Changing between standard PCs and embedded devices is easily possible.
17

Please find the video at https://www.youtube.com/watch?v=bVc27OYAvh4.
The following is shown in the video:
With a shell using PuTTY, we are connected with a Raspberry Pi. First, we set some environment variables
with setup.env. Then, we start the application. The goal of the application is to find an arbitrary number of
SD cards in an image, which is grabbed with the camera module of the Raspberry Pi. First, there are no
matches, then 1, then 2. However, when we place more than 2 SD cards under the camera, the number
doesn‘t change…a bug!
Now, we open HDevelop, and connect with the process which is running on the Raspberry Pi. Here, we can
see the code which is running on the Raspberry Pi, stop the program and step through it, inspect the
variables, and even save variables for further debugging. After we have saved an image, we stop the
debugging – the application is running again.
Now, we open the same HDevelop program on our local computer. Here, we can again step through the
program and look for the bug in detail. First, some morphology is performed to reduce the region where
we look for the SD cards. Then, we notice that find_shape_model, which looks for the trained shape of the
logo on the SD cards, does return only two Score values – only two SD cards are found. One of the
parameters of find_shape_model is NumMatches – the maximum number of returned matches. It is 2,
which is probably our bug. We can either set it to three to find a maximum number of 3 matches, or 0, to
find all matches.
Lastly, using WinSCP, we copy this procedure on our Raspberry Pi, and restart the application – without
recompiling! Now, all three matches are found. Problem solved.
18

The following is shown in the video:
With a shell using PuTTY, we are connected with a Raspberry Pi. First, we set some environment
variables with setup.env. Then, we start the application. The goal of the application is to find an
arbitrary number of SD cards in an image, which is grabbed with the camera module of the
Raspberry Pi. First, there are no matches, then 1, then 2. However, when we place more than 2
SD cards under the camera, the number doesn‘t change…a bug!
Now, we open HDevelop, and connect with the process which is running on the Raspberry Pi.
Here, we can see the code which is running on the Raspberry Pi, stop the program and step
through it, inspect the variables, and even save variables for further debugging. After we have
saved an image, we stop the debugging – the application is running again.
Now, we open the same HDevelop program on our local computer. Here, we can again step
through the program and look for the bug in detail. First, some morphology is performed to
reduce the region where we look for the SD cards. Then, we notice that find_shape_model, which
looks for the trained shape of the logo on the SD cards, does return only two Score values – only
two SD cards are found. One of the parameters of find_shape_model is NumMatches – the
maximum number of returned matches. It is 2, which is probably our bug. We can either set it to
three to find a maximum number of 3 matches, or 0, to find all matches.
Lastly, using WinSCP, we copy this procedure on our Raspberry Pi, and restart the application –
without recompiling! Now, all three matches are found. Problem solved.
19

Focusing on industrial vision applications
MVTec focuses on industrial applications. Open source software often lacks features like measure tools,
blob analysis, or matching.
Comprehensive documentation and fast development
MVTec HALCON offers documentation for every user and level – ranging from "Quick Guides" up to
"Solution Guides“, which offer extensive step-by-step tutorials for solving various vision problems (for
example how to read 2D data codes, or how to perform 3D vision and image mosaicking). In the reference
documentation every single one of the more than 2000 HALCON operators is being referenced extensively.
In some cases the reference even includes mathematical backgrounds and scientific publications on which
the method is based upon. In addition, MVTec regularly releases tutorial videos for both HALCON and
HALCON Embedded.
HALCON saves money through continuous maintenance
Maintaining software is a big cost driver. Once the software has been delivered to the customer, he or she
starts working with it. From this life cycle stage on, maintenance will be necessary, too. As this stage of the
product life cycle also tends to be the longest, maintenance costs play a large role in the long term.
However, many customers are not aware of these „hidden“ costs.
MVTec‘s software products are continuously being adapted with respect to, e.g. new architectures like
AVX2, ARM 64 etc. MVTec also ensures compatibility of its software for various versions over a long period
of time.
You can rely on HALCON
Our customers' products require high quality and performance. So we have to provide the best vision
library possible to match their needs.
The software engineering behind HALCON meets the highest standards: in 2015, one error per 12,600 lines
of code was found. This corresponds to 0.08 errors per 1000 lines of code.
20

It is essential to point customers‘ attention towards the total costs of software. The
price of a single SDK or runtime license is merely relevant to the overall costs. Other cost
factors have a much bigger impact, like training costs (for new customers),
implementation costs (for new projects) or maintenance and opportunity costs (for
products that already have been delivered).
With HALCON, you can plan and calculate with all these costs from the beginning.
21

MVTec HALCON is the comprehensive standard software for machine vision with an integrated
development environment (HDevelop) that is used worldwide.
It enables cost savings and improved time to market. HALCON’s flexible architecture facilitates
rapid development of any kind of machine vision application.
MVTec HALCON provides outstanding performance and a comprehensive support of multi-core
platforms, special instruction sets like AVX2 and NEON, as well as GPU acceleration. It serves all
industries, with a library used in hundreds of thousands of installations in all areas of imaging
like blob analysis, morphology, matching, measuring, identification, and 3D vision.
The software secures your investment by supporting a wide range of operating systems and
providing interfaces to hundreds of industrial cameras and frame grabbers, in particular by
supporting standards like GenICam, GigE Vision, and USB3 Vision.
Machine vision software today has to fulfill many requirements:
The faster the software, the faster the inspection task can be performed, the higher the
throughput of the machine, which results in lower production costs.
The need of cost reduction leads to more and more sophisticated and complex machine vision
tasks. To tackle all upcoming challenges and to be one step before the competitor, it is
important that the machine vision software has tools for many different tasks which can be
combined easily to solve even complex tasks.
Last but not least, the results should of course be as accurate as possible and the software must
be approved to run even under difficult industrial environments without errors.
HALCON fulfills all of these requirements.
24

What is HALCON Embedded?
• HALCON Embedded means HALCON running on your special platform. HALCON is
portable to various microprocessors/DSPs, operating systems, and compilers.
• HALCON Embedded lets you exploit the power of a comprehensive machine vision
library on embedded systems.
• HALCON Embedded allows you to develop the software part of your machine vision
application on a standard platform and thereby greatly eases the programming of an
embedded system. Short said: Develop on a PC, and let the application run on an
embedded system.
25

MVTec is the only software manufacturer worldwide solely focusing on developing
software for machine vision.
MVTec employs highly qualified experts for machine vision with up to 30 years of
experience in this technology.
We love to solve vision problems!
MVTec is dedicated to machine vision software.
Therefore, the company's vision statement is built on this dedication:
• Maintain the technological market leadership for machine vision software
• Remain a successful manufacturer of world-wide standard-software products
for the machine vision industry
• Represent a unique competence center for image processing algorithms
26

MVTec is dedicated to machine vision software: This passion for machine vision
is the driving force of the entire MVTec staff, including the management. We
personally stand behind the high quality of our products and services.
27

This example shows how fast and easy application development can be using HDevelop.
Here, we grab live images of a camera and count the number of bottles in a crate with
just 10 lines of code.
In HDevelop, we write this script very simply: We grab live images from a camera,
smooth the image to reduce the impact of image noise, apply a threshold, and compute
the connected regions to get the separate bottles.

HDevelop's integrated Image Acquisition Assistant lets you easily detect, connect, and
configure all available cameras and framegrabbers with just a few mouse clicks.
Parameters can be adjusted using an intuitive graphical user interface and can be
verified directly in a live image.
Finally, the settings can be added to the script as automatically generated code
segments.

In addition to a comfortable full text editor that actively gives suggestions and supports
auto-completion of code, HDevelop comes with a wide range of helpful features.
HDevelop also offers the dialog-based operator and parameter selection, and a
structured menu hierarchy that enables you to find the best operator for your
application as fast as possible.

HDevelop offers a lot of intuitive debugging features. Values of used variables are always
accessible through the variables window, and the results of image processing operations
are instantly visualized in the graphics window. Furthermore, the control flow of the
program can be controlled easily via breakpoints and variable stepping options.

Once the image processing part of your program is ready, it can easily be exported to
the programming language of your choice using the file menu.

The exported code can easily be integrated into your project.

Finally, the application is ready and running.

"Embedded Vision Made Smart: Introduction to the HALCON Embedded Machine Vision Library," a Presentation from MVTec

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to "Embedded Vision Made Smart: Introduction to the HALCON Embedded Machine Vision Library," a Presentation from MVTec

Similar to "Embedded Vision Made Smart: Introduction to the HALCON Embedded Machine Vision Library," a Presentation from MVTec (20)

More from Edge AI and Vision Alliance

More from Edge AI and Vision Alliance (20)

Recently uploaded

Recently uploaded (20)

"Embedded Vision Made Smart: Introduction to the HALCON Embedded Machine Vision Library," a Presentation from MVTec