1. Agricultural and Biosystems Engineering
Plant Recognition through the Fusion of
2D and 3D Images for Robotic Weeding
Jingyao Gai (M.S)
Agricultural and Biosystems Engineering
Iowa State University, IA, USA
July 27nd, 2015
2. Agricultural and Biosystems Engineering
Weeds: Important issues in farming
A weed is a plant considered undesirable in a
particular situation.
• Compete with crops for
nutrients, water, and light
• Reduce yields
• Harbor pest insects and
diseases
Weed numbers or size
2
3. Agricultural and Biosystems Engineering
How to prevent: Weed controlling
Mechanical weeding:
• Non-chemical and
ecologically sound
• Environmental friendly
Challenge:
Needs more labor than
chemical weed control
Difficult to remove weeds
within crop rows
Robotic weed control
3
4. Agricultural and Biosystems Engineering
How to recognize: Sensing
Microsoft Kinect V2:
Features Value
Depth sensor Time of Flight (IR)
Depth spatial
resolution
512 x 424
Color spatial
resolution
1920 x 1024
Angle of view 70 deg x 60 deg
Frame rate 30fps
Pixel density at
750mm
2.05mm/pixel x
2.04mm/pixel
Price $199
Kinect V2 sensor
Point cloud fused with color
Competent to work outdoor, if under
shaded sunlight or overcast.
4
5. Agricultural and Biosystems Engineering
How to recognize: Image processing
Depth
image
Color
image
Filtered
depth
image Point
cloud of
plants
Separated
clusters
for plants
Recognition
(Weed or not)
Size
Height
Leaf shape
Position
…
Location
Outline:
Preprocessing
Segmentation Clustering
Feature
extraction
Classification
Localization
5
6. Agricultural and Biosystems Engineering
Preprocessing:
Neighbor count filter:
Neighbor searching strategy Corn point cloud example before and after noise removal
Reliable area:
Potential neighbors
Testing point
Example depth image with noise at corners
Affected by ambient
sunlight
Reliable
area
Reliable area mask
6
7. Agricultural and Biosystems Engineering
Segmentation:
7
Color based plant detection:
• Non-uniform light condition outdoor
• Illuminant-invariant space method (GD
Finlayson et al. 2002)
Image with changing light condition
Illuminant-invariant image
Lambertian surface
Planckian light Illuminant-invariant space and projection
8. Agricultural and Biosystems Engineering
Segmentation:
Depth based ground detection:
• Random Sample consensus(RANSAC):
• Find the ground (plane) in point cloud
Principle for ground detection using RANSAC
Example of corn point cloud1. For j = 1 to N do
2. Randomly select 3 points and fit
plane model Mj
3. Evaluate the model Mj with a score
Cj based on outliers
4. End for
5. Select Mopt to maximize the score Cj:
Arg max(Mj) Cj
𝑎 𝑏 𝑐 𝑑 𝑥 𝑦 𝑧 1 𝑇 = 0
9. Agricultural and Biosystems Engineering
Feature extraction:
• Plant size
• Plant height
• Leaf shape
• Plant shape
• Plant position
Classification
Clustering & feature extraction:
Clustering:
Non-parametric clustering：
Region growing /
Superparamagnetic clustering
&
Refinement clustering:
9
Clustering result for corn example
Localization
12. Agricultural and Biosystems Engineering
Future work:
12
1. Optimize the code to speed up recognition and
localization.
2. Use algorithms more robust to deal with the
unexpected conditions in the field.
3. More features will be extracted and more species
will be applied to.
Hello, every one, my name is…, from…. Today my topic of the presentation is….
Weeds are bad guys in farming. The weeds are hateful because…..
As a result, we need to kill the weeds to prevent these.
There are three main methods to realize weed controlling: mechanical, biological and chemical.
Mechanical weeding is what we like best because:…. But:……
So to free the labors from those tedious work, we want the weeder to find and remove the weeds by itself.
That is robotic weed controlling.
The robotic weeder must have the ability to see and recognize the plants, tell which is weed and which is not, then kill the weed.
The eye we select for the robot is Kinect v2 camera. It is a sensor that can realize fusing both 2D color information and 3D spatial information.
Why we want to use this sensor for both 2D and 3D images?
2D color images have higher resolution, more detail
3D image have the spatial information of the plants
If fusing them together, we can take both of their advantages.
Some of the features are in this table， we can see it has a higher depth spatial resolution than most of the 3D cameras. The pixels are dense enough, and the price is cheap.
The Kinect v2 use Time Of Flight method to measure the distance. It emits near inferred light and measure the time span when receive the light. This light is also included in the sunlight. So if work outdoor, sunlight must be shaded or overcast.
After acquiring the data, the next step is to process them. Here is the outline of the algorithm:
Depth & Color, Depth has noise. Preprocessing to remove the noise.
For our TOF sensors working outdoor, there is a reliable area, which means out side this area, their will less noise outside this area. This is because of the sensor structure, as well as the ambient sunlight outdoor.
So remove the pixels outside the area
Neighbor count filter is a fast way to remove sparse noise in the point cloud. It searches the surrounding pixels to find the neighbors. If pixels have low neighbor count, it is spare noise.
In color based segmentation, the strategy is to find the green pixels. However, it is hard if directly using RGB channels, because of the light condition problem.
Finlayson developed the way using illuminant invariant space method to remove the shadows in 2002. It transfer the RGB space into ill-inv space, in which Y axis is…, X axis is… In this space, the same color in different light conditions forms a line, and all lines of different colors are approximately parallel. So if project them onto the invariant axis, all colors are separated.
Segmentation is to find the points belonging to plants in the point cloud. This is done with both depth and color information.
Strategy is to find the ground in the point cloud, which is assumed to be a plane. The algorithm used is RANSAC
Iterative method to estimate the parameters of a model, with in data set containing outliers.
After finding the points belonging to plants, then we need separate them into different clusters.
Superaramagnetic clustering is an optimization method by animating the ferromagnet movement.
Some times the one plant will be separated into different clusters, then use refinement clustering to fix it.
The next is to extract the features from the plants. Different species have different way to extract them.
Currently, the experiment is performed to 2 species.
