2013-06-26 laboratory seminar

1. Laboratory Seminar
A Multi-Sensor based Uncut Crop Edge Detection Method
for Head-Feeding Combine Harvester
2013-06-26
Wonjae Cho*
Lab. of Field Robotics
Div. of Environmental Science and Technology
Graduate School of Agriculture
Kyoto University
Wonjae CHO, Michihisa IIDA, Masahiko SUGURI,
Ryohei MASUDA, Hiroki KURITA

2. 0Kyoto University
Index
1. Introduction
2. Materials and Methods
3. Results and Discussion
4. Summary and Conclusions

3. 0Kyoto University
Introduction
Ground plane
rd
cd
Uncut crop edge
RTK-GPS
LRF
GPS compass
Body information
(Travel speed, Header position, Etc.)

4. No. Sensor Figure
1 Laser Range Finder
: SICK Inc., LMS111
2 GPS Antenna
: Topcon LEGACY+
3 GPS Compass
: Hemisphere V110
0Kyoto University
Materials and Methods
1) Experimental setup – Navigation sensors
1
2
3
The head-feeding combine harvester
(VY50CLAM, Mitsubishi Agricultural Machinery Co., Ltd.)
Technical Specification of the sensors

5. 0Kyoto University
Materials and Methods
1) Experimental setup – Control system

6. 0Kyoto University
Materials and Methods
1) Experimental setup – Integrated sensor control platform (ISCP)
Sensor Interface
(Machine Vision, LRF)
ECU-KUx Interface
(Combine Body, RTK-GPS, GPS compass)
Driver
Layer
Platform
Layer
Steering
(Speed, Heading)
Sensor
Fusion
Image
Processing
Combine Body Control
(Header, Unloading Auger, Etc.)
Algorithm
Layer
Path
Planning
Uncut Crop Edge Detection 3D Mapping
Task Definition
(Harvest, Turn)
User
Interface
Layer
Sensors Display
(Machine Vision, LRF, RTK-GPS)
Combine Body
Display
3D Map
Display
Remote
Monitoring
The ISCP can be accessed via the following address
https://github.com/FiroKyoto/IntegratedSensorControlPlatform.git

7. 0Kyoto University
Materials and Methods
2) 3D terrain mapping – The extrinsic parameters of the navigation sensors
Parameter Value
Tilt angle ( , °) 50.5
The height from ground ( ,m) 2.5
‘Longitudinal’ distance of
LRF from the origin of xy Plane ( ,m)
0.45
‘Transverse’ distance of
LRF from the origin of xy Plane ( ,m)
0.59
‘Longitudinal’ distance of
the divider from the LRF ( ,m)
0.6
‘Transverse’ distance of
the divider from the LRF ( ,m)
0.07

h
al
bl
cl
dl

8. 0Kyoto University
Materials and Methods
2) 3D terrain mapping – Sensors data fusion
a. The polar coordinates system data ( , ) from LRF need to be transformed
into Cartesian coordinates system data (x, y, z).
b. The absolute position obtained from RTK-GPS should be transformed
into a position on the Transverse Mercator (TM) coordinates system.
c. Each of the transformed data of the navigation sensors should be applied
to equation and expressed as 3D points on space W.
cos
sin sin
sin cos
i i i
LMS
i i i i i
i i i i
x
p y
z h
 
  
  
   
       
      
_ _
_ _
_
cos sin 0
sin cos 0
0 0 0 1
MAP TM Heading Laser
i i i
TM LMS
x i c x i
TM LMS
y i a y i
LMS
z i
P P R P
P l p
P l p
p
 
 
 
    
           
        


9. 0Kyoto University
Materials and Methods
2) 3D terrain mapping – The discrimination of the 3D points
LRF data distribution expressed on xz plane.
     
   
2 2
n h h k h k n h
k
n h n h
x x y y x x y y
d
x x y y
    

  
(Kimberling, 1998)
Parameter Signification
z of LRF data set becomes
maximum.
End points of LRF data set.
Perpendicular to the data
set.
Outermost boundary point
of the uncut crop area.
hp
np
kp
kd

10. 0Kyoto University
Materials and Methods
2) 3D terrain mapping – Expressed 3D map using OpenGL

11. 0Kyoto University
Materials and Methods
3) Uncut crop edge detection – Using RANSAC algorithm
RANdom SAmple Consensus (RANSAC) algorithm is a method that predicts the model parameters
from the original data that has severe measurement noise (Fischler and Bolles, 1981).
Pseudo code
Given:
data – a set of observed data points
model – a model that can be fitted to data points
m – the minimum number of data values required to fit the model
N – the maximum number of iterations allowed in the algorithm
t – a threshold value for determining when a data point fits a model
d – the number of close data values required to assert that model fits well to data
Return:
bestfit – model parameters which best fit the data (or nil if no good model is found)
iterations = 0
bestfit = nil
besterr = something really large

12. 0Kyoto University
Materials and Methods
3) Uncut crop edge detection – Using RANSAC algorithm
while iteration < N {
% Hypothesis Step
maybeinliers = m randomly selected values from data
maybemodel = model parameters fitted to maybeinliers
alsoinliers = empty set
for every point in data not in maybeinliers {
if point fits maybemodel with an error smaller than t
add point to alsoinliers
}
if the number of elements in alsoinliers is > d {
% this implies that we may have found a good model
% now test how good it is
bettermodel = model parameters fitted to all points in maybeinliers and
alsoinliers
thiserr = a measure of how well model fits these points
if thiserr < besterr {
bestfit = bettermodel
besterr = thiserr
}
}
increment iterations
}
return bestfit

13. 0Kyoto University
Materials and Methods
3) Uncut crop edge detection – Using RANSAC algorithm
a. data – Calculated set of the outermost boundary points ( ) of the uncut crop area,
configured in a dynamic array.
b. m – The number of sample data as 5.
c. t – The threshold value is set as 0.1.
d. N – The number of repeats of RANSAC algorithm can be calculated from the equation.
[Where]
The probability ( ), where at least one sample set includes valid data, as 0.99.
Probability ( ) of the data validity as 0.5.
kp
log(1 )
log(1 )m
p
N




p


14. 0Kyoto University
Materials and Methods
3) Uncut crop edge detection – Expressed uncut crop edge on W space

15. 0Kyoto University
Results and Discussion
1) Results – Experimental conditions
Traveling path of the head-feeding combine harvester
on Google Maps.
Date 2012-09-20
Location Nantan city, Kyoto Prefecture, Japan
Field Rice paddy field
Harvester
Model
VY50CLAM
Harvester
Control
By an operator (human)
Traveling
Speed [m/s]
0.6
Movement
Direction
Counter-clockwise
Data set Four dataset (A, B, C, D)

16. 0Kyoto University
Results and Discussion
1) Results – The result movie Clip
The result movie clip can be accessed via the following address
http://youtu.be/juyYuQfvgkk

17. 0Kyoto University
Results and Discussion
1) Results – Uncut crop edge extracted by using RANSAC
Section (A) Section (B)
Section (C) Section (d)

18. 0Kyoto University
Results and Discussion
1) Results – By the proposed method
3D terrain map of the experiment field.
Dataset Movement
Direction
Average of
Crop Height
[m]
Lateral
Offset
[m]
A South 0.514 0.294
B East 0.503 0.139
C North 0.549 0.067
D West 0.580 0.119
Average 0.537 0.154
Average processing speed of 35 ms

19. 0Kyoto University
Results and Discussion
2) Discussion

20. 0Kyoto University
Summary and Conclusions
a. The proposed method was able to generate 3D maps of the terrain to be
harvested at a processing speed of 35 ms by using navigation sensors.
b. The averages of lateral offset value of uncut crop edge and crop height,
extracted by the proposed method, were 0.154 m and 0.537 m, respectively.
c. While the proposed method of the study was robust in extracting the uncut
crop edge in general.
d. Nevertheless, it’s performance displayed a tendency to decline when the target
path was obscured by the lodging of rice plants.
e. Hence, in order to enhance the performance of the proposed method, the
mounting position of LRF needs to be modified to allow the accurate scanning
of the target path, together with the addition of an algorithm that can adjust
for the error in results.

21. Remote Monitoring

22. 0Kyoto University
Results and Discussion
1) Results – The result movie Clip
The result movie clip can be accessed via the following address
http://youtu.be/eL6cX5_9o-0

23. 小麦の収穫実験
2013-06-10 (月)

24. 0Kyoto University
Introduction
Header height of head feeding combine
In case header height is lower than ground
In case header height is too high

25. 0Kyoto University
Materials and Methods
1) Header control – Range designation
Travel Speed [m/s] = V
Distance [m] = S = Vt
Elapsed time [sec] = t
Start position Destination position

26. 0Kyoto University
Materials and Methods
1) Header control – Relationship between header height and potentiometer
y = 596.2x + 365.7
R² = 0.977
0
100
200
300
400
500
600
700
800
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7
Potentiometer(ADvalue)
Hedaer Height (m)
p1
pNpk
d
l1
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
-3 -2 -1 0 1 2 3 4
Height(m)
Lateral direction (m)
Ground Scope (1m)

27. 0Kyoto University
Results and Discussion
1) Results – Experimental conditions
Traveling path of the head-feeding combine harvester
on Google Maps.
Date 2013-06-10
Location 奈良県大桜井市西地区
Field Wheat field
Harvester
Model
VY446
Harvester
Control
By an Autonomous mode
Traveling
Speed [m/s]
0.5
Movement
Direction
Counter-clockwise
Data set One dataset

28. 0Kyoto University
Results and Discussion
1) Results – The result movie Clip
The result movie clip can be accessed via the following address

29. 0Kyoto University
Results and Discussion
2) Discussion
Rank weeds
Low density

30. 0Kyoto University
Future work
a. Auto-Steering algorithm
b. Turn algorithm
c. Advanced header control algorithm
d. Autonomous guidance system for head-feeding combine harvester
e. Advanced Remote Monitoring Application
: Participation of Telerobotics Summer School 2013 (Support by IEEE)
 July 8 - 12 2013 in Yokohama, Hiyoshi Campus, Keio University, Japan
 https://sites.google.com/site/teleroboticssummerschool2013/
f. Migration Integrated Sensor Control Platform (ISCP)
: Visual C# to C++ (native C++ or visual C++)

31. 0Kyoto University
Contact us
Thank you for listening to my presentation.
If you have any questions, please electronic mail me!
 cho@elam.kais.kyoto-u.ac.jp