Content based image retrieval for agriculture crops

CONTENT-BASED IMAGE
RETRIEVAL FOR AGRICULTURAL
CROPS
Esraa M. Elhariri, MSc
Scientific Research Group n Egypt (SRGE)
www.egyptscience.net

Agenda
 Introduction
 Problem Definition
 Scope of work & Proposed Solutions
 Crops Ripeness
 The Proposed Ripeness Classification System
 Experimental Results
 Conclusions & Future Work

Introduction
 Product quality is one of the prime factors in
ensuring consistent marketing of crops.
Determining ripeness stages is a very
important issue in produce (fruits and
vegetables) industry for getting high quality.
 Optimal harvest dates and prediction of
storage life are still mainly based on
subjective interpretation and practical
experience.

Introduction Cont…
 Monitoring and controlling ripeness is a
very important issue in the fruit industry
since the state of ripeness during harvest,
storage and market distribution determines
the quality of the final product measured in
terms of customer satisfaction.

Problem Definition
 Crops at greenhouse, storage containers are affected by
changes at temperature and humidity levels.
 these changes can cause diseases, lost of nutritional value,
over-ripen and changes at texture, appearance and flavor and
this affect the quality of the agricultural crops.
 Monitor the ripeness process of crops is very important issue
at agricultural and industry.

Problem Definition Cont…
 So the human labors were required to visit the greenhouse and
check the humidity and temperature levels at specifics times
in according to plant type (e.g. strawberry need the
temperature level and humidity to be checked every 3 hours).
 Doing this manually will be time consuming, cost high and
needs a lot of work and efforts.

Scope of work & Proposed
Solutions
 This thesis concerns the study and development of different
approaches to help Labors via providing an automated
ripeness assessment system for agricultural crops.
 Developing an automatic content based image retrieval
system for agriculture crops, which is tackling the problem
of ripeness assessment.
 Having a comparative study among different classification
algorithms for the problem of ripeness classification.

Sweeter
Less
Green
Softer
More palatable
Crops Ripeness

Choosing a crop
 We choose Bell Pepper and Tomato for our research.
 Different Ripeness Stages are described at the following
Charts.

The proposed classification
system: System Architecture

• Resizing images to 250x250 pixels for the following
reason:
• Reducing their color index to avoid high
computation.
 During this phase, the proposed approach prepares images
for the features extraction phase, The first step is :
system: Pre-processing Phase

• Remove gray shadow pixels.
• Convert image from RGB to Gray
image.
• Subtract image from background
image and compute a binary
mask.
• Apply the binary mask to original
image.
Background Removal Algorithm
Original Images
Images after background Removal
 The second step is removing
image background to get region
of interest by using the following
procedure:

 The third step is : converting each image from
RGB to HSV color space and extract HSV
components for the following reasons:
 When applying classification using PCA and
Euclidean distance using both RGB and HSV color
features we got better results using HSV color.
 From Survey, HSV color space as it is widely used
in the field of color vision and close to the
categories of human color perception
where R, G and B are color
component of RGB color
space

 The pre-processing phase can
be summarized as the
following:
• Resize images to 250 X 250
pixels,
• Remove images background,
• Convert images from RGB color
space to HSV color space,
•Extract HSV components.

 Since tomato surface color is the
most important characteristic to
observe ripeness of tomato, this
system uses color features for
classifying ripeness stages.
 Two color descriptors which were
used here are color moments and
HSV colored histogram.
system: Feature Extraction Phase

Standard Equation:
 The first descriptor is color
moments: we used mean, standard
deviation, and skewness which have
been proved to be efficient and
effective way for representing color
distribution in any image.
 Mean, standard deviation and
skewness for a colored image of
size NxM pixels are defined by the
following equations:
Mean Equation:
Skewness Equation:

 Colored histogram is the second color descriptor that shows representation of the
distribution of colors in an image. It represents the number of pixels that have
colors in each range of colors.
 We used 1D 16x4x4 HSV colored histogram, with 16 level for hue and 4 level
for each of saturation and value.
 Then we compute a combination 1D Features vector of the three color moments
and HSV histogram.

 Principal component analysis is a
statistical common technique, which is
widely used in image recognition and
compression for a dimensionality
reduction, data representation and
features extraction tool as it ensures
better classification.
 we use Principal Component
Analysis(PCA) for features vector
Transformation to extract good features
for the classification purpose.
IM1 F1 F2 : : : : : : Fk
IM2 : : : : : : : : :
::: : : : : : : : : :
IMn F1 F2 Fk
IM1 F1
: : Fm
IM2 :
: : :
::: :
: : :
IMn F1
: : Fm
PCA Algorithm
Whole Features
Extracted Sub-set of Features

 During this phase, the proposed
approach uses Support Vector
Machine(SVM) as a classifier, this
phase is responsible for classify data
into 5 classes mentioned before.
system: Classification Phase

 The Support Vector Machine (SVM) is a
Machine Learning (ML) algorithm that is
used for classification and regression of
high dimensional datasets with great
results. SVM solves the classification
problem via trying to find an optimal
separating hyperplane between classes.
 SVM algorithm seeks to maximize the
margin around a hyperplane that separates
a positive class from a negative class.

 Since SVM is a binary classifier, we have
used two different approaches to use it as
multi-class classifier:
 One-against-All
 One-against-One
 Also we have tested our SVM classifier
using different kernel functions includes:
 Linear kernel function,
 Polynomial kernel function,
 MLP kernel function,
 RBF kernel function.

One against All:
1. Construct N binary SVM.
2. Each SVM separates one class
from the rest classes.
3. Train the ith SVM with all
training samples of the ith class
with positive labels, and
training samples of other classes
with negative labels.
system: Classification Phase (One-against-All
Approach)

 At the begin, we have used One-
against-All multi-class SVM with
different kernel functions and total
of 230 images used for the
classification problem, it was
divided into 175 images as a
training dataset and 55 images as
a testing dataset.
Experimental Results:
First Approach using One-against-All Approach

 Training dataset is divided into 5 classes representing the different
stages of tomato ripeness.
Training Dataset

 Classification accuracy
using different kernel
function and different
sizes of training
classes:
Results of First Approach

Results of First Approach

 From the previous figure we
found that classification
accuracy using SVM with
linear kernel function
increased by increasing
number of training images per
class.
Experimental Results: Accuracy of
linear Kernel Function using Different Training Size

Publications List
• Implementation of one-against-all multi-calss SVM for tomato
• Publication of Multiclass SVM Based Classification Approach for
Tomato Ripeness
Submission : Jul 2013
Publication :Aug 2013
• Implementation of one-against-all & one-against-one multi-calss
SVM
• Publication of Automated Ripeness Assessment System of Tomatoes
Using PCA and SVM Techniques(book chapter)
Submission : Nov 2013
Publication: June 2014
• Implementation of one-against-one multi-calss SVM for bell pepper
• Publication of Bell Pepper Ripeness Classification based on Support Vector
Machine
Submission : Dec 2013
Presentation: Apr 2014
• Implementation of different machine learning techniques for tomato as an
extension of the first paper and the book chapter.
• Publication of Using machine learning techniques for evaluating tomato
ripeness
Submission : Feb 2014
Publication: Oct 2014
• Comparative Study of different classification techniques for both crops.
• Publication of Random Forests Based Classification for Crops Ripeness Stages
Submission: Apr 2014
Publication : June 2014
Esraa Elhariri, Nashwa El-Bendary, Mohamed Mostafa M. Fouad, Jan Plato’s,
Aboul Ella Hassanien, Ahmed M. M. Hussein, “Multiclass SVM Based
Classification Approach for Tomato Ripeness.” In Proceedings of the Fourth
International Conference on Innovations in Bio-inspired Computing and
Applications(IBICA 2013), 22 – 24 August, Ostrava, Czech Republic, pp.175-186 ,
2013, Springer International Publishing.

 Then, we have used both of
One-against-All & One-against-
One with 10-fold cross-
validation to generalize model
accuracy, using total of 250
images for both of training and
testing.
 This figure shows how cross-
validation works.
TrainingTesting
Experimental Results: Second
Approach using both of OAO and OAA with Cross-
Validation

One-against-One:
1-train k(k − 1)/2 binary SVMs
(1, 2), (1, 3), . . . , (1, k), (2, 3), (2,
4), . . . , (k − 1, k),
2-Select the one with the largest
vote.
system: Classification Phase (One-against-One
Approach)

 Classification accuracy using different kernel function and total number of
250 image(used for both training and testing using 10 folds cross-
validation) using one against one approach for multi-class SVM
classification:
Experimental Results: One-against-
One(OAO) with 10 fold Cross-Validation Approach

 Classification accuracy using different kernel function and total number of
250 image(used for both training and testing using 10 folds cross-
validation) using one against all approach for multi-class SVM
classification:
All(OAA) with 10 fold Cross-Validation Approach

 The following figure show a comparison between the classification
accuracy of One-against-All(OAA) multi-class SVM and One-against-
One(OAO) multi-class SVM Approaches:

Experimental Results: Comparison
between of OAO & OAA Approaches

Publications List
Tomato Ripeness
SVM
Machine
ripeness
Esraa Elhariri, Nashwa El-Bendary, Aboul Ella Hassanien, Amr Badr:
"Automated Ripeness Assessment System of Tomatoes Using PCA and SVM
Techniques." Computer Vision and Image Processing in Intelligent Systems and
Multimedia Technologies. IGI Global, 2014. 101-130. Web. 9 Oct. 2014.
doi:10.4018/978-1-4666-6030-4.ch006.

Bell Pepper Ripeness
Assessment
 Then we have worked on bell pepper:

 we have used One-against-One
multi-class SVM with different
kernel functions and total of 175
images used for the classification
problem, they were used for both
training and testing dataset using
10 fold Cross-validation.
One-against-One Approach

 Training dataset is divided into 5 classes
representing the different stages of Bell Pepper
ripeness.
Training Dataset

One (OAO) with 10 Cross-validation Approach
 Classification accuracy using
different kernel function and
10-folds cross-validation.

Publications List
Tomato Ripeness
SVM
Machine
ripeness
Esraa Elhariri, Nashwa El-Bendary, Ahmed M. M. Hussein, Aboul Ella
Hassanien, and Amr Badr, “Bell Pepper Ripeness Classification based on Support
Vector Machine”, In Proceeding of IEEE ICET, 19 – 21 April, Cairo, Egypt.

system: System Architecture (Comparison
between SVM and LDA)

 Then, we have used both of
One-against-All & One-against-
One, In addition to, LDA with
10-fold cross-validation to
generalize model accuracy,
using total of 250 images for
both of training and testing.
 This figure shows how cross-
validation works.
TrainingTesting
Experimental Results: Second Approach
using both of OAO, OAA and LDA with Cross-Validation

Linear Discriminant Analysis
(LDA)
 LDA is a commonly used
technique for data classification
and dimensionality reduction.
 It’s basic idea is to find a linear
transformation that best
discriminate among classes, then
classification can be performed in
transformed space based on some
metrics(Euclidean distance)

 Classification accuracy using different kernel function and total number of 250
image(used for both training and testing using 10 folds cross-validation) using one
against one approach for multi-class SVM classification:
One (OAO) with 10 Cross-validation Approach

 Classification accuracy using different kernel function and total number of 250
image(used for both training and testing using 10 folds cross-validation) using one
against all approach for multi-class SVM classification:
Experimental Results: One-against-All
(OAA) with 10 Cross-validation Approach

 Classification accuracy using total number of 250 image(used for both training
and testing using 10 folds cross-validation) using LDA classification:
Accuracy= 84 %
Experimental Results LDA with 10
Cross-validation Approach

 The following figure show a comparison between the
classification accuracy of One-against-All(OAA) multi-class
SVM and One-against-One(OAO) multi-class SVM and LDA
Approaches:

Experimental Results: Comparison
between of OAO & OAA and LDA Approaches

Publications List
Tomato Ripeness
SVM
• Publication of Bell Pepper Ripeness Classification based on Support
Vector Machine
• Implementation of different machine learning techniques for
tomato as an extension of the first paper and the book chapter.
• Publication of Using machine learning techniques for evaluating
tomato ripeness
• Publication of Random Forests Based Classification for Crops Ripeness
Stages
Nashwa El-Bendary, Esraa El Hariri, Aboul Ella Hassanien, Amr Badr, “Using
machine learning techniques for evaluating tomato ripeness”, In International
Journal of Expert Systems with Applications, Volume 42, Issue 4, March 2015,
Pages 1892-1905, Elsevier, ISSN 0957-4174,
http://dx.doi.org/10.1016/j.eswa.2014.09.057.

system: System Architecture
Random
ForestsOAO-SVM

One-against-One Approach
 we have used One-against-One multi-
class SVM and Random Forests with
different kernel functions and total of 175
and 250 images for bell pepper and
tomato, respectively used for the
classification problem, they were used
for both training and testing dataset.

 The Random Forests (RF) is one of the best
known classification and regression
techniques, which has the ability to classify
large dataset with excellent accuracy.
 Random Forests algorithm generates an
ensemble of decision trees. Ensemble
methods main principle is to group weak
learners together to build a strong learner.

 The input is entered at the top and as it traverses
down the tree, the original data is sampled in
random, but with replacement into smaller and
smaller sets.
 The class of sample is determined using random
forests trees, which are of an arbitrary number.

Training Dataset
 For Bell Pepper, Training dataset is divided into 5 classes
representing the different stages of Bell Pepper ripeness.

Training Dataset
 For Tomato, Training dataset is divided into 5 classes
representing the different stages of Tomato ripeness.

Experimental Results: One-against-One
(OAO) & RF with 10 Cross-validation Approaches
 Bell Pepper Ripeness
Classification accuracy using
Random Forest and SVM
using different kernel function
and 10-folds cross-validation.

Experimental Results: One-against-One
(OAO) & RF with 10 Cross-validation Approaches
 Tomato Ripeness Classification
accuracy using Random Forest
and SVM using different kernel
function and 10-folds cross-
validation.

Publications List
Tomato Ripeness
SVM
Machine
ripeness
Esraa Elhariri, Nashwa El-Bendary, Aboul Ella Hassanien, Amr Badr, Ahmed M.
M. Hussein, Václav Snásel, “Random Forests Based Classification for Crops
Ripeness Stages”, In Proceedings of the Fifth International Conference on
Innovations in Bio-Inspired Computing and Applications (IBICA 2014), vol. 303,
23 – 25 June, Ostrava, Czech Republic, pp.205-215, 2014, Springer International
Publishing.

Conclusions & Future Work
 In this Thesis, a system for classifying the ripeness stages of
both tomato and bell pepper has been developed.
 For tomato ripeness assessment, the ripeness classification
accuracy obtained by the OAO multi-class SVM approach is
better than ripeness classification accuracy obtained by the
OAA multi-class SVM and LDA approaches.
 For bell pepper ripeness assessment, the ripeness classification
accuracy obtained by the OAO multi-class SVM approach is
better than ripeness classification accuracy obtained by the
OAA multi-class SVM approach.

Conclusions & Future Work
 A number of future research could be achieved via classifying
different objects or crops by involving other features (texture,
shape, size, ... etc.) according to the classified objects nature.
 Other Machine Learning approaches could be employed in
order to address the advantages and limitations of applying
each of them.
 Another direction of research is to use nondestructive/non-
invasive detection technologies of food quality/maturity such
as hyperspectral imaging systems, colorimetric, Near Infrared
Spectroscopy, and non-invasive smart sensing technologies

For further questions:
Esraa M. El-hariri
Eng.esraa.elhariri@gmail.com

Content based image retrieval for agriculture crops

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (11)

Similar to Content based image retrieval for agriculture crops

Similar to Content based image retrieval for agriculture crops (20)

More from Aboul Ella Hassanien

More from Aboul Ella Hassanien (20)

Recently uploaded

Recently uploaded (20)

Content based image retrieval for agriculture crops