Identification and Recognition of Snack Foods from Cluttered Scene for Dietary Assessment

Abstract—In this paper, we propose an easy approach of
identification and classification of high calorie snacks for dietary
assessment using machine learning. As an object detection
technique we have use point features matching algorithm to
identify the object of interest from a cluttered scene. After
detecting the object, a Bag of Features (BoF) model is created by
extracting Speed up Robust features (SURF) features. This BoF
model is used to recognize and classify the snacks items of different
categories. We have used three types of snacks images named: Ice-
cream, Chips and Chocolate for experimental purpose. Depending
on the experimental results our proposed algorithm is able to
detect and classify different types of snacks with around 85%
accuracy.
Keywords—Point feature matching, BoF, SURF, Food
identification.
I. INTRODUCTION
BESITY is becoming a great health issue day by day.
Researchers said that junk foods and processed foods are
responsible for increasing the childhood obesity, heart
disease, diabetes and other chronic diseases. Brain-derived
neurotropic factor (BDNF) helps in the learning and memory
formation of a human brain can be suppressed due to some
foods containing high sugar and fat. Eating extra calories can
harm the healthy production and functioning of the synapses of
our brain. Ice-cream, chips and chocolates are very common
and favorite snacks for both child and adults. People often buy
these low cost high calorie foods to control their appetite
especially when they are busy and unable to take their meal in
time. A news was published in "The New England Journal of
Medicine" on 2011 reveals that the daily consumption of a
single ounce of potato chips led to an average weight gain of
1.69 pounds over four years. On the other hand, a half cup of
vanilla ice cream contains around 25mg cholesterol. Excess
consumption of high cholesterol can increase our blood-
cholesterol levels and increase the risk of heart disease.
Chocolate is another source of high fat and sugar which is
responsible for acne, obesity, high blood pressure, coronary
artery disease, and diabetes. To live a healthy life, it is important
to provide more attention in this regards and the great thing is
that the public’s view towards taking junk food has undergone
severe changes day by day. Today’s people are more conscious
about their health issues and try to maintain a healthy diet.
So an easy but effective calorie measurement techniques can
help them to identify the amount of junk food and snacks they
can intake as well as to decide whether the food is harmful or
not good for their health. Through our research we try to
introduce a new technique for identifying high calorie snacks:
Chocolate, Ice cream and chips from our menu and estimate
their nutrition value.
II. LITERATURE REVIEW
Many researches have taken place to identify the food and
measure the calorie of a food. In our paper we propose methods
to recognize some popular snacks such as, chocolate, ice cream
and chips. Obesity is conceding a great problem in today’s life.
The preeminent reason of obesity is consuming more calories
than we burn which can seriously undermine the quality of life.
Researchers says, accurately assessing dietary intake is an
important factor to reduce this risk. To meet this exigency,
Chang et al. (2016) [1] proposed a computer-aided technical
method to measure the amount of calorie intake using
Convolutional Neural Network (CNN)-based food image
recognition algorithm. Probst et al. (2015) [2] is motivated to
introduce another prototype for dietary assessment with the
help of smart phone as well as the features of image processing
and pattern recognition. Scale invariant feature transformation
(SIFT), local binary patterns (LBP), color etc. common visual
features are used for espying food images. The bag-of-words
(BoW) model is used to perceive the images taken by the phone.
Chen et al. (2012) [3] also focused on this major issue and
proposed a method of food identification and quantity
estimation for dietary assessment. They use Gabor and color
features to represent food items. A multi-label SVM classifier
combined with multi-class Adaboost algorithm is used to show
that the new technique can successfully improve the
performance of original SIFT and LBP feature descriptors.
To increase the classification accuracy rate Baxter [4]
considered each pixel as a certain ingredient to analyze and
classify a food item at the pixel-level, and then using statistics
and spatial relationships between those pixels that make up the
food as features in an SVM classifier.
Sun et al. (2003) [5] declared a method for investigating
the quality of pizza base and tomato sauce spread using
computer vision technology. A fuzzy logic system was applied
for classifying the sauce spread on the samples.
Recognition and Classification of Snack Foods
from Cluttered Scene for Dietary Assessment
Amatul Bushra Akhi 1
, Farzana Akter 2
and Tania Khatun 3
Department of Computer Science and Engineering
Daffodil International University, Dhaka
Emai: akhi.cse@diu.edu.bd, farzana.cse@diu.edu.bd and tania.cse@diu.edu.bd
O
International Journal of Computer Science and Information Security (IJCSIS),
Vol. 16, No. 5, May 2018
195 https://sites.google.com/site/ijcsis/
ISSN 1947-5500

Hafiz et al. (2016) [6] introduced a method to detect and
recognize different types of drinks using vision based
algorithms. Thresholding technique is used to segment out and
Bag of Feature is used to recognize the drinks.
Kalaivani et al. (2013) [7] proposed a hierarchical grading
method applied to the tomatoes to identify of good and bad
tomatoes using MATLAB. Thresholding, segmentation and k-
means clustering is used to extract feature.
Patel et al. (2011) [8] introduced a multiple features based
algorithm to detect the fruit, for efficient feature extraction. The
proposed technique can be used for targeting fruits for robotic
fruit harvesting.
Recognition of fruit depends on four basic features:
intensity, color, shape and texture. In [9], an efficient fusion of
color and texture feature is proposed for fruit recognition.
Wavelet transformed sub- bands is used to derive some
statistical and co-occurrence features for recognition which is
done by the minimum distance classifier.
In [10], a vision based techniques have been applied to
identify fruits and vegetables diseases. In their research,
Rozario et al. (2016) propose a computer vision-based approach
for segmentation to identify defected regions from various
fruits and vegetables. Savakar et al. (2012) [11] use artificial
neural networks for identification and classification of different
types of bulk fruit images. Back Propagation Neural Network
(BPNN) Algorithm is used to classify and recognize the fruit
image samples, using different types of feature set like color,
texture, combination of both color and texture features.
III. METHODOLOGY
At the very beginning of our experimental method it is very
important to correctly identify the object of interest because our
experimental images contain a cluttered scene with different
objects. Several preprocessing are required to make the images
ready for work properly. Fig 1 shows the complete
methodology of our proposed system.
Fig. 1. Block diagram of the proposed methodology
A. Preprocessing
A raw image contains of certain factors such as noise,
climatic conditions, poor resolution and unwanted background
for which it is not suitable enough to classification and
identification. So it is important to improve image quality and
prepare the image for further processing to detect the object as
accurately as possible. In this paper the pre-processing process
consists of noise reduction and contrast enhancement.
1) Noise reduction using adaptive fuzzy switching median filter:
The contamination of digital image by salt-and-pepper noise
Vol. 16, No. 5, May 2018
ISSN 1947-5500

is largely caused by error in image acquisition. Thus, noise
reduction is essential for the accuracy of further processing. In
salt-and-pepper noise a certain percentage of individual pixels
in digital image are randomly digitized into two extreme
intensities. To remove this kind of noise effectively we use two
stage noise adaptive fuzzy switching median filter in which salt-
and-pepper noise intensities will be identified before
identifying the locations of possible noise pixels. Detected
“noise pixels” will then be subjected to the second stage of the
filtering action, while “noise-free pixels” are retained and left
unprocessed.
2) Contrast enhancement:
Image contrast is an important factor which is used to evaluate
image quality in addition to distinguish one object from another
as well as background. In image processing, contrast
enhancement is used to improve the appearance of an image for
human visual analysis or subsequent machine analysis. It is
created by the difference in luminance reflected from two
adjacent surfaces as well as the difference in the color and
brightness of the object. In this paper, to contrast the test image
we use histogram equalization technique.
B. Detection of Object of Interest
To identify a specific object in a cluttered scene, it is very
important to correctly detect the object of interest in the image.
Here we have used the most recent and efficient point feature
matching algorithm to detect the object we intend to recognize.
Here we have used point feature matching algorithm for
detecting a specific object based on finding point
correspondences between the reference and the target image.
This method of object detection works best for objects that
exhibit non-repeating texture patterns, which give rise to unique
feature matches. This technique is not likely to work well for
uniformly-colored objects, or for objects containing repeating
patterns
First of all, this algorithm will read the reference image
containing the object of interest and a target image containing a
cluttered scene and perform feature detection process on both
of these images to extract features, which are unique to the
objects in the image, so that an object can be detected based on
its features in different images. In this algorithm, speeded up
robust features (SURF) is used as a patented local feature
detector and descriptor. It is partly inspired by the scale-
invariant feature transform (SIFT) descriptor but much robust
and faster than SIFT. To find points of interest, SURF uses a
Hessian matrix based blob detector. The determinant of the
Hessian matrix used to evaluate the local change around the
points. The points are choses as feature points where this
determinant is maximum. Given a point X= (x, y) in an image
I, the Hessian matrix H(X, σ) at point X and scale σ, is:
H(X, σ) =
( , ) ( , )
( , ) ( , )
-------------------- (1)
Where,
Lxx(X, σ) = I(X) *
2
2
(σ) ----------------------------- (2)
Lxy(X, σ) = I(X) *
2
(σ) ---------------------------- (3)
= current filter size × (

)
Where Lxx(X, σ) in equation (2) is the convolution of the
image, I(x, y) at the point x with second derivative of the
Gaussian. Non-maximal-suppression of the determinants of the
hessian matrices is the main part of the SURF detection process.
The main objective of the descriptor is to provide a unique and
robust description of features in an image like describing the
intensity distribution of the pixels within the surroundings of
the point of interest.
After getting the feature descriptors this algorithm finds
putative point matches between target image and reference
image by comparing the descriptors obtained from both of these
images and locate the object with a bounding box in the scene
based on the matched putative points. Thus entire object
detection process has been completed.
C. Recognition and Classification
This section describes the way to recognize the test image
obtained from the cluttered scene and classify the image
category by using well established computer vision approach
called bag of features (BoF). Object recognition using bag of
features is one of the most successful object classification
techniques and our target is to classify a given image into one
of the pre-determined training objects. The process generates a
histogram of visual word occurrences that represent an image
which is then used to train an image category classifier.
We have chosen speeded up robust features (SURF)
detector to extract features because it is speedup version of
SIFT, which uses an approximated DoG and the integral image
trick to provide greater scale invariance. If an image is
computed into an integral image, using just 6 calculations block
subtraction between any 2 blocks can be computed. With this
advantages, finding SURF features could be several order faster
than the traditional SIFT features.
To train our classifier we have selected a random subset of
images from the dataset. To create visual words of detected
object k-means clustering algorithm is used on the feature
descriptors extracted from training sets. The algorithm
iteratively groups the descriptors into k mutually exclusive
clusters. The resulting clusters are compact and separated by
similar characteristics. Each cluster center represents a feature,
or visual word.
The SURF algorithm consists of both feature detection and
representation aspects. First, we need to find out the point of
interest which we can use for further processing and this step is
Vol. 16, No. 5, May 2018
ISSN 1947-5500

called feature detection. Feature detection selects the regions of
an image that have unique content, such as corners or blobs.
SURF uses a Binary Large Object (BLOB) detector which is
based on the Hessian matrix to find the points of interest. Blob
is a group of connected pixels in an image that share some
common property (e.g.: grayscale value). Feature extraction is
done around detected features by transforming a local pixel
neighborhood into a compact vector representation. This new
representation permits comparison between neighborhoods
regardless of changes in scale or orientation.
In order to represent an object, system need to know which
part belongs to which. After extracting all the features from all
our training images, K-Means clustering is used to group all the
features together and extracts the representative centroids. After
finding the SURF features of an object we need to find the
nearest parts matching these features. So every feature is
categorized into one of the part based on the distance from
SURF feature to the cluster centers. A histogram is formed for
all cluster centers, which represents the frequency of parts for
each image. This histogram basically counts the number of
times each features has appeared in the given image. We use
this to categorize every new image that comes in by computing
the Euclidean distance.
IV. EXPERIMENTAL DATA AND RESULTS
Our data set is contrived by three different types of snacks
such as chips, chocolate and ice cream collected. To perform
this experiment we use 100 chips, 100 chocolate and 100 ice
cream images collected from internet. A set of training and
testing images are used here where the training set contains 150
images and testing set contains 150 images. We have trained
our classifier engine by extracting color, shape, texture and
SURF features. Some sample images are given below:
Chocolate Ice-cream Chips
Fig. 2: Sample images from training data set
Chocolate Ice cream and Chocolate
Chips Ice cream
Fig. 3: Sample images from validation data set
Our segmentation is performed on different types of images
of chips, chocolate and ice-cream after successful identification
of desired object. Our proposed system create a classifier
depending on the extracted features of identified objects and
calculate the nutrition values.
An error matrix also called a confusion matrix is a
contingency table that comprise of the information about actual
and predicted classifications done by a classification system.
Table I to III shows the confusion matrix that appraises the
Accuracy rate of the classification. The entries in the matrix are
True Positive (TP) rate, True Negative (TN) rate, False Positive
(FP) rate, False Negative (FN) rate for each type of dataset. The
accuracy (AC) is the ratio of the total number of predictions that
were correct. It is derived by the equation:
Accuracy =
( )
( )
………………..(4)
TABLE I
CONFUSION MATRIX FOR CHIPS IMAGES
TABLE II
CONFUSION MATRIX FOR CHOCHOLATE IMAGES
Actual Class
Predicted
class
Sample
image=300
Chips Not
Chips
Accuracy
86 %
Chips 77
(TP)
24
(FP)
Not Chips 17
(FN)
182
(TN)
Vol. 16, No. 5, May 2018
ISSN 1947-5500

TABLE III
CONFUSION MATRIX FOR ICE-CREAM IMAGES
Fig. 4 represents true detected result of sample image. Our algorithm sometimes
provide False Negative and False Positive predicted result. The output image
shows this is a chips but the sample image contain a chocolate. This error is
occurred due to different texture, shape or color feature.Fig 5 shows step by
step output image for detection of object of interest:
i.
i. Reference Image ii. Target Image
iii. Detect feature points and extract feature descriptors
iv. Matching points between reference and target image
v. Detected Object
vi. Frequency of visual word occurance
vii. Final recognition
Actual ClassPredictedclass
Sample
image=30
0
Chocolate
s
Not
Chocolat
e Accurac
y
80 %Chocolate
s
60(TP) 40(FP)
Not
Chocolate
20 (FN) 180 (TN)
Actual Class
Predictedclass
Sample
image=300
Ice-
cream
Not
Ice-
cream Accuracy
90%Ice-
cream
79(TP) 22 (FP)
Not
Ice-cream
8 (FN) 192
(TN)
Vol. 16, No. 5, May 2018
ISSN 1947-5500

V. CONCLUSION
In this paper, we proposed a method to classify and to identify
high calorie snacks (Ice-cream, Chips and Chocolate) from the
test image to measure the amount of calories has taken. Using
point feature matching algorithm, our target object is detected
in a cluttered scene, with a given reference image of the object.
Image category is classified using bag of features approach
through finding point correspondences between the reference
and the target image. From our experimentation, we found our
proposed model is able to detect and classify different types of
snacks with around 85% accuracy. The accuracy is good and
false positive rate is not so high. People today are very
conscious about their health. So, along with the patient, the
health conscious person who have a major effect of food
calories can be benefitted with this approach. In future, we will
try to improve the accuracy by building a robust system which
will identify all kinds of snacks more accurately.
REFERENCES
[1] Liu, C., Cao, Y., Luo, Y., Chen, G., Vokkarane, V., & Ma, Y. (2016,
May). Deepfood: Deep learning-based food image recognition for
computer-aided dietary assessment. In International Conference on Smart
Homes and Health Telematics (pp. 37-48). Springer, Cham.
[2] Probst, Y., Nguyen, D. T., Tran, M. K., & Li, W. (2015). Dietary
assessment on a mobile phone using image processing and pattern
recognition techniques: Algorithm design and system
prototyping. Nutrients, 7(8), 6128-6138.
[3] Chen, M. Y., Yang, Y. H., Ho, C. J., Wang, S. H., Liu, S. M., Chang, E.,
... & Ouhyoung, M. (2012, November). Automatic chinese food
identification and quantity estimation. In SIGGRAPH Asia 2012
Technical Briefs (p. 29). ACM.
[4] Baxter, J. (2012). Food recognition using ingredient-level features.
[5] Sun, D. W. (2000). Inspecting pizza topping percentage and distribution
by a computer vision method. Journal of food engineering, 44(4), 245-
249.
[6] Hafiz, R., Islam, S., Khanom, R., & Uddin, M. S. (2016, December).
Image based drinks identification for dietary assessment.
In Computational Intelligence (IWCI), International Workshop on (pp.
192-197). IEEE.
[7] Kalaivani, A., Nath, S. S., & Chitrakala, S. (2013). AUTOMATIC
dominant region segmentation for natural images. ICCSEA, SPPR, CSIA,
WimoA-2013, 273-280.
[8] Patel, H. N., Jain, R. K., & Joshi, M. V. (2011). Fruit detection using
improved multiple features based algorithm. International journal of
computer applications, 13(2), 1-5.
[9] Arivazhagan, S., Shebiah, R. N., Nidhyanandhan, S. S., & Ganesan, L.
(2010). Fruit recognition using color and texture features. Journal of
Emerging Trends in Computing and Information Sciences, 1(2), 90-94.
[10] Rozario, L. J., Rahman, T., & Uddin, M. S. (2016). Segmentation of the
Region of Defects in Fruits and Vegetables. International Journal of
Computer Science and Information Security, 14(5), 399.
[11] Savakar, D. G., & Anami, B. S. (2009). Recognition and classification of
food grains, fruits and flowers using machine vision. International Journal
of Food Engineering, 5(4).
Amatul Bushra Akhi has completed her
B.Sc (honors) and M.Sc from the
Department of Computer Science and
Engineering, Jahangirnagar University.
Currently she is working as a lecturer of
Computer Science and Engineering
Department at Daffodil International
Universality. Her research interest
includes image processing, IoT, cyber
security, database etc.
Farzana Akter completed her B.Sc
(honors) and M.Sc from the
Department of Computer Science and
Engineering, Jahangirnagar University.
Currently she is working as a lecturer of
Computer Science and Engineering
Department at Daffodil International
Universality. Her research interests
includes image processing, Database,
Telemedicine, IoT etc.
Tania Khatun has completed her B.Sc
(honors) and M.Sc from the Department
of Computer Science and Engineering,
Jahangirnagar University. She is now
working as a lecturer at Daffodil
International Universality. Her research
interests includes image processing,
machine learning, distributed databases,
data mining, telemedicine etc.
Vol. 16, No. 5, May 2018
ISSN 1947-5500

Identification and Recognition of Snack Foods from Cluttered Scene for Dietary Assessment

Recommended

Recommended

More Related Content

Similar to Identification and Recognition of Snack Foods from Cluttered Scene for Dietary Assessment

Similar to Identification and Recognition of Snack Foods from Cluttered Scene for Dietary Assessment (20)

Recently uploaded

Recently uploaded (20)

Identification and Recognition of Snack Foods from Cluttered Scene for Dietary Assessment