Application for plant recognition. Scanning leafs of plans will tell you its name Application development: Sipos Andor UX/UI Design: Andor Nagl, Documentation and Presentation: Sinisa Vukovic

1. Team Nepenthes
Subotica Tech - College
of Applied Sciences,
Subotica
Plant
Recognition
System
PRS
Team members:
Siniša Vuković
Andor Sípos
Andor Nagl
Mentor:
Zlatko Čović

2. 1 | P a g e
Plant Recognition System

3. 2 | P a g e
Contents
1. Introduction ................................................................................................................3
1.1 Team members..................................................................................................3
1.2. Problem ..................................................................................................................4
2. Solution.......................................................................................................................5
3. System Architecture..................................................................................................6
3.1. Administration Interface .......................................................................................7
3.1.1. Plant Control Interface ..................................................................................7
3.1.2. Neural Network Interface ..............................................................................8
3.1.3. Recognition Interface ..................................................................................10
3.1.4. Model View ViewModel (MVVM) ...............................................................11
3.2. Database design .................................................................................................13
4. Windows Phone Application..................................................................................14
5. Recognition Theory.................................................................................................17
5.1. Image edge detection.........................................................................................17
5.2. Thinning................................................................................................................18
5.3. Leaf image token ................................................................................................18
5.4. Neuronal Network ...............................................................................................19
6. Used Technologies.................................................................................................21
6.1. Microsoft Visual Studio ......................................................................................21
6.2. Microsoft Expression Blend...............................................................................21
6.3. Windows Azure Platform ...................................................................................21
6.4. Windows Phone 7...............................................................................................22
6.5. Windows Communication Foundation (WCF)................................................22
6.6. Microsoft Silverlight ............................................................................................22
6.7. MVVM Light Toolkit ............................................................................................22
7. Conclusion................................................................................................................23
8. Future Plans.............................................................................................................23
9. References...............................................................................................................24

4. 3 | P a g e
1. Introduction
1.1 Team members
Name: Siniša Vuković
Date of birth: November 16, 1990.
Place of birth: Subotica, Serbia
School: Subotica Tech
College of Applied Sciences,
Subotica
Name: Andor Sípos
Date of birth: July 20, 1988.
Place of birth: Subotica, Serbia
School: Subotica Tech
College of Applied Sciences,
Subotica
Name: Andor Nagl
Date of birth: December 13, 1990.
Place of birth: Subotica, Serbia
School: Subotica Tech
College of Applied Sciences,
Subotica

5. 4 | P a g e
1.2. Problem
We live in a crazy, mixed up world with a lot of problems. There are some crucial
problems that need to be solved right away, before it is too late. Unfortunately
one of these problems is education.
Technology rapidly improves every day, which makes our lives much easier. Our
generation was (and still is) part of this quick improvement, and we still
remember what life was like before these high-tech gadgets and technologies.
But that is not the case with the younger generations. This highly developed
future will spoil them. They will get use to this luxury, and it will affect their whole
life.
Unfortunately this is happening right now. People are wasting their precious time
surfing the internet, playing games, and watching television all day. Surveys
show that this is a real problem that is not to be ignored. Our opinion is the same
and we cannot take this problem lightly!
The sad thing is, that this problem affects the young generations the most. A
child’s mind is easily deceived by these things. This type of entertainment can be
very dangerous, and it can also lead to addiction. Sadly we have seen it a lot of
times with our own eyes.
As we said it earlier, this problem can have a negative impact on their lives.
Unfortunately we will see that in their education as well.

6. 5 | P a g e
2. Solution
Our aim is to make the learning world exciting and interesting for the new
generations, and to take education itself to the next level.
With this goal we hope that we can grab the young ones’ attention, and show
them the beauty of this world.
We know that this is a giant task, which cannot be solved with one idea or one
project. This is one long road that we have to walk through together to achieve
our goal. Our software is just a piece of the puzzle, but it is one step forward to
victory.

7. 6 | P a g e
3. System Architecture
SQL Azure
Windows Phone ApplicationAdministration Interface
Windows Communication
Foundation
Neural NetworkLINQ to SQL
Image 3-1: The System’s Architecture

8. 7 | P a g e
3.1. Administration Interface
3.1.1. Plant Control Interface
In order to recognize a plant, we need to train the system. First we need to select
the type of the plant and then load the images.
If it’s possible upload as many pictures as possible of the selected plant kind.
Because system will be more effective, and there will be less chance of mistaking
the type of the plant.
After every uploaded picture you will have to click on “Detect token” button. The
uploaded image will only show its edges in vectors.
Image 3-2: Plant Control Interface

9. 8 | P a g e
3.1.2. Neural Network Interface
Artificial Neural Networks (ANNs) are a new approach that follow a different way
from traditional computing methods to solve problems. Since conventional
computers use algorithmic approach, if the specific steps that the computer
needs to follow are not known, the computer cannot solve the problem. That
means, traditional computing methods can only solve the problems that we have
already understood and knew how to solve. However, ANNs are, in some way,
much more powerful because they can solve problems that we do not exactly
know how to solve. That's why, of late, their usage is spreading over a wide
range of area including, virus detection, robot control, intrusion detection
systems, pattern (image, fingerprint, noise) recognition and so on.
ANNs have the ability to adapt, learn, generalize, cluster or organize data. There
are many structures of ANNs including, Percepton, Adaline, Madaline, Kohonen,
BackPropagation and many others. Probably, BackPropagation ANN is the most
commonly used, as it is very simple to implement and effective. In this work, we
will deal with BackPropagation ANNs.
BackPropagation ANNs contain one or more layers each of which are linked to
the next layer. The first layer is called the "input layer" which meets the initial
input (e.g. vectors of a leaf) and so does the last one "output layer" which usually
holds the input's identifier. The layers between input and output layers are called
"hidden layer(s)" which only propagate the previous layer's outputs to the next
layer and (back) propagates the following layer's error to the previous layer.
Actually, these are the main operations of training a BackPropagation ANN which
follows a few steps.
A typical BackPropagation ANN is as depicted below. The black nodes (on the
extreme left) are the initial inputs. Training such a network involves two phases.
In the first phase, the inputs are propagated forward to compute the outputs for
each output node. Then, each of these outputs are subtracted from its desired
output, causing an error (an error for each output node). In the second phase,
each of these output errors is passed backward and the weights are fixed. These
two phases is continued until the sum of (square of output errors) reaches an
acceptable value.

10. 9 | P a g e
Image 3-4: Neural Network Interface
On Image 3-4 we can see the Neural network Interface. This is where we train
and configure the Neural Network.
Based on the amount of images and network properties you normally need to
specify around 500-1000 Training Cycles to get a good result in the recognition
later. If the error rate drops below 0.01 you normally should encounter no
problem in recognizing different leaf images.
The amount of input neurons for the network is normally twice the amount of
tokens because of the sinus & cosine value for one token.
If the training is complete then we will see result on the left side of the interface.

11. 10 | P a g e
3.1.3. Recognition Interface
Now that we trained our system to detect the plants, we can test it by uploading a
picture of a leaf of the plant that we trained. If everything went well it will show us
the name of the plant in the upper left corner.
Image 3-5: Recognition Interface

12. 11 | P a g e
3.1.4. Model View ViewModel (MVVM)
The Model View ViewModel (MVVM) is an architectural pattern used in software
engineering that originated from Microsoft as a specialization of the presentation
model design pattern.
Image 3-6: MVVM Diagram
Elements of the MVVM pattern include:
Model: as in the classic MVC pattern, the model refers to either (a) a domain
model which represents the real state content (an object-oriented approach), or
(b) the data access layer that represents that content (a data-centric approach).
View: as in the classic MVC pattern, the view refers to all elements displayed by
the GUI such as buttons, labels, and other controls
View model: the view model is a “model of the view” meaning it is an abstraction
of the view that also serves in mediating between the view and the model which
is the target of the view data bindings
.

13. 12 | P a g e
We chose the MVVM design pattern because we wanted to separate the user
interface and the business logic. This is very useful because you can easily
modify the program afterwards.
Image 3-7: ViewModel classes

14. 13 | P a g e
3.2. Database design
The database is responsible for storage of all the information required for the
system to run correctly. Microsoft SQL Azure was chosen as data storage,
because it is easy to use and it is a free database management system.
The database diagram is shown on Image 3-5. There are a lot of tables in our
database, but the most important ones are the following: The “Leaves” table and
the tables that are in relationships with it, and the “NeuralNetwork” table.
Image 3-8: Database Design

15. 14 | P a g e
4. Windows Phone Application
In this part of the documentation we will introduce the Plant Recognition System
application for Windows Phone.
Image 4-1: PRS’s First Page
On Image 4-1 we can see the main page of the program. In the middle of the
screen we can see a white camera inside a circle. If we press this button, we will
activate a telephone’s camera view. Then we can take start taking pictures about
the plants. It is important to put the leaves on a white paper, that way the
program can easily recognize the plant. You can see that in the “how to” option,
which is located in the application bar.

16. 15 | P a g e
Image 4-2: Plant’s Image Image 4-3: Plant’s Information
If we successfully took a picture of a plant, and the program recognized it, then
we will see one picture of the captured plant from our database, and this page
will also display the plant’s name, type and family (See Image 4-2).
If we scroll to the next page, it will contain many useful information about the
captured plant. That way the user can learn a lot of interesting things about that
plant (See Image 4-3).

17. 16 | P a g e
Image 4-4: PRS’s Second Page
Image 4-4 is the second page of the application. This page will display the user’s
latest photographed plants, with their names and types.

18. 17 | P a g e
5. Recognition Theory
5.1. Image edge detection
On of the main tasks of this application is the detection of specific tokens in a leaf
image. This tokens will then be the basis of the neuronal network calculations.
Assuming that the image is a full 2D scan of a single leaf like in the examples
below, we considered to use the well-known Prewitt Edge detection algorithm
which we will explain further.
Prewitt edge detection produces an image where higher grey-level values
indicate the presence of an edge between two objects. The Prewitt Edge
Detection filter computes the root mean square of two 3x3 templates. It is one of
the most popular 3x3 edge detection filters.
The Prewitt edge detection filter uses these two 3x3 templates to calculate the
gradient value:
-1 0 1 1 1 1
-1 0 1 0 0 0
-1 0 1 -1 -1 -1
X Y
Now consider the following 3x3 image window:
+------------+
| a1 a2 a3 |
| a4 a5 a6 |
| a7 a8 a9 |
+------------+
where:
 a1 .. a9 - are the grey levels of each pixel in the filter window
 X = -1*a1 + 1*a3 - 1*a4 + 1*a6 - 1*a7 + 1*a9
 Y = 1*a1 + 1*a2 + 1*a3 - 1*a7 - 1*a8 - 1*a9
 Prewitt gradient = SQRT(X*X + Y*Y)
All pixels are filtered. In order to filter pixels located near the edge of an image,
edge pixels values are replicated to give sufficient data.

19. 18 | P a g e
5.2. Thinning
As discussed earlier, the idea of identifying a specific leaf image`s species here
is that the outer frame of a leaf is enough to specify the species it belongs to. To
accomplish that, it is necessary to identify this outer frame exactly. The
previously applied Prewitt Edge detection normally just identify the edges with a
preconfigured threshold and after this edge detection we have to perform a
thinning algorithm to minimize this threshold-based edge to a one-line frame
where we then can apply a sort of token recognition as discussed later.
The used thinning algorithm here processed the image recursively and minimizes
the found lines to a one-pixel wide one by comparing the actual pixel situation
with specific patterns and then minimizes it.
5.3. Leaf image token
The central part of this application are the tokens of each leaf image that are
found after the image processing is through with it. This runs in Windows Azure
Cloud Platform. What exactly stands behind the idea of this tokens and how we
defined this tokens should be explained here in detail.
The idea behind the transfer of the leaf image shape into a neuronal network
usable form is, that the cosine and sinus angles of the shape represents the
criterias of a recognition pattern.
Image 5-1: Leaf Token
Image 5-1 shows a part of a leaf image that was already processed through the
above mentioned edge detection and thinning algorithms.

20. 19 | P a g e
To give you an idea of what you see in this image, here is a short list:
 Green line: The shape of the leaf image after successful edge detection &
thinning.
 Red Square: This square represents a point on the shape of the leaf
image from which we are going to draw a line to the next square.
 Blue line: The compound of the center of two squares from which we are
going to calculate the cosine and sinus angle. Such a blue line is a
representation of a leaf token.
If you now take a deeper view on the small triangle zoom on this image you
should recognize that it shows a right-angled triangle. This and the summary of
all triangles of a leaf image are the representation of the tokens of a leaf from
which we can start the neuronal network calculations.
Image 5-2: Triangle which represents the tokens
On Image 5-2 you see a small image of the right-angled triangle which
represents a token of a single leaf image. Here it should be clear now that the
angles A and B are the two necessary parts which will be fit into the neuronal
network layers.
With these two angles we can exactly represent the direction of the hypotenuse
from point P1 to P2 which is absolutely necessary for the representation of a leaf
image.
5.4. Neuronal Network
Another main part of this work is the integration of a feed-forward
Backpropagation neuronal network. As described earlier the inputs for this
neuronal network are the individual tokens of a leaf image, and as a token

21. 20 | P a g e
normally consists of a cosine and sine angle, the amount of input layers for this
network are the amount of tokens multiplied by two.
Image 5-3: Neural Network
The image on the right side should give you an idea of the neuronal network that
takes place in the Leaves Recognition application.
We have chosen a feed-forward Backpropagation network because it was part
of the task to show that just a Backpropagation network and the shape of a leaf
image is enough to specify the species of a leaf.
The implemented network also just has one input, hidden and output layer to
simplify and speed-up the calculations on that java implementation.
To fill the input neurons of the network, we use the previous calculated leaf
tokens like discussed in section 2.3. The number of output neurons is normally
specified by the amount of different species because we use a encoded form to
specify the outputs.
All other behaviours of the network is specified by the normal mathematical
principals of a Backpropagation network. If you want to get an idea of how such
a Backpropagation network works, please refer to deeper explanations about the
Backpropagation algorithm.

22. 21 | P a g e
6. Used Technologies
6.1. Microsoft Visual Studio
Microsoft Visual Studio is an integrated development environment. It is used to
develop console and graphical user interface applications. Microsoft Visual
Studio includes a code editor supporting IntelliSense as well as code refactoring.
The integrated debugger works both as a source-level debugger and a machine-
level debugger. Other built-in tools include a forms designer for building GUI
applications, web designer, class designer, and database schema designer.
6.2. Microsoft Expression Blend
Expression Blend, Visual Studio, Silverlight and .NET provide the most
compelling and seamless design and development workflow on the market today.
Rapidly iterate on both the user experience and core architecture, evolving your
ideas quickly from initial prototype through to completed project.
6.3. Windows Azure Platform
Microsoft Windows Azure Platform is a Microsoft cloud computing platform used
to build, host and scale web applications through Microsoft data centers. Azure is
classified as platform as a service and forms part of Microsoft's cloud computing
strategy, along with its software as a service offering, Microsoft Online Services.
The platform consists of various on-demand services hosted in Microsoft data
centers and commoditized through three product brands.

23. 22 | P a g e
6.4. Windows Phone 7
Windows Phone 7 is a mobile operating system developed by Microsoft, and is
the successor to its Windows Mobile platform. Unlike its predecessor, it is
primarily aimed at the consumer market rather than the enterprise market.
Applications and games for Windows Phone 7 must be based on XNA or a WP7
specific version of Silverlight.
6.5. Windows Communication Foundation (WCF)
WCF is an application programming interface (API) in the .NET Framework for
building connected, service-oriented applications. WCF is implemented primarily
as a set of classes on top of the .NET Framework’s Common Language Runtime
(CLR). This lets .NET developers build service-oriented applications in a familiar
way. WCF allows creating clients that access services. Both the client and the
service can run in pretty much any Windows process — WCF doesn’t define a
required host. Wherever they run, clients and services can interact via SOAP, via
a WCF-specific binary protocol, and in other ways.
6.6. Microsoft Silverlight
Microsoft Silverlight is an application framework for writing and running rich
Internet applications, with features and purposes similar to those of Adobe Flash.
The run-time environment for Silverlight is available as a plug-in for web
browsers running under Microsoft Windows and Mac OS X. While early versions
of Silverlight focused on streaming media, current versions support multimedia,
graphics and animation, and give developers support for CLI languages and
development tools. Silverlight is also one of the two application development
platforms for Windows Phone.
6.7. MVVM Light Toolkit
The main purpose of the toolkit is to accelerate the creation and development of
MVVM applications in WPF, Silverlight and in the Windows Phone.
Like other MVVM implementations, the toolkit helps you to separate your View
from your Model which creates applications that are cleaner and easier to
maintain and extend. It also creates testable applications and allows you to have
a much thinner user interface layer (which is more difficult to test automatically).

24. 23 | P a g e
7. Conclusion
Plant Recognition System is an educational and entertaining software for
everyone. Our particular targets are the younger generations, and the
adventurers who love spending time in the great outdoors, but it can be
interesting for everyone.
Like we said it earlier, our goal is to draw people’s attention towards education, to
show its beauty, and to make education itself more entertaining, interactive and
exciting for everyone.
We hope that our project can be a building block to this dream, or maybe it will
inspire others to pursue our goal, and together we can make the world a better
place.
8. Future Plans
Although the program is complete, there are many things that we can improve
and add to it. We still have many useful and interesting ideas that we can use,
like the following:
 Upload the database with as many plants as possible.
 Improve the recognition algorithm.
 Develop the application for Windows 8 Metro.

25. 24 | P a g e
9. References
[01] http://en.wikipedia.org/wiki/Neural_network
[02] http://en.wikipedia.org/wiki/Artificial_neural_network
[03] http://msdn.microsoft.com/
[04] http://edis.ifas.ufl.edu/
[05] http://www.codeproject.com/Articles/19323/Image-Recognition-with-
NeuralNetworks
[06] http://flavia.sourceforge.net/
[07] http://galasoft.ch/mvvm/