The document describes DiscoverNet, an adaptive simulation-based learning environment for designing neural networks. DiscoverNet allows users to build neural network simulators by manipulating neural objects. It includes a consultant module to provide educational guidance during the design process. The consultant reproduces the user's initial neural network design and identifies any issues to help the user correct misconceptions. Experimental results show that DiscoverNet supports learning and helps users develop an understanding of neural network mechanisms.
NETWORK LEARNING AND TRAINING OF A CASCADED LINK-BASED FEED FORWARD NEURAL NE...ijaia
Presently, considering the technological advancement of our modern world, we are in dire need for a system that can learn new concepts and give decisions on its own. Hence the Artificial Neural Network is all that is required in the contemporary situation. In this paper, CLBFFNN is presented as a special and intelligent form of artificial neural networks that has the capability to adapt to training and learning of new ideas and be able to give decisions in a trimodal biometric system involving fingerprints, face and iris biometric data. It gives an overview of neural networks.
Artificial Neural Networks: Applications In ManagementIOSR Journals
With the advancement of computer and communication technology, the tools used for management decisions have undergone a gigantic change. Finding the more effective solution and tools for managerial problems is one of the most important topics in the management studies today. Artificial Neural Networks (ANNs) are one of these tools that have become a critical component for business intelligence. The purpose of this article is to describe the basic behavior of neural networks as well as the works done in application of the same in management sciences and stimulate further research interests and efforts in the identified topics.
Analysis of Neocognitron of Neural Network Method in the String RecognitionIDES Editor
This paper aims that analysing neural network method
in pattern recognition. A neural network is a processing device,
whose design was inspired by the design and functioning of
human brain and their components. The proposed solutions
focus on applying Neocognitron Algorithm model for pattern
recognition. The primary function of which is to retrieve in a
pattern stored in memory, when an incomplete or noisy version
of that pattern is presented. An associative memory is a
storehouse of associated patterns that are encoded in some
form. In auto-association, an input pattern is associated with
itself and the states of input and output units coincide. When
the storehouse is incited with a given distorted or partial
pattern, the associated pattern pair stored in its perfect form
is recalled. Pattern recognition techniques are associated a
symbolic identity with the image of the pattern. This problem
of replication of patterns by machines (computers) involves
the machine printed patterns. There is no idle memory
containing data and programmed, but each neuron is
programmed and continuously active.
NETWORK LEARNING AND TRAINING OF A CASCADED LINK-BASED FEED FORWARD NEURAL NE...ijaia
Presently, considering the technological advancement of our modern world, we are in dire need for a system that can learn new concepts and give decisions on its own. Hence the Artificial Neural Network is all that is required in the contemporary situation. In this paper, CLBFFNN is presented as a special and intelligent form of artificial neural networks that has the capability to adapt to training and learning of new ideas and be able to give decisions in a trimodal biometric system involving fingerprints, face and iris biometric data. It gives an overview of neural networks.
Artificial Neural Networks: Applications In ManagementIOSR Journals
With the advancement of computer and communication technology, the tools used for management decisions have undergone a gigantic change. Finding the more effective solution and tools for managerial problems is one of the most important topics in the management studies today. Artificial Neural Networks (ANNs) are one of these tools that have become a critical component for business intelligence. The purpose of this article is to describe the basic behavior of neural networks as well as the works done in application of the same in management sciences and stimulate further research interests and efforts in the identified topics.
Analysis of Neocognitron of Neural Network Method in the String RecognitionIDES Editor
This paper aims that analysing neural network method
in pattern recognition. A neural network is a processing device,
whose design was inspired by the design and functioning of
human brain and their components. The proposed solutions
focus on applying Neocognitron Algorithm model for pattern
recognition. The primary function of which is to retrieve in a
pattern stored in memory, when an incomplete or noisy version
of that pattern is presented. An associative memory is a
storehouse of associated patterns that are encoded in some
form. In auto-association, an input pattern is associated with
itself and the states of input and output units coincide. When
the storehouse is incited with a given distorted or partial
pattern, the associated pattern pair stored in its perfect form
is recalled. Pattern recognition techniques are associated a
symbolic identity with the image of the pattern. This problem
of replication of patterns by machines (computers) involves
the machine printed patterns. There is no idle memory
containing data and programmed, but each neuron is
programmed and continuously active.
International Journal of Engineering Research and Applications (IJERA) is an open access online peer reviewed international journal that publishes research and review articles in the fields of Computer Science, Neural Networks, Electrical Engineering, Software Engineering, Information Technology, Mechanical Engineering, Chemical Engineering, Plastic Engineering, Food Technology, Textile Engineering, Nano Technology & science, Power Electronics, Electronics & Communication Engineering, Computational mathematics, Image processing, Civil Engineering, Structural Engineering, Environmental Engineering, VLSI Testing & Low Power VLSI Design etc.
A systematic review on sequence-to-sequence learning with neural network and ...IJECEIAES
We develop a precise writing survey on sequence-to-sequence learning with neural network and its models. The primary aim of this report is to enhance the knowledge of the sequence-to-sequence neural network and to locate the best way to deal with executing it. Three models are mostly used in sequence-to-sequence neural network applications, namely: recurrent neural networks (RNN), connectionist temporal classification (CTC), and attention model. The evidence we adopted in conducting this survey included utilizing the examination inquiries or research questions to determine keywords, which were used to search for bits of peer-reviewed papers, articles, or books at scholastic directories. Through introductory hunts, 790 papers, and scholarly works were found, and with the assistance of choice criteria and PRISMA methodology, the number of papers reviewed decreased to 16. Every one of the 16 articles was categorized by their contribution to each examination question, and they were broken down. At last, the examination papers experienced a quality appraisal where the subsequent range was from 83.3% to 100%. The proposed systematic review enabled us to collect, evaluate, analyze, and explore different approaches of implementing sequence-to-sequence neural network models and pointed out the most common use in machine learning. We followed a methodology that shows the potential of applying these models to real-world applications.
Artificial Neural Network and its Applicationsshritosh kumar
Abstract
This report is an introduction to Artificial Neural
Networks. The various types of neural networks are
explained and demonstrated, applications of neural
networks like ANNs in medicine are described, and a
detailed historical background is provided. The
connection between the artificial and the real thing is
also investigated and explained. Finally, the
mathematical models involved are presented and
demonstrated.
APPLYING NEURAL NETWORKS FOR SUPERVISED LEARNING OF MEDICAL DATAIJDKP
Constructing a classification model based on some given patterns is a form of learning from the environment perception. This modelling aims to discover new knowledge embedded in the input observations. Learning behaviour of the neural network model enhances the classification properties. This paper considers artificial neural networks for learning two different medical data sets in term of number of instances. The experiment results confirm that the back-propagation supervised learning algorithm has proved its efficiency for such non-linear classification issues.
This presentation guide you through Neural Networks, use neural networksNeural Networks v/s Conventional
Computer, Inspiration from Neurobiology, Types of neural network, The Learning Process, Hetero-association recall mechanisms and Key Features,
For more topics stay tuned with Learnbay.
International Journal of Engineering Research and Applications (IJERA) is an open access online peer reviewed international journal that publishes research and review articles in the fields of Computer Science, Neural Networks, Electrical Engineering, Software Engineering, Information Technology, Mechanical Engineering, Chemical Engineering, Plastic Engineering, Food Technology, Textile Engineering, Nano Technology & science, Power Electronics, Electronics & Communication Engineering, Computational mathematics, Image processing, Civil Engineering, Structural Engineering, Environmental Engineering, VLSI Testing & Low Power VLSI Design etc.
A systematic review on sequence-to-sequence learning with neural network and ...IJECEIAES
We develop a precise writing survey on sequence-to-sequence learning with neural network and its models. The primary aim of this report is to enhance the knowledge of the sequence-to-sequence neural network and to locate the best way to deal with executing it. Three models are mostly used in sequence-to-sequence neural network applications, namely: recurrent neural networks (RNN), connectionist temporal classification (CTC), and attention model. The evidence we adopted in conducting this survey included utilizing the examination inquiries or research questions to determine keywords, which were used to search for bits of peer-reviewed papers, articles, or books at scholastic directories. Through introductory hunts, 790 papers, and scholarly works were found, and with the assistance of choice criteria and PRISMA methodology, the number of papers reviewed decreased to 16. Every one of the 16 articles was categorized by their contribution to each examination question, and they were broken down. At last, the examination papers experienced a quality appraisal where the subsequent range was from 83.3% to 100%. The proposed systematic review enabled us to collect, evaluate, analyze, and explore different approaches of implementing sequence-to-sequence neural network models and pointed out the most common use in machine learning. We followed a methodology that shows the potential of applying these models to real-world applications.
Artificial Neural Network and its Applicationsshritosh kumar
Abstract
This report is an introduction to Artificial Neural
Networks. The various types of neural networks are
explained and demonstrated, applications of neural
networks like ANNs in medicine are described, and a
detailed historical background is provided. The
connection between the artificial and the real thing is
also investigated and explained. Finally, the
mathematical models involved are presented and
demonstrated.
APPLYING NEURAL NETWORKS FOR SUPERVISED LEARNING OF MEDICAL DATAIJDKP
Constructing a classification model based on some given patterns is a form of learning from the environment perception. This modelling aims to discover new knowledge embedded in the input observations. Learning behaviour of the neural network model enhances the classification properties. This paper considers artificial neural networks for learning two different medical data sets in term of number of instances. The experiment results confirm that the back-propagation supervised learning algorithm has proved its efficiency for such non-linear classification issues.
This presentation guide you through Neural Networks, use neural networksNeural Networks v/s Conventional
Computer, Inspiration from Neurobiology, Types of neural network, The Learning Process, Hetero-association recall mechanisms and Key Features,
For more topics stay tuned with Learnbay.
CCS355 Neural Networks & Deep Learning Unit 1 PDF notes with Question bank .pdfAsst.prof M.Gokilavani
UNIT I INTRODUCTION
Neural Networks-Application Scope of Neural Networks-Artificial Neural Network: An IntroductionEvolution of Neural Networks-Basic Models of Artificial Neural Network- Important Terminologies of
ANNs-Supervised Learning Network.
A Parallel Framework For Multilayer Perceptron For Human Face RecognitionCSCJournals
Artificial neural networks have already shown their success in face recognition and similar complex pattern recognition tasks. However, a major disadvantage of the technique is that it is extremely slow during training for larger classes and hence not suitable for real-time complex problems such as pattern recognition. This is an attempt to develop a parallel framework for the training algorithm of a perceptron. In this paper, two general architectures for a Multilayer Perceptron (MLP) have been demonstrated. The first architecture is All-Class-in-One-Network (ACON) where all the classes are placed in a single network and the second one is One-Class-in-One-Network (OCON) where an individual single network is responsible for each and every class. Capabilities of these two architectures were compared and verified in solving human face recognition, which is a complex pattern recognition task where several factors affect the recognition performance like pose variations, facial expression changes, occlusions, and most importantly illumination changes. Experimental results show that the proposed OCON structure performs better than the conventional ACON in terms of network training convergence speed and which can be easily exercised in a parallel environment.
Prediction of Student's Performance with Deep Neural NetworksCSCJournals
The performance of education has a big part in people's life. The prediction of student's performance in advance is very important issue for education. School administrators and students' parents impact on students' performance. Hence, academic researchers have developed different types of models to improve student performance. The main goal to reveal of this study is to search the best model of neural network models for the prediction of the performance of the high school students. For this purpose, five different types of neural network models have been developed and compared to their results. The data set obtained from Taldykorgan Kazakh Turkish High School (in Kazakhstan) students was used. Test results show that proposed two types of neural network model are predicted students' real performance efficiently and provided better accuracy when the test of today’s and future’s samples have similar characteristics.
With the surge in modern research focus towards Pervasive Computing, lot of techniques and challenges
needs to be addressed so as to effectively create smart spaces and achieve miniaturization. In the process of
scaling down to compact devices, the real things to ponder upon are the Information Retrieval challenges.
In this work, we discuss the aspects of multimedia which makes information access challenging. An
Example Pattern Recognition scenario is presented and the mathematical techniques that can be used to
model uncertainty are also presented for developing a system that can sense, compute and communicate in
a way that can make human life easy with smart objects assisting from around his surroundings.
AN ADAPTIVE REUSABLE LEARNING OBJECT FOR E-LEARNING USING COGNITIVE ARCHITECTUREacijjournal
Nowadays, a huge amount of ambiguous e-learning materials are available in World Wide Web
irrespective of various objectives. These digital educational resources can be reused and shared from
centralized online repository and it will avoid the redundant learning material. The main goal is to design
consistent adaptable e-learning course material for web-based education system with emphasis on the
quality of learning. This can be done by organizing learning object in a prescribed manner and it can be
reused in feature. Such reusable learning objects are enhanced further to become adaptive reusable
learning objects that are virtually cognitive and responsive towards the specific needs of the user/customer.
This paper proposes the cognitive architecture to offer an adaptive reusable objects (RLO) based on
individual profile of e-learner besides their cognitive behaviour while learning.
Proposing a new method of image classification based on the AdaBoost deep bel...TELKOMNIKA JOURNAL
Image classification has different applications. Up to now, various algorithms have been presented
for image classification. Each of these methods has its own weaknesses and strengths. Reducing error rate
is an issue which many researches have been carried out about it. This research intends to optimize
the problem with hybrid methods and deep learning. The hybrid methods were developed to improve
the results of the single-component methods. On the other hand, a deep belief network (DBN) is a generative
probabilistic modelwith multiple layers of latent variables and is used to solve the unlabeled problems. In
fact, this method is anunsupervised method, in which all layers are one-way directed layers except for
the last layer. So far, various methods have been proposed for image classification, and the goal of this
research project was to use a combination of the AdaBoost method and the deep belief network method to
classify images. The other objective was to obtain better results than the previous results. In this project, a
combination of the deep belief network and AdaBoost method was used to boost learning and the network
potential was enhanced by making the entire network recursive. This method was tested on the MINIST
dataset and the results were indicative of a decrease in the error rate with the proposed method as compared
to the AdaBoost and deep belief network methods.
Soft Computing based Learning for Cognitive Radioidescitation
Over the last decade the world of wireless communications has been undergoing
some crucial changes, which have brought it at the forefront of international research and
development interest, eventually resulting in the advent of a multitude of innovative
technologies and associated products such as WiFi, WiMax, 802.20, 802.22, wireless mesh
networks and Software Defined Radio. Such a highly varying radio environment calls for
intelligent management, allocation and usage of a scarce resource, namely the radio
spectrum. One of the most prominent emerging technologies that promise to handle such
situations is Cognitive Radio. Cognitive Radio systems are based on Software Defined Radio
technology and utilize intelligent software packages that enrich their transceivers with the
highly attractive properties of self-awareness, adaptability and capability to learn. The
Cognitive Engine, the intelligent system behind the Cognitive Radio, combines sensing,
learning, and optimization algorithms to control and adapt the radio system from the
physical layer and up the communication stack. The integration of a learning engine can be
very important for improving the stability and reliability of the discovery and evaluation of
the configuration capabilities. To this effect, many different learning techniques are
available and can be used by a Cognitive Radio ranging from pure lookup tables to
arbitrary combinations of soft Computing techniques, which include among others:
Artificial Neural Networks, evolutionary/Genetic Algorithms, reinforcement learning, fuzzy
systems, Hidden Markov Models, etc. The proposed work contributes in this direction,
aiming to develop a learning scheme and work towards solving problems related to learning
phase of Cognitive Radio systems. Interesting scenarios are to be mobilized for the
performance assessment work, conducted in order to design and use an appropriate
structure, while indicative results need to be presented and discussed in order to showcase
the benefits of incorporating such learning schemes into Cognitive Radio systems.
Subsequently feasibility of such learning schemes could be tested with simulations. In the
near future, such learning schemes are expected to assist a Cognitive Radio system to
compare among the whole of available, candidate radio configurations and finally select the
best one to operate in.
1. DiscoverNet: Adaptive Simulation-Based Learning Environment
And Support System for Designing Neural Networks
Safia Belkada, Alexandra I. Cristea and Toshio Okamoto
AI & Knowledge Engineering Lab., Graduate School of Information Systems
University of Electro-Communications
Choufugaoka 1-5-1, Chofu-shi, Tokyo 182-8585, Japan
safia@ai.is.uec.ac.jp
Abstract
For the greatest part, intelligent tutoring systems (ITS) have attempted to implement traditional
methods of learning, teaching, train-and-practice, in which students solve relatively simple problems,
chosen in advance by the teacher. Recently however, adaptive educational systems and flexible learning
systems are being considered, as an alternative, powerful new strategy of teaching/learning. This paper
presents the development of an adaptive simulation-based learning environment for designing neural
networks (DiscoverNet), for discovery and self-learning. We chose ‘neural networks’ (NN) as
application domain, due to the fact that their study is particularly suitable for an adaptive, learner level
adjusted approach, as we will show in the paper. We embedded in the NN Computer Aided Design
(CAD) teaching system also a consultant module that is responsible for the educational orientation
during the design of neural networks. Our consulting function is based on storing and processing the
domain knowledge on NN, as well as the current user’s NN knowledge, in a knowledge based neural
network (KBNN) form. This paper presents our DiscoverNet system, with a special focus on the
adaptive features of the KBNN consultant module. We also present a case study for the beginner user,
as well as the results obtained. We will also discuss the educational methods used, as well as the
advantages we are aiming at with these methods.
1 Introduction
1.1 Background and Related Work
Recently, the trend towards developing interactive learning environments or intelligent tutoring systems is gaining
in strength, and, as McArthur et al. [5] pointed out, there are various examples of such researches now a days.
Simulation-learning environments have been proven to be an effective learning strategy [2,5]. This kind of strategy
allows learning by using simulation, by engaging learners to manipulate, freely and directly, concrete objects in the
experimental environment, and observe the changes induced by their manipulations. The aim of this type of
functionality is to foster the integration of theory and practice. The various possible situations, which are implied
implicitly or explicitly in the environment, build up the learner’s experience and, through trial and error, attract the
1
2. learner and support him/ her in establishing his/ her independence and develop his/ her fundamental comprehension
[1,3,10].
Building support environments using only symbolic, knowledge-based systems often leads to the well-known
knowledge acquisition bottleneck. A sub-symbolic neural network model can be trained to perform reasonably well -
based on the training data sets - but it cannot readily incorporate domain knowledge or provide explanation of the
results. A hybrid system made of a combination of the two approaches cited above has been shown [6] to outperform
both, when used separately. There are different approaches to constructing hybrid intelligent systems, which aim to
exploit the features of knowledge-based systems and neural network methods [6,8,11]. One way [11] is to translate the
domain theory into a neural network, and thereby determine the neural network’s topology, then refine the reformulated
rules, by using, for instance, backpropagation. The KBNN algorithm falls in this category [11]. We are building on the
KBNN approach, but with the difference that we use the KBNN network space as a search space for the correct network
topology, closest to the user designed network, as we will show further on.
1.2 Neural Networks as a Problem Solving Tool
There has been a great interest in artificial neural networks over the last few years, as researchers from diverse
backgrounds have produced theoretical foundations and demonstrated numerous useful applications of this field [9].
Neural networks are tools for solving many types of problems. These problems may be characterized as mapping,
clustering, and constrained optimization. There are several neural networks available for each type of problem. For
many applications, the mapping of the problem on a neural network requires detailed understanding of neural networks,
which can be achieved only by performing calculations by oneself, and by developing some simulators for extremely
simple examples. Most students that have to use neural networks, often have to face great difficulties to even acquire a
working knowledge of the very basics. Such problems, indeed very common in many areas, require however knowledge
about highly abstract concepts. When using neural networks for different domain applications, this involves
constructing simulators, and this can be very difficult to do, for example, during a NN lecture. For the same reasons,
students cannot really be expected to solve anything by trivial exercises during tutorials. This is not so much due to the
conceptual difficulties inherent in the neural networks, but a simple consequence of the representational complexity of
such a model, unrelated to the pedagogical method used. To overcome these problems, we build an adaptive learning
environment, to encourage the student to develop a deep understanding of the mechanisms of training and applying
neural networks.
2
3. 1.3 Our Research Goal and Methodology
In this paper we propose a framework based on a hybrid approach, integrating a knowledge-based system with the
neural network paradigm, to support learning neural networks within the DiscoverNet system. DiscoverNet allows
learners (for the sake of convenience only, we will use in the remaining the word “user” to refer to learner) to build
their neural network simulators, by blending theory and praxis within the environment, and facilitate learning through a
tutoring program. The system is based on the concept that neural networks can be broken down into a fundamental set
of neural components [4]. The user constructs his/her domain theory in the environment where s/he can choose some
components and find the pattern of relationships among these components. We embedded a consultant module in the
system that is responsible for the educational orientation during the design of neural networks. The consultant
reproduces the initial neural network, which is made by the user during the manipulations of neural objects, by using a
set of inference rules, to help determine the network's architecture and initial weight values. The consultant examines
which information is lacking or wrong, and sends it to the user, together with the initial network, showing the user
his/her misconceptions. The system has an object-oriented architecture, in which an adaptive user interface adjusts to the
user's skills, and has the role of the motivator in the discovery learning process. I.e., the interface allows visualizing
neural models as concrete neural objects. A user builds his/her models by making use of the neural objects and methods
provided by the library of objects. The system is based on a network of networks, adapting itself to the current user. The
order of constructing neural networks was chosen to reflect the complexity of the course material. However, the
material at each level is independent, so that the user may enter the design session at any level and in almost any order
s/he desires.
In this paper we will first show the difficulties that appear in teaching neural networks, and thereby justify the need
of an adaptive learning environment for this field. We identify the interaction with the design mechanism and the
visualization of abstract concepts as two different components of the pedagogical tool for teaching neural networks. Our
students are seniors, or graduate students, in science and engineering. It is assumed that the user is familiar with
calculus and some vector-matrix notions and operations. The mathematical treatment has been kept at a minimal level
in the system, allowing the users to build their neural networks by blending theory and praxis within the environment,
without too much delay or overhead.
The paper is organized as follows. Section 2 describes the basic features of the CAD learning system and its
underlying learning environment. In Section 3 we describe the computational aspects of the support system, via the
domain knowledge representation, followed by an illustration of the advice process. Experimental results and their
3
4. evaluation are reported in section 4, highlighting learning capabilities and properties of the system. Finally, we draw
some conclusions and evaluate our system from the educational point of view.
2. Simulation-Based Learning Environment
2.1 Neural Network Models and Preliminary Research
The most distinctive element in neural networks, as opposed to traditional computational structures, is denoted as
learning. Learning represents the adaptation of some characteristic neural network parameters, by using a set of
examples (known as outcomes of the problem for given conditions) so that, for given input patterns, the outputs reflect
the value of specific functions. The final data-modeling goal is to be able to generalize from prior examples (to be able
to estimate the outcomes of the problem under unknown conditions). Neural networks represent the combination of
architecture (topology) and appropriate learning techniques. Most NNs are often classified as single layer or multilayer
NNs. A neural network model is formally defined by a quadruple < S, W, L, F >, where W is a set of weights (wji), L is
a set of learning algorithms (la), F is a set of external activation functions (fact), and S is further defined by a pair (N, C),
where N is a set of units (ni) and C a set of connections (ci). Assume n units in a neural network model. We have the
following definitions:
N = {ni| 1 ≤ n}
C ⊆ {(ni→ nj) | 1 ≤ (i, j) ≤ n}
W ⊆ {(ni→ nj, wji) | 1 ≤ (i, j) ≤ n}
L = {la1, la2…}
F = {fact1, fact2…} (1)
The number of connections between the units defines the number of layers in the NN. Each two connected units have
an associated weight. The weights represent information being used by the model to solve the problem. Each learning
algorithm is a mapping that transforms one neural network state, which is defined by the set of W, to another neural
network state. For the most typical neural network setting, training is accomplished by presenting a sequence of vectors
or patterns. The weights are then adjusted according to a learning algorithm, i.e., in backpropagation, by summing the
weighted inputs and applying an external activation function. Typically, the same activation function is used for all units
in any particular layer. For instance, let F be a set of activation functions F ={f}, and Xi, Yj a sets of input and output
units. Assume that xi ∈ Xi and yj ∈ Yj. The output, yj, is expressed as:
yinj=Σxiwii ( 2)
4
5. yj =fact(yinj) ( 3)
We have based our CAD teaching environment on some knowledge extraction from three domain specialists, teachers
and professors of Neural Networks at university level and higher. The information based on their experience was
refined and used for the implementation of our system.
2.2 Knowledge Representation and Object Manipulation in DiscoverNet
Abstract
Class
neural
Learning algorithms
Backprop
Kohonen
-addNeuronLayer
-createMapLayer
-connectLayers
-getActivationArea
-learn�
.
Weight matrices Layers Units
-changeWeights -activateSigmoid -activation funct
--compOutputErr
Unit map
Comment objects
Specific
objects
pattern Input value
The domain knowledge of neural networks is represented in the system as a KBNN set of neural objects stored in
Inputs objects
the object library (figure 1). The object library consists of a package of classes, which provide meta- representations for
the graphical objects. The architecture of each class has two levels. The object knowledge level is based on objects and
Figure 1: Neural Net Knowledge Base
contains the core of formal and procedural concepts related to the neural network models. The user creates instances of
a class when s/he manipulates objects. For instance (figure 2), in the feedforward neural networks [7], the initial
instances are the creation of activation functions instances, initial matrix weight instances, and control parameters
instances. The user can interactively create instances and specify the values associated with the relevant functions. The
structural knowledge level allows the exploration of the design stage by defining concepts and relations (e.g., learning
algorithm) over the object knowledge. The structural knowledge is a set of rules defined on sets of instances from the
domain knowledge. The modeling of the domain knowledge of neural network consists of hierarchical relations
between various neural network models, as described in Figure 1. We construct models of knowledge for various neural
network models and various design models for a particular knowledge model.
5
6. Each object encapsulates a state and events. Modeling knowledge domain consists of hierarchical relationships of
various neural network models: we construct knowledge models for the various neural network models, and various
design models established for a particular knowledge model. The levels shown in Figure 1 do not represent levels of
abstraction, they only show, which classes are present and what classes have relationships to each other. The abstract
class “Neural” contains generic methods, e.g., a method to set the initial learning rate of a neural net. Note that the
system prototype contains classes suitable for the Backpropagation network and the Kohonen Feature Map network.
These types of neural networks are considered as most powerful, yet basic, models [7]. We chose these models because
they are the most used, and the equations that describe the models are simple compared to other models. Multilayer
perceptrons (MLP) are feedforward networks trained with static backpropagation [7]. These networks are used in many
applications requiring static pattern classification. Backpropagation is a learning algorithm used to change the weights
connected to the network’s hidden layer(s) [7]. The backpropagation uses a computed output error to change the
weights values in backward direction. It is vitally important to choose the correct algorithm, since the choice of the
algorithm can affect the final neural network performance. For the first steps in the neural network domain, a novice
user can try to handle similar problems of growing complexity.
Figure 2 shows a simplified user action model constructed by the consultant. Each node represents a design
object instance. Each design object tree can be decomposed into components required to accomplish the higher-level
node restrictions. Actions are at the lowest level in the graph. The decomposition may lead to nodes that are sequential.
Sequential nodes represent actions that must be performed in the prescribed order. Arcs represent events that describe
the initiating and terminating state of an action.
Major learner actions
Supervised Unsupervise
learning d learning
Actions to design a specific neural network
Learning Algorithm,
e.g.,backpropagation Input data
Architectu (training/testing data)
re
Parameters’
specification
Related components actions
6
7. Weight matrix
Number of layers/ Learning rate
units
Threshold
specifications
Connections’ type
e.g., full_connection
Figure2: User’s Design Model
In the following, the state and event concepts that we used will be explained.
2.2.1 State
A state, X, is a class variable which is a numerical attribute (e.g., an initial weight matrix), a characteristic of a
neural object (e.g., activation function) or a process (e.g., learning algorithm). It can be computed from predefined
formulas.
2.2.2 Event
Our definition of events is analogue to the definition of events in object-oriented computation. The event handling
determines the communication between interactive objects. Our communication model is a connective model, which
allows any object to communicate with any other one. However, for instance, in order for the “full_connection” object
to communicate with a unit object, it must have a pointer to that object. In such a case, the “full_ connection” object
can invoke some method on the unit object that handles the communication. The neural network model is constructed
from a collection of objects and the appropriate relationship between them. The overall design is provided by the object
interaction. However, although breaking down neural models into a set of neural objects reduces their complexity, it
sometimes leads to larger sets of objects. To manage these objects we determine the relationship existing between them
by using the inheritance mechanism.
Figure 3 shows an example of object interaction mechanism via messages that lead to object state transition.
Message
corresponding to
the trigging
simulation stage Simulation trace file
Message
analyzer
Message object transmission
Activate object
7
Figure 3: objects interactions
8. 2.3 DiscoverNet System Architecture
With DiscoverNet the user designs a neural network model through different manipulations of objects existing in the
library and different requests to the system for helps and hints. Figure 4 shows a simplified control flow of DiscoverNet,
which consists of the following parts:
(i) Adaptive user interface: it consists of the learning and authoring environment related to the design of the neural
network models. It is in this part also, where the user model representing the individual’s understanding level is first
analyzed and built.
(ii) Knowledge based neural network: it accumulates neural network models. Each neural model consists of the
respective architectures, learning algorithm and set of parameters [7]. The selection of the appropriate parameters and
architecture for a given application is made after many trials.
(iii) Consultant module: it manages the communication between different parts and gives assistance to the user
whenever it is needed.
(iv) In Figure 4, the design database stores representations of the design description made by the user of the solving
procedures of one specific application.
(v) The object library contains components of neural networks.
(vi) The system gradually builds a user model, and uses it for the analysis and understanding of the real user’s needs.
The consultant module uses this model, in particular, to generate useful guidance and correct the user mistakes.
User Model
Object Library
Adaptive user Interface
User
Knowledge Base
CONSULTANTMODULE
Neural Network
(KBNN)
Advisor
Design
Database
8
9. Figure 4. Architecture of DiscoverNet System
3. Discovery Learning Support System
In this section, we analyze the approach used for the consultant. The consultant module gives advice to the user
considering different possible concept descriptions, by integrating a knowledge-based system with a neural network that
helps the former to refine the initial domain knowledge. The consultant has two main parts: the rule base and the
advice algorithm.
3.1 The Rule Base System
A rule-based system captures the decision-making of the design. The decision-making rules consist of packages of
classes that attempt to identify objects or decide on actions:
(i) Design classes contain a set of rules, which describe the construction of a neural network model according to what
constraints should be given in the design stage and what information is desired ( An example can be found in figure 5).
(ii) For each Design class we define a set of rules that describe a specific neural network model and we name it
Specification class (Please find and example in figure 6).
(iii) Advisor classes manage the communication between the user and the whole system.
Rule MLFFwithBP = new Rule(rb, "MLFF_with_BP",
new Clause(supervisedType,cEquals, "Norecurrent") ,
new Clause(activfunc,cEquals, "nolinear"),
new Clause(learning,cEquals, "Backpropagation"),
new Clause(training,cEquals, 巴 atch_training")) ;
�
Figure 5. Definition of rules in the Design class
RuleVariable supervisedType = new RuleVariable("Supervised_Type") ;
RuleVariable learning = new RuleVariable("learning") ;
RuleVariable activfunc = new RuleVariable("ActivFunc") ;
learning.setLabels("Delta_rule Error_correction Competitive_learning Backpropagation") ;
Figure 6. Definition of rule 痴 variables in the specification class
3.2 The Advice Process
In the following we examine the computational approach of the advice process by using our algorithm, which has
9
10. two phases:
1) Searching through the space of neural net architectures H.
2) Training the neural network and providing feedback information.
Figure 7 shows the advice process in the consultant. The consultant diagnoses a neural model made by the user by
constructing a script file during the user's manipulations of objects (considered as initial domain knowledge). The
algorithm uses the initial knowledge to help guide the search through the space H and determine a starting point in the
weights space. The algorithm then tries to translate the script file into a set of units represented as an AND/OR
dependency tree. From the tree, the consultant chooses a proper index, related to the learning algorithm, and examines
whether the architecture and the corresponding parameters can be approximated in the space H. The following
algorithm describes the searching method:
(i) Assume I <S2, W2,L2,F2> is a set of instances representing the initial knowledge (user input).
(ii) Select h <S1,W1,L1,F1> with q layers in H,
(iii) I= I+ U I-: I+ is the set of instances satisfying h, and I-,those that do not satisfy h,
(iv) Map h <S1,W1,L1,F1> with I <S2,W2,L2,F2>
(v) Compute the probability P to choose the best initial weights in h to be included in I+,
∑ w ji
( j ,i )∈I
(vi) Pi = , Where wji ∈ I is weight value from unit i to unit j. The probability of belonging to the
∑ w ji + ∑ w ji
( j ,i )∈I ( j , i )∈H
instance I- is (1-this probability). The consultant tends to assign most of the nodes that are heavily linked together. This
helps to minimize the destruction of the design model I made by the user, since nodes belonging to the designed model
are connected by heavily linked weights.
(vii) I can be mapped to h if card (I+)= card (h)
(viii)
(ix)and
Pi ≠ 0.
That is, h can induce all possible partitions of I+ instance if the dimension of h is equal to the one of I+.
: File transformation
: Algorithm 10
NN made by the system and
: Input/output Candidates of best parameters for
Script :file: Analyzer algorithm
Learner
Rulebase the neural model presented to the
Domain
Actions trace file transformation learner
Searcher/
Knowledge : Database the NN) H)
(Train
Algorithm (space
11. Design/specification
classes
Rule1: {MLP,
superv}
Rule2:
{MLP,Bp,input/outpu
Initial neural model
made by a user
AND/OR Tree
Consultant Module
Constructed Neural
Model: graphical
representation
Figure 7 Advice Process in the Consultant Module
The induced network is a fully connected neural network with L ={l}, where l is the learning algorithm chosen by the
user, F = {f1, f2…fq}, where F is the set of activation functions. The weight values are confined to the range from –1 to –1;
that is, -1 ≤wji ≤ 1 for all wji, 1≤(i, j) ≤n, where wji is the weight associated with the connection pointing from unit i to
unit j. The consultant optimizes the induced neural network as follows:
(i) Neurons chosen by the user are initialized with heavy weight connections,
(ii) Lightweight connections are added to the network to facilitate the correction of the user's NN,
(iii) Additional input neurons may be automatically added by the system to the network, to incorporate features,
which do not appear in the network made by the user,
(iv) The consultant initializes the weights and biases if needed (i.e., if not already initialized by the user).
The consultant examines which information is lacking or wrong and sends the result to the user together with the initial
network showing the user’s misconceptions.
4. Implementation and Results
11
12. The experimental results are organized into three subsections. The first part explains the functionality of the learning
environment while the second part illustrates an application example for parameter computation, and the last part
represents a case study of parameter settings for the novice user training. For the experimental purpose, we have used
simple logic functions as problem applications.
4.1 Sample Assistance Process
At the start of the session, the main screen pops up as shown by Figure 8. The user builds his/her network by
assembling objects. The system constrains the network to be physically possible, by graphical representation restrictions
(see figure 10). When the design is satisfactorily completed, the user is required to train the network by specifying a
learning algorithm accompanied with its specific parameters. The environment shows the output data of the network
and the corresponding graph. In fact, to build a neural model, starting with the data, the user constructs the topology of
his/her neural model and specifies its initial weight matrices and learning parameters. Each layer of the constructed
model has to contain vectors of data for input/output units. When the learning algorithm is selected from a list of
options contained within menus, the training session can be started. At the end of the training session, the output pattern
of the data and the corresponding graph are shown to the user as final results.
Since the selection of appropriate parameters is crucial for a fast convergence of the algorithm [8], the user may
impose some constraints on the parameters. It is practical that the user specifies how much correction should be applied
to the weights by tuning the parameters.
Figure 8. Neural Network Design Session
12
13. 4.2 A MLP Weight Computation Example
This section presents a weight computation case study. Suppose that we have the following 3-layered MLP:
Input layer Hidden layer Patterns to be learned: Input
target
Output layer 01 0
Input Values
11 1
Output value
Weight matrix 2
Weight matrix 1 0.35
0.62 0.42 0.81
− 0.17 0.55
Figure 9. Weight change according to input patterns
First, the weight values are set to random values in weight matrix 1 and weight matrix 2. The learning rate of the net
is set to 0.25. Next, the values of the first input pattern (0 1) are set to the neurons of the input layer (the output of the
input layer is the same as its input). The neurons in the hidden layer are activated:
Input of the hidden neuron 1 : 0 × 0.62 + 1 × 0.52 = 0.55
Input of the hidden neuron 2 : 0 × 0.42 + 1 * (-0.17) = -0.17
Output of the hidden neuron 1 : 1 1 + e −0.55 = 0.63
Output of the hidden neuron 2 : 1 1 + e +0.17 = 0.46 (4)
The neuron in the output layer are activated:
Input of output neuron : 0.35 × 0.63 + 0.81 × 0.46 = 0.59
Output of output neuron : 1 1 + e -0.6 = 0.64 (5)
Computation final error : - 0.64
0.32659
By applying the backpropagation algorithm, the weight matrix2 is changed to: , and the weight matrix1
0.79311
0.62 0.42
is changed to .
− 0.21 0.5126
The first input pattern has been therefore propagated through the net. The same procedure is used to for the next input
pattern, but with the changed weight values. After forward and backward propagation of the second pattern, one
learning step is complete and the net error can be calculated by adding up the square output errors of each pattern. By
13
14. performing this procedure repeatedly, this error value gets smaller and smaller. The algorithm is successfully finished,
if the net error value reaches zero or is below some error margin (threshold).
4.3 The Configuration Menu as a Key to Embedding NNs’ Simulators
Most of NNs have a number of learning parameters that have to be appropriately specified. The selection of the
appropriate parameters for large-scale applications is an important experiment problem for a fast convergence of the
algorithm [12]. The user may impose some constraints on the parameters. Consequently, in training novice users, it is
necessary to develop techniques to properly set these parameters. In our system, from the perspective of the user, the
differences that appear between various models are reflected in the NN initialization and configuration menu. As
previously mentioned, this menu depends on the architecture and the learning algorithm selected in the network item
menu. The parameter settings for the backpropagation learning algorithm are as follows: (i) the parameter panel
allows the setting of the specific backpropagation learning parameters, known as learning rate, momentum, and
threshold. (ii) the external activation function menu choice allows the setting of the units to either a sigmoid, or
hyperbolic tangent function. (iii) the threshold input allows a brute force elimination of training examples with the
outputs above or below a certain level. (iv) the train panel allows the selection of the training data, as well as of the
number of training epochs (the number of repetitive passes through the set of examples). (v) from the test panel, the
selection of the testing data set is possible, to test the generalization ability of the NN. As can be seen in the left part of
Figure 10, the user can specify parameters such as external activation function, error threshold, and/or iterations,
learning rate, I/O pairs of values. Figure 10 displays the simulation for a beginner user, simulating the XOR problem.
The architecture of the network is presented on the right side of figure 10. The weights and their dynamic change during
training are displayed next to the network architecture.
14
15. Figure 10: Parameters Setting’s Screen
From these first tests and results, after a first feedback from users, we have concluded that some of the layout
proprieties of the display should be changed, in order to improve the understanding, especially for the beginner user.
For instance, the representation of weights next to the network architecture could reflect the layer structure of the
network, enhancing comprehension and making the following of the learning procedure easier for the inexperienced
user. Moreover, the consultant functions should be refined, especially with regard to the dialogue model between
system (represented by the consultant) and the user.
5. Discussions and Conclusion
In this paper, we have presented the DiscoverNet system, which automatically generates a user-specific neural
network environment, and interacts to support the user in the designing process. The system is based on two main
components: the graphical CAD representation of an initial network and its outputs, and an interactive tutor that checks
the user’s actions during the design sessions. The methodology that underlies the design of the system is based on
constructing assistant script classes to follow the user's design steps; a script defines the sequence of assistance. An
initial neural network is projected in the space of neural network architectures H, only if it matches approximately some
rules of the space H; otherwise the initial neural network cannot be trained. We backed up our approach with
convincing results given by the knowledge based neural network approach. In this latter approach, prior domain
knowledge is used, and complete initial knowledge is not required; the initial knowledge is translated into a neural
network, and then refined into a correct knowledge representation. We presented an object-oriented approach for the
design of an adaptive learning environment for discovery and self-learning learning. The system is extended with a
consultant module that contributes in the learning stage. We focused on the description of the advice process algorithm
in the consultant. As the first step, in order to study the feasibility of such a system, we considered the typical
components and building blocks of the feedforward neural network and Kohonen Feature Map network. We chose
these models because they are the most used, and the equations that describe the models are simple compared to other
models.
15
16. From some first tests and results, we have concluded that some of the layout proprieties of the display should be
changed, in order to improve the understanding, especially for the beginner user. Moreover, the consultant functions
should be refined, especially with regard to the dialogue model between the consultant and the user.
From educational point of view, the system implements the “learning-by-discovery”, “learning-by-doing”,
“exploratory learning” and “reinforcement learning” teaching strategies. In our implementation, discovery learning
means to search for the appropriate neural network for a specific application, based on the information provided by
samples in the knowledge base. Our consultant function is based on gathering NN knowledge in a KBNN meta-
knowledge structure.
This tool is aimed both at students, as well as at in-service workers and researchers in different fields, who need a
quick tutorial of NN design and usage. Therefore, our ultimate goal of DiscoverNet development is to assist senior and
graduate students or engineers in designing NN, and performing NN experiments on different applications.
The DiscoverNet prototype has been successfully tested, however, more extensive in-class tests are needed. The
system is intended for the use in courses at the Graduate School of Information Systems at the University of Electro-
Communications, Japan. From these diverse groups of students we expect to gain valuable suggestions on future
extensions to the current version of DiscoverNet. Embedding additional NNs would be extremely beneficial for the
improvement of the current implementation.
As the system is implemented in Java, it can be easily transferred to the Internet, for long-distance and life-long-
based education purposes. Especially, with the so-called high-capacity, high-speed Japan Gigabit Network introduced
between several national universities, as well as private companies in Japan, this system will be just one of several
distance teaching tools that our laboratory will be offering to the research and education community involved in this
Gigabit project, which is supported by the Japanese Ministry of Posts and Telecommunications, and implemented by the
Telecommunications Advancement Organization (TAO).
We believe that that with this system, we are tackling the problem and serving the need of a systematic, automatic and
adaptive approach to neural network teaching.
References
[1] Goel, A., Lilly project Report: Teaching Design Skills, International Memo, AI Group, College of Computing,
Georgia Institute of Technology, October 1993.
16
17. [2] Belkada, S., Cristea, A., Okamoto, T.:Development of an intelligent simulation-based learning environment to
design and tutor neural networks, Paper presented in the seventh international conference on computer in Education
(ICCE99), vol.2, pp. 291-298, Tokyo,1999.
[3] Cerbin, B.: The course portfolio as a Tool for Continuous Improvement of teaching and learning. Paper presented at
American Association for Higher Education Double Feature Conference: the 8th Annual assessment conference and
the 1st Continuous Quality Improvement Conference, Chicago, June 1993.
[4] Chenoweth, T., Obradovic, Z.: A multi-component non-linear prediction system for the Sp 500 Index.
Neurocomputing, J., vol. 10, no3.3, pp. 275-290, 1996.
[5] McArthur, D., Lewis, M., and Bishay, M.: The Roles of Artificial Intelligence in Education: current progress and
future prospects. RAND, 1700 Main St. Santa Monica, CA 90407-2138, 1998.
[6] Fu, L. et al: Integration of neural heuristics into knowledge-based inference, Connection Science, 1, 325-340, 1989.
[7] Haykin, S.: Neural Networks: A comprehensive Foundation, New York, Macmillan, 1994.
[8] Pazzani, M., & Kibler, D.: The utility of knowledge in inductive learning. Machine Learning, 9, 57-94, 1992.
[9] Stephen, I.G.: Neural network learning and expert system, A Bradford book, the MIT Press Cambridge,
Massachusetts London, England, 1999.
[10]Shank, R.C., et al: Discovery systems. PROC. of AI-ED, Amsterdam, IOS, pp. 237, 1989.
[11]Towell, G.G., Shavlik, J.W., and Noordewier, M.O.: Refinement of approximately correct domain theories by
knowledge-based neural networks. Proceedings of the eighth national conference on Artificial intelligence, pages
861-866, Boston, MA. MIT Press, 1990.
[12]Venkateswaran, R., Obradovic, Z.: Efficient learning through cooperation, in Proceeding world Congress, Neural
Networks, San Diego, CA, Vol.3, pp. 390-395, 1997
17