Working Paper from the EduMAP project by Jyrki Rasku, Paula Kuusipalo and Henry Joutsijoki Revised 26/7/2018: Grant number added

1. Decision Support Systems Review
Working Paper from the EduMAP project
Jyrki Rasku, Paula Kuusipalo and Henry Joutsijoki
The project is funded under the European Union’s Horizon 2020,
Research and Innovation programme, Inclusive, Innovative and Reflective Societies
under grant agreement No 693388
This report reflects only the author’s views and the European Union is not liable for any use that may be made of the
information contained therein.

2. Contents
Decision Support Systems Review ......................................................................................................1
1. Introduction......................................................................................................................................1
2. Intelligent Decision Support System (IDSS) ...................................................................................2
2.1 Research and Implementation ....................................................................................................3
3. Application Domains .......................................................................................................................4
3.1 Computer Platform Control........................................................................................................4
3.2 Industrial Efficiency...................................................................................................................5
3.3 Road maintenance ......................................................................................................................5
3.4 Clinical Diagnosing....................................................................................................................6
3.5 Searching for a Job.....................................................................................................................7
3.6. Search Engines and Recommendation Systems........................................................................7
4. Elements of an EduMAP IDSS........................................................................................................8
4.1 Users...........................................................................................................................................8
4.2 Experts........................................................................................................................................8
4.3 Data Used in Decision Support Systems....................................................................................9
4.4 Models Used in Decision Support Systems ...............................................................................9
4.4.1 Decision Tree.....................................................................................................................10
4.4.2 Neural Networks................................................................................................................11
4.4.3 Support Vector Machines ..................................................................................................13
5. Human versus Computer Program.................................................................................................14
6. Previous Use of the IDSS in the Field of Education......................................................................14
7. Design and Implementation of an IDSS for Adult Education .......................................................15
7.1 Pilot Projects in the Finnish and European Contexts ...............................................................15
7.2 Future Implementation Outlines in the EU Context.................................................................16
7.3 Technical Implementation Outlines .........................................................................................17
8. Conclusion .....................................................................................................................................17
9. References......................................................................................................................................19

3. 1
Decision Support Systems Review
This paper will review the Intelligent Decision Support Systems (IDSS). IDSS is a computer program
that supports stakeholders in their decisions making. The existing IDS systems will be utilized in the
preparation of an IDSS for policy makers, educators and other relevant stakeholders within the
Horizon project Adult Education as a Means to Active Participatory Citizenship (EduMAP) 1.
The aim of the IDSS is to support educational policies and practices to better meet the needs of young
adults at risk of social exclusion. We begin with describing the situation of adult education in EU
countries and outline what an IDSS is. Then we present a selected example of systems that are used
in different occupational fields including education. We continue with describing how data and
different mathematical models are used in existing systems. These models lay the basis for
intelligence in the systems. After pondering positive and negative aspects of computer programs we
outline the design and potential tools of system to be implemented in the EduMAP project. The
current challenges are linked to defining the proper database that contains the core information about
the entire complexity of the issue at hand: vulnerable young people, and various adult education
practices and policies across the different EU Member States.
1. Introduction
In modern societies the role of education is central for socialization of the new generations of citizens.
The basis for productive labor force and economy is laid in initial education during childhood and
youth while adult education has a central role in updating the skills of labor as well as in enhancing
the participation of low educated and marginalized individuals in the society. Improving the EU’s
performance is one of the key objectives of the Europe 2020 strategy, which includes both increasing
the population with tertiary education and decreasing the share of early school leavers2
. Even if both
indicators show steady development towards expected goals, some challenges remain concerning the
equal right to education for many vulnerable groups. The aim of the EduMAP project is to explore
the potential of adult education for enhancing active participatory citizenship in Europe. Adult
education is expected to provide means of preventing unemployment, social exclusion and political
frustration among vulnerable people, but structural, institutional and situational barriers still remain
that exclude vulnerable groups from seizing opportunities for further studies and active political,
social and economic participation.
Previous European wide research3
,4
points to clear social and economic benefits of engaging adults
into continued learning activities. Research also shows that low educational level is a genuine risk of
unemployment5
, exploitation in the labor market6
, health problems7
and crime8
. Even in Nordic
welfare countries, Denmark, Finland and Sweden, with high overall participation in adult education,
the provision is largely targeting the population that already has high education and good position in
the labor market. Despite the high expectations for including the less established population, the
evidence shows that adult education serves best those who have already benefited from educational
goods. In other words, highly educated who have well-established position in the labor market have
easiest access to adult education. Accumulating social and economic resources cater further
opportunities and in turn raise individual expectations that upgrading can open better prospects in the
work life9
.
2
Eurostat news release 57/2014 – 11 April 2014: Europe 2020 education indicators in the EU28 in 2013: Share of young
adults having completed tertiary education up to 37 percent. Share of early leavers from education and training down to
12 percent.

4. 2
Across Europe the participation rate in adult learning programs has not increased during the last
decade, despite political commitment at both European and national levels. Only 4.4 percent of the
approximately 66 million adults with low education attainment participate in learning activities
pointing to crucial need to improve both the design and the implementation of current adult learning
schemes (Education and Training Monitor)10
. To improve the schemes could mean influencing
national and European policy by setting explicit targets and more rigorous frameworks for policy
evaluation. Some suggestions for efficient models include co-financing schemes to support
employers’ investment in adult learning provision, financing of learning programs for disadvantaged
groups, and the alignment of training provision with the identified future skills needs of employers.
However, more accurate information of the mechanisms is still needed to ensure evidence based
policymaking in designing and implementing accurate policies and practices.
In search for potential of adult education for serving the needs of low-educated and less established
populations we need information and understanding of the mechanisms that maintain and accelerate
exclusion of the most vulnerable groups: people with low levels of basic or functional literacy, with
deficient language and cultural skills (ethnic minority groups, foreign newcomers), school dropouts,
injured, handicapped and mentally or intellectually challenged people. In the following we explore
the use and applicability of computer programming to support our quest for more accurate and
sufficient information. The exploration of IDSS logic elements and the available database is needed
to create tools for evidence-based policymaking and inclusive practices that would enhance the
participation of the vulnerable youth in the field of adult education. The purpose of this report is to
give readers an idea about the possibilities and limitations of existing decision support systems for
the use of adult education. For that purpose we present selected examples of commercial and
academic systems and ponder their applicability in constructing the EduMAP IDSS.
2. Intelligent Decision Support System (IDSS)
To understand the characteristics of IDSS a following everyday example might be helpful. When a
person starts to plan a project where his/her own knowledge is insufficient, he/she needs help from
someone else. For instance if a young couple starts to build a house they usually need help from
several different people. Within a certain budget they have to make a series of decisions. From one
expert the couple gets information for building the house foundations while another expert can help
the couple in selecting the proper insulation material. On the other hand, a person could be able to do
a certain task without knowing the details. For example, a hand held calculator can assist a person in
making a decision without knowing the details of an algorithm that actually makes the calculation. In
other words we don’t compromise but trust the outcome. Similar approach is valid for many types of
application domains and it has been used throughout human history.
While the example above concerns only few people, there are situations where larger groups of people
from different fields need support for their decisions. For instance, Kyoto Climate Protocols has set
strict limits for greenhouse gas emissions. For many countries this meant urgent need to change the
energy production methods. On the basis of the latest knowledge the use of renewable energy sources
is highly inefficient compared to the fossil and nuclear energy. This inefficiency originates from the
fact that there are only limited amount of places where renewable energy is available all the time. If
we consider this from global perspective, wind turbines and solar panels can operate only limited
periods of time when installed on regions that have moderate weather conditions. Despite the
limitations of renewable energy forms, they could be used more efficiently if people and governments

5. 3
would invest in them. According to (Yue and Yang)11
policy makers would benefit a lot if they could
have information for how to make renewable energy sources attractive to the investors. Global
warming is an example that reveals complexity of conflicting interest. Nevertheless with accurate and
reliable knowledge, at least some improvement in energy production and environmental protection
could be done.
We are faced with variety of tasks in everyday life that fall somewhere between the examples above
in complexity. However, whenever a policy maker has to decide something, he/she needs the best
possible knowledge to which to base his/her decisions on.
Previously the knowledge was exclusively possessed by experts, but nowadays large amount of
information is available for vast audience in digital form. Certain authors have begun to encode this
digital material into such form that it can be queried for retrieving information. In the simplest form
a www-query can provide enough information for making a decision, but in more advanced cases
certain amount of logic and techniques are required. In early days of Internet various news groups
and chat programs made it easy to get information from other people. These simple techniques made
it possible to set up early peer support groups and similar forums. On the other hand, the information
in such forum can be easily distorted and the participants’ input can turn the nature of conversation
negative. Nowadays there are computer programs in the Internet that do not allow the distortion of
information. If necessary, the information is added to the system in a controlled way. A computer
program that contains a domain specific logic, rules and knowledge is the essential component of an
intelligent decision support system (IDSS).
2.1 Research and Implementation
Research of intelligent decision support systems has begun in the mid-1960s. Daniel J Power has
published a book12
and large number of scientific articles that describe the evolution of decision
support systems. He has also introduced a classification of different IDSS structures in his studies.
The systems that have been implemented have followed quite tightly the existing technologies. In
early systems the machine environments have been described in a detailed manner. For instance, the
name of a single computer with a certain modem and monitor was mentioned. Since the hardware
was earlier significantly more expensive compared to what it is now, a smaller number of people or
companies had afforded to purchase them. Hence, devices such as processors, modems, monitors etc.
were described in detail. A very important aspect related to early decision support systems is that they
operated mainly in closed environments in which only a small number of people had access to. Reason
for this kind of behavior was linked to technological limitations. Internet and WWW have been
available for wider audience only around 20 years whereas intelligent decision support systems have
been examined and implemented over 50 years. Trend between the past and the future has changed.
Present systems are not described so detailed. Most of the current intelligent decision support systems
apply the latest web technologies, so the technical implementations of systems are quite similar.
Research of intelligent decision support systems has continued actively, but the research questions
are more detailed and the whole systems are not usually presented in a single article. This is natural,
because the latest systems are extremely complex. The earlier and current intelligent decision support
systems still share the in fundamental principles, despite rapid technological development. This
reflects well the fact that mathematical models do not change in general. The structure of intelligent
decision support system has not changed over the years even if the technologies have changed.
The starting point in using an IDSS is that we have a single computer or server including the necessary
hardware (hard disk, RAM, processor(s), graphic card etc.) where the intelligent decision support

6. 4
system will be installed. In addition we need a database. In many occasions SQL-based database
solutions are used due to their flexibility, good experiences and good technical support. In human
computer interaction a user interface is required which enables the use of IDSS. Designing the layout
of user interface is always a domain-specific question. The needs of application define what kind of
user interface will be constructed. As field of research the human-computer-interaction examines the
design of user interface. Several technologies and approaches such as multimodal interaction e.g.
gaze movements, haptic devices or speech recognition can be utilized as elements of the use of user
interface. However, design guidelines of user interfaces are not the focus on this report and, thus, we
will not consider them more closely. Finally, an interface is required which connects the user interface
and the database to each other so that a user can do queries from the database and obtain the
knowledge collected to the database. The technology of how to construct the interface depends on the
situation. If we have a web-based intelligent decision support system where user interface may
currently equal to simple web page, we can easily build connection between the user interface and
database using PHP- or Java languages. If we have an IDSS in a closed environment, other techniques
such as Java-based solutions can be used.
Since decision support systems have been under active research for several decades, various
publication fora have been developed. For instance, Journal of Decision Systems by Taylor & Francis
has concentrated on decision support systems from different perspectives. Another well-known
journal focused on decision support systems is published by Elsevier and called Decision Support
Systems. This journal is focused on theoretical and technical issues in the support of enhanced
decision making. Articles in the journal deal for instance, the issues of security13
and algorithms14
.
Also, the user interfaces15
of the systems and data storage issues16
have been addressed.
Theory behind the intelligent decision support systems is quite mature and there are plenty of
implemented systems that are used in a daily basis. In a web site that lists top decision support
systems17
43 software packages are presented and in a site that focus on clinical systems18
23
implementations are listed. With careful study one of these implementations can be applicable in
many different applications, not just one.
Numerous decision support systems are targeted to deal with medical issues and have been used
sufficiently long time, have been evaluated for instance, in academic review papers19, 20
.
3. Application Domains
Majority of intelligent decision support systems are targeted to business purposes, but the list of new
application domains is growing fast. Working systems can be found for instance in the field of
Computer Platform Control21
Industrial Efficiency22
, Clinical diagnosing23
, Road Maintenance24
,
Factory maintenance25
, Insurance Risk Analysis26
, Investments 27
, Waste Disposal and Recycling28
and Justice29
just to mention few. Next we review some systems that are used in different application
domains. We begin with commercial systems and in chapter 6 we deal with systems that are targeted
to educational purposes. While the domains are different, the basic idea of an IDSS remains the same.
They try to act like a human expert.
3.1 Computer Platform Control
IBM is focused on building supercomputers and large scale servers. They have also participated to
the research of intelligent decision support systems almost from its beginning. As one might
anticipate, the data flow through a supercomputer is huge. The data contain information about the

7. 5
users, which services they used and which configuration the computer used. Data logging is an old
concept, but nowadays vast log data is almost impossible to utilize without dedicated software.
Although large part of the log data does not contain any interesting patterns, certain patterns contain
useful information for the root user provided that the patterns can be found. Patterns that differ from
normal log patterns can reveal system misuse or early marks of incoming hardware failure. IBM
application system Tivoli [21] provides a set of decision support systems that are designed to mine
information from log data. Use of Tivoli system can support in making decisions with system
performance, resource availability, maintenance, workload prediction, workload balancing and cost
related questions.
Figure 1. Maintenance of supercomputers is impossible without a system that prioritizes the tasks.
(tgdaily.com)
3.2 Industrial Efficiency
ABB company is famous for their selection of electrical motors. Nowadays they also offer DSS800
decision support system22
for improving the output of a paper mill production line. The system aims
to predict the need for maintenance and to reveal the most probable problem areas. For instance,
successful prediction of exact time for replacing the paper mill wire can yield considerable savings.
Replacement too early is not reasonable while too late replacement can cause an unpredictable halt
to the production. In addition to maintenance scheduling DSS800 provides accurate follow up reports
that support decision making of adopting new best practices and incremental performance
improvement methods.
Figure 2. A question to an expert. ABB DSS800 brochure.
3.3 Road maintenance
Vaisala is a well-known Finnish company that has been manufacturing reliable sensors and

8. 6
measurement devices for decades. They have also built an intelligent decision support system that
combines information from a sensor grid assembled near a road system. Different sensors measure
weather related variables that help predicting, for instance, snow fall and black ice formation. The
system contains also road cameras in certain locations that help visual evaluation of weather
conditions. Sensor data are combined in the system and the extracted information is used in predicting
the weather conditions in different parts or road system. Based on the prediction, a correct number of
plow trucks can be sent to the right locations. Similarly correct amount of de-icing material can be
delivered only to such locations that need de-icing. Idaho Transportation Department30
has managed
to reduce road maintenance costs and number of traffic accidents using the system.
Figure 3. Correct timing in road maintenance increases safety and helps reduce material costs.
3.4 Clinical Diagnosing
Siemens is a world-wide company that is primarily known from their automation solutions. They also
provide a wide selection of different health care related products and services. These services contain
two decision support systems, Protis23
and Prisca systems23
.
Protis system optimizes the laboratory workflow and combines multiple blood test results from
patients. Protis combines the results of a single patient on a document in a form that a physician can
easily interpret. The document is designed to support a physician in making diagnoses and planning
treatment.
Prisca system is targeted to predict a prenatal risk and help a physician to make informed decisions
when taking care of a patient. Input data for the Prisca system contains normal ranges for different
weeks of gestation, multiple correction factors based on patient demographics, lab test and ultrasound
results. This data is used to predict the age-related risks, biochemical risks for trisomy 21 and trisomy
18, combined risks with nuchal translucency for trisomy 21 and trisomy 18 and neural tube defect
risk. In addition Prisca system provides various patient report formats.

9. 7
Figure 4. A question to an expert. (Siemens Prisca brochure)
3.5 Searching for a Job
Foredata31
is a Finnish service provider that has a DSS with a user interface foreammatti.fi that
provides information about open job positions. In addition, they have a data base that contains
combinations of different qualifications and diplomas that are required in variety of fields and
occupations. For instance, an electrician using the service can see that with similar qualifications
people are working as a salesman or machinist. This type of service can help a person to decide if
he/she can apply a job for which he/she does not have a formal education. Foredata has also a service
that predicts open positions in different fields near future. This service helps a person to decide
whether he/she should apply for a course if needed in order to get a job.
3.6. Search Engines and Recommendation Systems
Previously intelligent decision support systems were run on a single computer or small local area
networks. Nowadays such systems are wide spread. Web users usually do not even know that their
actions may be tracked by a recommendation system, while reading web pages. Web browser presents
commercials of products and services to the user. These commercials are ultimately generated by an
underlying recommendation system that is one form of an intelligent decision support system.
Amazon book store32
among other web stores has used recommendation systems for a long time.
When a person searches for a book from Amazon web store, he/she is notified what other books other
users have bought in addition to the book in the web search. Google Maps33
can also be considered
as an intelligent decision support system, because it provides an opportunity to plan routes between
locations.
The amount of data and information is growing exponentially. We are facing information flood every
day and we are not able to process all the available information. The problem is, how to distinguish
between the useful and non-useful information? The way of life in our modern society is hectic. We
demand the necessary information immediately and it must be obtained easily. Hence, a common
practice is to go to Internet and to give a keyword to Google and to select the first link what the search
engine offers or to seek the Wikipedia for the needed information. These applications have become
the everyday IDSS for us. However, in many cases the first website/news group/discussion forum

10. 8
does not give us the necessary information. This leads to a situation where people get frustrated and
start to click almost randomly different links in order to find what they want. When considering the
bigger picture, we notice that a lot of information is available but it is highly scattered. In other words,
we have huge amount of available information but we actually know less about it. Intelligent decision
support systems offer one way to control the scattered information. We can collect the scattered
knowledge and information into IDSS and offer it in a compact and easily understandable form for
the user.
4. Elements of an EduMAP IDSS
EduMap is a project about active citizenship. This means that our goal is to search for means that can
prevent young adults to fall outside the societal activities. While it is not straightforward to define
exactly what active citizenship means, at least two efforts can be made to support young adults
towards it. First task is to find out the reasons that prevent people in getting education they need.
Second task is to find out the methods that better aligns the need and supply of adult education. Active
citizenship can also be seen as a security factor. When young adults are engaged in the society and
feel that they have their lives in balance, they are not easily drawn into dangerous groups or otherwise
behave in destructive manner.
4.1 Users
Potential end users of EduMAP IDSS are providers of adult education and policy makers. The
purpose of an IDSS is to support the users in their decision making processes such that different
vulnerable groups could be taken into account in planning and implementing educational practices.
A useful IDSS can provide support to the user in such cases where his/her own knowledge is
insufficient. End users of an IDSS should be involved into system design, because only they can
present such questions that should be answered.
Users of EduMAP IDSS can be found at every level of education. A teacher, a manager of a single
school, a head of education at municipal level and authorities of education in national level. Certain
joint educational goals are implemented also at European and international level. The users in each
level need information that help them to make decisions. The information needed in each level is
different, but we should start from the lowest level, because from there we can find the root level
problems. Partial solutions of lower level problems can then “bubble” up and help upper levels to get
the information they need.
The set of important questions have to be collected from the people that are the potential users of the
system. It is not reasonable to have answers to all possible questions, but there must be a subset in
the question list that is common to most people that present the questions. The questions that are
considered to be important to the end users are answered by domain experts. Finally the mapping
between questions and answers is converted to a computer program.
4.2 Experts
In order to be useful, an IDSS must be able to provide answers to the question that the end user of the
system presents. In EduMAP project the experts are researchers of education theory, sociology and
computer science. However, expertise in relation to vulnerability in society and for finding the best
solutions of educational practices is not as easily achieved as expertise in the previously mentioned
industrial, physical or medical examples. The signals of a computer, measuring of weather conditions
and observations based on laboratory tests or previous users’ choices are easier to observe and analyse

11. 9
than the extent of human activity that has multiple constructions. People interact and participate in
relations that can be described as communicative practices and societal conditions. The experts in our
case are the vulnerable individuals themselves, teachers, administrators and various authorities.
Although, there are no generic computer systems that can answer all the questions for which answers
can be found, it is still possible to construct a system that can answer to the limited amount of the
questions that have already been answered by domain experts. Depending on the system, the answer
is usually a plain number that is decoded to human readable form.
4.3 Data Used in Decision Support Systems
Depending on the application domain, the data can be distributed in several different places. However,
most of the decision support systems use data in the form of an observation matrix. Rows in the
observation matrix are the individual observations and columns are the variables that are used to
encode the domain specific information.
M=[
𝑥11 ⋯ 𝑥1𝑝
⋮ ⋱ ⋮
𝑥 𝑛1 ⋯ 𝑥 𝑛𝑝
]
Matrix M could contain, for instance, frequencies of words in text documents. The first row contains
the word frequencies of document 1 while the last row contains the word frequencies of document n.
In this case we are interested only in p words that are searched for over all documents. One row in
the observation matrix is also called a feature vector. The content of this vector should be selected
carefully such that it contains only useful and not redundant information.
This kind of text mining was tried out for example in our search of the addressed target groups and
implemented measures of the ongoing Youth Guarantee- program that aims at providing solutions for
the young that are not participating in employment, education or training. Based on the experiment
we could trace the available databases containing information of development projects or best
practices in the field of adult education as one possible source of information for the IDSS.
4.4 Models Used in Decision Support Systems
Machine learning is a field of research that studies different ways to find structured patterns from data
sets. It is usually divided into two categories; unsupervised and supervised learning. Unsupervised
methods operate purely on data and they aim to divide datasets into such subsets that have some
common property inside in a group. On the other hand, there is certain property that can tell the
subsets apart. Supervised machine learning methods involve an expert that has labeled the data such
that the group of each observation is known. Machine learning in supervised mode means the process
of finding such model parameters that according to a certain criteria find the mapping between
observed variables and labels in the data. Supervised machine learning scheme is usually used in
building the intelligent decision support systems. However, unsupervised methods are usually used
in selecting important variables and visualization of data.
A running decision support system is a complex software entity with a rich user interface and data
store. Its intelligence is encoded into a set of algorithms that are realizations of selected mathematical
models. The purpose of the algorithms is to extract the information contained in the data and present
it to the user in an understandable form for a human. Model structure can vary a lot. Typical models
used, for instance, are decision trees34
, artificial neural networks35
and support vector machines36
.
Sometimes it is sufficient just to find the “most similar” case from the set of observations. This can

12. 10
be found using the nearest neighbor searching rule37
. The similarity measure between two cases is
often Euclidean distance, while there are plenty of other measures. For instance, if we consider which
of documents a or b is closer to document c, we calculate distances d(a,c) and d(b,c) and select the
smaller distance. When calculating “distance” between text documents x and y, we usually use cosine
measure.
𝑑(𝑥, 𝑦) = 1 −
𝒙 𝑻 𝒚
|𝒙||𝒚|
if d=0 the documents are identical. For simplicity documents can be considered as p dimensional
vectors that begin from origin. If they point to the same direction, they are the same.
4.4.1 Decision Tree
A decision tree is an intuitive model. It divides an original observation matrix such that the leaf nodes
of the tree contain observations that belong to the same group. Internal nodes contain the rules for
successive divisions for the data of the observation matrix. A classic example of a decision tree is
based on a history about weather conditions and whether a person has played tennis or not. Given the
observations in the following table presented in Figure 5, we can construct a decision tree that presents
the observations in tree form.
Figure 5. Quinlan’s example of a decision tree.
The first column in the observation matrix contains the variable values for outlook. From the
observation matrix, we can see that the player has always played when the outlook has been overcast.
This variable can split the observation matrix such that one part contains cases where the player
always plays and another part that still contains some uncertainty. In sunny or rainy days the player
has used one additional variable in decision making. In sunny days the player has played only, when
the humidity has been normal and in rainy days the player has played only when the wind has not
been present. Decision trees have been used for instance in [38]. Research around decision tree has
been active for several decades and numerous variations on decision tree algorithms have been
presented. The most famous and most used algorithm is Quinlan’s ID339
. Quinlan developed ID3

13. 11
algorithm further and as a result he invented C4.538
and C5.039
algorithms. Another slightly different
approach for decision trees is classification and regression trees (CART) algorithm which is one of
the top 10 algorithms in data mining according to Wu et al38
. There are several reasons behind the
success of decision trees. Firstly, is the simplicity of decision trees. Example given in Fig. 5 shows
how easy is to understand the structure and the principle of decision tree. Every inner node in a tree
corresponds to some variable in a data and the leaf node describes the predicted class label for the
feature vector (row in an observation matrix) which is classified. Secondly, visualization is another
big advantage for decision trees. We can always form a two dimensional presentation for decision
tree even if the number of variables in a dataset would be high. Thirdly, decision trees can handle
missing values. This is a very important asset in real-world application which encourages us to use
decision trees. We have noticed that datasets, for instance, in the field of adult education are scattered
and the quality of data varies highly. In many cases variables contain large amount of missing values
which make the analysis more difficult to do. Fourthly, from the computational point of view decision
trees are very efficient to use compared with many black-box algorithms which require evaluation of
highly sophisticated optimization algorithms. Since we are living now in the era of “big data”,
efficiency of the algorithm is one crucial point which must be taken into account.
Decision tree algorithms produce only one tree which is then used in classification or regression tasks.
However, one tree does not always give satisfactory results. Breiman who developed CART
algorithm introduced in 2001 a new method called Random Forests40
. Random Forests extends the
concept of decision tree algorithms and belongs as a method to a class of ensemble learning. In
Random Forests method (can be used both classification and regression tasks) a set of individual
decision trees is constructed and each one of the decision trees is built using different variable subset.
Final decision related to feature vector to be classified (also known as test set instances) is made
according majority voting principle. In other words, every decision tree in a forest gives a predicted
output for the test instance and where the most frequent result is selected as a final output. Overall,
Random Forests method has gained wide popularity among practitioners and researchers and used in
various applications.
In science an ultimate goal has always been to model our surrounding environment and actions of
human beings. One of the most interesting, challenging and researched subjects are human brains and
their functionality. How do we learn? How do we process information through our senses? How do
we make decisions? How do we solve problems? These are just some examples of the questions
which have been examined widely in different fields. From the computational point of view imitation
of human problem solving has interested researchers for several decades. The history on this subject
goes back to 1940s when the first attempts to create mathematical models called neural networks
were developed. However, the real breakthrough for the neural networks happened in the 1950s when
Rosenblatt introduced an algorithm called perceptron which stabilized its footprint to the theory of
neural networks.
4.4.2 Neural Networks
A neural network is a mathematical simplification of interconnected human brain cells. A single
neuron (cell) gets input from several nerves and creates an output depending on the pattern in inputs.
Neurons are arranged to three or more layers such that the first layer receives the input vector and the
last layer provides the output. The layers after the first layer take care of the required processing.
Resulting outputs of the hidden layer are used as an input for the neurons in the next layer. Figure 6
presents an illustration of an artificial neural network. Numbers of layers and neurons depend on the
intended application. Structure like given in Fig. 6 belongs to the group of multi-layer perceptron
(MLP). MLP neural networks41
are widely examined and used in various applications such as image
classification42
. Brains are learnable and adaptive organ. Similarly, MLP can learn and adapt based

14. 12
on the inputs what the network receives. By this means network learns patterns from the input data
and can recognize if similar kind of inputs are given again. The actual learning process of an MLP
network returns to mathematical algorithms. In the literature a large number of learning algorithms
have been presented and in one of our article42
15 different methods were investigated in image
classification task.
Figure 6. Illustration of an artificial neural network from tex.stackexchage.com.
Inputs in Figure 6 are the components of single row from observation matrix. If we interpret the
observations as text documents as before, an artificial neural network can be used to classify the text
documents into several different classes. The classification rule can be, for example, the topic of a
document.
Although we have discussed now only on MLP networks, they are not the only neural network
techniques developed. Other commonly encountered neural network techniques are Probabilistic
Neural Network41
, Radial Basis Function Networks41
and Recurrent Neural Network (e.g. Elman
Neural Networks 43
and Jordan Neural Networks43
). The most recent interest in Neural Networks is
called Deep Learning. Deep learning43
approach has taken neural networks into a new level. Deep
learning architectures are highly complex (more complex than MLP networks are for instance) and
from the computational point of view relatively time-consuming. Nevertheless, performance of deep
learning has been excellent in many situations. For instance, in 2016 a computer program called
AlphaGo45
played against South-Korean 9-dan professional Go player Lee Sudol in a five-match
game of Go. After five games AlphaGo won Sudol with a result of 4-1 and doing a historical result
since never before a computer program has won a 9-dan Go professional without handicap. This
example shows how computational neural network methods have developed significantly and their
learning process have ameliorated to the extent that they are now at the same time challenging the
capabilities of human brains. A common feature for all aforementioned neural network methods is
that they represent supervised learning paradigm. Neural networks can be used with another learning
paradigm called unsupervised learning. One of the most famous unsupervised neural network method
is Self-Organizing Maps44
(also known as Kohonen maps or simply SOM). Deep learning methods
and MLP networks can be considered as black-box techniques in which understanding of the relation
between neural network structure and the nature of the data can be difficult to illustrate. SOM instead
is a method where visualization of the data is an essential part. In SOM data is mapped to a two

15. 13
dimensional lattice where different parts of the map represent similar groups of instances. SOM uses
color coding to present which parts of the map contain more instances than the other. Color coding
helps again in understanding the nature of the data.
4.4.3 Support Vector Machines
If deep learning is a state-of-the-art neural network method, there are also other relevant techniques
which can be applied based machine learning tasks for adult education. Support Vector Machines
(SVMs)36,46,47
are a machine learning methods which can be applied to both classification and
regression tasks. SVMs obtained the final mathematical formulation in the 1990s although the ideas
behind SVMs go back to researches from the 1960s. The original purpose of SVM was targeted to
solve problems where only two classes are present. However, they can be easily applied to multiclass
problems as well. The extensions for multi-class cases have a common basis since they are all
constructed by combining several binary classifiers. Combining individual binary SVM classifiers
may be done using several approaches such as tree-based, graph-based or using error correcting
output codes. Lorena et al48
have written an extensive review on various approaches how to combine
binary classifiers in multi-class problems.
Support vector machines are widely used method in pattern recognition. They are targeted to solve
such problems where only two classes are present. However, they can be easily applied to multiclass
problems as well. The idea behind support vector machines is to search optimal “hyperplane” that
maximally separates two classes in feature space. We remember that the feature space is constructed
by the columns of an observation matrix. The intelligence in SVM lies in kernel trick by which a
nonlinear classification task can be reduced to a linear classification problem in a higher dimensional
space. Figure 7 presents an over simplification of a support vector machine classifier in a two-
dimensional case. In real world data the area of squares and circles is overlapping, but the idea is
same within a certain criterion.
Figure 7. Squares and circles are separated with support vectors that yield maximum margin
between the classes. (rsif.royalsocietypublishing.org)
Previously in this report, we mentioned the concept of nearest neighbor searching classification that
needs only a distance measure and data to work. More advanced methods like decision trees, artificial
neural networks, support vector machines and many others need to be “trained” before they can be
used. In addition to training, the models have to be evaluated whether they are good for the job they
are targeted to or not.

16. 14
Model training means the search of model parameters. Training data is part of the whole observation
matrix that is usually divided into subsets with cross validation process39
. One part of the data is used
in model training and the rest of the data are used to measure the model performance. Using the model
means that we present a new observation for the model and try to predict its class using the model
parameters. Our intelligence in this context is that we use our best knowledge of previous
observations in predicting the class of new observations.
All models presented above are mathematical formulations of wide range of applications. They can
be considered as multidimensional functions from feature vectors to classification. If we consider this
from the point of intelligent decision support system, the mathematical model is created by a
computer, but the selection of feature vector components and the respective classes to which the
vectors are mapped is the knowledge of domain experts.
5. Human versus Computer Program
Humans are capable of making complex designs in different branches. This is something that a
computer cannot do. On the other hand, a computer can make repetitive tasks extremely fast and the
result is always the same.
Humans get easily tired with repetitive tasks and the result may vary. For instance, if the task is to
make a billion multiplications of decimal numbers, computer does this within a millisecond without
any errors. This task is clearly such that a human would not even try it. There are many repetitive
tasks in everyday lives that are still made by humans, although a computer could fit better to the job.
For instance, in Finland the handling process of a construction permit may take for a year. A computer
could do this task within a nanosecond or so. Of course there are plenty of special cases that have to
be dealt by humans, but majority of cases could be dealt by a computer. Another advantage of
computer program is that it does not have to rest at all. A program can be run all the time and it can
serve hundreds of users simultaneously, while a human expert can only serve one client at the time.
Usefulness of a computer program is usually evaluated on the basis how well it does a job it is
implemented for. Terms validity and reliability are often used in this context. Validity is used to
measure how well a program solves the problem it is designed for. Reliability depicts the extent how
consistent the results of a program are given the input data. Low validity in complex problems mean
that the implemented program gives only a partial solution to the problem or solves entirely wrong
problem as a worst case. Low reliability mean that a “tiny” change in an input cause “large” change
in respective output.
6. Previous Use of the IDSS in the Field of Education
Although in the field of education the use of IDSS is still limited, some examples can be found. For
instance, authors have proposed a system that can be used in predicting the educational capacity
utilization over faculties in large universities trying to solve the problems of fair allocation of
resources54
. In a University consisting of several faculties that can further be divided into smaller
units, students can select between major and minor courses over unit and faculty borders. The
variation in numbers of students and their course selections influences on the demand of teaching
staff required to meet the needs. Previous research54
clearly states that certain units should hire more
teachers while other units could provide more courses to students. The system can also be used in
planning new courses that require personnel from many faculties. This type of system could easily be

17. 15
modified so that it could be used in scheduling workloads for production lines where setups change
a lot.
Another example is a situation where the selection of best possible university may appear problematic
for a new student. Certain general ranking lists exist, but they are usually too superficial from a
student’s point of view. Authors have applied a personalized ranking method for selecting a university
in Britain55
. This idea could be used in broader context, too. For instance, students that use a
wheelchair could find the system very useful if information about the structure and accessibility of
school environment would be provided.
An example from Finland is the centralized application system for secondary and tertiary education.
The national service portal Studyinfo56
contains the information and means that is needed for applying
to initial secondary and tertiary education. The portal contains information on adult education in
national languages Finnish and Swedish. The development of the portal is targeted to broadening the
online service to a user interface that serves both the students’, service providers’ and administrators’
needs. As part of the ongoing national program for eGovernment in Finland, the National Board of
Education is in charge of producing services and user interface to enhance digital solutions and
services for all potential learners and organizers of education. The portal, Studyinfo, already provides
information on the educational system and a search engine that can be used for finding study options
in Finland. The portal links to various service providers’ webpages. One of the future objectives is to
provide data-transfer protocol from the organizers’ client register to central register to avoid self-
reported information that is currently executed every six months or once a year depending on the type
of education. However, primary aim is to create a qualifications register.
7. Design and Implementation of an IDSS for Adult Education
In our quest to find answers to the question of what policies and practices are needed in order to
enhance participation of the vulnerable youth to adult education, we are facing a complex field
of interests, practices and policies at various levels of individual, institutional and national activities.
For the purposes of IDSS development we have begun to tackle this complexity by identifying,
finding and processing the available data. First, we searched for sources of information concerning
the target population, second, we pinpointed the data sources concerning the provision of educational
activities, and third, we experimented with available data to explore the conditions that allow
participation of the vulnerable youth in adult education. As a starting point we applied access to both
Eurostat microdata (Adult Education Survey, Labour Force Survey) and Finnish register data.
7.1 Pilot Projects in the Finnish and European Contexts
In our quest for testing the information value of various data for the purposes of EduMAP IDSS, we
identified the public sector authorities in Finland who gather administrative data on participation in
education and the beneficiaries of various public subsidies. The administrative register data is built
on a unique personal identification number indicating the individual participants and service users.
These national statistics are derived of the register data and are largely used in policymaking at
national and European level.
Next we contacted the institutional actors that administrate national register data consisting
information on young people in various vulnerable positions. Several institutional actors produce
relevant register data and statistics that indicate challenging life situations. These life situations are
possible causes of vulnerability and could be in relation to participation in education and hence in
working life and other spheres of active citizenship. The institutional actors whom we contacted in

18. 16
search for the data were the National Institute for Health and Welfare (THL) 49
, the Social Insurance
Institution of Finland (Kela) 50
, the Ministry of Employment (TEM)51
and Criminal Sanctions
Agency52
. They are respectively the main institutional actors of allowances like income support;
various benefits covering particular life situations that allow financial support like child home care,
single parenthood, handicap, rehabilitation; unemployment and participation in labor force training;
and lastly, imprisonment. In addition, Statistics Finland51
as the key institutional actor carrying out
national surveys and accumulation of register data was contacted. In Finland the various institutional
actors all apply different procedures and conditions for the use of data that we needed to follow to
gain access to microdata. With these data we intend to look for intersection of different vulnerable
situations and test data for indicating dependency and conditions of vulnerability vs. participation in
adult education.
In addition we are planning to conduct data mining on education portals and education providers’
webpages in order to seek the target groups’ visibility and how entrance conditions are presented.
Information about the available courses and study programs in the field of adult education is widely
available in internet webpages and portals. Through internet, text mining can be applied with certain
attributes to all available information of provision and entrance conditions of education. The
information concerning access to and ways of performing education and the applicants’ needs are the
kind of information that could be processed by the EduMAP IDSS.
Moreover, many adult education funding schemes at European and national level uphold databases
including information on the funded projects and best practices extracted from them. As a pilot we
will conduct data mining to ESF –project descriptions’ national database. We focus especially on the
projects in Youth Guarantee, a European wide program that aims at providing good quality education
or job opportunities to all unemployed youth under 25 years. We will conduct a search of the target
groups, actors and planned educational measures to test how the databases could be investigated by
IDSS methods and how the information could be further exploited in the decision making.
Using the register data, we will explore those situations of vulnerability in relation to participation in
adult education that can be traced with data collected for administrative purposes. However, all
situations of vulnerability do not become authenticated in these statistics. Human life, in other than
biological existence is a complex entity that entails human activity with all its rational and irrational
aspects. Moreover, institutional practices have been developed in various historical situations to serve
the particular interests and needs of national economics within different welfare state regimes. To
overcome the shortcomings of data concerning the vulnerable youth and adult education, we will
conduct empirical research and further data gathering in all European member states. Our intentions
during the field research phase is to explore in detail the situations of various vulnerable individuals
and conditions that have ensured their successful participation in adult education. These data will
contribute to formulation of the core information in the IDS system database.
7.2 Future Implementation Outlines in the EU Context
The available data has restrictions in describing vulnerability and covering the range of vulnerable
situations in all European Union Member States. The categories used for administrative purposes do
not necessarily equal to individual life situations nor do they cover the range of conditions of human
experience of vulnerability. For instance, many conditions that may hamper participation and active
citizenship are invisible. Minor deviation of the norm or “normality”, as in case of minor handicaps,
illnesses and other situations can turn into experienced barriers depending on the conditions of
society. It is not possible to find exhaustive defined categories that would cover all existing cases of
vulnerability. It is obvious that the available register and survey data is not sufficient in covering the
total variation of life-situations and experiences of individuals, realities in adult education and societal

19. 17
norms.
In addition, the ways of conducting data gathering vary in different countries as do the practices and
policies that provide or hamper opportunities of participation. In our attempt to overcome some of
the shortcomings of existing data we will experiment with different data sources and address the
different stakeholders while conducting empirical data gathering on practices that work well for some
vulnerable individuals in certain societal context. The good practices for purposes of finding the
solutions that have proven successful will be searched in all the Member States and even outside
Europe. The system intelligence will be built on these good practices.
In our quest for finding solutions for IDSS that could support decision making in the complex field
and different levels we are facing the real life situations in various regional circumstances that are
conditioned in historical contexts. Furthermore, at European level communication in the variety of
languages adds to complexity and has to be taken into account.
To begin with we have formulated the relevant questions for the IDSS at various action levels as
follows:
(1) What structural factors are topical for organizing appropriate adult education for vulnerable
youth?
(e.g. societal conditions and labor market conditions in EU28); (2) What policies and practices are
topical for organizing appropriate adult education for vulnerable youth (e.g. education policies at
national and European levels, educational practices in the Member States); and (3) What individual
factors are topical for organizing appropriate adult education for vulnerable youth (e.g. qualification
of teachers, capability of learners, motivational factors, employability of vulnerable youth)?
7.3 Technical Implementation Outlines
The implementation of an industrial strength IDSS like presented earlier in this paper is a huge
project. Fortunately the present software tools and programming environments can save a lot of work.
A proper IDSS can serve thousands of users at the same time. This requirement can be fulfilled, for
example, using such server applications that follow Oracle’s Java EE757
specifications. Use of Java
EE7 specifications helps the user to focus only on developing the application logic. Technical details,
like data base connections are taken care by the Java EE7. One such a server is Payara 58
that can be
used with NetBeans59
integrated development environment (IDE). With these tools we can easily
implement a web based system that can use a data base and transfer input from a user to the server
and in the opposite direction. Application logic can be implemented using R-server. R environment
contains wide selection of machine learning algorithms that can be combined to fulfill the possible
custom needs. If necessary, these algorithms can later on be rewritten using C or C++ if necessary.
8. Conclusion
IDSS is a computer program that emulates a human expert. The domain knowledge is mined from
the data using machine learning algorithms after the data has carefully been labelled by humans. A
working IDSS is ultimately used in predicting the outcome of a data unit that is not used in training
the system. This process resembles the way humans make predictions. While a computer learns from
the data, humans learn from experience, trials and errors. The learned patterns will finally be used in
making decisions in everyday lives.

20. 18
Research of intelligent decision systems continues, but it has already provided information enough
for implementation of programs that are found useful. Although many implementations have been
executed in academic projects and have been discontinued, big companies have adopted the ideas and
provide many commercial IDSS:s that are in regular use. Such companies as Intel, ABB, Siemens
and IBM among the others have their own IDSS products for sale. A common feature for all reviewed
systems is that they combine information from several aspects of a problem at hand. Although, the
system logic could be easy, it would be impossible to a human being to remember all aspects at the
same time in order to make a decision. Clearly the use of computer is a reasonable choice in dealing
with complex and data intensive task as is our goal in better aligning the meet and need of adult
education.
Our task in the field of adult education is to find practical, user friendly IDSS solution that will support
policy makers in finding educational policies and practices appropriate to the need of vulnerable
youth. Despite the huge challenge and complexities in relation to proper database, addressing and
facing the various levels and scope of the activities and facing challenges in relation to the variety of
European realities it is nevertheless a task worth pursuing.

21. 19
9. References
1 http://www.uta.fi/edu/en/research/projects/edumap/index.html
3 BeLL – Benefits of Lifelong Learning (2014) Final Report. http://www.bell-project.eu/cms/wp-
content/uploads/2014/06/Final-Report1.pdf
4 Blossfeld, Hans-Peter, Kilpi-Jakonen, Elina, Vono de Vilhena, Daniela and Buchholz, Sandra
(eds.) (2014) Adult Learning in Modern Societies. An International Comparison from a Life-course
Perspective. UK: Edward Elgar Publishing.
5 Niemi, Helena, Ruuskanen, Timo and Seppänen Tarja (2014) Osallistuminen aikuiskoulutukseen
vuonna 2012. Helsinki: Tilastokeskus.
http://www.stat.fi/tup/julkaisut/tiedostot/julkaisuluettelo/ykou_aku_201200_2014_12393_net.pdf
6 Arajärvi, Pentti (1994) Oikeus sivistykseen: sivistykselliset perusoikeudet, oppivelvollisuus ja
oikeuksien toteutumisen takeet. Helsinki: Lakimiesliiton kustannus.
7 Koskinen, Seppo, Castaneda, Anu E, Koponen Päivikki, Rask, Shadia (2012) 15.2 Koulutustaso
ja asuinalue In Castaneda, Anu E. et al (eds.) Maahanmuuttajien terveys ja hyvinvointi. Tutkimus
venäläis-, somalaialais- ja kurditaustaisista Suomessa. Helsinki: Terveyden ja hyvinvoinnin laitos.
8 Aaltonen, Mikko & Kivivuori, Janne & Martikainen, Pekka (2011): Social Determinants of Crime
in a Welfare State: Do They Still Matter? Acta Sociologica 54(2): 119–138
9 Kilpi-Jakonen, Elina, Vono de Vilhena, Daniela, Kosyakova, Yuliya, Stenberg, Anders and
Blossfeld Hans-Peter (2012) The Impact of Formal Adult Education on the Likelihood of Being
Employed: a Comparative Overview. Studies of Transtition States and Societies. Vol 4:1. pp. 48-
68.
10 Education and Training Monitor (2015) Luxembourg: Publications Office of the European
Union.
http://ec.europa.eu/dgs/education_culture/repository/education/library/publications/monitor15_en.p
df
11 C. Yue, G. Yang. Decision support system for exploiting local renewable energy sources: A case
study of the Chigu area of southwestern Taiwan. Energy Policy, 2005, pp. 383-394.
12 D. Power. Decision support systems: concepts and resources for managers. Quorum books,
Westport, 2002.
13 K. Muralidhar, R. Sharaty, H. Li. Secure attribute sharing of linked microdata. Decision Support
Systems, 2016, pp. 20-29.
14 H. Chi et al. Machine learning and genetic algorithms in pharmaceutical development and
manufacturing processes. Decision Support Systems, 2009, pp. 69-80.
15 C. Sankar, N. Ford, M. Bauer. A DSS user interface model to provide consistency and
adaptability. Decision Support Systems, 1995, pp. 93-104.
16 N. Gupta, G. Dutta, R. Fourer. An expanded database structure for a class of multi-period,
stochastic mathematical programming models for process industries. Decision Support Systems,
2014, pp. 43-56.
17http://www.capterra.com/decision-support-software/
18http://www.openclinical.org/dss.html
19 A. Garg et al. Effects of Computerized Clinical Decision Support Systems on Practitioner
Performance and Patient Outcomes. American Medical Association, 2005, pp. 1223-1238.

22. 20
20 T. Bright et al. Effect of clinical decision-support systems: a systematic review. Annals of
Internal Medicine, 2012, pp. 29-43.
21 https://www.ibm.com/software/tivoli
22 http://www.abb.fi/industries/db0003db001873/a939570fb3554e7ec125736f0026b7d4.aspx
23 https://www.healthcare.siemens.com/diagnostics-it
24 http://www.vaisala.com/en/roads/products/forecastinganddecisionsupport/Pages/mdss.aspx
25 http://www.intel.com/content/www/us/en/embedded/software/decision-solutions/trend-analytics-
software.html
26http://www.sas.com/en_us/software/risk-management/firmwide-risk-solvency-ii.html
27http://www.ortec-finance.com/Private-Wealth/Investment-Decision-Support.aspx
28 H. Bargava, C. Tettelbach. A web-based decision support system for waste disposal and
recycling, Computers, Environment and Urban Systems, 1997, pp. 47-65.
29 J. Zeleznikow. Using Web-based Legal Decision Support Systems to Improve Access to Justice,
Information and Communication Technology Law, 2002, pp. 15-33.
30 https://itd.idaho.gov/
31http://www.foredata.fi/
32 www.amazon.com
33 www.google.com
34 J. Quinlan. Induction of Decision Trees, Machine Learning, 1986, pp. 81-106.
35 C. Bishop. Neural Networks for Pattern Recognition, Oxford University Press, New York, 1995.
36 C. Cortes, V. Vapnik. Support-Vector Networks, Machine Learning, 1995, pp. 273-297.
37 T. Cover, P. Hart, Nearest Neigbor Pattern Classification, Transactions of Information Theory,
1967, pp. 21-27.
38 X. Wu, V. Kumar, J.R. Quinlan, J. Ghosh, Q, Yang, H. Motoda, G.F. McLachlan, A. Ng, B. Liu,
P.S. Yu, Z.-H. Zhou, M. Steinbach, D.J. Hand, D. Steinberg. Top 10 algorithms in data mining,
Knowledge Information Systems, 14, 2008, pp. 1-37.
39 R.O. Duda, P.E. Hart, D.G. Stork. Pattern Classification, John Wiley & Sons, New York, 2nd ed,
2001.
40 L. Breiman. Random Forests, Machine Learning, 45(1), 2001, pp. 5-32.
41 S. Haykin. Neural Networks: A Comprehensive Foundation, Prentice-Hall, New Jersey, 2nd ed,
1999.
42 H. Joutsijoki, K. Meissner, M. Gabbouj, S. Kiranyaz, J. Raitoharju, J. Ärje, S. Kärkkäinen, V.
Tirronen, T. Turpeinen, M. Juhola. Evaluating the performance of artificial neural networks for the
classification of freshwater benthic macroinvertebrates, Ecological Informatics, 20, 2014, pp. 1-12.
43 N.D. Lewis. Deep Learning Made Easy with R: A Gentle Introduction For Data Scientist,
AusCov, 2016.
44 T. Kohonen. Self-Organising Maps, Springer-Verlag, Berlin, 3rd ed, 2001.
45 https://deepmind.com/research/alphago/
46 C.J.C. Burges. A tutorial on Suppor t Vector Machines for pattern recognition, Data Mining and

23. 21
Knowledge Discovery, 2(2), 1998, pp. 121-167.
47 V. Vapnik. The Nature of Statistical Learning, Springer-Verlag, New York, 2nd ed, 2000.
48 A.C. Lorena, A.C.P.L.F. de Carvalho, J.M.P. Gama. A review on the combination of binary
classifiers in multiclass problems, Artificial Intelligence Review, 30, 2009, pp. 1-19.
49The National Institute for Health and Welfare (THL) https://www.thl.fi/en/web/thlfi-en
50 The Social Insurance Institution of Finland (Kela) http://www.kela.fi/web/en
51 Criminal Sanctions Agency (RISE) http://www.rikosseuraamus.fi/en/index.html
52 Ministry of Employment (TEM) http://tem.fi/en/frontpage
53 Statistics Finland http://tilastokeskus.fi/index_en.html
54 S. Mansmann, M. Scholl. Decision Support System for Managing Educational Capacity
Utilization in Universities, International Conference on Engineering and Computer Education,
2005.
55 C. Giannoulis, A. Ishizaka. A Web-based Decision Support System with ELECTRE III for a
Personalized Ranking of British Universities, Decision Support Systems ,2010, pp. 488-497.
56 National Studyinfo portal. https://studyinfo.fi/wp2/en/
57 http://www.oracle.com/technetwork/java/javaee/overview/index.html
58http://www.payara.fish/
59https://netbeans.org/