1. COMPARISON OF MACHINE
LEARNING TECHNIQUES USING WEKA
ENVIRONMENT
2011 20th IEEE International Workshops on Enabling
Technologies: Infrastructure for Collaborative Enterprises
Dwi Riyono D10207801
Dani Pranata M10307804
Nurul Retno Nurwulan D10301807

2. INTRODUCTION
• Correct diagnosis for further treatment or a potential therapy
change of a specific patient could be assisted with the use
of machine learning.
• The medical data given was employed in order to evaluate
the performance of a number of classification techniques.
• This study analyzed and evaluated the decision making task
of therapy change which a doctor suggests, when a number
of blood test parameters – mainly Prostate Specific Antigen
(PSA) – are measured every 3 months.
• Prostate cancer is the most common non-cutaneous cancer
and the second-leading cause of death in men in USA. It
prevalent in many countries and exhibits a wide spectrum of
aggressiveness.

3. Material and Methods
• Medical Problem Description and Data
• The WEKA environment
• Techniques

4. MEDICAL PROBLEM
DESCRIPTION AND DATA
• Challenge to physician who treats patients with
prostate cancer in advising effective treatment.
• Selection of appropriate treatment requires assessment
of the tumor’s potential aggressiveness and the
general health, life expectancy, and quality of life
preferences of the patient.
• Parameters chosen: Hematocrit (HCT), White Blood
Cells (WBC), free Prostate Specific Antigen (PSA free),
total Prostate Specific Antigen (PSA total), ratio PSA
(PSAfree/PSAtotal), Prostatic Acidic Phospatase (PAP),
and potential therapy change decision (yes/no).
• Real data of 40 patients were obtained. There were
1960 unique instances consisting of 280 rows and 7
columns.

5. THE WEKA ENVIRONMENT
• WEKA implements various machine learning
classification techniques, algorithm for regression and
clustering along with a number of visualization tools
that has been accepted as powerful and adequate
environment for data mining.
• All data analyzed and mined with the aim of WEKA is
saved in ARFF file format, which consists of special tags
in order to designate between attributes, values, and
names of the data given.
• All of the parameter chosen (blood test parameters)
were numerical values and the change therapy
decision of the doctor in the simple format of a yes/no.

6. TECHNIQUES
• Decision Trees – J48
• Neural Network (Multilayer Perceptron –
MLP)
• Naïve Bayes
• Radial Basis Function (RBF)
• K-Nearest Neighbor (IBk)

7. TECHNIQUES (2)
• Decision Trees – J48
It represents a mapping of the attributes given and
consists of nodes which link to two or more sub-trees. A
node calculates a specific outcome which is based on
the value of the instance and each possible outcome
is linked with one of the sub-trees. The J48 algorithm is
an efficient method for estimation and classification of
fuzzy data.

8. TECHNIQUES (3)
• Neural Network (Multilayer Perceptron – MLP)
An adaptive system that changes its structure based
on external or internal information which flows through
the network during an initial learning phase. In more
practical terms, NN is a non-linear statistical data
modeling tools. It can be used to model complex
relationships between inputs and outputs or to find
patterns in data.
The back propagation algorithm MLP was applied in
order to categorize a practitioner’s decision (therapy
change) was applied, using two input nodes (no = 0,
yes =1)

9. TECHNIQUES (4)
• Naïve Bayes
A representation of the Bayesian classifier that
produces probabilistic rules and received noteworthy
attention when used for classification purposes.
Classification is performed when the well-known Bayes
rule is applied to each attribute of the model and the
probability over an independent class variable is
computed. Although the model is straightforward, it
provides quite promising results on many real world
datasets.

10. TECHNIQUES (5)
• Radial Basis Function (RBF)
Initially introduced in order to address a variety of
problems (old pattern recognition techniques,
clustering, functional approximation, etc.). It is now
acknowledged to be one of the most important NN
models for classification. The basic function is based on
two-layer feed-forward model with a hidden layer
between the sets of input and output. Gaussian
function is preferred for classification and a key factor
for the successful implementation is to find a suitable
center.

11. TECHNIQUES (6)
• K-Nearest Neighbor (IBk)
One of the simplest forms of classification algorithms,
depicted as statistical learning algorithms and are
generated by simply storing the given data. Distance
metric is chosen and any new data is compared
against all-ready “memorized” data items, for the
classification purpose. The new item is assigned to the
class which is most common amongst its k nearest
neighbors. IBk is an implementation of the k-nearest
neighbor, which the number of nearest neighbor (k)
can be set manually or determined automatically
using cross-validation.

12. EXPERIMENT AND
RESULT

13. DATA SOURCE
Blood test are used to collect the data from patients
Six parameters that would be measured :
1. Hct (Hematocrit), volume percentage (%) of red
blood cells in blood
2. WBC (White blood cell)
3. PAP (Prostatic Acidic Phosphatase),
4. PSA Free (Prostate-Specific Antigen)
5. PSA Total
6. PSAf/PSAt
The percentage of PSA in the free or complex isoforms,
were used to predict the patient’s state over a period of
2 years.

14. BLOOD TEST PARAMETERS
Blood test are used to collect some health information
from each patients with the diagnosis of prostate
cancer.
Six parameters that will be measured :
1. Hct (Hematocrit), volume percentage (%) of red
blood cells in blood
2. WBC (White blood cell)
3. PAP (Prostatic Acidic Phosphatase),
4. PSA Free (Prostate-Specific Antigen)
5. PSA Total
6. PSAf/PSAt
The percentage of PSA in the free or complex isoforms,
were used to predict the patient’s state over a period
of 2 years.

15. TABLE 1.
BLOOD TEST PARAMETERS AND THEIR CRITICAL VALUES
Blood Tes Parameters Critical Values
HCT >28%
WBC >4000/mL
PSA Free 0.03ng/dl
PSA Total 0.05ng/dl
PSAf/PSAt >0.2
Prostatic Acid Phosfatase <3.5ng/ml
The difference value between each parameters and their critical value for every
quarter, would be used to decide a potential therapy plan change, along with
patient’s history and previous blood test results.

16. CLASSIFICATION PROCESS OF THE TARGET
PARAMETER (THERAPY CHANGE)
These difference value, would be provided to WEKA
toolbox for classification of the target parameter
(therapy change).
5 machine learning algorithm are used to obtain the
result.
1. J48
2. MLP
3. Naïve Bayes
4. RBF
5. IBk

17. CLASSIFICATION RESULTS FOR EACH
EXAMINED ALGORITHM FOR QUARTER 1
WEKA
Techniques
Simulation Results for Quarter 1
Correctly
Classified
Incorrectly
Classified
Time taken
(sec)
Kappa
statistic
J48 85% (34) 15% (6) 0.03 0.4146
MLP 85% (34) 15% (6) 0.13 0.4146
Naïve Bayes 90% (36) 10% (4) 0.01 0.6098
RBF 90% (36) 10% (4) 0.11 0.6098
IBk 82.5% (33) 17.5% (7) 0.01 0.2708
The table above, is mainly summarizes the accuracy of each machine learning
algorithm for all 40 patients, along with the time taken and Kappa statistic for each
algorithm.

18. TRAINING AND SIMULATION
ERROR FOR QUARTER 1
WEKA
Techniques
Simulation Results for Quarter 1
Mean
Absolute Error
Root Mean
Squared Error
Relative
Absolute Error
(%)
Root Relative
Squared Error (%)
J48 0.1737 0.3638 57.409 94.407
MLP 0.1899 0.3651 62.735 95.039
Naïve Bayes 0.1014 0.3163 33.494 81.334
RBF 0.1423 0.3127 47.007 81.406
IBk 0.1921 0.408 63.478 106.212
The table above is an overall synopsis based on different error rates.

19. DISCUSSION
&
CONCLUSIONS AND FUTURE WORK

20. DISCUSSION
Based on the results obtained for the 1st quarter of the therapy
plan for all patients examined, a number of useful conclusions
could be yielded, concerning the performance and error rates
of the algorithms chosen.
1. Naïve Bayes and RBF Network algorithms succeed to
obtain a relatively high accuracy rate (90%) with Kappa
score of 0.6098
2. Between the two of them, Naïve Bayes performs very fast
only 0.01 seconds comparing to 0.11 seconds that RBF
takes.
3. IBk algorithm has the worst accurracy with a small Kappa
Statitisic score of 0.2708, although time taken only 0.01 s.
4. This study observe that Naïve Bayes has the lowest mean
absolute, relative absolute and root relative squared error
rates, therefore it has more powerful classification
capabilities.
5. This study appointing that Naïve Bayes and RBF are the
best algorithm.
6. IBk is the algorithm with the highest error rate.

22. DISCUSSION
Perfomance J48 worth to mention, with accuracy rate
85% and the visualization tree which derived from the
execution of the algorithm for Q1. This decision tree
given in Figure 1 : For any patient given :
1. If the difference of PSA free is greater than 2.07
then there is definitely a necessity for therapy
change.
2. If not, then a ratioPSA (i.e.PSAfree/PSAtotal) and if
it’s greater than 0.15 (0.17 for a physician) then
there may not be need to change therapy.
3. If ratioPSA is lower than 0.15 then the last
parameter that the algorithm takes into
consideration is Prostatic Acidic Phosfatase and
characterizes a decision made according to a
difference of 2.3ng/ml.

24. DISCUSSION (2)
• Based on table III, concerning in classification error.
Naïve bayes (powerful classification
capabilities)has the lowest mean absolute, relative
absolute and root relative squared error rates.
• The algorithm with the highest error rates as can be
easily seen is IBk.

25. DISCUSSION (3)
• Based on table III, concerning in classification
error. Naïve bayes (powerful classification
capabilities)has the lowest mean absolute,
relative absolute and root relative squared error
rates.
• The algorithm with the highest error rates as can
be easily seen is IBk.

27. DISCUSSION
• This study, from (Table IV) a few observation can be
appointed most important aspect that a physician rarely
changes therapy to many patients during the period of a
quarter, therefore only one or two patients, therapy plan
was changed show in Table IV (Q2,Q4,Q6, and Q7).
• After performing a closer look to the value of the
parameters measured, moreover discussing this discovery
with the physicians (doctors in Urology), it turned out that
case considered to be ‘problematic’ in terms of measured
values. Specifically, these patients were not responding to
the treatment given and constantly the blood parameters
measured were extremely high or very low.
• Mean classification accuracy for each algorithm, for all
quarters examined, was: 92% for J48, 89% for MLP, 86% for
Bayes, 95% for RBF network and 92% for IBk.

29. CONCLUSIONS AND FUTURE WORK
• This study a comparison of five machine learning algorithms
upon real medical data was presented.
• Useful results were obtained concerning the performance and
error rates of the algorithms.
• The experiments performed showed that the best algorithm
based on the prostate cancer data given, is RBF Network
technique.
• RBF algorithm performed quite well in terms of classification
accuracy and Kappa score, as well as has given relatively low
error rates for the Q1 presented.
• One way of improving the result is the proposal of a new hybrid
algorithm.(algorithm which comprises of both the difference
between value s measured and critical values as well as the
difference in the values measured between two subsequent
quarters.
• Future work more clinical cases have to be evaluated to justify
these results as more will become available from the Dept.of
Urology.

30. REFERENCES
• Nils J. Nilsson (1999) Introduction to Machine Learning. California, United States of America
• Jemal A et al: Cancer statistics, 2005. CA Cancer J Clin 2005; 55(1):10–30.
• Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, Ian H. Witten (2009); The
WEKA Data Mining Software: An Update; SIGKD Explorations, Volume 11, Issue 1
• Quinlan, J.R. (1990). Decision trees and decision making. IEEE Trans System, Man and Cybernetics 20, (2),
339–346.
• Duda et al. Pattern classification, John Wiley & Sons, 2001
• Langley, P., Iba, W. Thompson, K. 1992 , An analysis of Bayesian classifiers, in Proceedings of the tenth
national conference on artificial intelligence, AAAI Press and MIT Press, pp. 223--228.
• A. G. Bors, "Introduction of the Radial Basis Function (RBF) Networks," Online Symposium for Electronics
Engineers, issue 1, vol. 1, DSP Algorithms: Multimedia, http://www.osee.net/, Feb. 13 2001, pp. 1-7.
• Aha, D. (1992) “Tolerating noisy, irrelevant, and novel attributes in instance-based learning algorithms.”
International Journal of Man-Machine Studies, Vol. 36, 267–287.
• Quinlan, J.R. (1993). C4.5: Programs for machine learning, Morgan Kaufmann, San Mateo, CA, USA.
• Jang, J.,-S., R., Sun, C., T., Mizutani, E ,: Neuro-fuzzy and soft computing: a computational approach to
learning and machine intelligence, Prentice Hall, Upper Saddle River, NJ, 1997.
• Bishop, C.M. (1996). Neural networks for pattern recognition, First edition, Oxford University Press, USA
• Bors, A.G., Pitas, I., (1996) “Median radial basis functions neural network,” IEEE Trans. on Neural Networks,
vol. 7, no. 6, pp. 1351-1364.