This document is a final examination report submitted to fulfill the requirements for an Information Technology Research and Innovation course. It presents the results of a systematic literature review of 44 studies on machine learning technique usage from 2010 to 2014. The review aimed to identify the most common machine learning techniques, their applications, appropriate data types, and strengths and weaknesses. The techniques most frequently used were found to be support vector machines, artificial neural networks, naive Bayes, decision trees, and k-nearest neighbors. The studies covered fields such as medicine, pharmacology, agriculture, archaeology, games, and business.

1. A SYSTEMATIC LITERATURE REVIEW OF
MACHINE LEARNING TECHNIQUE USAGE
A final-term examination report
to fullfill the requirements for
Information Technology Research and Innovation
Lecture: Dr.-Ing. Ir. Suhardi
by:
Arrahman Adnani, S.ST
NIM.23514052
SCHOOL OF INFORMATICS AND ENGINEERING
INSTITUTE TECHNOLOGY BANDUNG
DECEMBER 2014

2. 2
A Systematic Literature Review of
Machine Learning Technique Usage
Arrahman Adnani, S.ST
NIM.23514052
School of Informatics and Engineering
Institute Technology Bandung
Bandung, Indonesia
adnani@s.itb.ac.id
Abstract
The development of machine learning technique is very fast now. Its usage
has spread to various fields, such as learning machines currently used in medical
science, pharmacology, agriculture, archeology, games, business and so forth.
Many researches has been performed to create a more intelligent machines that
can replace or relieve human tasks such as analyzing, communicating, learning, or
making decisions. In this research performed a systematic review of research from
2010 to 2014 in the literature about the use of the machine learning technique.
The purpose of this study is to determine the techniques and problems in the use
of machine learning that may be used as a reference for conducting research in the
future.
Keywords: artificial neural network, classification, clustering, machine learning
technique, prediction, support vector machines, systematic literature
review

3. 3
1. Introduction
Currently very rapid development of machine learning and its use has been
expanded to various fields. For example, the current machine learning is used in
medical science to measure health [1,2] or diagnose a disease such as cancer [3,4].
In the pharmacology, is not only used to find the right formula and reliable drugs
to incapacitate disease virus [5,6], machine learning is also used to determine the
effective therapeutic treatment [7]. Besides being used in medical science and
pharmacology, or better known as bioinformatics, machine learning is also used in
other fields such as agriculture, archeology, games, business and others.
The researchers have done many studies on machine learning in order to
become more intelligent machines that can replace or relieve human task. With
the use of machine learning techniques, the machine has been able to better
analyze, communicate, learn, make decisions, or make predictions. For example,
in agriculture, machine learning is used to increase agricultural production as with
predicting pest plants [8]. Another example is in the business world, where
machine learning is used to predict the stock market and stock price index
movement [9,10].
Considering the rapid use of machine learning and the many studies about
it, this research needs to be conducted. This research aims to determine the
techniques and problems in the use of machine learning. The result of this study is
expected to be a reference to conduct research in the future.
2. Methodology
The method of this research is performed by adapting the systematic
literature review procedure including planning, conducting and reporting the
review given by [11]. In the planning phase will be designed the review protocol
to be conducted including the following steps: research questions identification,
search strategy design, data collection and data analysis. The process of
systematic literature review can be seen in Figure 1.

4. 4
Figure 1. Systematic Literature Review Process (adapted from [11])
In the first step necessary to identify research questions that will be
answered in the systematic literature review. In the second step will be described
search strategy including search keyword identification and selection of sources to
be searched. In the third step will be carried out the collection of relevant studies
based on the research questions. In the next step, will be conducted an analysis of
the data which previously collected.
a. Research question
There are 4 questions to be answered on this research. Here is a list of
questions and objectives of this research.
Table 1. Research questions and objectives
ID Research Questions Objectives
RQ1
What are the techniques of machine
learning?
Identify the machine learning
techniques commonly being used
RQ2
What are the uses of machine
learning techniques?
Identify the use of machine
learning techniques
RQ3
Which data types are used for
machine learning techniques?
Identify appropriate data type for
machine learning techniques
Planning the review
- Identi fy interest of review
- Designing review protocol
Conducting the review
1. Identify research question
2. Designing search strategy
3. Collecting data
4. Data analysis
Reporting the result of review

5. 5
RQ4
What are the strengths and
weaknesses of the machine learning
techniques?
Identify the performances of
machine learning techniques.
b. Search strategy
In this study used a keyword search “machine learning technique” on three
electronic databases online that are Elsevier, ProQuest, and IEEE. Searches are
limited from 2010 through 2014. Here are the search results for the three sources.
Table 2. Electronic database and search results
ID Databases URL Results (literatures)
DS1 Elsevier http://sciencedirect.com/ 3869
DS2 ProQuest http://search.proquest.com/ 2047
DS3 IEEE http://ieeexplore.ieee.org/ 965
Research on machine learning techniques seem more rapidly seen from the
many studies that have been performed. The amount of research on machine
learning looks increasingly from year to year. Growth in the total of research on
machine learning techniques can be seen in Figure 2. Its composition per year can
be seen in Figure 3.
Figure 2. Total research year 2010-2014
0
250
500
750
1000
1250
2010 2011 2012 2013 2014
Elsevier
ProQuest
IEEE

6. 6
Figure 3. Percentage of total research year 2010-2014
c. Data collection
Details of data to be collected from each selected literature should be able
to answer all the questions of the research. Here are the fields of the data to be
collected from the literature.
Table 3. Fields of data collection
ID Fields
F1 Author (s)
F2 Title
F3 Keyword
F4 Year
F5 Type (Journal/Conference)
F6 Publisher
F7 Machine learning techniques
F8 Use of machine learning techniques
F9 Appropriate data type for machine learning techniques
12,15 13,34 16,48
15,74 16,37
18,76
17,47
23,20
18,45
22,51
26,28 27,88
32,13
20,81 18,45
0%
20%
40%
60%
80%
100%
Elsevier ProQuest IEEE
2014
2013
2012
2011
2010

7. 7
F10 Strengths of the machine learning techniques
F11 Weaknesses of the machine learning techniques
d. Data analysis
In this research as much as 44 selected literatures will be analyzed which
17 literatures taken from Elsevier, 12 from ProQuest, and 15 of the IEEE. While
the composition of the literature based on the year are as follows: 2014 as 17;
2013 as 10; 2012 as 9; 2010 as 5; and 2010 as 3 literatures. Here is a list of
literature that will be analyzed.
Table 4. Selected literatures
ID Author Year Source Type Ref
S1 Paokanta et al. 2010 IEEE Conference [1]
S2 Martínez et al. 2014 Elsevier Journal [2]
S3 Kourou et al. 2014 Elsevier Journal [3]
S4 Asadi et al. 2014 ProQuest Journal [4]
S5 Danger et al. 2010 Elsevier Journal [5]
S6 Urquiza et al. 2012 Elsevier Journal [6]
S7 Caravaca et al. 2013 Elsevier Journal [7]
S8 Kim et al. 2013 Elsevier Conference [8]
S9 Patel et al. 2014 Elsevier Journal [9]
S10 Patel et al. 2014 Elsevier Journal [10]
S11 Ajila et al. 2013 IEEE Conference [12]
S12 Alsri et al. 2014 IEEE Conference [13]
S13 Bal et al. 2014 ProQuest Journal [14]
S14 Bélisle et al. 2014 Elsevier Journal [15]
S15 Betrie et al. 2012 ProQuest Journal [16]
S16 Bhutani 2014 ProQuest Journal [17]
S17 Bohn et al. 2013 Elsevier Conference [18]

8. 8
S18 Chaturvedi et al. 2012 IEEE Conference [19]
S19 Chaudhary et al. 2012 IEEE Conference [20]
S20 Costea 2014 Elsevier Conference [21]
S21 Daybelge et al. 2010 ProQuest Journal [22]
S22 Delen et al. 2012 Elsevier Journal [23]
S23 Fernandes et al. 2014 Elsevier Journal [24]
S24 Frid et al. 2014 IEEE Conference [25]
S25 Holzinger et al. 2014 IEEE Conference [26]
S26 Hosseinifard et al. 2011 IEEE Conference [27]
S27 Huang et al. 2013 ProQuest Journal [28]
S28 Kanewala et al. 2013 IEEE Conference [29]
S29 Kazemian et al. 2014 Elsevier Journal [30]
S30 Lin et al. 2014 IEEE Conference [31]
S31 Ludtke et al. 2011 Elsevier Journal [32]
S32 Oudendag et al. 2012 ProQuest Journal [33]
S33 Oztekin et al. 2013 Elsevier Journal [34]
S34 Panigrahi 2012 IEEE Conference [35]
S35 Pereira et al. 2012 Elsevier Journal [36]
S36 Sarina et al. 2011 ProQuest Journal [37]
S37 Sathyadevan et al. 2014 IEEE Conference [38]
S38 Schuster et al. 2011 IEEE Conference [39]
S39 Silva et al. 2013 IEEE Conference [40]
S40 Singh et al. 2013 IEEE Conference [41]
S41 Suh et al. 2011 ProQuest Journal [42]
S42 Talbi 2013 ProQuest Journal [43]
S43 Yuan et al. 2014 ProQuest Journal [44]
S44 Zhang et al. 2012 ProQuest Journal [45]

9. 9
Furthermore, from the selected literature can be taken keywords used. The
most frequently used keywords can be seen in Figure 4. Machine learning is used
in keyword of almost all the selected literature.
Figure 4. Keywords from selected literatures
3. Result and Discussion
After analyzing the data collected from a variety of selected literature. It
will answer some of the questions that have been the main objective of this
research. Here is the result and discussion.
a. RQ1: What are the techniques of machine learning?
Objective of this research question is to identify the machine learning
technique commonly being used. Five of the techniques often used are Support
Vector Machines (SVM), Artificial Neural Network (ANN), Naive-Bayes (NB),
Decision Trees (DT), and k-Nearest Neighbors (k-NN). The distribution of
machine learning techniques which used in several selected literature can be seen
in Table 5.

10. 10
Table 5. Distribution of machine learning techniques
No Machines Learning Techniques %
1 Support Vector Machines (SVM) 14.46
2 Artificial Neural Network (ANN) 10.84
3 Naive-Bayes (NB) 10.24
4 Decision Trees (DT) 7.83
5 k-Nearest Neighbors (k-NN) 5.42
6 Random Forest (RF) 4.22
7 Bayesian Networks (BN) 3.61
8 Multi-Layer Perceptron (MLP) 3.01
9 k-Means 2.41
10 Logistic Regression (LR) 2.41
11 Multiple Linear Regression (MLR) 1.81
12 Adaboost 1.20
13 Bootstrap Aggregation (Bagging) 1.20
14 Case Based Reasoning (CBR) 1.20
15 Classification and Regression Trees (CART) 1.20
16 Others 28.92
Total 100.00
The first position, a technique often used is SVM while predominantly
developed by Vapnik in 1998, Cherkassky and Mulier in 2007 [16]. SVM
increasingly successfully used in real world applications because it has a good
theoretical basis [20]. SVM theoretical foundation derived from statistical
learning theory which then combined with machine learning techniques [23].

11. 11
b. RQ2: What are the uses of machine learning techniques?
Machine learning techniques are used for classification, prediction,
clustering, ranking, and feature selection. The proportion of machine learning
usage can be seen in Figure 5. The five techniques are often used in the
classification is NB, SVM, DT, k-NN, BN, and MLP. Furthermore, the five
techniques that are often used in prediction are SVM, ANN, DT, RF, k-NN. While
in the clustering, k-Means is the technique most often used.
Figure 5. Proportion of machine learning usage
c. RQ3: Which data types are used for machine learning techniques?
The data type which often used in machine learning technique is structured
data that is in interval, nominal or ordinal scale. However, several studies have
been conducted to use machine learning technique in unstructured data. For
example, is used for identifying moving bodies from videos [38]. In addition,
machine learning technique is also used to analyze other data types such as image
[32,36,44], audio [40], and other e-document [31,35,37,39].
Classification
46,67%
Clustering
8,89%
Feature
Selection
2,22%
Prediction
37,78%
Rangking
4,44%

12. 12
d. RQ4: What are the strengths and weaknesses of the machine learning
techniques?
Some research indicates that SVM is a machine learning technique with
the best accuracy in comparison with other techniques [9,16,19,32,40,45]. But
besides SVM, several other studies showed that ANN is better, especially for
calculating the correlation between variables [34,35]. To get better results in the
accuracy and correlation, can be used hybrid method by combining several
techniques in several stages [10]. However, besides having strength in accuracy
and correlation, SVM and ANN technique has its own weakness where this
technique requires sufficient time to complete the process [16].
4. Conclusion
The use of machine learning technique in the analysis has been growing
rapidly especially to perform classification, prediction, and clustering. Machine
learning technique can be used in structured or unstructured data type. The results
of this research showed that the most often used technique is the SVM because
this technique gives better accuracy than other techniques. Some techniques such
as SVM and ANN can be combined into several stages of analysis to provide
better accuracy and correlation. However, this technique has limitations in terms
of time to complete the process which still requires considerable time.
5. Future Work
Direction of future work can be focused to obtain larger data. Large data
needs not only to obtain the number of samples to be used in the machine learning
process. But it is also to obtain more statistical parameters as input to find a much
better correlation. Currently, the need for larger data represented in big data
analysis. When dealing with big data, not only pay attention to the technique used
to perform the analysis, but also need to pay attention to develop the feature
selection technique in order to obtain the effective and efficient parameters that
will be used in the analysis.

13. 13
References
[1] P. Paokanta, M. Ceccarelli, S. Srichairatanakool, “The Effeciency of Data
Types for Classification Performance of Machine Learning Techniques for
Screening β-Thalassemia”, International Symposium on Applied Sciences
in Biomedical and Communication Technologies, 2010.
[2] J.M. Martínez, P.E. Montero, C. Barbieri, E.S.Olivas, F. Mari, M.M.
Sober, C. Amato, A.J.S. López, M. Bassi, R.M. Benedito, A. Stopper,
J.D.M. Guerrero, E. Gatti, “Prediction of the hemoglobin level in
hemodialysis patients using machine learning techniques”, International
Journal of Computer Methods and Programs in Biomedicine, 2014.
[3] K. Kourou, T.P. Exarchos, K.P. Exarchos, M.V. Karamouzis, D.I.
Fotiadis, “Machine learning applications in cancer prognosis and
prediction”, Computational and Structural Biotechnology Journal, 2014.
[4] H. Asadi, R. Dowling, B. Yan, P. Mitchell, “Machine Learning for
Outcome Prediction of Acute Ischemic Stroke Post Intra-Arterial
Therapy”, Public Library of Science Scholarly Journal, 2014.
[5] R. Danger, I.S. Bedmar, P. Martínez, P. Rosso, “A comparison of machine
learning techniques for detection of drug target articles”, International
Journal of Biomedical Informatics, 2010.
[6] J.M. Urquiza, I.Rojas, H.Pomares, J.Herrera, J. P.Florido, O.Valenzuela,
M.Cepero, “Using machine learning techniques and genomic/proteomic
information from known databases for defining relevant features for PPI
classification”, International Journal of Computers in Biology and
Medicine, 2012.
[7] J. Caravaca, E.S. Olivas, M. Bataller, A.J. Serrano, L.S. Miquel, J.V.
Francés, J.F. Guerrero, “Application of machine learning techniques to
analyse the effects of physical exercise in ventricular fibrillation”,
International Journal of Computers in Biology and Medicine, 2013.
[8] Y.H. Kim, S.J. Yoo, Y.H. Gu, J.H. Lim, D. Han, S.W. Baik, “Crop Pests
Prediction Method Using Regression and Machine Learning Technology
Survey”, International Conference on Future Software Engineering and
Multimedia Engineering, 2013.
[9] J. Patel, S. Shah, P. Thakkar, K. Kotecha, “Predicting stock and stock price
index movement using Trend Deterministic Data Preparation and machine
learning techniques”, International Journal of Expert System with
Applications, 2014.
[10] J. Patel, S. Shah, P. Thakkar, K. Kotecha, “Predicting stock market index
using fusion of machine learning techniques”, International Journal of
Expert System with Applications, 2014.

14. 14
[11] B.A. Kitchenham, “Guidelines for performing systematic literature review
in software engineering”, Technical report EBSE-2007-001, UK, 2007.
[12] S.A. Ajila, A.A. Bankole, “Cloud Client Prediction Models Using
Machine Learning Techniques”, Annual Computer Software and
Applications Conference, 2013.
[13] Alsri, S. Almuhammadi, S. Mahmood, “A model for work distribution in
global software development based on machine learning techniques”,
Science and Information Conference, 2014.
[14] M. Bal, M.F. Amasyali, H. Sever, G. Kose, A. Demirhan, “Performance
Evaluation of the Machine Learning Algorithms Used in Inference
Mechanism of a Medical Decision Support System”, The Scientific World
Journal, 2014.
[15] E. Bélisle, Z. Huang, S.L. Digabel, A.E. Gheribi, “Evaluation of machine
learning interpolation techniques for prediction of physical properties”,
International Journal of Computational Materials Science, 2014.
[16] G.D. Betrie, S. Tesfamariam, K.A. Morin, R. Sadiq, “Predicting copper
concentrations in acid mine drainage a comparative analysis of five
machine learning techniques”, Springer Science & Business Media
Scholarly Journals, 2012.
[17] G. Bhutani, “Application of Machine-Learning Based Prediction
Techniques in Wireless Networks”, International Journal of
Communications, Network and System Sciences, 2014.
[18] B. Bohn, J. Garcke, R.I. Teran, A. Paprotny, B. Peherstorfer, U.
Schepsmeier, C.A. Thole, “Analysis of Car Crash Simulation Data with
Nonlinear Machine Learning Methods”, International Conference on
Computational Science, 2013.
[19] K.K. Chaturvedi, V.B. Singh, “Determining Bug severity using machine
learning techniques”, International Conference on Software Engineering,
2012.
[20] Chaudhary, S. Kolhe, R. Kamal, “Machine learning techniques for Mobile
Intelligent Systems: A study”, International Conference on Wireless and
Optical Communications Networks, 2012.
[21] Costea, “Applying Fuzzy Logic and Machine Learning Techniques in
Financial Performance Predictions”, International Conference on Applied
Statistics, 2014.
[22] T. Daybelge, I. Cicekli, “A ranking method for example based machine
translation results by learning from user feedback”, Springer Science &
Business Media Scholarly Journals, 2010.
[23] D. Delen, H. Zaim, C. Kuzey, S. Zaim, “A comparative analysis of
machine learning systems for measuring the impact of knowledge

15. 15
management practices”, International Journal of Decision Support
Systems, 2012.
[24] J.A. Fernandes, X. Irigoien, J.A. Lozano, I. Inza, N. Goikoetxea, A. Pérez,
“Evaluating machine-learning techniques for recruitment forecasting of
seven North East Atlantic fish species”, International Journal of Ecological
Informatics, 2014.
[25] Frid, E.J. Safra, H. Hazan, L.L. Lokey, D. Hilu, L. Manevitz, L.O. Ramig,
S. Sapir, “Computational Diagnosis of Parkinson's Disease Directly from
Natural Speech Using Machine Learning Techniques”, International
Conference on Software Science, Technology and Engineering, 2014.
[26] Holzinger, D. Blanchard, M. Bloice, K. Holzinger, V. Palade, R. Rabadan,
“Darwin, Lamarck, or Baldwin: Applying Evolutionary Algorithms to
Machine Learning Techniques”, International Joint Conferences on Web
Intelligence and Intelligent Agent Technologies, 2014.
[27] Hosseinifard, M.H. Moradi, R. Rostami, “Classifying depression patients
and normal subjects using machine learning techniques”, Iranian
Conference on Electrical Engineering, 2011.
[28] M.J. Huang, H.K. Chiang, P.F. Wu, Y.J. Hsieh, “A multi-strategy machine
learning student modeling for intelligent tutoring systems Based on
blackboard approach”, Emerald Group Publishing Limited Scholarly
Journals, 2013.
[29] U. Kanewala, J.M. Bieman, “Using Machine Learning Techniques to
Detect Metamorphic Relations for Programs without Test Oracles”,
International Symposium on Software Reliability Engineering, 2013.
[30] H.B. Kazemian, S. Ahmed, “Comparisons of machine learning techniques
for detecting malicious webpages”, International Journal of Expert System
with Applications, 2014.
[31] M. Lin, D.W. Cheung, “Automatic Tagging Web Services Using Machine
Learning Techniques”, International Joint Conferences on Web
Intelligence and Intelligent Agent Technologies, 2014.
[32] Ludtke, K. Jerosch, O. Herzog, M. Schluter, “Development of a machine
learning technique for automatic analysis of seafloor image data: Case
example, Pogonophora coverage at mud volcanoes”, International Journal
of Computers & Geosciences, 2011.
[33] Oudendag, Z. Szlávik, V.V. Hennie, “The use of Machine Learning
techniques to predict farm size change - An implementation in the Dutch
Dairy sector”, American Academic & Scholarly Research Journal, 2012.
[34] Oztekin, D. Delen, A. Turkyilmaz, S. Zaim, “A machine learning-based
usability evaluation method for eLearning systems”, International Journal
of Decision Support Systems, 2013.

16. 16
[35] P.K. Panigrahi, “A Comparative Study of Supervised Machine Learning
Techniques for Spam E-mail Filtering”, International Conference on
Computational Intelligence and Communication Networks, 2012.
[36] L.A.M. Pereira, R.Y.M. Nakamura, G.F.S. de Souza, D. Martins, J. P.
Papa, “Aquatic weed automatic classification using machine learning
techniques”, International Journal of Computers and Electronics in
Agriculture, 2012.
[37] S. Sarina, S.S. Mariyam, A. Ajith, S. Shahida, “Intelligent Web Caching
using Machine Learning Methods”, Institute of Computer Science Czech
Republic Scholarly Journals, 2011.
[38] S. Sathyadevan, A.K. Balakrishnan, S. Arya, S.A. Raghunath, “Identifying
Moving Bodies from CCTV Videos using Machine Learning Techniques”,
International Conference on Networks & Soft Computing, 2014.
[39] E.W. Schuster, S. Kumar, S.E. Sarma, J.L. Willers, G.A. Milliken,
“Infrastructure for Data-Driven Agriculture: Identifying Management
Zones for Cotton using Statistical Modeling and Machine Learning
Techniques”, International Conference & Expo on Emerging Technologies
for a Smarter World, 2011.
[40] D.F. Silva, V.M.A. de Souza, G.E.A.P.A. Batista, E. Keogh, D.P.W. Ellis,
“Applying Machine Learning and Audio Analysis Techniques to Insect
Recognition in Intelligent Traps”, International Conference on Machine
Learning and Applications, 2013.
[41] S. Singh, S.P. Lal, “Educational courseware evaluation using Machine
Learning techniques”, Conference on e-Learning, e-Management and e-
Services, 2013.
[42] M. Suh, A. Nahapetian, J. Woodbridge, M. Rofouei, M. Sarrafzadeh,
“Machine Learning-Based Adaptive Wireless Interval Training Guidance
System”, Springer Science & Business Media Scholarly Journals, 2011.
[43] E.G. Talbi, “Combining metaheuristics with mathematical programming,
constraint programming and machine learning”, Springer Science &
Business Media Scholarly Journals, 2013.
[44] T. Yuan, X. Zheng, X. Hu, W. Zhou, W. Wang, “A Method for the
Evaluation of Image Quality According to the Recognition Effectiveness
of Objects in the Optical Remote Sensing Image Using Machine Learning
Algorithm”, Public Library of Science Scholarly Journal, 2014.
[45] M. Zhang, G. Johnson, J. Wang, “Predicting Takeover Success Using
Machine Learning Techniques”, Journal of Business & Economics
Research, 2012.