ORIGINAL ARTICLEBig data analytics capabilities a systema.docx

ORIGINAL ARTICLE
Big data analytics capabilities: a systematic literature
review and research agenda
Patrick Mikalef1 • Ilias O. Pappas1 • John Krogstie1 •
Michail Giannakos1
Received: 15 November 2016 / Revised: 3 July 2017 / Accepted:
12 July 2017 /
Published online: 15 July 2017
� Springer-Verlag GmbH Germany 2017
Abstract With big data growing rapidly in importance over the
past few years,
academics and practitioners have been considering the means
through which they
can incorporate the shifts these technologies bring into their
competitive strategies.
To date, emphasis has been on the technical aspects of big data,
with limited
attention paid to the organizational changes they entail and how
they should be
leveraged strategically. As with any novel technology, it is
important to understand

the mechanisms and processes through which big data can add
business value to
companies, and to have a clear picture of the different elements
and their interde-
pendencies. To this end, the present paper aims to provide a
systematic literature
review that can help to explain the mechanisms through which
big data analytics
(BDA) lead to competitive performance gains. The research
framework is grounded
on past empirical work on IT business value research, and
builds on the resource-
based view and dynamic capabilities view of the firm. By
identifying the main areas
of focus for BDA and explaining the mechanisms through which
they should be
leveraged, this paper attempts to add to literature on how big
data should be
examined as a source of competitive advantage. To this end, we
identify gaps in the
extant literature and propose six future research themes.
Keywords Big data � Dynamic capabilities � Resource-based
view � Competitive
performance � IT strategy

& Patrick Mikalef
[email protected]
1
Norwegian University of Science and Technology, Trondheim,
Norway
123
Inf Syst E-Bus Manage (2018) 16:547–578
https://doi.org/10.1007/s10257-017-0362-y
http://crossmark.crossref.org/dialog/?doi=10.1007/s10257-017-
0362-y&domain=pdf
http://crossmark.crossref.org/dialog/?doi=10.1007/s10257-017-
0362-y&domain=pdf
https://doi.org/10.1007/s10257-017-0362-y
1 Introduction
The application of big data in driving organizational decision
making has attracted
much attention over the past few years. A growing number of
firms are focusing
their investments on big data analytics (BDA) with the aim of
deriving important
insights that can ultimately provide them with a competitive
edge (Constantiou and
Kallinikos 2015). The need to leverage the full potential of the
rapidly expanding

data volume, velocity, and variety has seen a significant
evolution of techniques and
technologies for data storage, analysis, and visualization.
However, there has been
considerably less research attention on how organizations need
to change in order to
embrace these technological innovations, as well as on the
business shifts they entail
(McAfee et al. 2012). Despite the hype surrounding big data,
the issue of examining
whether, and under what conditions, big data investments
produce business value,
remains underexplored, severely hampering their business and
strategic potential
(McAfee et al. 2012). Most studies to date have primarily
focused on infrastructure,
intelligence, and analytics tools, while other related resources,
such as human skills
and knowledge, have been largely disregarded. Furthermore,
orchestration of these
resources, the socio-technological developments that they
precipitate, as well as
how they should be incorporated into strategy and operations
thinking, remains an

underdeveloped area of research (Gupta and George 2016).
Over the past few years, several research commentaries have
stressed the
importance of delving into the whole spectrum of aspects that
surround BDA
(Constantiou and Kallinikos 2015; Markus 2015). Nevertheless,
exploratory
empirical literature on the topic is still quite scarce (Gupta and
George 2016;
Wamba et al. 2017). Past literature reviews on the broader
information systems (IS)
domain have demonstrated that there are multiple aspects that
should be considered
when examining the business potential of IT investments
(Schryen 2013).
Furthermore, the particularities of each technological
development need to be
thoroughly examined in order to fully capture the
interdependencies that develop
between them, and how they produce value at a firm level. Past
literature on IT
business value has predominantly used the notion of IT
capabilities to refer to the

broader context of technology within firms, and the overall
proficiency in leveraging
and mobilizing the different resources and capabilities
(Bharadwaj 2000). It is
therefore important to identify and explore the domain-specific
aspects that are
relevant to BDA within the business context (Kamioka and
Tapanainen 2014).
While there is a growing stream of literature on the business
potential of BDA,
there is still limited work grounded on established theories used
in the IT-business
value domain (Gupta and George 2016). The lack of empirical
work in this direction
significantly hinders research concerning the value of BDA, and
leaves practitioners
in unchartered territories when faced with implementing such
initiatives in their
firms. Hence, in order to derive meaningful theoretical and
practical implications, as
well as to identify important areas of future research, it is
critical to understand how
the core artifacts pertinent to BDA are shaped, and how they
lead to business value

(Constantiou and Kallinikos 2015). Therefore, we employ a
systematic literature
review grounded in the established resource-based view (RBV)
of the firm, as well
548 P. Mikalef et al.
123
as the emerging dynamic capabilities view (DCV). We select
these theoretical
groundings since the former provides a solid foundation upon
which all relevant
resources can be identified and evaluated towards their
importance, while the latter
enables examination of the organizational capabilities towards
which these
resources should be directed in order to achieve competitive
performance gains
(Mikalef et al. 2016a, b). As such, the DCV exerts
complementarities in relation to
the RBV by providing an explanation of the rent-yielding
properties of organiza-
tional capabilities that can be leveraged by means of BDA
(Makadok 2001). Our

theoretical framework that guides the systematic literature
review uncovers some
initial findings on the value of BDA, while also providing a
roadmap on several
promising research streams.
The rest of the paper is structured as follows. In Sect. 2, we
describe the research
methodology used to conduct the systematic literature review,
and outline the main
steps followed. Next, in Sect. 3, we distinguish between the
concepts of big data,
BDA, and BDA capability, and present some definitions as
described in literature
for each. In Sect. 4, we proceed to describe the main theoretical
foundations upon
which we build on and develop the proposed research
framework. We then
summarize existing work on the business value of BDA
according to the identified
themes. In Sect. 5, we outline a series of areas that are currently
under-researched
and propose appropriate theoretical stances that could be
utilized in their
examination. In closing, Sect. 6 presents some concluding

remarks on the area of
BDA and their application to the strategic domain.
2 Research methodology
Following the established method of a systematic literature
review (Kitchenham
2004, 2007; Kitchenham et al. 2009), we undertook the review
in distinct stages.
First, we developed the review protocol. Second, we identified
the inclusion and
exclusion criteria for relevant publications. Third, we performed
an in-depth search
for studies, followed by critical appraisal, data extraction and a
synthesis of past
findings. The next sub-sections describe in detail the previously
mentioned stages
(Fig. 1).
2.1 Protocol development
The first step of the systematic literature review was to develop
a protocol for the
next steps. In accordance with the guideline, procedures, and
policies of the
Cochrance Handbook for Systematic Reviews of Intervention
(Higgins and Green

2008), the protocol established the main research question that
guided the selection
of papers, the search strategy, inclusion and quality criteria, as
well as the method of
synthesis. The review process was driven by the following
research question: What
are the definitional aspects, unique characteristics, challenges,
organizational
transformations, and business value associated with big data?
By focusing on these
elements of the research question, the subject areas and relevant
publications and
materials were identified.
Big data analytics capabilities: a systematic literature… 549
123
2.2 Inclusion and exclusion criteria
Due to the importance of the selection phase in determining the
overall validity of
the literature review, a number of inclusion and exclusion
criteria were applied.
Studies were eligible for inclusion if they were focused on the

topic of how big data
can provide business value. Publications were selected from
2010 onwards, since
that is when the term gained momentum in the academic and
business communities.
The systematic review included research papers published in
academic outlets, such
as journal articles and conference proceedings, as well as
reports targeted at
business executives and a broader audience, such as scientific
magazines. In-
progress research and dissertations were excluded from this
review, as were studies
that were not written in English. Finally, given that our focus
was on the business
transformation that big data entails, along with performance
outcomes, we included
quantitative, qualitative, and case studies. Since the topic of
interest is of an
interdisciplinary nature, a diversity of epistemological
approaches was opted for.
2.3 Data sources and search strategy
The search strategy started by forming search strings that were
then combined to

form keywords. In addition, during the search we employed
wildcard symbols in
Fig. 1 Stages of the study selection process
123
order to reduce the number of search strings. Combinations of
two sets of keywords
were used, with the first term being ‘big data,’ and the second
term being one of 12,
which were reviewed by a panel of five experts. These search
terms included:
analytics capability, competitive performance, firm
performance, organizational
performance, dynamic capabilities, resource-based view, human
skills, managerial
skills, analytics ecosystems, data scientist, competencies, and
resource management.
Keywords were searched within the title, abstract, and keyword
sections of the
manuscripts. The search strategy included electronic databases
such as Scopus,

Business Source Complete, Emerald, Taylor & Francis,
Springer, Web of
Knowledge, ABI/inform Complete, IEEE Xplore, and the
Association of Informa-
tion Systems (AIS) library. To further complement our search,
we applied the search
terms in the search engine Google Scholar, as well as the AIS
basket of eight
journals.
The search was initiated on September 5, 2016 and ended on
February 26, 2017.
At stage 1, 459 papers were identified and entered into the
reference manager
EndNote. At stage 2, all authors went through the titles of the
studies of stage 1 in
order to determine their relevance to the systematic review. At
this stage, studies
that were clearly not about the business aspects of big data were
excluded,
independently of whether they were empirical. In addition,
articles that were
focused on big data for public administration were not included
in the next stage.
The number of retained articles after the abovementioned

process was 228. In the
third stage, all remaining articles were examined in terms of
their abstracts and their
focus in relation to the research question we had defined.
However, some abstracts
were of varying quality, some were lacking information about
the content of the
article, while others had an apparent disconnect with their title
and did not fit our
focus. At this stage, each papers’ abstract was reviewed
independently by each
author. From the 228 abstracts assessed, 101 were omitted,
leaving 127 papers to be
further analyzed.
2.4 Quality assessment
Each of the 127 papers that remained after stage 3 was assessed
independently by
the authors in terms of several quality criteria. Studies were
examined in terms of
scientific rigor, so that appropriate research methods had been
applied; credibility,
to assess whether findings were well presented; and relevance,
which was assessed

based on whether findings were useful for companies engaging
in big data projects,
as well as the academic community. Taken together, these
criteria provided a
measure of the extent to which a publication would make a
valuable contribution to
the review. At this stage another 43 papers were excluded,
leaving 84 papers for
data extraction and synthesis. These papers were then coded
according to their area
of focus, allowing a categorization to be constructed. The
derived categories were a
result of identifying the main research areas that papers aimed
to contribute towards.
By categorizing papers, we were able to extract the details
needed to answer each of
the posed research questions.
123
2.5 Data extraction and synthesis of findings
In order to synthesize findings and categorize studies based on
their scope, an

analysis of the different research streams was performed. The
first step was to
identify the main concepts from each study, using the authors’
original terms. The
key concepts were then organized in a spreadsheet in order to
enable comparison
across studies and translation of findings into higher-order
interpretations. An
analysis was conducted based on the following areas of focus:
organizational
performance outcomes of big data, human skills and knowledge,
tangible and
intangible resources, team orchestration and project
management, adoption and
diffusion of big data initiatives, governance in big data projects,
as well as ethical
and moral issues related to big data within the business domain.
For empirical
studies, the the authors also recorded the type of study
conducted (e.g. qualitative,
quantitative, case study), the sample size, the instruments used
(e.g. surveys,
interviews, observations), as well as contextual factors
surrounding the study (e.g.

industry, country, firm size). Constant consensus meetings of all
researchers
established the data extraction stage and the categorization of
publications. The
remaining 84 papers were analyzed in detail in accordance with
the coding scheme,
and relevant data were extracted, analyzed, and synthesized.
3 Defining big data in the business context
Big data is becoming an emerging topic of interest in IS,
computer and information
sciences, management, and social sciences (Constantiou and
Kallinikos 2015). This
phenomenon is largely attributed to the widespread adoption of
social media,
mobile devices and sensors, integrated IS, and artifacts related
to the Internet of
Things. The surging interest in big data is also reflected in the
academic literature,
which spans multiple disciplinary domains (Chen et al. 2016).
While the different
epistemological domains provide an alternative perspective on
the notion of big
data, the definitions and key concepts put forth by each differ

significantly (Wamba
et al. 2015). As such, the first step of the systematic literature
review is to identify
the key concepts and develop integrative definitions of each.
Notions such as big
data, BDA, and BDA capability are often used interchangeably
in the literature.
However, their theoretical underpinnings reflect a different
perspective in how they
are perceived and measured (Cao and Duan 2014a). Therefore,
it is imperative to
clearly define the meaning of these concepts, and that aspects
they encompass.
3.1 Big data
As a starting point, we provide an overview of how big data
have been defined in
past studies, as well as what attributes are integral to the
concept. Several definitions
of big data have been put forth to date in attempts to distinguish
the phenomenon of
big data from conventional data-driven or business analytics
approaches (Table 1).
Some scholars focus on the origin of the data, emphasizing the
various channels

from which they are collected, such as enterprise IS, customer
transactions,
123
Table 1 Sample definitions of big data
Author(s) and date Definition
Russom (2011) Big data involves the data storage, management,
analysis, and visualization
of very large and complex datasets
White (2011) Big data involves more than simply the ability to
handle large volumes of
data; instead, it represents a wide range of new analytical
technologies and
business possibilities. These new systems handle a wide variety
of data,
from sensor data to Web and social media data, improved
analytical
capabilities, operational business intelligence that improves
business agility
by enabling automated real-time actions and intraday decision
making,

faster hardware and cloud computing including on-demand
software-as-a
service. Supporting big data involves combining these
technologies to
enable new solutions that can bring significant benefits to the
business
Beyer and Laney (2012) Big data: high-volume, high-velocity,
and/or high-variety information assets
that require new forms of processing to enable enhanced
decision making,
insight discovery, and process optimization
McAfee et al. (2012) Big data, like analytics before it, seeks to
glean intelligence from data and
translate that into business advantage. However, there are three
key
differences: Velocity, variety, volume
Gantz and Reinsel (2012) Big data focuses on three main
characteristics: the data itself, the analytics of
the data, and presentation of the results of the analytics that
allow the
creation of business value in terms of new products or services
Boyd and Crawford

(2012)
Big data: a cultural, technological, and scholarly phenomenon
that rests on
the interplay of (1) Technology: maximizing computation power
and
algorithmic accuracy to gather, analyze, link, and compare large
datasets.
(2) Analysis: drawing on large datasets to identify patterns in
order to make
economic, social, technical, and legal claims. (3) Mythology:
the
widespread belief that large datasets offer a higher form of
intelligence and
knowledge that can generate insights that were previously
impossible, with
the aura of truth, objectivity, and accuracy
Schroeck et al. (2012) Big data is a combination of volume,
variety, velocity and veracity that
creates an opportunity for organizations to gain competitive
advantage in
today’s digitized marketplace
Bharadwaj et al. (2013) Big data refers to datasets with sizes
beyond the ability of common software

tools to capture, curate, manage, and process the data within a
specified
elapsed time
Kamioka and Tapanainen
(2014)
Big data is large-scale data with various sources and structures
that cannot be
processed by conventional methods and that is intended for
organizational
or societal problem solving
Bekmamedova and
Shanks (2014)
Big data involves the data storage, management, analysis, and
visualization
of very large and complex datasets. It focuses on new data-
management
techniques that supersede traditional relational systems, and are
better
suited to the management of large volumes of social media data
Davis (2014) Big data consists of expansive collections of data
(large volumes) that are
updated quickly and frequently (high velocity) and that exhibit

a huge
range of different formats and content (wide variety)
Sun et al. (2015) Big data: the data-sets from heterogeneous and
autonomous resources, with
diversity in dimensions, complex and dynamic relationships, by
size that is
beyond the capacity of conventional processes or tools to
effectively
capture, store, manage, analyze, and exploit them
123
machines or sensors, social media, cell phones or other
networked devices, news
and network content, as well as GPS signals (Chen et al. 2016;
Opresnik and Taisch
2015). The majority of scholars emphasize the ‘‘three Vs’’ that
characterize big data:
volume, velocity, and variety (McAfee et al. 2012; Davis 2014;
Sun et al. 2015).
Volume refers to the sheer size of the dataset due to the
aggregation of a large

number of variables and an even larger set of observations for
each variable.
(George et al. 2016). In addition, many definitions highlight the
growing rate at
which the quantity of data increases, commonly expressed in
petabytes or exabytes,
used by decision makers to aid strategic decisions (Akter et al.
2016a). Velocity
reflects the speed at which these data are collected, updated,
and analyzed, as well as
the rate at which their value becomes obsolete (Davis 2014;
George et al. 2016).
The ‘newness’ of data that decision makers are able to collect,
as well as the
capacity to analyze these data-streams, is an important factor
when it comes to
improving business agility and enabling real-time actions and
intraday decision
making (White 2011; Boyd and Crawford 2012). Variety refers
to the plurality of
structured and unstructured data sources, which, amongst
others, include text, audio,
images, video, networks, and graphics (Constantiou and
Kallinikos 2015; George

et al. 2016). While there are no universal benchmarks for
defining the volume,
velocity, and variety of big data, the defining limits are
contingent upon size, sector,
and location of the firm, and are subject to changing limits over
time (Gandomi and
Haider 2015).
Adding to the existing body of definitions, several scholars
have included
different aspects of big data in their conceptualizations (Table
2). For instance, a
commonly acknowledged aspect of big data is its veracity
(Akter et al. 2016a, b;
Table 1 continued
Author(s) and date Definition
Opresnik and Taisch
(2015)
Big data typically refers to the following types of data: (1)
traditional
enterprise data, (2) machine-generated/sensor data (e.g.
weblogs, smart
meters, manufacturing sensors, equipment logs), and (3) social
data

Constantiou and
Kallinikos (2015)
Big data often represents miscellaneous records of the
whereabouts of large
and shifting online crowds. It is frequently agnostic, in the
sense of being
produced for generic purposes or purposes different from those
sought by
big data crunching. It is based on varying formats and modes of
communication (e.g. text, image, and sound), raising severe
problems of
semiotic translation and meaning compatibility. Big data is
commonly
deployed to refer to large data volumes generated and made
available on
the Internet and the current digital media ecosystems
Akter et al. (2016a) Big data is defined in terms of five ‘Vs:’
volume, velocity, variety, veracity,
and value. ‘Volume’ refers to the quantities of big data, which
are
increasing exponentially. ‘Velocity’ is the speed of data
collection,

processing and analyzing in the real time. ‘Variety’ refers to the
different
types of data collected in big data environments. ‘Veracity’
represents the
reliability of data sources. Finally, ‘value’ represents the
transactional,
strategic, and informational benefits of big data
Abbasi et al. (2016) Big data differs from ‘regular’ data along
four dimensions, or ‘4 Vs’—
volume, velocity, variety, and veracity
123
Abbasi et al. 2016). Veracity refers to the degree to which big
data is trusted,
authentic, and protected from unauthorized access and
modification (Demchenko
et al. 2013). Analyzing high-quality and reliable data is
imperative in enabling
management to make cognizant decisions and derive business
value (Akter et al.
2016b). Hence, big data used for business decisions should be
authenticated and

have passed through strict quality-compliance procedures before
being analyzed
(Dong and Srivastava 2013; Gandomi and Haider 2015). This
vast amount of data is
argued to be an important enabler of creating value for
organizations (Gandomi and
Haider 2015). Oracle introduced value as a defining aspect of
big data. According to
Oracle’s (2012) definition, big data are frequently characterized
by low value
density, meaning that the value of the processed data is
proportionately low
compared to its volume. Seddon and Currie (2017) included two
additional
dimensions in the definition of big data: variability and
visualization. Variability
refers to the dynamic opportunities that are available by
interpreting big data, while
visualization has to do with the representation of data in
meaningful ways through
artificial intelligence methods that generate models (Seddon and
Currie 2017).
3.2 Big data analytics

Some definitions of big data focus solely on the data and their
defining
characteristics (Davis 2014; Akter et al. 2016a, b; Abbasi et al.
2016); others
extend and include the analytical procedures, tools, and
techniques that are
Table 2 Defining characteristics of big data
Attribute Definition
Volume Volume represents the sheer size of the dataset due to
the aggregation of a large number
of variables and an even larger set of observations for each
variable. (George et al.
2016)
Velocity Velocity reflects the speed at which data are collected
and analyzed, whether in real time
or near real time from sensors, sales transactions, social media
posts, and sentiment data
for breaking news and social trends. (George et al. 2016)
Variety Variety in big data comes from the plurality of
structured and unstructured data sources
such as text, videos, networks, and graphics among others.
(George et al. 2016)
Veracity Veracity ensures that the data used are trusted,

authentic, and protected from
unauthorized access and modification. (Demchenko et al. 2013)
Value Value represents the extent to which big data generates
economically worthy insights
and/or benefits through extraction and transformation. (Wamba
et al. 2015)
Variability Variability concerns how insight from media
constantly changes as the same information
is interpreted in a different way, or new feeds from other
sources help to shape a
different outcome. (Seddon and Currie 2017)
Visualization Visualization can be described as interpreting the
patterns and trends that are present in
the data. (Seddon and Currie 2017)
3Vs: volume, velocity, variety (Chen and Zhang 2014)
4Vs: volume, velocity, variety, veracity (Zikopoulos and Eaton
2011; Schroeck et al. 2012; Abbasi et al.
2016)
5Vs: volume, velocity, variety, veracity, value (Oracle 2012;
Sharda et al. 2013)
7Vs: volume, velocity, variety, veracity, value variability,
visualization (Seddon and Currie 2017)

123
employed (Russom 2011; Bharadwaj et al. 2013); while some
even go on to
describe the type of impact that the analysis and presentation of
big data can yield in
terms of business value (White 2011; Beyer and Laney 2012;
Schroeck et al. 2012;
De Mauro et al. 2015). This point is made very clear by the
definition provided by
Gantz and Reinsel (2012), who state that BDA revolve around
three main
characteristics: the data itself, the analytics applied to the data,
and the presentation
of results in a way that allows the creation of business value. In
this definition, the
process of analyzing the data is outlined without linking it to
any tangible or
intangible business outcome. George et al. (2016) posit that big
data refers to large
and varied data that can be collected and managed, whereas data
science develops

models that capture, visualize, and analyze the underlying
patterns in the data. To
make this distinction more apparent, some scholars use the term
BDA to emphasize
the process and tools used in order to extract insights from big
data. In essence,
BDA encompasses not only the entity upon which analysis in
performed—i.e. the
data—but also elements of tools, infrastructure, and means of
visualizing and
presenting insight. This distinction is quite eloquently put in the
definitions of Kwon
et al. (2014), and Lamba and Dubey (2015). Nevertheless, while
the definitions of
BDA encompass a wider spectrum of elements critical to the
success of big data,
they do not include the organizational resources that are
required to transform big
data into actionable insight. Becoming a data-driven
organization is a complex and
multifaceted task, and necessitates attention at multiple levels
from managers. To
address the transition to a data-driven era and provide
practitioners with guidelines

on how to deploy their big data initiatives, scholars have begun
utilizing the term
‘BDA capability’ to reference a company’s proficiency in
leveraging big data to
gain strategic and operational insight (Table 3).
Table 3 Sample definitions of big data analytics
Authors and date Definition
Loebbecke and Picot
(2015)
Big data analytics: a means to analyze and interpret any kind of
digital
information. Technical and analytical advancements in BDA,
which—in large
…
© Kitchenham, 2004
Procedures for Performing
Systematic Reviews
Barbara Kitchenham

e-mail: [email protected]
Joint Technical Report
Software Engineering Group
Department of Computer Science
Keele University
Keele, Staffs
ST5 5BG, UK
Keele University Technical Report TR/SE-0401
ISSN:1353-7776
and
Empirical Software Engineering
National ICT Australia Ltd.
Bay 15 Locomotive Workshop
Australian Technology Park
Garden Street, Eversleigh
NSW 1430, Australia
NICTA Technical Report 0400011T.1

July, 2004
i
0. Document Control Section
0.1 Contents
0. Document Control
Section....................................................................................
..i
0.1 Contents
...............................................................................................
...........i
0.2 Document Version Control
.......................................................................... iii
0.3 Executive Summary
......................................................................................iv
1.
Introduction............................................................................
................................1

2. Systematic Reviews
...............................................................................................
1
2.1 Reasons for Performing Systematic Reviews
................................................1
2.2 The Importance of Systematic Reviews
........................................................2
2.3 Advantages and disadvantages
......................................................................2
2.4 Feature of Systematic Reviews
......................................................................2
3. The Review Process
.............................................................................................. .
3
4. Planning
...............................................................................................
..................3
4.1 The need for a systematic
review...................................................................3
4.2 Development of a Review Protocol
...............................................................4
4.2.1 The Research Question
..........................................................................5
4.2.1.1 Question Types
..................................................................................5
4.2.1.2 Question Structure
.............................................................................6
4.2.1.2.1 Population
....................................................................................6
4.2.1.2.2 Intervention
..................................................................................6

4.2.1.2.3 Outcomes
.....................................................................................6
4.2.1.2.4 Experimental designs
...................................................................7
4.2.2 Protocol
Review....................................................................................
.7
5. Conducting the review
...........................................................................................7
5.1 Identification of Research
..............................................................................7
5.1.1 Generating a search strategy
..................................................................7
5.1.2 Publication Bias
.....................................................................................8
5.1.3 Bibliography Management and Document Retrieval
............................9
5.1.4 Documenting the Search
........................................................................9
5.2 Study Selection
..............................................................................................
9
5.2.1 Study selection
criteria...........................................................................9
5.2.2 Study selection process
........................................................................10
5.2.3 Reliability of inclusion
decisions.........................................................10
5.3 Study Quality Assessment
...........................................................................10
5.3.1 Quality
Thresholds..............................................................................

.11
5.3.2 Development of Quality
Instruments...................................................15
5.3.3 Using the Quality
Instrument...............................................................16
5.3.4 Limitations of Quality Assessment
......................................................16
5.4 Data Extraction
............................................................................................1
7
5.4.1 Design of Data Extraction Forms
........................................................17
5.4.2 Contents of Data Collection
Forms......................................................17
5.4.3 Data extraction procedures
..................................................................17
5.4.4 Multiple publications of the same data
................................................18
5.4.5 Unpublished data, missing data and data requiring
manipulation .......18
5.5 Data
Synthesis.................................................................................
.............18
ii
5.5.1 Descriptive synthesis
...........................................................................19
5.5.2 Quantitative Synthesis
.........................................................................19
5.5.3 Presentation of Quantitative Results
....................................................20

5.5.4 Sensitivity
analysis...............................................................................21
5.5.5 Publication bias
....................................................................................21
6. Reporting the
review.....................................................................................
.......22
6.1 Structure for systematic review
...................................................................22
6.2 Peer Review
................................................................................ ...............
..22
7. Final remarks
...............................................................................................
........25
8.
References..............................................................................
..............................25
Appendix 1 Steps in a systematic review
................................................................27
iii
0.2 Document Version Control
Document
status
Version
Number

Date Changes from previous version
Draft 0.1 1 April 2004 None
Published 1.0 29 June 2004 Correction of typos
Additional discussion of
problems of assessing evidence
Section 7 “Final Remarks”
added.
iv
0.3 Executive Summary
The objective of this report is to propose a guideline for
systematic reviews
appropriate for software engineering researchers, including PhD
students. A
systematic review is a means of evaluating and interpreting all
available research
relevant to a particular research question, topic area, or
phenomenon of interest.
Systematic reviews aim to present a fair evaluation of a
research topic by using a
trustworthy, rigorous, and auditable methodology.
The guideline presented in this report was derived from three
existing guidelines used
by medical researchers. The guideline has been adapted to

reflect the specific
problems of software engineering research.
The guideline covers three phases of a systematic review:
planning the review,
conducting the review and reporting the review. It is at a
relatively high level. It does
not consider the impact of question type on the review
procedures, nor does it specify
in detail mechanisms needed to undertake meta-analysis.
1
1. Introduction
This document presents a general guideline for undertaking
systematic reviews. The
goal of this document is to introduce the concept of rigorous
reviews of current
empirical evidence to the software engineering community. It is
aimed at software
engineering researchers including PhD students. It does not
cover details of meta-
analysis (a statistical procedure for synthesising quantitative
results from different
studies), nor does it discuss the implications that different types
of systematic review
questions have on systematic review procedures.
The document is based on a review of three existing guidelines
for systematic
reviews:
1. The Cochrane Reviewer’s Handbook [4].

2. Guidelines prepared by the Australian National Health and
Medical Research
Council [1] and [2].
3. CRD Guidelines for those carrying out or commissioning
reviews [12].
In particular the structure of this document owes much to the
CRD Guidelines.
All these guidelines are intended to aid medical researchers.
This document attempts
to adapt the medical guidelines to the needs of software
engineering researchers. It
discusses a number of issues where software engineering
research differs from
medical research. In particular, software engineering research
has relatively little
empirical research compared with the large quantities of
research available on
medical issues, and research methods used by software
engineers are not as rigorous
as those used by medical researchers.
The structure of the report is as follows:
1. Section 2 provides an introduction to systematic reviews as a
significant
research method.
2. Section 3 specifies the stages in a systematic review.
3. Section 4 discusses the planning stages of a systematic
review
4. Section 5 discusses the stages involved in conducting a
systematic review
5. Section 6 discusses reporting a systematic review.

2. Systematic Reviews
A systematic literature review is a means of identifying,
evaluating and interpreting
all available research relevant to a particular research question,
or topic area, or
phenomenon of interest. Individual studies contributing to a
systematic review are
called primary studies; a systematic review is a form a
secondary study.
2.1 Reasons for Performing Systematic Reviews
There are many reasons for undertaking a systematic review.
The most common
reasons are:
• To summarise the existing evidence concerning a treatment or
technology e.g. to
summarise the empirical evidence of the benefits and
limitations of a specific
agile method.
2
• To identify any gaps in current research in order to suggest
areas for further
investigation.
• To provide a framework/background in order to appropriately
position new
research activities.

However, systematic reviews can also be undertaken to examine
the extent to which
empirical evidence supports/contradicts theoretical hypotheses,
or even to assist the
generation of new hypotheses (see for example [10]).
2.2 The Importance of Systematic Reviews
Most research starts with a literature review of some sort.
However, unless a literature
review is thorough and fair, it is of little scientific value. This
is the main rationale for
undertaking systematic reviews. A systematic review
synthesises existing work in
manner that is fair and seen to be fair. For example, systematic
reviews must be
undertaken in accordance with a predefined search strategy. The
search strategy must
allow the completeness of the search to be assessed. In
particular, researchers
performing a systematic review must make every effort to
identify and report research
that does not support their preferred research hypothesis as well
as identifying and
reporting research that supports it.
2.3 Advantages and disadvantages
Systematic reviews require considerably more effort than
traditional reviews. Their
major advantage is that they provide information about the
effects of some
phenomenon across a wide range of settings and empirical
methods. If studies give
consistent results, systematic reviews provide evidence that the
phenomenon is robust

and transferable. If the studies give inconsistent results, sources
of variation can be
studied.
A second advantage, in the case of quantitative studies, is that it
is possible to
combine data using meta-analytic techniques. This increases the
likelihood of
detecting real effects that individual smaller studies are unable
to detect. However,
increased power can also be a disadvantage, since it is possible
to detect small biases
as well as true effects.
2.4 Feature of Systematic Reviews
Some of the features that differentiate a systematic review from
a conventional
literature review are:
• Systematic reviews start by defining a review protocol that
specifies the research
question being addressed and the methods that will be used to
perform the review.
• Systematic reviews are based on a defined search strategy that
aims to detect as
much of the relevant literature as possible.
• Systematic reviews document their search strategy so that
readers can access its
rigour and completeness.
• Systematic reviews require explicit inclusion and exclusion
criteria to assess each
potential primary study.

• Systematic reviews specify the information to be obtained
from each primary
study including quality criteria by which to evaluate each
primary study.
• A systematic review is a prerequisite for quantitative meta-
analysis
3
3. The Review Process
A systematic review involves several discrete activities.
Existing guidelines for
systematic reviews have different suggestions about the number
and order of activities
(see Appendix 1). This documents summarises the stages in a
systematic review into
three main phases: Planning the Review, Conducting the
Review, Reporting the
Review.
The stages associated with planning the review are:
1. Identification of the need for a review
2. Development of a review protocol.
The stages associated with conducting the review are:
1. Identification of research
2. Selection of primary studies
3. Study quality assessment
4. Data extraction & monitoring
5. Data synthesis.
Reporting the review is a single stage phase.

Each phase is discussed in detail in the following sections.
Other activities identified
in the guidelines discussed in Appendix 1 are outside the scope
of this document.
The stages listed above may appear to be sequential, but it is
important to recognise
that many of the stages involve iteration. In particular, many
activities are initiated
during the protocol development stage, and refined when the
review proper takes
place. For example:
• The selection of primary studies is governed by inclusion and
exclusion criteria.
These criteria are initially specified when the protocol is
defined but may be
refined after quality criteria are defined.
• Data extraction forms initially prepared during construction of
the protocol will
be amended when quality criteria are agreed.
• Data synthesis methods defined in the protocol may be
amended once data has
been collected.
The systematic reviews road map prepared by the Systematic
Reviews Group at
Berkley demonstrates the iterative nature of the systematic
review process very
clearly [15].
4. Planning

4.1 The need for a systematic review
The need for a systematic review arises from the requirement of
researchers to
summarise all existing information about some phenomenon in a
thorough and
unbiased manner. This may be in order to draw more general
conclusion about some
phenomenon than is possible from individual studies, or as a
prelude to further
research activities.
4
Prior to undertaking a systematic review, researchers should
ensure that a systematic
review is necessary. In particular, researchers should identify
and review any existing
systematic reviews of the phenomenon of interest against
appropriate evaluation
criteria. CRC [12] suggests the following checklist:
• What are the review’s objectives?
• What sources were searched to identify primary studies? Were
there any
restrictions?
• What were the inclusion/exclusion criteria and how were they
applied?
• What criteria were used to assess the quality of primary
studies and how were
they applied?

• How were the data extracted from the primary studies?
• How were the data synthesised? How were differences
between studies
investigated? How were the data combined? Was it reasonable
to combine the
studies? Do the conclusions flow from the evidence?
From a more general viewpoint, Greenlaugh [9] suggests the
following questions:
• Can you find an important clinical question, which the review
addressed?
(Clearly, in software engineering, this should be adapted to
refer to an important
software engineering question.)
• Was a thorough search done of the appropriate databases and
were other
potentially important sources explored?
• Was methodological quality assessed and the trials weighted
accordingly?
• How sensitive are the results to the way that the review has
been done?
• Have numerical results been interpreted with common sense
and due regard to the
broader aspects of the problem?
4.2 Development of a Review Protocol
A review protocol specifies the methods that will be used to
undertake a specific

systematic review. A pre-defined protocol is necessary to
reduce the possibility
researcher bias. For example, without a protocol, it is possible
that the selection of
individual studies or the analysis may be driven by researcher
expectations. In
medicine, review protocols are usually submitted to peer
review.
The components of a protocol include all the elements of the
review plus some
additional planning information:
• Background. The rationale for the survey.
• The research questions that the review is intended answer.
• The strategy that will be used to search for primary studies
including search terms
and resources to be searched, resources include databases,
specific journals, and
conference proceedings. An initial scoping study can help
determine an
appropriate strategy.
• Study selection criteria and procedures. Study selection
criteria determine criteria
for including in, or excluding a study from, the systematic
review. It is usually
helpful to pilot the selection criteria on a subset of primary
studies. The protocol
should describe how the criteria will be applied e.g. how many
assessors will
evaluate each prospective primary study, and how
disagreements among assessors
will be resolved.

5
• Study quality assessment checklists and procedures. The
researchers should
develop quality checklists to assess the individual studies. The
purpose of the
quality assessment will guide the development of checklists.
• Data extraction strategy. This should define how the
information required from
each primary study would be obtained. If the data require
manipulation or
assumptions and inferences to be made, the protocol should
specify an
appropriate validation process.
• Synthesis of the extracted data. This should define the
synthesis strategy. This
should clarify whether or not a formal meta-analysis is intended
and if so what
techniques will be used.
• Project timetable. This should define the review plan.
4.2.1 The Research Question
4.2.1.1 Question Types
The most important activity during protocol is to formulate the
research question. The
Australian NHMR Guidelines [1] identify six types of health
care questions that can
be addressed by systematic reviews:
1. Assessing the effect of intervention.
2. Assessing the frequency or rate of a condition or disease.

3. Determining the performance of a diagnostic test.
4. Identifying aetiology and risk factors.
5. Identifying whether a condition can be predicted.
6. Assessing the economic value of an intervention or
procedure.
In software engineering, it is not clear what the equivalent of a
diagnostic test would
be, but the other questions can be adapted to software
engineering issues as follows:
• Assessing the effect of a software engineering technology.
• Assessing the frequency or rate of a project development
factor such as the
adoption of a technology, or the frequency or rate of project
success or failure.
• Identifying cost and risk factors associated with a technology.
• Identifying the impact of technologies on reliability,
performance and cost
models.
• Cost benefit analysis of software technologies.
Medical guidelines often provide different guidelines and
procedures for different
types of question. This document does not go to this level of
detail.
The critical issue in any systematic review is to ask the right
question. In this context,
the right question is usually one that:
• Is meaningful and important to practitioners as well as
researchers. For example,
researchers might be interested in whether a specific analysis
technique leads to a

significantly more accurate estimate of remaining defects after
design inspections.
However, a practitioner might want to know whether adopting a
specific analysis
technique to predict remaining defects is more effective than
expert opinion at
identifying design documents that require re-inspection.
• Will lead either to changes in current software engineering
practice or to
increased confidence in the value of current practice. For
example, researchers
6
and practitioners would like to know under what conditions a
project can safely
adopt agile technologies and under what conditions it should
not.
• Identify discrepancies between commonly held beliefs and
reality.
Nonetheless, there are systematic reviews that ask questions
that are primarily of
interest to researchers. Such reviews ask questions that identify
and/or scope future
research activities. For example, a systematic review in a PhD
thesis should identify
the existing basis for the research student’s work and make it
clear where the
proposed research fits into the current body of knowledge.
4.2.1.2 Question Structure

Medical guidelines recommend considering a question from
three viewpoints:
• The population, i.e. the people affected by the intervention.
• The interventions usually a comparison between two or more
alternative
treatments.
• The outcomes, i.e. the clinical and economic factors that will
be used to compare
the interventions.
In addition, study designs appropriate to answering the review
questions may be
identified.
4.2.1.2.1 Population
In software engineering experiments, the populations might be
any of the following:
• A specific software engineering role e.g. testers, managers.
• A type of software engineer, e.g. a novice or experienced
engineer.
• An application area e.g. IT systems, command and control
systems.
A question may refer to very specific population groups e.g.
novice testers, or
experienced software architects working on IT systems. In
medicine the populations
are defined in order to reduce the number of prospective
primary studies. In software
engineering far less primary studies are undertaken, thus, we
may need to avoid any
restriction on the population until we come to consider the
practical implications of
the systematic review.

4.2.1.2.2 Intervention
Interventions will be software technologies that address specific
issues, for example,
technologies to perform specific tasks such as requirements
specification, system
testing, or software cost estimation.
4.2.1.2.3 Outcomes
Outcomes should relate to factors of importance to practitioners
such as improved
reliability, reduced production costs, and reduced time to
market. All relevant
outcomes should be specified. For example, in some cases we
require interventions
that improve some aspect of software production without
affecting another e.g.
improved reliability with no increase in cost.
A particular problem for software engineering experiments is
the use of surrogate
measures for example, defects found during system testing as a
surrogate for quality,
7
or coupling measures for design quality. Studies that use
surrogate measures may be
misleading and conclusions based on such studies may be less
robust.
4.2.1.2.4 Experimental designs
In medical studies, researches may be able to restrict systematic
reviews to primary of

studies of one particular type. For example, Cochrane reviews
are usually restricted to
randomised controlled trials (RCTs). In other circumstances, the
nature of the
question and the central issue being addressed may suggest that
certain studies design
are more appropriate than others. However, this approach can
only be taken in a
discipline where the amount of available research is a major
problem. In software
engineering, the paucity of primary studies is more likely to be
the problem for
systematic reviews and we are more likely to need protocols for
aggregating
information from studies of widely different types. A starting
point for such
aggregation is the ranking of primary studies of different types;
this is discussed in
Section 5.3.1.
4.2.2 Protocol Review
The protocol is a critical element of any systematic review.
Researchers must agree a
procedure for reviewing the protocol. If appropriate funding is
available, a group of
independent experts should be asked to review the protocol. The
same experts can
later be asked to review the final report.
PhD students should present their protocol to their supervisors
for review and
criticism.
5. Conducting the review
Once the protocol has been agreed, the review proper can start.

This involves:
1. Identification of research
2. Selection of studies
3. Study quality assessment
4. Data extraction and monitoring progress
5. Data synthesis
Each of these stages will be discussed in this section. Although
some stages must
proceed sequentially, some stages can be undertaken
simultaneously.
5.1 Identification of Research
The aim of a systematic review is to find as many primary
studies relating to the
research question as possible using an unbiased search strategy.
For example, it is
necessary to avoid language bias. The rigour of the search
process is one factor that
distinguishes systematic reviews from traditional reviews.
5.1.1 Generating a search strategy
It is necessary to determine and follow a search strategy. This
should be developed in
consultation with librarians. Search strategies are usually
iterative and benefit from:
• Preliminary searches aimed at both identifying existing
systematic reviews and
assessing the volume of potentially relevant studies.
8

• Trial searchers using various combinations of search terms
derived from the
research question
• Reviews of research results
• Consultations with experts in the field
A general approach is to break down the question into
individual facets i.e.
population, intervention, outcomes, study designs. Then draw
up a list of synonyms,
abbreviations, and alternative spellings. Other terms can be
obtained by considering
subject headings used in journals and data bases. Sophisticated
search strings can then
be constructed using Boolean AND’s and OR’s.
Initial searches for primary studies can be undertaken initially
using electronic
databases but this is not sufficient. Other sources of evidence
must also be searched
(sometimes manually) including:
• Reference lists from relevant primary studies and review
articles
• Journals (including company journals such as the IBM Journal
of Research and
Development), grey literature (i.e. technical reports, work in
progress) and
conference proceedings
• Research registers
• The Internet.
It is also important to identify specific researchers to approach
directly for advice on

appropriate source material.
Medical researchers have developed pre-packaged research
strategies. Software
Engineering Researchers need to develop and publish such
strategies including
identification of relevant electronic databases.
5.1.2 Publication Bias
Publication bias refers to the problem that positive results are
more likely to be
published than negative results. The concept of positive or
negative results sometimes
depends on the viewpoint of the researcher. (For example,
evidence that full
mastectomies were not always required for breast cancer was
actually an extremely
positive result for breast cancer sufferers). However,
publication bias remains a
problem particularly for formal experiments, where failure to
reject the null
hypothesis is considered less interesting than an experiment that
is able to reject the
null hypothesis.
Publication bias can lead to systematic bias in systematic
reviews unless special
efforts are made to address this problem. Many of the standard
search strategies
identified above are used to address this issue including:
• Scanning the grey literature
• Scanning conference proceedings
• Contacting experts and researches working in the area and
asking them if they
know of any unpublished results.

In addition, statistical analysis techniques can be used to
identify the potential
significance of publication bias (se Section 5.5.5).
9
5.1.3 Bibliography Management and Document Retrieval
Bibliographic packages such as Reference Manager or Endnote
are very useful to
manage the large number of reference that can be obtained from
a …

ORIGINAL ARTICLEBig data analytics capabilities a systema.docx

Recommended

Recommended

More Related Content

Similar to ORIGINAL ARTICLEBig data analytics capabilities a systema.docx

Similar to ORIGINAL ARTICLEBig data analytics capabilities a systema.docx (20)

More from aman341480

More from aman341480 (20)

Recently uploaded

Recently uploaded (20)

ORIGINAL ARTICLEBig data analytics capabilities a systema.docx