Iwsm2014 evaluating software product quality (ali idri)

+
Evaluating Software Product Quality:
A systematic mapping study
Sofia Ouhbi, Ali Idri, José Luis Fernández-Alemán, Ambrosio Toval
IWSM MENSURA Conference
October 6-8,2014, Rotterdam
Presented by: Ali Idri

+ Presentation Outline
1. Motivations and Objectives
2. Software Product Quality: Introduction
3. Research Methodology
4. Results
5. Principal findings
6. Implications
7. Conclusion and Future work
2
Evaluating Software Product Quality: A systematic mapping study

+
Motivations and Objectives
10/29/2014 Evaluating Software Product Quality: A systematic mapping study 3

+ Motivations and Objectives
4
Motivations
• Obtain the current approaches used to SPQE
• No systematic mapping study of SPQE approaches has been
published
Objectives
• Summarize the existing SPQE approaches and whether or not
these approaches are based on well-known SQ models
• Discover if SPQE is done through one or many SQ characteristics
• Identify the research types and empirical types that exist in SQPE
literature
• List the data sets used in the evaluation of the SQPE approaches
Evaluating Software Product Quality: A systematic mapping study 10/29/2014

+
Software Product Quality
Introduction

+ Software Product Quality
6
“Although quality is pursued in order to satisfy users, it is
important to note that the quality desired by users is not
universal but rather subject to constant change”
SQuBOK, JUSE 2007
Software product quality could be defined as “the totality of
characteristics of an entity that bear on it’s ability to satisfy
stated and implied needs”
ISO/IEC 9126-1

+ Software Product Quality
o McCall
o Boehm
o Dromey
o ISO/IEC 9126 standard, which has been
replaced with ISO/IEC 25010
7
 Several SQ models and standards with which to improve Software
Product Quality have been proposed such as:
 These models have some common SQ characteristics, such as:
o Efficiency
o Reliability
o Portability
o Maintainability

+ ISO/IEC 250nn (SQuaRE)
8
2501n
Quality Model Division
2500n
Quality Management
Division
2502n
Quality Measurement
Division
2503n
Quality
Requirement
Division
2504n
Quality
Evaluation
Division

+
Research Methodology
Systematic mapping study

+ Systematic mapping study
10
 The systematic mapping study principal goal is to provide an
overview of a research area, and identify the quantity and type of
research and results available within it.
This method focuses on classification, conducting a thematic
analysis and identifying publication fora.
 A mapping process consists of three activities:
1) the search for relevant publications,
2) the definition of a classification scheme,
3) the mapping of publications.

+ Systematic mapping process
11
Process Steps
Definition of
Research
Questions
Conduct
Search
Screening of
Papers
Key wording
using
Abstracts
Data Extraction
and Mapping
Process
Review Scope All papers
Relevant
Papers
Classification
Scheme
Systematic
Map
Outcomes

+ Systematic mapping questions
• Which publication MQ1 channels are the main targets for SPQE research?
• How has the frequency of SPQE research dissemination changed over
time? MQ2
MQ3 • In which research types are SPQE papers classified?
MQ4 • Which approaches have been used for SPQE?
MQ5 • Are the SPQE selected studies empirically validated?
MQ6 • What are the datasets that were used in SPQE literature?
MQ7 • Which artifacts have been reported in SPQE selected studies?
• What are the well-known SQ models that have been mentioned in SPQE
literature? MQ8
MQ9 • Which characteristics were used to evaluate SPQ?
12

+ Search string
“Software quality" AND (evaluat* OR measur* OR assess*) AND
(technique* OR approach* OR method* OR model* OR framework*
OR tool* OR data mining OR learning OR artificial intelligence OR
pattern recognition OR analogy OR case based reasoning OR
nearest neighbo* OR decision tree* OR classification tree* OR
neural net* OR genetic programming OR genetic algorithm* OR
bayesian belief network* OR bayesian net* OR association rule* OR
support vector machine* OR regression OR fuzzy logic)
13

+ Study selection
IC1
• The studies that address evaluation, measurement or assessment of the
quality of software product in overall or through SQ characteristics
EC1
• Papers that focus on system quality not on SPQ
EC2
• Papers whose subject was one or many SQ characteristics (e.g.
maintainability, reliability) which were not used in the evaluation of SPQ
EC3
• Papers that were published before the nineties and after 2013
14

+ Study selection
15

+ Data extraction
16
Empirical type
Research type
Artifact
SQPE Approach
-
- Data mining technique
- Process: A serie of
actions, or functions leading
to an SPQE result and
performing operations on
data.
- Method: A regular and
systematic means of
accomplishing SPQE.
- Tool-based technique: A
technique based on a
software tool to accomplish
SPQE tasks.
- Model: A system
representation that allows
SPQE to be investigated
through a hierarchical
structure.
- Framework: A real or
conceptual structure
intended to serve as a
support or guide for SPQE.
- Other, e.g. guidelines.
- Evaluation research:
Existing SPQE
approaches are
implemented in practice
and an evaluation of
them is conducted.
- Solution proposal:
An SPQE solution is
proposed. This solution
may be a new SPQE
approach or a
significant extension of
an existing approach.
- Other, e.g. experience
paper, review.
- Case study: An empirical
inquiry that investigates an
SPQE approach within its real-life
context.
- Survey: A method for
collecting quantitative
information concerning an
SPQE approach.
- Experiment: An empirical
method applied under controlled
conditions, in order to evaluate
a SPQE approach.
- History-based evaluation:
Studies evaluating SPQE
approaches in previously
completed software projects.
- Other
- Documentation
- Design module
- Source code
- Other

+
Results

+ MQ1. Publication channels
Publication Channel Total
Journal 33
Conference 12
Workshop 6
Symposium 3
Other 3
18
 The Journal of Systems and Software is a recurrent publication source for
SPQE techniques, followed by the Information and Software
Technology Journal.
• There is no obvious conference target that attracts publication concerning
SPQE approaches

+ MQ2. Publication trend
19
25
20
15
10
5
0
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
No of papers
Selected studies Empirical studies

+ MQ3,4,5. Research types, approaches and empirical types
20
20 4 4 4 1
1
7 1 4 3
3 16 2 10
2 9 3
2
2 4
1
1
8
2
Research
type
Solution
proposal
68%
Evaluation
research
28%
SPQE
Approaches
Empirical
type
Case
study
18%
Experiment
9%
History-based
evaluation
44%
Survey
9%
Theory
21%
Data mining
techniques
47%
Framework
7%
Method
11%
Model
14%
Other
7%
Process
5%
Tool
9%
Review
4%

+ MQ4. SPQE Data Mining Approaches
21
DT
3%
GP
3%
Learning
4%
ANN
7%
BN
7%
Classification
7%
Clustering
11%
Pattern
matching
11%
Fuzzy
29%
Regression
18%

+ MQ6. Data Sets
22
0 5 10 15 20 25 30
History-based evaluation
Experiment
Case study
DSP NASA OSS Other

+ MQ7. Artifacts
23
0 5 10 15 20 25 30 35 40 45
Documentations
Design module
Source code
Papers reporting only one artifact Papers reporting more than one artifact

+ MQ8. SQ Models
24
 around 50% of SPQE papers do not cite any well-known SQ model
ISO/ IEC
9126
59%
Dromey
10%
Other
12%
ISO/ IEC
25010
2%
Boehm
10%
McCall
7%

+ MQ9. SQ Characteristics
25
 25% of SPQE papers did not mention any SQ characteristics
 The main characteristic used in the evaluation of SPQ in literature is
reliability including fault-proneness, followed by maintainability and
functionality
 23% of the selected studies had used all the ISO/IEC 9126
characteristics to evaluate SPQ

+
Principal findings

+ Principal findings
27
 SPQE is taken very seriously by researchers
 The interest in evaluating SPQ has increased dramatically last year
 Researchers are still searching for good approaches with which to enhance
SPQE
 Few researchers chose to evaluate existing SPQE techniques
 The majority of SPQE approaches that were found are data mining-based
techniques, principally fuzzy logic, regression and clustering techniques
 Data used was extracted from open source projects, domain specific projects
or from existing data sets such as NASA to evaluate SPQE solutions

+ Principal findings
28
 Case studies were used to evaluate SPQE approaches and a few experiments
were conducted in the SPQE papers selected
 Source code was the main artifact concerned by SPQE approaches
 Researchers are more interested to evaluate internal SQ
 Half of the selected studies based their solutions on SQ models, particularly
the ISO/IEC 9126 standard
 50% of the selected studies have not mentioned any well-known SQ models,
however they have evaluated SQ by using an SQ characteristic
 Reliability was the most frequently reported SQ characteristic in SPQE
literature, and was in most cases achieved using fault-proneness

+
Implications

+ Implications
30
 This study will allow researchers and practitioners to discover the existing
SPQE approaches and techniques in literature.
 The empirical studies presented may provide an overview of the efficiency of
each approach.
 More studies involving recent SQ models are needed to develop approaches
that will meet SQ standards, particularly the ISO/IEC 25010 standard.
 The evaluation of SQ characteristics which have an impact on the emerging
market of mobile software applications, should receive more attention from
researchers.

+
Conclusions & Future work

+ Conclusions and Future work
32
 Our study has summarized the existing knowledge as regards SPQE
approaches
 Papers dealing with SPQE approaches between 1990 and 2013 were
identified. 57 papers were selected
 Ongoing research is based on proposing an empirical method with
which to evaluate SPQ in some specific environments

+
Thank you for your attention!
Any questions?

Iwsm2014 evaluating software product quality (ali idri)

More Related Content

What's hot

Viewers also liked

Similar to Iwsm2014 evaluating software product quality (ali idri)

More from Nesma

Recently uploaded

Iwsm2014 evaluating software product quality (ali idri)