PhD Thesis Presentation - Evaluating Requirements Engineering Tools and Catalogue-Based Reuse of Natural-Language Requirements in Global Software Engineering

Evaluating Requirements Engineering Tools and
Catalogue-Based Reuse of Natural-Language
Requirements in Global Software Engineering
A dissertation presented by
Juan Manuel Carrillo de Gea
and supervised by
Dr. Joaquín Nicolás Ros
Faculty of Computer Science
University of Murcia
Spain
5 February 2016

5 February 2016
Agenda
• Introduction
– Problem
– Hypothesis
– Goals
• Development
– Overview
– Contributions
• Conclusions
– Context
– Results
– Further work
2

5 February 2016
Agenda
• Introduction
– Problem
– Hypothesis
– Goals
• Development
– Overview
– Contributions
• Conclusions
– Context
– Results
– Further work
3

5 February 2016
• Software companies are moving towards
globally-distributed development teams
• Many benefits, but…
– Temporal, geographic, socio-cultural and linguistic
distance
– Specific challenges for Requirements Eng. (RE)
Global Software
Engineering (GSE)
Requirements
specification
Requirements
reuse
Introduction
Problem
Introduction – Problem
4

5 February 2016
• Unstructured natural language (NL) is very
common in industry
– Easier to understand than non-textual notations
– Inherently ambiguous, different interpretations!
• Exacerbated by (globally) distributed stakeholders
– Lack of tacit knowledge and informal communication
– Distance
Requirements
specification
Global Software
Engineering (GSE)
Requirements
reuse
Introduction
Problem
5

5 February 2016
• Use preexisting requirements artifacts rather
than build from scratch
–  time and effort to build software and  quality
–  time and effort to maintain software
– The earlier it is applied, the better
Requirements
reuse
Requirements
specification
Global Software
Engineering (GSE)
Introduction
Problem
6

5 February 2016
• Requirements reuse can help to interpret the
real meaning of the reqs. in a given context
by leaving little room for misunderstandings
and misconceptions in GSE
• No other proposals addressing catalogue-
based NL requirements reuse in GSE
Requirements
specification
Global Software
Engineering (GSE)
Requirements
reuse
Introduction
Hypothesis
Introduction – Hypothesis
7

5 February 2016
Requirements
specification
Global Software
Engineering (GSE)
Requirements
reuse
If adequate catalogue-based reuse techniques and an automated tool support
were defined for NL requirements specification when stakeholders are globally
distributed, then relevant gains of effectiveness and productivity in software
development projects would be obtained
Introduction
Hypothesis
Propose catalogue-based reuse techniques and an automated tool support for
NL requirements specification in GSE, and validate them empirically
HYPOTHESIS
Introduction – Hypothesis
8

5 February 2016
Requirements
specification
Global Software
Engineering (GSE)
Requirements
reuse
Introduction
Goal
Introduction – Goal
9
GOAL

5 February 2016
Provide automated support for the requirements reuse
techniques by using a prototypical tool
Devise reuse-based techniques for NL requirements
specification in GSE
Introduction
Goal refinement
Validate the techniques and the prototype empirically in a
GSE scenario
Goal 1
Goal 2
Goal 3
Introduction – Goal
10

5 February 2016
Agenda
• Introduction
– Problem
– Hypothesis
– Goals
• Development
– Overview
– Contributions
• Conclusions
– Context
– Results
– Further work
11

5 February 2016
Development
Overview: Goal 1
36
12
18
35
8
23
20
6
11
23
19
3
0
5
10
15
20
25
30
35
40
Com
m
unication
and
dista
nce
Know
ledge
m
anagem
ent and
awareness
Cultural diffe
re
nces
M
anagem
entand
pro
je
ctcoord
in
ation
Tools
whichsupport
th
eprocesses
Custo
m
er-supplier dista
nce
Risks
Safeguards
Challenges for RE
Risks and safeguards
for RE in GSE
• State-of-the-art on RE in GSE
– Specific challenges for RE
• Inherent collaborative nature
– Risks and safeguards for RE in GSE
• SLR and quantitative approach
Development – Overview
12

5 February 2016
Development
Overview: Goal 1
PANGEA reuse-based
techniques
PANGEA repository of
reusable reqs.
• Reuse-based techniques for NL reqs.
– In the context of the PANGEA method
• Requirements reference model
• Repository of reusable catalogues
13

5 February 2016
Development
Overview: Goal 2
State-of-the-practice
on RE tools
Classification
framework
14
• Classification framework for RE tool
capabilities
– ISO/IEC TR 24766
• State-of-the-practice on commercial RE tools
– DESMET and survey

5 February 2016
• Prototypical tool support for the reuse-based
techniques for NL reqs.
– Needs analysis and SRS
– High-level design
– PANTALASA RE
tool (proof-of-
concept)
• Social networking software, (semantic) wiki and CMS
Development
Overview: Goal 2
PANTALASA RE tool
(proof-of-concept)
15

5 February 2016
• Empirical validation of the proposal
– Experiment planning,
design and execution
• Two experiments
• Distributed university students
– Nearshore GSE scenario
• Rabat (Morocco) and Murcia (Spain)
– Data analysis
Development
Overview: Goal 3
Validation (two
experiments)
16

5 February 2016
Development
Overview
State-of-the-practice
on RE tools
Classification
framework
36
12
18
35
8
23
20
6
11
23
19
3
0
5
10
15
20
25
30
35
40
Com
m
unication
and
dista
nce
Know
ledge
m
anagem
ent and
awareness
Cultural diffe
re
nces
M
anagem
entand
pro
je
ctcoord
in
ation
Tools
whichsupport
th
eprocesses
Custo
m
er-supplier dista
nce
Risks
Safeguards
Challenges for RE
Risks and safeguards
for RE in GSE
PANGEA reuse-based
techniques
PANGEA repository of
reusable reqs.
PANTALASA RE tool
(proof-of-concept)
Validation (two
experiments)
17

5 February 2016
• Systematic Literature Review (SLR)
– 148 risks and 90 safeguards
– 6 categories (concerns):
• Communication and distance
• Knowledge management and awareness
• Cultural differences
• Management and project coordination
• Tools which support the process
• Customer-supplier distance
Risks and safeguards for RE in GSE
A contribution for Goal 1
Development – Contributions
18

5 February 2016
• Customizable repository
– Help in finding strategies to face problems that
can arise in GSE
– Specially useful for organisations without
previous experience in GSE
– Initial and extendable knowledge base
19

5 February 2016
20

5 February 2016
• Quantitative analysis of the results of the SLR
– Relationships between canonical risks and
safeguards
– The ratio assesses each canonical risk
• 𝑟𝑎𝑡𝑖𝑜 𝑟𝑠𝑘𝑖 = 𝑠𝑎𝑓𝑒𝑔𝑢𝑎𝑟𝑑𝑠 𝑟𝑠𝑘𝑖 𝑟𝑖𝑠𝑘 𝑟𝑠𝑘𝑖
– ratio > 1  the ith canonical risk is well covered by canonical
safeguards
– 0 ≤ ratio < 1  the ith canonical risk is not well covered
– ratio = 1  balanced scenario
21

5 February 2016
• Quantitative analysis of the results of the SLR
– Bivariate correlation tests (Pearson correlation
coefficient) measures linear dependence b/w the
risks and the safeguards of each concern
• r-value indicates strength and direction (±)
1. Communication and distance (0.017)
2. Knowledge management and awareness (0.937)
3. Cultural differences (-1.000**)
4. Management and project coordination (0.411)
5. Tools which support the process (1.000**)
6. Customer-supplier distance (0.644)
22

5 February 2016
• Most populated canonical risks:
1. Conflict between client and vendor
2. Communication problems, Lack of face-to-face meetings
and Lack of informal communication
3. Not well-defined processes
• Risks that are not faced by any safeguards:
– Lack of speed when communicating changes in
requirements
– Difficulty in identifying the key users and High number of
users and requirements
• Uncovered specific challenges for RE in GSE
– Our proposal: a reqs. reuse approach
A contribution for Goal 1 – Conclusions
23

5 February 2016
NL reqs. reuse techniques in GSE
Security
Privacy
i18n
CARE
Development
for reuse
Development
with reuse
Guidelines,
recommendations,
regulations, laws,
standards
Domain knowledge
(experts, previous
projects, docs…) IRS SRS STS
SyRS SyTS
24

5 February 2016
Security
Privacy
i18n
CARE
Development
for reuse
Development
with reuse
Guidelines,
recommendations,
regulations, laws,
standards
Domain knowledge
(experts, previous
SyRS SyTS
Elicitation Analysis and
negotiation
Validation Documentation
or specification
(for reuse)
25

5 February 2016
Security
Privacy
i18n
CARE
Development
for reuse
Development
with reuse
Guidelines,
recommendations,
regulations, laws,
standards
Domain knowledge
(experts, previous
SyRS SyTS
Elicitation Analysis and
negotiation
or specification
(for reuse)
Elicitation
(with reuse)
Analysis and
negotiation
(with reuse)
or specification
(with reuse)
26

5 February 2016
• PANGEA extends the SIREN reqs. reuse method to GSE
starting from the repository of risks and safeguards
– Changes in the process model
• New activities and tasks
• New set of roles
– Changes in part of the techniques
• The requirements reference model
– New requirements attributes
– New attribute values
– New traceability relationships
– No changes in another part of the techniques
• The documents hierarchy
• The repository of reusable catalogues
27

5 February 2016
28

5 February 2016
• Requirements reference model
– SIREN metadata
• Requirements attributes (e.g. Text, UniqueIdentifier)
• Parameterised requirements
• Traceability model (i.e. Parent-child, Requires,
RelatedTo, MutuallyExclusive)
– PANGEA metadata
• SIREN requirements +
• Additional attributes (i.e. SourceTeam, SourceAnalyst,
DestinationTeam) and attribute values (i.e. State:
Deallocated discussion pending)
29

5 February 2016
• We postulate that certain problems in GSE could
be mitigated by means of reuse:
– Knowledge availability (gathering and understanding)
– Communication (overhead, misunderstandings,
interpretation)
• PANGEA is a reuse-based RE method for GSE
– Specifies knowledge in the form of NL reqs.
– Encompasses process model, requirements reference
model and RE tool (PANTALASA)
• No other proposals are aimed at both GSE and
catalogue-based requirements reuse
30

5 February 2016
State-of-the-practice on RE tools
Define a classification framework for RE tool
capabilities
Collect data about RE tools
Depict the state-of-the-art on RE tools (i):
descriptive analysis
Define a statistical approach to assess and
categorise RE tools
Depict the state-of-the-art on RE tools (ii):
inferential analysis
31

5 February 2016
Category (classification
framework)
No. of
capabilities
1. Elicitation 20
2. Analysis 11
3. Specification 7
4. Modelling 13
5. V&V 11
6. Management 18
7. Traceability 29
8. Collaboration and GSD 42
9. Other tool capabilities 17
Total 168
• Goal/Question Metric
(GQM) framework
– Goal: The goal is to depict
the state-of-the-art of RE
tools
– Question: Do current RE
tools adequately support
the RE process?
– Metric: The capabilities
supported by the RE tools
within distinct categories
of features in an ISO/IEC TR
24766-based classification
framework
32

5 February 2016
• DESMET method for evaluating SE methods and
tools
– Feature Analysis─Survey
• 188-item (168 tech. + 20 gen.) questionnaire
– ISO/IEC TR 24766 categories of features (6) +
Modelling + Traceability + Collaboration and GSD
• Web-based survey using LimeSurvey
– 38 participants out of 94 RE tool vendors invited
• Another survey already performed
– From CFv1 to CFv2: CFv1 + reuse
– Data analysis: work-in-progress
33

5 February 2016
• Bivariate correlation tests
(Pearson correlation
coefficient)
– Strong direct correlation
between each distinct
category of features
– Cost per individual license–
• Analysis (0.336*)
• Modelling (0.404*)
• Traceability (0.329*)
• Specification (0.545**)
• Global score (0.358)
– Number of licenses in use–
• Other tool capab. (0.513**)
• Cost per license (0.243)
• Global score (0.183)
34

5 February 2016
• Goal/Question Metric (GQM)
framework
– Goal: The goal is to investigate the
commonalities and differences b/w
current RE tools empirically,
through the assessment of their
features, the application of a
clustering process to group them,
and the execution of a formal
experiment
– Question: Have the clusters of RE
tools got significant differences in
coverage of features?
– Metric: The capabilities supported
by the RE tools within the distinct
categories of features included in
our classification framework
35

5 February 2016
• The RE process is well-covered by RE tools
– Association b/w the scores achieved in each category
– The more expensive, the better analysis, specification,
modelling and traceability support
– No association b/w cost and global score
– No association b/w amount of licenses in use and cost nor
global score
• The quantitative approach analyses commonalities and
differences b/w RE tools
– High variability in functionality
• The RE tool landscape pointed out desirable
capabilities to take into account later on
36

5 February 2016
Tool for NL reqs. reuse in GSE
ISO/IEC TR 24766 Classification framework SRS
37

5 February 2016
Vision
(38 FEAT)
Use Case
Specification
(16 Use Cases)
Supplementary
Specification
(195 Suppl.
Requirements)
FEAT25 The system shall be able
to manage at reuse time both the
horizontal and vertical
traceability relationships that are
associated to a requirement
SUPL122 The system shall be able
to reuse requirements following
existing parent-child traceability
relationships (i.e. the child
requirement refines the parent
requirement)
UC16 Reuse Requirements from a
Catalogue
…
3.1.4 REUSE REQUIREMENTS
1. For each requirements document in
the catalogue, if the document type
does not exist in the project, the
system creates it and copies the
requirements from the requirements
catalogue to the document.
2. If the document type already exists,
the system copies the requirements
to the document.
3. In both cases, the system controls
the order of the identification
number of each reused requirement.
4. The system copies all the traceability
relationships and requirements
attributes.
5. In the case of requirements without
associated document, the system
copies them, their traceability
relationships and attributes from the
requirements catalogue to the
project.
38

5 February 2016
39

5 February 2016
Social networking sw. + wiki Adding enterprise wiki sw.
SMW
repository
SMW
PHP +
JavaScript
Go-b/w
app.
JS
Facebook
PHP
Web
browser
HTML +
JavaScript
40

5 February 2016
Content management system
41

5 February 2016
• Information sources for the PANTALASA SRS:
– Risks and safeguards for RE in GSE
– NL reqs. reuse techniques in GSE
– State-of-the-practice on RE tools (standard + classification
framework + evaluation of RE tools)
• PANTALASA is a prototypical RE tool that…
– Main features:
• Access control, roles, users, teams; Projects, catalogues, documents,
sections, requirements; Baselines; Discussion threads; Reqs. reuse
• …allowed us to test our hypothesis (proof-of-concept)
– What: catalogue-based reuse techniques and tool support
– Where: NL reqs. specification in GSE
– Why: effectiveness and productivity gains
42

5 February 2016
• Experimentation with students
1. Find a suitable GSE scenario for reqs. reuse
• Murcia-Rabat: nearshore development
– Temporal distance (different time zones)
– Physical distance (almost 700 km)
– Socio-cultural distance (linguistic/cultural differences)
2. Identify factors that could improve reqs.
• Reuse techniques, reqs. catalogues, tool support
3. Carry out the experiment
• Non-reuse versus reuse-based techniques in GSE
Empirical validation of the proposal
43

5 February 2016
Group formation
• Teams of two people
• Participation modalities:
– Global (GLO): UMU and MVU
– Co-located in Murcia (CLM)
– Co-located in Rabat (CLR)
• Working language: English
• Informal communication in
the native language (co-
located modalities)
Provided materials
1. Exercise statement
2. Group assignments
3. Summary on the
techniques and the tool
4. I18n standards (e.g.
ISO/IEC 24751)
5. I18n reqs. catalogue
6. Questionnaire
(demographics,
techniques, tool)
44

5 February 2016
Non-reuse task (example)
• Original text (Paragraph 4, Article
12, Spanish Regulation on
Security Measures):
– Only the authorized personnel in
the security document may grant,
alter or cancel authorized access to
data and resources, according to
the criteria established by
responsible for the file
• Elicited req.:
– The system shall allow authorized
personnel in the security
document, grant, alter or cancel
permissions and access privileges
of users
Reuse task (example)
• S3. The person in charge of the file will
define and verify the process of data
backup and recovery
– Article “14. Backup and recovery” from the
Spanish PDP/RMS
• C3. The person in charge of the file will
use [hardware backup] for the backup
– Refined from the Spanish PDP domain req.
S3, and included in the PDP catalogue
– Parameter [hardware backup] with
alternatives DVD, CD and USB memory
• R3. The person in charge of the file will
use DVD for the backup
– From parameterised req. C3
– Parameter instantiated with value DVD
45

5 February 2016
• Questionnaire
– Demographic information
(e.g. age, gender, course,
previous experience)
– Perception of the
techniques applied during
the tasks (e.g. difficulty,
speed, understanding)
– Perception of the
automated tool support
(e.g. ease of use,
functionality, robustness)
• Outcome measures
– Productivity,
effectiveness
– Difficulty, Speed, Quality,
Understanding
46

5 February 2016
First experiment Family of experiments
47

5 February 2016
First experiment
• In general, no significant
differences (non-reuse vs.
reuse)
– Lack of proper tool support for
searching the catalogue,
managing traceability and
variation points
• RE tool support is required if
the expected benefits of reqs.
reuse are to be achieved
• Sufficient specific training is
required to effectively apply
the reuse-based reqs.
specification techniques
Family of experiments
• Significant differences in
productivity and effectiveness
(non-reuse vs. reuse)
– Reuse provides benefits over
traditional reqs. specification
– No significant interaction
between the process (non-
reuse/reuse) and the co-factor
distribution (co-located/
global)
• The reuse techniques are not
(so much) hampered by
distance as other RE activities,
tasks and/or techniques
48

5 February 2016
Agenda
• Introduction
– Problem
– Hypothesis
– Goals
• Development
– Overview
– Contributions
• Conclusions
– Context
– Results
– Further work
49

5 February 2016
• According to our results the hypothesis is valid
– It is feasible to apply our proposal
• The catalogue-based reuse techniques are simple, practical
• The NL reqs. are widespread in industry
• Globally distributed stakeholders benefit from the
mitigation of knowledge and communication problems
• The gains of effectiveness and productivity are satisfying
Conclusions
The hypothesis revisited
HYPOTHESIS
Conclusions
50

5 February 2016
• National research projects and technology transfer
contracts
– PEGASO/PANGEA (January 2010–December 2012)
– ORIGIN. Technology transfer contract in the context of the
ORIGIN project (March 2012–October 2012)
– GEODAS/GEODAS-REQ (January 2013–December 2015)
• International mobility scholarships and research stays
– Mediterranean Office for Youth (July–November 2013)
• Mohammed V University (Rabat, Morocco)
• Dr. Ali Idri (Full Professor at the Mohammed V University)
– German Academic Exchange Service (February–April 2014)
• University of Stuttgart (Stuttgart, Germany)
• Dr. Christof Ebert (Managing Director at Vector Consulting Sevices;
Professor at the University of Stuttgart)
Conclusions
Context
Conclusions – Context
51

5 February 2016
Conclusions
Results
Contribution type Amount
Journal indexed in JCR 7 (5 + 2 accepted with revision)
International book chapter 1
National book chapter 1
International conference or workshop 4
National conference or workshop 1
Intellectual property registry 4
National research project 2
Technology transfer contract 1
International mobility scholarship 2
International research stay (≥ 3 moths) 2
Conclusions – Results
52

5 February 2016
Conclusions
Publication results
• *: accepted with revision
• **: accepted
• ***: in press
No. of contribution Contribution type Goal 1 Goal 2 Goal 3
Contribution 1* Journal article (JCR Q1) 
Contribution 2 Journal article (JCR Q1) 
Contribution 5* Journal article (JCR Q4)   
Contribution 6** Journal article (JCR Q4)  
Contribution 7*** Journal article (JCR Q2)   
Contribution 8 Intl. book chapter  
Contribution 9 Natl. book chapter  
Contribution 10 Intl. conf. or workshop 
Contribution 11 Intl. conf. or workshop  
Contribution 12 Intl. conf. or workshop   
Contribution 13 Intl. conf. or workshop  
Contribution 14 Natl. conf. or workshop 
Contribution 15 Intellect. prop. regist. 
53

5 February 2016
• Empirical studies (survey, case study and experiment)
– The survey allows to describe a situation using data
obtained from a sample of the population under study
– The experiment enables manipulation of variables and
measurement of their effects
• Systematic Literature Review (SLR)
– Rigorously synthesises the scientific literature in relation to
a question asked by the researcher
– It allows to characterise the evidence on a particular issue
and helps identify gaps in the state-of-the-art
• DESMET
– Method designed for evaluating SE methods and tools
Conclusions
Training results
54

5 February 2016
• A theory for RE tools
– Dr. Anil Thurimella (Requirements Manager at Harman Intl.;
Researcher at the Technical University of Munich)
– Commonality and variability analysis
• Data gathering process
• Clustering and statistical analysis for variations across tools
• Guidelines on making strategic decisions
– Application of a theory building framework to develop
guidelines for implementing RE tool plug-ins
• Use theory building concepts from social sciences
• Also analysis of RE tool capabilities and their grouping
• Guidelines on plug-in development
– Extend commonality and variability analysis of RE tools with a
theory building framework
Conclusions
Further work
Conclusions – Further work
55

5 February 2016
• RE tools evaluation
– Dr. Christof Ebert
– Criteria ranking
• Relative value of RE tool capabilities
• The classification framework could be too generic
• Supporting all capabilities in the CF is not optimal from
a cost/benefit perspective
– Update classification framework and dataset
– Recommendation system
• Expert guidance is usually missing or very expensive
Conclusions
Further work
56

5 February 2016
• Integration architecture between analysis and
design artefacts
– Dr. Javier Bermúdez and Dr. Jesús García Molina
(Lecturer and Full Professor respectively at the
University of Murcia)
– Automatic synchronisation b/w reqs. (what) and
design diagrams (how)
• Catalogue of reqs. templates (ReqIF representation)
• Intermediate models (MOF, Ecore representation)
• Design diagrams (UML class diagrams, sequence diagrams)
– Powerful traceability throughout lifecycle artefacts
– Data engineering domain (initially)
Conclusions
Further work
57

5 February 2016
Acknowledgments
• Software Engineering
Research Group, University
of Murcia
• Mediterranean Office for
Youth
• German Academic Exchange
Service
58

Thank you for your attention!
Questions?

5 February 2016
Results – Journals indexed in JCR
• Juan Manuel Carrillo de Gea, Joaquín Nicolás, José Luis Fernández Alemán,
Ambrosio Toval, Christof Ebert, and Aurora Vizcaíno. Requirements engineering
tools. IEEE Softw., 28(4):86–91, Jul./Aug. 2011. ISSN 0740-7459.
Ambrosio Toval, Christof Ebert, and Aurora Vizcaíno. Requirements engineering
tools: capabilities, survey and assessment. Inf. Softw.Technol., 54(10):1142–1157,
Oct. 2012. ISSN 0950-5849.
Ambrosio Toval, Christof Ebert, and Aurora Vizcaíno. Commonalities and
differences between requirements engineering tools: a quantitative approach.
Comp. Sci. Inf. Syst., 12(1):257–288, Jan. 2015. ISSN 1820-0214.
Ambrosio Toval, Sofia Ouhbi, and Ali Idri. Co-located and distributed natural-
language requirements specification: traditional versus reuse-based techniques. J.
Softw.-Evol. Proc., 28(3):205–227, Mar. 2016. ISSN 2047-7473.
• José Luis Fernández Alemán, Juan Manuel Carrillo de Gea, Joaquín Vidal, Joaquín
Nicolás, Ambrosio Toval, and Ali Idri. Effects of using requirements catalogs on
effectiveness and productivity of requirements specification in a software project
management course. IEEE Trans. Educ., 59(2):105–118, May 2016. ISSN 0018-9359.
60

PhD Thesis Presentation - Evaluating Requirements Engineering Tools and Catalogue-Based Reuse of Natural-Language Requirements in Global Software Engineering

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (17)

Similar to PhD Thesis Presentation - Evaluating Requirements Engineering Tools and Catalogue-Based Reuse of Natural-Language Requirements in Global Software Engineering

Similar to PhD Thesis Presentation - Evaluating Requirements Engineering Tools and Catalogue-Based Reuse of Natural-Language Requirements in Global Software Engineering (20)

Recently uploaded

Recently uploaded (20)

PhD Thesis Presentation - Evaluating Requirements Engineering Tools and Catalogue-Based Reuse of Natural-Language Requirements in Global Software Engineering