SlideShare a Scribd company logo
1 of 29
Download to read offline
Technische Universität München
In Quest for Requirements Engineering Oracles
DependentVariables and Measurements for (good) RE	

Joint work with	

Jakob Mund	

Henning Femmer	

AntonioVetrò
Daniel Méndez Fernández	

!
Technische Universität München	

Germany	

!
EASE 2014	

London, UK	

13.05.2014
@mendezfe
A regular case study in requirements engineering…
Goal: Test the sensitivity of a new RE method in company context	

» Introduce a new RE method in project environment	

» Conduct action research workshops following the new method 	

» Rate the method outcome according to assessment criteria defined using GQM	

! [Mendez Fernandez et al.A Case Study on the Application of an 

Artefact-Based Requirements Engineering Approach]
A regular case study in requirements engineering…
What conclusions can I draw beyond the chosen context?	

!
» How representative are the variables / how important are they to other contexts?	

» How can the variables be efficiently measured?	

» What relations exist between the variables?
Dependent variables in requirements engineering	

Why?
Status quo in requirements engineering research
• Development of methods/tools for problems too often not properly understood	

» Control of validity / threat mitigation?	

!
Research objectives
• Get a better understanding on RE phenomena, their dependencies and their
measurability	

!
Our motivation
Understand RE phenomena and research implications on evidence-based RE research	

• Difficulties arising from cause-effect relationships 	

• Measurability and actionability	

• Implications on threat to validity (e.g. possibilities of generalisation)
Study design
Research questions
!RQ 1. Which RE-related phenomena exist, where in the project ecosystem 

do they manifest themselves, and how do they relate to each other?	

RQ 2. Are the phenomena measurable?	

RQ 3. Are the phenomena actionable?
Study design
Research questions
!
Requirements Engineering survey
NaPiRE www.re-survey.org
Globally distributed, replicated survey 

on status quo and problems in RE	

RQ 1. Which RE-related phenomena exist, where in the project ecosystem 

do they manifest themselves, and how do they relate to each other?	

RQ 2. Are the phenomena measurable?	

RQ 3. Are the phenomena actionable?
Study design
Research questions
!
Requirements Engineering survey
NaPiRE www.re-survey.org
Globally distributed, replicated survey 

on status quo and problems in RE	

RQ 1. Which RE-related phenomena exist, where in the project ecosystem 

do they manifest themselves, and how do they relate to each other?	

Researchers
Levels of dependent variables (“Dimensions”)
[Gorschek et al. Requirements Eng.: In Search of the dependent variables]
RQ 1
RQ 2. Are the phenomena measurable?	

RQ 3. Are the phenomena actionable?
RE Phenomena inferred from open 

questions on problems, 

their causes and their effects
Study design
Research questions
!
Requirements Engineering survey
NaPiRE www.re-survey.org
Globally distributed, replicated survey 

on status quo and problems in RE	

RQ 1. Which RE-related phenomena exist, where in the project ecosystem 

do they manifest themselves, and how do they relate to each other?	

Researchers
Levels of dependent variables (“Dimensions”)
[Gorschek et al. Requirements Eng.: In Search of the dependent variables]
RQ 1
RQ 2. Are the phenomena measurable?	

RQ 3. Are the phenomena actionable?
Voting of	

• Measurability (artefacts? activities? both? none?)	

• Actionability
RQ 2 & 3
RE Phenomena inferred from open 

questions on problems, 

their causes and their effects
Results	

RQ 1: Dependent variables
!
!
!
!
• 93 variables with 167 dependencies	

» Full result set: http://goo.gl/W3A1tu
Company
Project
Engineering
2
2
22
2
2
2
3
2
2
4
2
3
8
3
2
2
4
2
2
2
2
2
2
2
2
2
3
2
3
3
Requirements
Engineering
M36
Name
Actionable (A)
Measurable (on aRtefacts, on aCtivities,
on aRtefacts & aCtivities)
M03
R
M08
R
R03
R
M09
R
RP06
R
M32
C
RP02
R
M29
C
M30
R&C
M05
C
M15
M20
M13
A
R11
R18
R&C
M24
R
RP10
R&C A
M17
C
RP03
C A
M02
R A
RP08
R A
RP11
A
RP07
R
RP04
C
R01
RP05
R A
M28
RP01
R
[Results with minimum occurrence in elements > 1]
Requirements Engineering
Results (Excerpt)	

RQ 1: Dependent variables
“Incomplete requirements”
“Implicit requirements not!
made explicit”
Engineering
Requirements Engineering
Results (Excerpt)	

RQ 1: Dependent variables
“Incomplete requirements”
“Implicit requirements not!
made explicit”
Engineering
“Increased effort in testing”
…
Requirements Engineering
Results (Excerpt)	

RQ 1: Dependent variables
“Incomplete requirements”
“Implicit requirements not!
made explicit”
SW Project
Engineering
“Increased effort in testing”
…
Requirements Engineering
Results (Excerpt)	

RQ 1: Dependent variables
“Incomplete requirements”
“Implicit requirements not!
made explicit”
SW Project
Engineering
“Increased effort in testing”
…
Requirements Engineering
Results (Excerpt)	

RQ 1: Dependent variables
“Incomplete requirements”
“Implicit requirements not!
made explicit” “Change requests”
“Effort and time overrun”
“Moving targets”
SW Project
Engineering
“Increased effort in testing”
…
Requirements Engineering
Results (Excerpt)	

RQ 1: Dependent variables
“Incomplete requirements”
“Implicit requirements not!
made explicit”
“Failed acceptance”
…
“Change requests”
“Effort and time overrun”
“Moving targets”
Company “Customer (dis-)satisfaction”
SW Project
Engineering
“Increased effort in testing”
…
Requirements Engineering
Results (Excerpt)	

RQ 1: Dependent variables
“Incomplete requirements”
“Implicit requirements not!
made explicit”
“Failed acceptance”
…
“Change requests”
“Effort and time overrun”
“Moving targets”
“Incomplete requirements”
“Failed acceptance”
“Increased effort in testing”
“Effort and time overrun”
“Implicit requirements not!
made explicit”
“Customer (dis-)satisfaction”
“Change requests”
Requirements Engineering
Engineering
SW Project
Company
…
…
“Moving targets”
Results (Excerpt)	

RQ 2: Measurability
• High degree of measurability for variables in RE dimension (59% of variables for RE)	

• Unmeasurable variables comprise social phenomena (e.g.,“Weak access customer needs”)	

• Most measurable variables are measurable on basis of artefacts 

(e.g.,“Underspecified requirements”)	

• Decreasing understanding of variables and measurability the “farther away” from RE
“Incomplete requirements”
“Failed acceptance”
“Increased effort in testing”
“Effort and time overrun”
“Implicit requirements not!
made explicit”
“Customer (dis-)satisfaction”
“Change requests”
Requirements Engineering
Engineering
SW Project
Company
…
…
“Moving targets”
Critical reflection	

The good: Measurability in RE context
“Incomplete requirements”
“Failed acceptance”
“Increased effort in testing”
“Effort and time overrun”
“Implicit requirements not!
made explicit”
“Customer (dis-)satisfaction”
“Change requests”
Requirements Engineering
Engineering
SW Project
Company
…
…
“Moving targets”
Critical reflection	

The good: Measurability in RE context
• High degree of measurability for RE-related variables	

• Measurability (to a large extent) on basis of artefacts	

» Support of independent and comparable measurements 	

» Already a better understanding of RE study context and its characteristics
“Incomplete requirements”
“Failed acceptance”
“Increased effort in testing”
“Effort and time overrun”
“Implicit requirements not!
made explicit”
“Customer (dis-)satisfaction”
“Change requests”
Requirements Engineering
Engineering
SW Project
Company
…
…
“Moving targets”
Critical reflection	

The bad: Limitations in RE context
“Incomplete requirements”
“Failed acceptance”
“Increased effort in testing”
“Effort and time overrun”
“Implicit requirements not!
made explicit”
“Customer (dis-)satisfaction”
“Change requests”
Requirements Engineering
Engineering
SW Project
Company
…
…
“Moving targets”
Critical reflection	

The bad: Limitations in RE context
?
• Incomplete dependencies	

» Investigation was performed “inside-out” from RE and needs to be extended
“Incomplete requirements”
“Failed acceptance”
“Increased effort in testing”
“Effort and time overrun”
“Implicit requirements not!
made explicit”
“Customer (dis-)satisfaction”
“Change requests”
Requirements Engineering
Engineering
SW Project
Company
…
…
“Moving targets”
Critical reflection	

The bad: Limitations in RE context
• Complex (yet incomplete) system of variables	

» Too many (especially unknown) variables intersect with each other
?
?
• Incomplete dependencies	

» Investigation was performed “inside-out” from RE and needs to be extended
“Incomplete requirements”
“Failed acceptance”
“Increased effort in testing”
“Effort and time overrun”
“Implicit requirements not!
made explicit”
“Customer (dis-)satisfaction”
“Change requests”
Requirements Engineering
Engineering
SW Project
Company
…
…
“Moving targets”
Critical reflection	

The bad: Limitations in RE context
• Complex (yet incomplete) system of variables	

» Too many (especially unknown) variables intersect with each other
• Even known variables strongly depend on subjectivity	

» Requirements engineering is also about beliefs, desires, experiences, expectations and politics	

?
?
• Incomplete dependencies	

» Investigation was performed “inside-out” from RE and needs to be extended
“Incomplete requirements”
“Failed acceptance”
“Increased effort in testing”
“Effort and time overrun”
“Implicit requirements not!
made explicit”
“Customer (dis-)satisfaction”
“Change requests”
Requirements Engineering
Engineering
SW Project
Company
…
…
“Moving targets”
Critical reflection	

The ugly: No RE oracle in sight!
No “generalisability” 

(by no means)
• Complex (yet incomplete) system of variables
• Even known variables strongly depend on subjectivity	

• Incomplete dependencies
Research implications (Excerpt)
General implications on evidence-based RE research
• No generalisation possible, but first set of variables and ideas for measurability	

» Aim for universal language rather than for universal truth	

• High degree of subjectivity	

» Qualitative studies (e.g. expert judgment)
Research implications (Excerpt)
General implications on evidence-based RE research
• No generalisation possible, but first set of variables and ideas for measurability	

» Aim for universal language rather than for universal truth	

• High degree of subjectivity	

» Qualitative studies (e.g. expert judgment)
Implications on research on RE methodologies
• Inference of first improvement goals	

» Calibrate… 	

– Study designs according to variables and their measurement 	

– Variables if aiming for tackling “commonly accepted” problems
Threats to Validity
• Variables are incomplete (needless to say)	

• Classification….	

• to some extent with a high disagreement	

• not the only possible way (and it’s positivistic)	

!
!
• However: It is a way to learn more and we are
confident to further continue it!
!
Full data disclosed to:
http://goo.gl/W3A1tu
!
• raw data
• full graphs
• R scripts
• …
@mendezfe
• Next replication of NaPiRE ongoing (www.re-survey.org)	

» Gather more RE phenomena 	

• from NaPiRE 	

• from other sources (e.g. via longitudinal studies)	

!
!
!
Future work
daniel.Mendez@tum.de
Let’s build a big picture of RE variables together!
Backup
* Cohen’s Kappa values for classification
A phenomenon is measurable, if its understanding is 

sufficiently mature such that 	

(i)  an existing or anticipated measure, i.e., objective map- 

ping to mathematical objects	

(ii) can efficiently (e.g., in justifiable time) and effectively 

(i.e., preserving empirical observations) capture the 

phenomenon 	

(iii)  under practical conditions and when applied on study 

objects which can be expected to be present in a soft- 

ware project ecosystem.

More Related Content

Viewers also liked

Case Studies in Industry - What We Have Learnt
Case Studies in Industry - What We Have LearntCase Studies in Industry - What We Have Learnt
Case Studies in Industry - What We Have LearntDaniel Mendez
 
Where do we stand in Requirements Engineering Improvement Today? First Result...
Where do we stand in Requirements Engineering Improvement Today? First Result...Where do we stand in Requirements Engineering Improvement Today? First Result...
Where do we stand in Requirements Engineering Improvement Today? First Result...Daniel Mendez
 
A Case Study on Artefact-based RE Improvement in Practice
A Case Study on Artefact-based RE Improvement in PracticeA Case Study on Artefact-based RE Improvement in Practice
A Case Study on Artefact-based RE Improvement in PracticeDaniel Mendez
 
Case studies in industry - fundamentals and lessons learnt
Case studies in industry - fundamentals and lessons learntCase studies in industry - fundamentals and lessons learnt
Case studies in industry - fundamentals and lessons learntDaniel Mendez
 
Who cares about Software Process Modelling? A First Investigation about the P...
Who cares about Software Process Modelling? A First Investigation about the P...Who cares about Software Process Modelling? A First Investigation about the P...
Who cares about Software Process Modelling? A First Investigation about the P...Daniel Mendez
 
Naming the Pain in Requirements Engineering - Design of a Global Family of Su...
Naming the Pain in Requirements Engineering - Design of a Global Family of Su...Naming the Pain in Requirements Engineering - Design of a Global Family of Su...
Naming the Pain in Requirements Engineering - Design of a Global Family of Su...Daniel Mendez
 
An Exploratory Study on Technology Transfer in Software Engineering
An Exploratory Study on Technology Transfer in Software EngineeringAn Exploratory Study on Technology Transfer in Software Engineering
An Exploratory Study on Technology Transfer in Software EngineeringDaniel Mendez
 
Artefact-based Requirements Engineering Improvement - Learning to Walk in Pra...
Artefact-based Requirements Engineering Improvement - Learning to Walk in Pra...Artefact-based Requirements Engineering Improvement - Learning to Walk in Pra...
Artefact-based Requirements Engineering Improvement - Learning to Walk in Pra...Daniel Mendez
 
Improving Requirements Engineering by Artefact Orientation
Improving Requirements Engineering by Artefact OrientationImproving Requirements Engineering by Artefact Orientation
Improving Requirements Engineering by Artefact OrientationDaniel Mendez
 
Theories in Empirical Software Engineering
Theories in Empirical Software EngineeringTheories in Empirical Software Engineering
Theories in Empirical Software EngineeringDaniel Mendez
 
Scientific software engineering methods and their validity
Scientific software engineering methods and their validityScientific software engineering methods and their validity
Scientific software engineering methods and their validityDaniel Mendez
 
Software Engineering Excellence - The key to mastering the Digital Transforma...
Software Engineering Excellence - The key to mastering the Digital Transforma...Software Engineering Excellence - The key to mastering the Digital Transforma...
Software Engineering Excellence - The key to mastering the Digital Transforma...Daniel Mendez
 
An Introduction into Philosophy of Science for Software Engineers
An Introduction into Philosophy of Science for Software Engineers An Introduction into Philosophy of Science for Software Engineers
An Introduction into Philosophy of Science for Software Engineers Daniel Mendez
 

Viewers also liked (13)

Case Studies in Industry - What We Have Learnt
Case Studies in Industry - What We Have LearntCase Studies in Industry - What We Have Learnt
Case Studies in Industry - What We Have Learnt
 
Where do we stand in Requirements Engineering Improvement Today? First Result...
Where do we stand in Requirements Engineering Improvement Today? First Result...Where do we stand in Requirements Engineering Improvement Today? First Result...
Where do we stand in Requirements Engineering Improvement Today? First Result...
 
A Case Study on Artefact-based RE Improvement in Practice
A Case Study on Artefact-based RE Improvement in PracticeA Case Study on Artefact-based RE Improvement in Practice
A Case Study on Artefact-based RE Improvement in Practice
 
Case studies in industry - fundamentals and lessons learnt
Case studies in industry - fundamentals and lessons learntCase studies in industry - fundamentals and lessons learnt
Case studies in industry - fundamentals and lessons learnt
 
Who cares about Software Process Modelling? A First Investigation about the P...
Who cares about Software Process Modelling? A First Investigation about the P...Who cares about Software Process Modelling? A First Investigation about the P...
Who cares about Software Process Modelling? A First Investigation about the P...
 
Naming the Pain in Requirements Engineering - Design of a Global Family of Su...
Naming the Pain in Requirements Engineering - Design of a Global Family of Su...Naming the Pain in Requirements Engineering - Design of a Global Family of Su...
Naming the Pain in Requirements Engineering - Design of a Global Family of Su...
 
An Exploratory Study on Technology Transfer in Software Engineering
An Exploratory Study on Technology Transfer in Software EngineeringAn Exploratory Study on Technology Transfer in Software Engineering
An Exploratory Study on Technology Transfer in Software Engineering
 
Artefact-based Requirements Engineering Improvement - Learning to Walk in Pra...
Artefact-based Requirements Engineering Improvement - Learning to Walk in Pra...Artefact-based Requirements Engineering Improvement - Learning to Walk in Pra...
Artefact-based Requirements Engineering Improvement - Learning to Walk in Pra...
 
Improving Requirements Engineering by Artefact Orientation
Improving Requirements Engineering by Artefact OrientationImproving Requirements Engineering by Artefact Orientation
Improving Requirements Engineering by Artefact Orientation
 
Theories in Empirical Software Engineering
Theories in Empirical Software EngineeringTheories in Empirical Software Engineering
Theories in Empirical Software Engineering
 
Scientific software engineering methods and their validity
Scientific software engineering methods and their validityScientific software engineering methods and their validity
Scientific software engineering methods and their validity
 
Software Engineering Excellence - The key to mastering the Digital Transforma...
Software Engineering Excellence - The key to mastering the Digital Transforma...Software Engineering Excellence - The key to mastering the Digital Transforma...
Software Engineering Excellence - The key to mastering the Digital Transforma...
 
An Introduction into Philosophy of Science for Software Engineers
An Introduction into Philosophy of Science for Software Engineers An Introduction into Philosophy of Science for Software Engineers
An Introduction into Philosophy of Science for Software Engineers
 

Similar to In Quest for Requirements Engineering Oracles: Dependent Variables and Measurements for a (good) RE

Requirement engineering evaluation
Requirement engineering evaluationRequirement engineering evaluation
Requirement engineering evaluationIshraq Al Fataftah
 
VA-VE Presentation.pptx
VA-VE Presentation.pptxVA-VE Presentation.pptx
VA-VE Presentation.pptxDevenJally1
 
Requirements Decision Making through Architecturally Significant Requirements
Requirements Decision Making through Architecturally Significant RequirementsRequirements Decision Making through Architecturally Significant Requirements
Requirements Decision Making through Architecturally Significant Requirementsspareuseratlero
 
'A critique of testing' UK TMF forum January 2015
'A critique of testing' UK TMF forum January 2015 'A critique of testing' UK TMF forum January 2015
'A critique of testing' UK TMF forum January 2015 Georgina Tilby
 
Mapping Project Management Work to DevOps - style Workflows
Mapping Project Management Work to DevOps - style WorkflowsMapping Project Management Work to DevOps - style Workflows
Mapping Project Management Work to DevOps - style WorkflowsChris Knotts
 
Requirements Analysis and Management using Innoslate
Requirements Analysis and Management using InnoslateRequirements Analysis and Management using Innoslate
Requirements Analysis and Management using InnoslateElizabeth Steiner
 
Testing Quality Requirements of a System-of-Systems in the Public Sector - Ch...
Testing Quality Requirements of a System-of-Systems in the Public Sector - Ch...Testing Quality Requirements of a System-of-Systems in the Public Sector - Ch...
Testing Quality Requirements of a System-of-Systems in the Public Sector - Ch...Markus Borg
 
Model-Based Software Engineering: A Multiple-Case Study on Challenges and Dev...
Model-Based Software Engineering: A Multiple-Case Study on Challenges and Dev...Model-Based Software Engineering: A Multiple-Case Study on Challenges and Dev...
Model-Based Software Engineering: A Multiple-Case Study on Challenges and Dev...Rodi Jolak
 
Key Measurements For Testers
Key Measurements For TestersKey Measurements For Testers
Key Measurements For TestersQA Programmer
 
Advances and recent trends in Modeling and Analysis of Bridges
Advances and recent trends in Modeling and Analysis of BridgesAdvances and recent trends in Modeling and Analysis of Bridges
Advances and recent trends in Modeling and Analysis of BridgesAIT Solutions
 
Requirements Engineering Process Improvement
Requirements Engineering Process ImprovementRequirements Engineering Process Improvement
Requirements Engineering Process ImprovementIan Sommerville
 
Mi 291 chapter 2 (engineering analsysis)
Mi 291 chapter 2 (engineering analsysis)Mi 291 chapter 2 (engineering analsysis)
Mi 291 chapter 2 (engineering analsysis)varun teja G.V.V
 
requirement engineering
requirement engineeringrequirement engineering
requirement engineeringanam singla
 
Predicting the quality of a survey question from its design characteristics: SQP
Predicting the quality of a survey question from its design characteristics: SQPPredicting the quality of a survey question from its design characteristics: SQP
Predicting the quality of a survey question from its design characteristics: SQPDaniel Oberski
 
Which Vertical Search Engines are Relevant? Understanding Vertical Relevance ...
Which Vertical Search Engines are Relevant? Understanding Vertical Relevance ...Which Vertical Search Engines are Relevant? Understanding Vertical Relevance ...
Which Vertical Search Engines are Relevant? Understanding Vertical Relevance ...Mounia Lalmas-Roelleke
 

Similar to In Quest for Requirements Engineering Oracles: Dependent Variables and Measurements for a (good) RE (20)

Requirement engineering evaluation
Requirement engineering evaluationRequirement engineering evaluation
Requirement engineering evaluation
 
VA-VE Presentation.pptx
VA-VE Presentation.pptxVA-VE Presentation.pptx
VA-VE Presentation.pptx
 
Requirements Decision Making through Architecturally Significant Requirements
Requirements Decision Making through Architecturally Significant RequirementsRequirements Decision Making through Architecturally Significant Requirements
Requirements Decision Making through Architecturally Significant Requirements
 
'A critique of testing' UK TMF forum January 2015
'A critique of testing' UK TMF forum January 2015 'A critique of testing' UK TMF forum January 2015
'A critique of testing' UK TMF forum January 2015
 
Chapter 8.ppt
Chapter 8.pptChapter 8.ppt
Chapter 8.ppt
 
Mapping Project Management Work to DevOps - style Workflows
Mapping Project Management Work to DevOps - style WorkflowsMapping Project Management Work to DevOps - style Workflows
Mapping Project Management Work to DevOps - style Workflows
 
Requirements Analysis and Management using Innoslate
Requirements Analysis and Management using InnoslateRequirements Analysis and Management using Innoslate
Requirements Analysis and Management using Innoslate
 
Testing Quality Requirements of a System-of-Systems in the Public Sector - Ch...
Testing Quality Requirements of a System-of-Systems in the Public Sector - Ch...Testing Quality Requirements of a System-of-Systems in the Public Sector - Ch...
Testing Quality Requirements of a System-of-Systems in the Public Sector - Ch...
 
Model-Based Software Engineering: A Multiple-Case Study on Challenges and Dev...
Model-Based Software Engineering: A Multiple-Case Study on Challenges and Dev...Model-Based Software Engineering: A Multiple-Case Study on Challenges and Dev...
Model-Based Software Engineering: A Multiple-Case Study on Challenges and Dev...
 
Key Measurements For Testers
Key Measurements For TestersKey Measurements For Testers
Key Measurements For Testers
 
Business Analyst interview Questions
Business Analyst interview QuestionsBusiness Analyst interview Questions
Business Analyst interview Questions
 
Advances and recent trends in Modeling and Analysis of Bridges
Advances and recent trends in Modeling and Analysis of BridgesAdvances and recent trends in Modeling and Analysis of Bridges
Advances and recent trends in Modeling and Analysis of Bridges
 
Requirements Engineering Process Improvement
Requirements Engineering Process ImprovementRequirements Engineering Process Improvement
Requirements Engineering Process Improvement
 
Mi 291 chapter 2 (engineering analsysis)
Mi 291 chapter 2 (engineering analsysis)Mi 291 chapter 2 (engineering analsysis)
Mi 291 chapter 2 (engineering analsysis)
 
requirement engineering
requirement engineeringrequirement engineering
requirement engineering
 
Predicting the quality of a survey question from its design characteristics: SQP
Predicting the quality of a survey question from its design characteristics: SQPPredicting the quality of a survey question from its design characteristics: SQP
Predicting the quality of a survey question from its design characteristics: SQP
 
Which Vertical Search Engines are Relevant? Understanding Vertical Relevance ...
Which Vertical Search Engines are Relevant? Understanding Vertical Relevance ...Which Vertical Search Engines are Relevant? Understanding Vertical Relevance ...
Which Vertical Search Engines are Relevant? Understanding Vertical Relevance ...
 
Shared position in a project
Shared position in a projectShared position in a project
Shared position in a project
 
QA in RE
QA in REQA in RE
QA in RE
 
SystemsAnalyst
SystemsAnalystSystemsAnalyst
SystemsAnalyst
 

More from Daniel Mendez

Empirical Software Engineering - What is it and why do we need it?
Empirical Software Engineering - What is it and why do we need it?Empirical Software Engineering - What is it and why do we need it?
Empirical Software Engineering - What is it and why do we need it?Daniel Mendez
 
Building and Evaluating Theories 
 in Software Engineering
Building and Evaluating Theories 
 in Software EngineeringBuilding and Evaluating Theories 
 in Software Engineering
Building and Evaluating Theories 
 in Software EngineeringDaniel Mendez
 
Requirements Engineering Research: How good are we at solving practical prob...
Requirements Engineering Research:  How good are we at solving practical prob...Requirements Engineering Research:  How good are we at solving practical prob...
Requirements Engineering Research: How good are we at solving practical prob...Daniel Mendez
 
Design Thinking for Requirements Engineering
Design Thinking for Requirements EngineeringDesign Thinking for Requirements Engineering
Design Thinking for Requirements EngineeringDaniel Mendez
 
In Quest of Requirements Engineering Research that Industry Needs
In Quest of Requirements Engineering Research that Industry NeedsIn Quest of Requirements Engineering Research that Industry Needs
In Quest of Requirements Engineering Research that Industry NeedsDaniel Mendez
 
Survey Research in Software Engineering
Survey Research in Software EngineeringSurvey Research in Software Engineering
Survey Research in Software EngineeringDaniel Mendez
 
Surveys in Software Engineering
Surveys in Software EngineeringSurveys in Software Engineering
Surveys in Software EngineeringDaniel Mendez
 
Theory Building in RE - The NaPiRE Initiative
Theory Building in RE - The NaPiRE InitiativeTheory Building in RE - The NaPiRE Initiative
Theory Building in RE - The NaPiRE InitiativeDaniel Mendez
 

More from Daniel Mendez (8)

Empirical Software Engineering - What is it and why do we need it?
Empirical Software Engineering - What is it and why do we need it?Empirical Software Engineering - What is it and why do we need it?
Empirical Software Engineering - What is it and why do we need it?
 
Building and Evaluating Theories 
 in Software Engineering
Building and Evaluating Theories 
 in Software EngineeringBuilding and Evaluating Theories 
 in Software Engineering
Building and Evaluating Theories 
 in Software Engineering
 
Requirements Engineering Research: How good are we at solving practical prob...
Requirements Engineering Research:  How good are we at solving practical prob...Requirements Engineering Research:  How good are we at solving practical prob...
Requirements Engineering Research: How good are we at solving practical prob...
 
Design Thinking for Requirements Engineering
Design Thinking for Requirements EngineeringDesign Thinking for Requirements Engineering
Design Thinking for Requirements Engineering
 
In Quest of Requirements Engineering Research that Industry Needs
In Quest of Requirements Engineering Research that Industry NeedsIn Quest of Requirements Engineering Research that Industry Needs
In Quest of Requirements Engineering Research that Industry Needs
 
Survey Research in Software Engineering
Survey Research in Software EngineeringSurvey Research in Software Engineering
Survey Research in Software Engineering
 
Surveys in Software Engineering
Surveys in Software EngineeringSurveys in Software Engineering
Surveys in Software Engineering
 
Theory Building in RE - The NaPiRE Initiative
Theory Building in RE - The NaPiRE InitiativeTheory Building in RE - The NaPiRE Initiative
Theory Building in RE - The NaPiRE Initiative
 

Recently uploaded

KCD Costa Rica 2024 - Nephio para parvulitos
KCD Costa Rica 2024 - Nephio para parvulitosKCD Costa Rica 2024 - Nephio para parvulitos
KCD Costa Rica 2024 - Nephio para parvulitosVictor Morales
 
March 2024 - Top 10 Read Articles in Artificial Intelligence and Applications...
March 2024 - Top 10 Read Articles in Artificial Intelligence and Applications...March 2024 - Top 10 Read Articles in Artificial Intelligence and Applications...
March 2024 - Top 10 Read Articles in Artificial Intelligence and Applications...gerogepatton
 
Theory of Machine Notes / Lecture Material .pdf
Theory of Machine Notes / Lecture Material .pdfTheory of Machine Notes / Lecture Material .pdf
Theory of Machine Notes / Lecture Material .pdfShreyas Pandit
 
tourism-management-srs_compress-software-engineering.pdf
tourism-management-srs_compress-software-engineering.pdftourism-management-srs_compress-software-engineering.pdf
tourism-management-srs_compress-software-engineering.pdfchess188chess188
 
The Satellite applications in telecommunication
The Satellite applications in telecommunicationThe Satellite applications in telecommunication
The Satellite applications in telecommunicationnovrain7111
 
"Exploring the Essential Functions and Design Considerations of Spillways in ...
"Exploring the Essential Functions and Design Considerations of Spillways in ..."Exploring the Essential Functions and Design Considerations of Spillways in ...
"Exploring the Essential Functions and Design Considerations of Spillways in ...Erbil Polytechnic University
 
Robotics Group 10 (Control Schemes) cse.pdf
Robotics Group 10  (Control Schemes) cse.pdfRobotics Group 10  (Control Schemes) cse.pdf
Robotics Group 10 (Control Schemes) cse.pdfsahilsajad201
 
70 POWER PLANT IAE V2500 technical training
70 POWER PLANT IAE V2500 technical training70 POWER PLANT IAE V2500 technical training
70 POWER PLANT IAE V2500 technical trainingGladiatorsKasper
 
Structural Integrity Assessment Standards in Nigeria by Engr Nimot Muili
Structural Integrity Assessment Standards in Nigeria by Engr Nimot MuiliStructural Integrity Assessment Standards in Nigeria by Engr Nimot Muili
Structural Integrity Assessment Standards in Nigeria by Engr Nimot MuiliNimot Muili
 
Detection&Tracking - Thermal imaging object detection and tracking
Detection&Tracking - Thermal imaging object detection and trackingDetection&Tracking - Thermal imaging object detection and tracking
Detection&Tracking - Thermal imaging object detection and trackinghadarpinhas1
 
Gravity concentration_MI20612MI_________
Gravity concentration_MI20612MI_________Gravity concentration_MI20612MI_________
Gravity concentration_MI20612MI_________Romil Mishra
 
Artificial Intelligence in Power System overview
Artificial Intelligence in Power System overviewArtificial Intelligence in Power System overview
Artificial Intelligence in Power System overviewsandhya757531
 
ADM100 Running Book for sap basis domain study
ADM100 Running Book for sap basis domain studyADM100 Running Book for sap basis domain study
ADM100 Running Book for sap basis domain studydhruvamdhruvil123
 
Comprehensive energy systems.pdf Comprehensive energy systems.pdf
Comprehensive energy systems.pdf Comprehensive energy systems.pdfComprehensive energy systems.pdf Comprehensive energy systems.pdf
Comprehensive energy systems.pdf Comprehensive energy systems.pdfalene1
 
Secure Key Crypto - Tech Paper JET Tech Labs
Secure Key Crypto - Tech Paper JET Tech LabsSecure Key Crypto - Tech Paper JET Tech Labs
Secure Key Crypto - Tech Paper JET Tech Labsamber724300
 
Indian Tradition, Culture & Societies.pdf
Indian Tradition, Culture & Societies.pdfIndian Tradition, Culture & Societies.pdf
Indian Tradition, Culture & Societies.pdfalokitpathak01
 
CS 3251 Programming in c all unit notes pdf
CS 3251 Programming in c all unit notes pdfCS 3251 Programming in c all unit notes pdf
CS 3251 Programming in c all unit notes pdfBalamuruganV28
 
Katarzyna Lipka-Sidor - BIM School Course
Katarzyna Lipka-Sidor - BIM School CourseKatarzyna Lipka-Sidor - BIM School Course
Katarzyna Lipka-Sidor - BIM School Coursebim.edu.pl
 
Uk-NO1 kala jadu karne wale ka contact number kala jadu karne wale baba kala ...
Uk-NO1 kala jadu karne wale ka contact number kala jadu karne wale baba kala ...Uk-NO1 kala jadu karne wale ka contact number kala jadu karne wale baba kala ...
Uk-NO1 kala jadu karne wale ka contact number kala jadu karne wale baba kala ...Amil baba
 
priority interrupt computer organization
priority interrupt computer organizationpriority interrupt computer organization
priority interrupt computer organizationchnrketan
 

Recently uploaded (20)

KCD Costa Rica 2024 - Nephio para parvulitos
KCD Costa Rica 2024 - Nephio para parvulitosKCD Costa Rica 2024 - Nephio para parvulitos
KCD Costa Rica 2024 - Nephio para parvulitos
 
March 2024 - Top 10 Read Articles in Artificial Intelligence and Applications...
March 2024 - Top 10 Read Articles in Artificial Intelligence and Applications...March 2024 - Top 10 Read Articles in Artificial Intelligence and Applications...
March 2024 - Top 10 Read Articles in Artificial Intelligence and Applications...
 
Theory of Machine Notes / Lecture Material .pdf
Theory of Machine Notes / Lecture Material .pdfTheory of Machine Notes / Lecture Material .pdf
Theory of Machine Notes / Lecture Material .pdf
 
tourism-management-srs_compress-software-engineering.pdf
tourism-management-srs_compress-software-engineering.pdftourism-management-srs_compress-software-engineering.pdf
tourism-management-srs_compress-software-engineering.pdf
 
The Satellite applications in telecommunication
The Satellite applications in telecommunicationThe Satellite applications in telecommunication
The Satellite applications in telecommunication
 
"Exploring the Essential Functions and Design Considerations of Spillways in ...
"Exploring the Essential Functions and Design Considerations of Spillways in ..."Exploring the Essential Functions and Design Considerations of Spillways in ...
"Exploring the Essential Functions and Design Considerations of Spillways in ...
 
Robotics Group 10 (Control Schemes) cse.pdf
Robotics Group 10  (Control Schemes) cse.pdfRobotics Group 10  (Control Schemes) cse.pdf
Robotics Group 10 (Control Schemes) cse.pdf
 
70 POWER PLANT IAE V2500 technical training
70 POWER PLANT IAE V2500 technical training70 POWER PLANT IAE V2500 technical training
70 POWER PLANT IAE V2500 technical training
 
Structural Integrity Assessment Standards in Nigeria by Engr Nimot Muili
Structural Integrity Assessment Standards in Nigeria by Engr Nimot MuiliStructural Integrity Assessment Standards in Nigeria by Engr Nimot Muili
Structural Integrity Assessment Standards in Nigeria by Engr Nimot Muili
 
Detection&Tracking - Thermal imaging object detection and tracking
Detection&Tracking - Thermal imaging object detection and trackingDetection&Tracking - Thermal imaging object detection and tracking
Detection&Tracking - Thermal imaging object detection and tracking
 
Gravity concentration_MI20612MI_________
Gravity concentration_MI20612MI_________Gravity concentration_MI20612MI_________
Gravity concentration_MI20612MI_________
 
Artificial Intelligence in Power System overview
Artificial Intelligence in Power System overviewArtificial Intelligence in Power System overview
Artificial Intelligence in Power System overview
 
ADM100 Running Book for sap basis domain study
ADM100 Running Book for sap basis domain studyADM100 Running Book for sap basis domain study
ADM100 Running Book for sap basis domain study
 
Comprehensive energy systems.pdf Comprehensive energy systems.pdf
Comprehensive energy systems.pdf Comprehensive energy systems.pdfComprehensive energy systems.pdf Comprehensive energy systems.pdf
Comprehensive energy systems.pdf Comprehensive energy systems.pdf
 
Secure Key Crypto - Tech Paper JET Tech Labs
Secure Key Crypto - Tech Paper JET Tech LabsSecure Key Crypto - Tech Paper JET Tech Labs
Secure Key Crypto - Tech Paper JET Tech Labs
 
Indian Tradition, Culture & Societies.pdf
Indian Tradition, Culture & Societies.pdfIndian Tradition, Culture & Societies.pdf
Indian Tradition, Culture & Societies.pdf
 
CS 3251 Programming in c all unit notes pdf
CS 3251 Programming in c all unit notes pdfCS 3251 Programming in c all unit notes pdf
CS 3251 Programming in c all unit notes pdf
 
Katarzyna Lipka-Sidor - BIM School Course
Katarzyna Lipka-Sidor - BIM School CourseKatarzyna Lipka-Sidor - BIM School Course
Katarzyna Lipka-Sidor - BIM School Course
 
Uk-NO1 kala jadu karne wale ka contact number kala jadu karne wale baba kala ...
Uk-NO1 kala jadu karne wale ka contact number kala jadu karne wale baba kala ...Uk-NO1 kala jadu karne wale ka contact number kala jadu karne wale baba kala ...
Uk-NO1 kala jadu karne wale ka contact number kala jadu karne wale baba kala ...
 
priority interrupt computer organization
priority interrupt computer organizationpriority interrupt computer organization
priority interrupt computer organization
 

In Quest for Requirements Engineering Oracles: Dependent Variables and Measurements for a (good) RE

  • 1. Technische Universität München In Quest for Requirements Engineering Oracles DependentVariables and Measurements for (good) RE Joint work with Jakob Mund Henning Femmer AntonioVetrò Daniel Méndez Fernández ! Technische Universität München Germany ! EASE 2014 London, UK 13.05.2014 @mendezfe
  • 2. A regular case study in requirements engineering… Goal: Test the sensitivity of a new RE method in company context » Introduce a new RE method in project environment » Conduct action research workshops following the new method » Rate the method outcome according to assessment criteria defined using GQM ! [Mendez Fernandez et al.A Case Study on the Application of an 
 Artefact-Based Requirements Engineering Approach]
  • 3. A regular case study in requirements engineering… What conclusions can I draw beyond the chosen context? ! » How representative are the variables / how important are they to other contexts? » How can the variables be efficiently measured? » What relations exist between the variables?
  • 4. Dependent variables in requirements engineering Why? Status quo in requirements engineering research • Development of methods/tools for problems too often not properly understood » Control of validity / threat mitigation? ! Research objectives • Get a better understanding on RE phenomena, their dependencies and their measurability ! Our motivation Understand RE phenomena and research implications on evidence-based RE research • Difficulties arising from cause-effect relationships • Measurability and actionability • Implications on threat to validity (e.g. possibilities of generalisation)
  • 5. Study design Research questions !RQ 1. Which RE-related phenomena exist, where in the project ecosystem 
 do they manifest themselves, and how do they relate to each other? RQ 2. Are the phenomena measurable? RQ 3. Are the phenomena actionable?
  • 6. Study design Research questions ! Requirements Engineering survey NaPiRE www.re-survey.org Globally distributed, replicated survey 
 on status quo and problems in RE RQ 1. Which RE-related phenomena exist, where in the project ecosystem 
 do they manifest themselves, and how do they relate to each other? RQ 2. Are the phenomena measurable? RQ 3. Are the phenomena actionable?
  • 7. Study design Research questions ! Requirements Engineering survey NaPiRE www.re-survey.org Globally distributed, replicated survey 
 on status quo and problems in RE RQ 1. Which RE-related phenomena exist, where in the project ecosystem 
 do they manifest themselves, and how do they relate to each other? Researchers Levels of dependent variables (“Dimensions”) [Gorschek et al. Requirements Eng.: In Search of the dependent variables] RQ 1 RQ 2. Are the phenomena measurable? RQ 3. Are the phenomena actionable? RE Phenomena inferred from open 
 questions on problems, 
 their causes and their effects
  • 8. Study design Research questions ! Requirements Engineering survey NaPiRE www.re-survey.org Globally distributed, replicated survey 
 on status quo and problems in RE RQ 1. Which RE-related phenomena exist, where in the project ecosystem 
 do they manifest themselves, and how do they relate to each other? Researchers Levels of dependent variables (“Dimensions”) [Gorschek et al. Requirements Eng.: In Search of the dependent variables] RQ 1 RQ 2. Are the phenomena measurable? RQ 3. Are the phenomena actionable? Voting of • Measurability (artefacts? activities? both? none?) • Actionability RQ 2 & 3 RE Phenomena inferred from open 
 questions on problems, 
 their causes and their effects
  • 9. Results RQ 1: Dependent variables ! ! ! ! • 93 variables with 167 dependencies » Full result set: http://goo.gl/W3A1tu Company Project Engineering 2 2 22 2 2 2 3 2 2 4 2 3 8 3 2 2 4 2 2 2 2 2 2 2 2 2 3 2 3 3 Requirements Engineering M36 Name Actionable (A) Measurable (on aRtefacts, on aCtivities, on aRtefacts & aCtivities) M03 R M08 R R03 R M09 R RP06 R M32 C RP02 R M29 C M30 R&C M05 C M15 M20 M13 A R11 R18 R&C M24 R RP10 R&C A M17 C RP03 C A M02 R A RP08 R A RP11 A RP07 R RP04 C R01 RP05 R A M28 RP01 R [Results with minimum occurrence in elements > 1]
  • 10. Requirements Engineering Results (Excerpt) RQ 1: Dependent variables “Incomplete requirements” “Implicit requirements not! made explicit”
  • 11. Engineering Requirements Engineering Results (Excerpt) RQ 1: Dependent variables “Incomplete requirements” “Implicit requirements not! made explicit”
  • 12. Engineering “Increased effort in testing” … Requirements Engineering Results (Excerpt) RQ 1: Dependent variables “Incomplete requirements” “Implicit requirements not! made explicit”
  • 13. SW Project Engineering “Increased effort in testing” … Requirements Engineering Results (Excerpt) RQ 1: Dependent variables “Incomplete requirements” “Implicit requirements not! made explicit”
  • 14. SW Project Engineering “Increased effort in testing” … Requirements Engineering Results (Excerpt) RQ 1: Dependent variables “Incomplete requirements” “Implicit requirements not! made explicit” “Change requests” “Effort and time overrun” “Moving targets”
  • 15. SW Project Engineering “Increased effort in testing” … Requirements Engineering Results (Excerpt) RQ 1: Dependent variables “Incomplete requirements” “Implicit requirements not! made explicit” “Failed acceptance” … “Change requests” “Effort and time overrun” “Moving targets”
  • 16. Company “Customer (dis-)satisfaction” SW Project Engineering “Increased effort in testing” … Requirements Engineering Results (Excerpt) RQ 1: Dependent variables “Incomplete requirements” “Implicit requirements not! made explicit” “Failed acceptance” … “Change requests” “Effort and time overrun” “Moving targets”
  • 17. “Incomplete requirements” “Failed acceptance” “Increased effort in testing” “Effort and time overrun” “Implicit requirements not! made explicit” “Customer (dis-)satisfaction” “Change requests” Requirements Engineering Engineering SW Project Company … … “Moving targets” Results (Excerpt) RQ 2: Measurability • High degree of measurability for variables in RE dimension (59% of variables for RE) • Unmeasurable variables comprise social phenomena (e.g.,“Weak access customer needs”) • Most measurable variables are measurable on basis of artefacts 
 (e.g.,“Underspecified requirements”) • Decreasing understanding of variables and measurability the “farther away” from RE
  • 18. “Incomplete requirements” “Failed acceptance” “Increased effort in testing” “Effort and time overrun” “Implicit requirements not! made explicit” “Customer (dis-)satisfaction” “Change requests” Requirements Engineering Engineering SW Project Company … … “Moving targets” Critical reflection The good: Measurability in RE context
  • 19. “Incomplete requirements” “Failed acceptance” “Increased effort in testing” “Effort and time overrun” “Implicit requirements not! made explicit” “Customer (dis-)satisfaction” “Change requests” Requirements Engineering Engineering SW Project Company … … “Moving targets” Critical reflection The good: Measurability in RE context • High degree of measurability for RE-related variables • Measurability (to a large extent) on basis of artefacts » Support of independent and comparable measurements » Already a better understanding of RE study context and its characteristics
  • 20. “Incomplete requirements” “Failed acceptance” “Increased effort in testing” “Effort and time overrun” “Implicit requirements not! made explicit” “Customer (dis-)satisfaction” “Change requests” Requirements Engineering Engineering SW Project Company … … “Moving targets” Critical reflection The bad: Limitations in RE context
  • 21. “Incomplete requirements” “Failed acceptance” “Increased effort in testing” “Effort and time overrun” “Implicit requirements not! made explicit” “Customer (dis-)satisfaction” “Change requests” Requirements Engineering Engineering SW Project Company … … “Moving targets” Critical reflection The bad: Limitations in RE context ? • Incomplete dependencies » Investigation was performed “inside-out” from RE and needs to be extended
  • 22. “Incomplete requirements” “Failed acceptance” “Increased effort in testing” “Effort and time overrun” “Implicit requirements not! made explicit” “Customer (dis-)satisfaction” “Change requests” Requirements Engineering Engineering SW Project Company … … “Moving targets” Critical reflection The bad: Limitations in RE context • Complex (yet incomplete) system of variables » Too many (especially unknown) variables intersect with each other ? ? • Incomplete dependencies » Investigation was performed “inside-out” from RE and needs to be extended
  • 23. “Incomplete requirements” “Failed acceptance” “Increased effort in testing” “Effort and time overrun” “Implicit requirements not! made explicit” “Customer (dis-)satisfaction” “Change requests” Requirements Engineering Engineering SW Project Company … … “Moving targets” Critical reflection The bad: Limitations in RE context • Complex (yet incomplete) system of variables » Too many (especially unknown) variables intersect with each other • Even known variables strongly depend on subjectivity » Requirements engineering is also about beliefs, desires, experiences, expectations and politics ? ? • Incomplete dependencies » Investigation was performed “inside-out” from RE and needs to be extended
  • 24. “Incomplete requirements” “Failed acceptance” “Increased effort in testing” “Effort and time overrun” “Implicit requirements not! made explicit” “Customer (dis-)satisfaction” “Change requests” Requirements Engineering Engineering SW Project Company … … “Moving targets” Critical reflection The ugly: No RE oracle in sight! No “generalisability” 
 (by no means) • Complex (yet incomplete) system of variables • Even known variables strongly depend on subjectivity • Incomplete dependencies
  • 25. Research implications (Excerpt) General implications on evidence-based RE research • No generalisation possible, but first set of variables and ideas for measurability » Aim for universal language rather than for universal truth • High degree of subjectivity » Qualitative studies (e.g. expert judgment)
  • 26. Research implications (Excerpt) General implications on evidence-based RE research • No generalisation possible, but first set of variables and ideas for measurability » Aim for universal language rather than for universal truth • High degree of subjectivity » Qualitative studies (e.g. expert judgment) Implications on research on RE methodologies • Inference of first improvement goals » Calibrate… – Study designs according to variables and their measurement – Variables if aiming for tackling “commonly accepted” problems
  • 27. Threats to Validity • Variables are incomplete (needless to say) • Classification…. • to some extent with a high disagreement • not the only possible way (and it’s positivistic) ! ! • However: It is a way to learn more and we are confident to further continue it!
  • 28. ! Full data disclosed to: http://goo.gl/W3A1tu ! • raw data • full graphs • R scripts • … @mendezfe • Next replication of NaPiRE ongoing (www.re-survey.org) » Gather more RE phenomena • from NaPiRE • from other sources (e.g. via longitudinal studies) ! ! ! Future work daniel.Mendez@tum.de Let’s build a big picture of RE variables together!
  • 29. Backup * Cohen’s Kappa values for classification A phenomenon is measurable, if its understanding is 
 sufficiently mature such that (i)  an existing or anticipated measure, i.e., objective map- 
 ping to mathematical objects (ii) can efficiently (e.g., in justifiable time) and effectively 
 (i.e., preserving empirical observations) capture the 
 phenomenon (iii)  under practical conditions and when applied on study 
 objects which can be expected to be present in a soft- 
 ware project ecosystem.