1.
Technische Universität München
Theory building in RE -
The NaPiRE* Initiative
* Naming the Pain in Requirements Engineering
(www.re-survey.org)
Daniel Méndez
Technical University of Munich
www.mendezfe.org
@mendezfe
RE@40 Seminar, 2017
Kappel am Albis, Switzerland

2.
“[…] judging a theory by assessing the
number, faith, and vocal energy of its
supporters […] basic political credo of
contemporary religious maniacs”
— Imre Lakatos, 1970

3.
‣ In Software Engineering, too, truth often lies in power
rather than in empirical evidence.
“[…] judging a theory by assessing the
number, faith, and vocal energy of its
supporters […] basic political credo of
contemporary religious maniacs”
— Imre Lakatos, 1970

4.
RE research and practice
often dominated by conventional wisdom
For a problem-driven research, we nee a
reliable body of knowledge about
industrial practices and problems.

5.
Naming the Pain in
Requirements Engineering (NaPiRE)
‣Foundation for problem-driven research
Overall goal: (First) Theory on industrial
practices and problems in Requirements
Engineering

6.
2017
22 countries
51 researchers from 43 institutions
2014
10 countries
23 researchers
2012
1 country,
2 researchers
Overall approach:
By the community for the community
Bi-yearly replicated, globally distributed family of
surveys (w/ synthesis) on practices and problems
‣ Collaborative instrument design
‣ Independent surveys in different countries
‣ Collaborative synthesis and publication
www.re-survey.org
Principles
‣ Theory-centric
‣ Anonymity, but closed
‣ Full open science practices

7.
2017
22 countries
51 researchers from 43 institutions
2014
10 countries
23 researchers
Overall approach:
By the community for the community
Bi-yearly replicated, globally distributed family of
surveys (w/ synthesis) on practices and problems
‣ Collaborative instrument design
‣ Independent surveys in different countries
‣ Collaborative synthesis and publication
www.re-survey.org
Principles
‣ Theory-centric
‣ Anonymity, but closed
‣ Full open science practices

8.
2017
22 countries
51 researchers from 43 institutions
Overall approach:
By the community for the community
Bi-yearly replicated, globally distributed family of
surveys (w/ synthesis) on practices and problems
‣ Collaborative instrument design
‣ Independent surveys in different countries
‣ Collaborative synthesis and publication
www.re-survey.org
Principles
‣ Theory-centric
‣ Anonymity, but closed
‣ Full open science practices

9.
Excerpt results NapiRE 2015/16
Country # Companies
(full survey completion)
Response rate
Austria 14 72 %
Germany 41 36,8 %
UK (Northern Ireland) 18 39,7 %
Canada 13 75 %
USA 15 36,2 %
Estonia 8 25,7 %
Finland 15 37,5 %
Norway 10 70,8 %
Sweden 20 51,8 %
Brazil 74 35,3 %
Total 228
crosscutting (60)
public (28)
sector (28)
enterprise (20)
finance(26)
planning (20)
resource (23)
automotive (15)
healthcare(14)
management (16)
business (9)
energy (11)
insurance(11)
logistics (8)
telecom(9) transportation(8)
aerospace (5)
education (5)
gas (5) geoprocessing (4)
informed(5)
intelligence (4) oil (5)
process (5)
aviation (3)
communication (3) construction (3)
defence (3) e-commerce (3)
entertainment (3) games (3)
human (3)
shipping (3)
workforce (3)
agriculture (2)
chemicals (2)
customer (2)
electronic (2)
railway (2) relationship (2)
scientific (2) software (2)
automation (1)
e-
government (1)
funerary (1)
goods (1) industrial (1)
networking (1)
pulp (1)
steel (1)
Distributed over
industry sectors

10.
Status quo in practices used [1]
‣ Requirements Elicitation
‣ Requirements Documentation
‣ Requirements Change and Alignment
‣ Requirements Engineering Standards
‣ Requirements Engineering Improvement
Problems in RE [2]
‣ Problems and criticality
‣ Causes
‣ Effects
[2] Mendez, Wagner, et al. Naming the pain in requirements engineering Contemporary problems, causes, and effects in practice , 2017
[1] Wagner, Mendez, et al. Status Quo in Requirements Engineering: A Theory and a Global Family of Surveys, 2017
Req Elicitation Technique
Interview
Scenario
Prototyping
Facilitated Meetings
Observation
Req Documentation Technique
Structured req list
Domain/business process model
Use case model
Goal model
Data model
Non-functional req
Textual
Semi-formal
Formal
Technology
Req Test Alignment Approach
Req review by tester
Coverage by tests
Acceptance criteria
Test derivation from models
Req Change Approach
Product backlog update
Change requests
Trace management
Impact analysis
Activity
Req Elicitation
Req Documentation
Req Change Management
Req Test Alignment
P 1-5
P 6-13
P 14-20
P 21-24
Actor
Req Engineer
Test Engineer
Req Standard Application
Practice
Control
Tailoring
Req Eng Process Standard
P 25-28
Req Standard Defintion
Compliance
Development
Tool support
Quality assurance
Project management
Knowledge transfer
Process complexity
Communication demand
Willigness to change
Possibility of standardisation
P 26-45
Req Improvement Means
Continuous improvement
Strengths/weaknesses
Own business unit/role
Req Eng ImprovementP 46--48
Excerpt results NapiRE 2015/16

11.
Status quo in practices used [1]
‣ Requirements Elicitation
‣ Requirements Documentation
‣ Requirements Change and Alignment
‣ Requirements Engineering Standards
‣ Requirements Engineering Improvement
Problems in RE [2]
‣ Problems and criticality
‣ Causes
‣ Effects
[2] Mendez, Wagner, et al. Naming the pain in requirements engineering Contemporary problems, causes, and effects in practice , 2017
[1] Wagner, Mendez, et al. Status Quo in Requirements Engineering: A Theory and a Global Family of Surveys, 2017
Req Elicitation Technique
Interview
Scenario
Prototyping
Facilitated Meetings
Observation
Req Documentation Technique
Structured req list
Domain/business process model
Use case model
Goal model
Data model
Non-functional req
Textual
Semi-formal
Formal
Technology
Req Test Alignment Approach
Req review by tester
Coverage by tests
Acceptance criteria
Test derivation from models
Req Change Approach
Product backlog update
Change requests
Trace management
Impact analysis
Activity
Req Elicitation
Req Documentation
Req Change Management
Req Test Alignment
P 1-5
P 6-13
P 14-20
P 21-24
Actor
Req Engineer
Test Engineer
Req Standard Application
Practice
Control
Tailoring
Req Eng Process Standard
P 25-28
Req Standard Defintion
Compliance
Development
Tool support
Quality assurance
Project management
Knowledge transfer
Process complexity
Communication demand
Willigness to change
Possibility of standardisation
P 26-45
Req Improvement Means
Continuous improvement
Strengths/weaknesses
Own business unit/role
Req Eng ImprovementP 46--48
Excerpt results NapiRE 2015/16
Teaser

12.
In scope of artefact-based standard (reference) models

13.
In scope of artefact-based standard (reference) models

14.
Unclear roles and responsabilities
at customer side
Weak qualification of
RE team members
Subjective
interpretations
Stakeholders lack
business vision and
understanding
(1.28%)
Customer does not
know what he wants
(6.41%)
Lack of experience of RE
team members
Complexity of domain
Insufficient
resources
Lack of time
Conflicting stakeholder
viewpoints
(8.97%)
Strict time schedule by
customer
Missing req.
specification
template
Missing IT project experience
at customer side
Weak qualification of
stakeholders
(6.41%)
Input (33%)
(3.85%)
(6.41%)
(2.56%)
(1.28%
)
(1.28%)
(1.28%)
(2.56%)
(10.26
%)
(2.56%)
(2.56%)
(1.28%)
(1.28%)(1.28%)
(1.10%)
Method (38%)
Req. too abstract
Missing completeness check
of requirements
Insufficient agility
Communication flaws
between team and customer
Lack of a well-defined RE
process
Complexity of RE
Unclear terminology
(11.54%)
(1.28%)(1.28%)
(2.56%) (2.56%)
(1.28%) (1.28%)
People (17%)
Tools (3%)
(1.28%)
Communication
flaws customer
Missing engagement by
customer
Missing customer
involvement (2.56%)
(1.28%)
(1.28%) Policy restrictions (1.28%)
(1.28%)
Missing RE awareness
at customer side (1.28%)
Weak management at
customer side
(1.28%)
Missing direct
communication to
customer
Missing involvement of
dev. (2.56%)
Poor project
management
Missing solution
approach
Organization (9%)
Too high team
distribution
Many
customer
s
Language Barriers
Not following the
communication
plan
Missing tool
support(1.28%)
Customer (19%)
Design or
Implementation (9%)
Product (22%)
Validation or
Verification (3%)
Increased number of req.
changes
(16.18%)
Incomplete Req.
(2.94%
)
Decreased user
acceptance
Customer
dissatisfaction
(2.94%)
Need for post
implementation
Poor product quality
Decreased business
value
(10.29%)
(8.82%)
(2.94%)
Increased
communication
Implementation of
irrelevant req.
Effort overrun
Time overrun
Budget overrun
Decreased efficiency
(overall)
Increased number of
change requests
Wrong estimates
Increased difficulty of
req. elicitation
Validation of
req. becomes
difficult
Project or
Organization (47%)
(2.94%)
(1.47%)
(13.24%)
(5.88%)
(4.41%)
(2.94%)(2.94%)
(2.94%)
(1.47%) (1.47%)
(1.47%) (1.47%)
(2.94%)
Poor (system) design
quality (1.47%)
Misunderstanding
(overall)
Late decision (2.94%)
(2.94%) (2.94%)
Overall
demotivation
Information gets
lost
Increased complexity
in RE

15.
)
Communication
flaws customer
Customer (19%)
Design or
Implementation (9%)
Product (22%)
Validation or
Verification (3%)
Increased number of req.
changes
(16.18%)
Incomplete Req.
(2.94%
)
Decreased user
acceptance
Customer
dissatisfaction
(2.94%)
Need for post
implementation
Poor product quality
Decreased business
value
(10.29%)
(8.82%)
(2.94%)
Increased
communication
Implementation of
irrelevant req.
Effort overrun
Time overrun
Budget overrun
Decreased efficiency
(overall)
Increased number of
change requests
Wrong estimates
Increased difficulty of
req. elicitation
Validation of
req. becomes
difficult
Project or
Organization (47%)
(2.94%)
(1.47%)
(13.24%)
(5.88%)
(4.41%)
(2.94%)(2.94%)
(2.94%)
(1.47%) (1.47%)
(1.47%) (1.47%)
(2.94%)
Poor (system) design
quality (1.47%)
Misunderstanding
(overall)
Late decision (2.94%)
(2.94%) (2.94%)
Overall
demotivation
Information gets
lost
Increased complexity
in RE

16.
Unclear roles and responsabilities
at customer side
Weak qualification of
RE team members
Subjective
interpretations
Stakeholders lack
business vision and
understanding
(1.28%)
Customer does not
know what he wants
(6.41%)
Lack of experience of RE
team members
Complexity of domain
Insufficient
resources
Lack of time
Conflicting stakeholder
viewpoints
(8.97%)
Strict time schedule by
customer
Missing req.
specification
template
Missing IT project experience
at customer side
Weak qualification of
stakeholders
(6.41%)
Input (33%)
(3.85%)
(6.41%)
(2.56%)
(1.28%
)
(1.28%)
(1.28%)
(2.56%)
(10.26
%)
(2.56%)
(2.56%)
(1.28%)
(1.28%)(1.28%)
(1.10%)
Method (38%)
Req. too abstract
Missing completeness check
of requirements
Insufficient agility
Communication flaws
between team and customer
Lack of a well-defined RE
process
Complexity of RE
Unclear terminology
(11.54%)
(1.28%)(1.28%)
(2.56%) (2.56%)
(1.28%) (1.28%)
People (17%)
Tools (3%)
(1.28%)
Communication
flaws customer
Missing engagement by
customer
Missing customer
involvement (2.56%)
(1.28%)
(1.28%) Policy restrictions (1.28%)
(1.28%)
Missing RE awareness
at customer side (1.28%)
Weak management at
customer side
(1.28%)
Missing direct
communication to
customer
Missing involvement of
dev. (2.56%)
Poor project
management
Missing solution
approach
Organization (9%)
Too high team
distribution
Many
customer
s
Language Barriers
Not following the
communication
plan
Missing tool
support(1.28%)

17.
Unclear roles and responsabilities
at customer side
Weak qualification of
RE team members
Subjective
interpretations
Stakeholders lack
business vision and
understanding
(1.28%)
Customer does not
know what he wants
(6.41%)
Lack of experience of RE
team members
Complexity of domain
Insufficient
resources
Lack of time
Conflicting stakeholder
viewpoints
(8.97%)
Strict time schedule by
customer
Missing req.
specification
template
Missing IT project experience
at customer side
Weak qualification of
stakeholders
(6.41%)
Input (33%)
(3.85%)
(6.41%)
(2.56%)
(1.28%
)
(1.28%)
(1.28%)
(2.56%)
(10.26
%)
(2.56%)
(2.56%)
(1.28%)
(1.28%)(1.28%)
(1.10%)
Method (38%)
Req. too abstract
Missing completeness check
of requirements
Insufficient agility
Communication flaws
between team and customer
Lack of a well-defined RE
process
Complexity of RE
Unclear terminology
(11.54%)
(1.28%)(1.28%)
(2.56%) (2.56%)
(1.28%) (1.28%)
People (17%)
Tools (3%)
(1.28%)
Communication
flaws customer
Missing engagement by
customer
Missing customer
involvement (2.56%)
(1.28%)
(1.28%) Policy restrictions (1.28%)
(1.28%)
Missing RE awareness
at customer side (1.28%)
Weak management at
customer side
(1.28%)
Missing direct
communication to
customer
Missing involvement of
dev. (2.56%)
Poor project
management
Missing solution
approach
Organization (9%)
Too high team
distribution
Many
customer
s
Language Barriers
Not following the
communication
plan
Missing tool
support(1.28%)
Causes rather not due to missing guidance via reference
models, but due to organisational and even social issues.

18.
Do teams applying agile practices avoid these issues?
Unclear roles and responsabilities
at customer side
Weak qualification of
RE team members
Subjective
interpretations
Stakeholders lack
business vision and
understanding
(1.28%)
Customer does not
know what he wants
(6.41%)
Lack of experience of RE
team members
Complexity of domain
Insufficient
resources
Lack of time
Conflicting stakeholder
viewpoints
(8.97%)
Strict time schedule by
customer
Missing req.
specification
template
Missing IT project experience
at customer side
Weak qualification of
stakeholders
(6.41%)
Input (33%)
(3.85%)
(6.41%)
(2.56%)
(1.28%
)
(1.28%)
(1.28%)
(2.56%)
(10.26
%)
(2.56%)
(2.56%)
(1.28%)
(1.28%)(1.28%)
(1.10%)
Method (38%)
Req. too abstract
Missing completeness check
of requirements
Insufficient agility
Communication flaws
between team and customer
Lack of a well-defined RE
process
Complexity of RE
Unclear terminology
(11.54%)
(1.28%)(1.28%)
(2.56%) (2.56%)
(1.28%) (1.28%)
People (17%)
Tools (3%)
(1.28%)
Communication
flaws customer
Missing engagement by
customer
Missing customer
involvement (2.56%)
(1.28%)
(1.28%) Policy restrictions (1.28%)
(1.28%)
Missing RE awareness
at customer side (1.28%)
Weak management at
customer side
(1.28%)
Missing direct
communication to
customer
Missing involvement of
dev. (2.56%)
Poor project
management
Missing solution
approach
Organization (9%)
Too high team
distribution
Many
customer
s
Language Barriers
Not following the
communication
plan
Missing tool
support(1.28%)
Customer (19%)
Design or
Implementation (9%)
Product (22%)
Validation or
Verification (3%)
Increased number of req.
changes
(16.18%)
Incomplete Req.
(2.94%
)
Decreased user
acceptance
Customer
dissatisfaction
(2.94%)
Need for post
implementation
Poor product quality
Decreased business
value
(10.29%)
(8.82%)
(2.94%)
Increased
communication
Implementation of
irrelevant req.
Effort overrun
Time overrun
Budget overrun
Decreased efficiency
(overall)
Increased number of
change requests
Wrong estimates
Increased difficulty of
req. elicitation
Validation of
req. becomes
difficult
Project or
Organization (47%)
(2.94%)
(1.47%)
(13.24%)
(5.88%)
(4.41%)
(2.94%)(2.94%)
(2.94%)
(1.47%) (1.47%)
(1.47%) (1.47%)
(2.94%)
Poor (system) design
quality (1.47%)
Misunderstanding
(overall)
Late decision (2.94%)
(2.94%) (2.94%)
Overall
demotivation
Information gets
lost
Increased complexity
in RE

19.
Small companies
(1-50)
Medium companies
(51-250)
Large companies
(> 251)
#1 Problem Incomplete/hidden
requirements
Communication
flaws customer
Incomplete/hidden
requirements
#2 Problem Communication
flaws customer
Incomplete/hidden
requirements
Moving targets
#3 Problem Underspecified
requirements
Communication
flaws project team
Communication
flaws customer
#4 Problem Communication
flaws project team
Separation reqs. /
solutions
Time boxing
#5 Problem Time boxing Weak access
customer needs
Underspecified
requirements
Do teams applying agile practices avoid these issues?

20.
Small companies
(1-50)
Medium companies
(51-250)
Large companies
(> 251)
#1 Problem Incomplete/hidden
requirements
Communication
flaws customer
Incomplete/hidden
requirements
#2 Problem Communication
flaws customer
Incomplete/hidden
requirements
Moving targets
#3 Problem Underspecified
requirements
Communication
flaws project team
Communication
flaws customer
#4 Problem Communication
flaws project team
Separation reqs. /
solutions
Time boxing
#5 Problem Time boxing Weak access
customer needs
Underspecified
requirements
No.
Do teams applying agile practices avoid these issues?

21.
Small companies
(1-50)
Medium companies
(51-250)
Large companies
(> 251)
#1 Problem Incomplete/hidden
requirements
Communication
flaws customer
Incomplete/hidden
requirements
#2 Problem Communication
flaws customer
Incomplete/hidden
requirements
Moving targets
#3 Problem Underspecified
requirements
Communication
flaws project team
Communication
flaws customer
#4 Problem Communication
flaws project team
Separation reqs. /
solutions
Time boxing
#5 Problem Time boxing Weak access
customer needs
Underspecified
requirements
Not at all.
Do teams applying agile practices avoid these issues?

22.
How can we make use of such insights?

23.
How can we make use of such insights?
‣ Support problem-driven research

24.
How can we make use of such insights?
Raw data
‣ Anonymised
‣ Sanitised
‣ Curated (w/ codebooks)
‣ Support problem-driven research

25.
How can we make use of such insights?
Open data repository
tera-PROMISE
http://openscience.us/repo/
(will probably change)
Raw data
‣ Anonymised
‣ Sanitised
‣ Curated (w/ codebooks)
‣ Support problem-driven research

26.
How can we make use of such insights?
Open data repository
tera-PROMISE
http://openscience.us/repo/
(will probably change)
Raw data
‣ Anonymised
‣ Sanitised
‣ Curated (w/ codebooks)
Problem-driven research
supports
‣ Support problem-driven research

27.
Phenomena and measurements

28.
How can we make use of such insights?
‣ Immediate operationalisation for practical environments

29.
How can we make use of such insights?
‣ Immediate operationalisation for practical environments
Example:
evidence-based Risk
Management*
* Based on two MSc. theses: Priscilla Mafra at UFF, Brazil; Michaela Tießler at TUM, Germany

30.
How can we make use of such insights?
1 Build holistic model with cause-effect relationships
‣ Immediate operationalisation for practical environments

31.
How can we make use of such insights?
1 Build holistic model with cause-effect relationships
‣ Immediate operationalisation for practical environments

32.
Implement Bayesian network
including context factors
2
How can we make use of such insights?
1 Build holistic model with cause-effect relationships
‣ Immediate operationalisation for practical environments

33.
How can we make use of such insights?
3
Operationalise as a
risk assessment approach
‣ Immediate operationalisation for practical environments
Build holistic model with cause-effect relationships1
2 Implement Bayesian network including context factors

34.
How can we make use of such insights?
3
Operationalise as a
risk assessment approach
‣ Immediate operationalisation for practical environments
Build holistic model with cause-effect relationships1
2 Implement Bayesian network including context factors

35.
How can we make use of such insights?
3
Operationalise as a
risk assessment approach
‣ Immediate operationalisation for practical environments
Build holistic model with cause-effect relationships1
2 Implement Bayesian network including context factors

36.
How can we make use of such insights?
‣ Immediate operationalisation for practical environments
Build holistic model with cause-effect relationships1
2 Implement Bayesian network including context factors
3 Operationalise as a risk assessment approach
4
Offer situation-spec. guidelines
(corrective/preventive measures)

37.
Preparation
Where do we stand right now?
•Preparation instruments about to be completed
Next steps
•Data disclosure previous runs
•Survey implementation & pilot phase next run
‣ Start independent data collections;
(before summer break)
ISERN 2017 (October)
• Discussion 1st Results
• Analysis strategy
Survey / Data Collection
(Coordinated local replications)
Survey ++
Data Analysis and Publication
(Synthesis, planning, joint publication)
We are here...
ISERN 2018
• Data analysis
and publication
(cont.)

38.
You are cordiallyinvited to join us!
www.re-survey.org
Thank you!
Daniel Méndez
Technical University of Munich
www.mendezfe.org
@mendezfe