The document discusses new opportunities for using natural language processing (NLP) techniques to support requirements conversations between analysts and stakeholders. It presents an initial tool called Trace2Conv that aims to trace formal requirements back to informal conversations to provide additional context. A second tool called Requirements Conversation Summarizer is proposed to facilitate analysts in exploring long recorded conversation transcripts by identifying questions and filtering information by relevance.
Vip profile Call Girls In Lonavala 9748763073 For Genuine Sex Service At Just...
Requirements Conversations: A New Frontier in AI-for-RE
1. Requirements Conversations:
A New Frontier in AI-for-RE
Fabiano Dalpiaz
Requirements Engineering Lab
Utrecht University, the Netherlands
August 16, 2022
f.dalpiaz@uu.nl @FabianoDalpiaz
2. Outline and Acks
@2022 Fabiano Dalpiaz
2
1. Context:
NLP4RE
2. Conversational
Req. Engineering
3. From reqs to
specs
4. Conversation
summarization
5. Outlook
3. Outline and Acks
@2022 Fabiano Dalpiaz
2
1. Context:
NLP4RE
2. Conversational
Req. Engineering
3. From reqs to
specs
4. Conversation
summarization
5. Outlook
Sjaak Brinkkemper
Tjerk Spijkman
Nikita van den Berg
Xavier de Bondt
4. 1. Context: NLP for Requirements
Engineering (NLP4RE)
Fabiano Dalpiaz,Alessio Ferrari, Xavier Franch, and Cristina Palomares. "Natural language processing
for requirements engineering:The best is yet to come." IEEE Software 35, no. 5 (2018): 115-119.
@2022 Fabiano Dalpiaz
3
5. RE practice: most reqs. are in natural language
@2022 Fabiano Dalpiaz
4
The <system name> shall <system response>.
WHILE <in a specific state> the <system name> shall <system
response>
WHEN <trigger> the <system name> shall <system response>
…
User stories
App store
reviews
EARS
Rupp’s template & ISO/IEC/IEEE 29148
Use cases
6. RE Research: 4 categories of NLP4RE tools
@2022 Fabiano Dalpiaz
5
1. Find defects /
deviations from
good practice
2. Generate models
from NL requirements
3. Infer trace links
between NL
requirements and
other artifacts
4. Identify key
abstractions
from NL
documents
Daniel Berry, Ricardo Gacitua, Pete Sawyer, and Sri Fatimah Tjong. "The case for dumb
requirements engineering tools." In InternationalWorking Conference on Requirements
Engineering: Foundation for Software Quality, pp. 211-217. 2012.
7. An active area of research!
@2022 Fabiano Dalpiaz
6
Liping Zhao,Waad Alhoshan, Alessio Ferrari,
Keletso J. Letsholo, Muideen A.Ajagbe, Erol-
Valeriu Chioasca, and Riza T. Batista-Navarro.
"Natural Language Processing (NLP) for
Requirements Engineering:A Systematic
Mapping Study." ACM Computing Surveys 54:3,
2022
8. An active area of research!
@2022 Fabiano Dalpiaz
6
Liping Zhao,Waad Alhoshan, Alessio Ferrari,
Keletso J. Letsholo, Muideen A.Ajagbe, Erol-
Valeriu Chioasca, and Riza T. Batista-Navarro.
"Natural Language Processing (NLP) for
Requirements Engineering:A Systematic
Mapping Study." ACM Computing Surveys 54:3,
2022
Specification
Main artifact for
AI-based tools
18. Background theory: Refinement in RE
@2022 Fabiano Dalpiaz
11
Pohl, Klaus. "The three dimensions of requirements
engineering: a framework and its applications."
Information systems 19.3 (1994): 243-258.
Specification
Representation
opaque
fair
complete
common view
informal semi-formal formal
personal view
Initial RE
input
Desired RE output
Agreement
19. Background theory: Refinement in RE
@2022 Fabiano Dalpiaz
11
Pohl, Klaus. "The three dimensions of requirements
engineering: a framework and its applications."
Information systems 19.3 (1994): 243-258.
Specification
Representation
opaque
fair
complete
common view
informal semi-formal formal
personal view
Initial RE
input
Desired RE output
Agreement
Refinement
path in
practice
20. Background theory: Refinement in RE
@2022 Fabiano Dalpiaz
11
Pohl, Klaus. "The three dimensions of requirements
engineering: a framework and its applications."
Information systems 19.3 (1994): 243-258.
Specification
Representation
opaque
fair
complete
common view
informal semi-formal formal
personal view
Initial RE
input
Desired RE output
Agreement
Refinement
path in
practice
RE research approaches,
including NLP4RE Tools
21. How do current NLP4RE tools work?
@2022 Fabiano Dalpiaz
12
23. NLP4RE Tools for Conversational RE
@2022 Fabiano Dalpiaz
13
} The tool supports the conversation between analyst and stakeholders
vs.
24. NLP4RE Tools for Conversational RE
@2022 Fabiano Dalpiaz
13
} The tool supports the conversation between analyst and stakeholders
} Elicitation and refinement as concurrent activities
vs.
25. Conversational RE
@2022 Fabiano Dalpiaz
14
“The (automated) analysis of requirements
elicitation conversations aimed at identifying and
extracting requirements-relevant information”
28. (Requirements) conversations vs. specifications
@2022 Fabiano Dalpiaz
15
2+ parties (here Analyst
and Stakeholder)
Informal: no “shall”
statements, user
stories, glossary
29. (Requirements) conversations vs. specifications
@2022 Fabiano Dalpiaz
15
2+ parties (here Analyst
and Stakeholder)
Informal: no “shall”
statements, user
stories, glossary
Relevant
information may
be sparse
30. (Requirements) conversations vs. specifications
@2022 Fabiano Dalpiaz
15
2+ parties (here Analyst
and Stakeholder)
Informal: no “shall”
statements, user
stories, glossary
Relevant
information may
be sparse
Includes persuasion,
uncertainty,
misunderstandings
31. Dissecting a conversation: turns and grounding acts
@2022 Fabiano Dalpiaz
16
Traum, David R., and Elizabeth A. Hinkelman.
"Conversation acts in task-oriented spoken dialogue."
Computational intelligence 8.3 (1992): 575-599.
32. Dissecting a conversation: turns and grounding acts
@2022 Fabiano Dalpiaz
16
Traum, David R., and Elizabeth A. Hinkelman.
"Conversation acts in task-oriented spoken dialogue."
Computational intelligence 8.3 (1992): 575-599.
Turns and utterance units as
atomic entities
33. Dissecting a conversation: turns and grounding acts
@2022 Fabiano Dalpiaz
16
Traum, David R., and Elizabeth A. Hinkelman.
"Conversation acts in task-oriented spoken dialogue."
Computational intelligence 8.3 (1992): 575-599.
Turns and utterance units as
atomic entities
Grounding acts determine
the effect of an
utterance unit
34. Dissecting a conversation: discourse units
@2022 Fabiano Dalpiaz
17
Traum, David R., and Elizabeth A. Hinkelman.
"Conversation acts in task-oriented spoken dialogue."
Computational intelligence 8.3 (1992): 575-599.
35. Dissecting a conversation: discourse units
@2022 Fabiano Dalpiaz
17
Cross-speaker interaction
defines the meaning
Traum, David R., and Elizabeth A. Hinkelman.
"Conversation acts in task-oriented spoken dialogue."
Computational intelligence 8.3 (1992): 575-599.
36. Traum, David R., and Elizabeth A. Hinkelman.
"Conversation acts in task-oriented spoken dialogue."
Computational intelligence 8.3 (1992): 575-599.
Dissecting a conversation: argumentation acts
@2022 Fabiano Dalpiaz
18
37. Traum, David R., and Elizabeth A. Hinkelman.
"Conversation acts in task-oriented spoken dialogue."
Computational intelligence 8.3 (1992): 575-599.
Dissecting a conversation: argumentation acts
@2022 Fabiano Dalpiaz
18
The purpose of a
conversation across
multiple turns:
argumentation acts
38. Traum, David R., and Elizabeth A. Hinkelman.
"Conversation acts in task-oriented spoken dialogue."
Computational intelligence 8.3 (1992): 575-599.
Dissecting a conversation: argumentation acts
@2022 Fabiano Dalpiaz
18
The purpose of a
conversation across
multiple turns:
argumentation acts
Q&A as a basic interaction,
clarifications, summary,
persuasion, …
39. Conversations vs. Specifications: not quite the same
@2022 Fabiano Dalpiaz
19
The parameters A and B shall be configured via
a configuration file in format XYZ
[specification]
[conversation]
40. Tools for Conversational RE: Two Examples
@2022 Fabiano Dalpiaz
20
Trace2Conv:
pre-RS traceability
Requirements Conversation
Summarizer
44. Requirements
conversations
Requirements Analyst
Own ideas
Budget / project
constraints
Design
decisions Domain-specific
documentation
Trace2Conv: Key Idea
@2022 Fabiano Dalpiaz
22
} Supports backward, pre-RS traceability
} Largely overlooked area of research
Specification
Trace2Conv
45. Requirements
conversations
Requirements Analyst
Own ideas
Budget / project
constraints
Design
decisions Domain-specific
documentation
Trace2Conv: Key Idea
@2022 Fabiano Dalpiaz
22
} Supports backward, pre-RS traceability
} Largely overlooked area of research
} Aims to find information that provides
additional context to a requirement
Specification
Trace2Conv
46. Requirements
conversations
Requirements Analyst
Own ideas
Budget / project
constraints
Design
decisions Domain-specific
documentation
Trace2Conv: Key Idea
@2022 Fabiano Dalpiaz
22
} Supports backward, pre-RS traceability
} Largely overlooked area of research
} Aims to find information that provides
additional context to a requirement
} Has to cope with an abstraction gap
} Formal to informal
Specification
Trace2Conv
47. What can we achieve with Trace2Conv?
@2022 Fabiano Dalpiaz
23
Trace requirements back to elicitation sessions
48. What can we achieve with Trace2Conv?
@2022 Fabiano Dalpiaz
23
Trace requirements back to elicitation sessions
Obtain client agreement by showing requirement sources
49. What can we achieve with Trace2Conv?
@2022 Fabiano Dalpiaz
23
Trace requirements back to elicitation sessions
Obtain client agreement by showing requirement sources
Enable iterative specification writing and review
50. What can we achieve with Trace2Conv?
@2022 Fabiano Dalpiaz
23
Trace requirements back to elicitation sessions
Obtain client agreement by showing requirement sources
Enable iterative specification writing and review
Provide extra context about a requirement for devs
53. Pre-processing and matching
@2022 Fabiano Dalpiaz
25
As a vendor user, I can use the password forgotten
functionality whenever I forgot or want to reset my
password, so that I always have a way to create a new
password
54. Pre-processing and matching
@2022 Fabiano Dalpiaz
25
As a vendor user, I can use the password forgotten
functionality whenever I forgot or want to reset my
password, so that I always have a way to create a new
password
55. Pre-processing and matching
@2022 Fabiano Dalpiaz
25
As a vendor user, I can use the password forgotten
functionality whenever I forgot or want to reset my
password, so that I always have a way to create a new
password
56. Pre-processing and matching
@2022 Fabiano Dalpiaz
25
As a vendor user, I can use the password forgotten
functionality whenever I forgot or want to reset my
password, so that I always have a way to create a new
password
58. Short demo of the Trace2Conv frontend
@2022 Fabiano Dalpiaz
27
59. Short demo of the Trace2Conv frontend
@2022 Fabiano Dalpiaz
27
60. Trace2Conv: Next Steps
@2022 Fabiano Dalpiaz
28
Requirements evolution over
multiple conversations
More advanced heuristics: what
are the most likely matches?
Matching segments rather
than speaker turns
62. } Aim: summarization before a specification exists
} Trigger: long recorded conversations, spanning over multiple hours
} How to facilitate the analyst in exploring the transcript?
Summarizing a transcript: ideas
@2022 Fabiano Dalpiaz
30
63. } Aim: summarization before a specification exists
} Trigger: long recorded conversations, spanning over multiple hours
} How to facilitate the analyst in exploring the transcript?
Summarizing a transcript: ideas
@2022 Fabiano Dalpiaz
30
Idea #1: Identify
the questions
64. } Aim: summarization before a specification exists
} Trigger: long recorded conversations, spanning over multiple hours
} How to facilitate the analyst in exploring the transcript?
Summarizing a transcript: ideas
@2022 Fabiano Dalpiaz
30
Idea #1: Identify
the questions
Idea #2: Filter by
question relevance
65. } Aim: summarization before a specification exists
} Trigger: long recorded conversations, spanning over multiple hours
} How to facilitate the analyst in exploring the transcript?
Summarizing a transcript: ideas
@2022 Fabiano Dalpiaz
30
Idea #1: Identify
the questions
Idea #2: Filter by
question relevance
Idea #3: Label by
relevance type [WiP]
66. How to identify the questions? (Idea #1)
@2022 Fabiano Dalpiaz
31
67. How to identify the questions? (Idea #1)
@2022 Fabiano Dalpiaz
31
Based on sequences of POS tags:
Wh-, yes/no, tag questions
68. How to identify the questions? (Idea #1)
@2022 Fabiano Dalpiaz
31
Based on sequences of POS tags:
Wh-, yes/no, tag questions
Based on pre-trained DistilBert:
28/38 question tags matched
69. How to identify the questions? (Idea #1)
@2022 Fabiano Dalpiaz
31
Based on sequences of POS tags:
Wh-, yes/no, tag questions
Based on pre-trained DistilBert:
28/38 question tags matched
Combination: question if either
approach says so
71. How to filter relevant questions? (Idea #2, version A)
@2022 Fabiano Dalpiaz
33
72. How to filter relevant questions? (Idea #2, version A)
@2022 Fabiano Dalpiaz
33
TF-IDF can be used to distinguish
questions with domain-specific
words from general ones
73. How to filter relevant questions? (Idea #2, version B)
@2022 Fabiano Dalpiaz
34
74. How to filter relevant questions? (Idea #2, version B)
@2022 Fabiano Dalpiaz
34
A context document can be, e.g., a
system/project definition document
75. What is our gold standard for relevance?
@2022 Fabiano Dalpiaz
35
76. What is our gold standard for relevance?
@2022 Fabiano Dalpiaz
35
77. What is our gold standard for relevance?
@2022 Fabiano Dalpiaz
35
78. Is Idea #2 effective?
@2022 Fabiano Dalpiaz
36
No large differences between the
approaches – Ideas #2A and #2B
are practically equivalent
80. Summarization: outlook
@2022 Fabiano Dalpiaz
37
} What does relevance mean?
} Large disagreement, especially on
questions
} How much can we summarize?
} End goal of the tool
82. Tools for Conversational RE: Two Examples
@2022 Fabiano Dalpiaz
39
Trace2Conv:
pre-RS traceability
Requirements Conversation
Summarizer
83. Direction #3: distilling requirements?
@2022 Fabiano Dalpiaz
40
} Can we automatically generate
requirements from conversations?
} Long-term direction
} High value
} Extremely challenging
} Rarely mentioned in an explicit way
(Spijkman, CAiSE’21)
84. Direction #3: distilling requirements?
@2022 Fabiano Dalpiaz
40
Spijkman & al.,
NLP4RE’21
Ruiz & Hasselman,
EMMSAD’20
} Can we automatically generate
requirements from conversations?
} Long-term direction
} High value
} Extremely challenging
} Rarely mentioned in an explicit way
(Spijkman, CAiSE’21)
85. The future of evaluation metrics
@2022 Fabiano Dalpiaz
41
} Most of the literature employs information retrieval metrics
} Precision, recall, F1, …
} Progressive shift toward quality-in-use with conversational RE tools?
86. The future of evaluation metrics
@2022 Fabiano Dalpiaz
41
} Most of the literature employs information retrieval metrics
} Precision, recall, F1, …
} Progressive shift toward quality-in-use with conversational RE tools?
ISO 25022
(quality in use)