Using Interactive GA for Requirements Prioritization

1. Using Interactive GA for Requirements
Prioritization
Paolo Tonella, Angelo Susi, Francis Palma
Fondazione Bruno Kessler
Software Engineering Research Unit
Trento, Italy
{tonella | susi}@fbk.eu, francis.palma@studenti.unitn.it

2. Outline
• The problem of requirements prioritization
• Approaches
• Our Interactive Genetic Algorithm approach
– The inputs (requirements, Domain knowledge / constraints, feedback
from human evaluators )
– The algorithm and evolution laws
• Algorithm explained via an execution
• Exploiting the approach on the field
• Conclusions
2 Benevento, Italy, September 7th
, 2010

3. The problem
• Prioritization of requirements: the activity of finding
an order relation on the set of requirements under
analysis
• Prioritize according to information concerning the
• available budget
• time constraints
• stakeholder expectations
• technical constraints
3 Benevento, Italy, September 7th
, 2010

4. Sorting approaches
The rank of a requirement can be expressed as its relative position with respect
to the other requirements in the set, as in Bubble sort or Binary search
procedures
Or as an absolute measure of the evaluation criterion for the requirement, as in
Cumulative voting
The methods are highly sensible to errors
Pairwise approaches
Analytic Hierarchy Process (AHP): stat-of-the-art pairwise comparison approach
(considers only user feedback on the set of alternative requirements)
The method is not scalable
CBRank: pairwise approach based on Machine Learning techniques that take into
account the user feedback and the domain knowledge
This algorithm does not allow to represent some kinds of constraints categories (such
as dependencies)
Genetic algorithm
Genetic Algorithm as NSGA- II algorithm
It that does not consider the possibility of exploiting user feedback
Approaches
4 Benevento, Italy, September 7th
, 2010

5. • Based on Interactive Genetic Algorithm (IGA)
– aims at minimizing the disagreement between a total
order of prioritized requirements and the various
constraints that are either encoded with the requirements
or that are expressed iteratively by the user during the
prioritization process
We try to overcame the limits of the other approaches:
considering user knowledge, minimizing the requests to users
to pay attention to scalability problems, considering the
possibility of arbitrary constraints and assuring robustness to
errors
Our Approach
5 Benevento, Italy, September 7th
, 2010

6. • Set of Requirements
• Domain Knowledge: available as requirements
documentation (e.g., cost of the implementation, value for
the stakeholders, dependencies between requirements) that
can be converted into total or partial rankings of the
requirements
• Evaluation from users in terms of orderings between pairs of
requirements
The Input
6 Benevento, Italy, September 7th
, 2010

7. 1. Acquisition and coding of set of Requirements and Domain Knowledge
2. Interactive Genetic Algorithm: computation of solutions (individuals), also
exploiting evaluations from users
3. Output of the ranking (the most promising individual)
The process
7 Benevento, Italy, September 7th
, 2010
R3
R3 R2
R2
R1
R1 R4
R4
R1
R1 R2
R2
R3
R3
R4
R4
R5
R5
Domain knowledge
(constraints)
User feedback
R1 R2
R3 R4
R1 R2
R3 R4
Set of requirements
Interactive Genetic
Algorithm
R2
R2
R1
R1
R3
R3
R4
R4

8. 2.1. computation of a first set of solutions (individuals)
2.2. identification of conflicts (ties) between individuals and
constraints
2.3. request of knowledge to users (to decide about conflicts)
2.4. computation of new solutions
– via evolution rules
2.5. If max number of iterations
– than exit
– else 2.2
The IGA algorithm: step 2.
8 Benevento, Italy, September 7th
, 2010

9. Executing the algorithm
9 Benevento, Italy, September 7th
, 2010

10. Req Priorities Dependencies
R1 High R2, R3
R2 Low R3
R3 Low
R4 Medium R3
R5 Medium
Domain Knowledge coding into
graphs
10 Benevento, Italy, September 7th
, 2010
Transform the domain knowledge into graphs
R3
R3R2
R2
R1
R1
R5
R5R4
R4
Priorities
R3
R3 R2
R2
R1
R1 R4
R4
Dependencies

11. Individual
ID
Requirements rankings
(Individual)
Disagree
Pr1 < R3,R2,R1,R4,R5 >
Pr2 < R3,R2,R1,R5,R4 >
Pr3 < R1,R3,R2,R4,R5 >
Pr4 < R2,R3,R1,R4,R5 >
Pr5 < R2,R3,R4,R5,R1 >
Pr6 < R2,R3,R5,R4,R1 >
Production of individuals and
identification of conflicts
11 Benevento, Italy, September 7th
, 2010
R3
R3R2
R2
R1
R1
R5
R5R4
R4
Priorities
R3
R3
R2
R2
R1
R1
R4
R4
R5
R5
Conflicts =
{(R3, R1),
(R3, R4),
(R3, R5)}
dis(pr1,pr2)={(r,s)∈pr1*|(r,s)∈pr2*}

12. Individual
ID
Requirements rankings
(Individual)
Disagree
Pr1 < R3,R2,R1,R4,R5 >
Pr2 < R3,R2,R1,R5,R4 >
Pr3 < R1,R3,R2,R4,R5 >
Pr4 < R2,R3,R1,R4,R5 >
Pr5 < R2,R3,R4,R5,R1 >
Pr6 < R2,R3,R5,R4,R1 >
Production of individuals and
identification of conflicts
12 Benevento, Italy, September 7th
, 2010
R3
R3R2
R2
R1
R1
R5
R5R4
R4
Priorities
R3
R3
R2
R2
R1
R1
R4
R4
R5
R5
Conflicts =
{(R2, R1),
(R2, R4),
(R2, R5),
(R3, R1),
(R3, R4),
(R3, R5)}
dis(pr1,pr2)={(r,s)∈pr1*|(r,s)∈pr2*}
… and so on …
6
6
7
9
9
6

13. TIE PAIRS
Pr1, Pr2, Pr3 (R4, R5), (R1, R2), (R1, R3)
Pr5,Pr6 (R4, R5)
Pairs to be evaluated to choose
the individual
13 Benevento, Italy, September 7th
, 2010
Individual ID Requirements rankings
(Individual)
Disagree
Pr1 < R3,R2,R1,R4,R5 > 6
Pr2 < R3,R2,R1,R5,R4 > 6
Pr3 < R1,R3,R2,R4,R5 > 6
Pr4 < R2,R3,R1,R4,R5 > 7
Pr5 < R2,R3,R4,R5,R1 > 9
Pr6 < R2,R3,R5,R4,R1 > 9
PR1 = < R3,R2,R1,R4,R5 >
vs
PR2 = < R3,R2,R1,R5,R4 >
(R4,R5)
Candidate pairs
to be asked to
decision maker
Ranked individuals
with respect to
disagreement

14. User feedback
14
Benevento, Italy, September 7th
, 2010
TIE PAIRS
Pr1, Pr2, Pr3 (R4, R5), (R1, R2), (R1, R3)
Pr5,Pr6 (R4, R5)
R3
R3R2
R2
R1
R1
R5
R5R4
R4
Priorities
R3
R3 R2
R2
R1
R1 R4
R4
Dependencies
Why (R4,R5) ?
Nothing is said about (R4,R5) in the
Priorities and Dependencies graphs
Why (R1,R2) and (R1,R3) ?
<>
?
R1
R1
R2
R2
R3
R3
R4
R4
R5
R5
User Preference Graph eliOrd
<
or
>
Contradiction
(R1,R3)
Contradiction
(R1,R2)

15. Population evolution: crossover
15 Benevento, Italy, September 7th
, 2010
cut-head/fill-in-tail and cut-tail/fill-in-head
R3
R3 R2
R2 R1
R1 R5
R5 R4
R4
R1
R1 R3
R3 R2
R2 R4
R4 R5
R5
Positions 2-3 as cut
points
to cut-head/fill-in-tail
R3
R3 R2
R2 R1
R1 R4
R4 R5
R5
Pr2
Pr3
Pr2’

16. Population evolution: crossover
16 Benevento, Italy, September 7th
, 2010
cut-head/fill-in-tail and cut-tail/fill-in-head
R3
R3 R2
R2 R1
R1 R5
R5 R4
R4
R1
R1 R3
R3 R2
R2 R4
R4 R5
R5
Positions 2-3 as cut
points
to cut-head/fill-in-tail
R2
R2 R5
R5 R4
R4R1
R1 R3
R3
Pr2
Pr3
Pr3’

17. Population evolution: mutation
17 Benevento, Italy, September 7th
, 2010
swap operators
R3
R3 R2
R2 R1
R1 R4
R4 R5
R5
Pr1
R3
R3 R2
R2 R4
R4 R1
R1 R5
R5
Pr1’
R1 and R4 to swap

18. • The new evolved population
• is compared against the new set of constraints
graphs
New round of the algorithm
18 Benevento, Italy, September 7th
, 2010
R3
R3R2
R2
R1
R1
R5
R5R4
R4
Priorities
R3
R3 R2R2
R1
R1 R4
R4
Dependencies
R1
R1
R2
R2
R3
R3
R4
R4
R5
R5
User Preference Graph eliOrd
R3
R3 R2
R2 R4
R4 R1
R1 R5
R5
Pr1’
R3
R3 R2
R2
R2
R2 R5
R5 R4
R4R1
R1 R3
R3
R1
R1 R4
R4 R5
R5
Pr3’
Pr2’

19. Evaluating the approach
19 Benevento, Italy, September 7th
, 2010

20. • Prioritize requirements for a real software system, as part of the project
ACube (Ambient Aware Assistance)
– designing a highly technological monitoring environment to be deployed in
nursing homes to support medical and assistance staff
• After user requirements analysis phase,
– 60 user requirements and 49 technical requirements
– Four macro-scenarios have been identified.
Case Study
20 Benevento, Italy, September 7th
, 2010
Id Macro-scenario # of requirements
FALL Monitoring falls 26
ESC Monitoring escapes 23
MON Monitoring dangerous behavior 21
ALL The three scenarios 49

21. • Two sets of technical constraints:
– Priority: a function that associates technical requirement to a number
indicating the priority of the technical requirement with respect to the priority
of the user requirements it is intended to address
– Dependency: is defined on the basis of the dependencies between
requirements
• For each of the four macro-scenarios, we obtained the Gold
Standard (GS) prioritization from the software architect of the
ACube project
– The GS prioritization is the ordering given by the software architect to
the requirements when planning their implementation during the
ACube project
Ingredients
21 Benevento, Italy, September 7th
, 2010

22. RQ1: convergence
22 Benevento, Italy, September 7th
, 2010
The best individual in each population converges toward a
low value of the final fitness function
RQ1 (Convergence) Can we observe convergence with
respect to the finally elicited fitness function?
IGA converges even though the fitness function is known in
its complete form only at the end of the elicitation process

23. RQ2: Role of interaction
23 Benevento, Italy, September 7th
, 2010
RQ2 (Role of interaction) Does IGA produce improved prioritizations
Compared to non-interactive requirement ordering?
IGA outperforms substantially GA (and RAND), especially when a higher
number of pairwise comparisons can be carried out

24. RQ3: Role of initial precedence
constraints
24 Benevento, Italy, September 7th
, 2010
RQ3 (Role of initial precedence constraints) How does initial availability
of precedence constraints affect the performance of IGA?
The improvement of IGA over GA is even higher when limited ranking
information is available a-priori

25. RQ4: Robustness
25 Benevento, Italy, September 7th
, 2010
RQ4 (Robustness) Is IGA robust with respect to errors committed by
the user during the elicitation of pairwise comparisons?
It seems (more complex error models needed)

26. • Cost/benefit trade off offered by IGA as compared to AHP is
extremely interesting
– With an elicitation effort reduced to 10% of the one required by AHP,
IGA produces an approximate ordering which has a quite low average
distance from the requirement positions in the GS
Threats
• External validity (generalization of the findings):
– we used (only one) real project with four scenarios
• Construct Validity:
– disagreement measure is used in other contexts
– error model is simple
Discussion
26 Benevento, Italy, September 7th
, 2010

27. • We proposed an interactive genetic algorithm to collect pairwise
information useful to prioritize the requirements for a software system
• Our approach scaled to the size of the considered case study
• We also verified the robustness of the algorithm with respect to
increasing user feedback errors
• Finally we evaluated the approach in a real project
What’s next?
• Algorithm:
– refining the algorithm
– the evolution operators (e.g., introducing new heuristics)
• Experiment
– off-line: comparisons with other approaches
– on-line: controlled experiments on the field
Conclusion and Future
27 Benevento, Italy, September 7th
, 2010

28. Thank you
28 Benevento, Italy, September 7th
, 2010

29. 29 Benevento, Italy, September 6th
, 2010

30. Population evolution: crossover
30 Benevento, Italy, September 6th
, 2010
cut-head/fill-in-tail and cut-tail/fill-in-head
R3R3 R2R2 R1R1 R4R4 R5R5Pr1
R3R3 R2R2 R1R1 R4R4 R5R5Pr1’
R1 and R4 to swap

31. Outline
• The problem
• Research baseline
– GA and IGA approaches
– Other approaches (?)
• Our approach
– The data
• requirements
• Domain knowledge / constraints (existing rankings and existing dependencies, e.g.
from human domain experts, domain laws, …)
• Feedback from human evaluators (to solve ties)
– The algorithm and evolution laws (operators) [to let universe evolve towards
the fitness specified by domain knowledge + human evaluators]
• Algorithm explained via an execution
• Exploiting the approach on the field
• Conclusions
31 Benevento, Italy, September 6th
, 2010

32. Population evolution: mutation
32 Benevento, Italy, September 6th
, 2010
cut-head/fill-in-tail and cut-tail/fill-in-head
R3
R3 R2
R2 R1
R1 R5
R5 R4
R4
R1
R1 R3
R3 R2
R2 R4
R4 R5
R5
Positions 2-3 as cut points to cut-head/fill-in-tail
R3
R3 R2
R2
R1
R1 R5
R5 R4
R4R1
R1 R3
R3
R2
R2 R4
R4 R5
R5
Pr2
Pr3
Pr3’
Pr2’

33. Population evolution: mutation
33 Benevento, Italy, September 6th
, 2010
cut-head/fill-in-tail and cut-tail/fill-in-head
R3
R3 R2
R2 R1
R1 R5
R5 R4
R4
R1
R1 R3
R3 R2
R2 R4
R4 R5
R5
Positions 2-3 as cut
points
to cut-head/fill-in-tail
R3
R3 R2
R2 R1
R1 R4
R4 R5
R5
Pr2
Pr3
Pr2’

34. Population evolution: mutation
34 Benevento, Italy, September 6th
, 2010
cut-head/fill-in-tail and cut-tail/fill-in-head
R3
R3 R2
R2 R1
R1 R5
R5 R4
R4
R1
R1 R3
R3 R2
R2 R4
R4 R5
R5
Positions 2-3 as cut
points
to cut-head/fill-in-tail
R2
R2 R5
R5 R4
R4R1
R1 R3
R3
Pr2
Pr3
Pr3’

35. Population evolution: crossover
35 Benevento, Italy, September 6th
, 2010
cut-head/fill-in-tail and cut-tail/fill-in-head operators
R3
R3 R2
R2 R1
R1 R5
R5 R4
R4
R1
R1 R3
R3 R2
R2 R4
R4 R5
R5
Positions 2-3 as cut points to cut-head/fill-in-tail
R3
R3 R2
R2 RR R5
R5 R4
R4
R1
R1 R3
R3 R2
R2 R4
R4 R5R5
Pr2
Pr3
Pr3’
Pr2’

36. Approaches
• process can be designed as an a-priori or as an a-
posteriori process.
– a-priori the preferences are formulated before the specification of the
set of requirements via predefined models (e.g., differential
equations)
– a-posteriori approaches, the ranking is formulated on the basis of the
characteristics of the set of requirements under analysis (e.g., via a
process of pairwise comparison allowing to define at the same time
which requirement and why it has to be preferred between two
alternatives)
36 Benevento, Italy, September 6th
, 2010

37. Four research questions:
RQ1 (Convergence) Can we observe convergence with respect to the finally
elicited fitness function?
RQ2 (Role of interaction) Does IGA produce improved prioritizations
compared to non-interactive requirement ordering?
RQ3 (Role of initial precedence constraints) How does initial availability of
precedence constraints affect the performance of IGA?
RQ4 (Robustness) Is IGA robust with respect to errors committed by the user
during the elicitation of pairwise comparisons?
Research Questions
37 Benevento, Italy, September 6th
, 2010

38. Outline
• The problem
• Research baseline
– GA and IGA approaches
– Other approaches (?)
• Our approach
– The data (requirements, Domain knowledge / constraints, Feedback from
human evaluators )
– The algorithm and evolution laws (operators) [to let universe evolve towards
the fitness specified by domain knowledge + human evaluators]
• Algorithm explained via an execution
• Exploiting the approach on the field
• Conclusions
38 Benevento, Italy, September 6th
, 2010