1. A New Collaborative Algorithm
for the Selection of Ideas
by Jana Goers, Graham Horton and Nadine Kempe

2. Group of experts screening ideas

3. Challenges in Idea Selection
Situation Problems
 Many ideas  Lack of consensus of idea
 Ill-defined ideas selection leads to a lack of
 Group of experts commitment for
implementation
 High cognitive load when
forced to select among 100 or
more ideas
Group Task  Rejecting good ideas by mistake
 Select the ideas which are
worth pursuing

4. Requirements For a Good Idea Selection
Requirements:
1. Fast selection process
2. Consensus about selection result
3. Low cognitive load
 No method that fulfills these requirements.

5. Alternative 1
We could use a group discussion.
This would deliver: But:
 High acceptance of result  High cognitive load
 Slow

6. Alternative 2
Or we could partition the task.
This would deliver: But:
 Fast selection procedure  Low acceptance of result
 Low cognitive load

7. Is There Something in Between?
A method that combines:
 Speed and cognitive load comparable to parallelised selection
 Consensus comparable to a group discussion
But what could this collaborative approach look like?

8. A Principle of Computer Science
"Divide and Conquer" principle:
 The given problem is too big to solve.
1. So, we divide the problem...
2. ... and solve each sub-problem.
3. We obtain the overall solution by putting together the sub-solutions.

9. Sketch of Our Approach
Adaption of "Divide and Conquer" for idea selection:
1 Partition the ideas among the participants.
2a) Perform local selections on each partition.
(Parallel work; establishes local idea quality)
3 Perform selection of partition representatives.
(Group work; establishes global idea quality)
2b) Revise the local selections.
(Parallel work; corrects local selections according to global quality)
4 The overall selection is now the unification of the local selections.

10. The Threshold Algorithm
Step 1: Partition pool of ideas into subsets of five

11. The Threshold Algorithm
Step 2: Assign each subset to a group member

12. The Threshold Algorithm
Step 3: Each group member performs local selection
locally selected
ideas
locally rejected
ideas

13. The Threshold Algorithm
Step 4: Each group member selects ideas for his subset
Mark worst idea of
your selected ideas! locally selected
ideas
local threshold idea
locally rejected
ideas

14. The Threshold Algorithm
Result (4): Five local selections based on local quality

15. The Threshold Algorithm
Step 5: Group discussion only of local threshold ideas

16. The Threshold Algorithm
Step 6: Group determines global threshold idea

17. The Threshold Algorithm

18. The Threshold Algorithm
Step 7: Each group member revises local selection

19. Experiments
Parallel Threshold Discussion
Group 1 Idea Set 1 Idea Set 2 Idea Set 3
Group 2 Idea Set 2 Idea Set 3 Idea Set 1
Group 3 Idea Set 3 Idea Set 1 Idea Set 2
Group 4 Idea Set 1 Idea Set 2 Idea Set 3
What were the measurements?
 Actual and perceived duration
 Cognitive load
 Acceptance of results
 Number of rejection errors

20. Actual Duration in Minutes
12:23:40 AM
Duration in Minutes
12:21:36 AM
12:14:24 AM
12:11:45 AM
12:07:12 AM 12:06:00 AM
12:00:00 AM
Parallel Threshold Discussion
The threshold approach ...
 needs twice the time of the parallelised method,
 but only half the time of the discussion.

21. But More Interesting Is the Perceived Speed
100
Summed Points of
90
Evaluation Form
81 80
80
70
60 58
50
Parallel Threshold Discussion
Even though the threshold needs twice the time of
the parallel method, ...
 it is on the same subjective speed level as the parallel method.

22. Cognitive Load
50 46
39
Summed Points of
40
Evaluation Form
30
30
20
10
0
Parallel Threshold Discussion
The cognitive load of the threshold approach ...
 is not as high as for the discussion but not as low as for the parallel
method.

23. Acceptance of Selection Results
100
Summed Points of
Evaluation Form
90 86
80 76
74
70
60
Parallel Threshold Discussion
Unfortunately, the threshold approach ...
 has a similar subjective acceptance as the parallel method.

24. Number of Rejection Errors
40
33
rejection errors
30
Number of
20 18
16
10
0
Parallel Threshold Discussion
However, the threshold approach ...
 produces as few rejection errors as the discussion.

25. Acceptance and Rejection Errors
But why such a bad performance regarding acceptance?
Our suspicion:
 Participants mistrusted the algorithm due to lack of understanding
A Master Thesis is currently investigating this.

26. Conclusion
Our approach was able to fulfill our requirements:
1. Fast selection process
2. Consensus in selection result  few rejection errors
3. Low cognitive load
 The threshold method is an appropriate trade-off.

27. Outlook
Our next steps:
 Could a better understanding of the method improve acceptance?
 Could we reduce cognitive load by using pairwise comparisons?
 What could a multi-criteria approach look like?
 Could we obtain more speed by using a computer-supported method?
 Are there other applications?

28. Two Insights
1. Individual and group tasks:
 We learned that a collaborative selection doesn't have to be
executed entirely by a group.
 Are there other collaborative tasks which can benefit from the
"Divide and Conquer" principle?
2. Abstraction of what we did:
 We designed a collaborative selection process by using a principle of
Computer Science.
 Which other principles could be of value for collaborative tasks?