Context: The study selection process is critical to improve the reliability of secondary studies. Goal: To evaluate the selection strategies commonly employed in secondary studies in software engineering. Method: Building on these strategies, a study selection process was formulated and evalu- ated in a systematic review. Results: The selection process used a more inclusive strategy than the one typically used in secondary studies, which led to additional relevant articles. Conclusions: The results indicates that a good-enough sample could be obtained by following a less inclusive but more efficient strategy, if the articles identified as relevant for the study are a representative sample of the population, and there is a homogeneity of results and quality of the articles.

1. Evaluating strategies for !
study selection in !
systematic literature studies "
Nauman bin Ali (nal@bth.se), "
Kai Petersen (kps@bth.se)"
"
Blekinge Institute of Technology, Sweden"

2. Outline"
• Study selection process and decision
rules based on existing guidelines
• Results of evaluating the selection
process and decision rules in a
systematic literature review
• Lessons learned

3. Selection process based on
identified strategies"
Selection criteria
Review of
selection
criteria
Specify
selection
criteria in the
review
protocol
`Think-aloud'
application of
criteria
Specify Update Update
Pilot
selection on
a subset
Acceptable
level of
agreement
Analyse
disagreements
& calculate
inter-rater
agreement
Perform
Selection
Calculate
inter-rater
agreement
No
Yes
Apply
decision
rules
Update
Petersen and Ali, “Identifying strategies for study selection in systematic reviews and maps”, International Empirical Software
Engineering and Measurement, pp.351,354, 22-23 Sept. 2011

4. Decision rules employed by
researchers"
D1: ‘‘Majority vote i.e. a group of reviewers take a vote
on the article and the decision of the majority is
followed”
D2: “At least one relevant then include”
D3: “All relevant then include”
D4: “At least one uncertain then include”
D5: “One irrelevant and one uncertain then exclude”
D6: “All researchers vote uncertain then include”
D7: “All irrelevant then exclude”
Petersen and Ali, “Identifying strategies for study selection in systematic reviews and maps”, International Empirical Software
Engineering and Measurement, pp.351,354, 22-23 Sept. 2011

5. Structuring the selection
decisions"
Reviewer 1
Relevant Uncertain Irrelevant
Reviewer 2
Relevant A B D
Uncertain B C E
Irrelevant D E F
• Level of inclusiveness – decision rules
• Highlights shortcomings of the criteria – too many articles in category `C’ or `D’
• Will discussion be useful? Or we need further information to make an informed
decision – category `C’
• Degree of disagreement should guide decision making and additional effort
expended – likely relevant `B’ and likely irrelevant ‘E’ – currently it is not used
• Improves repeatability as decisions are unambiguous

6. Contribution vs. Overhead"
Cat. ID # of
articles
Primary
studies
Overhead Contribution
A 96 51 46 % 59 %
B 34 11 67 % 13 %
C 122 16 86 % 19 %
D 30 3 90 % 4 %
E 82 6 92 % 7 %
F 1542 0 - -
Articles: 1906
Reviewers: 2
Primary studies: 87
% agreement: 92.50
Cohen Kappa: 0.73
• With only one reviewer up to 11% of the articles could have
been lost.
• Major contribution 59% comes from category A
• 19% of the primary studies come from articles where a
decision could not be made based on title and abstract alone.

7. Analyzing decision rules"
Reviewer 2
Relevant A B D
Uncertain B C E
Irrelevant D E F
D2: “At least one relevant then include”
D3: “All relevant then include”
D4: “At least one uncertain then include”
D5: “One irrelevant and one uncertain then exclude”
D6: “All researchers vote uncertain then include”
D7: “All irrelevant then exclude”
Decision rule # of articles Primary
studies
Reviewer 1
Relevant Uncertain Irrelevant
Overhead Contribution
D3: {A} 96 51 46 % 59 %
D2: {A,B,D} 160 65 59 % 75 %
D6: {A,B,C} 252 78 69 % 90 %
D5: {A,B,C,D} 282 81 71 % 93 %
D4: {A,B,C,E} 334 84 74 % 97 %
D7: {A,B,C,D,E} 364 87 76 % 100 %

8. Lessons learned"
• Interpret the inter-rater statistic carefully
• Use multiple reviewers for selection
• Improve reliability by documenting the selection
criteria process and decision rules to resolve
disagreements.
• We need further studies to derive any general
conclusions, but until then more inclusive
strategies should be pursued.
• Utilize the level of disagreement on relevance of
articles to make an informed decision about
selection and investing additional effort.

9. Thanks!"