1. Evidence Briefings:
Towards a Medium to Transfer Knowledge
from Systematic Reviews to Practitioners
@brunocartaxo @gustavopinto @scbs@soueltonvieira
moving to
1

2. Before we start..
2

3. Evidence-Based Medicine
EBM is the conscientious, explicit,
judicious and reasonable use of modern,
best evidence in making decisions about
the care of individual patients.
It integrates the clinical experience and
patient values with the best available
research
It aims to increase the use of high
quality clinical research in clinical
decision making
Acta Informatica Medica’2008 3

4. Evidence-Based Medicine
One of the greatest
achievements of EBM has been
the development of systematic
reviews and meta-analyses,
methods by which researchers
identify multiple studies on a
topic, separate the best ones and
then critically analyze them to
come up with a summary of the
best available evidence.
Acta Informatica Medica’2008 4

5. Evidence-Based Practice
5
EB Psychology EB Psychiatry
EBM
EB Education
extends

6. Evidence-Based Practice
6
EB Psychology EB Psychiatry
EBM
EB Education
extends
EB Software Eng.

7. Evidence-Based Software Engineering
Evidence-Based Software
Engineering (EBSE) is a way
to integrate the best research
evidence with practice.
Barbara Kitchenham
Keele University
ICSE’2004
7

8. 1. Convert the need for information into
answerable questions
2. Identify the best evidence with which to
answer these questions
3. Appraise the evidence critically: assess
its validity and usefulness
4. Implement the results of this appraisal in
software engineering practice
5. Evaluate the performance of this
implementation
The 5 Steps of Evidence-Based Practice
}Systematic
Literature
Reviews
(SRs)
8

9. How to find the best evidence?
Software Engineering
conferences
Software Engineering
journals
9

10. How to find the best evidence?
X, Y, and Z: A Systematic
Literature Review
10

11. It involves reading the right
papers and then changing
behavior in the practice of the
discipline.
Trisha Greenhalgh
Oxford University
Evidence-Based practice is not only about reading
and summarizing papers
11

12. Fábio Queda
CIn @ UFPE
There is a lack of connection
between systematic reviews
and software engineering
practice
IST’2011
12

13. SRs do not provide guidelines for practitioners
Most of SR authors
affirmed that they
hadn’t a direct impact
on industrial practice
Lack of connection
with industry is the
6th top barrier
13

14. In addition
14

15. Evidence-Based Medicine (EBM)
researchers argue that SRs:
Focus on a narrow question
15
Are time consuming
they usually require
between 6 months and 2
years to complete
(1,139 hours, on average)
Policymakers often
require access to
contextualized
resources that
address a broader
scope of scientific
evidence quickly
On the other hand…

16. 16
Rapid Reviews Briefings and Summaries
56% of the rapid
reviews were conducted
in the last 3 years
“Rapid reviews
simplify the
process of SRs
to produce
information in
a timely
manner”
“[Briefings] translates
existing SRs into actionable
messages in the form of
short accessible briefings”

17. 17
EBM
Researchers
Health
decision-makers

18. 18
EBSE
Researchers
Software
Industry

19. How to transfer knowledge
from systematic reviews to
software engineering
practice?
19

20. 20

21. Evidence Briefings:
Towards a Medium to Transfer Knowledge
from Systematic Reviews to Practitioners
@brunocartaxo @gustavopinto @scbs@soueltonvieira
moving to
21

22. “Evidence Briefings”
22
An one-page
document, extracted
from a systematic
review, that contains
findings useful for
practitioners

23. Approach
23
1
Systematic
Reviews Selection
2
Systematic Review
Data Extraction
3
Evidence Briefings
Generation
4
Evidence Briefings
Evaluation

24. Approach
24
1
Systematic
Reviews Selection
2
Systematic Review
Data Extraction
3
Evidence Briefings
Generation
4
Evidence Briefings
Evaluation

25. 25
1 Systematic Reviews Selection
120
Systematic Reviews

26. 26
1 Systematic Reviews Selection
120
Systematic Reviews
Fábio Queda
CIn @ UFPE
YAY!
citation++

27. 27
1 Systematic Reviews Selection
120
Systematic Reviews
32
Systematic Reviews
with guidelines
Fábio Queda
CIn @ UFPE
YAY!
citation++

28. 28
1 Systematic Reviews Selection
120
Systematic Reviews
32
Systematic Reviews
with guidelines
24
Systematic Reviews
with search strings
Fábio Queda
CIn @ UFPE
YAY!
citation++

29. 29
1 Systematic Reviews Selection
120
Systematic Reviews
32
Systematic Reviews
with guidelines
24
Systematic Reviews
with search strings
12
Selected SRs
Fábio Queda
CIn @ UFPE
YAY!
citation++
# Subject
2 Global software development
3 Agile software development
1 Software testing
1 Software requirements
1 Model based software development
1 Software development productivity
1 Cost and effort estimation
1 Code duplication
1 Software engineering knowledge management

30. Approach
30
1
Systematic
Reviews Selection
2
Systematic Review
Data Extraction
3
Evidence Briefings
Generation
4
Evidence Briefings
Evaluation

31. 31
2 Systematic Reviews Data Extraction
Paper title:
Original
The effectiveness of pair programing: A meta-analysis
Briefing
The effectiveness of pair programming

32. 32
2 Systematic Reviews Data Extraction
Paper title:
Research goals:
Original
The effectiveness of pair programing: A meta-analysis
Briefing
The effectiveness of pair programming
Template
This briefing reports evidence on <GOAL> based on scientific
evidence from a systematic review.
Briefing
This briefing reports evidence on the effectiveness of pair
programming around quality duration and effort based on
scientific evidence from a systematic review.

33. 33
2 Systematic Reviews Data Extraction
Paper title:
Research goals:
Research findings:
Original
The effectiveness of pair programing: A meta-analysis
Briefing
The effectiveness of pair programming
Template
This briefing reports evidence on <GOAL> based on scientific
evidence from a systematic review.
Briefing
This briefing reports evidence on the effectiveness of pair
programming around quality duration and effort based on
scientific evidence from a systematic review.
Finding 1
Pairing up of individuals seems to elevate the junior pairs up to
near senior pair performance
Finding 2
If you do not know the seniority or skill levels of your
programmers, but do have a feeling for task complexity, then
employ pair programing either when task complexity is low and
time is of the essence, or when task complexity is high and
correctness if important

34. Approach
34
1
Systematic
Reviews Selection
2
Systematic Review
Data Extraction
3
Evidence Briefings
Generation
4
Evidence Briefings
Evaluation

35. 35
3 Evidence Briefings Generation: Principles
Similarity: elements that are similar
are more likely to be organized
together
Proximity: closer elements are more
likely to be perceived as a group
Continuation: elements will be
grouped as a whole if they are co-
linear
Unity: elements that have a visual
connection should belong to a uniform
group

36. 36
3 Evidence Briefings Generation
1. The title of the briefing
2. The goal of the briefing
3. The findings extracted
from the original review
4. An informative box
with general
information
5. The reference to the
original review
6. The logos of our research
group and university

37. 37
3 Evidence Briefings Generation
1. The title of the briefing
2. The goal of the briefing
3. The findings extracted
from the original review
4. An informative box
with general
information
5. The reference to the
original review
6. The logos of our research
group and university
The template of Evidence Briefings is licensed under
CC-BY license!

38. Approach
38
1
Systematic
Reviews Selection
2
Systematic Review
Data Extraction
3
Evidence Briefings
Generation
4
Evidence Briefings
Evaluation

39. RQ: How practitioners and
researchers perceive the
content and format of
Evidence Briefings?
39
4 Evidence Briefings Evaluation

40. 40
4 Evidence Briefings Evaluation: Practitioners
473
StackExchange Users
who asked questions
related to the SRs

41. 41
4
1 SRs Search Strings
Queried a
StackExchange website
2
Removed false-positive
questions
3
Classified questions
using open card sort
4
programmers.stackexchange.com
sqa.stackexchange.com
pm.stackexchange.com
reverseengineering.stackexchange.com
softwarerecs.stackexchange.com
1,738 related questions
473 related questions
Kappa: 0.72
Evidence Briefings Evaluation: Practitioners

42. 42
4
473
StackExchange Users
who asked questions
related to the SRs
Evidence Briefings Evaluation: Practitioners

43. 43
4
473
StackExchange Users who
asked questions related
to the SRs
only 146 of them had
public profile
(LinkedIn, Github,
Twitter, etc)
Evidence Briefings Evaluation: Practitioners

44. 44
4
The 22 authors of the 12
Systematic Reviews
Evidence Briefings Evaluation: Researchers

45. 45
4 Evidence Briefings Evaluation: Survey
Survey principles:
Reciprocity (e.g., we raffled a 100 USD Amazon card gift)
Authority & Credibility (e.g, Ph.D., University professors)
Liking (e.g., personalized emails)
Scarcity: (e.g., we defined a deadline)
Brevity (e.g., we asked closed questions as much as possible)
Social Benefit (e.g., 1 USD for the Brazilian Red Cross)

46. Evidence Briefings Evaluation: Survey
46
4
Survey principles:
Reciprocity (e.g., we raffled a 100 USD Amazon card gift)
Authority & Credibility (e.g, Ph.D., University professors)
Liking (e.g., personalized emails)
Scarcity: (e.g., we defined a deadline)
Brevity (e.g., we asked closed questions as much as possible)
Social Benefit (e.g., 1 USD for the Brazilian Red Cross)

47. 47
4
Survey with Practitioners (17 questions, 2 open-ended)
Survey with Researchers (8 questions, 2 open-ended)
Evidence Briefings Evaluation: Survey
7 respondents (31% of response rate)
32 respondents (22% of response rate)

48. Results
48
Survey with Practitioners
Survey with Researchers

49. Results
49
Survey with Practitioners
Survey with Researchers

50. 50
Survey with Practitioners: Demographics
Q1. What is your current position?
Q2. How many years of experience do you have in your current
position?

51. 51
Survey with Practitioners: Demographics
Q1. What is your current position?
Q2. How many years of experience do you have in your current
position?

52. 52
Survey with Practitioners: Demographics
Q3. Where do you work?
Q4. What is your level of educational attainment?

53. 53
Survey with Practitioners: Demographics
Q3. Where do you work?
Q4. What is your level of educational attainment?

54. 54
Survey with Practitioners: Acquiring Knowledge
Q5. How often do you use StackExchange websites?
Q6. How often do you read software engineering research
papers?

55. 55
Q7. Have you ever read a systematic review paper?
Q8. For what reason you read a systematic review paper?
Survey with Practitioners: Acquiring Knowledge

56. 56
Q10. To what degree do you think the information available in the
briefing we sent to you can answer your question on StackExchange?
Survey with Practitioners: Briefings’ Content

57. 57
Q10. To what degree do you think the information available in the
briefing we sent to you can answer your question on StackExchange?
Q11. Why?
Survey with Practitioners: Briefings’ Content
“The question is too specific”
“The question expected more than one
answer”
“The question touched a slightly
different issue”
“The briefing lacks details”

58. 58
Survey with Practitioners: Briefings’ Content
Q12. Regardless the briefing answers or not your question, how
important do you think is the research presented on the briefing?

59. 59
Q12. Regardless the briefing answers or not your question, how
important do you think is the research presented on the briefing?
Survey with Practitioners: Briefings’ Content
Q13. Why?
“Agile is not a one size fits all methodology. To make it
work you need to see what works for you and your
team. [...] Making bold high level statistical
statements about Agile software development will only
hurt it where as it can shine in truly Agile
organizations.”

60. 60
Q14. How do you compare the answers from the StackExchange
community to the findings presented in the briefing?
Survey with Practitioners: Briefings’ Content

61. 61
Survey with Practitioners: Briefings’ Format
Q15. How easy was to find the information in the briefing?
Q16. Is the briefing interface clear and understandable?
Q17. Does the briefing look reliable?

62. 62
Q15. How easy was to find the information in the briefing?
Q16. Is the briefing interface clear and understandable?
Q17. Does the briefing look reliable?
Survey with Practitioners: Briefings’ Format

63. Results
63
Survey with Practitioners
Survey with Researchers

64. 64
Survey with Researchers: Sharing Knowledge
Q1. How important for you is to share research results to
practitioners?
?

65. 65
Survey with Researchers: Sharing Knowledge
Q1. How important for you is to share research results to
practitioners?

66. 66
Survey with Researchers: Sharing Knowledge
Q2. How often do you share research results to practitioners?

67. 67
Survey with Researchers: Sharing Knowledge
Q2. How often do you share research results to practitioners?

68. 68
Survey with Researchers: Sharing Knowledge
Q2. How often do you share research results to practitioners?
Q3. How do you do that?
“Teaching”
“Seminars”
“Writing”
“Advisory work”
“Social Networks”

69. 69
Survey with Researchers: Briefings’ Content
Q4. How does the briefing that we sent to you cover the main
findings of your paper?

70. 70
Survey with Researchers: Briefings’ Content
Q4. How does the briefing that we sent to you cover the main
findings of your paper?
Q5. Why?
YAY! :-)

71. 71
Survey with Researchers: Briefings’ Format
Q6. How easy was to find the information in the briefing?
Q7. Is the briefing interface clear and understandable?
Q8. Does the briefing look reliable?

72. Discussions
72

73. Revisiting Findings
73
Practitioners rarely use research papers as
mediums to acquire knowledge.
Software engineering practice still has many
beliefs with no evidence basis.

74. Revisiting Findings
74
Both researchers and practitioners positively
evaluated the evidence briefings

75. Revisiting Findings
75
Both researchers and practitioners positively
evaluated the evidence briefings
The briefings well covered the main findings of
the original systematic reviews

76. The Yin-Yang of Research and Practice
76
Researchers want to transfer knowledge.
But not all of them do so.
Practitioners want to be more aware of software
engineering research. But few of them do so.

77. Implications
77

78. Researchers
78

79. Researchers
79

80. Practitioners
80

81. Tool builders
81

82. Tool builders
82

83. Educators
83

84. Conference and Publicity Chairs
84

85. One last thing..
85

86. 86
Search Strings are
your best

87. 87
Assess your search
strings on
StackExchange on early
stages of a systematic
review planning!

88. 88
Are your search strings
well-designed?
“quality + model”, “quality + model driven”
and “model driven + experience”
(software AND ((cost OR effort OR
productivity) WITH (factors OR indicators
OR drivers OR measure)))

89. 89
Thanks!

90. Evidence Briefings:
Towards a Medium to Transfer Knowledge
from Systematic Reviews to Practitioners
@brunocartaxo @gustavopinto @scbs@soueltonvieira
moving to
90