Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Evidence Briefings:
Towards a Medium to Transfer Knowledge
from Systematic Reviews to Practitioners
@brunocartaxo @gustavop...
Before we start..
2
Evidence-Based Medicine
EBM is the conscientious, explicit,
judicious and reasonable use of modern,
best evidence in makin...
Evidence-Based Medicine
One of the greatest
achievements of EBM has been
the development of systematic
reviews and meta-an...
Evidence-Based Practice
5
EB Psychology EB Psychiatry
EBM 

EB Education

extends
Evidence-Based Practice
6
EB Psychology EB Psychiatry
EBM 

EB Education

extends
EB Software Eng.
Evidence-Based Software Engineering
Evidence-Based Software
Engineering (EBSE) is a way
to integrate the best research
evi...
1. Convert the need for information into
answerable questions
2. Identify the best evidence with which to
answer these que...
How to find the best evidence?
Software Engineering
conferences
Software Engineering
journals
9
How to find the best evidence?
X, Y, and Z: A Systematic
Literature Review
10
It involves reading the right
papers and then changing
behavior in the practice of the
discipline.
Trisha Greenhalgh
Oxfor...
Fábio Queda
CIn @ UFPE
There is a lack of connection
between systematic reviews
and software engineering
practice
IST’2011...
SRs do not provide guidelines for practitioners
Most of SR authors
affirmed that they
hadn’t a direct impact
on industrial ...
In addition
14
Evidence-Based Medicine (EBM)
researchers argue that SRs:
Focus on a narrow question
15
Are time consuming
they usually re...
16
Rapid Reviews Briefings and Summaries
56% of the rapid
reviews were conducted
in the last 3 years
“Rapid reviews
simplif...
17
EBM
Researchers
Health
decision-makers
18
EBSE
Researchers
Software
Industry
How to transfer knowledge
from systematic reviews to
software engineering
practice?
19
20
Evidence Briefings:
Towards a Medium to Transfer Knowledge
from Systematic Reviews to Practitioners
@brunocartaxo @gustavop...
“Evidence Briefings”
22
An one-page
document, extracted
from a systematic
review, that contains
findings useful for
practiti...
Approach
23
1
Systematic
Reviews Selection
2
Systematic Review
Data Extraction
3
Evidence Briefings
Generation
4
Evidence B...
Approach
24
1
Systematic
Reviews Selection
2
Systematic Review
Data Extraction
3
Evidence Briefings
Generation
4
Evidence B...
25
1 Systematic Reviews Selection
120
Systematic Reviews
26
1 Systematic Reviews Selection
120
Systematic Reviews
Fábio Queda
CIn @ UFPE
YAY!
citation++
27
1 Systematic Reviews Selection
120
Systematic Reviews
32
Systematic Reviews
with guidelines
Fábio Queda
CIn @ UFPE
YAY!...
28
1 Systematic Reviews Selection
120
Systematic Reviews
32
Systematic Reviews
with guidelines
24
Systematic Reviews
with ...
29
1 Systematic Reviews Selection
120
Systematic Reviews
32
Systematic Reviews
with guidelines
24
Systematic Reviews
with ...
Approach
30
1
Systematic
Reviews Selection
2
Systematic Review
Data Extraction
3
Evidence Briefings
Generation
4
Evidence B...
31
2 Systematic Reviews Data Extraction
Paper title:
Original
The effectiveness of pair programing: A meta-analysis
Briefin...
32
2 Systematic Reviews Data Extraction
Paper title:
Research goals:
Original
The effectiveness of pair programing: A meta...
33
2 Systematic Reviews Data Extraction
Paper title:
Research goals:
Research findings:
Original
The effectiveness of pair ...
Approach
34
1
Systematic
Reviews Selection
2
Systematic Review
Data Extraction
3
Evidence Briefings
Generation
4
Evidence B...
35
3 Evidence Briefings Generation: Principles
Similarity: elements that are similar
are more likely to be organized
togeth...
36
3 Evidence Briefings Generation
1. The title of the briefing
2. The goal of the briefing
3. The findings extracted
from the...
37
3 Evidence Briefings Generation
1. The title of the briefing
2. The goal of the briefing
3. The findings extracted
from the...
Approach
38
1
Systematic
Reviews Selection
2
Systematic Review
Data Extraction
3
Evidence Briefings
Generation
4
Evidence B...
RQ: How practitioners and
researchers perceive the
content and format of
Evidence Briefings?
39
4 Evidence Briefings Evaluat...
40
4 Evidence Briefings Evaluation: Practitioners
473
StackExchange Users
who asked questions
related to the SRs
41
4
1 SRs Search Strings
Queried a
StackExchange website
2
Removed false-positive
questions
3
Classified questions
using o...
42
4
473
StackExchange Users
who asked questions
related to the SRs
Evidence Briefings Evaluation: Practitioners
43
4
473
StackExchange Users who
asked questions related
to the SRs
only 146 of them had
public profile
(LinkedIn, Github,
...
44
4
The 22 authors of the 12
Systematic Reviews
Evidence Briefings Evaluation: Researchers
45
4 Evidence Briefings Evaluation: Survey
Survey principles:
Reciprocity (e.g., we raffled a 100 USD Amazon card gift)
Auth...
Evidence Briefings Evaluation: Survey
46
4
Survey principles:
Reciprocity (e.g., we raffled a 100 USD Amazon card gift)
Auth...
47
4
Survey with Practitioners (17 questions, 2 open-ended)
Survey with Researchers (8 questions, 2 open-ended)
Evidence B...
Results
48
Survey with Practitioners
Survey with Researchers
Results
49
Survey with Practitioners
Survey with Researchers
50
Survey with Practitioners: Demographics
Q1. What is your current position?
Q2. How many years of experience do you have...
51
Survey with Practitioners: Demographics
Q1. What is your current position?
Q2. How many years of experience do you have...
52
Survey with Practitioners: Demographics
Q3. Where do you work?
Q4. What is your level of educational attainment?
53
Survey with Practitioners: Demographics
Q3. Where do you work?
Q4. What is your level of educational attainment?
54
Survey with Practitioners: Acquiring Knowledge
Q5. How often do you use StackExchange websites?
Q6. How often do you re...
55
Q7. Have you ever read a systematic review paper?
Q8. For what reason you read a systematic review paper?
Survey with P...
56
Q10. To what degree do you think the information available in the
briefing we sent to you can answer your question on St...
57
Q10. To what degree do you think the information available in the
briefing we sent to you can answer your question on St...
58
Survey with Practitioners: Briefings’ Content
Q12. Regardless the briefing answers or not your question, how
important do...
59
Q12. Regardless the briefing answers or not your question, how
important do you think is the research presented on the b...
60
Q14. How do you compare the answers from the StackExchange
community to the findings presented in the briefing?
Survey wi...
61
Survey with Practitioners: Briefings’ Format
Q15. How easy was to find the information in the briefing?
Q16. Is the briefin...
62
Q15. How easy was to find the information in the briefing?
Q16. Is the briefing interface clear and understandable?

Q17. ...
Results
63
Survey with Practitioners
Survey with Researchers
64
Survey with Researchers: Sharing Knowledge
Q1. How important for you is to share research results to
practitioners?
?
65
Survey with Researchers: Sharing Knowledge
Q1. How important for you is to share research results to
practitioners?
66
Survey with Researchers: Sharing Knowledge
Q2. How often do you share research results to practitioners?
67
Survey with Researchers: Sharing Knowledge
Q2. How often do you share research results to practitioners?
68
Survey with Researchers: Sharing Knowledge
Q2. How often do you share research results to practitioners?
Q3. How do you...
69
Survey with Researchers: Briefings’ Content
Q4. How does the briefing that we sent to you cover the main
findings of your ...
70
Survey with Researchers: Briefings’ Content
Q4. How does the briefing that we sent to you cover the main
findings of your ...
71
Survey with Researchers: Briefings’ Format
Q6. How easy was to find the information in the briefing?
Q7. Is the briefing in...
Discussions
72
Revisiting Findings
73
Practitioners rarely use research papers as
mediums to acquire knowledge.
Software engineering prac...
Revisiting Findings
74
Both researchers and practitioners positively
evaluated the evidence briefings
Revisiting Findings
75
Both researchers and practitioners positively
evaluated the evidence briefings
The briefings well cov...
The Yin-Yang of Research and Practice
76
Researchers want to transfer knowledge.
But not all of them do so.
Practitioners ...
Implications
77
Researchers
78
Researchers
79
Practitioners
80
Tool builders
81
Tool builders
82
Educators
83
Conference and Publicity Chairs
84
One last thing..
85
86
Search Strings are
your best
87
Assess your search
strings on
StackExchange on early
stages of a systematic
review planning!
88
Are your search strings
well-designed?
“quality + model”, “quality + model driven”
and “model driven + experience”
(sof...
89
Thanks!
Evidence Briefings:
Towards a Medium to Transfer Knowledge
from Systematic Reviews to Practitioners
@brunocartaxo @gustavop...
Upcoming SlideShare
Loading in …5
×

0

Share

Download to read offline

Evidence Briefings: Towards a Medium to Transfer Knowledge from Systematic Reviews to Practitioners

Download to read offline

Presentation of our ESEM'16 paper

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

Evidence Briefings: Towards a Medium to Transfer Knowledge from Systematic Reviews to Practitioners

  1. 1. Evidence Briefings: Towards a Medium to Transfer Knowledge from Systematic Reviews to Practitioners @brunocartaxo @gustavopinto @scbs@soueltonvieira moving to 1
  2. 2. Before we start.. 2
  3. 3. Evidence-Based Medicine EBM is the conscientious, explicit, judicious and reasonable use of modern, best evidence in making decisions about the care of individual patients. It integrates the clinical experience and patient values with the best available research It aims to increase the use of high quality clinical research in clinical decision making Acta Informatica Medica’2008 3
  4. 4. Evidence-Based Medicine One of the greatest achievements of EBM has been the development of systematic reviews and meta-analyses, methods by which researchers identify multiple studies on a topic, separate the best ones and then critically analyze them to come up with a summary of the best available evidence. Acta Informatica Medica’2008 4
  5. 5. Evidence-Based Practice 5 EB Psychology EB Psychiatry EBM 
 EB Education
 extends
  6. 6. Evidence-Based Practice 6 EB Psychology EB Psychiatry EBM 
 EB Education
 extends EB Software Eng.
  7. 7. Evidence-Based Software Engineering Evidence-Based Software Engineering (EBSE) is a way to integrate the best research evidence with practice. Barbara Kitchenham Keele University ICSE’2004 7
  8. 8. 1. Convert the need for information into answerable questions 2. Identify the best evidence with which to answer these questions 3. Appraise the evidence critically: assess its validity and usefulness 4. Implement the results of this appraisal in software engineering practice 5. Evaluate the performance of this implementation The 5 Steps of Evidence-Based Practice }Systematic Literature Reviews (SRs) 8
  9. 9. How to find the best evidence? Software Engineering conferences Software Engineering journals 9
  10. 10. How to find the best evidence? X, Y, and Z: A Systematic Literature Review 10
  11. 11. It involves reading the right papers and then changing behavior in the practice of the discipline. Trisha Greenhalgh Oxford University Evidence-Based practice is not only about reading and summarizing papers 11
  12. 12. Fábio Queda CIn @ UFPE There is a lack of connection between systematic reviews and software engineering practice IST’2011 12
  13. 13. SRs do not provide guidelines for practitioners Most of SR authors affirmed that they hadn’t a direct impact on industrial practice Lack of connection with industry is the 6th top barrier 13
  14. 14. In addition 14
  15. 15. Evidence-Based Medicine (EBM) researchers argue that SRs: Focus on a narrow question 15 Are time consuming they usually require between 6 months and 2 years to complete (1,139 hours, on average) Policymakers often require access to contextualized resources that address a broader scope of scientific evidence quickly On the other hand…
  16. 16. 16 Rapid Reviews Briefings and Summaries 56% of the rapid reviews were conducted in the last 3 years “Rapid reviews simplify the process of SRs to produce information in a timely manner” “[Briefings] translates existing SRs into actionable messages in the form of short accessible briefings”
  17. 17. 17 EBM Researchers Health decision-makers
  18. 18. 18 EBSE Researchers Software Industry
  19. 19. How to transfer knowledge from systematic reviews to software engineering practice? 19
  20. 20. 20
  21. 21. Evidence Briefings: Towards a Medium to Transfer Knowledge from Systematic Reviews to Practitioners @brunocartaxo @gustavopinto @scbs@soueltonvieira moving to 21
  22. 22. “Evidence Briefings” 22 An one-page document, extracted from a systematic review, that contains findings useful for practitioners
  23. 23. Approach 23 1 Systematic Reviews Selection 2 Systematic Review Data Extraction 3 Evidence Briefings Generation 4 Evidence Briefings Evaluation
  24. 24. Approach 24 1 Systematic Reviews Selection 2 Systematic Review Data Extraction 3 Evidence Briefings Generation 4 Evidence Briefings Evaluation
  25. 25. 25 1 Systematic Reviews Selection 120 Systematic Reviews
  26. 26. 26 1 Systematic Reviews Selection 120 Systematic Reviews Fábio Queda CIn @ UFPE YAY! citation++
  27. 27. 27 1 Systematic Reviews Selection 120 Systematic Reviews 32 Systematic Reviews with guidelines Fábio Queda CIn @ UFPE YAY! citation++
  28. 28. 28 1 Systematic Reviews Selection 120 Systematic Reviews 32 Systematic Reviews with guidelines 24 Systematic Reviews with search strings Fábio Queda CIn @ UFPE YAY! citation++
  29. 29. 29 1 Systematic Reviews Selection 120 Systematic Reviews 32 Systematic Reviews with guidelines 24 Systematic Reviews with search strings 12 Selected SRs Fábio Queda CIn @ UFPE YAY! citation++ # Subject 2 Global software development 3 Agile software development 1 Software testing 1 Software requirements 1 Model based software development 1 Software development productivity 1 Cost and effort estimation 1 Code duplication 1 Software engineering knowledge management
  30. 30. Approach 30 1 Systematic Reviews Selection 2 Systematic Review Data Extraction 3 Evidence Briefings Generation 4 Evidence Briefings Evaluation
  31. 31. 31 2 Systematic Reviews Data Extraction Paper title: Original The effectiveness of pair programing: A meta-analysis Briefing The effectiveness of pair programming
  32. 32. 32 2 Systematic Reviews Data Extraction Paper title: Research goals: Original The effectiveness of pair programing: A meta-analysis Briefing The effectiveness of pair programming Template This briefing reports evidence on <GOAL> based on scientific evidence from a systematic review. Briefing This briefing reports evidence on the effectiveness of pair programming around quality duration and effort based on scientific evidence from a systematic review.
  33. 33. 33 2 Systematic Reviews Data Extraction Paper title: Research goals: Research findings: Original The effectiveness of pair programing: A meta-analysis Briefing The effectiveness of pair programming Template This briefing reports evidence on <GOAL> based on scientific evidence from a systematic review. Briefing This briefing reports evidence on the effectiveness of pair programming around quality duration and effort based on scientific evidence from a systematic review. Finding 1 Pairing up of individuals seems to elevate the junior pairs up to near senior pair performance Finding 2 If you do not know the seniority or skill levels of your programmers, but do have a feeling for task complexity, then employ pair programing either when task complexity is low and time is of the essence, or when task complexity is high and correctness if important
  34. 34. Approach 34 1 Systematic Reviews Selection 2 Systematic Review Data Extraction 3 Evidence Briefings Generation 4 Evidence Briefings Evaluation
  35. 35. 35 3 Evidence Briefings Generation: Principles Similarity: elements that are similar are more likely to be organized together Proximity: closer elements are more likely to be perceived as a group Continuation: elements will be grouped as a whole if they are co- linear Unity: elements that have a visual connection should belong to a uniform group
  36. 36. 36 3 Evidence Briefings Generation 1. The title of the briefing 2. The goal of the briefing 3. The findings extracted from the original review 4. An informative box with general information 5. The reference to the original review 6. The logos of our research group and university
  37. 37. 37 3 Evidence Briefings Generation 1. The title of the briefing 2. The goal of the briefing 3. The findings extracted from the original review 4. An informative box with general information 5. The reference to the original review 6. The logos of our research group and university The template of Evidence Briefings is licensed under CC-BY license!
  38. 38. Approach 38 1 Systematic Reviews Selection 2 Systematic Review Data Extraction 3 Evidence Briefings Generation 4 Evidence Briefings Evaluation
  39. 39. RQ: How practitioners and researchers perceive the content and format of Evidence Briefings? 39 4 Evidence Briefings Evaluation
  40. 40. 40 4 Evidence Briefings Evaluation: Practitioners 473 StackExchange Users who asked questions related to the SRs
  41. 41. 41 4 1 SRs Search Strings Queried a StackExchange website 2 Removed false-positive questions 3 Classified questions using open card sort 4 programmers.stackexchange.com sqa.stackexchange.com pm.stackexchange.com
 
 reverseengineering.stackexchange.com 
 softwarerecs.stackexchange.com 
 1,738 related questions 473 related questions Kappa: 0.72 Evidence Briefings Evaluation: Practitioners
  42. 42. 42 4 473 StackExchange Users who asked questions related to the SRs Evidence Briefings Evaluation: Practitioners
  43. 43. 43 4 473 StackExchange Users who asked questions related to the SRs only 146 of them had public profile (LinkedIn, Github, Twitter, etc) Evidence Briefings Evaluation: Practitioners
  44. 44. 44 4 The 22 authors of the 12 Systematic Reviews Evidence Briefings Evaluation: Researchers
  45. 45. 45 4 Evidence Briefings Evaluation: Survey Survey principles: Reciprocity (e.g., we raffled a 100 USD Amazon card gift) Authority & Credibility (e.g, Ph.D., University professors) Liking (e.g., personalized emails) Scarcity: (e.g., we defined a deadline) Brevity (e.g., we asked closed questions as much as possible) Social Benefit (e.g., 1 USD for the Brazilian Red Cross)
  46. 46. Evidence Briefings Evaluation: Survey 46 4 Survey principles: Reciprocity (e.g., we raffled a 100 USD Amazon card gift) Authority & Credibility (e.g, Ph.D., University professors) Liking (e.g., personalized emails) Scarcity: (e.g., we defined a deadline) Brevity (e.g., we asked closed questions as much as possible) Social Benefit (e.g., 1 USD for the Brazilian Red Cross)
  47. 47. 47 4 Survey with Practitioners (17 questions, 2 open-ended) Survey with Researchers (8 questions, 2 open-ended) Evidence Briefings Evaluation: Survey 7 respondents (31% of response rate) 32 respondents (22% of response rate)
  48. 48. Results 48 Survey with Practitioners Survey with Researchers
  49. 49. Results 49 Survey with Practitioners Survey with Researchers
  50. 50. 50 Survey with Practitioners: Demographics Q1. What is your current position? Q2. How many years of experience do you have in your current position?
  51. 51. 51 Survey with Practitioners: Demographics Q1. What is your current position? Q2. How many years of experience do you have in your current position?
  52. 52. 52 Survey with Practitioners: Demographics Q3. Where do you work? Q4. What is your level of educational attainment?
  53. 53. 53 Survey with Practitioners: Demographics Q3. Where do you work? Q4. What is your level of educational attainment?
  54. 54. 54 Survey with Practitioners: Acquiring Knowledge Q5. How often do you use StackExchange websites? Q6. How often do you read software engineering research papers?
  55. 55. 55 Q7. Have you ever read a systematic review paper? Q8. For what reason you read a systematic review paper? Survey with Practitioners: Acquiring Knowledge
  56. 56. 56 Q10. To what degree do you think the information available in the briefing we sent to you can answer your question on StackExchange? Survey with Practitioners: Briefings’ Content
  57. 57. 57 Q10. To what degree do you think the information available in the briefing we sent to you can answer your question on StackExchange? Q11. Why? Survey with Practitioners: Briefings’ Content “The question is too specific” “The question expected more than one answer” “The question touched a slightly different issue” “The briefing lacks details”
  58. 58. 58 Survey with Practitioners: Briefings’ Content Q12. Regardless the briefing answers or not your question, how important do you think is the research presented on the briefing?
  59. 59. 59 Q12. Regardless the briefing answers or not your question, how important do you think is the research presented on the briefing? Survey with Practitioners: Briefings’ Content Q13. Why? “Agile is not a one size fits all methodology. To make it work you need to see what works for you and your team. [...] Making bold high level statistical statements about Agile software development will only hurt it where as it can shine in truly Agile organizations.”
  60. 60. 60 Q14. How do you compare the answers from the StackExchange community to the findings presented in the briefing? Survey with Practitioners: Briefings’ Content
  61. 61. 61 Survey with Practitioners: Briefings’ Format Q15. How easy was to find the information in the briefing? Q16. Is the briefing interface clear and understandable?
 Q17. Does the briefing look reliable?
  62. 62. 62 Q15. How easy was to find the information in the briefing? Q16. Is the briefing interface clear and understandable?
 Q17. Does the briefing look reliable? Survey with Practitioners: Briefings’ Format
  63. 63. Results 63 Survey with Practitioners Survey with Researchers
  64. 64. 64 Survey with Researchers: Sharing Knowledge Q1. How important for you is to share research results to practitioners? ?
  65. 65. 65 Survey with Researchers: Sharing Knowledge Q1. How important for you is to share research results to practitioners?
  66. 66. 66 Survey with Researchers: Sharing Knowledge Q2. How often do you share research results to practitioners?
  67. 67. 67 Survey with Researchers: Sharing Knowledge Q2. How often do you share research results to practitioners?
  68. 68. 68 Survey with Researchers: Sharing Knowledge Q2. How often do you share research results to practitioners? Q3. How do you do that? “Teaching” “Seminars” “Writing” “Advisory work” “Social Networks”
  69. 69. 69 Survey with Researchers: Briefings’ Content Q4. How does the briefing that we sent to you cover the main findings of your paper?
  70. 70. 70 Survey with Researchers: Briefings’ Content Q4. How does the briefing that we sent to you cover the main findings of your paper? Q5. Why? YAY! :-)
  71. 71. 71 Survey with Researchers: Briefings’ Format Q6. How easy was to find the information in the briefing? Q7. Is the briefing interface clear and understandable?
 Q8. Does the briefing look reliable?
  72. 72. Discussions 72
  73. 73. Revisiting Findings 73 Practitioners rarely use research papers as mediums to acquire knowledge. Software engineering practice still has many beliefs with no evidence basis.
  74. 74. Revisiting Findings 74 Both researchers and practitioners positively evaluated the evidence briefings
  75. 75. Revisiting Findings 75 Both researchers and practitioners positively evaluated the evidence briefings The briefings well covered the main findings of the original systematic reviews
  76. 76. The Yin-Yang of Research and Practice 76 Researchers want to transfer knowledge. But not all of them do so. Practitioners want to be more aware of software engineering research. But few of them do so.
  77. 77. Implications 77
  78. 78. Researchers 78
  79. 79. Researchers 79
  80. 80. Practitioners 80
  81. 81. Tool builders 81
  82. 82. Tool builders 82
  83. 83. Educators 83
  84. 84. Conference and Publicity Chairs 84
  85. 85. One last thing.. 85
  86. 86. 86 Search Strings are your best
  87. 87. 87 Assess your search strings on StackExchange on early stages of a systematic review planning!
  88. 88. 88 Are your search strings well-designed? “quality + model”, “quality + model driven” and “model driven + experience” (software AND ((cost OR effort OR productivity) WITH (factors OR indicators OR drivers OR measure)))
  89. 89. 89 Thanks!
  90. 90. Evidence Briefings: Towards a Medium to Transfer Knowledge from Systematic Reviews to Practitioners @brunocartaxo @gustavopinto @scbs@soueltonvieira moving to 90

Presentation of our ESEM'16 paper

Views

Total views

380

On Slideshare

0

From embeds

0

Number of embeds

3

Actions

Downloads

1

Shares

0

Comments

0

Likes

0

×