1. Postgraduate Course
Getting and making sense of “the
best available” scientific evidence
EBMgt
Helping Managers Make Better
Decisions

2. Postgraduate Course
Evidence is not the same as ‗proof‘ or ‗hard facts‘
Evidence can be
- so strong that no one doubts its correctness, or
- so weak that it is hardly convincing at all
What is evidence?

3. Postgraduate Course
Evidence of effect (do!)
Evidence of no effect (don’t!)
No evidence of effect (research!)
Don’t confuse

4. Postgraduate Course
Assignment
CAT
Critical Appraised Topic

5. Postgraduate Course
CAT: Critical Appraised Topic
A critically appraised topic (or CAT) is a structured, short (2
pages max) summary of evidence on a topic of
interest, usually focused around a practical problem or
question. A CAT is like a “quick and dirty” version of a
systematic review, summarizing the best available research
evidence on a topic. Usually more than one study is included
in a CAT.

6. Postgraduate Course
CAT: structure
1) Question
2) PICOC
3) Background
4) Search strategy
5) Results
6) Conclusion
7) Comments
8) Recommendation
9) References

7. Postgraduate Course
Asking the right questions
Part 2

8. Postgraduate Course
5-step approach
Gathering Best Scientific Evidence is
a 5-step approach
1. Formulate an answerable question (PICOC)
2. Search for the best available evidence
3. Critical appraise the quality of the found
evidence
4. Integrate the evidence with managerial
expertise and organizational concerns and
apply
5. Monitor and evaluate the results

9. Postgraduate Course
Asking the right question?
Does team-building work?
Does leadership development training work?
Does management development improve the
performance of managers?
Does employee participation prevent
resistance to change?
Is 360 degree feedback effective?

10. Postgraduate Course
P = Problem or population
I = Intervention or successfactor
C = Comparison
O = Outcome
C = Context
Answerable question: PICOC

11. Postgraduate Course
Scenario: You are a consultant, your client is an insurance
company, there are plans for a merger, you have heard that the other
company has a different culture, you want to know if this will effect
the outcome
P = Organizations with a different corporate culture
I = Merger
C = Organizations with a similar corporate culture
O = Long term profitability
(C)) = Profit organizations, competitive market)
Answerable question: PICO(C)

12. Postgraduate Course
Searching evidence
The problem with finding evidence:
the abundance of literature

13. Postgraduate Course
Searching evidence
Article 1
Article 2
Article 3
Article 4
Article 5
Article 6
Article 7
Article 8
There are about 1350 articles published
on HRM every year. For an HR
professional to keep up this means
reading 3 to 4 articles every day!
(most of these publications are not valid
or irrelevant)

14. Postgraduate Course
Searching evidence
Evidence-based searching
In a systematic and transparant way
searching for the ―best‖ evidence
Part of EBMgt where decision maker is
not a ‗subject matter expert‘

15. Postgraduate Course
Searching for scientific
evidence
What kind of evidence are we looking for?
1. Studies with a design that best suits the research question
2. Studies with the highest level of evidence

16. Postgraduate Course
Explanation
Which design for which question?

17. Postgraduate Course
Levels of internal validity

18. Postgraduate Course
Searching evidence
Where do we search?

19. Postgraduate Course
Peer reviewed journals

20. Postgraduate Course
Which journals are peer reviewed?

21. Postgraduate Course
Type of Information Source
Current Information
Wall street Journal, Financial
Times, Business week, Financieel
Dagblad
Overview of a subject
General background
Academic Information
Statistical Information
Textbooks and popular books
Encyclopedias, yearbooks & book reviews
ABI/INFORM, Business Source
Premier, Emerald, PsychInfo, Science
Direct
CBS Statline, Eurostat
Theories about a subject Textbooks and encyclopedias
Information sources
Company information
Company Annual
Reports, Datastream, Factiva.com, Amad
eus

22. Postgraduate Course
Searching evidence
How do we search?
Search Strategy

23. Postgraduate Course
Why do we need a search strategy?
Promotes deeper learning about your question
Leads to better yield of quality research.
Saves time in the long run.
Source: Inky Bob, www.flickr.com, Creative Commons, April 2006.
Search strategy

24. Postgraduate Course
Two types of search strategies
Search strategy
Building blocks methodSnowball method

25. Postgraduate Course
Snowball method
Starting from one book or article, you search
for other literature on the same topic.
 Snowballing to older publications by finding out which
publications were used by the author (see bibliography of
book or article).
 Snowballing to more recent publications by finding out
how often that book or article has been
cited by other authors (see Web of
Knowledge or Google Scholar).

26. Postgraduate Course
Synonyms or
related terms
….
….
….
….
Synonyms or
related terms
….
….
….
….
Synonyms or
related terms
….
….
….
….
Building blocks method
Synonyms or
related terms
….
….
….
….
Keyword 1 Keyword 2 Keyword 3 Keyword 4
AND AND AND
OR OR OR

27. Postgraduate Course
P = back office employees
I = merger, integration, back office
C = status quo
O = economy of scale
C = healthcare, different organizational culture, unequal
Answerable question: PICOC
1. Underline the keywords
2. Number the order of importance from 1-4

28. Postgraduate Course
P = back office employees
I = 1. merger, 3. integration, back office
C = status quo
O = 4. economy of scale
C = 5. healthcare, different 2. organizational culture, unequal
Answerable question: PICOC
1. Underline the keywords
2. Number the order of importance

29. Postgraduate Course
Search terms
Operationalise your Pico elements!
O = long term profitability?

30. Postgraduate Course
corporate culture: organizational behavior/character, corporate identity
merger: acquisition, take-over, fusion, combination, unification
profitability: profit, advantage, return on investment, shareholder value
The keywords of your PICOC may be enough.
If not, select more words by using:
Select keywords
 synonyms
 alternate spelling, translations
 related terms / words / subjects
 narrower or broader terms

31. Postgraduate Course
Search Query
1. Search with #1 PICOC term (incl. alternative terms,
synonyms, alternate spellings, truncations, etc.) in the
thesaurus, title or abstract
2. Combine the results with OR (use the history function!)
3. Search with #2 PICOC term (incl. synonyms, etc.)
4. Combine the results with OR
5. Combine the results of step 2 and 4 with AND

32. Postgraduate Course
• Merger
• Fusion
• Combination
• Take over
• Acquisition
• Unification
• …
1. Merger 3. Integration
• Healthcare
organization
• Non profit
• Not for profit
4. Health care
organization
AND
Search Query: an example
I I O
OR OR
• Integration •Corporate culture
•Organizational
behavior
•Organizational
character
•Corporate
identity
•Core beliefs
•Shared values
2. Corporate
culture
C
AND
OR

33. Postgraduate Course
Use the history function to combine results
Search Query

34. Postgraduate Course
Search Query

35. Postgraduate Course
Boolean operators
 AND = both terms (apples AND oranges)
 OR = either one of these terms (apples OR oranges)
 NOT = without this term (fruit NOT oranges)
 NEAR = near this term (apples NEAR oranges)
 * = replaces 0,1 or more characters (apple*=
apple, apples, applejack, applejuice, applepie, etc.)
 ?= replaces 1 character (organi?ation=
organisation, organization)

36. Postgraduate Course
orangesapples
apples AND oranges apples OR oranges
orangesapples
Boolean operators

37. Postgraduate Course
Justify your search strategy
Why?
To help the reader of your paper:
 Follow the steps of your search process
 Understand the end results
How?
 Including keywords used for the search actions
 Justify information sources used (literature list)
Search Strategy

38. Postgraduate Course
Include literature references
Why?
 To give other authors the credit they are due.
 To show that you have made use of reliable sources
 To show the relationship between your work and that of others.
 To show that you have studied the subject in depth
 To make it possible to check your work.
 To avoid committing plagiarism !!!
How?
 Cite & include references to acknowledge all your sources carefully.
 Include sufficient own / new ideas in your work.
 You can make use of Reference Manager or Endnote
Search Strategy

39. Postgraduate Course
Start up
select ‗advanced‘ select ‗peer-reviewed‘ select ‗ABI/INFORM Global‘

40. Postgraduate Course
ABI / Inform

41. Postgraduate Course
Learning through play !
Try all buttons
Make lots of mistakes
Have fun !
Go do it & report back next week.

42. Postgraduate Course
Levels of evidence = A hierarchical order for
research designs based on their internal validity
Internal validity = Degree the results may be
unbiased. Higher when conditions demonstrating
causality are present (1. control over ―cause‖, 2.
temporal order, and 3. control over or no
plausible alternative explanation for findings).
Careful design of primary studies promotes these three conditions but
seldom eliminates them. Threats to internal validity are overcome when
accumulated studies with different designs yield comparable findings.
Levels of Scientific Evidence

43. Postgraduate Course
Levels of internal validity

44. Postgraduate Course
Levels of internal validity
It is shown that …
It is likely that …
Experts are of the
opinion that …
There are signs
that …

45. Postgraduate Course
But … sometimes observational
studies are as good as RCT‘s
Internal validity
When the size of effect is very large (swamps
the bias)

46. Postgraduate Course
Generalizability
 Degree findings hold across
populations, settings, procedures etc. (external
validity).
 Reasons for rejecting generalizability must be
logical and evidence-based (not mere dislike of
findings)
 Logical threats to generalizability include:
 Person/Treatment interactions: e.g., incentives based
on dice throw that work for gamblers and not
Baptists
 File draw problem: Studies only published if show
significant effects (why unpublished sources matter)

47. Postgraduate Course
These treatments have not been tested in RCTs:
are they supported by poor evidence?
Internal validity
Heimlich manoeuvre Dehydration: drinking water Cardiac arrest: AED

48. Postgraduate Course
Better than a single study:
a replication study
Better than a replication study:
a systematic review / meta analysis
If there were 100 studies, 99 of which gave a ‘negative’
result (where, say, the new intervention appeared to be
not effective), while one had a ‘positive’ result (were the
intervention appeared effective), it would obviously be a
mistake to consider only the single positive study.
But ….

49. Postgraduate Course
Research designs
Systematic review or meta-analysis
Randomized controlled study (experiment)
Non-randomized controlled study (quasi-experiment)
Observational research: cohort-, panel-, case-control and cross-
sectional study
Before-after study (pretest – posttest design)
Qualitative research

50. Postgraduate Course
Systematic review
The intention behind a systematic review is to identify as fully
as possible all the scientific studies of relevance to a
particular subject and to assess the validity and authority of
the evidence of each study separately. As the name
indicates, a systematic review takes a systematic approach to
identifying studies and has the methodological quality
critically appraised by multiple researchers independently of
each other, as a consequence of which the review is
transparent and reproducible and can be monitored. The use
of statistical analysis techniques in a systematic review to
pool the results of the individual studies numerically in order
to achieve a more accurate estimate of the effect is termed a
―meta-analysis‖.

51. Postgraduate Course

52. Postgraduate Course
Founded in 1993 and named after the British epidemiologist Archy Cochrane
International non-profit and independen organization
Mission: to enable people to make well-informed decisions abouth healthcare
Dedicated to making up-to-date, accurate information about the effects of
healthcare readily available worldwide.
Main product: The Cochrane Database of Systematic Reviews
 1995: 36 reviews
 1999: 500 reviews
 2001: 1000 reviews
 2004: 2000 reviews + 1400 published protocols
Reviews prepared by healthcare professionals who volunteer
(10.000 people worldwide)
Application of rigorous quality standards
Cochrane Collaboration

53. Postgraduate Course
Controlled study
In a controlled study two or more groups are
compared with each other, usually comprising
one group in which an intervention is carried out
(experimental group) and one group where no
or an alternative intervention is conducted
(control group).

54. Postgraduate Course
In the case of randomization, the groups compared with each
other are selected entirely randomly, for example by drawing
lots. This means that each participant (or other unit such as
a team, department or company) has an equal chance of
being in the intervention or control group. In this way, the
influence of any distorting factors is spread over both groups
so that these groups are as comparable as possible with
each other with the exception of the intervention.
Randomization
Controlled study

55. Postgraduate Course
Observational research
Cohort/panel study—entities followed over time
(classic longitudinal study of AT&T managers)
Case-control study—comparisons between entities
with different outcomes (Collins‘ ―Good to Great‖)
Cross-sectional study—one-time assess‘t (turnover
rates of high performing employees in 2012
Observational research refers to studies where the researcher merely
observes but does not intervene, with the intention of finding correlations
among the observed data (synonym: naturalistic study, non-intervention trial)

56. Postgraduate Course
Case-control study
Longitudinal study in which one group of people or companies with a
particular outcome (for example, above-average performance) is compared
subsequently (= retrospective) with a group that does not have this outcome.

57. Postgraduate Course
Cross-sectional study
Study in which data of a statistically significant sample of a population
(managers, CEO‘s, employees) is gathered at one point in time. It provides
a snapshot of the current condition but does not explain cause and effect.
Cross-sectional studies include
surveys

58. Postgraduate Course
Assessing the quality
of a study

59. Postgraduate Course
Intermezzo
How to read a research article?

60. Postgraduate Course
1. Title
2. Abstract
3. Introduction
4. Background / review of literature
5. Organizational context
6. Methodology
7. Results
8. Discussion
Structure of an article

61. Postgraduate Course
In general
 Don‘t let yourself be taken in by scientific jargon and
complex use of language!! Good articles are written in
plain English.
 Even authorative journals with a high impact factor
contain bad articles and vice versa.
 Focus on research question, study design and outcome.
 Don‘t worry about statistics!
 Be critical!! Always ask yourself: does this make sense?

62. Postgraduate Course
Methodological pitfalls
Randomization errors
Bias
Confounding
Reverse Causation

63. Postgraduate Course
Bias: distortion of the outcome due to
systematic errors caused by the way the
study is designed or conducted.
NB: If bias is not taken into account then any
conclusions drawn may be wrong!
Bias

64. Postgraduate Course
1. Selection bias
2. Information (detection) bias
3. Performance bias
4. Exclusion (attrition) bias
5. Publication bias
…
…
30. …..
Forms of bias

65. Postgraduate Course
Selection bias
Error in the way participants in a study were selected. Means comparison
groups differ in measured or unmeasured baseline characteristics.
Types of selection bias:
Sampling bias (selecting only successful departments or individuals who
have committed crimes)
Participation bias (self-selection, non-response, etc.)

66. Postgraduate Course
Distortion of the outcome due to misinterpretation of information
or systematic errors in the the measurement of research
variables which leads to misclassification.
Information bias can be prevented by the use of standardized
measurement instruments, hard outcome measures, validated
questionnaires and objective, independent and blinded
assessors.
Types of information bias:
 Reporting bias (recall bias)
 Observer bias (interviewer bias, halo-effect)
Information bias

67. Postgraduate Course
Confounding
Confounding is the idea that a 3rd variable can distort or confuse (or
confound..) a relationship between two other variables.
Let‘s say that a college education is strongly positively correlated to
successful completion of firefighter training. Is it true that people
with less than four years of college don‘t make good fire fighters?
Or cannot fulfill the requirements of well-trained firefighter?
Confounding could exist if the reading materials use in firefighter
training are written at a much higher level than the job actually
requires.

68. Postgraduate Course
Confounding

69. Postgraduate Course
Reverse causation

70. Postgraduate Course
Critical appraisal of studies
Appendix: Appraisal Questions
Making sense of evidence

71. Postgraduate Course
Effect Size
 Strength or meaningfulness of relationship
between two variables (cause/effect)…several
metrics exist:
 Practical value (average $ saved, weight lost, gain in
test scores)
 Effect strength (standardized indicate of d difference
between treatments or r strength of relationship
across multiple studies)
 Judgment required: Small effects with low cost can
be of practical value (e.g. can be relatively easy to
create identification or ingroup/outgroup effects)

72. Postgraduate Course
Standard appraisal questions
1. Did the study address a clearly focused issue?
2. Is the sample size justified?
3. Is the design appropriate to the stated aims?
4. Are the measurements likely to be valid and reliable?
5. Are the statistical methods described?
6. Did untoward events occur during the study?
7. Were the basic data adequately described?
8. Do the numbers add up?
9. Was the statistical significance assessed?
10. What do the findings mean?
11. Are important effects overlooked?
12. What implications does the study have for your practice?

73. Postgraduate Course
Appraisal of a controlled study
1. Did the study address a clearly focused issue?
2. Were subjects randomly allocated to the experimental and control
group? If not, could this have introduced bias?
3. Are objective inclusion / exclusion criteria used?
4. Were groups comparable at the start of the study?
5. Are objective and validated measurement methods used and were
they similar in the different groups? (misclassification bias)
6. Were outcomes assessed blind? If not, could this have introduced
bias?
7. Is the size of effect practically relevant?
8. Are the conclusions applicable?

74. Postgraduate Course
Appraisal of a cohort / panel study
1. Did the study address a clearly focused issue?
2. Was the cohort / panel recruited in an acceptable way? (selection
bias)
3. Was the cohort/ panel representative of a defined population?
4. Was a control group used? Should one have been used?
5. Are objective and validated measurement methods used and were
they similar in the different groups? (misclassification bias)
6. Was the follow up of cases/subjects long enough?
7. Could there be confounding?
8. Is the size of effect practically relevant?
9. Are the conclusions applicable?

75. Postgraduate Course
1. Did the study address a clearly focused issue?
2. Were the cases and controls defined precisely?
3. Was the selection of cases and controls based on external, objective
and validated criteria? (selection bias)
4. Are objective and validated measurement methods used and were
they similar in cases and controls? (misclassification bias)
5. Did the study incorporate blinding where feasible? (halo-effect)
6. Was there data-dredging?
7. Could there be confounding?
8. Is the size of effect practically relevant?
9. Are the conclusions applicable?
Appraisal of a case-control study

76. Postgraduate Course
Assessment of a survey
1. Did the study address a clearly focused issue?
2. Was the sample size justified?
3. Could the way the sample was obtained introduce (selection)bias?
4. Is the sample representative and reliable?
5. Are the measurements (questionnaires) likely to be valid and
reliable?
6. Was the statistical significance assessed?
7. Are important effects overlooked?
8. Can the results be generalized?

77. Postgraduate Course
Don‘t confuse: representativeness and reliability
The number of respondents has no direct relationship with
representativeness; even a large random sample can be insufficiently
representative. However, the number of respondents does have an
impact on the reliability of the results.
Assessment of a survey

78. Postgraduate Course
1. Is your organization / division / population so different from
those in the study that its results cannot apply?
2. How relevant is the study to what you are seeking to understand
or decide?
3. What are your organization‘s potential benefits and
harms from the intervention?
4. Is the intervention feasible in your setting?
5. What are your executive‘s (or client‘s) concerns,
preferences and expectations for both the
outcome you are trying to prevent and the
intervention you are offering?
Organization concerns
Organization
Concerns
Ask yourself to what extent the evidence is
applicable in your situation:

79. Postgraduate Course
Turning evidence into practice
Evidence based management:
closing the gap between research and practice