CEBMa center forEvidence-Based Management CEBMa G.docx

CEBMa
center for
Evidence-Based Management
CEBMa Guideline for
Critically
Appraised Topics
in Management and Organizations
Version 1.1
Editors: Eric Barends, Denise M. Rousseau, Rob B. Briner
CEBMa
center for
Guideline for Critically Appraised Topics in Management and
Organizations
Contents
Critically Appraised Topics 3
Steps in the CAT process 4

Step 1. Background: What is the context of the CAT question 5
Step 2. Formulating the CAT question: What does the CAT
answer? 6
Step 3. Defining inclusion criteria: Which studies will be taken
into account? 9
Step 4. Search strategy: How should the studies be sought? 10
Step 5. Study selection: How should you select the studies? 12
Step 6. Data extraction: What information should be extracted?
13
Step 7. Critical appraisal: How should the quality of the studies
be judged? 15
Step 8. Results: What did you find? 18
Step 9. Conclusion 22
Step 10. Limitations 24
Step 11. Implications and recommendations 25
Checklist 28
References 29
Appendix: Summarizing scientific literature 30
© CEBMa, 2017
Pubished by the Center for Evidence-Based Management,

Amsterdam, The Netherlands
No rights reserved. All parts of this publication may be
reproduced in any form or by any
means without the prior permission of CEBMa. First published
2017
ISBN: 978-90-826957-1-7
For further information on CEBMa, visit our website:
www.cebma.org
http://www.cebma.org
CEBMa
center for
Organizations
Critically Appraised Topics
A Critically Appraised Topic (CAT) provides a quick and
succinct assessment of what is known (and
not known) in the scientific literature about an intervention or
practical issue by using a systematic
methodology to search and critically appraise primary studies.
However, in order to be quick, a CAT
makes concessions in relation to the breadth, depth and
comprehensiveness of the search. Aspects
of the search may be limited to produce a quicker result:

• Searching: a limited number of databases may be consulted,
and unpublished research
can be excluded. Sometimes a CAT may be limited to only
meta-analyses and/or
controlled studies.
• Data Extraction: only a limited amount of key data may be
extracted, such as year,
population, sector, sample size, main findings, and effect size.
• Critical Appraisal: quality appraisal is often limited to
methodological appropriateness.
Due to these limitations, a CAT is more prone to selection bias
than a systematic review or rapid
evidence assessment.
CEBMa
center for
Organizations
Steps in the CAT process
A CAT involves the following steps:
1. Background
2. Question

3. Inclusion Criteria
4. Search Strategy
5. Study Selection
6. Data Extraction
7. Critical Appraisal
8. Results
8.1. Definitions
8.2. Causal Mechanism
8.3. Main Findings
9. Conclusion
10. Limitations
11. Implications and recommendations
6
7
8
9
10
11

5
4
3
2
1
CEBMa
center for
Organizations
Step 1. Background: What is the context of the CAT
question
The background should clearly state what the rationale for the
CAT was and explain why the question
being asked is important. You may also indicate how it might
relate to a wider understanding of a
general problem. Most CATs occur in the context of a specific
organization. You should address
this context (e.g. sector, history, characteristics), help specify
the rationale for the CAT, and explain
why the question is important for the organization, its members
or its clients. In formulating the CAT
question, it is important to reflect on the potential stakeholders
relevant to the general problem being

addressed in order to tap deeper insight into the issues involved
(e.g., internal stakeholders such as
employees at different organizational levels or external
stakeholders like clients or the community).
EXAMPLE 1
As a change consultant, I am expected to contribute to the
realization of organizational
change. The outcomes of change can be both positive and
negative, depending on
the type of change and the specific individual or group affected.
Particularly when the
change has predominantly negative outcomes (e.g., lay-offs), I
think it is of utmost
importance that the change process is fair and just. I am curious
about the impact
procedural justice has on the way people perceive the outcomes
of organizational
change.
EXAMPLE 2
Interviewing and who got what job how are the most talked
about subject on campus.
Most students are getting ready to either get internships or full
time jobs this summer.
It is widely believed that smiling during a job interview may
increase your chances of
getting hired. This CAT was conducted to understand whether
this claim is supported
by scientific evidence.
EXAMPLE 3
Hoping to imitate the innovative and flexible work

environments found at start-ups
and companies like Google, my organization’s Executive
Director is considering
implementing a bullpen style, open-plan layout. Currently, our
office is divided into
individual workspaces with half walls. The Executive Director
and Controller both have
private offices. With 15 employees working in a relatively small
space, I worry that the
distractions created by a new, open layout may undermine our
ability to focus and to
be productive at work. To draw a more informed conclusion on
the effect that such a
layout might have at our office, I have gathered and assessed
the quality of available
scientific evidence, outlined key findings, and summarized their
practical implications.
CEBMa
center for
Organizations
Step 2. Formulating the CAT question: What does the
CAT answer?
You can use a CAT to answer many different types of questions.
For the purposes of this guideline,
we split these into ‘impact’ and ‘non-impact’ questions. This
distinction is not ideal but reflects the

fact that the most common CAT question are about
• the effect an intervention, factor, or independent variable.
• the drivers (antecedents) of a certain outcome.
EXAMPLE 1
What is known in the scientific literature about the impact of
goal setting on the task
performance of sales agents?
EXAMPLE 2
What is known in scientific literature about the impact of
smiling during job interviews:
Do people who smile more have better chances of getting hired?
EXAMPLE 3
flexible work schedules on
task performance?
EXAMPLE 4
What is known in the scientific literature about the effect of
open-office layouts on
workers’ task performance?
CEBMa
center for

Organizations
Non-impact questions
You can, however, use CATs to answer a range of other
questions, which can be grouped as:
• Needs: What do people want or need?
• Attitude: What do people think or feel?
• Experience: What are peoples’ experiences?
• Prevalence: How many / often do people / organizations ...?
• Procedure: How can we implement ...?
• Process: How does it work?
• Exploration: Why does it work?
EXAMPLE
Main question
What is known in the scientific literature about the prevalence
of burnout among
nurses in the US?
Supplementary questions
• What is burnout?

• What are the symptoms of burnout more widely and for nurses
more specifically?
• Are there reliable and valid instruments available to measure
burnout?
PICOC
A PICOC is a mnemonic used to assist reviewers to search for
studies that are relevant to the
professional context described in Step 1 (Background). The
PICOC acronym stands for:
Population Who?
Type of employee, subgroup,
people who may be affected by the
outcome
I ntervention What or How?
Management technique/method,
factor, independent variable
Comparison Compared to what?
Alternative intervention, factor,
variable
Outcome
What are you trying to accomplish /
improve / change?
Purpose, objective, dependent
variable
Context
In what kind of organization /

circumstances?
Type of organization, sector, relevant
contextual factors
CEBMa
center for
Organizations
Each element of the PICOC is vital in focusing the question and
searching effectively for the most
relevant evidence.
EXAMPLE 1
P: physicians
I: 360-degree feedback
C: coaching
O: increased task performance
C: university hospital that has recently undergone significant
organizational
restructuring
EXAMPLE 2

P: software developers
I: agile working
C: business as usual / status quo
O: reduced software development costs
C: large international IT firm in a highly competitive market
In addition, your PICOC will help you to determine whether the
findings of a study will be generalizable
and applicable to your organizational context. More
specifically, your PICOC helps answer the
question of whether your population, outcome of interest, and
organizational characteristics are so
different from those in the study that its results may be difficult
to apply. After all, some psychological
principles are generalizable to all human beings, but sometimes
what works in one narrowly defined
setting might not work in another.
CEBMa
center for
Organizations
Step 3. Defining inclusion criteria: Which studies will be

taken into account?
One of the features that distinguishes a CAT from a traditional
review is the pre-specification of criteria
for including and excluding studies. The inclusion criteria (also
referred to as eligibility criteria) help the
reviewer(s) to determine whether a study will be included in the
CAT when reviewing its abstract and/
or full text. The inclusion criteria should be guided by your
CAT question and objectives, and by the
outcome measures that you will be considering to answer your
question. They define the studies that
the search strategy is attempting to locate.
EXAMPLE
Inclusion criteria
1. Date: published in the period 1980 to 2016 for meta-analyses
and the period 2000
to 2016 for primary studies
2. Language: articles in English
3. Type of studies: quantitative, empirical studies.
4. Study design: only meta-analyses or controlled studies
5. Measurement: a) studies in which the effect of goal setting on
organizational
outcomes was measured or b) studies in which the effect of
moderators and/or
mediators on the outcome of goal setting was measured.
6. Outcome: task performance
7. Context: studies related to workplace settings.

Exclusion criteria
1. Studies including goal setting as part of health-, lifestyle- or
treatment-related
interventions.
CEBMa
center for
Organizations
Step 4. Search strategy: How should the studies be
sought?
Based on the question, you next have to conduct a structured
search for all relevant studies in the
international research literature. In the first instance, you
should concentrate your search on relevant
bibliographical databases using clearly defined search terms. At
the very least, conduct your search
using ABI/INFORM from ProQuest and Business Source
Premier from EBSCO. Depending on the
CAT question, you may also need to search in databases that are
aimed at neighbouring disciplines
such as psychology (PsycINFO), education (ERIC) or healthcare
(PubMed).
EXAMPLE

The following four databases were used to identify studies:
ABI/INFORM Global and
PsycINFO. The following generic search filters were applied to
all databases during the
search:
1. Scholarly journals, peer-reviewed
2. Published in the period 1980 to 2016 for meta-analyses and
the period 2000 to
2016 for primary studies
3. Articles in English
A search was conducted using combinations of different search
terms, such as ‘job
interview’, ‘employment interview’, ‘selection interview’ and
‘smiling’.
CEBMa
center for
Organizations
Documentation of the search
It is important that the search conducted is transparent,
verifiable and reproducible. For this reason,
the search process should be clearly documented, preferably in

the form of a table that shows which
search terms were used, how search terms were combined, and
how many studies were found at
every step. An example is provided below.
ABI/Inform Global, PsycINFO
peer reviewed, scholarly journals, July 2016
Search terms ABI PSY
S1: ti(“job interview*”) OR ab(“job interview*”) 76 231
S2: ti(“employment interview*”) OR ab(“employment
interview*”) 122 368
S3: ti(“selection interview*”) OR ab(“selection interview*”) 70
130
S4: S1 OR S2 OR S3 259 583
S5: ti(smil*) OR ab(smil*) 736 2,673
S6: S4 AND S5 7 5
CEBMa
center for
Organizations

Step 5. Study selection: How should you select the
studies?
In general, a search will yield a large number of studies –
sometimes more than hundred. Some
studies will not be directly relevant to the research question and
PICOC. Hence, the next step is to
screen them to check that they meet the inclusion criteria.
Screening is usually a two-stage process,
the first involves reviewing the abstracts and the second,
reviewing the full studies.
Review abstracts
This involves reading the abstracts that have been found
through the searching. Each abstract should
be compared against the inclusion criteria and if the abstract
meets the criteria then the full study
should be read. Not all abstracts will contain information on all
the inclusion criteria (this is particularly
a problem with electronic searching). In these cases, decisions
need to be made on whether or not to
include the study on the information available. When in doubt,
the study should be included.
Review full studies
You should read the full article and compare it against the
inclusion criteria.
EXAMPLE
Selection took place in two phases. First, the titles and abstracts
of 83 studies
identified were screened for their relevance to this CAT. In case
of doubt, the study
was included. Duplicate publications were removed. This first

phase yielded 2 meta-
analyses and 12 controlled and/or longitudinal studies.
CEBMa
center for
Organizations
Step 6. Data extraction: What information should be
extracted?
Data extraction involves the collation of the results and other
information of the studies included. From
each study, information relevant to the CAT question, such as
year of publication, research design,
sample size, population (e.g., industry, type of employees),
outcome measures, main findings, effect
sizes, weaknesses, and the final level of trustworthiness (see 7)
should be reported, preferably in the
form of a clearly structured table. An example is provided on
the next page.
CEBMa
center for

Organizations
A
u
th
o
r
&
ye
a
r
S
e
c
to
r
/
P
o
p
u
la
ti
o

n
D
e
si
g
n
+
sa
m
p
le
s
iz
e
M
a
in
fi
n
d
in
g
s
E

ff
e
c
t
si
ze
L
im
it
a
ti
o
n
s
L
e
ve
l
A
b
ra
h
am
&
G

ra
h
am
-
R
o
w
e
(2
0
0
9
)
S
ys
te
m
at
ic
re
vi
ew
;
2

/3
R
C
T;
1
/3
q
u
as
i-
ex
p
er
im
en
ta
l
W
o
rk
si
te
;
8

st
u
d
ie
s;
N
=
6
2
4
e
m
p
lo
ye
es
W
o
rk
si
te
p
h

ys
ic
al
a
ct
iv
ity
in
te
rv
en
tio
n
s
w
h
ic
h
in
cl
u
d
e
sp
ec

ifi
c
g
o
al
se
tt
in
g
,
g
o
al
r
ev
ie
w
s
(i.
e.
f
o
llo
w

-u
p
) a
n
d
g
ra
d
ed
t
as
ks
h
av
e
a
sm
al
l,
p
o
si
tiv
e

im
p
ac
t
o
n
fi
tn
es
s-
re
la
te
d
o
u
tc
o
m
es
S
m
al
l

L
im
ite
d
re
le
va
n
ce
t
o
th
e
re
vi
ew
q
u
es
tio
n

A
A
B
an
d
u
ra
&
L
o
ck
e
(2
0
0
3
)
G
en
er
al
p
o
p

u
la
tio
n
Tr
ad
iti
o
n
al
lit
er
at
u
re
r
ev
ie
w
D
is
cu
ss

es
t
h
e
im
p
o
rt
an
ce
o
f
se
lf-
ef
fic
ac
y
fo
r
u
n
d
er
st

an
d
in
g
,
p
re
d
ic
tin
g
an
d
c
h
an
g
in
g
p
eo
p
le
's

p
er
fo
rm
an
ce
o
r
g
o
al
a
tt
ai
n
m
en
t.
S
el
f-
ef
fic
ac

y
is
st
at
ed
t
o
b
e
re
la
te
d
(b
as
ed
o
n
m
et
a-
an
al
yt

ic
al
fi
n
d
in
g
s
fr
o
m
p
re
vi
o
u
s
st
u
d
ie
s)
,
am
o

n
g
o
th
er
s,
t
o
m
o
re
p
ro
ac
tiv
e
(s
el
f-
se
t)
g
o
al
-s

et
tin
g
,
ch
al
le
n
g
in
g
g
o
al
s,
a
n
d
f
as
te
r
g
o
al

a
tt
ai
n
m
en
t,
a
s
w
el
l a
s
ef
fo
rt
a
n
d
p
er
fo
rm

an
ce
.
N
o
e
ffe
ct
s
iz
es
p
ro
vi
d
ed
N
o
s
ys
te
m
at

ic
se
ar
ch
,
n
o
in
fo
rm
at
io
n
re
g
ar
d
in
g
d

es
ig
n
o
f
in
cl
u
d
ed
st
u
d
ie
s
D
B
ro
w
n
,
2
0
0
5

C
an
ad
ia
n
g
o
ve
rn
m
en
t
em
p
lo
ye
es
in
a
t
ra
in

in
g
p
ro
g
ra
m
R
an
d
o
m
iz
ed
co
n
tr
o
lle
d
t
ra

il,
N
=
7
4
,
fie
ld
se
tt
in
g
B
o
th
p
ar
tic
ip
an
ts
w
h
o

w
er
e
u
rg
ed
t
o
d
o
t
h
ei
r
b
es
t
an
d
t
h
o
se
w
h

o
se
t
p
ro
xi
m
al
(s
h
o
rt
er
-t
er
m
) a
s
w
el
l a
s
d
is

ta
l (
=
lo
n
g
er
-t
er
m
) g
o
al
s
h
ad
in
cr
ea
se
d
t
ra

n
sf
er
o
f
tr
ai
n
in
g
(=
m
ai
n
te
n
an
ce
o
f
le
ar
n
ed
m

at
er
ia
l o
ve
r
tim
e
an
d
g
en
er
al
iz
at
io
n
o
f
le
ar
n
ed

m
at
er
ia
l f
ro
m
t
h
e
cl
as
sr
o
o
m
t
o
t
h
e
w
o
rk
p
la

ce
c
o
n
te
xt
) r
el
at
iv
e
to
t
h
o
se
w
h
o
s
et
o
n
ly
d

is
ta
l o
u
tc
o
m
e
g
o
al
s.
T
h
er
e
w
as
n
o
s
ig
n
ifi

ca
n
t
d
iff
er
en
ce
in
t
h
e
tr
an
sf
er
le
ve
l o
f
p
ar
tic
ip
an

ts
u
rg
ed
t
o
d
o
t
h
ei
r
b
es
t
an
d
t
h
o
se
w
h
o

s
et
p
ro
xi
m
al
p
lu
s
d
is
ta
l g
o
al
s.
In
ad
d
iti
o
n
,

th
er
e
w
as
n
o
d
iff
er
en
ce
b
et
w
ee
n
t
h
e
ex
p
er
im

en
ta
l c
o
n
d
iti
o
n
s
re
g
ar
d
in
g
t
h
e
ef
fe
ct
o
n

s
el
f-
ef
fic
ac
y.
T
h
is
s
u
g
g
es
ts
t
h
at
s
u
p
p
o
rt
t

h
e
co
n
cl
u
si
o
n
t
h
at
d
is
ta
l o
u
tc
o
m
e
g
o
al
s

ar
e
n
o
t
ef
fe
ct
iv
e
in
b
rin
g
in
g
a
b
o
u
t
an
in
cr
ea

se
in
t
ra
n
sf
er
w
h
en
p
ar
tic
ip
an
ts
a
re
le
ar
n
in
g
n

ew
s
ki
lls
S
m
al
l
S
h
o
rt
t
im
e
fr
am
e
b
et
w
ee
n

tr
ai
n
in
g
a
n
d
m
ea
su
re
m
en
t
(s
ix
w
ee
ks
)
A
F

u
,
2
0
0
9
In
d
u
st
ria
l s
al
es
p
eo
p
le
B
ef
o
re
a
ft
er

,
w
ith
d
o
u
b
le
p
o
st
-t
es
t
(3
m
o
n
th
s
an
d
6

m
o
n
th
s)
N
=
1
4
3
T
h
e
st
u
d
y
in
d
ic
at
es
f
u
rt

h
er
t
h
at
s
el
f-
se
t
g
o
al
s
fu
lly
m
ed
ia
te
t
h
e
re
la

tio
n
sh
ip
b
et
w
ee
n
a
ss
ig
n
ed
g
o
al
s
an
d
s
el
lin
g
e

ffo
rt
(a
ss
g
o
al
s
im
p
ac
t
ss
g
a
n
d
t
h
en
s
el
lin
g
e

ffo
rt
).
In
a
d
d
iti
o
n
,
th
e
lo
n
g
itu
d
in
al
d
at
a
in
d

ic
at
e
th
at
c
o
m
p
an
y-
as
si
g
n
ed
g
o
al
s,
s
el
f-
se
t

g
o
al
s,
a
n
d
s
el
lin
g
e
ffo
rt
al
l p
o
si
tiv
el
y
in
flu

en
ce
f
u
tu
re
n
ew
p
ro
d
u
ct
s
al
es
,
b
u
t
n
o
t
se
lf-

ef
fic
ac
y
(n
o
t
si
g
n
ifi
ca
n
t)
.
In
te
re
st
in
g
ly
,
th
e
re

su
lts
o
f
th
e
st
u
d
y
fa
il
to
c
o
n
fir
m
a
n
in
ve
rt
ed

,
U
-s
h
ap
ed
r
el
at
io
n
sh
ip
b
et
w
ee
n
a
ss
ig
n
ed
g
o
al

s
an
d
e
ffo
rt
M
o
d
er
at
e
N
o
s
er
io
u
s
lim
ita
tio
n
s

C
S
ch
w
ei
tz
er
et
a
l,
2
0
0
4
U
n
d
er
g
ra
d
u
at
e

st
u
d
ie
s,
U
S
R
C
T,
n
=
1
5
9
R
es
u
lts
o
f
a
la
b
o

ra
to
ry
e
xp
er
im
en
t
u
til
iz
in
g
h
ig
h
,
lo
w
,
in
cr
ea

si
n
g
,
d
ec
re
as
in
g
,
an
d
‘‘
d
o
y
o
u
r
b
es
t’’
g
o

al
s
tr
u
ct
u
re
s
ac
ro
ss
m
u
lti
p
le
r
o
u
n
d
s
p
ro
vi

d
e
ev
id
en
ce
t
h
at
d
ep
le
tio
n
m
ed
ia
te
s
th
e
re
la
tio

n
sh
ip
b
et
w
ee
n
g
o
al
st
ru
ct
u
re
s
an
d
u
n
et
h
ic

al
b
eh
av
io
r,
an
d
t
h
at
t
h
is
e
ffe
ct
is
m
o
d
er
at
ed
b

y
th
e
n
u
m
b
er
o
f
co
n
se
cu
tiv
e
g
o
al
s
as
si
g
n
ed
.

V
er
y
sm
al
l
A
rt
ifi
ci
al
st
u
d
en
ts
A
CEBMa
center for

Organizations
Step 7. Critical appraisal: How should the quality of the
studies be judged?
Methodological appropriateness
You can usually find a study to support or refute almost any
theory or claim. It is thus important that
you determine which studies are trustworthy (i.e. valid and
reliable) and which are not. You should
first determine the trustworthiness of a scientific study by its
methodological appropriateness.
For cause-and-effect claims (i.e. if we do A, will it result in
B?), a study has a high methodological
appropriateness when it fulfills the three conditions required for
causal inference: co-variation, time-
order relationship, and elimination of plausible alternative
causes (Shaughnessy & Zechmeister, 2006).
A study that uses a control group, random assignment and a
before-and-after measurement is
therefore regarded as the ‘gold standard’ for effect studies1.
Non-randomized studies and before-after studies come next in
terms of appropriateness. Cross-
sectional studies (surveys) and case studies are regarded as
having the greatest chance of showing
bias in their results and thus fall lower in the ranking of
appropriateness. Systematic reviews or meta-
analyses based on pooled results of randomized controlled
studies are thus regarded as the most
appropriate design.
To determine the methodological appropriateness of effect
studies and impact evaluations, we

recommend using the following six levels of appropriateness,
which are based on the classification
system of Shadish, Cook and Campbell (2002), and Petticrew
and Roberts (2006).
Design Level
Systematic review or meta-analysis of randomized controlled
studies AA
Systematic review or meta-analysis of non-randomized
controlled and/or before-after
studies A
Randomized controlled study
Systematic review or meta-analysis of controlled studies
without a pretest or
uncontrolled study with a pretest
BNon-randomized controlled before-after study
Interrupted time series
Systematic review or meta-analysis of cross-sectional studies
C
Controlled study without a pretest or uncontrolled study with a
pretest
Cross-sectional study (survey) D
Case studies, case reports, traditional literature reviews,
theoretical papers E
1 It should be noted that randomized controlled studies are

often conducted in an artificial (lab-type) setting – with
students
carrying out prescribed work tasks – which may restrict their
generalizability. Non-randomized studies in a field setting –
with employees carrying out their normal tasks within an
organizational setting – on the other hand, have a lower level of
trustworthiness, but can still be useful for management practice.
CEBMa
center for
Organizations
Note, however, that the level of methodological appropriateness
as explained above is only relevant
in assessing the validity of a study that examines a cause-and-
effect relationship that might exist
between an intervention (e.g. performance appraisal) and its
outcomes (e.g performance). When
the CAT question concerns a non-effect or non-impact question,
for example the prevalence of
phenomenon (‘How many / often do people / organizations
...?’), a cross-sectional study may be the
most appropriate design (Petticrew and Roberts, 2003).
EXAMPLE 1
The overall quality of the studies included was high. Of the 4
meta-analyses, 3 included

randomized and/or non-randomized controlled studies and were
therefore qualified
as level A or AA. The remaining meta-analysis was graded as
level C, because it was
insufficiently clear what type of studies were included. The
actual level of evidence of
this meta-analysis (and as result the overall quality of the
studies included in this CAT)
may therefore be higher. All 3 primary studies used a cross-
sectional design and were
therefore graded level D.
EXAMPLE 2
After critical appraisal of the 24 studies, only four studies were
included. Most studies
were excluded because they had serious methodological
shortcomings. One of the
studies included concerned a systematic review, representing
the results of 18 studies.
The overall quality of the included studies, however, was low.
For instance, all but two
of the studies included in the systematic review were self-report
surveys, and due to
heterogeneity between studies it was not possible to calculate a
pooled estimate of
effect. The three single primary studies used a cross-sectional
design. As a result, the
trustworthiness of the scientific evidence supporting the
following main findings is very
limited.
Effect Sizes
An outcome can be statistically significant, but it may not
necessarily be practically relevant. Note

that even a trivial effect can be statistically significant if the
sample size is large. This works the other
way around as well: even a large, practical relevant effect can
be statistically non-significant if the
sample size is small. Also, keep in mind that p-values do NOT
measure the probability that the studied
hypothesis is true, or the probability that the data were
produced by random chance alone (Ziliak,
2016). Nor can a p-value tell you the size of an effect, the
strength of the evidence or the importance
of a result.
For this reason, you should pay little attention to the p-value
but instead assess the ‘effect size’ – a
standard measure of the magnitude of the effect – of the studies
included when addressing impact
CEBMa
center for
Organizations
questions. To determine the magnitude of an effect, apply
Cohen’s rules of thumb (Cohen, 1988; see
below). According to Cohen a ‘small’ effect is an effect that is
only visible through careful examination.
A ‘medium’ effect, however, is one that is ‘visible to the naked
eye of the careful observer’. Finally, a
‘large’ effect is one that anybody can easily see because it is

substantial.
Effect size Small Medium Large
Standardized mean difference: d, ∆, g ≤ .20 .50 ≥ .80
Correlation: r, ρ ≤ .10 .30 ≥ .50
Correlation: r2 ≤ .01 .09 ≥ .25
ANOVA: η2, ω2 ≤ .01 .06 ≥ .14
Chi-square: ω2 ≤ .10 .30 ≥ .50
Simple regression: β ≤ .10 .30 ≥ .50
Multiple regression: β ≤ .20 .50 ≥ .80
Multiple regression: R2 ≤ .02 .13 ≥ .26
Note, however, that Cohen’s rules of thumb were meant to be
exactly that - ‘rules of thumb’ - and
are for many reasons arbitrary (Cooper & Lindsay, 1998). For
example, a standard mean difference
of .20 may be regarded as ‘small’ when the outcome concerns
job satisfaction but ‘large’ when the
outcome concerns fatal medical errors. When assessing impact,
it is therefore important to relate the
effect size directly to the outcome that was measured.
CEBMa
center for

Organizations
Step 8. Results: What did you find?
Step 8.1. Definition: What is meant by X?
Most CAT questions include one or more key elements /
constructs, for which several definitions are
available. In this step, you should provide an overview of the
most common definition(s).
EXAMPLE 1
An incentive is commonly defined as “something that arouses
action or activity” (http://
www.merriam-webster.com). In the domain of management,
incentives can be defined
as “…plans that have predetermined criteria and standards, as
well as understood
policies for determining and allocating rewards” (Doe, 2011, p.
219). Incentives include
all forms of rewards (and punishments) that are based on an
employee’s performance
or behavior. Promotions, grades, awards, praise, and recognition
are therefore all
incentives. However, financial incentives such as money, bonus
plans, or stock options
are the most commonly used (Doe, 2014). Formally, incentives
differ from rewards.
Incentives refer to all stimuli that are provided in advance,
whereas rewards are offered
after a given performance (Doe, 2014). In the scientific
literature and management

practice, however, these terms are used interchangeably.
EXAMPLE 2
A smile is defined as a pleased, kind, or amused expression,
typically with the corners
of the mouth turned up and the front teeth exposed. A neutral
expression is a blank
facial expression characterized by neutral positioning of the
facial features, implying a
lack of strong emotion.
EXAMPLE 3
The concept of self-managing teams is referred to in various
ways, using terms such
as ‘autonomous groups’, ‘shared’ or ‘self-directed teams’; all of
these terms refer
to teams that are hallmarked by autonomy. We use the term
‘self-managing teams’
to cover all of the different descriptions of this concept. Doe et
al (2012) refer to the
standard definition of autonomous groups as “groups
responsible for a complete
product or service, or a major part of a production process.
They control members’
task behavior and make decisions about task assignment and
work methods” (Doe,
2005: p. 341).
http://www.merriam-webster.com
http://www.merriam-webster.com
CEBMa
center for

Organizations
Step 8.2. Causal mechanism: How is X assumed to have an
effect
on Y?
A causal mechanism spells out the process by which an
intervention, construct or phenomenon is
expected to have an effect on a certain outcome. The causal
mechanism is often based on one or
more (social or psychological) theories that explain why the
assumed effect occurs and under which
circumstances.
EXAMPLE 1
As Doe et al. describe, the presumed causal mechanism behind
open-plan offices
works as follows:
• Breaking down physical office barriers increases one’s
exposure to teammates
• Increased exposure facilitates communication among
teammates
• Facilitated communication increases collaboration
• Increased collaboration increases overall productivity and
performance

However, one can also presume that increased exposure to
teammates might also
cause increases in visual and auditory distractions,
counteracting positive benefits to
some degree.
EXAMPLE 2
The construct of perceived supervisory support stems from the
norm of reciprocity,
which states that people treat others as they would like to be
treated, repaying
kindness with kindness and retaliating against those who inflict
harm (Doe et al.,
2013; Doe, 1960). Put differently, when a manager helps his or
her employees in
times of need or recognizes their extra effort, the employees
will feel inclined to act to
benefit that manager (e.g., by meeting goals and objectives) and
thus the organization
as a whole (Doe, 2013; Doe et al, 1986). Not surprisingly,
physicians experiencing
inadequate supervisory support tend to describe their
commitment to the hospital and
its patients in negative terms, which in turn negatively affects
their performance (Doe et
al, 2013).
CEBMa
center for

Organizations
Step 8.3. Main findings
In this section, you should provide an overview of the main
findings relevant to the CAT question. For
each finding, you should present the main evidence from the
CAT, including its level of trustworthiness
and (if available) effect size.
EXAMPLE 1
Smiling is weakly correlated with job interview success (level
D)
Putting on a smile may be advantageous by comparison with
remaining neutral, which
may be seen as reflecting a lack of interest or involvement. But
there are other factors
at play too during the interview and just smiling is weakly
correlated to success (Doe et
al, 2009).
EXAMPLE 2
Overall, financial incentives have a moderate positive effect on
performance (level A)
There is strong evidence that financial incentives tend to have a
moderate positive
effect on performance (Doe, et al., 2014; Doe & Doe, 2014;
Doe, et al., 2010). This
positive effect is often referred to as the ‘price effect’: the

financial incentive increases
the intention to perform well because of the monetary benefit.
However, this effect
differs between forms of incentives, types of motivation, and
performance outcomes,
as described in the following findings. Moreover, when
financial incentives are high they
can hamper performance by interfering with learning and
promoting inappropriate risk
taking (Doe et al., 2009).
EXAMPLE 3
When employees must first acquire requisite knowledge or
skills to perform the task,
specific and challenging goals can have a large negative effect
on performance
(level A)
Several randomized controlled studies have demonstrated that
when a task requires
the acquisition of knowledge before it can be performed
effectively, a general goal
(e.g. ‘do your best’) leads to higher performance than a specific
high goal (Doe & Doe,
1989; Doe, 1995). In fact, when knowledge acquisition is
necessary for effectively
performing a task, setting a specific but extremely high
performance goal can lead
people to ruminate on the potential negative consequences of
failure rather than focus
on task-relevant ways to attain the goal (Doe et al, 2002).
CEBMa

center for
Organizations
EXAMPLE 4
Arousing fear leads employees to engage in short-term thinking
(level B)
Fear can be aroused in people with a specific goal in mind; to
take preventive
measures, or to stimulate proactive behavior. However, there is
strong evidence
indicating that fear (e.g. job insecurity) tends to lead people to
engage in short-term
thinking, narrowing their attention to immediate consequences
(Doe, 1999).
EXAMPLE 5
The outcome of the managers’ own performance appraisal has a
large effect on how
they evaluate their employees (level A)
A combination of studies (including a randomized controlled
study) demonstrated that
managers who receive positive feedback about their
performance subsequently rate
their employee higher than managers who receive negative
feedback regarding their
own performance (Doe, 2008). Surprisingly, this effect even

occurred when managers
knew their own evaluation was bogus.
EXAMPLE 6
Managers’ power level has a large to moderate effect on how
they rate both others
and themselves (level A)
A meta-analysis of 46 studies indicates that as a manager’s
power level grows, his/her
evaluation of others becomes increasingly negative, whereas
evaluations of him/herself
become ever more positive (Doe & Doe, 1998). This finding
suggests that performance
evaluations by supervisors should be considered in light of their
hierarchical position
and power level.
EXAMPLE 7
Managing the customer experience may have a moderate
positive effect on a
company’s financial performance (level D)
Findings from cross-sectional studies suggest that proactive
customer experience
management (CEM), such as customer touch points, customer
focus, goals for
customer experiences, customer insight, employee recruitment
and training, etc., may
have a positive effect on a company’s financial performance
(Doe, 2015).

CEBMa
center for
Organizations
Step 9. Conclusion
You should make the conclusion of your CAT a concise
statement (of two or three sentences) on the
main findings on the CAT question.
EXAMPLE 1
Scientific research literature supports my assumption that a fair
change process
is important to realizing successful change, given the moderate
positive effect of
procedural justice on organizational outcomes. Although the
effects are mostly small
to medium, the indications are that there is indeed a positive
relationship between
procedural justice and acceptance, commitment and behavior
during organizational
change.
EXAMPLE 2
We can conclude that financial incentives can have a positive
impact on performance,
also known as the ‘price effect’. However, financial incentives
also have a negative
impact on the intrinsic motivation of employees, which is

known as the ‘crowding-
out’ effect. The net result of these two opposing effects
determines a possible gain or
loss in performance. In addition, the net effect is influenced by
several mediating and
moderating variables.
EXAMPLE 3
Goal-setting is one of the most powerful and evidence-based
interventions for
enhancing performance, provided that moderating factors such
as goal attribute, type
of task, organizational context and employee characteristics are
carefully taken into
account.
EXAMPLE 4
The scientific literature does not support the claim that
organizational change requires
leaders with strong emotional intelligence.
CEBMa
center for
Organizations
EXAMPLE 5

While I was unable to derive a direct cause-effect relationship
based on this CAT, we
can observe general trends that can inform practice. For
example, taking proactive
action to meet the needs of volunteers is likely to improve their
well-being, foment
loyalty among volunteer corps, and tend to improve the work of
nonprofit and
voluntary organizations. However, the studies are too limited
and flawed to recommend
wholesale changes across the nonprofit sector, but their
conclusions can at least
prompt some internal assessment for the nonprofit manager
considering further
investment in a volunteer program.
EXAMPLE 6
Corporate social responsibility has been shown to have a
positive correlation
with corporate financial performance as defined by both
accounting and market
performance. That being said, causality has yet to be proven.
There is a bi-directional
correlation between the two variables, meaning that financial
performance is correlated
with social performance and vice versa. Furthermore, while firm
size and year of
observation have been shown not to have an effect on the CSP-
CFP relationship,
other confounding variables such as national culture, research
methodology, and types
of interventions may still effect the observed correlation.

CEBMa
center for
Organizations
Step 10. Limitations
In a CAT you are aiming to provide a balanced assessment of
what is known (and not known) in the
scientific literature about an intervention or practical issue by
using a systematic methodology to
search and critically appraise empirical studies. Nevertheless,
all CAT have limitations. In your CAT you
should explicitly describe any limitations and discuss how they
possibly impacted the findings of the
assessment. Below is an example of a description of limitations
that are inherent to CATs.
EXAMPLE
Concessions were made in relation to the breadth and depth of
the search process. As
a consequence, some relevant studies may have been missed.
A second limitation concerns the critical appraisal of the studies
included. This CAT
did not conduct a comprehensive review of the psychometric
properties of the tests,
scales and questionnaires used.
A third limitation concerns the fact that the evidence on several

moderators is often
based on a limited number (sometimes only one) of studies.
Although most of these
studies were well controlled or even randomized, no single
study can be considered to
be strong evidence – it is merely indicative.
Finally, this CAT focused only on meta-analyses. As a
consequence, relevant findings
may have been missed.
Given these limitations, care must be taken not to present the
findings presented in a
CAT as conclusive.
CEBMa
center for
Organizations
Step 11. Implications and recommendations
Once you have used the evidence found to answer the CAT’s
main question, you should use the final
part of the assessment to relate the findings to the background
of the CAT and the PICOC described
in step 1 and 2. For example: Is the evidence supportive of
current practice? What are the estimated
benefits and harms? What, based on the evidence found, are
your specific recommendations for

action? Importantly, how might you consider the concerns and
interests of different organizational
stakeholders in your recommendations?
EXAMPLE 1
The fair process effect in organizations is observed when
change leaders increase
aspects of their decision-making process, specifically
consistency, accuracy, lack
of bias, and openness to employee input. When procedural
justice is not taken into
account, employees may feel treated unfairly and resistance to
change may increase.
To actively design a fair change process, the six classic criteria
for procedural justice
specified by Doe (1980) may serve as a useful checklist. These
criteria can be
turned into practical guidelines for the purpose of
organizational change as follows:
(a) the change approach needs to be consistently applied to all
employees at all
times; (b) it needs to be impartial, i.e. prejudice or stereotyping
are eliminated; (c) the
information on which decisions are based needs to be accurate;
(d) opportunities
should be provided to correct or change plans or processes; (e)
those responsible for
the organizational change (the change managers or leaders) need
to represent the
interests of all stakeholders affected by the change; and (f) the
ethical standards and
values of those involved should never be disregarded.
EXAMPLE 2

Financial incentives can be used to increase the employee
motivation and performance
needed to support change. However, upper management should
have a clear vision
about the change in performance or behavior that it desires, as it
requires different
approaches to incentivizing. Intrinsically motivated employees
executing interesting
tasks and quality outcomes should be encouraged by indirect
incentives (e.g.,
opportunities to do valued activities) in order to avoid eroding
that motivation. Direct
financial incentives are effective when extrinsic motivation and
quantitative performance
need to be stimulated. Upper management should therefore
frequently ‘calculate’ the
proposed net effect (positive price effect versus negative
crowding-out effect) when
defining a pay plan. Lastly, if the plan is designed to increase
team performance, all
incentives should not be distributed equally, as this may harm
individual motivation.
CEBMa
center for
Organizations
EXAMPLE 3

The continuation of the client’s mentoring program should be
seriously weighed
at this time, given the evidence that (a) the effect of youth
mentoring on academic
outcomes is small and (b) fidelity to matching relationships and
sustaining them over
the long term may provide greater benefit. Given that the
organization’s program is
school-based and effective long-term mentors are challenging to
recruit and retain,
the organization may want to consider suspending its mentor
program. Given that the
organization’s staff is overextended and that budget allocations
across programs are
at-risk, I would recommend that mentoring activities be
suspended at the end of the
school year.
EXAMPLE 5
This CAT demonstrates that Emotional Intelligence (EI) is not a
radical new construct
in leadership. Even though EI has (some) positive effects, these
effects can also be
explained by the overlap with other psychological constructs. In
addition, the claims
made by well-known consultancy firms such as Hay Group that
“EI can make the
difference between a highly effective and an average
professional contributor” is not
supported by the outcome of this CAT. For this reason, I advise
against investing in
training courses that claim to develop our executives’ EI.
EXAMPLE 4

Since the research evidence does not provide us with a decisive
answer to the
question whether self-managing teams perform better, no clear
recommendations for
practice can be given on whether our organization should
implement self-managing
teams. Neither can it be determined whether self-managing
teams will help our
organization to change more successfully. Instead, we suggest
that our organization
should be extra careful about implementing self-managing
teams when the divisions
involved have high levels of hierarchy, centralized decision
making or bureaucratic
formalization. Divisions characterized by high-tech novelty and
radical innovation,
however, do not need to be as cautious in implementing self-
managing teams. The
latter do need to be aware of the potential for conflict within
their teams and the
negative impact that this could have on team performance. The
available knowledge
on these contextual factors is still limited due to the current
embryonic state of
research evidence on this subject. Any introduction of self-
managing teams should
therefore be considered carefully.
CEBMa
center for

Organizations
EXAMPLE 6
Over the last 20 years, there has been a huge increase in
consumer awareness,
tighter environmental regulations, and a shift towards more
environmentally friendly
business practices. This review demonstrates that corporate
social responsibility (CSR)
strategies are both required to mitigate environmental and
regulatory risk and correlate
with higher financial performance. This relationship is
recognized regardless of industry,
firm size, or the year of study. Therefore, I would recommend
that managers explore
and assess opportunities to unlock value from CSR strategies
throughout business
operations.
CEBMa
center for
Organizations
Checklist

1. Have you clearly described the background and context of the
CAT question?
2. Does the CAT address a clearly focused question? Is it clear
what the CAT will answer?
3. Have you used the PICOC framework to focus the CAT
question?
4. Have you clearly defined the inclusion criteria (e.g.
population, outcomes of interest, study design)?
5. Have you conducted a comprehensive literature search using
relevant research databases
(i.e. ABI/INFORM, Business Source Premier, PsycINFO, Web
of Science, etc.)?
6. Is the search systematic and reproducible (e.g. were searched
information sources listed, were
search terms provided, were search results reported)?
7. Have you selected the studies using explicit inclusion and
exclusion criteria?
8. Have you clearly described the key features (year of
publication, population, sample size, study
design, outcome measures, effect sizes, limitations, level of
trustworthiness) of all studies included?
9. Have you assessed the methodological appropriateness of
each study using predetermined
quality criteria?
10. Have you provided definitions of the key
elements/constructs in the CAT question?

11. Have you clearly described the assumed causal mechanism?
12. Have you provided an overview with the main findings,
including their level of trustworthiness and
effect size?
13. Have you provided a clear, succinct conclusion on the main
findings on the CAT question?
14. Have you clearly described all limitations and discussed
how they may impact on the findings of
the CAT?
15. Have you clearly described what the implications for
practice are?
CEBMa
center for
Organizations
References
Antman, E. M. (1992). A comparison of results of meta-
analyses of randomized controlled trials and
recommendations of clinical experts. Journal of the American
Medical Association, 286(2), pp. 240-
248.

Ariely, D., Gneezy, U., Loewenstein, G., & Mazar, N. (2009).
Large stakes and big mistakes. The
Review of Economic Studies, 76(2), 451-469.
Bushman, B., & Wells, G. (2001). Narrative impressions of
literature: The availability bias and
corrective properties of meta-analytic approaches. Personality
and Social Psychology Bulletin, 27(9),
1123-1130.
Chalmers, I., Enkin, M., & Keirse, M. (1993). Preparing and
updating systematic reviews of
randomized controlled trials of health care. Millbank Quarterly
(71), 411-437.
Cohen, J. (1988). Statistical power analysis for the behavioral
sciences (2nd ed.). Hillsdale, NJ:
Lawrence Earlbaum Associates.
Cooper, H. M., & Lindsay, J. L. L. (1998). Research synthesis
and meta-analysis. p. 271-285. Sage
Publications.
Denyer, D (2014) The stages of a systematic review. Retrieved
from http://www.restore.ac.uk/
logicofenquiry/logicofenquiry/gst/SR/stages/Pages/default.html
Fink, A. (1998). Conducting Research Literature Reviews: From
Paper to the Internet. London:
Sage Publications.
Gough, D., Oliver, S., & Thomas, J. (Eds.). (2012). An
introduction to systematic reviews. Sage.
Government Social Research Service (2009) GSR rapid
evidence assessment toolkit (Online), www.

civilservice.gov.uk/my-civil-
service/networks/professional/gsr/resources/gsr-rapid-evidence-
assessment-toolkit.aspx
Hallgren, K. A. (2012). Computing inter-rater reliability for
observational data: an overview and tutorial.
Tutorials in quantitative methods for psychology, 8(1), 23.
Higgins, J., & Green, S. (Eds.). (2006). Cochrane Handbook for
Systematic Reviews of Interventions
(Vol. Version 5.1.0. (updated March 2011)). Chichester, UK:
John Wiley & Sons, Ltd.
Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2009).
Preferred reporting items for systematic
reviews and meta-analyses: the PRISMA statement. Annals of
internal medicine, 151(4), 264-269.
Moher, D., Schulz, K. F., & Altman, D. G. (2001). The
CONSORT statement: revised recommendations
for improving the quality of reports of parallel group
randomized trials. BMC Medical Research
Methodology, 1(1)
Newman M, Elbourne D (2005) Improving the usability of
educational research: guidelines for the
reporting of primary empirical research Studies in Education
(the REPOSE guidelines). Evaluation and
Research in Education 18 (4), 201-212.
Petticrew, M. (2001). Systematic reviews from astronomy to
zoology: myths and misconceptions.
British Medical Journal, 322(January), 98-101.
Petticrew, M., & Roberts, H. (2008). Systematic reviews in the
social sciences: A practical guide.

John Wiley & Sons.
Shadish, W., Cook, T., & Campbell, D. (2002). Experimental
and Quasi-Experimental Designs for
Generalized Causal Inference. Boston: Houghton Mifflin
Company.
Ziliak, S. (2016). Statisticians Found One Thing They Can
Agree On: It's Time To Stop Misusing
P-Values. Retrieved January 04, 2017, from
http://fivethirtyeight.com/features/statisticians-found-
one-thing-they-can-agree-on-its-time-to-stop-misusing-p-values/
http://www.restore.ac.uk/logicofenquiry/logicofenquiry/gst/SR/s
tages/Pages/default.html
http://www.restore.ac.uk/logicofenquiry/logicofenquiry/gst/SR/s
tages/Pages/default.html
CEBMa
center for
Organizations
Appendix: Summarizing scientific literature
Evidence summaries come in many forms. One of the best-
known types is the conventional literature
review, which provides an overview of the relevant literature
published on a topic. However, a
conventional literature review’s trustworthiness is often low:
clear criteria for inclusion are often lacking

(including whether cited work is peer reviewed), studies are
selected based on the researcher’s
individual preferences, and the research methodology is
generally not subjected to a critical appraisal
(Antman, 1992; Bushman & Wells, 2001; Chalmers, Enkin, &
Keirse, 1993; Fink, 1998). As a result,
most conventional literature reviews are prone to severe bias
and are therefore largely considered
untrustworthy as an answer to questions relevant to practice.
For this reason, many evidence-based
disciplines use so-called ‘systematic reviews’ instead. This type
of review is a specific methodology
that aims to comprehensively identify all relevant studies on a
specific topic, and to select appropriate
studies based on explicit criteria. In addition, the
methodological quality of the studies included is
assessed by on the basis of explicit criteria, such as the
presence of a pre-test or a control group
(Higgins & Green, 2006; Petticrew & Roberts, 2006). In
contrast to a conventional literature review,
a systematic review (SR) is transparent, verifiable, and
reproducible, and, as a result, the likelihood
of bias is considerably smaller. Many SRs also include a meta-
analysis, in which statistical analysis
techniques are used to combine the results of individual studies
to arrive at a more accurate estimate
of effects 2.
Although the SR methodology was originally developed in the
field of medicine, its added value
is evident in disciplines such as nursing, education, policing,
criminology, public policy, and
management (Petticrew, 2001). In disciplines in which
evidence-based practice is well established,
SRs are provided by global communities such as the Cochrane
and Campbell collaborations, and

by organizations such as the EPPI Centre. In management,
however, the SR methodology is not yet
widely adopted, and systematic reviews are consequently
scarce.
Rapid Evidence Assessments (REAs) and Critically Appraised
Topics (CATs) are two other types of
evidence summaries that can inform practice. Both apply the
same systematic approach to selecting
the studies – the methodological quality and practical relevance
of the studies are assessed based
on explicit criteria; thus, summaries are transparent, verifiable
and reproducible. The main way in
which these three types of summaries vary is in relation to the
time and resources used to produce
them and the scope and depth of the results produced. CATs are
the quickest to produce and may
take one skilled person a few days to produce. REAs might take
two skilled persons several days to
weeks. A SR usually takes a team of academics several months
to produce, as it aims to identify all
published and unpublished relevant studies (see table on the
next page).
In general, an organization will not have time or financial
means to hire a team to conduct a SR on a
managerial topic of interest. A CAT, on the other hand, may be
a good way to get a quick impression
of the available scientific evidence regarding the effect of a
specific intervention, but it may be lacking
rigor. As a result of these practical limitations, an REA is the
most widely used method of reviewing
the scientific literature within Evidence-Based Management.
2 The difference between a systematic review and a meta-
analysis is not always clear. Many studies defined as systematic

reviews include a meta-analysis aggregating statistical effects;
conversely, studies defined as meta-analyses often include
a systematic review of the literature.
CEBMa
center for
Organizations
Quality criteria SR REA CAT LR
The search for studies is systematic and reproducible 4 4 4 7
All relevant research databases are searched 4 4 7 7
Attempts are made to locate unpublished research 4 4/7 7 7
Studies are selected based on explicit inclusion and exclusion
criteria 4 4 4 7
The study selection is conducted by two reviewers,
independently
of each other 4 4 7 7
The selection process is clearly documented, for example, in the
form of a flow chart that shows how many studies are excluded
and why
4 4 7 7

The process to extract data are clearly defined and is presented
in a table 4 4 7 7
The methodological appropriateness of each study included is
appraised using predetermined quality criteria 4 4 4 7
The methodological quality of each study included is appraised
using predetermined quality criteria 4 4 7 7
The appraisal of the methodological quality is conducted by two
reviewers, independently of each other 4 4/7 7 7
The practical and methodological heterogeneity of the studies is
assessed. 4 4 7 7
The statistical heterogeneity of the studies is assessed. 4 7 7 7
Statistical analysis techniques (meta-analyses) are used to
combine the results of individual studies to arrive at a more
accurate estimate of effects
4/7 7 7 7
CEBMa
center for
Center for Evidence Based Management
www.cebma.org

Amsterdam
Critically Appraised TopicsSteps in the CAT processStep
1.Background: What is the context of the CAT questionStep 2.
Formulating the CAT question: What does the CAT answer?Step
3. Defining inclusion criteria: Which studies will be taken into
account?Step 4. Search strategy: How should the studies be
sought?Step 5. Study selection: How should you select the
studies?Step 6. Data extraction: What information should be
extracted?Step 7. Critical appraisal: How should the quality of
the studies be judged?Step 8. Results: What did you find?Step
9. ConclusionStep 10. LimitationsStep 12. Implications and
recommendationsChecklistReferencesAppendix: Summarizing
scientific literature
Button 3: Button 2: Page 2: Page 31: Page 42: Page 53: Page 64:
Page 75: Page 86: Page 97: Page 108: Page 119: Page 1210:
Page 1311: Page 1412: Page 1513: Page 1614: Page 1715: Page
1816: Page 1917: Page 2018: Page 2119: Page 2220: Page 2321:
Page 2422: Page 2523: Page 2624: Page 2725: Page 2826: Page
2927: Page 3028: Page 3129: Button 4:
The Impact of Experiential Design Elements on the Attendee
Experience
A. Introduction
As an aspiring event professional, it is important to stay up- to-
date
with the latest event industry trends. As I approach graduation
and
begin the process of searching for full-time positions in the
events
industry, employers are more likely to seek out candidates that
have at
least some familiarity or experience with the latest industry

trends.
Recently, there is a growing trend for the incorporation of
‘design
thinking’ or experiential design elements into event production,
particularly amongst in trade shows. Incorporation of
experiential design
elements is important to both internal and external event
stakeholders.
For organizations and event planners, there is a consideration of
whether
incorporating experiential design elements is necessary for
event
success. While for event attendees, experiential design elements
have
the potential to enhance an event experience and create an
overall more
memorable event experience. But what is known in the
scientific
literature about the impact of experiential design elements on
attendees’
event experience.
elements have a positive impact on the attendee experience?
-Question: What are considered experiential design
elements?
B. Search Description
PICOC
Population Event attendees
Intervention Incorporation of experiential design elements
Comparison Lack of experiential design elements
Outcome Improved attendee experience

Contex t Trade shows, special events, sports events
Inclusion Criteria
◊ Date: Published between 1980-2016
Commented [LDJP1]: Two things here Lindsay: It would
have been good to have an explanation or short description of
what experiential design elements would entail according to
you before you conduct the CAT! Second, you need a claim
about the relationship between experiential design and event
experience to test before you create your question. This is
missing here
◊ Language: English
◊ Type: Empirical, mixed method approach, literature reviews,
and
theoretical frameworks
◊ Measure: Studies where the effect of experiential design
elements
was measured against reports on the attendee experience
◊ Outcome: Feedback on the attendee experience as a result of
usage
of experiential design elements
◊ Context: Trade shows, special events, sports events, other
miscellaneous events
Ex clusion Criteria

◊ Impacts of experiential design elements that do not relate to
the
attendee experience
◊ Discussion or evaluation of experiential design or design
thinking
approaches throughout an organization generally
Search Strategy
The following databases were used to identify relevant studies:
ABI/INFORM, Business Source Complete, Leisure Tourism
(CABI), and the
Journal of Convention & Event Tourism. When conducting my
search for
relevant studies, the following filters were applied:
-December 31, 2016
-reviewed articles
In conducting my search, I used a combination of search terms
including:
“experiential design*”, “design think*”, “design* and
element*”, attendee* or
visitor* or guest* or tourist* or consumer* or customer*,
impact* or metric*
or measure* or assess*, and experienc*. When looking for
relevant studies
within the Journal of Convention & Event Tourism, I only used
the search
term “impact of design on attendee experience” because the
journal would
not allow entry of multiple search terms at once.
Search Details

Search Terms ABI
INFORM
Business
Source
Complete
(EBSCO)
Leisure
Tourism
(CABI)
J ournal of
Convention
& Event
Tourism
S1 Noft(“ ex periential
design*” )
38 7 4 -
S2 Noft(“ ex periential
design*” ) AND
noft (attendee* or
visitor* or guest*
or tourist* or
consumer* or
customer*)
7 4 3 -
S3 Noft(“ design

think*” ) AND
noft(attendee* or
visitor* or guest*
or tourist* or
consumer* or
customer*)
72 34 1 -
S4 Noft(“ design
think*” ) AND
(attendee* or
visitor* or guest*
or tourist* or
consumer* or
customer*) AND
noft(ex perienc*)
27 15 1 -
S5 Noft(“ design
think*” ) AND noft
(impact* or
metric* or
measure* or
assess*) AND
noft(ex perienc*)
38 8 0 -
S6 Ti(design* and
element*) AND
noft(attendee* or
visitor* or guest*

or tourist* or
consumer* or
customer*) AND
noft (impact* or
metric* or
measur* or
assess*)
20 7 0 -
S7 “ impact of design
on attendee
ex perience”
- - - 383
Study Selection
The selection of relevant studies took place over two phases.
First, I filtered
through the titles and abstracts of all articles that populated in
all seven
searches. In case of doubt, the study was included, and
duplicate
publications were removed. The first screening resulted in the
identification
of seven studies that were relevant to experiential design
elements or the
concept of design thinking. Secondly, I read through all seven
studies to
determine which provided explicit information concerning the
effects of
experiential design elements on the attendee experience, as it
relates to
some type of event. This resulted in only two studies: 1
literature review and

1 case study.
Data Ex traction
Author &
Year
Sector/Population Design &
Sample Size
Main Findings Effect
Size
Limitations Level
Kathleen
Beard
Nelson
(2009)
Review of three
theoretical
frameworks
Literature
review of
three
theoretical
frameworks
on the
Three
theories are
proposed
that discuss
the many
different

N/a Theoretical
perspective
with no
empirical
evidence
E
Commented [LDJP2]: Well explained!
Formatted Table
relationship
of
experiential
design
elements and
event
experience
elements of
experiential
design.
These
elements can
be physical,
visual,
auditory, and
ambient and
if
implemented
can help to
overcome a

consumer’s
innate bias
and can
provide
organizations
a
competitive
advantage.
Heekyung
Sung &
Woojin
Lee
(2015)
Attendees at the
International
City/County
Management
Association
Convention at
the Phoenix
Convention
Center in
October 2012
Quantitative
experiment.
Case study
that issued a
self-
administered
questionnaire
to 400
attendees
that

generated
217 useable
Basic and
excitement
services were
both show to
positively
influence the
attendee’s
individual
experiential
value while
performance
services were
shown to
Small Only a single
sample of
attendees’
perspective
on individual
consumption
value and
satisfaction
was
measured;
service
elements of
the
B Commented [LDJP3]: no
Commented [LDJP4R3]: t sure why the table appeared like
this…In any cases, good information in it.

survey
responses.
have no
significant
impact.
Individual
consumption
value had an
impact on
raising an
attendee’s
overall
satisfaction.
convention
center were
used to
evaluate
individual’s
satisfaction
with the
event
C. Critical Appraisal
After a critical appraisal of the seven studies deemed relevant,
only two
were included. This is because most of the studies found did not
directly
discuss the effects of experiential design elements on the
attendee’s
experience and rather focused on general implications and
effects of
implementation of design thinking proponents throughout an

organization or on the implementation of design thinking
proponents as
it relates to the service experience, an environment where the
consumer
interacts with an organization’s personnel and product for
purchase.
The overall quality of the studies included were low. Of the two
studies
deemed relevant, one study only surveyed a single sample, and
the
other study consisted of mere theories on what experiential
design
elements are considered and listed potential implications of
these
elements, with no empirical evidence to support these claims.
Because
these studies survey such limited sample populations, the results
have
very little validity or reliability and therefore are not very
trustworthy.
Additionally, neither of the two studies included the most
preferred
methodological approaches for cause and effect research
questions.
However, the studies did provide good insight as to what are
experiential design elements.
Commented [LDJP5]: Indeed!
D. Results
Definitions
In looking to define experiential design elements, there were

three
theories that were referenced, which were Goffman’s
dramaturgy theory
(1959), Kotler’s atmospheric theory (1973), and Bitner’s
servicescape
theory (1992). Dramaturgy is centered around all things
theatrical and
themed and include elements related to the event setting or
environment (Nelson, 2009). Atmospherics are defined as
“elements of
the service environment that influence affective reaction to
place” that
includes visual, aural, olfactory, and tactile elements. (Nelson,
2009).
Lastly, servicescapes focus on the “relationship of physical
complexity to
action performance” that includes ambient conditions, elements
of
spacial functionality, and visual signs, symbols, and artifacts
(Nelson,
2009).
Convention services are defined as services provided by the
convention
facility including both tangible and intangible services that
research has
shown can impact an attendee’s experiential consumer value and
overall
event satisfaction (Sung, Lee, 2015). For the purpose of this
study,
convention services were divided into three types: basic
services,
performance services, and excitement services. Basic services
are an
attendee’s minimum requirements from a facility that cause
dissatisfaction if not fulfilled including maintenance of

facilities and
comfort of seating in meeting rooms (Sung, Lee, 2015).
Performance
services are directly linked to an attendee’s need and desires
including
convenience of parking and local food & beverage facilities
(Sung, Lee,
2015). Lastly, excitement services are services that increase
attendee
satisfaction if provided but do not negatively impact attendee
satisfaction if not provided such as unique architecture design
of the
facility or organic food availability (Sung, Lee, 2015).
Main Findings
The presence of basic and excitement services were shown to
positively
impact an attendee’s experiential consumption value, but
performance
Commented [LDJP6]: Music to my ears!
services are shown to have no effect on experiential
consumption value.
(Level B)
This study revealed there is a direct relationship between the
presence of
basic and excitement services and an attendee’s experiential
consumption value (Sung, Lee, 2015). Essentially, high service
quality has
a positive impact on experiential value (Sung, Lee, 2015). As it
relates to
the specific research question posed in this paper, excitement

factors
could be considered a form of experiential design elements, as
excitement factors are not considered critical to an event’s
function, but
rather add components to simply enhance an attendee’s
experience.
Therefore, from this finding, one could proclaim advance that in
this
study experiential design elements were shown to positively add
to an
attendee’s experiential consumption. However, this finding may
be very
limited in its reliability as this finding was determined from the
analysis
of a single case study.
An attendee’s perception of service quality is shown to directly
impact
overall attendee satisfaction. (Level B)
This study also revealed that a relationship between an
attendee’s
evaluation of quality of services provided and the attendee’s
overall
satisfaction with the event and the convention center. As it
relates to the
specific research question posed in this paper, one could
proclaim that
an increase in the attendee’s perception of services, which
includes the
aforementioned experiential design elements, positively impact
the
attendee’s experience. However, this finding may be very
limited in its
reliability as this finding was determined from the analysis of a
single

case study.
E. Conclusion
Scientific literature does not support the claim that experiential
design
elements positively impact an attendee’s experience. This is
primarily
because there is such littleso few research that was found that
specifically illustrates the relationship of experiential design
elements on
Commented [LDJP7]: Incredible!
Commented [LDJP8]: Or you could say: may need further
confirmation with new studies that would consider….
Commented [LDJP9]: Here is my claim!!!!
the attendee event experience to be able to draw a trustworthy
conclusion.
F. Limitations
As this was intended to be a mini-CAT assignment, concessions
were
made in both the breadth and depth of the search process.
Database
searches were aimed to be inclusive yet limited in scope. This
conclusion
was also reached from the analysis of only two relevant studies,
which
definitely effected the results. Additionally, of the two relevant
studies,
neither had the preferred methodological approach and only
included

the analysis of theories and a single case study. Lack of analysis
of
studies with more preferred methodological approaches could
have also
contributed to other relevant findings being missed. Lastly, the
main
findings are based on such a small sample size that the findings
cannot
be considered truly trustworthy or generalizable.
G. Recommendations & Implications
This CAT concludes there is no positive impact on the attendee
experience from the incorporation of experiential design
elements. This
suggests that the growing trend of the incorporation of design
thinking
is currently perceived as an additional benefit to event attendees
but is
not yet considered integral for an event’s success. Therefore,
the ability
to incorporate experiential design elements into the planning
and
execution phase of an event may still be seen as a bonus feature
throughout the industry and not yet a necessary skillset to
receive
employment. As this trend continues to grow throughout the
industry, I
believe there will be more research conducted on the
relationship
between experiential design elements and the attendee
experience.
Furthermore, I believe there will come a point where event
success will
be based on the successful incorporation of experiential design
elements.

Commented [LDJP10]: Well said1
H. References
Calgren, L., Elmquist, M., & Rauth, I. (2016). The Challenges
of Using Design
Thinking in Industry- Experiences from Five Large Firms.
Creativity
and Innovation Management.
doi:https://doi.org/10.1111/caim.12176 [1st
phase]
Chen, R. (2016). Fuzzy dual experience-based design evaluation
model for
integrating engineering design into customer
responses.10:439. https://doi.org/10.1007/s12008-016-0310-y
[1st phase]
Commented [LDJP11]: This section needs a better
presentation: it is cluttered!
https://doi.org/10.1007/s12008-016-0310-y
Cole-Colander, C. (2003). Designing the Customer Experience.
Building
Research & Information, 31(5),357.
https://doi.org/10.1080/0961321032000088025 [1st phase]
Lin, C. J., & Cheng, L.-Y. (2015). An integrated model of
service experience
design improvement. Service Industries Journal, 35(1–2), 62–
80.
https://doi.org/10.1080/02642069.2014.979407 [1st phase]
Nelson, K. (2009). Enhancing the Attendee's Experience

through Creative
Design of the Event Environment: Applying Goffman's
Dramaturgical Perspective. Journal of Convention & Event
Tourism.
10:2, 120 -133, DOI: 10.1080/15470140902922023 [2nd phase]
Sung, H. & Lee, W.(2015) The Effect of Basic, Performance and
Excitement
Service Factors of a Convention Center on Attendees'
Experiential Value and Satisfaction: A Case Study of the
Phoenix Convention
Center. Journal of Convention & Event Tourism. 16:3, 175-
199, DOI: 10.1080/15470148.2015.1034908 [2nd phase]
Wattanasupachoke, T. (2012). Design thinking, innovativeness
and
performance: An empirical examination. International Journal
of
Management and Innovation, 4(1), 1-14. Retrieved from
http://proxygw.wrlc.org/login?url=https://search.proquest.com/d
ocview/10
19286671?accountid=11243 [1st phase]
https://doi.org/10.1080/0961321032000088025
https://doi.org/10.1080/02642069.2014.979407
https://doi.org/10.1080/15470140902922023
https://doi.org/10.1080/15470148.2015.1034908
ocview/1019286671?accountid=11243
ocview/1019286671?accountid=11243Chen, R. (2016). Fuzzy
dual experience-based design evaluation model for integrating
engineering design into customer responses.10:439.

https://doi.org/10.1007/s12008-016-0310-y [1st phase]
CRITICALLY APPRAISED TOPIC:
IMPACT OF EMPLOYEE DIVERSITY ON ORGANIZATIOAL
PERFORMANCE
Background:
As the rapid growth of globalization and the increased economic
interactions between the multinational companies all over the
world, the flow and allocation of human resource, many
companies have hired employees from different countries with
different human capital backgrounds and biological-

demographical backgrounds. International human resource
management is gradually regarded as one of the best choices for
multinational companies to improve their competitions and
improve the performance of the programs/ projects. Besides, as
the developments of human rights and disabled people’s rights,
more women and disabled people could start their commercial
career. The diversity of employees is becoming the mainstream
for modern multinational companies. Although the diversity of
employees could make corporations more influential and
competitive, it could also cause many problems such as internal
conflicts, tension, language barriers and the increase of
disagreements. The diversity of employees has two face, one
face is positive and beneficial to improve the performance of
corporations in projects/ programs, however, the other face is
negative and controversial. Many experts and studies have
researched the effect of diversity to the performance of
multinational companies, some of them encourage the
development of diversity in multinational companies because
they claim that the diversity of employee could encourage the
internationalization of the companies and improve the outcomes
of those companies. Meanwhile, many experts doubt that the
effects of diversity of employees are not clear, and the potential
negative impacts of diversity of the employees could happen in
offices and daily management. In business and management
area, the impacts of diversity of employees need to be further
explored.
To develop deeper understand to the impacts of the diversity of
the employees to the outcomes of a group, I divided the
diversity to two major parts, the human capital diversity of
employees and the biological-demographical diversity of the
employees. I will use the Critical Appraisal Topic (CAT)
assessment to evaluate the positive and negative impacts from
these two types diversity. And my CAT report is focusing on
answering a question:

What is known in scholar literature about the impact of human
capital diversity and biological-demographical diversity of the
employees on the performance of the organizations?
Moreover, my Critical Appraisal Topic assessment will not
focus on specific area or organizations due to the diversity of
employees is the main stream in the global human resource
management area and almost all the multinational companies or
institutions are facing the same challenge no matter retailing
organizations or medical corporations or engineering
companies, this CAT provides the common suggestions to them:
Is the diversity of employees beneficial for these organizations
and What kind of diversity will be positive and what kind of
diversity will be negative?
· PICOC
· Population: Employees in organizations
· Intervention: The human capital diversity of employees
· Comparison: The biological-demographical diversity of
employees
· Outcome: Figure out the how the two types diversity will
impact the performance of multinational companies or
groups.
· Context: Multinational companies or international groups in
the international human resource market
· Search strategies:
· Anu Super search database were searched using the
keywords/terms such as “employee diversity *” Global human
resource management”, “meta-analysis”, “organizational
outcomes”. The search yielded over 80000 studies which
published in peer-reviewed journals, journals articles. and the
filters behind will be applied during the search:
· Searching for the title and abstract

· Scholarly peer-reviewed
· Journal articles
· Published date from 2001 – 2018
· Only English studies will be reviewed
· 6. Full text online
· Selection of studies:
· I have strictly followed the inclusion criteria and the search
strategy to make sure that my strategies are relevant to the
impacts of the two kinds of diversity of employees on the
performance of organizations. And 12 studies are selected to
identify the relationship between the two type of employee
diversity and the organizational performance.
· Most of my chosen studies are based on cross-sectional and
primary research studies also with one meta-analysis study in
the study group.
· I have carefully chosen the studies which research on the
relationship between the diversity of employees and the
outcomes of firms/companies instead of small group or
individuals.
· I have checked that these studies based on actual firms not
fictitious organizations.
· I carefully follow the specific requirements in inclusion
criteria.
Set
Search
Database
Results
S1
ab(employee diversity) and ti(employee diversity) and
ab(organizational performance)
ANU Super Search
9
S2

ab(human capital diversity) and ti(employee diversity) and
ANU Super Search
1
S3
ab(biological and demographical diversity) and ti(employee
diversity) and ab(organizational performance)
ANU Super Search
0
S4
ab(employee diversity) and ti(meta-analysis) and
ANU Super Search
2
Data extraction
AUTHOR & YEAR
Design
Effect Size
Main Findings
Weakness
Level & Trust
worthiness
Ozgen Peters, Niebuhr, Nijkamp & Poot
2014
Cross-sectional / Primary Research
494 (Large)
Cultural diversity could lead to better organizational
performance.
The uncertain relationship between immigration diversity in
organizations and the performance of organizations.
None
B

Sanchez, Vinces & Guillen, 2018
Cross-sectional / Primary Research
902(large)
Positive effect of cultural/human capital diversity on
organizational performance.
Positive effect of demographical/ biological diversity on
organizational performance.
The research and just focus on one region the Brazi which could
not represent the common impacts globally.
D
Garib, 2013
Cross-sectional/ primary research
NA
Apparently positive effect of hu diversity on organizational
performance
Very Small positive effect of demographical like gender and sex
diversity on organizational performance
The research only covers one country Holland.
D
Ostergaard, Timmermans & Kristinsson,
2010
Cross-sectional/ Primary Research
1684(large)
Gender diversity of employees lead to better organizational
performance.
The age diversity of employees lead to worse organizational
performance.
The educational diversity leads to better creativity and
innovation and organizational Performance.
This evidence only researcher the major department or plants of
the firms.
And the measurement of organizational performance is based on

the innovation of these firms.
D
Horwitz & Horwitz,
2007
Meta-analysis
78
Without obvious correlation between biological-demographical
diversity on organizational performance.
None
B
Critical Appraisal:
Through the critical appraisal of four primary researches and
one meta-analysis, the trust worthiness is not high but close to
the moderate quality. And from the 5 studies above it could be
concluded that the relationship between human capital diversity
of employees / biological-demographical diversity of employees
and the outcomes of the organizations is moderate no matter it
is negative or positive. In addition, I did not set the control
groups for the studies in the data extraction above due to the
main target of this CAT report is figuring out the negative and
positive relationship between the two types diversity of
employees and the performance of the organizations , the
accurate results for each study is not necessary for this CAT
report.
Definition:
1. The concept diversity of employee represents that the
company hires employees with different characteristics. And
there are two major diversity the human capital diversity and
the biological-demographical diversity. The human capital
diversity includes educational background, family background
working experience, professional expertise, social ranks etc.

And the biological-demographical diversity includes the
differences in gender, race or language.
2. The performance of multinational organizations normally
includes external innovations, internal harmony, managerial
abilities, problem solving ability and decision-making ability
also healthy development of organizations and the profits from
global operations.
Causal Mechanism:
The primary research and meta-analysis studies above in the
data extraction demonstrate that the performance and outcomes
of the multinational companies rely on the global context and
the situations of these organizations. However, the human
capital and biological-demographical diversity could deeply
affect the organizations such as communication, negotiation,
conflicts, promotions, problem solving and decision making,
and all of these elements could decide the outcomes and
performance of organizations.
In fact, one of the humanity for most of people is avoiding
uncertainty and diversity because most of us prefer to
communicate and corporate with people who are similar with us
because when employees communicate and work with people
have similar characteristics with them like similar educational
background, similar social ranks and similar value or people
have same gender, race or age with them. When people work
with similar people, they tend to have higher communicative
efficiency and less barrier, also they will feel comfortable when

work with people who are easier to understand. Since most of us
like staying in the comfort zone and lock ourselves in our own
rooms.
On the other hand, there are uncertainty and risks hidden in the
diversity, but when our partners who have different educational
background different working experience or different gender or
race with us work with us, they may have wider vision and
better knowledge base and managerial capabilities. Therefore, it
is possible that when we work in a diverse group/organization,
the diverse employees will improve the efficiency and outcomes
of these organizations. The risks and uncertainty sometimes
bring better outcomes to organizations.
Main Findings:
1. The Human capital diversity tend to have positive
relationship with the performance of the multinational
organizations. The human capital diversity contains the
educational background, professional knowledge, expertise and
working experience, these kinds of elements regarded as
beneficial to improve the decision making and efficiency of the
projects/programs from the studies in CAT report. (Level, D)
2. The Gender diversity ( the gender balance) and cultural
diversity in companies could improve the managerial level,
creativity and innovation in the multinational companies, then,
they have more positive impacts to the organizational
performance while age diversity tend to has negative impacts to
the organizational performance. (Level, D)
3. Studies demonstrate that two types of diversity of employee
will not have permanent impacts to the organizational
performance, the impacts from same diversity could be different
in different areas. In some developing countries both the human
capital and biological-demographical diversity could have
negative impacts to the outcomes of organizations. Meanwhile,

in some developing countries like Holland and Germany, both
the two types diversity could have positive impacts to the
performance of the multinational companies. No findings
illustrate that the two types diversity of employee will have
same impacts to every context.(Level, B)
4. Although the gender diversity has positive impacts to the
organizational performance, however, the other elements of
biological tend to have very small positive or negative impacts
to the performance of organizations. Therefore, in the two types
of diversity, the human capital diversity is more beneficial and
positive for organizational outcomes than the biological-
demographical diversity.
Conclusion:
Through the research to the critical appraisal scholar literature
on human capital and diversity’s impact on organizational
performance, three main points could be provided. First of all,
the impacts of a type of diversity might change when this kind
of diversity was applied to another area/ country So, it is not
appropriate to define that a kind of diversity will always be
positive or beneficial to the organizational performance or
outcomes. Secondly, in these two types of diversity of
employees, the human capital diversity tends to have more
positive impacts than biological-demographical diversity. The
biological-demographical diversity might be more uncertain and
less positive to the organizational performance. Third, the
cultural and gender diversity could be more positive and
beneficial to the organizational performance, and the age
diversity is obviously negative to the organizational outcomes.
To sum up, organizations could hire employees with more
human capital diversity especially the gender and cultural
diversity and less biological-demographical diversity especially
the age diversity.

Limitations:
There are three major limitations in my CAT report. They are:
1. The trustworthiness of my studies is just close to moderate,
they are high level and strongly reliable studies. Therefore, the
accuracy and reliance of my research will be negatively
influenced.
2. There are no control groups in my CAT which means my
results could not be that accurate.
3. I divided different many kinds of diversity to two major
diversity, the human capital diversity and the biological-
demographical diversity, and the impacts to the organizational
performance are based on these two types of major diversity.
Some kinds of diversity may not be considered or ignored, and
my result are two general which could be further explored.

Reference list
Garib, G. (2013). Leisure managers’ perceptions of employee
diversity and impact of employee diversity. International
Journal of Hospitality Management, 32, 254-260.
doi:10.1016/j.ijhm.2012.07.002
Horwitz, S. K., & Horwitz, I. B. (2007). The effects of team
diversity on team outcomes: A meta-analytic review of team
demography. Los Angeles, CA: Sage Publications.
doi:10.1177/0149206307308587
Triguero-Sánchez, R., Peña-Vinces, J., & Guillen, J. (2018).
How to improve firm performance through employee diversity
and organisational culture. Revista Brasileira de Gestão de

Negócios, 20(3). Retrieved from:
https://www.researchgate.net/profile/Jesus_Pena-
Vinces/publication/326486344_How_to_improve_firm_performa
nce_through_employee_diversity_and_organisational_culture/li
nks/5b57052faca27217ffb74499/How-to-improve-firm-
performance-through-employee-diversity-and-organisational-
culture.pdf
Østergaard, C. R., Timmermans, B., & Kristinsson, K. (2011).
Trrsp;arsrst879R 78
a esearch Policy, 40(3), 500-509.
doi:10.1016/j.respol.2010.11.004
Ozgen, C., Peters, C., Niebuhr, A., Nijkamp, P., & Poot, H. J.
(2014). Does cultural diversity of migrant employees affect
innovation? International Migration Review,48(S1), S377-S416.
doi:10.1111/imre.12138
CHEN 3
PROBLEM DEFINITION REPORT
I - Background and Context
Australian universities have been experiencing discrimination
issues on campus. Most of the universities enjoy a long-
standing motto which celebrate diversity as a critical value.

However, the increasing population of transgenders on campus
has imposed a challenge on the school administration and the
student body due to the limited prior experience and possible
bullying targeting the transgender community. Hence, the
school administration is looking for ways to prevent bullying
and better support the transgender and the LGBTI population at
large.
II - Justification of Interest and Evidence for the Problem
The 2016 Census has revealed that transgenders occupied
approximately 14.8% of the Australian population (Australian
Bureau of Statistics, 2018.) With such a high percentage, the
universities have provided limited support to the group and
somehow neglected the diversity awareness education in the
curriculum. As a result, bullying, including cyberbullying,
would occur at high chances. Indeed, a survey of an American
college has indicated that 78% of the transgender students had
experienced harassments (Stolzenberg and Hughes, 2017.) If the
discriminatory behaviors led to on-campus violence, the school
would take much burden and responsibilities. Furthermore, the
deficiency in diversity awareness would hurt the university’s
reputation, which could lead to substantial financial losses.
III - Questions: Causes of the Problem
PICOC
Population = Enrolled students of the universities
Intervention = Diversity awareness education and Support for
transgenders
Comparison = No education or support
Outcome = Incidence of discriminatory violence or bullying
Context = Australian universities
Table 1 - Questions to explore the problem from scientific
sources
Questions
Justifications

What is the impact of not providing diversity awareness
education program on the occurrence of discriminatory violence
or bullying incidence in the scientific literature?
It attempts to estimate the probability of the occurrence of the
outcome situation in the case of no intervention.
not providing administrative support for transgenders on the
occurrence of discriminatory violence or bullying incidence ?
Framed in the same way as the previous question, this question
evaluates the occurrence of the outcome in the case of lacking
another intervention.
IV - Questions: Potential
Solution
s to the Problem
Table 2 - Questions to explore possible solutions from
scientific sources
Questions
Justifications
What is known in the scientific literature about the impact (to
what degree) of the diversity awareness education program
(Intervention) on preventing incidence of discriminatory
violence or bullying (outcome) ?
It probes the causality between the “Intervention” and
“Outcome.” The question focuses on one intervention method,
and also investigates the degree of influence, i.e., the Moderator

variable.
the administrative support for transgenders (Intervention) on
preventing incidence of discriminatory violence or bullying
(outcome) ?
It explores the causality between another intervention method
and the ideal outcome. The question also critically examines the
academic evidence.
V - References:
Australian Bureau of Statistics. (2018, November 6). 2071.0 -
Census of Population and Housing: Reflecting Australia -
Stories from the Census, 2016. Retrieved from
https://www.abs.gov.au/ausstats/[email protected]/Lookup/by%2
0Subject/2071.0~2016~Main%20Features~Sex%20and%20Gend
er%20Diversity%20in%20the%202016%20Census~100
Stolzenberg, E. B., & Hughes, B. (2017). The Experiences of
Incoming Transgender College Students: New Data on Gender
Identity. Liberal Education, 103(2), n2.

CEBMa center forEvidence-Based Management CEBMa G.docx

CEBMa center forEvidence-Based Management CEBMa G.docx

Recommended

Recommended

More Related Content

Similar to CEBMa center forEvidence-Based Management CEBMa G.docx

Similar to CEBMa center forEvidence-Based Management CEBMa G.docx (20)

More from keturahhazelhurst

More from keturahhazelhurst (20)

Recently uploaded

Recently uploaded (20)

CEBMa center forEvidence-Based Management CEBMa G.docx