This document discusses hierarchical linear modeling (HLM) and how it can be used to analyze data from program evaluations with nested structures. HLM accounts for the non-independence of observations by modeling variability at multiple levels, such as students within classrooms or repeated measures over time within individuals. It allows evaluators to determine how program outcomes vary across these levels and which participant or site characteristics most influence outcomes. The presenters provide examples of how HLM can be applied to evaluation questions to identify evidence-based factors for improving a program's effectiveness.

1. Improving
Evalua/ons
and
U/liza/on
with
Sta/s/cal
Edge:
Nested
Data
Designs
and
Hierarchical
Linear
Modeling
(HLM)
CES
Conference
-­‐
June
10
2013
Marci
Pernica
-­‐
Ministry
of
Community
and
Social
Services
Judith
Godin
–
J
Godin
Consul7ng

2. • The
goal
of
this
presenta0on
is
to
introduce
the
concept
of
HLM
and
explain
how
it
can
be
used
in
program
evalua0on
Introduc0on

3. What
is
Hierarchical
Linear
Modeling
(HLM)
• HLM
is
a
sta0s0cal
technique
to
analyze
data
that
is
structured
in
hierarchies
(or
“nested”)
• To
account
for
the
fact
that
people
that
are
“clustered”
or
“nested”
within
the
same
group
have
more
in
common
than
if
they
were
independent
random
samples
Classroom
1
Classroom
2
Classroom
3
Student
4
Student
2
Student
5
Student
3
Student
8
Student
7
Student
6
Student
9
Student
10
Student
11
Student
12
Nested
data
designs

4. Student
Class
School
District
Hierarchical
Structure
–
mul0-­‐level
age
I.Q
Measuring
test
scores
(dependent
variable)
Independent
variables

5. • HLM
enables
a
more
robust
analy0c
approach
for
nested
data
(than
regression
or
ANOVA)
• Data
in
evalua0on
are
oZen
nested
• To
determine
success
condi*ons
for
the
program
–
e.g.
is
the
program
more
suitable
for
certain
sub-­‐popula0ons
or
more
successful
if
delivered
in
a
certain
way
Why
Use
HLM
in
Evalua0on

6. Program
design
structure
Data
structure
Evalua0on
ques0ons
-­‐
Which
par0cipant
or
site-­‐level
characteris0cs
are
most
influen0al
in
explaining
the
varia0on
in
test
scores
among
the
program
par0cipants?
-­‐
What
program
delivery
characteris0cs
(site
level
prac0ces)
seem
to
be
having
the
most
posi0ve
impact
on
the
par0cipants’
test
scores?
-­‐
Are
some
program
features
more
suited
to
certain
sub-­‐popula0ons
(e.g.
gender,
age
group,
ethnic
or
cultural
group)
Applying
HLM
in
Evalua0on

7. Par0cipant
1
Site
1
Site
2
Site
3
Par0cipant
4
Par0cipant
2
Par0cipant
5
Par0cipant
3
Par0cipant
8
Par0cipant
7
Par0cipant
6
Par0cipant
9
Par0cipant
10
Par0cipant
11
Par0cipant
12
Example
of
levels
of
a
hierarchical
model
Par0cipants
(level
1)
nested
within
sites
(level
2)

8. Assessing
test
scores
by
age
from
site
to
site
Test
Score
Age
Four
different
program
sites
Although
the
test
scores
differ
from
site
to
site,
the
rela0on
between
age
and
test
score
is
the
same
at
different
sites

9. Student
1
Student
2
Student
3
Baseline
Month
7
Month
1
Month
12
Month
6
Month
9
Month
8
Month
6
Baseline
Month
2
Month
4
Month
6
Assessing
change
over
0me
Assessments
across
0me
(level
1)
are
nested
within
individuals
(level
2)
(i.e.
repeated
measures
design)

10. Assessing
improved
performance
over
0me
Test
Score
Time
Four
different
study
par0cipants
Although
some
individuals
have
higher
test
scores
to
start
with,
the
rate
of
change
(improved
performance)
is
comparable
among
the
par0cipants

11. Tradi0onal
Methods
Test
Score
Age
Rela0on
between
age
and
test
score
es0mated
once
for
all
sites
together

12. Advantages
of
HLM
Test
Score
Age
Four
different
program
sites
Here,
the
rela0on
between
age
and
test
score
varies
across
sites.
Are
there
any
site
level
variables
associated
with
the
strength
of
this
rela0on?

13. Design
Considera0ons
for
Using
HLM
• Sample
size
– Par0cipant
level
– Site
level
– Repeated
measures
• Missing
Data
– Can
be
easy
or
difficult
to
deal
with
• Number
of
variables
– Comprehensive
coverage
– Parsimony

14. Final
Thoughts
• Applying
HLM
in
evalua0ons
with
nested
data
enables
more
robust
results
and
conclusions
• U0liza0on-­‐focused
– Iden0fy
evidence-­‐based
success
factors
or
condi0ons
for
improving
the
program
delivery
model,
to
ul0mately
achieve
beger
program
effec0veness
– Promo0ng
the
value
in
evalua0on
(gathering
the
evidence
to
determine
the
‘success
factors’
for
the
interven0on
to
be
effec0ve)

15. Ques0ons?
Marci
Pernica
Marci.Pernica@Ontario.ca
Ministry
of
Community
and
Social
Services
Judith
Godin
sta/s/cs@jgodin.com
Independent
Consultant