Improving	
  Evalua/ons	
  and	
  U/liza/on	
  with	
  
Sta/s/cal	
  Edge:	
  Nested	
  Data	
  Designs	
  and	
  
Hierarc...
•  The	
  goal	
  of	
  this	
  presenta0on	
  is	
  to	
  introduce	
  
the	
  concept	
  of	
  HLM	
  and	
  explain	
  ...
What	
  is	
  Hierarchical	
  Linear	
  Modeling	
  (HLM)	
  
•  HLM	
  is	
  a	
  sta0s0cal	
  technique	
  to	
  analyze...
Student	
  
Class	
  
School	
  
District	
  
Hierarchical	
  Structure	
  –	
  mul0-­‐level	
  	
  
age	
   I.Q	
  
Measu...
•  HLM	
  enables	
  a	
  more	
  robust	
  analy0c	
  approach	
  
for	
  nested	
  data	
  (than	
  regression	
  or	
  ...
Program	
  design	
  structure	
  
Data	
  structure	
  
Evalua0on	
  ques0ons	
  
-­‐	
  Which	
  par0cipant	
  or	
  sit...
Par0cipant	
  
1	
  
Site	
  1	
   Site	
  2	
   Site	
  3	
  
Par0cipant	
  
4	
  
Par0cipant	
  
2	
  
Par0cipant
5	
  
...
Assessing	
  test	
  scores	
  by	
  age	
  from	
  site	
  to	
  site	
  
Test	
  Score	
  
Age	
  
Four	
  different	
  
...
Student	
  1	
   Student	
  2	
   Student	
  3	
  
Baseline	
  
Month	
  7	
  
Month	
  1	
  
Month	
  12	
  
Month	
  6	
...
Assessing	
  improved	
  performance	
  over	
  0me	
  
Test	
  Score	
  
Time	
  
Four	
  different	
  
study	
  par0cipan...
Tradi0onal	
  Methods	
  
Test	
  Score	
  
Age	
  
Rela0on	
  between	
  age	
  and	
  
test	
  score	
  es0mated	
  once...
Advantages	
  of	
  HLM	
  
Test	
  Score	
  
Age	
  
Four	
  different	
  
program	
  sites	
  
Here,	
  the	
  rela0on	
 ...
Design	
  Considera0ons	
  for	
  Using	
  HLM	
  
•  Sample	
  size	
  
– Par0cipant	
  level	
  
– Site	
  level	
  
– R...
Final	
  Thoughts	
  	
  
•  Applying	
  HLM	
  in	
  evalua0ons	
  with	
  nested	
  data	
  
enables	
  more	
  robust	
...
Ques0ons?	
  
	
  
Marci	
  Pernica	
  	
  	
  
Marci.Pernica@Ontario.ca	
  	
  
Ministry	
  of	
  Community	
  and	
  Soc...
Upcoming SlideShare
Loading in …5
×

Improving evaluations and utilization with statistical edge nested data designs and hierarchical linear modeling

456 views
323 views

Published on

Published in: Technology, Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
456
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
11
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Improving evaluations and utilization with statistical edge nested data designs and hierarchical linear modeling

  1. 1. Improving  Evalua/ons  and  U/liza/on  with   Sta/s/cal  Edge:  Nested  Data  Designs  and   Hierarchical  Linear  Modeling  (HLM)     CES  Conference  -­‐  June  10  2013   Marci  Pernica  -­‐  Ministry  of  Community  and  Social  Services   Judith  Godin  –  J  Godin  Consul7ng  
  2. 2. •  The  goal  of  this  presenta0on  is  to  introduce   the  concept  of  HLM  and  explain  how  it  can  be   used  in  program  evalua0on       Introduc0on  
  3. 3. What  is  Hierarchical  Linear  Modeling  (HLM)   •  HLM  is  a  sta0s0cal  technique  to  analyze  data   that  is  structured  in  hierarchies  (or  “nested”)   •  To  account  for  the  fact  that  people  that  are  “clustered”  or   “nested”  within  the  same  group  have  more  in  common  than   if  they  were  independent  random  samples       Classroom  1   Classroom   2   Classroom  3   Student  4   Student     2   Student  5   Student  3   Student  8  Student  7  Student  6   Student  9   Student   10   Student   11   Student   12   Nested   data   designs  
  4. 4. Student   Class   School   District   Hierarchical  Structure  –  mul0-­‐level     age   I.Q   Measuring   test  scores   (dependent  variable)   Independent   variables  
  5. 5. •  HLM  enables  a  more  robust  analy0c  approach   for  nested  data  (than  regression  or  ANOVA)   •  Data  in  evalua0on  are  oZen  nested   •  To  determine  success  condi*ons  for  the   program  –  e.g.  is  the  program  more  suitable   for  certain  sub-­‐popula0ons  or  more  successful   if  delivered  in  a  certain  way   Why  Use  HLM  in  Evalua0on  
  6. 6. Program  design  structure   Data  structure   Evalua0on  ques0ons   -­‐  Which  par0cipant  or  site-­‐level  characteris0cs  are  most  influen0al  in   explaining  the  varia0on  in  test  scores  among  the  program   par0cipants?   -­‐  What  program  delivery  characteris0cs  (site  level  prac0ces)  seem  to  be   having  the  most  posi0ve  impact  on  the  par0cipants’  test  scores?   -­‐  Are  some  program  features  more  suited  to  certain  sub-­‐popula0ons  (e.g.   gender,  age  group,  ethnic  or  cultural  group)     Applying  HLM  in  Evalua0on  
  7. 7. Par0cipant   1   Site  1   Site  2   Site  3   Par0cipant   4   Par0cipant   2   Par0cipant 5   Par0cipant   3   Par0cipant   8   Par0cipant   7   Par0cipant   6   Par0cipant   9   Par0cipant   10   Par0cipant   11   Par0cipant   12   Example  of  levels  of  a  hierarchical  model   Par0cipants  (level  1)  nested  within  sites  (level  2)  
  8. 8. Assessing  test  scores  by  age  from  site  to  site   Test  Score   Age   Four  different   program  sites   Although  the  test  scores  differ  from  site  to  site,  the   rela0on  between  age  and  test  score  is  the  same  at   different  sites  
  9. 9. Student  1   Student  2   Student  3   Baseline   Month  7   Month  1   Month  12   Month  6   Month  9  Month  8  Month  6   Baseline   Month  2   Month  4   Month  6   Assessing  change  over  0me   Assessments  across  0me  (level  1)  are  nested  within  individuals   (level  2)  (i.e.  repeated  measures  design)  
  10. 10. Assessing  improved  performance  over  0me   Test  Score   Time   Four  different   study  par0cipants   Although  some  individuals  have  higher  test  scores   to  start  with,  the  rate  of  change  (improved   performance)  is  comparable  among  the   par0cipants  
  11. 11. Tradi0onal  Methods   Test  Score   Age   Rela0on  between  age  and   test  score  es0mated  once  for   all  sites  together  
  12. 12. Advantages  of  HLM   Test  Score   Age   Four  different   program  sites   Here,  the  rela0on  between  age  and  test  score  varies   across  sites.   Are  there  any  site  level  variables  associated  with  the   strength  of  this  rela0on?  
  13. 13. Design  Considera0ons  for  Using  HLM   •  Sample  size   – Par0cipant  level   – Site  level   – Repeated  measures   •  Missing  Data   – Can  be  easy  or  difficult  to  deal  with   •  Number  of  variables   – Comprehensive  coverage   – Parsimony    
  14. 14. Final  Thoughts     •  Applying  HLM  in  evalua0ons  with  nested  data   enables  more  robust  results  and  conclusions   •  U0liza0on-­‐focused     – Iden0fy  evidence-­‐based  success  factors  or   condi0ons  for  improving  the  program  delivery   model,  to  ul0mately  achieve  beger  program   effec0veness     – Promo0ng  the  value  in  evalua0on  (gathering  the   evidence  to  determine  the  ‘success  factors’  for   the  interven0on  to  be  effec0ve)    
  15. 15. Ques0ons?     Marci  Pernica       Marci.Pernica@Ontario.ca     Ministry  of  Community  and  Social  Services     Judith  Godin     sta/s/cs@jgodin.com   Independent  Consultant  

×