Improving evaluations and utilization with statistical edge nested data designs and hierarchical linear modeling

  • 178 views
Uploaded on

 

More in: Technology , Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
178
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
7
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Improving  Evalua/ons  and  U/liza/on  with   Sta/s/cal  Edge:  Nested  Data  Designs  and   Hierarchical  Linear  Modeling  (HLM)     CES  Conference  -­‐  June  10  2013   Marci  Pernica  -­‐  Ministry  of  Community  and  Social  Services   Judith  Godin  –  J  Godin  Consul7ng  
  • 2. •  The  goal  of  this  presenta0on  is  to  introduce   the  concept  of  HLM  and  explain  how  it  can  be   used  in  program  evalua0on       Introduc0on  
  • 3. What  is  Hierarchical  Linear  Modeling  (HLM)   •  HLM  is  a  sta0s0cal  technique  to  analyze  data   that  is  structured  in  hierarchies  (or  “nested”)   •  To  account  for  the  fact  that  people  that  are  “clustered”  or   “nested”  within  the  same  group  have  more  in  common  than   if  they  were  independent  random  samples       Classroom  1   Classroom   2   Classroom  3   Student  4   Student     2   Student  5   Student  3   Student  8  Student  7  Student  6   Student  9   Student   10   Student   11   Student   12   Nested   data   designs  
  • 4. Student   Class   School   District   Hierarchical  Structure  –  mul0-­‐level     age   I.Q   Measuring   test  scores   (dependent  variable)   Independent   variables  
  • 5. •  HLM  enables  a  more  robust  analy0c  approach   for  nested  data  (than  regression  or  ANOVA)   •  Data  in  evalua0on  are  oZen  nested   •  To  determine  success  condi*ons  for  the   program  –  e.g.  is  the  program  more  suitable   for  certain  sub-­‐popula0ons  or  more  successful   if  delivered  in  a  certain  way   Why  Use  HLM  in  Evalua0on  
  • 6. Program  design  structure   Data  structure   Evalua0on  ques0ons   -­‐  Which  par0cipant  or  site-­‐level  characteris0cs  are  most  influen0al  in   explaining  the  varia0on  in  test  scores  among  the  program   par0cipants?   -­‐  What  program  delivery  characteris0cs  (site  level  prac0ces)  seem  to  be   having  the  most  posi0ve  impact  on  the  par0cipants’  test  scores?   -­‐  Are  some  program  features  more  suited  to  certain  sub-­‐popula0ons  (e.g.   gender,  age  group,  ethnic  or  cultural  group)     Applying  HLM  in  Evalua0on  
  • 7. Par0cipant   1   Site  1   Site  2   Site  3   Par0cipant   4   Par0cipant   2   Par0cipant 5   Par0cipant   3   Par0cipant   8   Par0cipant   7   Par0cipant   6   Par0cipant   9   Par0cipant   10   Par0cipant   11   Par0cipant   12   Example  of  levels  of  a  hierarchical  model   Par0cipants  (level  1)  nested  within  sites  (level  2)  
  • 8. Assessing  test  scores  by  age  from  site  to  site   Test  Score   Age   Four  different   program  sites   Although  the  test  scores  differ  from  site  to  site,  the   rela0on  between  age  and  test  score  is  the  same  at   different  sites  
  • 9. Student  1   Student  2   Student  3   Baseline   Month  7   Month  1   Month  12   Month  6   Month  9  Month  8  Month  6   Baseline   Month  2   Month  4   Month  6   Assessing  change  over  0me   Assessments  across  0me  (level  1)  are  nested  within  individuals   (level  2)  (i.e.  repeated  measures  design)  
  • 10. Assessing  improved  performance  over  0me   Test  Score   Time   Four  different   study  par0cipants   Although  some  individuals  have  higher  test  scores   to  start  with,  the  rate  of  change  (improved   performance)  is  comparable  among  the   par0cipants  
  • 11. Tradi0onal  Methods   Test  Score   Age   Rela0on  between  age  and   test  score  es0mated  once  for   all  sites  together  
  • 12. Advantages  of  HLM   Test  Score   Age   Four  different   program  sites   Here,  the  rela0on  between  age  and  test  score  varies   across  sites.   Are  there  any  site  level  variables  associated  with  the   strength  of  this  rela0on?  
  • 13. Design  Considera0ons  for  Using  HLM   •  Sample  size   – Par0cipant  level   – Site  level   – Repeated  measures   •  Missing  Data   – Can  be  easy  or  difficult  to  deal  with   •  Number  of  variables   – Comprehensive  coverage   – Parsimony    
  • 14. Final  Thoughts     •  Applying  HLM  in  evalua0ons  with  nested  data   enables  more  robust  results  and  conclusions   •  U0liza0on-­‐focused     – Iden0fy  evidence-­‐based  success  factors  or   condi0ons  for  improving  the  program  delivery   model,  to  ul0mately  achieve  beger  program   effec0veness     – Promo0ng  the  value  in  evalua0on  (gathering  the   evidence  to  determine  the  ‘success  factors’  for   the  interven0on  to  be  effec0ve)    
  • 15. Ques0ons?     Marci  Pernica       Marci.Pernica@Ontario.ca     Ministry  of  Community  and  Social  Services     Judith  Godin     sta/s/cs@jgodin.com   Independent  Consultant