Upcoming SlideShare
×

# Predicting Student Performance on the MSP-HSPE: Understanding, Conducting, and Using Alignment Studies

780 views
608 views

Published on

Presented at Washington Educational Research Association (WERA) conference.

Presenters:
Highline Public Schools and Vancouver Public Schools
Sarah Johnson Sarah.Johnson@highlineschools.org
Paul Stern Paul.Stern@vansd.org

Presentation Overview:
- Background/The Value of Alignment Studies
- Highline’s Regression Study
- Multi-District Regression Study
- Conclusions
- Applying the Results

Published in: Education
0 Likes
Statistics
Notes
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

• Be the first to like this

Views
Total views
780
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
2
0
Likes
0
Embeds 0
No embeds

No notes for slide
• Hidden:Doctors use blood pressure to predict general healthBlood tests Future:Colleges use SAT &amp; GPA to predict college successInsurance companies use credit scores to predict the risk a driver will get into an accidentWe want to use the alignment results to identify which students are at risk for not meeting proficiency on the state test.
• With regard to Washington State Proficiency – like other states, we have a moving target from grade to grade. A student who is proficient one year may not be proficient the next only because the state’s performance expectation is different.10th grade is at the 26th percentile!Hypothetical 41st percentile student…
• Other half served by EvergreenUsing MAP for about 5 years
• NWEA provides the value at which a student has 50/50 odds of passing the MSP. Also, odds for each of our 4 levels… but all WE really care about is passing. Values for Spring RIT are interesting – but there’s no predictive power there…
• Fortunately there are values for Fall, but not Winter.
• The values from the prior two tables can be plotted – similar to the Highline modelNote – there’s a similar chart for Reading too.
• NWEA also provides the probability of passing for RIT scores in 5 RIT bands. This is more useful – but can be overwhelming…. And what do you do with a score of 202?
• Questions in the back of my mind…Which gives better results?Worked with friends in other districts… can’t be that hard, can it? Data sharing agreements Defining dates – what does Fall 2011 mean? Data errors – grade level reported for THIS year, not year the data reflectsI know regression methodologies and I’m suspicious of Equipercentile.Highline appeared to have better ideas for applying the informationPooled 2 years of data from 7 districts
• I tested all kinds of variables…
• I tested all kinds of variables and all provided very good correlation values.I selected Winter HIMAP because…
• Need lead time, so everything “Spring” was outMissing data was a problem. Especially if we want to use results for prediction and for many reasons, HIMAP provided the largest NI liked the logic of HIMAP – easy to not care and get an under-report. Hard to cheat to a high score.And Winter is an optional test window for most grades in Vancouver
• The Cut Scores were almost identical across the 3 methods in Reading
• … and in Math
• Then I applied Highline’s plus/minus 4 RIT to create 3 bands – same idea, but I used different names for the 3 resulting categories.Explain chart – “Of the kids that Multi-Dist analysis predicted to likely not be proficient in Reading, only 15% surprised us. Of those in the 50/50 range, 55% were proficient. Of those that we predicted to be proficient, 92% met expectations.
• So, taking this chart we looked at earlier………..
• And taking the values from this chart, we can create a teacher report….
• This is a sample teacher report I plan to distribute after the Fall window closes.
• ### Predicting Student Performance on the MSP-HSPE: Understanding, Conducting, and Using Alignment Studies

1. 1. Predicting Student Performance on the MSP-HSPEUnderstanding, Conducting, and Using Alignment StudiesHighline Public Schools andVancouver Public SchoolsPresenters:Sarah Johnson Sarah.Johnson@highlineschools.orgPaul Stern Paul.Stern@vansd.org
2. 2. Overview• Background/The Value of Alignment Studies• Highline’s Regression Study• NWEA’s Linking Study• Multi-District Regression Study• Conclusions• Applying the Results
3. 3. About the MAP Assessments• Computerized• Adaptive assessment – increases in difficulty when answers are correct and decreases in difficulty when answers are incorrect.• Rasch Units (RIT) Scale • Equal Interval • Vertical scaling • Has the same meaning regardless of grade or age of the student.• For the purposes of this presentation, we will be looking at Reading & Math only.
4. 4. Value of Alignment StudiesResearchers align scales for one of two purposes:• Use results from measure “X” to predict the value of a harder-to-observe measure or outcome “Y”.• Use results from measure “X” to predict the value of a future measure or outcome “Y”.In our case, faculty and administration are interested inidentifying students who are likely to struggle on futurestate performance measures. By intervening early, wecan target resources to students who may not meet“proficiency”.
5. 5. http://kingsburycenter.org/gallery
6. 6. About Vancouver Public Schools• About 22,000 enrolled students• 6 High Schools (4 comprehensive, 1 magnet, 1 alternative)• 18% of students speak a language other than English at home• 53% eligible for free or reduced price lunch• The district serves half of the city of Vancouver, WA (across the river from Portland)
7. 7. About Highline Public Schools• About 18,000 enrolled students• 15 High Schools (2 comprehensive, 6 small learning community, 1 magnet, 5 alternative, 1 skills center)• 43% of students speak a language other than English at home - 21% ELL.• 68% eligible for free or reduced price lunch• The district serves neighborhoods of White Center, Burien, Des Moines, SeaTac and Normandy Park just south of Seattle.
8. 8. Highline’s Regression Study• In 2007, School and District Administration had been requesting ways to interpret student MAP scores in context of (then) WASL testing. One concern in particular was that students had been above average on the national norms, but yet were not meeting standard on the state assessment.• School staff also requested a way to quickly identify if a student was on track or not.
9. 9. Highline’s Regression Study• Decided to do a regression analysis to predict WASL performance.• Ran correlations on multiple variables, and found that “HiMap” (max of last 3 test administrations) had a higher correlation with WASL than a single MAP score. • Weeds out test “bombs” and missing data
10. 10. “HIMAP” Variable Defined Fall Winter SPRING HIMAP HIMAP HIMAP 5th Grade 5th Grade 5th Grade
11. 11. Highline’s Regression Study• Rather than make a straight out prediction of whether a student will meet/not meet standard, we wanted to emphasize the possible predictive error.• Decided to find a cut on the MAP assessment to predict 400 on WASL, and then generate an error band around that where students would be considered “too close to call”• Used 4 points as a generous estimate of the standard error of the assessment (usually between 3-3.5)
12. 12. Intervention Categories: 3 “Bands”• “Above Benchmark” students were those who performed more than 4 RIT points above the cut score. These students are considered on track to meet standard.• “Strategic” students were those who performed within 4 RIT points of the cut. These students are “too close to call” and should receive strategic intervention to meet standard.• “Intensive” students were those who performed more than 4 RIT points below the cut score. These students are unlikely to meet standard without intensive intervention.
13. 13. Cuts for Fall, Winter and Spring• When the study was first done in 2008, regression analyses were performed using Spring MAP scores and WASL.• Growth norms were utilized to back track to get cuts for Fall and Winter• Cut scores and ranges were disseminated to teachers and administrators, along with an explanation of the scores.• Excel files for schools began including MAP scores, along with each students’ “BSI Indicator”, color coded in Red, Yellow and Green.
14. 14. Predictive Validity• When a student’s indicator is compared to their actual performance: • Approximately 90% of students identified as “Above Benchmark” actually met standard. • Approximately 50% of students identified as “Strategic” actually met standard. • Approximately 10% of students identified as “Intensive” actually met standard.• These were generally true within about 10 percentage points
15. 15. 2010 MSP• The analysis was re-run in 2010 following the first year of transition from WASL to MSP.• During the second analysis, regressions were run on each test window individually in each grade level, finding individual cuts, rather than using growth norms.• District budget cuts made high school MAP testing optional, and therefore High School was excluded.
16. 16. 2012 MSP• Due to online testing for MSP along with other district initiatives requiring lab time, our Spring window was moved from May to March beginning in Spring 2012. Also, Winter testing became an optional window.• Therefore, cuts were created one more time in 2012 for Highline. Again, regressions were done between Fall MAP and MSP, and Spring MAP and MSP. Because Winter was optional, the cuts for Winter were determined using the 2/3 point between the Fall and Spring cuts.
17. 17. NWEA’s Linking Study • Most recently updated in Feb, 2011 • Based on a sample of 271 schools in the Spring of 2010 • NWEA uses an Equi-percentile method to equate test results
18. 18. Equipercentile Method ofAlignment • NWEA used a sample of students from 271 schools taking the 2010 spring assessment in WA. • For each grade and subject, identify the percentage of students in the study sample that met standard. • For each grade and subject, identify the RIT associated with the equivalent percentile from within the study sample.“If 40% of the study population in grade 3 mathperformed below the proficient level on the statetest, we would find the RIT score that would beequivalent to the 40th percentile for the studypopulation”
19. 19. Multi-District Regression Study• Included 7 districts including Seattle, Bellingham, Vancouver, Highline, Sumn er, Auburn, and Clover Park• Data covered the 2009-10 and 2010-11 academic years• The “cut score” for proficiency was consistent across both years at each grade level, so data from both years was pooled• Overall N of approximately 80,000
20. 20. Independent Variables Created• Math Spring RIT (Winter and Fall as well)• Math Spring HIMAP (Winter and Fall as well)• Combined Spring HIMAP (sum of Read & Math) (Winter and Fall as well)• Math Winter HIMAP + Math MSP• Math Fall HIMAP + Math MSP(Comparable variables were also created for Reading)
21. 21. Quality of CorrelationBest: (Corr: 0.78) • Spring RIT (but no predictive value, so Spring indicators will be ignored)Next Best: (Corr: 0.73-0.75) • Winter RIT • Winter HIMAP + MSP scale score (275-500) • Winter HIMAPThird Best: (Corr: 0.70) • Read Winter HIMAP + Math Winter HIMAP
22. 22. Rationale for Selecting Winter HIMAP• Spring MAP test window overlaps MSP/HSPE test window.• Prior Year MSP scores not available for grades 3 and 10.• New students in district are missing MSP scores.• Not all students perform to their best ability on every test.• Many students do not take the Winter MAP.
23. 23. Rationale for Selecting Winter HIMAPWinter HIMAP …• Is not very different in the quality of the correlation as compared to other options,• Maximizes the number of students for whom it can be applied, and• Is relatively easy to explain
24. 24. Predictive Validity: Percent ofStudents Meeting Standard by Band Multi- 10%-20% ofREADING District NWEA Highline “Likely NotLikely Not Proficient 15% 17% 20% Proficient” Students MetAt Risk 55% 59% 65% Standard.Likely Proficient 92% 93% 94% 50%-65% of “At Risk Students Met Multi- Standard.MATH District NWEA Highline 90%-95% ofLikely Not Proficient 10% 14% 14% “Likely Proficient”At Risk 50% 62% 61% Students MetLikely Proficient 92% 95% 95% Standard.
25. 25. Pro and Con of Do-It-YourselfPro: • Data are based on “our kids” (this is an emotional argument, not a statistical one). • Winter and prior spring estimates can be computed rather than estimated.Con:• It is a lot of work.• Controlling for test windows is complex.• NWEA results are very similar to DIY results.• Teachers who encounter the NWEA linking study will be confused about why our cut points are different.• … and did I say it was a lot of work?
26. 26. Read Met MSP MSP Math Met MSP MSP Growth Read 2013 2013 Read Growth Math 2013 2013 Math LAST FIRST Target Read Fall Read Read Spring Target Math Fall Math Math SpringID# NAME NAME Grade 2012 Fall RIT Pctile Categ. Odds Target 2012 Fall RIT Pctile Categ. Odds Target10944 5 Yes 202 34 46% 207 No 196 12 17% 20313455 5 No 202 34 46% 207 Yes 208 40 36% 21613980 5 Yes 215 73 75% 219 No 228 88 81% 23517713 5 No 217 78 80% 221 No 215 61 52% 22317716 5 No 192 14 24% 199 Yes 184 3 5% 19217719 5 No 206 45 54% 211 No 204 29 27% 21217728 5 Yes 211 61 66% 215 Yes 208 40 36% 21617732 5 Yes 213 67 70% 217 Yes 208 40 36% 21617736 5 Yes 216 76 78% 220 Yes 227 87 80% 23417804 5 Yes 203 36 48% 208 Yes 217 66 56% 22418312 5 Yes 205 42 52% 210 Yes 201 21 22% 20818328 5 Yes 212 64 68% 216 Yes 202 24 24% 21018578 5 No 203 36 48% 208 Yes 201 21 22% 20818624 5 Yes 216 76 78% 220 Yes 206 34 32% 21419057 5 No 225 93 89% 228 Yes 212 52 46% 22019128 5 Yes 217 78 80% 221 Yes 222 78 68% 22921036 5 No 176 2 5% 18624125 5 No 215 73 75% 219 Yes 210 46 40% 21826414 5 No 194 17 27% 201 Yes 209 43 38% 21727807 5 No 180 4 8% 189 Yes 185 3 5% 19330737 5 No 205 42 52% 210 No 209 43 38% 21736075 5 No 201 31 43% 206 No 185 3 5% 19336376 5 Yes 171 1 3% 182 191 7 9% 19941166 5 No 184 6 12% 192 No 197 14 18% 20443584 5 No 230 97 93% 233 Yes 224 82 72% 23146978 5 Yes 197 22 34% 203 Yes 197 14 18% 204