Loading…

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

Like this presentation? Why not share!

Measuring Growth Toward College Readiness

on

  • 1,836 views

Measuring Growth Toward College Readiness ...

Measuring Growth Toward College Readiness
Robert Theaker, Sr. Research Associate, Michael Dahlin, Research Specialist with NWEA Kingsbury Center
Fusion 2012, the NWEA summer conference in Portland, Oregon

Researchers at the NWEA Kingsbury Center have conducted two studies to examine the predictive relationship between the RIT scales of NWEA’s MAP® assessments in reading, language usage, and mathematics to the college readiness benchmarks of the EXPLORE, PLAN, and ACT achievement tests in reading, English, and mathematics.

The objective of the first study was to identify cut scores on the MAP reading, language usage, and general mathematics tests that correspond to the published college readiness benchmarks on the EXPLORE, PLAN, and ACT assessments (ACT, 2010). A secondary objective was to create a series of probability tables that estimate the likelihood of meeting the designated college readiness benchmark, given an observed MAP score.

The second study expanded the research toward accurately estimating college readiness benchmarks to include elementary and middle grades in mathematics and reading. Benchmark scores established by ACT® (22 in mathematics and 21 in reading) were used as score criteria. The MAP® assessments were used in order to connect third through 11th grade student growth with college readiness. A large, multistate sample allowed for the extension of the authors previous analyses. The use of 11th grade MAP and ACT score correlations to backward map benchmark scores resulted in predictive accuracy beginning in grade 3.


Learning outcome:
- The results of these two studies whose final goal is predicting college readiness

Audience:
- Experienced data user
- District leadership
- Curriculum and Instruction

Statistics

Views

Total Views
1,836
Views on SlideShare
1,836
Embed Views
0

Actions

Likes
2
Downloads
48
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Measuring Growth Toward College Readiness Measuring Growth Toward College Readiness Presentation Transcript

  • Measuring Growth Toward College Readiness Two Studies of Alignment of the RIT Scales of NWEA’s MAP® Assessments with the College Readiness Benchmarks of EXPLORE®, PLAN®, and ACT® Michael Dahlin Robert Theaker Fusion 2012
  • Agenda• Organizational Background• Objectives of Study – NWEA Linking – Theaker & Johnson• Methods• Findings• Uses
  • Methods: Sample (cont)• Active NWEA districts that use EXPLORE, PLAN, and ACT were recruited.• ACT data was matched to corresponding MAP data at the individual level• Total number of matched record pairs 108,000• No formal sampling strategies employed other than to cut extreme residuals
  • Methods: Sample Test Grade Unique Unique Unique Unique State District School Student Count Count Count CountACT 11 3 9 36 3680Explore 8 3 5 51 11822Plan 10 3 7 54 13915Total Matched Record Pairs 108,000
  • 15 Models Explored in Total• MAP Reading to EXPLORE/PLAN/ACT English• MAP Reading to EXPLORE/PLAN/ACT Reading• MAP Language Usage to EXPLORE/PLAN/ACT English• MAP Language Usage to EXPLORE/PLAN/ACT Reading• MAP Math to EXPLORE/PLAN/ACT Math
  • Model Selection Process• Explored several classes of models – Linear OLS – Nonlinear OLS – Linear random effects (HLM) – Non linear random effects (“HNLM”?)• Examined fit statistics (AIC, BIC) for each• Cut standardized residuals >2.0 and refit• Overall best fit models were quadratic OLS: 2 y Ax Bx C
  • Sample Scatterplot:ACT Math from MAP Math
  • Estimated Math Cut Scores MAP Mathematics RIT Score as Predictor – Same Season Cut Scores and Normative Percentile Ranks on MAP Corresponding to College Readiness BenchmarksGra Mathematics College MAP Cut MAPde Readiness Test Score Normative Benchmark Percentile Rank8 EXPLORE Math 17 245 7210 PLAN Math 19 251 7711 ACT Math 22 258 84
  • Variance Accounted for by Quadratic Regression Model MAP Mathematics Test as PredictorGrade College Readiness Test Correlations Percent of Variance (R) Accounted for (r2)8 EXPLORE Mathematics .825 68.0%10 PLAN Mathematics .802 64.3%11 ACT Mathematics .870 75.7% SSE Note : R 1 SST
  • Benchmark Prediction Accuracy MAP Mathematics as Predictor Percentage of Sample whose College Readiness Status was Accurately Predicted by MAP ScoreGrade College Sample Size Percentage Percentage Percentage Readiness Correctly of False of False Test Predicted Positives Negatives8 EXPLORE 12753 82% 4% 14% Mathematics10 PLAN 9516 86% 4% 9% Mathematics11 ACT 2948 91% 1% 7% Mathematics
  • Probability Calculations MAP Reading to ACT Reading100%90%80%70%60%50%40%30%20%10% 0% 145 165 185 205 225 245 265 Empirical Probabilities Model-based Probabilities Adjusted Empirical
  • Probability Tables MAP Reading RIT Score as Predictor Reading Benchmark English BenchmarkMAP Reading EXPLORE PLAN ACT EXPLORE PLAN ACTRIT Range 8th Grade 10th Grade 11th Grade 8th Grade 10th Grade 11th Grade145 0% 0% 0% 0% 0% 0%150 0% 0% 0% 0% 0% 0%155 0% 0% 0% 0% 0% 0%160 0% 0% 0% 0% 0% 0%165 0% 0% 0% 0% 0% 0%170 1% 0% 0% 0% 0% 0%175 1% 0% 0% 1% 0% 0%180 0% 0% 0% 1% 0% 0%185 0% 1% 0% 1% 2% 0%190 0% 1% 0% 4% 2% 0%195 2% 1% 0% 5% 7% 0%200 2% 2% 0% 10% 10% 1%205 2% 2% 1% 14% 12% 6%210 6% 6% 3% 25% 18% 11%215 14% 10% 6% 39% 30% 22%220 27% 17% 10% 61% 44% 39%225 45% 30% 16% 77% 60% 56%230 64% 44% 32% 91% 76% 79%235 82% 66% 54% 99% 90% 93%240 96% 82% 80% 100% 96% 99%245 100% 92% 93% 100% 100% 100%250 100% 100% 96% 100% 100% 100%255 100% 100% 100% 100% 100% 100%260 100% 100% 100% 100% 100% 100%265 100% 100% 100% 100% 100% 100%270 100% 100% 100% 100% 100% 100%275 100% 100% 100% 100% 100% 100%280 100% 100% 100% 100% 100% 100%285 100% 100% 100% 100% 100% 100%290 100% 100% 100% 100% 100% 100%295 100% 100% 100% 100% 100% 100%300 100% 100% 100% 100% 100% 100%
  • Estimated Reading Cut Scores MAP Reading RIT Score as Predictor – Same Season Cut Scores and Normative Percentile Ranks on MAP Corresponding to College Readiness BenchmarksGrade Reading MAP MAP English MAP MAP College Cut Normed College Cut Normed Bench Bench Readiness Score Percentile Readiness Score Percentile mark mark Test Rank Test Rank8 EXPLORE 15 230 70 EXPLORE 13 220 44 Reading English10 PLAN 17 234 73 PLAN 15 227 58 Reading English11 ACT 21 237 78 ACT English 18 232 68 Reading
  • Estimated Language Usage Cut Scores MAP Language Usage RIT Score as Predictor – Same Season Cut Scores and Normative Percentile Ranks on MAP Corresponding to College Readiness BenchmarksGrade Reading MAP MAP English MAP MAP College Cut Normed College Cut Normed Bench Bench Readiness Score Percentile Readiness Score Percentile mark mark Test Rank Test Rank8 EXPLORE 15 229 72 EXPLORE 13 219 43 Reading English10 PLAN 17 232 73 PLAN 15 225 56 Reading English11 ACT 21 234 75 ACT English 18 228 62 Reading
  • SECTION Title Here in Archer Bold Subtitle here if needed in Arial Regular
  • Theaker and Johnson CR Benchmark Methods: Sample• ACT data was matched to corresponding MAP data at the individual level• Sample contained over 201,000 matched pairs of scores• No formal sampling strategies employed other than to cut extreme residuals• One Model vs. 15 was employed• Utilized a quadratic regression mode.
  • College Readiness Benchmark Model StatisticsTable 3 Correlations Between MAP Benchmarks and 11th Grade ACT Performance Mathematics Fall Spring Fall Spring Grade n r n r n r n r3 469 0.75 483 0.71 473 0.73 495 0.734 915 0.74 1,046 0.75 923 0.73 1,057 0.725 1,534 0.78 1,766 0.77 1,528 0.72 1,770 0.746 2,046 0.79 2,434 0.81 2,082 0.75 2,425 0.777 4,081 0.84 4,446 0.84 4,328 0.78 4,332 0.798 5,779 0.86 6,048 0.85 5,980 0.78 6,021 0.779 5,990 0.89 7,536 0.87 6,019 0.80 7,422 0.7810 5,727 0.91 6,502 0.88 5,917 0.79 7,012 0.7811 2,362 0.89 2,468 0.86 2,386 0.77 2,359 0.77Note. All coefficients are significant at p < .001.
  • College Readiness Benchmarkby Grade Correct Prediction % Table 4 Quadratic Regression Model Fit Statistics for Mathematics and Reading Scores Grade level Correct prediction-Math Correct prediction-Reading Fall testing season 3 82% 78% 4 80% 79% 5 81% 77% 6 80% 77% 7 81% 79% 8 83% 79% 9 86% 81% 10 87% 81% 11 91% 82% Correct prediction-Math Correct prediction-Reading Spring testing season 3 79% 78% 4 81% 79% 5 81% 79% 6 82% 80% 7 82% 80% 8 83% 80% 9 83% 79% 10 85% 80% 11 90% 84% Note. All Pearson’s chi-squared tests were significant at p < .001.
  • CR Benchmark Percentile Rankby Grade 90 NWEA Percentile Ranking of College Readiness Benchmarks MAP Scores Grades 3-12 80 70 60Percenitle 50 40 30 20 10 0 3f 3s 4f 4s 5f 5s 6f 6s 7f 7s 8f 8s 9f 9s 10f 10s 11f 11s Grade and Season Math Reading
  • College Readiness Benchmark RIT Score by Grade 280 College Readiness Benchmarks for MAP Scores, Grades 3-12 260 258 254 251 249 255 244 250 242MAP RIT Scale Scores 243 246 240 236 238 235 238 231 233 235 237 228 233 225 230 224 225 228 221 220 224 216 220 216 213 215 209 204 208 200 199 180 3f 3s 4f 4s 5f 5s 6f 6s 7f 7s 8f 8s 9f 9s 10f 10s 11f 11s Grade and Testing Season Math Reading
  • College Readiness Benchmarksby Institution Type Reading Mathematics Open State Top public Open State Top public Ivy League Ivy LeagueGrade & Testing Season enrollmen universitie universitie enrollmen universitie universitie = ACT 32 = ACT 32 t = ACT 16 s = ACT 24 s = ACT 29 t = ACT 16 s = ACT 24 s = ACT 29Grade 3 Fall 179 205 216 221 179 207 217 222Grade 3 Spring 189 213 224 229 188 220 234 241Grade 4 Fall 190 214 225 231 189 217 228 234Grade 4 Spring 197 221 231 236 197 227 240 246Grade 5 Fall 196 220 231 236 197 228 241 247Grade 5 Spring 204 226 235 240 207 240 253 260Grade 6 Fall 203 226 236 241 205 239 252 258Grade 6 Spring 209 230 239 244 214 246 258 265Grade 7 Fall 207 229 239 244 211 242 254 261Grade 7 Spring 212 232 241 246 217 248 260 267Grade 8 Fall 212 232 241 246 216 247 259 266Grade 8 Spring 215 235 244 249 221 253 265 272Grade 9 Fall 214 234 244 249 220 250 262 268Grade 9 Spring 216 237 246 251 223 255 268 274Grade 10 Fall 217 237 246 251 224 253 265 272Grade 10 Spring 219 239 248 253 225 258 271 277Grade 11 Fall 220 242 252 257 228 258 270 277Grade 11 Spring 219 243 254 260 229 262 275 282Note. Values were calculated by ACT from empirical data to identify the scores of students with a 50% likelihood of achieving a B average in afreshman-level course (ACT, 2009).
  • College Readiness By InstitutionType
  • Possible Student Level College Readiness ReportStudent Name Amanda Chang Grade 6 RIT Percentile Rank Probability of Meeting ACT College Readiness BenchmarkFall 2010 226 66th 27%Spring 2011 238 78th 49% Actual Growth Growth Needed to Growth Target Met Meet ACT BenchmarkGrowth 12 10 Yes
  • A prototype graphic display of a college readiness report270 Chad Johnsons Mathematics Growth 8th Grade College Readiness Benchmark260250240 College Readiness Tranectory Johnsons Trajectory230 Johnsons Growth Normal Growth220210200 F Gr 6 S Gr 6 S Gr 7 S Gr 8
  • Questions