Technical report page 558 560

2,041 views
1,934 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,041
On SlideShare
0
From Embeds
0
Number of Embeds
25
Actions
Shares
0
Downloads
11
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Technical report page 558 560

  1. 1. California Department of Education Statewide Assessment Division California Standards Tests Technical Report Spring 2009 Administration Submitted March 1, 2010 Educational Testing Service Contract No. 5417
  2. 2. STAR ProgramTable of Contents (Summarized)Chapter 1: Introduction......................................................................................................................................... 1 Background ........................................................................................................................................................................... 1 Test Purpose.......................................................................................................................................................................... 1 Test Content .......................................................................................................................................................................... 2 Intended Population .............................................................................................................................................................. 2 Intended Use and Purpose of Test Scores.......................................................................................................................... 3 Testing Window ..................................................................................................................................................................... 3 Significant STAR Developments in 2009 ............................................................................................................................. 4 Limitations of the Assessment ............................................................................................................................................ 4 Groups and Organizations Involved with the STAR Program ........................................................................................... 5 Overview of the Technical Report ........................................................................................................................................ 5 References ............................................................................................................................................................................. 7Chapter 2: An Overview of CST Processes ........................................................................................................ 8 Item Development ................................................................................................................................................................. 8 Test Variations, Accommodations, and Modifications .................................................................................................... 11 Scores .................................................................................................................................................................................. 12 References ........................................................................................................................................................................... 20Chapter 3: Item Development........................................................................................................................... 196 Rules for Item Development ............................................................................................................................................. 196 Selection of Item Writers .................................................................................................................................................. 198 Item Review Process ......................................................................................................................................................... 199 Field Testing ...................................................................................................................................................................... 203 Data Review Meetings ....................................................................................................................................................... 205 Item Banking ...................................................................................................................................................................... 206 References ......................................................................................................................................................................... 207Chapter 4: Test Assembly ................................................................................................................................ 208 Test Length ........................................................................................................................................................................ 208 Rules for Item Selection.................................................................................................................................................... 208 Reference ........................................................................................................................................................................... 213 Appendix 4.B—Cluster Targets ........................................................................................................................................ 223Chapter 5: Test Administration ........................................................................................................................ 253 Test Security and Confidentiality ..................................................................................................................................... 253 Procedures to Maintain Standardization ......................................................................................................................... 258 Accommodations for Students with Disabilities ............................................................................................................ 260 Demographic Data Corrections ........................................................................................................................................ 261 Testing Irregularities ......................................................................................................................................................... 261 Test Administration Incidents .......................................................................................................................................... 261 References ......................................................................................................................................................................... 262Chapter 6: Performance Standards ................................................................................................................. 263 Background ....................................................................................................................................................................... 263 Standard Setting Procedure ............................................................................................................................................. 263 Results ............................................................................................................................................................................... 265 References ......................................................................................................................................................................... 268Chapter 7: Scoring and Reporting ................................................................................................................... 269 Procedures for Maintaining and Retrieving Individual Scores ...................................................................................... 269 Types of Scores and Subscores ...................................................................................................................................... 270 Score Verification Procedures ......................................................................................................................................... 271 Overview of Score Aggregation Procedures .................................................................................................................. 274 Reports to be Produced and Scores for Each Report .................................................................................................... 278 Criteria for Interpreting Test Scores ................................................................................................................................ 280 Criteria for Interpreting Score Reports ............................................................................................................................ 280 Reference ........................................................................................................................................................................... 281Chapter 8: Analyses .......................................................................................................................................... 372 Samples Used for the Analyses ....................................................................................................................................... 372 Classical Item Analyses .................................................................................................................................................... 373 Reliability Analyses ........................................................................................................................................................... 375 Decision Classification Analyses .................................................................................................................................... 379 Validity Evidence ............................................................................................................................................................... 382 Page i
  3. 3. STAR Program IRT Analyses ...................................................................................................................................................................... 394 DIF Analyses ...................................................................................................................................................................... 399 References ......................................................................................................................................................................... 401Chapter 9: Quality Control Procedures ...........................................................................................................609 Quality Control of Item Development .............................................................................................................................. 609 Quality Control of the Item Bank ...................................................................................................................................... 611 Quality Control of Test Materials ..................................................................................................................................... 611 Quality Control of Scanning ............................................................................................................................................. 612 Quality Control of Image Editing ...................................................................................................................................... 613 Quality Control of Answer Document Processing and Scoring .................................................................................... 613 Quality Control of Psychometric Processes ................................................................................................................... 615 Quality Control of Reporting ............................................................................................................................................ 617Chapter 10: Historical Comparisons ............................................................................................................... 618 Examinee Performance ..................................................................................................................................................... 619 Test Characteristics .......................................................................................................................................................... 620 References ......................................................................................................................................................................... 621 Page ii
  4. 4. STAR ProgramTable of Contents (Full)Acronyms and Initialisms Used in the CST Technical Report.............................................................................. xivChapter 1: Introduction......................................................................................................................................... 1 Background ........................................................................................................................................................................... 1 Test Purpose.......................................................................................................................................................................... 1 Test Content .......................................................................................................................................................................... 2 Intended Population .............................................................................................................................................................. 2 Intended Use and Purpose of Test Scores.......................................................................................................................... 3 Testing Window ..................................................................................................................................................................... 3 Significant STAR Developments in 2009 ............................................................................................................................. 4 Elimination of California Achievement Tests, Sixth Edition Survey (CAT/6 Survey) .............................................................. 4 Release of CST Exemplars .................................................................................................................................................... 4 Limitations of the Assessment ............................................................................................................................................ 4 Score Interpretation ............................................................................................................................................................... 4 Out-of-Level Testing .............................................................................................................................................................. 4 Score Comparison ................................................................................................................................................................. 4 Groups and Organizations Involved with the STAR Program ........................................................................................... 5 State Board of Education (SBE)............................................................................................................................................. 5 California Department of Education (CDE) ............................................................................................................................ 5 Contractors ............................................................................................................................................................................ 5 Overview of the Technical Report ........................................................................................................................................ 5 References ............................................................................................................................................................................. 7Chapter 2: An Overview of CST Processes ........................................................................................................ 8 Item Development ................................................................................................................................................................. 8 Item Formats .......................................................................................................................................................................... 8 Item Development Specifications ........................................................................................................................................... 8 Item Banking .......................................................................................................................................................................... 8 Item Refresh Rate and Released Test Questions (RTQ) ....................................................................................................... 9 Test Length ............................................................................................................................................................................ 9 Test Blueprint ......................................................................................................................................................................... 9 Content Rules and Item Selection.......................................................................................................................................... 9 Psychometric Criteria ........................................................................................................................................................... 10 Test Security and Confidentiality.......................................................................................................................................... 10 Procedures to Maintain Standardization .............................................................................................................................. 11 Test Variations, Accommodations, and Modifications .................................................................................................... 11 Test Variations, Accommodation and Modification Summaries ........................................................................................... 12 Scores .................................................................................................................................................................................. 12 Individual Scores.................................................................................................................................................................. 13 Group Scores ....................................................................................................................................................................... 13 Calibration ............................................................................................................................................................................ 14 Scaling ................................................................................................................................................................................. 15 True Score Equating ............................................................................................................................................................ 15 Equating Samples ................................................................................................................................................................ 17 Equating the Braille Versions of the CSTs ........................................................................................................................... 18 Scaling the CSTs for Integrated/Coordinated Science ......................................................................................................... 18 References ........................................................................................................................................................................... 20 Appendix 2.A—CST Item and Estimated Time Chart ....................................................................................................... 21 Appendix 2.B—Reporting Clusters .................................................................................................................................... 22 English–Language Arts ........................................................................................................................................................ 22 History–Social Science ........................................................................................................................................................ 24 Mathematics......................................................................................................................................................................... 24 Science ................................................................................................................................................................................ 27 Appendix 2.C—2009 Test Variations, Accommodations, and Modifications ................................................................. 32 Appendix 2.D—Special Services Summary Tables .......................................................................................................... 35Chapter 3: Item Development........................................................................................................................... 196 Rules for Item Development ............................................................................................................................................. 196 Item Development Specifications ....................................................................................................................................... 196 Expected Item Ratio ........................................................................................................................................................... 197 Selection of Item Writers .................................................................................................................................................. 198 Criteria for Selecting Item Writers ...................................................................................................................................... 198 Item Writer Training ........................................................................................................................................................... 198 Page iii
  5. 5. STAR Program Item Review Process ......................................................................................................................................................... 199 Contractor Review ............................................................................................................................................................. 199 Content Expert Reviews .................................................................................................................................................... 200 Statewide Pupil Assessment Review (SPAR) Panel .......................................................................................................... 203 Field Testing ...................................................................................................................................................................... 203 Stand-Alone Field Testing .................................................................................................................................................. 203 Embedded Field-Test Items ............................................................................................................................................... 204 Data Review Meetings ....................................................................................................................................................... 205 Item Banking ...................................................................................................................................................................... 206 References ......................................................................................................................................................................... 207Chapter 4: Test Assembly ................................................................................................................................ 208 Test Length ........................................................................................................................................................................ 208 Rules for Item Selection.................................................................................................................................................... 208 Test Blueprint ..................................................................................................................................................................... 208 Content Rules and Item Selection...................................................................................................................................... 208 Psychometric Criteria ......................................................................................................................................................... 209 Projected Psychometric Properties of the Assembled Tests .............................................................................................. 211 Rules for Item Sequence and Layout ................................................................................................................................. 212 Reference ........................................................................................................................................................................... 213 Appendix 4.A—Technical Characteristics....................................................................................................................... 214 Appendix 4.B—Cluster Targets ........................................................................................................................................ 223Chapter 5: Test Administration ........................................................................................................................ 253 Test Security and Confidentiality ..................................................................................................................................... 253 ETS’s Office of Testing Integrity (OTI) ............................................................................................................................... 253 Test Development .............................................................................................................................................................. 253 Item Review ....................................................................................................................................................................... 253 Item Bank ........................................................................................................................................................................... 254 Transfer of Forms and Items to the CDE ........................................................................................................................... 254 Firewall .............................................................................................................................................................................. 254 Printing............................................................................................................................................................................... 255 Test Administration ............................................................................................................................................................ 255 Test Delivery ...................................................................................................................................................................... 255 Processing and Scoring ..................................................................................................................................................... 256 Data Management ............................................................................................................................................................. 256 Transfer of Scores via Secure Data Exchange .................................................................................................................. 257 Statistical Analysis ............................................................................................................................................................. 257 Reporting and Posting Results........................................................................................................................................... 257 Student Confidentiality ....................................................................................................................................................... 257 Test Results ....................................................................................................................................................................... 257 Procedures to Maintain Standardization ......................................................................................................................... 258 Test Administrators ............................................................................................................................................................ 258 Directions for Administration (DFA) .................................................................................................................................... 259 District and Test Site Coordinator Manual ......................................................................................................................... 259 STAR Management System Manuals ................................................................................................................................ 259 Test Booklets ..................................................................................................................................................................... 260 Accommodations for Students with Disabilities ............................................................................................................ 260 Identification ....................................................................................................................................................................... 260 Scoring............................................................................................................................................................................... 261 Demographic Data Corrections ........................................................................................................................................ 261 Testing Irregularities ......................................................................................................................................................... 261 Test Administration Incidents .......................................................................................................................................... 261 References ......................................................................................................................................................................... 262Chapter 6: Performance Standards ................................................................................................................. 263 Background ....................................................................................................................................................................... 263 Standard Setting Procedure ............................................................................................................................................. 263 Standard Setting Methodologies ........................................................................................................................................ 264 Results ............................................................................................................................................................................... 265 References ......................................................................................................................................................................... 268Chapter 7: Scoring and Reporting ................................................................................................................... 269 Procedures for Maintaining and Retrieving Individual Scores ...................................................................................... 269 Scoring and Reporting Specifications ................................................................................................................................ 269 Scanning and Scoring ........................................................................................................................................................ 270 Types of Scores and Subscores ...................................................................................................................................... 270 Raw Score ......................................................................................................................................................................... 270 Page iv
  6. 6. STAR Program Subscore............................................................................................................................................................................ 270 Scale Score........................................................................................................................................................................ 270 Performance Levels ........................................................................................................................................................... 270 Writing Score ..................................................................................................................................................................... 271 Score Verification Procedures ......................................................................................................................................... 271 Scoring Key Verification Process ....................................................................................................................................... 271 Monitoring and Quality Control of Writing Scoring ............................................................................................................. 272 Score Verification Process ................................................................................................................................................. 274 Overview of Score Aggregation Procedures .................................................................................................................. 274 Individual Scores................................................................................................................................................................ 274 Reports to be Produced and Scores for Each Report .................................................................................................... 278 Types of Score Reports ..................................................................................................................................................... 278 Score Report Contents ...................................................................................................................................................... 278 Score Report Applications.................................................................................................................................................. 279 Criteria for Interpreting Test Scores ................................................................................................................................ 280 Criteria for Interpreting Score Reports ............................................................................................................................ 280 Reference ........................................................................................................................................................................... 281 Appendix 7.A—ELA for Writing (Grades Four and Seven) Rubrics .............................................................................. 282 Grade Four Scoring Rubric ................................................................................................................................................ 282 Grade Seven Scoring Rubric ............................................................................................................................................. 285 Appendix 7.B—Scale Score Distributions....................................................................................................................... 288 Appendix 7.C—Demographic Summaries ....................................................................................................................... 292 Appendix 7.D—Types of Score Reports .......................................................................................................................... 368Chapter 8: Analyses .......................................................................................................................................... 372 Samples Used for the Analyses ....................................................................................................................................... 372 Classical Item Analyses .................................................................................................................................................... 373 Multiple-Choice Items ........................................................................................................................................................ 373 Writing Tasks ..................................................................................................................................................................... 374 Reliability Analyses ........................................................................................................................................................... 375 Intercorrelations, Reliabilities, and SEMs for Reporting Clusters ....................................................................................... 377 Subgroup Reliabilities and SEMs ....................................................................................................................................... 377 Conditional Standard Errors of Measurement .................................................................................................................... 377 Decision Classification Analyses .................................................................................................................................... 379 Writing Score Reliability ..................................................................................................................................................... 380 Prompt and Rater Agreement Summary ............................................................................................................................ 381 Validity Evidence ............................................................................................................................................................... 382 Purpose of the CST ........................................................................................................................................................... 382 The Constructs to Be Measured ........................................................................................................................................ 382 The Scores Generated and the Interpretations and Uses of these Scores ........................................................................ 383 Intended Test Population(s) ............................................................................................................................................... 383 Validity Evidence Collected ................................................................................................................................................ 384 Evidence Based on Response Processes ......................................................................................................................... 391 Evidence Based on Internal Structure ................................................................................................................................ 392 Evidence Based on Consequences of Testing................................................................................................................... 394 IRT Analyses ...................................................................................................................................................................... 394 IRT Model-Data Fit Analyses ............................................................................................................................................. 394 Model Fit Assessment Results ........................................................................................................................................... 396 Evaluation of Scaling ......................................................................................................................................................... 396 Summaries of Scaled IRT b-values.................................................................................................................................... 398 Post-Equating Results ....................................................................................................................................................... 398 DIF Analyses ...................................................................................................................................................................... 399 References ......................................................................................................................................................................... 401 Appendix 8.A—Classical Analyses .................................................................................................................................. 403 Appendix 8.B—Reliability Analyses ................................................................................................................................ 415 Appendix 8.C—Validity Analyses .................................................................................................................................... 497 Appendix 8.D—IRT Analyses ........................................................................................................................................... 541 Scaling and Post-Equating Results .................................................................................................................................... 557 Appendix 8.E—DIF Analyses............................................................................................................................................ 579Chapter 9: Quality Control Procedures ...........................................................................................................609 Quality Control of Item Development .............................................................................................................................. 609 Item Specifications ............................................................................................................................................................. 609 Item Writers........................................................................................................................................................................ 609 Internal Contractor Reviews ............................................................................................................................................... 609 Assessment Review Panel (ARP) Review ......................................................................................................................... 610 Statewide Pupil Assessment Review (SPAR) Panel Review ............................................................................................. 610 Page v
  7. 7. STAR Program Data Review of Field Tested Items .................................................................................................................................... 610 Quality Control of the Item Bank ...................................................................................................................................... 611 Quality Control of Test Materials ..................................................................................................................................... 611 Collecting Test Materials .................................................................................................................................................... 611 Processing Test Materials .................................................................................................................................................. 612 Quality Control of Scanning ............................................................................................................................................. 612 Post-scanning Edits ........................................................................................................................................................... 612 Quality Control of Image Editing ...................................................................................................................................... 613 Quality Control of Answer Document Processing and Scoring .................................................................................... 613 Accountability of Answer Documents ................................................................................................................................. 613 Processing of Answer Documents ..................................................................................................................................... 613 Scoring and Reporting Specifications ................................................................................................................................ 614 Matching Information on CST Answer Documents ............................................................................................................ 614 Matching Multiple-choice and Writing Scores for ELA Grades Four and Seven ................................................................ 614 Storing Answer Documents................................................................................................................................................ 614 Quality Control of Psychometric Processes ................................................................................................................... 615 Score Key Verification Procedures .................................................................................................................................... 615 Quality Control of Item Analyses, DIF and Equating Process ............................................................................................ 615 Score Verification Process ................................................................................................................................................. 616 Offloads to Test Development............................................................................................................................................ 616 Quality Control of Reporting ............................................................................................................................................ 617 Excluding Student Scores from Summary Reports ............................................................................................................ 617Chapter 10: Historical Comparisons ............................................................................................................... 618 Examinee Performance ..................................................................................................................................................... 619 Test Characteristics .......................................................................................................................................................... 620 References ......................................................................................................................................................................... 621 Appendix 10.A—Historical Comparisons Tables ........................................................................................................... 622 Appendix 10.B—Historical Comparisons Tables ........................................................................................................... 632Tables Table 2.1 Subgroup Definitions ........................................................................................................................................... 13 Table 2.2 Scale Scores Ranges for Performance Levels .................................................................................................... 16 Table 2.3 Sample Selected to Establish Reference Form ................................................................................................... 17 Table 2.4 Proficiency-Level Cut Points for Science End-of-Course CSTs .......................................................................... 19 Table 2.5 Computation of Proficiency-Level Cut Points on Integrated/Coordinated Science 1 ........................................... 19 Table 2.C.1 Matrix of Test Variations, Accommodations, and Modifications ........................................................................ 32 Table 2.C.2 Matrix of Test Variations for English Learners .................................................................................................. 34 Table 2.D. 1 Special Services Summary for ELA, Grades Two and Three ......................................................................... 35 Table 2.D.2 Special Services Summary for ELA, Grade Four............................................................................................. 40 Table 2.D.3 Special Services Summary for ELA, Grades Five and Six .............................................................................. 45 Table 2.D.4 Special Services Summary for ELA, Grade Seven .......................................................................................... 49 Table 2.D.5 Special Services Summary for ELA, Grades Eight and Nine ........................................................................... 54 Table 2.D.6 Special Services Summary for ELA, Grades Ten and Eleven ......................................................................... 58 Table 2.D.7 Special Services Summary for History–Social Science, Grade Eight .............................................................. 62 Table 2.D.8 Special Services Summary for World History .................................................................................................. 67 Table 2.D.9 Special Services Summary for History–Social Science, Grade Eleven ........................................................... 72 Table 2.D.10 Special Services Summary for Mathematics, Grades Two and Three........................................................... 77 Table 2.D.11 Special Services Summary for Mathematics, Grades Four and Five............................................................. 82 Table 2.D.12 Special Services Summary for Mathematics, Grades Six and Seven............................................................ 87 Table 2.D.13 Special Services Summary for General Mathematics.................................................................................... 92 Table 2.D.14 Special Services Summary for Algebra I ....................................................................................................... 97 Table 2.D.15 Special Services Summary for Geometry .................................................................................................... 102 Table 2.D.16 Special Services Summary for Algebra II .................................................................................................... 107 Table 2.D.17 Special Services Summary for Summative High School Mathematics ........................................................ 112 Table 2.D.18 Special Services Summary for Integrated Mathematics 1 ........................................................................... 118 Table 2.D.19 Special Services Summary for Integrated Mathematics 2 ........................................................................... 123 Table 2.D.20 Special Services Summary for Integrated Mathematics 3 ........................................................................... 128 Table 2.D.21 Special Services Summary for Science, Grade Five ................................................................................... 133 Table 2.D.22 Special Services Summary for Science, Grade Eight .................................................................................. 138 Table 2.D.23 Special Services Summary for Grade Ten Life Science .............................................................................. 143 Table 2.D.24 Special Services Summary for Biology ........................................................................................................ 148 Table 2.D.25 Special Services Summary for Chemistry ................................................................................................... 154 Table 2.D.26 Special Services Summary for Earth Science ............................................................................................. 160 Table 2.D.27 Special Services Summary for Physics ....................................................................................................... 166 Table 2.D.28 Special Services Summary for Integrated/Coordinated Science 1 .............................................................. 172 Page vi
  8. 8. STAR ProgramTable 2.D.29 Special Services Summary for Integrated/Coordinated Science 2 .............................................................. 178Table 2.D.37 Special Services Summary for Integrated/Coordinated Science 3 ............................................................... 184Table 2.D.30 Special Services Summary for Integrated/Coordinated Science 4 .............................................................. 190Table 3.1 Field-test Percentages for the CSTs ................................................................................................................. 198Table 3.2 CST ARP Member Qualifications, by subject and total ..................................................................................... 201Table 3.3 Summary of Items and Forms Presented in 2009 CSTs ................................................................................... 204Table 4.1 Target Statistical Specifications for the CSTs ................................................................................................... 210Table 4.A.1 Summary of 2009 CST Projected Technical Characteristics ......................................................................... 214Table 4.A.2 Summary of 2009 CST Projected Statistical Attributes .................................................................................. 215Table 6.1 Scale Scores Ranges for Performance Levels .................................................................................................. 265Table 6.2 Percentage of Examinees in Each Performance Level ..................................................................................... 266Table 7.1 Mean and Standard Deviation of Raw and Scale Scores for CSTs .................................................................. 275Table 7.2 Percentage of Examinees in Proficiency Levels for CSTs ................................................................................ 276Table 7.3 Subgroup Definitions ......................................................................................................................................... 277Table 7.B.1 Distribution of CST Scale Scores for ELA ...................................................................................................... 288Table 7.B.2 Distribution of CST Scale Scores for History–Social Science ........................................................................ 288Table 7.B.3 Distribution of CST Scale Scores for Mathematics, Grades Two through Seven .......................................... 289Table 7.B.4 Distribution of Scale Scores for Mathematics End-of-Course CSTs .............................................................. 289Table 7.B.5 Distribution of CST Scale Scores for Science ................................................................................................ 290Table 7.B.6 Distribution of CST Scale Scores for a Grade-Specific Population ................................................................ 291Table 7.C.1 Demographic Summary for ELA, Grade Two ................................................................................................ 292Table 7.C.2 Demographic Summary for ELA, Grade Three .............................................................................................. 294Table 7.C.3 Demographic Summary for ELA, Grade Four ................................................................................................ 296Table 7.C.4 Demographic Summary for ELA, Grade Five ................................................................................................ 298Table 7.C.5 Demographic Summary for ELA, Grade Six .................................................................................................. 300Table 7.C.6 Demographic Summary for ELA, Grade Seven ............................................................................................. 302Table 7.C.7 Demographic Summary for ELA, Grade Eight ............................................................................................... 304Table 7.C.8 Demographic Summary for ELA, Grade Nine ................................................................................................ 306Table 7.C.9 Demographic Summary for ELA, Grade Ten ................................................................................................. 308Table 7.C.10 Demographic Summary for ELA, Grade Eleven .......................................................................................... 310Table 7.C.11 Demographic Summary for History–Social Science, Grade Eight ............................................................... 312Table 7.C.12 Demographic Summary for World History ................................................................................................... 314Table 7.C.13 Demographic Summary for History–Social Science, Grade Eleven ............................................................ 316Table 7.C.14 Demographic Summary for Mathematics, Grade Two ................................................................................. 318Table 7.C.15 Demographic Summary for Mathematics, Grade Three .............................................................................. 320Table 7.C.16 Demographic Summary for Mathematics, Grade Four ................................................................................ 322Table 7.C.17 Demographic Summary for Mathematics, Grade Five ................................................................................. 324Table 7.C.18 Demographic Summary for Mathematics, Grade Six................................................................................... 326Table 7.C.19 Demographic Summary for Mathematics, Grade Seven ............................................................................. 328Table 7.C.20 Demographic Summary for General Mathematics ....................................................................................... 330Table 7.C.21 Demographic Summary for Algebra I .......................................................................................................... 332Table 7.C.22 Demographic Summary for Geometry ......................................................................................................... 334Table 7.C.23 Demographic Summary for Algebra II ......................................................................................................... 336Table 7.C.24 Demographic Summary for Summative High School Mathematics ............................................................. 338Table 7.C.25 Demographic Summary for Integrated Mathematics 1 ................................................................................ 340Table 7.C.26 Demographic Summary for Integrated Mathematics 2 ................................................................................ 342Table 7.C.27 Demographic Summary for Integrated Mathematics 3 ................................................................................ 344Table 7.C.28 Demographic Summary for Science, Grade Five ........................................................................................ 346Table 7.C.29 Demographic Summary for Science, Grade Eight ....................................................................................... 348Table 7.C.30 Demographic Summary for Grade Ten Life Science ................................................................................... 350Table 7.C.31 Demographic Summary for Biology ............................................................................................................. 352Table 7.C.32 Demographic Summary for Chemistry ......................................................................................................... 354Table 7.C.33 Demographic Summary for Earth Science .................................................................................................. 356Table 7.C.34 Demographic Summary for Physics ............................................................................................................ 358Table 7.C.35 Demographic Summary for Integrated/Coordinated Science 1 ................................................................... 360Table 7.C.36 Demographic Summary for Integrated/Coordinated Science 2 ................................................................... 362Table 7.C.37 Demographic Summary for Integrated/Coordinated Science 3 ................................................................... 364Table 7.C.38 Demographic Summary for Integrated/Coordinated Science 4 ................................................................... 366Table 7.D.1 Score Reports Reflecting CST Results .......................................................................................................... 368Table 8.1 Mean and Median Proportion Correct and Point-Biserial .................................................................................. 374Table 8.2 Reliabilities and Standard Errors of Measurement for the CSTs ....................................................................... 376Table 8.3 Scale Score CSEM at Cut Points ...................................................................................................................... 378Table 8.4 2009 CST Content Area Correlations (All Valid Scores) ................................................................................... 388Table 8.5 Evaluation of Common Items between New and Reference Test Forms .......................................................... 397Table 8.A.1 Item-by-item p-value and Point-Biserial for ELA ............................................................................................ 403Table 8.A.2 Item-by-item p-value and Point-Biserial for History–Social Science .............................................................. 405 Page vii
  9. 9. STAR Program Table 8.A.3 Item-by-item p-value and Point-Biserial for Mathematics ............................................................................... 406 Table 8.A.4 Item-by-item p-value and Point-Biserial for Mathematics [continued] ............................................................ 407 Table 8.A.5 Item-by-item p-value and Point-Biserial for Science ...................................................................................... 408 Table 8.A.6 Item-by-item p-value and Point-Biserial for Science [continued] .................................................................... 409 Table 8.A.7 Item-by-item p-value and Point-Biserial for Grade-Specific CSTs ................................................................. 411 Table 8.A.8 Distribution of Essay Scores for ELA (Grade Four)—Overall and by Subgroup (all %) ................................. 412 Table 8.A.9 Distribution of Essay Scores for ELA (Grade Seven)—Overall and by Subgroup (all %) .............................. 412 Table 8.A.10 Mean Scores for ELA (Grade Four Essay)—Overall and by Subgroup ....................................................... 413 Table 8.A.11 Mean Scores for ELA (Grade Seven Essay)—Overall and by Subgroup .................................................... 413 Table 8.A.12 Effect Sizes for ELA (Grades Four and Seven Essay)—by Subgroup ......................................................... 414 Table 8.B.1 Reliabilities and Correlations for ELA ............................................................................................................ 415 Table 8.B.2 Reliabilities and Correlations for History–Social Science ............................................................................... 416 Table 8.B.3 Subscore Reliabilities and Correlations for Mathematics ............................................................................... 417 Table 8.B.4 Subscore Reliabilities and Correlations for Science ...................................................................................... 420 Table 8.B.5 Reliabilities and Correlations for Grade-Specific Tests .................................................................................. 422 Table 8.B.6 Reliabilities and SEMs for the CSTs by Gender ............................................................................................ 423 Table 8.B.7 Reliabilities and SEMs for the CSTs by Economic Status ............................................................................. 424 Table 8.B.8 Reliabilities and SEMs for the CSTs by Special Services .............................................................................. 425 Table 8.B.9 Reliabilities and SEMs for the CSTs by English Fluency ............................................................................... 426 Table 8.B.10 Reliabilities and SEMs for the CSTs by English Fluency ............................................................................. 427 Table 8.B.11 Reliabilities and SEMs for the CSTs by Primary Ethnicity ........................................................................... 428 Table 8.B.12 Reliabilities and SEMs for the CSTs by Primary Ethnicity ........................................................................... 429 Table 8.B.13 Reliabilities and SEMs for the CSTs by Primary Ethnicity ........................................................................... 430 Table 8.B.14 Reliabilities and SEMs for the CSTs by Primary Ethnicity-for-Not Economically Disadvantaged ................ 431 Table 8.B.15 Reliabilities and SEMs for the CSTs by Primary Ethnicity-for-Not Economically Disadvantaged ................ 432 Table 8.B.16 Reliabilities and SEMs for the CSTs by Primary Ethnicity-for-Not Economically Disadvantaged ................ 433 Table 8.B.17 Reliabilities and SEMs for the CSTs by Primary Ethnicity-for-Economically Disadvantaged ....................... 434 Table 8.B.18 Reliabilities and SEMs for the CSTs by Primary Ethnicity-for-Economically Disadvantaged ....................... 435 Table 8.B.19 Reliabilities and SEMs for the CSTs by Primary Ethnicity-for-Economically Disadvantaged ....................... 436 Table 8.B.20 Subscore Reliabilities and SEM for ELA by Gender/Economic Status ........................................................ 437 Table 8.B.21 Subscore Reliabilities and SEM for History–Social Science by Gender/Economic Status ........................... 438 Table 8.B.22 Subscore Reliabilities and SEM for Mathematics by Gender/Economic Status ........................................... 439 Table 8.B.23 Subscore Reliabilities and SEM for Science by Gender/Economic Status .................................................. 441 Table 8.B.24 Subscore Reliabilities and SEM for Grade-Specific CSTs by Gender/Economic Status.............................. 443 Table 8.B.25 Subscore Reliabilities and SEM for ELA by Special Services/English Fluency ........................................... 444 Table 8.B.26 Subscore Reliabilities and SEM for History–Social Science by Special Services/English Fluency .............. 446 Table 8.B.27 Subscore Reliabilities and SEM for Mathematics by Special Services/English Fluency .............................. 447 Table 8.B.28 Subscore Reliabilities and SEM for Science by Special Services/English Fluency...................................... 450 Table 8.B.29 Subscore Reliabilities and SEM for Grade-Specific CSTs by Special Services/English Fluency ................. 452 Table 8.B.30 Subscore Reliabilities and SEM for ELA by Primary Ethnicity ..................................................................... 453 Table 8.B.31 Subscore Reliabilities and SEM for History–Social Science by Primary Ethnicity........................................ 455 Table 8.B.32 Subscore Reliabilities and SEM for Mathematics by Primary Ethnicity ........................................................ 456 Table 8.B.33 Subscore Reliabilities and SEM for Science by Primary Ethnicity ............................................................... 459 Table 8.B.34 Subscore Reliabilities and SEM for Grade-Specific CSTs by Primary Ethnicity........................................... 461 Table 8.B.35 Subscore Reliabilities and SEM for ELA by Primary Ethnicity-for-Not Economically Disadvantaged .......... 462 Table 8.B.36 Subscore Reliabilities and SEM for History–Social Science by Primary Ethnicity-for-Not Economically Disadvantaged .......................................................................................................................................... 464 Table 8.B.37 Subscore Reliabilities and SEM for Mathematics by Primary Ethnicity-for-Not Economically Disadvantaged ................................................................................................................................................................ 465 Table 8.B.38 Subscore Reliabilities and SEM for Science by Primary Ethnicity-for-Not Economically Disadvantaged ................................................................................................................................................................ 468 Table 8.B.39 Subscore Reliabilities and SEM for Grade-Specific CSTs by Primary Ethnicity-for-Not Economically Disadvantaged ................................................................................................................................................................ 470 Table 8.B.40 Subscore Reliabilities and SEM for ELA by Primary Ethnicity-for-Economically Disadvantaged ................. 471 Table 8.B.41 Subscore Reliabilities and SEM for History–Social Science by Primary Ethnicity-for-Economically Disadvantaged ................................................................................................................................................................ 473 Table 8.B.42 Subscore Reliabilities and SEM for Mathematics by Primary Ethnicity-for-Economically Disadvantaged ................................................................................................................................................................ 474 Table 8.B.43 Subscore Reliabilities and SEM for Science by Primary Ethnicity-for-Economically Disadvantaged ........... 477 Table 8.B.44 Subscore Reliabilities and SEM for Grade-Specific CSTs by Primary Ethnicity-for- Economically Disadvantaged .......................................................................................................................................... 479 Table 8.B.45 Reliability of Classification for ELA, Grade Two ........................................................................................... 480 Table 8.B.46 Reliability of Classification for ELA, Grade Three ........................................................................................ 480 Table 8.B.47 Reliability of Classification for ELA, Grade Four (Reading only) .................................................................. 480 Table 8.B.48 Reliability of Classification for ELA, Grade Four (Reading and Writing) ...................................................... 481 Table 8.B.49 Reliability of Classification for ELA, Grade Five ........................................................................................... 481 Page viii
  10. 10. STAR ProgramTable 8.B.50 Reliability of Classification for ELA, Grade Six............................................................................................. 481Table 8.B.51 Reliability of Classification for ELA, Grade Seven (Reading only) ............................................................... 482Table 8.B.52 Reliability of Classification for ELA, Grade Seven (Reading and Writing) .................................................... 482Table 8.B.53 Reliability of Classification for ELA, Grade Eight ......................................................................................... 482Table 8.B.54 Reliability of Classification for ELA, Grade Nine .......................................................................................... 483Table 8.B.55 Reliability of Classification for ELA, Grade Ten ........................................................................................... 483Table 8.B.56 Reliability of Classification for ELA, Grade Eleven....................................................................................... 483Table 8.B.57 Reliability of Classification for History–Social Science, Grade Eight ............................................................ 484Table 8.B.58 Reliability of Classification for World History ................................................................................................ 484Table 8.B.59 Reliability of Classification for History–Social Science, Grade Eleven ......................................................... 484Table 8.B.60 Reliability of Classification for Mathematics, Grade Two ............................................................................. 485Table 8.B.61 Reliability of Classification for Mathematics, Grade Three ........................................................................... 485Table 8.B.62 Reliability of Classification for Mathematics, Grade Four ............................................................................. 485Table 8.B.63 Reliability of Classification for Mathematics, Grade Five ............................................................................. 486Table 8.B.64 Reliability of Classification for Mathematics, Grade Six ............................................................................... 486Table 8.B.65 Reliability of Classification for Mathematics, Grade Seven .......................................................................... 486Table 8.B.66 Reliability of Classification for General Mathematics ................................................................................... 487Table 8.B.67 Reliability of Classification for Algebra I ....................................................................................................... 487Table 8.B.68 Reliability of Classification for Geometry ..................................................................................................... 487Table 8.B.69 Reliability of Classification for Algebra II ...................................................................................................... 488Table 8.B.70 Reliability of Classification for Summative High School Mathematics .......................................................... 488Table 8.B.71 Reliability of Classification for Integrated Mathematics 1 ............................................................................. 488Table 8.B.72 Reliability of Classification for Integrated Mathematics 2 ............................................................................. 489Table 8.B.73 Reliability of Classification for Integrated Mathematics 3 ............................................................................. 489Table 8.B.74 Reliability of Classification for Science, Grade Five ..................................................................................... 489Table 8.B.75 Reliability of Classification for Science, Grade Eight ................................................................................... 490Table 8.B.76 Reliability of Classification for Grade 10 Life Science .................................................................................. 490Table 8.B.77 Reliability of Classification for Biology ......................................................................................................... 490Table 8.B.78 Reliability of Classification for Chemistry ..................................................................................................... 491Table 8.B.79 Reliability of Classification for Earth Science ............................................................................................... 491Table 8.B.80 Reliability of Classification for Physics ......................................................................................................... 491Table 8.B.81 Reliability of Classification for Integrated/Coordinated Science 1 ................................................................ 492Table 8.B.82 Reliability of Classification for Integrated/Coordinated Science 2 ................................................................ 492Table 8.B.83 Reliability of Classification for Integrated/Coordinated Science 3 ................................................................ 492Table 8.B.84 Reliability of Classification for Integrated/Coordinated Science 4 ................................................................ 493Table 8.B.85 Reliability of Classification for Algebra I – Grade 8 ...................................................................................... 493Table 8.B.86 Reliability of Classification for Geometry – Grade 9..................................................................................... 493Table 8.B.87 Reliability of Classification for Algebra II – Grade 10 ................................................................................... 494Table 8.B.88 Reliability of Classification for Summative High School Mathematics – Grade 11 ....................................... 494Table 8.B.89 Reliability of Classification for World History-Grade 10 ................................................................................ 494Table 8.B.90 Inter-Rater Agreement Analyses for ELA (Grade Four) ............................................................................... 495Table 8.B.91 Inter-Rater Agreement Analyses for (ELA Grade Seven) ............................................................................ 495Table 8.B.92 Descriptive Statistics for the Ratings by the Two Raters ............................................................................. 495Table 8.B.93 Generalizability Analyses for Grade Four Essay—[(Person: Essay) x Rater] .............................................. 496Table 8.B.94 Generalizability Analyses for Grade Seven Essay—[(Person: Essay) x Rater] ........................................... 496Table 8.C.1 2009 CST Content Area Correlations (Male) ................................................................................................. 497Table 8.C.2 2009 CST Content Area Correlations (Female) ............................................................................................. 499Table 8.C.3 2009 CST Content Area Correlations (Gender unknown) ............................................................................. 501Table 8.C.4 2009 CST Content Area Correlations (American Indian) ............................................................................... 503Table 8.C.5 2009 CST Content Area Correlations (Asian American) ................................................................................ 505Table 8.C.6 2009 CST Content Area Correlations (Pacific Islander) ................................................................................ 507Table 8.C.7 2009 CST Content Area Correlations (Filipino) ............................................................................................. 509Table 8.C.8 2009 CST Content Area Correlations (Hispanic) ........................................................................................... 511Table 8.C.9 2009 CST Content Area Correlations (African American) ............................................................................. 513Table 8.C.10 2009 CST Content Area Correlations (White).............................................................................................. 515Table 8.C.11 2009 CST Content Area Correlations (Ethnicity unknown) .......................................................................... 517Table 8.C.12 2009 CST Content Area Correlations (English Only) ................................................................................... 519Table 8.C.13 2009 CST Content Area Correlations (Initially Fluent English Proficient) .................................................... 521Table 8.C.14 2009 CST Content Area Correlations (English Learner) .............................................................................. 523Table 8.C.15 2009 CST Content Area Correlations (Reclassified Fluent English Proficient) ............................................ 525Table 8.C.16 2009 CST Content Area Correlations (English Fluency unknown) .............................................................. 527Table 8.C.17 2009 CST Content Area Correlations (Not Economically Disadvantaged) .................................................. 529Table 8.C.18 2009 CST Content Area Correlations (Economically Disadvantaged) ......................................................... 531Table 8.C.19 2009 CST Content Area Correlations (Unknown Economic Status) ............................................................ 533Table 8.C.20 2009 CST Content Area Correlations (No Special Education Services) ...................................................... 535Table 8.C.21 2009 CST Content Area Correlations (Special Education Services) ........................................................... 537 Page ix
  11. 11. STAR Program Table 8.C.22 2009 CST Content Area Correlations (Special Education unknown) ........................................................... 539 Table 8.D.1 IRT Model Data Fit Distribution for ELA ......................................................................................................... 541 Table 8.D.2 IRT Model Data Fit Distribution for History–Social Science ........................................................................... 541 Table 8.D.3 IRT Model Data Fit Distribution for Mathematics, Grades Two through Seven ............................................. 541 Table 8.D.4 IRT Model Data Fit Distribution for Mathematics End-of-Course Tests ......................................................... 541 Table 8.D.5 IRT Model Data Fit Distribution for Science ................................................................................................... 542 Table 8.D.6 IRT Model Data Fit Distribution for ELA (field-test items) .............................................................................. 542 Table 8.D.7 IRT Model Data Fit Distribution for History–Social Science (field-test items) ................................................ 542 Table 8.D.8 IRT Model Data Fit Distribution for Mathematics, Grades Two through Seven (field-test items) ................... 542 Table 8.D.9 IRT Model Data Fit Distribution for Mathematics End-of-Course Tests (field-test items) ............................... 543 Table 8.D.10 IRT Model Data Fit Distribution for Science (field-test items) ...................................................................... 543 Table 8.D.11 IRT b-values for ELA, Grade Two................................................................................................................ 543 Table 8.D.12 IRT b-values for ELA, Grade Three ............................................................................................................. 543 Table 8.D.13 IRT b-values for ELA, Grade Four ............................................................................................................... 544 Table 8.D.14 IRT b-values for ELA, Grade Five................................................................................................................ 544 Table 8.D.15 IRT b-values for ELA, Grade Six ................................................................................................................. 544 Table 8.D.16 IRT b-values for ELA, Grade Seven ............................................................................................................ 544 Table 8.D.17 IRT b-values for ELA, Grade Eight .............................................................................................................. 545 Table 8.D.18 IRT b-values for ELA, Grade Nine ............................................................................................................... 545 Table 8.D.19 IRT b-values for ELA, Grade Ten ................................................................................................................ 545 Table 8.D.20 IRT b-values for ELA, Grade Eleven ........................................................................................................... 545 Table 8.D.21 IRT b-values for History–Social Science, Grade Eight ................................................................................ 546 Table 8.D.22 IRT b-values for World History..................................................................................................................... 546 Table 8.D.23 IRT b-values for History–Social Science Grade Eleven ............................................................................... 546 Table 8.D.24 IRT b-values for Mathematics, Grade Two .................................................................................................. 546 Table 8.D.25 IRT b-values for Mathematics, Grade Three ............................................................................................... 547 Table 8.D.26 IRT b-values for Mathematics, Grade Four ................................................................................................. 547 Table 8.D.27 IRT b-values for Mathematics, Grade Five .................................................................................................. 547 Table 8.D.28 IRT b-values for Mathematics, Grade Six .................................................................................................... 547 Table 8.D.29 IRT b-values for Mathematics, Grade Seven ............................................................................................... 548 Table 8.D.30 IRT b-values for General Mathematics ........................................................................................................ 548 Table 8.D.31 IRT b-values for Algebra I ............................................................................................................................ 548 Table 8.D.32 IRT b-values for Geometry .......................................................................................................................... 548 Table 8.D.33 IRT b-values for Algebra II ........................................................................................................................... 549 Table 8.D.34 IRT b-values for Summative High School Mathematics ............................................................................... 549 Table 8.D.35 IRT b-values for Integrated Mathematics 1 .................................................................................................. 549 Table 8.D.36 IRT b-values for Integrated Mathematics 2 .................................................................................................. 549 Table 8.D.37 IRT b-values for Integrated Mathematics 3 .................................................................................................. 550 Table 8.D.38 IRT b-values for Science, Grade Five.......................................................................................................... 550 Table 8.D.39 IRT b-values for Science, Grade Eight ........................................................................................................ 550 Table 8.D.40 IRT b-values for Grade 10 Life Science ....................................................................................................... 550 Table 8.D.41 IRT b-values for Biology .............................................................................................................................. 551 Table 8.D.42 IRT b-values for Chemistry .......................................................................................................................... 551 Table 8.D.43 IRT b-values for Earth Science .................................................................................................................... 551 Table 8.D.44 IRT b-values for Physics .............................................................................................................................. 551 Table 8.D.45 Distribution of IRT b-values for ELA ............................................................................................................ 552 Table 8.D.46 Distribution of IRT b-values for History–Social Science ............................................................................... 552 Table 8.D.47 Distribution of IRT b-values for Mathematics, Grades Two through Seven ................................................. 553 Table 8.D.48 Distribution of IRT b-values for Mathematics End-of-Course CSTs ............................................................. 553 Table 8.D.49 Distribution of IRT b-values for Science ...................................................................................................... 554 Table 8.D.50 Distribution of IRT b-values for ELA (field-test items) .................................................................................. 554 Table 8.D.51 Distribution of IRT b-values for History–Social Science (field-test items) .................................................... 555 Table 8.D.52 Distribution of IRT b-values for Mathematics, Grades Two through Seven (field-test items) ....................... 555 Table 8.D.53 Distribution of IRT b-values for Mathematics End-of-Course CSTs (field-test items) .................................. 556 Table 8.D.54 Distribution of IRT b-values for Science (field-test items) ............................................................................ 556 Table 8.D.55 2009 New Conversions for ELA, Grades Two and Three ............................................................................ 557 Table 8.D.56 New Conversions for ELA, Grade Four (with and without essay) ................................................................ 558 Table 8.D.57 2009 New Conversions for ELA, Grades Five and Six ................................................................................ 559 Table 8.D.58 2009 New Conversions for ELA, Grade Seven (with and without essay) .................................................... 560 Table 8.D.59 2009 New Conversions for ELA, Grades Eight and Nine ............................................................................ 561 Table 8.D.60 2009 New Conversions for ELA, Grades Ten and Eleven ........................................................................... 562 Table 8.D.61 2009 New Conversions for History–Social Science, Grade Eight and World History .................................. 563 Table 8.D.62 2009 New Conversion for History–Social Science, Grade Eleven ............................................................... 564 Table 8.D.63 2009 New Conversions for Mathematics, Grades Two and Three .............................................................. 565 Table 8.D.64 2009 New Conversions for Mathematics, Grades Four and Five ................................................................ 566 Table 8.D.65 2009 New Conversions for Mathematics, Grades Six and Seven ............................................................... 567 Page x

×