Summary of Technical Adequacy Evidence
Part 1: Technical Quality (What is the technical quality of the assessment tool?)
1.1 Provide a general description of the knowledge and skills, including their relative emphasis, that the
assessment tool is designed to measure.
1.2 Describe (or reference) the steps used to develop the assessment tool and support materials.
1.3 Describe (or reference) the procedures used to ensure that the assessment tool is fair and free of bias.
1.4 Provide an estimate of the reliability of student scores (e.g. Internal Consistency such as Coefficient α,
KR-20, or KR-21) or decision consistency.
1.5 If multiple test forms were used, describe the evidence obtained to ensure their equivalence.
1.6 If achievement levels (or performance standards) were used, provide the verbal descriptors of student
performance for each level and describe the procedures used to create these verbal descriptors and to establish
the corresponding cut points.
3/12/03 Technical Adequacy Evidence: Locally-Developed Assessment Page 2
Part 2. Implementation (How was the assessment administered and scored?)
2.1 What administration procedures were followed for the assessment, and what steps were taken to ensure that
these procedures were uniformly followed across schools (classes) and in consecutive years?
a. Describe (or reference) the administration procedures.
b. If the administration procedures were modified for any class or building within the district or for different
school years, describe how and why the administration procedures differed.
c. From the list below, mark the steps taken to ensure/verify that the standard conditions were followed
across settings and occasions:
* Provided training or instruction to all test administrators on the use of standard conditions.
* Observed test administrations in selected classrooms in the building.
* Documented any irregularities or incidences that might distort the quality of the data.
If optional conditions were possible (e.g., calculator availability), the conditions were applied to
* all students.
d. From the list below, mark the steps taken to ensure/verify that student scores were not compromised (at
any setting or on any occasion).
* Prior to the test administration, students did not have access to assessment items or tasks.
During the test administration, students did not receive assistance that could lead to inflated scores
* (e.g., cheating, cueing, extra help from teacher, or inappropriate accommodation or modification).
* After the test administration, student responses/answers were not altered.
2.2 Were any accommodations or modifications made when this assessment was administered? If so, describe
the accommodations or modifications used.
2.3 When objective scoring is incorporated manually, what is done to ensure the accuracy of the scores?
a. Describe the qualifications of the scorers.
b. From the list below, mark the steps taken to ensure the accuracy of the scores.
* Checked the consistency of the scoring (e.g., scored responses twice) for a sample.
Checked the accuracy of the score conversions, as appropriate (e.g., raw score to scaled score, or
* percent correct, conversions) for a sample.
* Verified the accuracy of the keyed entries for database updates/creation.
3/12/03 Technical Adequacy Evidence: Locally-Developed Assessment Page 3
2.4 When constructed-response tasks require raters to assign scores to student responses, what were the scoring
procedures and what was done to ensure the quality of the scores across raters and occasions?
a. Describe the panel of raters participating in the scoring by providing the following: 1) number of raters
who evaluated student responses to tasks from this assessment, 2) qualifications of the raters in terms of
the sufficiency of their content expertise and the experiences they have had working with students at the
relevant grade level.
b. Describe (or reference) the training opportunities provided to raters by providing descriptions of the
following: 1) training procedures and/or materials used to assist raters in developing a common
understanding of the scoring criteria, and 2) procedures used to monitor the scoring process to determine
if/when retraining was necessary.
c. Describe (or reference) the process followed to score student responses by providing a description of the
following: 1) procedure used to conceal student identities from the raters—if the identities were not
concealed, describe what was done to reduce the likelihood that this knowledge influenced the ratings;
and 2) procedure used to determine the final score assigned to a student’s response—if a given response
received more than one rating, describe how differences were adjusted (if at all) when the ratings
assigned to this response were not the same.
d. Describe (or reference) the steps taken to ensure the comparability of the ratings across raters and over
time by providing a description of the following: 1) aspects of the scoring materials and or procedures
that were used to increase comparability of ratings across raters (e.g., task-specific scoring criteria,
anchor papers, training papers, facilitators/table leaders received standardized training); and
2) procedures used to ensure the consistency of the scoring and training procedures from one setting
or year to the next.
2.5 When constructed-response tasks require raters to assign scores to student responses, what is the consistency
(reliability) of scores across raters (i.e., rater consistency), and how was this determined?
a. Describe the method used to estimate rater consistency by answering the following questions:
1) Was rater consistency computed based on independent scores, or based on a consensus or arbitration
process? 2) If only one rater was used, how was rater consistency determined?
b. For each constructed-response task on this assessment scored during the scoring session what was the
total number of student responses scored and what percent of these responses were used in computing
c. For the constructed-response task(s) on this assessment, provide data summarizing the extent of rater
consistency achieved and describe how these data were obtained. At a minimum, summarize the extent
of “percent exact agreement.”
3/12/03 Technical Adequacy Evidence: Locally-Developed Assessment Page 4
Part 3. Interpretation (How are the assessment results interpreted and reported?)
3.1 From the list below, mark the steps taken to establish the integrity of the scores prior to making
interpretations with them.
Score data were checked for completeness to ensure that the results of all students tested were
* included. If scores were missing, corrective action was taken.
Suspicious scores (i.e., extremely high or low scores) were checked for accuracy, and if appropriate,
* erroneous scores were corrected.
Group scores were checked to ensure that scores of students with IEPs and scores from all other
* subgroups were included in district- and building-wide scores, and that the number of students in each
of these subgroups was reasonable.
3.2 Describe how the results from this assessment tool were reported in the Annual Progress Report.
3.3 From the list below, mark the steps taken to ensure that the displays/presentations (e.g., graphs, tables,
figures, or text) of assessment information in the Annual Progress Report were accurate and clear.
An accurate verbal description of the achievement domain measured by the instrument was provided
* (e.g., “reading comprehension” not just “reading”).
* Graphs or other visuals were constructed using size and scale that do not mislead.
All displays of data were labeled completely and accurately (e.g., student vs. school norms,
* Iowa vs. national norms, mean vs. median grade equivalent, etc.).
* Group sizes were included with all displays of data.
Where achievement levels were used, the verbal descriptors of the performance associated with the
* achievement levels were also given (e.g., description of what “proficient” means)
3.4 From the list below, mark the steps taken to ensure that the conclusions/interpretations made in the Annual
Progress Report are supported by the data reported.
The conclusions/interpretations were examined to verify that they were consistent with the data that
* were reported.
The generalizations were examined to verify that they did not go beyond the knowledge/skills actually
* measured by the instrument (e.g. just “reading comprehension” instead of all content standards in
The generalizations were examined to verify that they did not go beyond the grade level for which
* data were reported.
Common or potential misinterpretations or misunderstandings were identified, and cautions were
* included in the APR to address them.
3/12/03 Technical Adequacy Evidence: Locally-Developed Assessment Page 5