REPORTING A TEST

1. Chapter 5
Instrument Selection, Administration,
Scoring, and Communicating Results

2. Selection of an Assessment
Instrument
 Identify possible alternatives
 Educational Testing Services may be
useful
 Mental Measurement Yearbook (MMY)
reviews many tests and is available at
Morris or reviews can be ordered on-line
for a fee.
 Client’s lives can be adversely
affected by selecting and using a
faulty instrument

3. Evaluating an Instrument
 Test Purpose
 Instrument
Development
 Appropriate Norm
Group or Criterion
 Reliability
 Validity
 Bias
 Interpretation and
Scoring
 User Qualifications

4. Test purpose
 Does instrument meet counseling
needs?
 Manual may state purpose
 Purpose may not relate to name
 Purpose may not be met because of
construction problems

5. Instrument development
 Construction of items crucial
 Look for detailed item analysis data
 Face validity

6. Selection of norm group
 Is norming group appropriate for
client?
 Age
 Gender
 Ethnicity
 Socioeconomic representation
 Educational level
 Geographic location

7. Selection of criterion
 How did developers determine
criterion?
 Standards from professional
organizations
 Reviews by leading researchers
 Studies of curriculum
 Manual must supply enough data to
be convincing

8. Reliability
 No clear guidelines for what
constitutes “good” reliability
 Type of instrument affects how
reliability should be viewed
 Can be estimated in different ways
and manuals will often provide range
of coefficients calculated using
different methods

9. Validity
 Look for preponderence of evidence
 Manual must provide enough
information to judge
 GO TO THE LITERATURE!

10. Bias
 Test bias in context of multicultural
issues
 Differential item functioning (item
difficulty differs for ethnic groups)
 Reliability coefficients may differ
ethnically
 Construct validity may be compromised
as the construct may be culturally based

11. Interpretation and scoring
materials
 Manual must provide adequate
description of what scores mean
 Hand vs. machine scoring

12. User qualifications
 Level A: manual is sufficient (Holland’s Self-
Directed Search)
 Level B: master’s degree or equivalent
training (Myers-Briggs, Strong Interest
Inventory)
 Level C: doctoral level or equivalent
training (Wechslers, Binet, MMPI-2)
 Code of Fair Testing Practices in Education
 ACA Code of Ethics, Section E

13. Practical issues
 Cost
 Time

14. Evaluating instruments
 Form in text, page 95
 Instrument evaluation paper

15. Administering Assessment
Instruments
 Pre-read administration materials
 Follow standardized procedures

16. Administrator Effects
 Expectancy (pygmalion effect)
 Some data support minor effects

17. Examiner/Examinee Relationship
 Some studies support impact of
relationship for kids and even more
so for lower SES kids
 Standardized test manuals usually
address relationship boundaries

18. Examiner race
 Research equivocal

19. Scoring
 Hand scoring
 Clients can score some assessments
(SDS) but may be clinical reason not to
 Some tests require multiple templates
 Computer scoring and interpretation
 Few to no errors
 Counselors ethically bound to know
integrity of service and steps taken to
develop scoring and interpretation
programs

20. Performance and Authentic
Assessment Scoring
 Associated with achievement testing
(rather than grammar test, write
business letter)
 Goal is to see if knowledge can be
applied
 Big issue is objectivity of scoring

21. Performance and Authentic
Assessment Scoring Guidelines
 Assessment has
specific focus
 Scoring plan is
based on
observable
qualities
 Scoring is designed
to reflect the
intended target
 Setting for
assessment is
appropriate
 Checklist or rating
scales are used
 Scoring procedures
have been field
tested before they
are used

22. Communicating Results
 Must know manual information
 Optimize power of test (It’s useful in
these ways for these reasons) rather
than allowing client to maximize (test
speaks truth)
 Use effective counseling skills
 Develop multiple methods of
explaining in “plain” English

23. Communicating Results (cont’d)
 Use descriptive terms rather than
numerical scores and tie them to the
reason for the assessment
 Put results in context of other client
information
 Involve client in interpretation and
ask for feedback often

24. Communicating Results (cont’d)
 Limitations are discussed in non-
technical terms
 Encourage client to ask questions
 Summarize results to iterate and
stress important points

25. Communicating Results to Parents
 Be empathic
 Monitor parents’ reactions so child
does not internalize as blame
 Provide therapeutic environment