Arte387Ch8

355 views

Published on

Published in: Education
  • Be the first to comment

  • Be the first to like this

Arte387Ch8

  1. 1. Chapter 8: Validity and Reliability Facilitated by Marissa Barclay
  2. 2. <ul><li>The art educator involved in assessment initiatives beyond the classroom needs to have a working knowledge of validity and reliability issues as well as item analysis-all of which are critical to assessment practice. </li></ul><ul><li>In layman's terms: YOU NEED TO KNOW IT! </li></ul><ul><li>**all information in this Power Point is courtesy of the author, Donna Kay Beattie** </li></ul>
  3. 3. <ul><li>Validity: In short, inferences drawn from a test or assessment score need to be validated. </li></ul><ul><li>There are twelve validation criteria that are useful for judging the validity of performance-based art assignments. </li></ul><ul><li>These criteria should be addressed during the given performance task. </li></ul>
  4. 4. <ul><li>Relevance refers to the quality of fit between the purpose of the assessment and selected performance format and tasks. </li></ul><ul><li>Art educator should look for the best possible match between purpose and assessment performance formats. </li></ul>
  5. 5. <ul><li>This refers to the authenticity of assessment task content. </li></ul><ul><li>The assessment task must faithfully reflect the integrity of the discipline and clearly show the field’s most time-tested and valuable content and processes. </li></ul>
  6. 6. <ul><li>This term refers to the scope and comprehensiveness of performance task content and underlying constructs are the internal qualities and behaviors that undergird a performance. </li></ul>
  7. 7. <ul><li>Refers to the level s of intellectual complexity the performance task requires of students. </li></ul><ul><li>An example of this might be a holistic performance –based task that covers each of the four visual arts disciplines, their interconnections, and their connections to the other academic disciplines. </li></ul>
  8. 8. <ul><li>Refers to the extent to which a task allows equal opportunities for all students to succeed. </li></ul><ul><li>The art educator can create a checklist of possible sources of bias to be used to review the task as it is being developed. </li></ul>
  9. 9. <ul><li>Refers to how motivating, challenging, and satisfying a task is to both students and others who might have interest in the task, such as parents, other teachers, administrators, and experts from the art disciplines. </li></ul>
  10. 10. <ul><li>Refers to students’ need to see and understand what is expected of them. In assessment terminology this is known as transparency. </li></ul>
  11. 11. <ul><li>Refers to the homogeneity of exercises in a performance. </li></ul><ul><li>If the holistic task score is to be interpreted as a valid measure of knowledge of the visual arts, then each exercise within the task should correlate well with the others. </li></ul>
  12. 12. <ul><li>Refers to both intended and unintended consequences of a task or of interpretation of task scores,. </li></ul><ul><li>The art educator can address the consequential criterion by anticipating possible assessment side effects (negative and positive), and hypothesizing potential testing outcomes. </li></ul>
  13. 13. <ul><li>Refers to the extent to which the task reflects the actual behavior of characteristic being examined. </li></ul><ul><li>A direct assessment of students abilities to criticize a work of art would be to have them write a critical review. </li></ul><ul><li>Knowledge and skills cannot be assessed directly, but are inferred from performances and products. </li></ul>
  14. 14. <ul><li>The value of the performance task must override the cost, and performance assessments generally cost more than multiple-choice or other pencil-and-paper test formats. </li></ul><ul><li>All aspects of a performance assessment should be carefully studied in an effort to devise ways to reduce costs and increase efficiency. </li></ul>
  15. 15. <ul><li>Refers to the degree to which the results of a performance assessment can be generalized across different domains. </li></ul><ul><li>Contextual can be considered here because it is concerned with issues of transfer and generalizability. </li></ul><ul><li>Classroom settings, classroom management effects, and even rater variability are contextual issues that need to be analyzed before making judgments of transfer. </li></ul>
  16. 16. <ul><li>This is a concept known in classical tests and measurement theory. </li></ul><ul><li>Can also be defined as the consistency of scores. </li></ul><ul><li>For the art educator it is worthwhile to know what factors might cause an unreliable assessment score. </li></ul><ul><li>Note: an assessment can be reliable without being valid, but it cannot be valid without being reliable. </li></ul><ul><ul><li>Meaning: reliability affects the quality of the decisions made on the basis of the derived score. </li></ul></ul>
  17. 17. <ul><li>There are twelve procedures to help improve the reliability and generalizability of art assessments: </li></ul><ul><li>**Also listed on the handout* </li></ul><ul><li>-Assess the same material -Broaden the scope of assessments </li></ul><ul><li>-Develop clear and concrete scoring criteria </li></ul><ul><li>-Make annotated examples showing each score </li></ul><ul><li>-Make scoring objective, use a scoring rubric </li></ul><ul><li>-When possible, and use more than one scorer </li></ul>
  18. 18. <ul><li>-if more than one judgment is needed, train scorers </li></ul><ul><li>-Check consistency, go back and check scores </li></ul><ul><li>-Score one question on all tests, then the next, etc. </li></ul><ul><li>- Provide practice and training assessment s </li></ul><ul><li>-Craft tests to fit each student’s needs </li></ul><ul><li>-Design tasks that help differentiate the most able from the least able students </li></ul>
  19. 19. <ul><li>This is a related psychometric issue to validity and reliability. </li></ul><ul><li>This term is defined as “the process of collecting summarizing, and using information from students’ responses to make decisions about each assessment task. </li></ul><ul><li>Three Steps of a simple item analysis on an important classroom test (written exam) </li></ul><ul><li>1) The teacher needs to group the assessments according to high and low scores. </li></ul><ul><li>2) Tallies are made of each group’s responses to each test item </li></ul><ul><li>3) Percentages for item responses are figured. </li></ul>
  20. 22. <ul><li>Validity refers to inferences drawn from a test or assessment score needed to be validated </li></ul>
  21. 23. <ul><li>Relevance </li></ul><ul><li>Content and fidelity </li></ul><ul><li>Exhaustiveness </li></ul><ul><li>Cognitive complexity </li></ul><ul><li>Equity </li></ul><ul><li>Meaningfulness </li></ul><ul><li>Straightforwardness </li></ul><ul><li>Cohesiveness </li></ul><ul><li>Consequences </li></ul><ul><li>Directness </li></ul><ul><li>Cost and efieiency </li></ul><ul><li>Generalizability </li></ul>
  22. 24. <ul><li>This is a concept known in classical tests and measurement theory. </li></ul><ul><li>Can also be defined as the consistency of scores. </li></ul>
  23. 25. <ul><li>FALSE! </li></ul><ul><li>an assessment can be reliable without being valid, but it cannot be valid without being reliable. </li></ul><ul><ul><li>Meaning: reliability affects the quality of the decisions made on the basis of the derived score. </li></ul></ul>

×