Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Basic Assessment Literacy Everything  You Wanted to Know About Testing But Was Never That Interested
<ul><li>Smartest kid in class </li></ul>
<ul><li>Sticky Ideas </li></ul>
<ul><li>Quality data  requires a quality test </li></ul><ul><li>Quality tests  are not easy to create </li></ul>
Assessment literacy means: 5 Blind Men & an Elephant?
Assessment literacy means: Gathering  dependable information about student achievement Using  the assessment process to pr...
What’s a “good test?” <ul><li>“ The most basic and obvious answer to that question is that good tests measure what you wan...
Questions to ask Why? Users/Uses What? Targets How Much? Sample How? Method Accurate? Avoid Bias
Why? Users/Uses What? Targets How Much? Sample How? Method Accurate? Avoid Bias
Why? <ul><li>Is the  Purpose  for the assessment clear? </li></ul><ul><ul><li>Why are these targets being assessed?  </li>...
Why? Users/Uses What? Targets How Much? Sample How? Method Accurate? Avoid Bias
What? <ul><li>Are the  Targets  clear and appropriate? </li></ul><ul><ul><li>Specific & clearly defined expectations </li>...
Four target types <ul><li>Knowledge </li></ul><ul><ul><li>George Washington / 1787 / US Constitution </li></ul></ul><ul><l...
<ul><li>Skills </li></ul><ul><ul><li>Analyze a primary source document </li></ul></ul><ul><li>Products </li></ul><ul><ul><...
<ul><li>An assessment that accurately reflects its intended target is  Valid </li></ul>
Why? Users/Uses What? Targets How Much? Sample How? Method Accurate? Avoid Bias
How? <ul><li>Different  targets require  different  methods </li></ul><ul><ul><li>Match appropriate method with intended p...
General categories <ul><li>Objective </li></ul><ul><ul><li>Student selects correct response from several alternatives /  s...
<ul><li>Subjective </li></ul><ul><ul><li>Student organizes / presents an original answer </li></ul></ul><ul><ul><li>Hard t...
<ul><li>Objective  vs.  Subjective   </li></ul><ul><ul><li>“ By definition, no test can be truly objective” </li></ul></ul>
<ul><li>Selected Response   </li></ul><ul><ul><li>Multiple Choice, True/False, Matching, Fill-in-the-blank </li></ul></ul>...
Mark Pett ©2001
Let’s try it! <ul><li>Name the parts of the human skeleton </li></ul><ul><ul><li>Selected response </li></ul></ul><ul><li>...
<ul><li>Cite four examples of satire in  Huckleberry Finn </li></ul><ul><ul><li>Short answer essay </li></ul></ul><ul><li>...
<ul><li>Diagnose a physical ailment </li></ul><ul><ul><li>Extended response essay </li></ul></ul><ul><li>List important me...
Why? Users/Uses What? Targets How Much? Sample How? Method Accurate? Avoid Bias
How much? <ul><li>How can we  Sample  appropriately? </li></ul><ul><ul><li>“ All assessments are just a small part of an ‘...
<ul><li>“ Have I gathered enough information of the right kind so that I can draw confident conclusions about student achi...
Why? Users/Uses What? Targets How Much? Sample How? Method Accurate? Avoid Bias
Mark Pett ©2001
Accurate? <ul><li>What are possible sources of  Interference  or  Bias? </li></ul>
<ul><li>Assessment </li></ul><ul><ul><li>Poorly written, poor target-method match, culturally biased, inappropriate readin...
<ul><li>Student </li></ul><ul><ul><li>Test anxiety, cheat, guess, interpretation different than intent </li></ul></ul><ul>...
<ul><li>Eliminating errors due to bias increase the  Reliability  of our assessments </li></ul>
<ul><li>What was  new? </li></ul><ul><li>What was  re-new? </li></ul>
Steps to Structured Response test <ul><li>Prepare  a blueprint </li></ul><ul><li>Select  material to assess </li></ul><ul>...
<ul><li>Prepare a blueprint </li></ul>Step 1
40 15 25 Total 13 5 8 Rights & responsibilities of citizens 12 5 7 Structure of US government 15 5 10 Forms of government ...
<ul><li>Select material to assess </li></ul>Step 2
<ul><li>“ It is always tempting to emphasize the parts of the course that are  easiest  to test, rather than the parts tha...
So what’s worth assessing?
<ul><li>Which test items,  if any, need to be changed? </li></ul><ul><li>Can you think  of a test that included items you ...
<ul><li>A select sub-set  of all possible important elements </li></ul><ul><li>Professional judgment  of relative importan...
Step 3 <ul><li>Build  test items from propositions </li></ul><ul><ul><li>“ Clearly stated sentences that reflect elements ...
Examples <ul><li>In a monarchy, the right to govern is secured through birth. </li></ul><ul><li>The executive and legislat...
True or False <ul><li>In a monarchy, the right to govern is secured through birth. </li></ul><ul><li>or </li></ul><ul><li>...
Fill in the Blank <ul><li>How is the right to govern secured in a monarchy? </li></ul>
Multiple Choice <ul><li>How is the right to govern secured in a monarchy? </li></ul><ul><li>A.  With military power </li><...
True or False <ul><li>The executive and legislative branches differ in that the latter is elected directly by the people. ...
Fill in the Blank <ul><li>Election of members of the executive and legislative branches differ in what way? </li></ul>
Multiple Choice <ul><li>Election of members of the executive and legislative branches differ in what way? </li></ul><ul><l...
<ul><li>Franzipanics Quiz </li></ul>
Specific assessment criteria
Multiple choice <ul><li>Advantages </li></ul><ul><ul><li>Effective measurement </li></ul></ul><ul><ul><li>Reduces guessing...
Multiple Choice suggestions <ul><li>Stem is a clear, explicit & singular question </li></ul><ul><li>Use at least four alte...
<ul><li>Alternatives are the same format and length </li></ul><ul><li>All alternatives should be plausible </li></ul><ul><...
Is the question any good? <ul><li>Student who knows the content will know the answer before reading the alternatives </li>...
<ul><li>“ Your goal  is not to trick students or require them to make difficult judgments about two nearly equal options.”...
<ul><li>“ Your goal  is to write questions that students who understand the material will answer correctly and students wh...
Bill Watterson ©1995
True or false <ul><li>Advantages </li></ul><ul><ul><li>Can measure large amounts of basic knowledge </li></ul></ul><ul><ul...
True & false suggestions <ul><li>Statements are absolutely true or false </li></ul><ul><li>Express item as simply & clearl...
Matching <ul><li>Advantages </li></ul><ul><ul><li>Measures large amounts of knowledge level content </li></ul></ul><ul><ul...
Matching suggestions <ul><li>Use items that share same foundation of knowledge </li></ul><ul><li>Avoid grammatical cues </...
Fill-in-the-Blank <ul><li>Advantages </li></ul><ul><ul><li>Good for 5 Ws and H </li></ul></ul><ul><ul><li>Helps minimize g...
Fill-in-the-Blank suggestions <ul><li>Omit only significant items from stem </li></ul><ul><li>Insure only one correct answ...
Things to discuss <ul><li>Possible accommodations </li></ul><ul><li>Student evaluations of test </li></ul><ul><li>Pilot ve...
Upcoming SlideShare
Loading in …5
×

Assessment Literacy For GIPS

1,612 views

Published on

Published in: Education, Technology
  • Be the first to comment

Assessment Literacy For GIPS

  1. 1. Basic Assessment Literacy Everything You Wanted to Know About Testing But Was Never That Interested
  2. 2. <ul><li>Smartest kid in class </li></ul>
  3. 3. <ul><li>Sticky Ideas </li></ul>
  4. 4. <ul><li>Quality data requires a quality test </li></ul><ul><li>Quality tests are not easy to create </li></ul>
  5. 5. Assessment literacy means: 5 Blind Men & an Elephant?
  6. 6. Assessment literacy means: Gathering dependable information about student achievement Using the assessment process to promote maximum achievement
  7. 7. What’s a “good test?” <ul><li>“ The most basic and obvious answer to that question is that good tests measure what you want to measure, and bad tests do not.” Ben Clay Is This a Trick Question? </li></ul>
  8. 8. Questions to ask Why? Users/Uses What? Targets How Much? Sample How? Method Accurate? Avoid Bias
  9. 9. Why? Users/Uses What? Targets How Much? Sample How? Method Accurate? Avoid Bias
  10. 10. Why? <ul><li>Is the Purpose for the assessment clear? </li></ul><ul><ul><li>Why are these targets being assessed? </li></ul></ul><ul><ul><li>Who will use the information? </li></ul></ul><ul><ul><li>What will the information be used for? </li></ul></ul>
  11. 11. Why? Users/Uses What? Targets How Much? Sample How? Method Accurate? Avoid Bias
  12. 12. What? <ul><li>Are the Targets clear and appropriate? </li></ul><ul><ul><li>Specific & clearly defined expectations </li></ul></ul><ul><ul><li>Center on the core of the discipline </li></ul></ul><ul><ul><li>Aligned to program / course / unit level curriculum </li></ul></ul>
  13. 13. Four target types <ul><li>Knowledge </li></ul><ul><ul><li>George Washington / 1787 / US Constitution </li></ul></ul><ul><li>Reasoning </li></ul><ul><ul><li>Evaluate differences between government types </li></ul></ul>
  14. 14. <ul><li>Skills </li></ul><ul><ul><li>Analyze a primary source document </li></ul></ul><ul><li>Products </li></ul><ul><ul><li>Create a reasoned argument for allowing immigration reform </li></ul></ul>
  15. 15. <ul><li>An assessment that accurately reflects its intended target is Valid </li></ul>
  16. 16. Why? Users/Uses What? Targets How Much? Sample How? Method Accurate? Avoid Bias
  17. 17. How? <ul><li>Different targets require different methods </li></ul><ul><ul><li>Match appropriate method with intended purpose and target </li></ul></ul>
  18. 18. General categories <ul><li>Objective </li></ul><ul><ul><li>Student selects correct response from several alternatives / supplies a word or short phrase to answer a question </li></ul></ul><ul><ul><li>Easy to grade / hard to create </li></ul></ul>
  19. 19. <ul><li>Subjective </li></ul><ul><ul><li>Student organizes / presents an original answer </li></ul></ul><ul><ul><li>Hard to grade / easy to create </li></ul></ul>
  20. 20. <ul><li>Objective vs. Subjective </li></ul><ul><ul><li>“ By definition, no test can be truly objective” </li></ul></ul>
  21. 21. <ul><li>Selected Response </li></ul><ul><ul><li>Multiple Choice, True/False, Matching, Fill-in-the-blank </li></ul></ul><ul><li>Essays </li></ul><ul><ul><li>Short Answer, Extended Response </li></ul></ul><ul><li>Performance Tasks </li></ul><ul><li>Personal Communication </li></ul>Specific categories
  22. 22. Mark Pett ©2001
  23. 23. Let’s try it! <ul><li>Name the parts of the human skeleton </li></ul><ul><ul><li>Selected response </li></ul></ul><ul><li>Appraise a composition on the basis of its organization </li></ul><ul><ul><li>Short answer essay </li></ul></ul><ul><li>Demonstrate safe lab skills </li></ul><ul><ul><li>Performance task </li></ul></ul>
  24. 24. <ul><li>Cite four examples of satire in Huckleberry Finn </li></ul><ul><ul><li>Short answer essay </li></ul></ul><ul><li>Design a logo for a web page </li></ul><ul><ul><li>Performance task </li></ul></ul><ul><li>Describe the impact of a Bull Market </li></ul><ul><ul><li>Extended response essay </li></ul></ul>
  25. 25. <ul><li>Diagnose a physical ailment </li></ul><ul><ul><li>Extended response essay </li></ul></ul><ul><li>List important mental attributes necessary in an athlete </li></ul><ul><ul><li>Short response essay </li></ul></ul><ul><li>Categorize great American fiction writers </li></ul><ul><ul><li>Selected response </li></ul></ul>
  26. 26. Why? Users/Uses What? Targets How Much? Sample How? Method Accurate? Avoid Bias
  27. 27. How much? <ul><li>How can we Sample appropriately? </li></ul><ul><ul><li>“ All assessments are just a small part of an ‘ideal’ assessment of infinite length.” </li></ul></ul>
  28. 28. <ul><li>“ Have I gathered enough information of the right kind so that I can draw confident conclusions about student achievement?” </li></ul>
  29. 29. Why? Users/Uses What? Targets How Much? Sample How? Method Accurate? Avoid Bias
  30. 30. Mark Pett ©2001
  31. 31. Accurate? <ul><li>What are possible sources of Interference or Bias? </li></ul>
  32. 32. <ul><li>Assessment </li></ul><ul><ul><li>Poorly written, poor target-method match, culturally biased, inappropriate reading level </li></ul></ul><ul><li>Assessor </li></ul><ul><ul><li>Incorrectly scored </li></ul></ul>
  33. 33. <ul><li>Student </li></ul><ul><ul><li>Test anxiety, cheat, guess, interpretation different than intent </li></ul></ul><ul><li>Environment </li></ul><ul><ul><li>Cold, hot, poor light, time restraints, noise, lunch, recess </li></ul></ul>
  34. 34. <ul><li>Eliminating errors due to bias increase the Reliability of our assessments </li></ul>
  35. 35. <ul><li>What was new? </li></ul><ul><li>What was re-new? </li></ul>
  36. 36. Steps to Structured Response test <ul><li>Prepare a blueprint </li></ul><ul><li>Select material to assess </li></ul><ul><li>Build test items from propositions </li></ul>
  37. 37. <ul><li>Prepare a blueprint </li></ul>Step 1
  38. 38. 40 15 25 Total 13 5 8 Rights & responsibilities of citizens 12 5 7 Structure of US government 15 5 10 Forms of government Total Process & Application Know & Understand Content Targets
  39. 39. <ul><li>Select material to assess </li></ul>Step 2
  40. 40. <ul><li>“ It is always tempting to emphasize the parts of the course that are easiest to test, rather than the parts that are important to test.” Ben Clay Is This a Trick Question? </li></ul>
  41. 41. So what’s worth assessing?
  42. 42. <ul><li>Which test items, if any, need to be changed? </li></ul><ul><li>Can you think of a test that included items you felt weren’t worth learning or assessing? </li></ul><ul><li>How do you make decisions about what’s worth assessing? </li></ul>Review the Case Study
  43. 43. <ul><li>A select sub-set of all possible important elements </li></ul><ul><li>Professional judgment of relative importance </li></ul>So . . . ?
  44. 44. Step 3 <ul><li>Build test items from propositions </li></ul><ul><ul><li>“ Clearly stated sentences that reflect elements of content aligned to Course / Unit EUs & EQs.” </li></ul></ul><ul><ul><li>Create more than blueprint requires </li></ul></ul>
  45. 45. Examples <ul><li>In a monarchy, the right to govern is secured through birth. </li></ul><ul><li>The executive and legislative branches differ in that the latter is elected directly by the people. </li></ul>
  46. 46. True or False <ul><li>In a monarchy, the right to govern is secured through birth. </li></ul><ul><li>or </li></ul><ul><li>In a monarchy, the right to govern is secured through the approval of those governed. </li></ul>
  47. 47. Fill in the Blank <ul><li>How is the right to govern secured in a monarchy? </li></ul>
  48. 48. Multiple Choice <ul><li>How is the right to govern secured in a monarchy? </li></ul><ul><li>A. With military power </li></ul><ul><li>B. Through birth </li></ul><ul><li>C. By popular vote </li></ul><ul><li>D. Through purchase </li></ul>
  49. 49. True or False <ul><li>The executive and legislative branches differ in that the latter is elected directly by the people. </li></ul><ul><li>or </li></ul><ul><li>Members of executive and legislative branches are both elected directly by the people. </li></ul>
  50. 50. Fill in the Blank <ul><li>Election of members of the executive and legislative branches differ in what way? </li></ul>
  51. 51. Multiple Choice <ul><li>Election of members of the executive and legislative branches differ in what way? </li></ul><ul><li>A. Legislators are restricted by term limits; presidents are not </li></ul><ul><li>B. Legislators are elected directly; presidents are not. </li></ul><ul><li>C. One must register to vote for legislators; not for president </li></ul><ul><li>D. One must register to vote for presidents; not for legislators. </li></ul>
  52. 52. <ul><li>Franzipanics Quiz </li></ul>
  53. 53. Specific assessment criteria
  54. 54. Multiple choice <ul><li>Advantages </li></ul><ul><ul><li>Effective measurement </li></ul></ul><ul><ul><li>Reduces guessing </li></ul></ul><ul><ul><li>Can cover wide range of content </li></ul></ul><ul><li>Disadvantages </li></ul><ul><ul><li>Difficult to create </li></ul></ul>
  55. 55. Multiple Choice suggestions <ul><li>Stem is a clear, explicit & singular question </li></ul><ul><li>Use at least four alternatives </li></ul><ul><li>Avoid grammatical / sentence structural cues </li></ul><ul><li>Highlight the negative in negatively stated stems </li></ul>
  56. 56. <ul><li>Alternatives are the same format and length </li></ul><ul><li>All alternatives should be plausible </li></ul><ul><li>Avoid answer patterns </li></ul>
  57. 57. Is the question any good? <ul><li>Student who knows the content will know the answer before reading the alternatives </li></ul>
  58. 58. <ul><li>“ Your goal is not to trick students or require them to make difficult judgments about two nearly equal options.” </li></ul>
  59. 59. <ul><li>“ Your goal is to write questions that students who understand the material will answer correctly and students who do not understand will answer incorrectly.” </li></ul><ul><li>Dr. John Johnson Penn State University </li></ul>
  60. 60. Bill Watterson ©1995
  61. 61. True or false <ul><li>Advantages </li></ul><ul><ul><li>Can measure large amounts of basic knowledge </li></ul></ul><ul><ul><li>Students can process many questions quickly </li></ul></ul><ul><li>Disadvantages </li></ul><ul><ul><li>Guessing works </li></ul></ul><ul><ul><li>Need many items for reliability </li></ul></ul>
  62. 62. True & false suggestions <ul><li>Statements are absolutely true or false </li></ul><ul><li>Express item as simply & clearly as possible </li></ul><ul><li>Measure only one idea per item </li></ul><ul><li>Use possible misconceptions as false items </li></ul><ul><li>Avoid negatively stated items </li></ul>
  63. 63. Matching <ul><li>Advantages </li></ul><ul><ul><li>Measures large amounts of knowledge level content </li></ul></ul><ul><ul><li>Useful for factual content areas </li></ul></ul><ul><li>Disadvantages </li></ul><ul><ul><li>Can be time consuming for students </li></ul></ul><ul><ul><li>Does not measure higher learning levels </li></ul></ul>
  64. 64. Matching suggestions <ul><li>Use items that share same foundation of knowledge </li></ul><ul><li>Avoid grammatical cues </li></ul><ul><li>Keep stem list to10-15 items </li></ul><ul><li>Response list should be larger than stem list </li></ul>
  65. 65. Fill-in-the-Blank <ul><li>Advantages </li></ul><ul><ul><li>Good for 5 Ws and H </li></ul></ul><ul><ul><li>Helps minimize guessing </li></ul></ul><ul><li>Disadvantages </li></ul><ul><ul><li>Difficult to assess high levels of learning </li></ul></ul><ul><ul><li>May have more than one correct answer </li></ul></ul><ul><ul><li>Time consuming scoring </li></ul></ul>
  66. 66. Fill-in-the-Blank suggestions <ul><li>Omit only significant items from stem </li></ul><ul><li>Insure only one correct answer </li></ul><ul><li>Avoid grammatical cues </li></ul><ul><li>Blank at stem’s end </li></ul><ul><li>Should count for more points </li></ul>
  67. 67. Things to discuss <ul><li>Possible accommodations </li></ul><ul><li>Student evaluations of test </li></ul><ul><li>Pilot version / full version </li></ul><ul><li>Who takes the test </li></ul><ul><li>Who administers the test </li></ul>

×