Basic A ssessment Literacy                       Everything             You W anted to Know           A bout Testing But W...
T est builders in empowered desktopFocus just on MC & D BQ s
Smartest kid in class
Sticky Ideas
Q uality data requires a quality testQ uality tests are not easy to create
A ssessment literacy means:5 ld e & n B Mn a  in E pat  l hn  e   ?
A ssessment literacy means:G t rg eedb in r a n b u a e dpnal f mt aot   h in      e o io s dn ah vmn  t etciee et   uU in...
W hat’s a “ good test?”“ The most basic and obvious  answer to that question is that  good tests measure what you  want to...
Q uestions to ask     W hy?                     W hat?  U sers/U ses     H ow?       T argets                  Method     ...
W hy?                     W hat?U sers/U ses     H ow?       T argets                Method               H ow Much?      ...
W hy?Is the Purpose for the assessment  clear?  oW hy are these targets being   assessed?  oW ho will use the information?...
W hy?                     W hat?U sers/U ses     H ow?       T argets                Method               H ow Much?      ...
W hat?A re the Targets clear and  appropriate?  oSpecific & clearly defined   expectations  oC enter on the core of the di...
Four target typesKnowledge oG eorge   W ashington / 1787 / U S   C onstitutionReasoning oEvaluatedifferences between   gov...
Skills  oA nalyze   a primary source documentProducts  oC reate a reasoned argument for    allowing immigration reform
A n assessment that accurately  reflects its intended target is Valid
W hy?                     W hat?U sers/U ses     H ow?       T argets                Method               H ow Much?      ...
H ow?D ifferent targets require different methods  oMatch appropriate method with   intended purpose and target
G eneral categoriesO bjective  oStudent  selects correct response   from several alternatives / supplies a   word or short...
Subjective  oStudent  organizes / presents an   original answer  oH ard to grade / easy to create
O bjective vs. Subjective  o“ Bydefinition, no test can be truly   objective”
Specific categoriesSelected Response  oMultipleC hoice, T rue/False,   Matching, Fill-in-the-blankEssays  oShort   A nswer...
Mark Pett ©2001
Let’s try it!N ame the parts of the human skeleton  oSelected   responseA ppraise a composition on the basis  of its organ...
C ite four examples of satire in  Huckleberry Finn  oShort   answer essayD esign a logo for a web page  oPerformance   tas...
D iagnose a physical ailment  oExtended   response essayList important mental attributes necessary in an athlete  oShort  ...
W hy?                     W hat?U sers/U ses     H ow?       T argets                Method               H ow Much?      ...
H ow much?H ow can we Sample appropriately? o“ A ll assessments are just a small   part of an ‘ideal’ assessment of   infi...
“ H ave I gathered enough  information of the right kind so  that I can draw confident  conclusions about student  achieve...
W hy?                     W hat?U sers/U ses     H ow?       T argets                Method               H ow Much?      ...
Mark Pett ©2001
A ccurate?W hat are possible sources of Interference or Bias?
A ssessment  oPoorly written, poor target-method   match, culturally biased,   inappropriate reading levelA ssessor  oInco...
Student oTest  anxiety, cheat, guess,   interpretation different than intentEnvironment oC old, hot, poor light, time rest...
Eliminating errors due to bias increase the Reliability of our assessments
W hat was new?W hat was re-new?
Steps to Structured ResponsetestPrepare a blueprintSelect material to assessBuild test items from propositions
Step 1Prepare a blueprint
Cnn   ot t    e         Ko &               nw     Poes                       rcs&     Tt                                 o...
Step 2Select material to assess
“ It is always tempting to emphasize   the parts of the course that are   easiest to test, rather than the   parts that ar...
So what’s worthassessing?
Review the C ase StudyW hich test items, if any, need to be  changed?C an you think of a test that included  items you fel...
So . . . ?A select sub-set of all possible important elementsProfessional judgment of relative importance
Step 3Build test items from propositions  o“ C learlystated sentences that   reflect elements of content aligned   to C ou...
ExamplesIn a monarchy, the right to govern is  secured through birth.The executive and legislative  branches differ in tha...
T rue or FalseIn a monarchy, the right to govern is  secured through birth.orIn a monarchy, the right to govern is  secure...
Fill in the BlankH ow is the right to govern secured in a monarchy?
Multiple C hoiceH ow is the right to govern secured in a monarchy? A . W ith military power B. T hrough birth C . By popul...
T rue or FalseThe executive and legislative  branches differ in that the latter is  elected directly by the people.orMembe...
Fill in the BlankElection of members of the executive and legislative branches differ in what way?
Multiple C hoiceElection of members of the executive  and legislative branches differ in what  way? A . Legislators are re...
Franzipanics Q uiz
Specific assessment criteria
Multiple choiceA dvantages  oEffectivemeasurement  oReduces guessing  oC an cover wide range of contentD isadvantages  oD ...
Multiple C hoice suggestionsStem is a clear, explicit & singular  questionU se at least four alternativesA void grammatica...
A lternatives are the same format  and lengthA ll alternatives should be plausibleA void answer patterns
Is the question any good?Student who knows the content will know the answer before reading the alternatives
“ Your goal is not to trick students  or require them to make difficult  judgments about two nearly  equal options.”
“ Your goal is to write questions that  students who understand the  material will answer correctly and  students who do n...
Bill W atterson ©1995
T rue or falseA dvantages  oC anmeasure large amounts of basic   knowledge  oStudents can process many questions   quickly...
T rue & false suggestionsStatements are absolutely true or falseExpress item as simply & clearly as possibleMeasure only o...
MatchingA dvantages  oMeasures   large amounts of   knowledge level content  oU seful for factual content areasD isadvanta...
Matching suggestionsU se items that share same  foundation of knowledgeA void grammatical cuesKeep stem list to10-15 items...
Fill-in-the-BlankA dvantages  oG ood  for 5 W s and H  oH elps minimize guessingD isadvantages  oD ifficult   to assess hi...
Fill-in-the-Blank suggestionsO mit only significant items from  stemInsure only one correct answerA void grammatical cuesB...
T hings to discussPossible accommodationsStudent evaluations of testPilot version / full versionW ho takes the testW ho ad...
Upcoming SlideShare
Loading in …5
×

Assessment literacy 2007

973 views

Published on

Published in: Education, Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
973
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • Sub-set is like a political survey / you only ask 250 people instead 25000
  • Assessment literacy 2007

    1. 1. Basic A ssessment Literacy Everything You W anted to Know A bout Testing But W as N ever T hat Interested
    2. 2. T est builders in empowered desktopFocus just on MC & D BQ s
    3. 3. Smartest kid in class
    4. 4. Sticky Ideas
    5. 5. Q uality data requires a quality testQ uality tests are not easy to create
    6. 6. A ssessment literacy means:5 ld e & n B Mn a in E pat l hn e ?
    7. 7. A ssessment literacy means:G t rg eedb in r a n b u a e dpnal f mt aot h in e o io s dn ah vmn t etciee et uU in t a es et rcst po oe sgh s s n poeso rm t e s m mx u ah vmn aimm ciee et
    8. 8. W hat’s a “ good test?”“ The most basic and obvious answer to that question is that good tests measure what you want to measure, and bad tests do not.” Ben C lay Is This a Trick Question?
    9. 9. Q uestions to ask W hy? W hat? U sers/U ses H ow? T argets Method H ow Much? Sample A ccurate? A void Bias
    10. 10. W hy? W hat?U sers/U ses H ow? T argets Method H ow Much? Sample A ccurate? A void Bias
    11. 11. W hy?Is the Purpose for the assessment clear? oW hy are these targets being assessed? oW ho will use the information? oW hat will the information be used for?
    12. 12. W hy? W hat?U sers/U ses H ow? T argets Method H ow Much? Sample A ccurate? A void Bias
    13. 13. W hat?A re the Targets clear and appropriate? oSpecific & clearly defined expectations oC enter on the core of the discipline oA ligned to program / course / unit level curriculum
    14. 14. Four target typesKnowledge oG eorge W ashington / 1787 / U S C onstitutionReasoning oEvaluatedifferences between government types
    15. 15. Skills oA nalyze a primary source documentProducts oC reate a reasoned argument for allowing immigration reform
    16. 16. A n assessment that accurately reflects its intended target is Valid
    17. 17. W hy? W hat?U sers/U ses H ow? T argets Method H ow Much? Sample A ccurate? A void Bias
    18. 18. H ow?D ifferent targets require different methods oMatch appropriate method with intended purpose and target
    19. 19. G eneral categoriesO bjective oStudent selects correct response from several alternatives / supplies a word or short phrase to answer a question oEasy to grade / hard to create
    20. 20. Subjective oStudent organizes / presents an original answer oH ard to grade / easy to create
    21. 21. O bjective vs. Subjective o“ Bydefinition, no test can be truly objective”
    22. 22. Specific categoriesSelected Response oMultipleC hoice, T rue/False, Matching, Fill-in-the-blankEssays oShort A nswer, Extended ResponsePerformance T asksPersonal C ommunication
    23. 23. Mark Pett ©2001
    24. 24. Let’s try it!N ame the parts of the human skeleton oSelected responseA ppraise a composition on the basis of its organization oShort answer essayD emonstrate safe lab skills
    25. 25. C ite four examples of satire in Huckleberry Finn oShort answer essayD esign a logo for a web page oPerformance taskD escribe the impact of a Bull Market oExtended response essay
    26. 26. D iagnose a physical ailment oExtended response essayList important mental attributes necessary in an athlete oShort response essayC ategorize great A merican fiction writers
    27. 27. W hy? W hat?U sers/U ses H ow? T argets Method H ow Much? Sample A ccurate? A void Bias
    28. 28. H ow much?H ow can we Sample appropriately? o“ A ll assessments are just a small part of an ‘ideal’ assessment of infinite length.”
    29. 29. “ H ave I gathered enough information of the right kind so that I can draw confident conclusions about student achievement?”
    30. 30. W hy? W hat?U sers/U ses H ow? T argets Method H ow Much? Sample A ccurate? A void Bias
    31. 31. Mark Pett ©2001
    32. 32. A ccurate?W hat are possible sources of Interference or Bias?
    33. 33. A ssessment oPoorly written, poor target-method match, culturally biased, inappropriate reading levelA ssessor oIncorrectly scored
    34. 34. Student oTest anxiety, cheat, guess, interpretation different than intentEnvironment oC old, hot, poor light, time restraints, noise, lunch, recess
    35. 35. Eliminating errors due to bias increase the Reliability of our assessments
    36. 36. W hat was new?W hat was re-new?
    37. 37. Steps to Structured ResponsetestPrepare a blueprintSelect material to assessBuild test items from propositions
    38. 38. Step 1Prepare a blueprint
    39. 39. Cnn ot t e Ko & nw Poes rcs& Tt oa l Tr t ae gs U drad ne t sn Ap a n pl t ic ioFr s fom o 1 0 5 1 5gvrmnoe et nS ut e f St c r oU r u 7 5 1 2gvrmnoe et nRh & igts 8 5 1 3r p nibitse os il s ieo cizn fitesTt oa l 2 5 1 5 4 0
    40. 40. Step 2Select material to assess
    41. 41. “ It is always tempting to emphasize the parts of the course that are easiest to test, rather than the parts that are important to test.” Ben C lay Is This a Trick Question?
    42. 42. So what’s worthassessing?
    43. 43. Review the C ase StudyW hich test items, if any, need to be changed?C an you think of a test that included items you felt weren’t worth learning or assessing?H ow do you make decisions about what’s worth assessing?
    44. 44. So . . . ?A select sub-set of all possible important elementsProfessional judgment of relative importance
    45. 45. Step 3Build test items from propositions o“ C learlystated sentences that reflect elements of content aligned to C ourse / U nit EU s & EQ s.” oC reate more than blueprint requires
    46. 46. ExamplesIn a monarchy, the right to govern is secured through birth.The executive and legislative branches differ in that the latter is elected directly by the people.
    47. 47. T rue or FalseIn a monarchy, the right to govern is secured through birth.orIn a monarchy, the right to govern is secured through the approval of those governed.
    48. 48. Fill in the BlankH ow is the right to govern secured in a monarchy?
    49. 49. Multiple C hoiceH ow is the right to govern secured in a monarchy? A . W ith military power B. T hrough birth C . By popular vote D . Through purchase
    50. 50. T rue or FalseThe executive and legislative branches differ in that the latter is elected directly by the people.orMembers of executive and legislative branches are both elected directly by the people.
    51. 51. Fill in the BlankElection of members of the executive and legislative branches differ in what way?
    52. 52. Multiple C hoiceElection of members of the executive and legislative branches differ in what way? A . Legislators are restricted by term limits; presidents are not B. Legislators are elected directly; presidents are not. C . O ne must register to vote for legislators; not for president D . O ne must register to vote for presidents;
    53. 53. Franzipanics Q uiz
    54. 54. Specific assessment criteria
    55. 55. Multiple choiceA dvantages oEffectivemeasurement oReduces guessing oC an cover wide range of contentD isadvantages oD ifficult to create
    56. 56. Multiple C hoice suggestionsStem is a clear, explicit & singular questionU se at least four alternativesA void grammatical / sentence structural cuesH ighlight the negative in negatively stated stems
    57. 57. A lternatives are the same format and lengthA ll alternatives should be plausibleA void answer patterns
    58. 58. Is the question any good?Student who knows the content will know the answer before reading the alternatives
    59. 59. “ Your goal is not to trick students or require them to make difficult judgments about two nearly equal options.”
    60. 60. “ Your goal is to write questions that students who understand the material will answer correctly and students who do not understand will answer incorrectly.” D r. John Johnson Penn State U niversity
    61. 61. Bill W atterson ©1995
    62. 62. T rue or falseA dvantages oC anmeasure large amounts of basic knowledge oStudents can process many questions quicklyD isadvantages oG uessing works o
    63. 63. T rue & false suggestionsStatements are absolutely true or falseExpress item as simply & clearly as possibleMeasure only one idea per itemU se possible misconceptions as false items
    64. 64. MatchingA dvantages oMeasures large amounts of knowledge level content oU seful for factual content areasD isadvantages oC an be time consuming for students oD oes not measure higher learning
    65. 65. Matching suggestionsU se items that share same foundation of knowledgeA void grammatical cuesKeep stem list to10-15 itemsResponse list should be larger than stem list
    66. 66. Fill-in-the-BlankA dvantages oG ood for 5 W s and H oH elps minimize guessingD isadvantages oD ifficult to assess high levels of learning oMay have more than one correct
    67. 67. Fill-in-the-Blank suggestionsO mit only significant items from stemInsure only one correct answerA void grammatical cuesBlank at stem’s endShould count for more points
    68. 68. T hings to discussPossible accommodationsStudent evaluations of testPilot version / full versionW ho takes the testW ho administers the test

    ×