Problematising Assessment

  • 284 views
Uploaded on

A discussion of some issues inherent in the practice of assessment in education; assessment drift and Type 1/Type 2 errors

A discussion of some issues inherent in the practice of assessment in education; assessment drift and Type 1/Type 2 errors

More in: Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
284
On Slideshare
0
From Embeds
0
Number of Embeds
6

Actions

Shares
Downloads
2
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Problematising Assessment (as if it needed it) James Atherton 11 March 13
  • 2. Problematising Assessment (as if it needed it) Balloons in this James Atherton colour are 11 March 13 additional notes for the online version
  • 3. This is the outcome to which the session relates3.3: Understand theories, principles and applications of formal and informal assessment
  • 4. 3.3: Understand theories, principles and applications of formal and informal assessment And if I were teaching Ofsted style I should now recite the objectives...
  • 5. And for once I will. At the end of thissession you should be– Confused
  • 6. Confused...but at a higher level than before
  • 7. It is frowned upon for you to confuse your students. Confused...but at a higher level than before Probably from Kelley, 1951, but attributed to various sources
  • 8. It is frowned upon for you to ...which may well confuse your be the biggest students. limitation on your teaching. Confused...but at a higher level than before Probably from Kelley, 1951, but attributed to various sources
  • 9. Confusion can beconstructive in teaching— like ploughing before planting
  • 10. The Problem of Proxies
  • 11. 1: The Problem of Proxies ...or surrogates, or substitutes, or stand-ins for the real thing
  • 12. 1: The Problem of ProxiesAssessment is rife with them, and diluted by ...or surrogates, ortheir use—but we are substitutes, or stuck with them stand-ins for the real thing
  • 13. whenThis is the essence of intuitive heuristics: faced with a difficult question, we often answer an easier one instead, usually without noticing the substitution Kahneman 2011: 12 Thinking Fast and Slow, Penguin
  • 14. whenThis is the essence of intuitive heuristics: faced with a difficult question, we often answer an easier one instead, usually without noticing the substitution Kahneman 2011: 12 And this is exactly what we do in assessment
  • 15. Assess-Content ment
  • 16. In principle our teaching is governed by content, and theassessment is just to check that it has been learned Assess- Content ment
  • 17. Assess-Content ment
  • 18. Assess-Content ment In practice, the demands of the assessment can all to easily take over
  • 19. Assess- Content ment “Will we be In practice, the demandstested on this?” of the assessment can all too easily take over
  • 20. PurposesForms Aspects
  • 21. Purposes Forms AspectsHere are some traditionalperspectives on assessment...
  • 22. Purposes Forms Aspects• Diagnosis• Feedback• Standards
  • 23. Purposes Forms Aspects• Diagnosis• Feedback Pre-teaching• Standards
  • 24. Purposes Forms Aspects• Diagnosis• Feedback During teaching• Standards
  • 25. Purposes Forms Aspects• Diagnosis• Feedback After teaching• Standards
  • 26. Purposes Forms Aspects• Validity• Reliability• Fairness• Security
  • 27. Purposes Forms Aspects• Validity• Reliability• Fairness Traditional criteria for evaluating• Security assessment
  • 28. Purposes Forms Aspects• Criterion-referenced• Norm-referenced• Ipsative
  • 29. Purposes Forms Aspects• Criterion-referenced• Norm-referenced• Ipsative Judging against fixed pre-specified criteria
  • 30. Purposes Forms Aspects• Criterion-referenced• Norm-referenced• Ipsative Judging against other people’s performance
  • 31. Purposes Forms Aspects• Criterion-referenced• Norm-referenced• Ipsative Judging against your own prior performance: personal best
  • 32. Purposes• Formative Forms Aspects• Summative ...etc. I could now test you on your knowledge of assessment, but
  • 33. See what I’ve done? I’ve reduced the whole topic to• 12 items of jargon
  • 34. Validity• Does it do what it says on the tin?• Is it really assessing the outcome?
  • 35. What the area of practice actually Let’s look at the requires whole process of assessment drift.
  • 36. What the area of practice actually Let’s look at the requires whole process of assessment drift. Based on the work of Howard Becker and Etienne Wenger, among others
  • 37. What the coursesets out to teach
  • 38. What the course sets out to teachThere’s about 80%overlap—never a perfect fit
  • 39. What the course actually does teach
  • 40. What the coursesets out to assess
  • 41. What the course actually does assess
  • 42. What the area of practice What the course actually actually does requires assessThat’s all the overlap left
  • 43. What the area of practice What the course actually actually does requires assess And if you don’t pass very well...
  • 44. 2: False positives and false negatives:the inherent limitations of testing
  • 45. 2: False positives and false negatives: the inherent limitations of testing I got into some trouble in this section! The maths are correct, but the problem comes with the labelling of the False Positives (or Type 1 errors) and what happens if you try toeliminate them simply by making the assessment stricter (rather than by targeting it more precisely), so to avoid unnecessary extra confusion, I’ve taken that out of this version.
  • 46. Take a hundred people and train them for something....
  • 47. In the real world, 80% are competent at it, and 20% aren’t
  • 48. Not competent (20%) Competent (80%) In the real world, 80% are competent at it, and 20% aren’t
  • 49. But we’re not in the real world—we’re in a college—and we have to devise a test to determine who can be let loose on the public
  • 50. Inaccurate (20%) Accurate (80%)... but tests aren’t always good predictors. You devise the best you can, but it may be only, say, 80% accurate.
  • 51. Inaccurate (20%) Not competent (20%) Accurate (80%) Competent (80%)So the 80% the test passes are not the same as the 80% who are genuinely competent
  • 52. False + (4%) False – (16%)True – True + (64%) 16%
  • 53. False + (4%) False – (16%)True – True + (64%) 16% These are the “true positives”—they passed the test, and so they should have
  • 54. These are the true negatives: theyfailed and so they False +should have done. (4%) False – (16%) True – True + (64%) 16%
  • 55. These are theunfortunates: the test failed them,but it was+ False wrong.That is technically (4%) False – (16%) a ‘Type 2’ error. True – True + (64%) 16%
  • 56. False + (4%) False – (16%)True – True + (64%) 16% These are the ‘Type 1’ errors: they should have failed, but the test passed them.
  • 57. This test will always be 20% wrong. So you can only reduce the FalsePositives at the cost of increasing the False Negatives. See the notes for more on this.
  • 58. So I hope you are nowconfused at a higher level than before...
  • 59. • Becker H (1963) “Why school is a lousy place to learn anything in” reprinted in R J Burgess (ed.) Howard Becker on Education Buckingham; Open University Press, 1998• Kahneman D (2011) Thinking, fast and slow London; Penguin• Kay J (2011) Obliquity; why our goals are best achieved indirectly London; Profile Books• Wenger E (1998) Communities of Practice; learning, meaning and identity Cambridge; C.U.P.
  • 60. www.bedspce.org.uk/cbc