Problematising Assessment

661 views

Published on

A discussion of some issues inherent in the practice of assessment in education; assessment drift and Type 1/Type 2 errors

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
661
On SlideShare
0
From Embeds
0
Number of Embeds
54
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Problematising Assessment

  1. 1. Problematising Assessment (as if it needed it) James Atherton 11 March 13
  2. 2. Problematising Assessment (as if it needed it) Balloons in this James Atherton colour are 11 March 13 additional notes for the online version
  3. 3. This is the outcome to which the session relates3.3: Understand theories, principles and applications of formal and informal assessment
  4. 4. 3.3: Understand theories, principles and applications of formal and informal assessment And if I were teaching Ofsted style I should now recite the objectives...
  5. 5. And for once I will. At the end of thissession you should be– Confused
  6. 6. Confused...but at a higher level than before
  7. 7. It is frowned upon for you to confuse your students. Confused...but at a higher level than before Probably from Kelley, 1951, but attributed to various sources
  8. 8. It is frowned upon for you to ...which may well confuse your be the biggest students. limitation on your teaching. Confused...but at a higher level than before Probably from Kelley, 1951, but attributed to various sources
  9. 9. Confusion can beconstructive in teaching— like ploughing before planting
  10. 10. The Problem of Proxies
  11. 11. 1: The Problem of Proxies ...or surrogates, or substitutes, or stand-ins for the real thing
  12. 12. 1: The Problem of ProxiesAssessment is rife with them, and diluted by ...or surrogates, ortheir use—but we are substitutes, or stuck with them stand-ins for the real thing
  13. 13. whenThis is the essence of intuitive heuristics: faced with a difficult question, we often answer an easier one instead, usually without noticing the substitution Kahneman 2011: 12 Thinking Fast and Slow, Penguin
  14. 14. whenThis is the essence of intuitive heuristics: faced with a difficult question, we often answer an easier one instead, usually without noticing the substitution Kahneman 2011: 12 And this is exactly what we do in assessment
  15. 15. Assess-Content ment
  16. 16. In principle our teaching is governed by content, and theassessment is just to check that it has been learned Assess- Content ment
  17. 17. Assess-Content ment
  18. 18. Assess-Content ment In practice, the demands of the assessment can all to easily take over
  19. 19. Assess- Content ment “Will we be In practice, the demandstested on this?” of the assessment can all too easily take over
  20. 20. PurposesForms Aspects
  21. 21. Purposes Forms AspectsHere are some traditionalperspectives on assessment...
  22. 22. Purposes Forms Aspects• Diagnosis• Feedback• Standards
  23. 23. Purposes Forms Aspects• Diagnosis• Feedback Pre-teaching• Standards
  24. 24. Purposes Forms Aspects• Diagnosis• Feedback During teaching• Standards
  25. 25. Purposes Forms Aspects• Diagnosis• Feedback After teaching• Standards
  26. 26. Purposes Forms Aspects• Validity• Reliability• Fairness• Security
  27. 27. Purposes Forms Aspects• Validity• Reliability• Fairness Traditional criteria for evaluating• Security assessment
  28. 28. Purposes Forms Aspects• Criterion-referenced• Norm-referenced• Ipsative
  29. 29. Purposes Forms Aspects• Criterion-referenced• Norm-referenced• Ipsative Judging against fixed pre-specified criteria
  30. 30. Purposes Forms Aspects• Criterion-referenced• Norm-referenced• Ipsative Judging against other people’s performance
  31. 31. Purposes Forms Aspects• Criterion-referenced• Norm-referenced• Ipsative Judging against your own prior performance: personal best
  32. 32. Purposes• Formative Forms Aspects• Summative ...etc. I could now test you on your knowledge of assessment, but
  33. 33. See what I’ve done? I’ve reduced the whole topic to• 12 items of jargon
  34. 34. Validity• Does it do what it says on the tin?• Is it really assessing the outcome?
  35. 35. What the area of practice actually Let’s look at the requires whole process of assessment drift.
  36. 36. What the area of practice actually Let’s look at the requires whole process of assessment drift. Based on the work of Howard Becker and Etienne Wenger, among others
  37. 37. What the coursesets out to teach
  38. 38. What the course sets out to teachThere’s about 80%overlap—never a perfect fit
  39. 39. What the course actually does teach
  40. 40. What the coursesets out to assess
  41. 41. What the course actually does assess
  42. 42. What the area of practice What the course actually actually does requires assessThat’s all the overlap left
  43. 43. What the area of practice What the course actually actually does requires assess And if you don’t pass very well...
  44. 44. 2: False positives and false negatives:the inherent limitations of testing
  45. 45. 2: False positives and false negatives: the inherent limitations of testing I got into some trouble in this section! The maths are correct, but the problem comes with the labelling of the False Positives (or Type 1 errors) and what happens if you try toeliminate them simply by making the assessment stricter (rather than by targeting it more precisely), so to avoid unnecessary extra confusion, I’ve taken that out of this version.
  46. 46. Take a hundred people and train them for something....
  47. 47. In the real world, 80% are competent at it, and 20% aren’t
  48. 48. Not competent (20%) Competent (80%) In the real world, 80% are competent at it, and 20% aren’t
  49. 49. But we’re not in the real world—we’re in a college—and we have to devise a test to determine who can be let loose on the public
  50. 50. Inaccurate (20%) Accurate (80%)... but tests aren’t always good predictors. You devise the best you can, but it may be only, say, 80% accurate.
  51. 51. Inaccurate (20%) Not competent (20%) Accurate (80%) Competent (80%)So the 80% the test passes are not the same as the 80% who are genuinely competent
  52. 52. False + (4%) False – (16%)True – True + (64%) 16%
  53. 53. False + (4%) False – (16%)True – True + (64%) 16% These are the “true positives”—they passed the test, and so they should have
  54. 54. These are the true negatives: theyfailed and so they False +should have done. (4%) False – (16%) True – True + (64%) 16%
  55. 55. These are theunfortunates: the test failed them,but it was+ False wrong.That is technically (4%) False – (16%) a ‘Type 2’ error. True – True + (64%) 16%
  56. 56. False + (4%) False – (16%)True – True + (64%) 16% These are the ‘Type 1’ errors: they should have failed, but the test passed them.
  57. 57. This test will always be 20% wrong. So you can only reduce the FalsePositives at the cost of increasing the False Negatives. See the notes for more on this.
  58. 58. So I hope you are nowconfused at a higher level than before...
  59. 59. • Becker H (1963) “Why school is a lousy place to learn anything in” reprinted in R J Burgess (ed.) Howard Becker on Education Buckingham; Open University Press, 1998• Kahneman D (2011) Thinking, fast and slow London; Penguin• Kay J (2011) Obliquity; why our goals are best achieved indirectly London; Profile Books• Wenger E (1998) Communities of Practice; learning, meaning and identity Cambridge; C.U.P.
  60. 60. www.bedspce.org.uk/cbc

×