Your SlideShare is downloading. ×
Assessment re-capitulated and questioned
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Assessment re-capitulated and questioned

187
views

Published on

Hunting assumptions about assessment; an up-dated replacement for a 2009 version.

Hunting assumptions about assessment; an up-dated replacement for a 2009 version.

Published in: Education, Technology

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
187
On Slideshare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Assessment Recap… … as ever with added complications! James Atherton Unit 3: 8 April 2014
  • 2. The blog page to which some notes refer is at: http://pce2013.blogspot.co.uk/2014/04/8-april-back-to-assessment.html
  • 3. Why? Requirements Approaches Assessment 08/04/2014 - v2 Feedback (Formative) Accreditation (Summative) Validity Reliability Fairness (Security) Norm-referenced Criterion-referenced Ipsative
  • 4. • “Assessment is central because it frames student learning” (Oxford Brookes U. Position Paper on Assessment, n.d.) • “From our students’ point of view, assessment always defines the actual curriculum” (Ramsden P,1992)
  • 5. • “Assessment is central because it frames student learning” (Oxford Brookes U. Position Paper on Assessment, n.d.) • “From our students’ point of view, assessment always defines the actual curriculum” (Ramsden P,1992) Not for nothing do students invariably cry, “Are we being assessed on this?”
  • 6. • “Assessment is central because it frames student learning” (Oxford Brookes U. Position Paper on Assessment, n.d.) • “From our students’ point of view, assessment always defines the actual curriculum” (Ramsden P,1992) Not for nothing do students invariably cry, “Are we being assessed on this?” And if the answer is “No”, then the learning is automatically worthless
  • 7. It is assessment which turns; • presenting • story-telling • demonstrating • … into teaching
  • 8. • Assessment of learning • (summative) • Assessment for learning • (formative) Which is privileged?
  • 9. • Assessment of learning • (summative) • Assessment for learning • (formative) Which is privileged? Accreditation and qualifications— important but not the whole story
  • 10. • Assessment of learning • (summative) • Assessment for learning • (formative) Which is privileged? Accreditation and qualifications— important but not the whole story Importance increasingly recognised in the rhetoric of education, if not the practice…
  • 11. • Assessment of learning • (summative) • Assessment for learning • (formative) Which is privileged? Accreditation and qualifications— important but not the whole story Importance increasingly recognised in the rhetoric of education, if not the practice… See link in notes
  • 12. Formative Summative but…
  • 13. Formative Summative but… All assessment, including most of what is designed for summative purposes, can be used formatively
  • 14. Formative Summative but… All assessment, including most of what is designed for summative purposes, can be used formatively… …but only if you can get the students to take their work back and read the
  • 15. PresenterPresenter AudienceLearner Information Feedback Teacher Teacher/ learner Learner/ teacher
  • 16. Experiential environment Theoretical representation Goal-oriented behaviour Conceptual representation Teacher Student Articulation/ Re-articulation Reflection/ Adaptation Reflection/ Adaptation Action/ Feedback Based on Laurillard, 2001 This is Diana Laurillard’s elaboration of the previous graphic—link in the notes
  • 17. Effect-size 2 • An effect-size of 1.0 indicates an increase of one standard deviation, typically associated with advancing [students’] achievement by one year, improving the rate of learning by 50%, or a correlation between some variable (e.g., amount of homework) and achievement of approximately .50. When implementing a new program, an effect-size of 1.0 would mean that approximately 95% of outcomes positively enhance achievement, or average students receiving that treatment would exceed 84% of students not receiving that treatment. Hattie, 1999:4 (my emphasis) Just a reminder of Hattie—now familiar territory, with a link in the notes.
  • 18. Interventions and Effect Size 0 50 100 150 200 250 300 350 400 -0 .8 -0 .6 -0 .4 -0 .2 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 Effect Size No.ofstudies
  • 19. Above average effect size Influence Effect Size •Feedback 1.13 • Students' prior cognitive ability 1.04 • Instructional quality 1.00 • Direct instruction .82 • Remediation/feedback .65 • Students' disposition to learn .61 • Class environment .56 • Challenge of Goals .52 • Peer tutoring .50 • Mastery learning .50 • Homework .43 • Teacher Style .42 • Questioning .41
  • 20. • Feed-back, or • Feed-forward ? Phil Race is the principal advocate for feed-forward. Link in notes
  • 21. Why? Requirements Approaches Assessment 08/04/2014 - v2 Feedback (Formative) Accreditation (Summative) Validity Reliability Fairness (Security) Norm-referenced Criterion-referenced Ipsative
  • 22. Requirements… • Validity: the assessment measures what it is supposed to.
  • 23. •What mismatches have you come across? • (i.e. between desired outcomes and methods of assessment)
  • 24. The boundary is constantly moving Common observation suggests the belief is unfounded; we seldom find hard evidence of such correlations. Instead, we rely on test results for want of anything better. In any event, the skills required to perform well on school examinations may not be the same skills required to perform adequately in the situations the school trains people for. (Becker, 1972) We seldom argue that conventional tests measure the actual skills students are supposed to have learned. Rather, we believe that the test, … is … highly correlated with the ability to exercise those skills, though the mechanism … is seldom investigated or demonstrated. Link to full paper in the notes
  • 25. Reference points for Assessment • Competitive (“Norm-referenced”) See link in notes
  • 26. Norm-Referenced Assessment 0 5 10 15 20 25 0 35 45 55 65 75 85 95 105 115 125 135 145 155 165 %ofcohort Mean This curve is based on a real reference curve used for the standardisation of IQ tests, which is why it is not quite as graceful as a classic “bell-curve”. See link in notes.
  • 27. •When might it be appropriate to use norm-referenced assessment in a PCE setting?
  • 28. Reference points for Assessment (2) • Competitive (“Norm-referenced”) • Competence (Criterion-referenced) [and • Personal best (Ipsative)]
  • 29. Criterion-Referencing Threshold of competence Object is to ensure minimum level of competence attained One problem is whether that threshold effectively sets a cap on the level of achievement: we briefly discussed the question of inability to deal with exceptional genius in the arts.
  • 30. A complication of “competence” Threshold of Competence Competence is as much a matter of consistency as of skill — what level of consistency is “competent”?
  • 31. Answer Facts Stimulus Idea Convergent thinking - science and technology(?) Divergent thinking - arts and humanities(?) See link in notes
  • 32. •How might you define achievement of intended outcomes in a divergent subject area?
  • 33. Requirements (2)… • Validity: the assessment measures what it is supposed to. • Reliability: it produces consistent and replicable results  Between judges  Between occasions  Between cohorts
  • 34. Why? Requirements Approaches Assessment 08/04/2014 - v2 Feedback (Formative) Accreditation (Summative) Validity Reliability Fairness (Security) Norm-referenced Criterion-referenced Ipsative
  • 35. • "Is an indistinct photograph a picture of a person at all? Is it even always an advantage to replace a blurred picture with a sharp one? Isn't the blurred one often exactly what we need?" Wittgenstein (1953) para. 71 • “OK, you’ve got your PhD; now don’t touch anything!” (source unknown) • “I claimed at the start that I would use the term 'ambiguity' to mean anything I liked,” (Empson, 1947; viii)16 Importance of clear criteria?
  • 36. Criteria a. Understanding and familiarity with the subject b. Breadth of reading c. Relevance of content to essay title d. Structure of the argument e. Independence of thought and critical evaluation f. Presentation, including fluency of writing, grammar and spelling g. Citation and referencing How would you weight these? (in relation to an HE1/NQF 4 task in a “soft” discipline)
  • 37. n 33223 erion Without necessarily being highly original, the work shows consistent critical evaluation of sources and ideas, and the author's own considered judgments The argument is pursued clearly and in a balanced manner, supported by evidence and informed judgement Essay addresses title and its underlying themes and assumptions, clearly explicated. Evidence of extensive reading beyond the recommended texts, including critical evaluation of sources Ability to contextualise subject within wider frameworks and critically explore connections Critical evaluation and independent thought are the norm, but are either not consistent or coherent throughout. A clear and generally structured argument let down by some unsound inferences or assertions without evidence Essay addresses title in a focused and organised way with some awareness of any more general issues raised by it. Recommended reading used, together with a range of other sources Familiarity with subject demonstrated, including complex connections between elements Evidence of occasional independent and critical thought, limited by either inconsistent or incoherent reasoning Continuity of argument evident throughout the essay supported by evidence Content clearly addresses set title as given. Effective use made of reading, largely confined to reading list Subject matter adequately understood, with some connections made Critical evaluation limited to expressions of opinion, using mainly second-hand ideas Argument patchy, inconsistent, or proceeding by assertion rather than evidence. Essay generally addresses title, but with lapses, digressions or padding. Limited use made of texts on set reading list Evidence of knowledge, but only basic connections made between items Uncritical and unreflective work, relying on undigested sources. Little clear structure or argument evident. Little or tenuous connection between title and essay content Little or no reading evident, even within the set texts Subject matter treated as set of discrete bits of knowledge Independence of thought and critical evaluation Structure of the argument Relevance of content to essay title Breadth of readingUnderstanding and familiarity with the subject Criterion ECriterion DCriterion CCriterion BCriterion A 1Academic levelEducation, Individual and SocietyEduc 1002 5 4 3 2 1 More (including download) via link in notes.
  • 38. Background of topic
  • 39. Background of topic Knowledge of topic
  • 40. Background of topic Knowledge of topic Research skills
  • 41. Background of topic Knowledge of topic Ability to conduct argument Research skills
  • 42. Background of topic Knowledge of topic Ability to conduct argument Research skills “Hard” and “soft” objectives are discussed in the notes.
  • 43. Too clear and specific? Task-2 An annotated diagram which illustrates how you contribute to the Quality Assurance and Quality Improvement Processes, procedures and initiatives in the area of work you are engaged in. A short report evaluating and reflecting upon this contribution.  Extended diagram(s) showing how quality framework systems work within own teaching area making links to validity and reliability of the data collected.  Comprehensive annotations including links to areas for improvements and research.  Excellent critical analysis of the current issues relating to quality assurance and quality improvement processes in own area of teaching.  Sound application of theory to practice in order to analyse and evaluate the success of the quality procedures (both positive and negative) Applied theory to practice to identify own CPD.  Critical evaluation of the long term impact of QA and QI systems and procedures.  Detailed bibliography with substantial use of journal research sources. Quotes are presented and fully engaged.  Sound structure. Few errors of presentation.
  • 44. Too clear and specific? Task-2 An annotated diagram which illustrates how you contribute to the Quality Assurance and Quality Improvement Processes, procedures and initiatives in the area of work you are engaged in. A short report evaluating and reflecting upon this contribution.  Extended diagram(s) showing how quality framework systems work within own teaching area making links to validity and reliability of the data collected.  Comprehensive annotations including links to areas for improvements and research.  Excellent critical analysis of the current issues relating to quality assurance and quality improvement processes in own area of teaching.  Sound application of theory to practice in order to analyse and evaluate the success of the quality procedures (both positive and negative) Applied theory to practice to identify own CPD.  Critical evaluation of the long term impact of QA and QI systems and procedures.  Detailed bibliography with substantial use of journal research sources. Quotes are presented and fully engaged.  Sound structure. Few errors of presentation. This is an actual rubric from a supposedly Master’s level course.
  • 45. Too clear and specific? Task-2 An annotated diagram which illustrates how you contribute to the Quality Assurance and Quality Improvement Processes, procedures and initiatives in the area of work you are engaged in. A short report evaluating and reflecting upon this contribution.  Extended diagram(s) showing how quality framework systems work within own teaching area making links to validity and reliability of the data collected.  Comprehensive annotations including links to areas for improvements and research.  Excellent critical analysis of the current issues relating to quality assurance and quality improvement processes in own area of teaching.  Sound application of theory to practice in order to analyse and evaluate the success of the quality procedures (both positive and negative) Applied theory to practice to identify own CPD.  Critical evaluation of the long term impact of QA and QI systems and procedures.  Detailed bibliography with substantial use of journal research sources. Quotes are presented and fully engaged.  Sound structure. Few errors of presentation. If you get this detailed and prescriptive, can you possibly hit the level of critical scholarship expected of a graduate level course?
  • 46. Too clear and specific? Task-2 An annotated diagram which illustrates how you contribute to the Quality Assurance and Quality Improvement Processes, procedures and initiatives in the area of work you are engaged in. A short report evaluating and reflecting upon this contribution.  Extended diagram(s) showing how quality framework systems work within own teaching area making links to validity and reliability of the data collected.  Comprehensive annotations including links to areas for improvements and research.  Excellent critical analysis of the current issues relating to quality assurance and quality improvement processes in own area of teaching.  Sound application of theory to practice in order to analyse and evaluate the success of the quality procedures (both positive and negative) Applied theory to practice to identify own CPD.  Critical evaluation of the long term impact of QA and QI systems and procedures.  Detailed bibliography with substantial use of journal research sources. Quotes are presented and fully engaged.  Sound structure. Few errors of presentation. If you get this detailed and prescriptive, can you possibly hit the level of critical scholarship expected of a graduate level course? Despite fatuous usage of the currently fashionable phrases.
  • 47. Requirements (4)… • Validity: the assessment measures what it is supposed to. • Reliability: it produces consistent and replicable results • Fairness • Security
  • 48. Requirements (4)… • Validity: the assessment measures what it is supposed to. • Reliability: it produces consistent and replicable results • Fairness • Security What does “inclusivity” mean in the context of assessment?
  • 49. Why? Requirements Approaches Assessment 08/04/2014 - v2 Feedback (Formative) Accreditation (Summative) Validity Reliability Fairness (Security) Norm-referenced Criterion-referenced Ipsative