Assessment: Achieving improved efficiency, effectiveness, educational integrity, equity and ethical practice Assoc Prof Janice Orrell Education Consultant

What is assessment? Difficult Subjectivity Complex Are comments right? Deciding if students have engaged with the course and met learning outcomes Source of income Grading and ranking Clarify standards Learn about the students Formative – engage and develop understanding Feedback Exams- quality control Credentialing – capacity as a profession Marking – Assessment; different Clinician – information gathering about capacity for understanding of learning outcomes – trigger intervention

Why do we do it? Purpose – defines what it is Process – integral to learning and teaching to guide and support learning (not just feedback) in light of the ‘orderly progression’

Overview What is Assessment? What are some of the problems and challenges associated with assessment? Designing assessment for greater efficiency , effectiveness , educational integrity , equity and ethical practice Feedback, the elusive element. Grading!

What is Assessment?

What are some of the problems and challenges associated with assessment?

Designing assessment for greater efficiency , effectiveness , educational integrity , equity and ethical practice

Feedback, the elusive element.

Grading!

Assessment is diversely regarded as….. Marking, Correcting and Grading Credentialing Sorting the …. from the ……. Directing learning Diagnosing learning Feedback on learning Feedback on teaching Opportunity for critical reflection

Marking, Correcting and Grading

Credentialing

Sorting the …. from the …….

Directing learning

Diagnosing learning

Feedback on learning

Feedback on teaching

Opportunity for critical reflection

In reality…Assessment is… A postscript in teaching and curriculum design An activity to Test what is known Pivotal in learning A common practice routine and ritual Time consuming A negotiated exchange (product for grades) High-stakes - high risk –high fallibility

A postscript in teaching and curriculum design

An activity to Test what is known

Pivotal in learning

A common practice routine and ritual

Time consuming

A negotiated exchange (product for grades)

High-stakes - high risk –high fallibility

Assessment practice is surrounded by uncertainty (Allen 1998) Academics design tasks, award grades and provide feedback that, They feel comfortable with, They believe the student will feel is fair, Can with stand robust scrutiny by colleagues. They make decisions that lead to preferred and expected outcomes using their experience to predict the best and worst outcomes!

Academics design tasks, award grades and provide feedback that,

They feel comfortable with,

They believe the student will feel is fair,

Can with stand robust scrutiny by colleagues.

They make decisions that lead to preferred and expected outcomes using their experience to predict the best and worst outcomes!

Assessment is the HE Triple Blindfolded High Jump!

Much assessment is grounded in: Implicit University and academic disciplinary traditions Tacit wisdom of experience (Schulman: 1989) Source of assessment “expertise” is its own practice. ( Practice Making Practice ; Britzman, 2003) Privacy and not open to scrutiny because of academic freedom Episodic routines and rituals

Implicit University and academic disciplinary traditions

Tacit wisdom of experience (Schulman: 1989)

Source of assessment “expertise” is its own practice. ( Practice Making Practice ; Britzman, 2003)

Privacy and not open to scrutiny because of academic freedom

Episodic routines and rituals

Assessment is high stakes for us all Students are defined by their grades Academic worth is often measured as capacity to grade Institutions depend on assessment outcomes to assure the quality or its programmes and graduates Professions and Society use them as code to recruit new staff.

Students are defined by their grades

Academic worth is often measured as capacity to grade

Institutions depend on assessment outcomes to assure the quality or its programmes and graduates

Professions and Society use them as code to recruit new staff.

Conservatism is evident in much of respected higher education assessment practice Longing for past glories Efforts focus on inventing better mouse traps

Longing for past glories

Efforts focus on inventing better mouse traps

Conservative responses to the challenges Seek assessment processes that are: Fixed, non-negotiable Within known boundaries Predictable Safe

Seek assessment processes that are:

Fixed, non-negotiable

Within known boundaries

Predictable

Safe

Examinations measure knowledge that is known & merely useful instead of really useful

measure knowledge that is known &

merely useful

instead of

really useful

“ Good practice” in assessment underpins fairness Appropriate tasks that encourage learning Clear descriptions of grading criteria & grade related performance standards Timely feedback that linked to the assessment criteria Decision-making guided by posted criteria and standards when grading (adapted from Carroll 2004)

Appropriate tasks that encourage learning

Clear descriptions of grading criteria & grade related performance standards

Timely feedback that linked to the assessment criteria

Decision-making guided by posted criteria and standards when grading

(adapted from Carroll 2004)

Designing the Assessment Task Efficiency. Is the work of staff and students on the assessment task the best use of time and effort or just helping to spin out a grade? Effectiveness: Does the assessment process account for the diverse ways in which learning occurs and produce the best most enduring learning : Assessment of learning? (postscript) Assessment as learning? (pivot) Educational integrity: Is there curriculum alignment? Equity: Is success possible for all students? Ethical practice: Are students educated about academic integrity and does the design minimize opportunities for plagiarism, cheating collusion or academic fraud?

Efficiency. Is the work of staff and students on the assessment task the best use of time and effort or just helping to spin out a grade?

Effectiveness: Does the assessment process account for the diverse ways in which learning occurs and produce the best most enduring learning :

Assessment of learning? (postscript)

Assessment as learning? (pivot)

Educational integrity: Is there curriculum alignment?

Equity: Is success possible for all students?

Ethical practice: Are students educated about academic integrity and does the design minimize opportunities for plagiarism, cheating collusion or academic fraud?

Assessment Choices Formative or summative Grade or no grade Teach to the test or encourage creativity Weekly work or end of semester exam Individual or group work Essays or authentic projects Who grades…student, peer, profession or teacher Open or closed book exams Who will grade? University or professional practitioners? Set or negotiated tasks? Assess participation in class or not? How many pieces of assessment is enough? Seen or unseen exams Should presentation impact on grades? Should all students have to do the same tasks? How important are “hand-in” dates ? How much feedback?

What attributes would you like to engender in your graduates? Knowledgeable disciplinary specialists? Ethical, efficient & trust worthy professionals? Collaborative, reliable colleagues and team players? Community & corporate citizens? Effective problem solvers? Creative & critical reasoning thinkers? Effective verbal & written communicators ?

Knowledgeable disciplinary specialists?

Ethical, efficient & trust worthy professionals?

Collaborative, reliable colleagues and team players?

Community & corporate citizens?

Effective problem solvers?

Creative & critical reasoning thinkers?

Effective verbal & written communicators ?

To what extent do your current assessments encourage this? Knowledge Skills Dispositions Expert capabilities Professional Behaviour Disciplinary expertise

Assessment as Mapping Learning for the Curriculum? Workplace Context Professional Action Professional dispositions Knowledge Skills

You asked about current practices and policies… Length of assignment per credit point? How to avoid over-assessing? compliance / motivated to learn. Measures or Indicators (How will they best demonstrate the learning you are seeking? Integrating more practical and problem-based assignments How can you make it assessment as learning?

Length of assignment per credit point?

How to avoid over-assessing?

compliance / motivated to learn.

Measures or Indicators (How will they best demonstrate the learning you are seeking?

Integrating more practical and problem-based assignments

How can you make it assessment as learning?

What designs best produce desired learning and graduate capabilities? Authentic Tasks: Assessment as Learning Eg. Library search (critical selection & team activity) Forensic role play (investigation/problem solving) Observation (noticing) Case presentations (theory/practice application) Debates and position statement (evidence-based, opinion formation) Mastery learning (Skill development)

Authentic Tasks: Assessment as Learning

Eg. Library search (critical selection & team activity)

Forensic role play (investigation/problem solving)

Observation (noticing)

Case presentations (theory/practice application)

Debates and position statement (evidence-based, opinion formation)

Mastery learning (Skill development)

+ FEEDBACK!

+ FEEDBACK!

Feedback Does it make a difference? Students always want more Do they use it to critically reflect on their performance? Does the typical feedback provided really inspire critical self reflection?

Does it make a difference?

Students always want more

Do they use it to critically reflect on their performance?

Does the typical feedback provided really inspire critical self reflection?

Spelling? Where is the intro? Analysis? Link? Meaning? Tense? Evidence? Reference? ? ?? ??? ? ! 5/10

Or Great! Well done! Excellent I like it! √ √√ 10/10

Quality of feedback Much is code or cryptic Few students are clear about what they need to do to improve Good students get least feedback Little is contingent up the feedback Student critical reflection and uptake on feedback is largely optional and dependent on personal disposition .

Much is code or cryptic

Few students are clear about what they need to do to improve

Good students get least feedback

Little is contingent up the feedback

Student critical reflection and uptake on feedback is largely optional and dependent on personal disposition .

Assessment & Feedback for Learning Design assessment to INCLUDE the act of feedback and critical reflection from the start ! Feedback has CONSEQUENCIAL expectations ( Utilisation of feedback an essential aspect of the grading process!) Ensure that there is a climate of collaborative critical reflection by using peer and self review (assessment NOT grading)

Design assessment to INCLUDE the act of feedback and critical reflection from the start !

Feedback has CONSEQUENCIAL expectations ( Utilisation of feedback an essential aspect of the grading process!)

Ensure that there is a climate of collaborative critical reflection by using peer and self review (assessment NOT grading)

Good practice in feedback: Delivers high quality information to correct & enhance learning and teaching. Makes grades contingent upon students acting on the information. Encourages positive motivational belief and self esteem. Facilitates student’s capacity for critical reflect and self assessment. Encourages dialogue about learning between teachers & students. Clarifies what is a good performance of both criteria and standards. Opportunity to close the gap between current & desired performance.

Delivers high quality information to correct & enhance learning and teaching.

Makes grades contingent upon students acting on the information.

Encourages positive motivational belief and self esteem.

Facilitates student’s capacity for critical reflect and self assessment.

Encourages dialogue about learning between teachers & students.

Clarifies what is a good performance of both criteria and standards.

Opportunity to close the gap between current & desired performance.

What is NOT Feedback NOT editing. Require them to do so and hand back in. NOT spell checking. Require students to do so and hand back in. NOT grammar checking Point out these issues ONCE and require students to undertake some activity to improve and report back to you once it has been done.

NOT editing. Require them to do so and hand back in.

NOT spell checking. Require students to do so and hand back in.

NOT grammar checking

Point out these issues ONCE and require students to undertake some activity to improve and report back to you once it has been done.

What can be feedback? Test at the beginning of semester on what needs to be known at the end Regular (weekly) independent, on-line, ‘low stakes’ tests for students to completely so that they can assess the accuracy of their understanding of key concepts. Weekly worked examples on problems at the beginning or end of lectures and ask students to compare their (or peers) work with the example Require students to nominate the kind of feedback they are seeking Require students to seek peer feedback & to indicate how they have acted on the peer advice. Provide generic one page handouts summarising how the class has performed overall and describe the qualities of performances that attracted specific grades.

Test at the beginning of semester on what needs to be known at the end

Regular (weekly) independent, on-line, ‘low stakes’ tests for students to completely so that they can assess the accuracy of their understanding of key concepts.

Weekly worked examples on problems at the beginning or end of lectures and ask students to compare their (or peers) work with the example

Require students to nominate the kind of feedback they are seeking

Require students to seek peer feedback & to indicate how they have acted on the peer advice.

Provide generic one page handouts summarising how the class has performed overall and describe the qualities of performances that attracted specific grades.

Use lectures Ask students to complete and hand up: ‘ Minute papers’ in which they outline their current understanding or a core concept ‘ Application papers’ in which they apply a theoretical concept to an everyday instance. ‘ Five main points’ summary of the lecture. Collect and quickly analyse student responses. (You could do 50% if the class is large or get a tutor to analyse them. Provide (i) on-line feedback or (ii) feedback at the next week’s lecture (5 minutes at the beginning, it gets the students there on time) to the class on your assessment or common misunderstandings.

Ask students to complete and hand up:

‘ Minute papers’ in which they outline their current understanding or a core concept

‘ Application papers’ in which they apply a theoretical concept to an everyday instance.

‘ Five main points’ summary of the lecture.

Collect and quickly analyse student responses. (You could do 50% if the class is large or get a tutor to analyse them.

Provide (i) on-line feedback or (ii) feedback at the next week’s lecture (5 minutes at the beginning, it gets the students there on time) to the class on your assessment or common misunderstandings.

Strategies Use a grading rubric to give students feedback on where they sit in a progression of performance standards (say 4-5 levels) on range of attributes (criteria) Invite students to establish so personal learning goals for their next assignment based on this feedback. Invite students to self-assess using the rubric Discuss with students the differences in your assessment and theirs on the different attributes in the rubric.

Use a grading rubric to give students feedback on where they sit in a progression of performance standards (say 4-5 levels) on range of attributes (criteria)

Invite students to establish so personal learning goals for their next assignment based on this feedback.

Invite students to self-assess using the rubric

Discuss with students the differences in your assessment and theirs on the different attributes in the rubric.

Grading! Marking!

Grading!

Marking!

A Test! True or False! Is assessment as simple as giving a grade? 248 X46 --------- 1448 992 ---------- 11368 -------- What grade out of 10 would you give a student for this? What feedback would you give to justify the grade?

Is assessment as simple as giving a grade?

248

X46

---------

1448

992

----------

11368

--------

What grade out of 10 would you give a student for this?

What feedback would you give to justify the grade?

Grade & feedback from other assessors! 9 A simple carrying error, otherwise it’s correct. 8 Simple error deduct 2 7 Three process & layout, errors deduct 1 mark each. 6 Should have known the process 5 Knows the process, but can’t reward carelessness 4 Three errors deduct 2 marks each. 3 Three processes correct 1 mark each 2 Careless work but partly correct 1 For trying 0 Wrong! No marks!

9 A simple carrying error, otherwise it’s correct.

8 Simple error deduct 2

7 Three process & layout, errors deduct 1 mark each.

6 Should have known the process

5 Knows the process, but can’t reward carelessness

4 Three errors deduct 2 marks each.

3 Three processes correct 1 mark each

2 Careless work but partly correct

1 For trying

0 Wrong! No marks!

Interpretation& grading is influenced by non-student performance factors Quality of other student’s work, Assessors’ personality Assessors’ experiences and beliefs Knowledge of the student and classroom events Surface features Introductory paragraph, Presentation factors

Quality of other student’s work,

Assessors’ personality

Assessors’ experiences and beliefs

Knowledge of the student and classroom events

Surface features

Introductory paragraph,

Presentation factors

Grading rubric standards Metacognitive & abstract Limited commitment Extended Abstract 4.Advanced Analysis & application Relative commitment Relational 3.Proficient Knowledge no structure Multiplistic Multi-structural 2.Functional Limited knowledge Dualistic Uni-structural 1.Developing Qualities Perry Biggs Level

What is assessment? Not optimal, dynamic – changed as use to standards Increasing complexity – need to look at whole program Scaffolded learning for students Formative – Summative; Passion; integrated across the program Testing understanding, rather than what they remember Assessment can undermine learning – rituals of memorisation

Why do we do it?