REAQ Lester Gilbert Gary Wills Bill Warburton Veronica Gale Report on E-Assessment Quality A JISC project in eLearning October 2009

THE PITCH

The system

Assessment

Quality management

Report questions

REAQ process

The team Management group Lester Gilbert, ECS, University of Southampton , PI Dr Gary Wills, ECS, University of Southampton Expert consultants group Cliff Beevers, Heriot-Watt Paul Booth, Question Tools John Kleeman & Greg Pope, Questionmark Harvey Mellor, IoE Chris Ricketts, Plymouth Denise Whitelock, OU The workers Veronica Gale, Consultant researcher Bill Warburton, iSolutions, University of Southampton

Management group

Lester Gilbert, ECS, University of Southampton , PI

Dr Gary Wills, ECS, University of Southampton

Expert consultants group

Cliff Beevers, Heriot-Watt

Paul Booth, Question Tools

John Kleeman & Greg Pope, Questionmark

Harvey Mellor, IoE

Chris Ricketts, Plymouth

Denise Whitelock, OU

The workers

Veronica Gale, Consultant researcher

Bill Warburton, iSolutions, University of Southampton

Interviewees HEIs Heriot-Watt University of Southampton Newcastle University University of Plymouth The Open University Edinburgh University Institute of Education Cambridge Assessment SQA Question Tools (Network Rail) Vrije Universiteit Amsterdam

HEIs

Heriot-Watt

University of Southampton

Newcastle University

University of Plymouth

The Open University

Edinburgh University

Institute of Education

Cambridge Assessment

SQA

Question Tools (Network Rail)

Vrije Universiteit Amsterdam

Questions asked (1) What denotes ‘high quality’ in summative e-assessment? What steps do you follow to create and use summative e-assessment? How do you ensure e-assessment: reliability, validity, security, and accessibility? How does the process of creating good quality e-assessment differ from the process of creating traditional assessment?

What denotes ‘high quality’ in summative e-assessment?

What steps do you follow to create and use summative e-assessment?

How do you ensure e-assessment:

reliability,

validity,

security, and

accessibility?

How does the process of creating good quality e-assessment differ from the process of creating traditional assessment?

Questions asked (2) Please give us examples of good e-assessment; why are these ‘good’? When you have heard of poor e-assessment, what has made it ‘poor’? What feedback have you received from students who have taken e-assessments? What advice would you give to others using summative e-assessment? What research or other work has informed your thinking about summative e-assessment? What further research would you like to see conducted?

Please give us examples of good e-assessment; why are these ‘good’?

When you have heard of poor e-assessment, what has made it ‘poor’?

What feedback have you received from students who have taken e-assessments?

What advice would you give to others using summative e-assessment?

What research or other work has informed your thinking about summative e-assessment?

What further research would you like to see conducted?

EXPERT EXPECTATIONS

We expected to hear about… … delivery issues…

And hear about… … psychometric measures… … intended learning outcomes …

As well as hearing about… … appropriate standards … … and if we were lucky, capability maturity …

WHAT WE HEARD

We did hear an (awful) lot about… Delivery issues: infrastructure, support, and how things go wrong …

But not much about… Point biserials, Cronbach Alphas, Kuder-Richardsons Content validity, Conformance to ILOs

And hardly anything about… Metrics, Capability maturity Practice standards

And when we did hear… Difficulty coefficients / facility values were often inappropriately used…

SO…

Conclusions Essentially, little support for e-assessment, in the areas of: Tools & toolkits Guidance & advice focused upon quality Exemplars of good practice Little evidence for: Maturity of good practice Expectations (‘demand characteristics’) of quality

Essentially, little support for e-assessment, in the areas of:

Tools & toolkits

Guidance & advice focused upon quality

Exemplars of good practice

Little evidence for:

Maturity of good practice

Expectations (‘demand characteristics’) of quality

Recommendations Tools & toolkits Exemplars of good practice Project to develop item bank quality statistics Suppliers to make quality reports more accessible Workshops, guidance, & advice focused upon quality: Quality management Standards Metrics & psychometrics Capability maturity JISC bids & project outputs to include, as relevant: Psychometric measures Standards Capability maturity modelling

Tools & toolkits

Exemplars of good practice

Project to develop item bank quality statistics

Suppliers to make quality reports more accessible

Workshops, guidance, & advice focused upon quality:

Quality management

Standards

Metrics & psychometrics

Capability maturity

JISC bids & project outputs to include, as relevant:

Psychometric measures

Standards

Capability maturity modelling

THANKS! Comments, questions, …

Comments, questions, …