ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 1
The UK approaches to
quality in e-learning
- as seen from the HE Academ...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 2
Topics
1. Introduction, disclaimers and
acknowledgements
2. The four ph...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 3
1. Introduction, disclaimers
and acknowledgements
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 4
Disclaimer: This talk is not on behalf of
any institution, agency or mi...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 5
Re.ViCa
(Review of Virtual Campuses)
Project supported by the European...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 6
2. The four phases of the UK HE
Benchmarking Programme
an overview
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 7
Benchmarking e-learning
At national level, started in UK and New Zeala...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 8
Benchmarking e-learning (UK)
Foreseen in HEFCE e-learning strategy 200...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 9
Documentation – very good
HE Academy reports on benchmarking
Evaluato...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 10
UK: benchmarking e-learning
“Possibly more important is for us [HEFCE]...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 11
Methodologies in UK HE
There were five methodologies used in UK but o...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 12
Pick&Mix overview
Focussed on e-learning, not general pedagogy
Draws...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 13
Pick&Mix history
Initial version developed in early 2005 in response
...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 14
Pick&Mix
Criteria and metrics
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 15
Criteria
Criteria are “statements of practice” which are
scored into ...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 16
Pick&Mix: 20 core criteria
Removed any not specific to e-learning
– I...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 17
Pick&Mix Scoring
Use a 6-point scale (1-6)
– 5 (cf Likert, MIT90s lev...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 18
Pick&Mix System: summary
Has taken account of “best of breed”
schemes...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 19
Institutional competences
University of Leicester used Pick&Mix in th...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 20
Pick&Mix
Three sample criteria
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 21
P01 “Adoption” (Rogers)
1. Innovators only
2. Early adopters taking it...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 22
P10 “Training”
1. No systematic training for e-learning
2. Some system...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 23
P05 “Accessibility”
1. VLE and e-learning material are not accessible
...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 24
Other methodologies
Members of the BELA team have run three
other met...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 25
National indicators
Pick&Mix is mapped to the HEFCE
Measures of Succe...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 26
Comparative work
A databank of scores from 10 HEIs is
public in anony...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 27
Carpets
3.5Quality Enhancement
2.7Decisions/Programmes
2.1Staff Recogn...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 28
Supplementary criteria - examples
IT reliability
Market research, co...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 29
Local criteria
Institutions can track their own “local
criteria”
But...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 30
Slices (departments etc)
As well as benchmarking the whole institutio...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 31
Evidence and Process
Iterative Self-Review
for public criterion systems
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 32
The Iterative Self-Review Process
For all the methodologies we deploy...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 33
Iterative Self-Review details
Introductory meeting
Initial collectio...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 34
How to handle evidence
Have a “file” for each criterion
Institutions...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 35
Peer review
Peer review exists in the Iterated Self
Review model:
– S...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 36
Distance Learning
Benchmarking Club
A work package in the JISC Curric...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 37
Members
University of Leicester (UK)
University of Liverpool (UK)
U...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 38
Process
Institutions will work in a virtual cohort
using teleconferen...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 39
4. Implications for
QA in e-learning
My thoughts
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 40
Too many concepts
Benchmarking
Standards?
Quality
Accreditation
/appro...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 41
My view - the pyramid
Critical Success
Factors -------------
Benchma...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 42
Benchmarking frameworks
It is implausible that there will be a global...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 43
Concordances
Mappings between systems are hard and
rarely useful (Bac...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 44
Experience on methodologies
Methodologies do not survive without
regu...
ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 45
References
A key paper on the international aspects is
“BENCHMARKING E...
Upcoming SlideShare
Loading in...5
×

Quality in e-learning - a view for ENQA

1,429

Published on

A view from a benchmarking e-learning perspective of how to initiate a synthesis of approaches to quality in e-learning for use Europe-wide within the ENQA Standards and Guidelines

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,429
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
11
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Quality in e-learning - a view for ENQA

  1. 1. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 1 The UK approaches to quality in e-learning - as seen from the HE Academy/JISC benchmarking programmes - and more recent developments including Re.ViCa and the DL benchmarking club Professor Paul Bacsich Matic Media Ltd
  2. 2. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 2 Topics 1. Introduction, disclaimers and acknowledgements 2. The four phases of the UK HE Benchmarking Programme 3. More recent developments in UK HE benchmarking e-learning 4. Implications for schemes on Quality of e-Learning
  3. 3. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 3 1. Introduction, disclaimers and acknowledgements
  4. 4. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 4 Disclaimer: This talk is not on behalf of any institution, agency or ministry – it is a personal expert view Thanks to HE Academy, JISC, EU Lifelong Learning Programme, Manchester Business School and University of Leicester for support - apologies to others omitted
  5. 5. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 5 Re.ViCa (Review of Virtual Campuses) Project supported by the European Union under the Lifelong Learning Programme - Erasmus/Virtual Campus – With International Advisory Committee Database of countries, agencies and Programmes (500) Nine case studies Set of 17 Critical Success Factors developed after wide international consultation – embedded in Pick&Mix scheme Organised post-secondary e-learning initiatives are found across the “G-100” (all except the Least Developed Countries)
  6. 6. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 6 2. The four phases of the UK HE Benchmarking Programme an overview
  7. 7. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 7 Benchmarking e-learning At national level, started in UK and New Zealand – Soon spread to Australia – Not closely linked initially to quality agenda At European level, developments include E-xcellence and UNIQUe – Some earlier work from OBHE, ESMU etc – but not in “public criterion” mode – Later, developments in other projects – Increasingly, links made to quality agenda
  8. 8. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 8 Benchmarking e-learning (UK) Foreseen in HEFCE e-learning strategy 2005 Higher Education Academy (HEA) oversaw it Four phases – 82 institutions – 5 methodologies Two consultant teams – BELA and OBHE Justified entry to HEA Pathfinder and Enhancement National initiatives - and useful for JISC initiatives also (Curriculum Design etc) Can be leveraged into update of learning and teaching strategy (e.g. Leicester U)
  9. 9. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 9 Documentation – very good HE Academy reports on benchmarking Evaluator reports on each phase Consultant team reports on each phase Conference papers (EADTU/ICDE each year – and ALT-C etc) Definitive book chapter (to appear) HE Academy blog and wiki (web 2.0) Specific HEI blogs and some public reports http://elearning.heacademy.ac.uk/wiki/index.php/ Bibliography_of_benchmarking
  10. 10. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 10 UK: benchmarking e-learning “Possibly more important is for us [HEFCE] to help individual institutions understand their own positions on e-learning, to set their aspirations and goals for embedding e-learning – and then to benchmark themselves and their progress against institutions with similar goals, and across the sector”
  11. 11. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 11 Methodologies in UK HE There were five methodologies used in UK but only two now have public criteria, are routinely updated and are available for single institutions (to use outside consortia): Pick&Mix – Used under HEA auspices in 24 UK institutions – Including 4 diverse institutions in Wales – Now being used in a further UK HEI and one in Australia – About to be used by the 7-institution Distance Learning Benchmarking Club (UK, Sweden, Australia, Canada, New Zealand) eMM – as used in New Zealand and Australia
  12. 12. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 12 Pick&Mix overview Focussed on e-learning, not general pedagogy Draws on several sources and methodologies – UK and internationally (including US) and from college sector Not linked to any particular style of e-learning (e.g. distance or on-campus or blended) Oriented to institutions with notable activity in e-learning Suitable for desk research as well as “in-depth” studies Suitable for single- and multi-institution studies
  13. 13. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 13 Pick&Mix history Initial version developed in early 2005 in response to a request from Manchester Business School for an international competitor study Since then, refined by literature search, discussion, feedback, presentations, workshops, concordance studies and four phases of use – fifth and sixth phases now Forms the basis of the current wording of the Critical Success Factors scheme for the EU Re.ViCa project
  14. 14. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 14 Pick&Mix Criteria and metrics
  15. 15. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 15 Criteria Criteria are “statements of practice” which are scored into a number of performance levels from bad/nil to excellent It is wisest if these statements are in the public domain – to allow analysis & refinement The number of criteria is crucial Pick&Mix currently has a core of 20 – based on analysis from the literature (ABC, BS etc) and experience in many senior mgt scoring meetings
  16. 16. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 16 Pick&Mix: 20 core criteria Removed any not specific to e-learning – Including those in general quality schemes (QAA in UK) Careful about any which are not provably success factors Left out of the core were some criteria where there was not yet UK consensus Institutions will wish to add some to monitor their KPIs and objectives. Recommended no more than 6. – Pick&Mix now has over 70 supplementary criteria to choose from – more can be constructed or taken from other schemes These 20 have stood the test of four phases of benchmarking with only minor changes of wording – originally 18 - two were split to make 20
  17. 17. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 17 Pick&Mix Scoring Use a 6-point scale (1-6) – 5 (cf Likert, MIT90s levels) plus 1 more for “excellence” Contextualised by “scoring commentary” There are always issues of judging progress especially “best practice” The 6 levels are mapped to 4 colours in a “traffic lights” system – red, amber, olive, green
  18. 18. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 18 Pick&Mix System: summary Has taken account of “best of breed” schemes Output and student-oriented aspects Methodology-agnostic but uses underlying approaches where useful (e.g. Chickering & Gamson, Quality on the Line, MIT90s) Requires no long training course to understand
  19. 19. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 19 Institutional competences University of Leicester used Pick&Mix in the very first phase of the HEA programme – And two phases of re-benchmarking Other universities with strong competence (with approved HEA Consultants) are University of Derby and University of Chester Several other universities have done excellent work and produced public papers and reports (e.g. Northumbria, Worcester)
  20. 20. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 20 Pick&Mix Three sample criteria
  21. 21. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 21 P01 “Adoption” (Rogers) 1. Innovators only 2. Early adopters taking it up 3. Early adopters adopted; early majority taking it up 4. Early majority adopted; late majority taking it up 5. All taken up except laggards, who are now taking it up (or retiring or leaving) 6. First wave embedded, second wave under way (e.g. m-learning after e-learning)
  22. 22. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 22 P10 “Training” 1. No systematic training for e-learning 2. Some systematic training, e.g. in some projects and departments 3. Uni-wide training programme but little monitoring of attendance or encouragement to go 4. Uni-wide training programme, monitored and incentivised 5. All staff trained in VLE use, training appropriate to job type – and retrained when needed 6. Staff increasingly keep themselves up to date in a “just in time, just for me” fashion except in situations of discontinuous change
  23. 23. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 23 P05 “Accessibility” 1. VLE and e-learning material are not accessible 2. VLE and much e-learning material conform to minimum standards of accessibility 3. VLE and almost all e-learning material conform to minimum standards of accessibility 4. VLE and all e-learning material conform to at least minimum standards of accessibility, much to higher standards 5. VLE and e-learning material are accessible, and key components validated by external agencies 6. Strong evidence of conformance with letter & spirit of accessibility in all countries where students study
  24. 24. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 24 Other methodologies Members of the BELA team have run three other methodologies: – MIT90s, eMM and ELTI for HE Academy And analysed most others: – Most US and European methodologies were analysed  QoL, E-xcellence, BENVIC, OBHE Insights from other methodologies are fed into Pick&Mix to improve it
  25. 25. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 25 National indicators Pick&Mix is mapped to the HEFCE Measures of Success (England) Similar mappings were done for the Welsh Indicators of Success – draft and final and for the Becta Balanced Scorecard (for colleges)
  26. 26. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 26 Comparative work A databank of scores from 10 HEIs is public in anonymous form Because each criterion is stable in concept, longitudinal comparisons (across time) are also possible – Old criteria are withdrawn if no longer relevant and new criteria introduced (e.g for Web 2.0 and work-based learning) – Several HEIs have done re-benchmarking
  27. 27. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 27 Carpets 3.5Quality Enhancement 2.7Decisions/Programmes 2.1Staff Recognition 2.8Quality Assurance 3.3Tech Support to Staff 2.9Organisation 3.4Evaluation (e-learning) 2.7Planning Annually 1.4Costing 1.6Academic Workload 3.1Training 2.0Learning Material 2.9Pedagogy 3.4Decisions/Projects 3.9e-Learning Strategy 2.0Accessibility 2.5Usability 2.8Tools 5.1VLE stage 3.6Adoption AvIHGFEDCBACriterion name 3.5Quality Enhancement 2.7Decisions/Programmes 2.1Staff Recognition 2.8Quality Assurance 3.3Tech Support to Staff 2.9Organisation 3.4Evaluation (e-learning) 2.7Planning Annually 1.4Costing 1.6Academic Workload 3.1Training 2.0Learning Material 2.9Pedagogy 3.4Decisions/Projects 3.9e-Learning Strategy 2.0Accessibility 2.5Usability 2.8Tools 5.1VLE stage 3.6Adoption AvIHGFEDCBACriterion name
  28. 28. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 28 Supplementary criteria - examples IT reliability Market research, competitor research IPR Research outputs from e-learning Help Desk Management of student expectations Student satisfaction Web 2.0 pedagogy
  29. 29. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 29 Local criteria Institutions can track their own “local criteria” But this is rarely done – It is actually very hard to craft good criterion statements
  30. 30. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 30 Slices (departments etc) As well as benchmarking the whole institution, it is wise to look at a few “slices”: Schools, Faculties,, Programmes… Useful to give a context to scores Do not do too many Slices need not be organisational – Distance learning… – Thematic or dimensional slices like HR, costs… Most other systems also now use this approach
  31. 31. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 31 Evidence and Process Iterative Self-Review for public criterion systems
  32. 32. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 32 The Iterative Self-Review Process For all the methodologies we deployed, we use an Iterative Self-Review Process The methodologies do NOT require it – it was what our UK institutions desired, for all the public criterion systems – strong resistance to documentary review It encourages a more senior level of participation from the institution: the result is theirs, not the assessors It allows them to get comfortable with the criteria as they apply to their institution And move directly to implementation of change But it selects against complex methodologies And requires more effort from assessors
  33. 33. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 33 Iterative Self-Review details Introductory meeting Initial collection of evidence Selection of supplementary criteria Mid-process meeting Further collection of evidence Scoring rehearsal meeting Final tweaks on and chasing of evidence Scoring meeting Reflection meeting – to move to change
  34. 34. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 34 How to handle evidence Have a “file” for each criterion Institutions normally group criteria according to their own L&T strategy or in terms of “owning” departments – We also supply some standard groupings, e.g. based on MIT90s, but few use these
  35. 35. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 35 Peer review Peer review exists in the Iterated Self Review model: – Specialist assessors (normally two nowadays) have experience in the sector – Often, the benchmarking is done in a benchmarking cohort and the leaders of each HEI in the cohort form a peer group
  36. 36. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 36 Distance Learning Benchmarking Club A work package in the JISC Curriculum Delivery project DUCKLING at the University of Leicester A number (7) of institutions in UK and beyond will be benchmarked this year – And again next year (Sept-Oct 2010) – The aim is to baseline and then measure incremental progress in e-learning
  37. 37. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 37 Members University of Leicester (UK) University of Liverpool (UK) University of Southern Queensland (Australia) Massey University (NZ) Thompson Rivers University (Canada) Lund University (Sweden) KTH (Sweden)
  38. 38. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 38 Process Institutions will work in a virtual cohort using teleconferencing Pick&Mix will be used – with an adjusted set of Core Criteria to take account of: – Updated analysis of earlier benchmarking phases – Critical Success Factors for large dual-mode institutions – The need for expeditious working
  39. 39. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 39 4. Implications for QA in e-learning My thoughts
  40. 40. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 40 Too many concepts Benchmarking Standards? Quality Accreditation /approval /kitemarking Critical Success Factors E-learning is only a small part of the quality process – how can agencies and assessors handle five variants of the concept across many separate methodologies?
  41. 41. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 41 My view - the pyramid Critical Success Factors ------------- Benchmarking ---- Quality -------------- Detailed pedagogic guidelines ---------- Criteria are placed at different layers in the pyramid depending on their “level” Leadership level Senior managers
  42. 42. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 42 Benchmarking frameworks It is implausible that there will be a global scheme or even continent-wide schemes for benchmarking But common vocabulary and principles can be enunciated – e.g. for public criterion systems: – Criteria should be public, understandable, concise and relatively stable – and not politicised or fudged – Criteria choice should be justified from field experience and the literature – Core and supplementary criteria should be differentiated for each jurisdiction – Core criteria should be under 40 in number – The number of scoring levels should be 4, 5 or 6
  43. 43. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 43 Concordances Mappings between systems are hard and rarely useful (Bacsich and Marshall, passim) Concordances of systems are easier and helpful – e.g. to reduce the burden of benchmarking with a new methodology – Such approaches will be used in the Distance Learning Benchmarking Club – for E-xcellence+/ESMU and ACODE
  44. 44. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 44 Experience on methodologies Methodologies do not survive without regular updating by a design authority – this is difficult in a leaderless group context Forking of methodologies needs dealt with by folding updates back to the core system – otherwise survival is affected Complex methodologies do not survive well A public criterion system allows confidence, transparency, and grounding in institutions
  45. 45. ENQA workshop, Sigtuna, Sweden, 7-8 October 2009 45 References A key paper on the international aspects is “BENCHMARKING E-LEARNING IN UK UNIVERSITIES: LESSONS FROM AND FOR THE INTERNATIONAL CONTEXT”, in Proceedings of the ICDE conference M-2009 at http://www.ou.nl/Docs/Campagnes/ICDE2009/Papers/Final_Paper_338Bacsich.pdf. A specific chapter on the UK HE benchmarking programme methodologies is: “Benchmarking e-learning in UK universities – the methodologies”, in Mayes, J.T., Morrison, D., Bullen, P., Mellar, H., and Oliver, M.(Eds.) Transformation in Higher Education through Technology-Enhanced Learning, York: Higher Education Academy, 2009 (expected late 2009)
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×