0
TRU seminar, Kamloops, Canada, 9 September 2010 1
UK approaches to
quality in e-learning
views from the UK benchmarking pr...
TRU seminar, Kamloops, Canada, 9 September 2010 2
Topics
1. Introduction
2. The four phases of the UK HE
Benchmarking Prog...
TRU seminar, Kamloops, Canada, 9 September 2010 3
1. Introduction
TRU seminar, Kamloops, Canada, 9 September 2010 4
The requirement
from ICDE/Athabasca/
Alberta Quality Council
To produce ...
TRU seminar, Kamloops, Canada, 9 September 2010 5
2. The four phases of the UK HE
Benchmarking Programme
an overview
TRU seminar, Kamloops, Canada, 9 September 2010 6
Benchmarking e-learning
 At national level, started in UK and New Zeala...
TRU seminar, Kamloops, Canada, 9 September 2010 7
Benchmarking e-learning (UK)
 Foreseen in HEFCE e-learning strategy 200...
TRU seminar, Kamloops, Canada, 9 September 2010 8
Documentation – very good
 HE Academy reports on benchmarking
 Evaluat...
TRU seminar, Kamloops, Canada, 9 September 2010 9
UK: benchmarking e-learning
“Possibly more important is for us [HEFCE] t...
TRU seminar, Kamloops, Canada, 9 September 2010 10
Methodologies in UK HE
 There were five methodologies used in UK but o...
TRU seminar, Kamloops, Canada, 9 September 2010 11
Pick&Mix overview
 Focussed on e-learning, not general pedagogy
 Draw...
TRU seminar, Kamloops, Canada, 9 September 2010 12
Pick&Mix history
 Initial version developed in early 2005 in response
...
TRU seminar, Kamloops, Canada, 9 September 2010 13
Pick&Mix
Criteria and metrics
TRU seminar, Kamloops, Canada, 9 September 2010 14
Criteria
 Criteria are “statements of practice” which are
scored into ...
TRU seminar, Kamloops, Canada, 9 September 2010 15
Pick&Mix: 20 core criteria
 Removed any not specific to e-learning
– I...
TRU seminar, Kamloops, Canada, 9 September 2010 16
Pick&Mix Scoring
Use a 6-point scale (1-6)
– 5 (cf Likert, MIT90s leve...
TRU seminar, Kamloops, Canada, 9 September 2010 17
Pick&Mix System: summary
Has taken account of “best of breed”
schemes
...
TRU seminar, Kamloops, Canada, 9 September 2010 18
Institutional competences
 University of Leicester used Pick&Mix in th...
TRU seminar, Kamloops, Canada, 9 September 2010 19
Pick&Mix
Three sample criteria
TRU seminar, Kamloops, Canada, 9 September 2010 20
P01 “Adoption” (Rogers)
1. Innovators only
2. Early adopters taking it ...
TRU seminar, Kamloops, Canada, 9 September 2010 21
P10 “Training”
1. No systematic training for e-learning
2. Some systema...
TRU seminar, Kamloops, Canada, 9 September 2010 22
P05 “Accessibility”
1. VLE and e-learning material are not accessible
2...
TRU seminar, Kamloops, Canada, 9 September 2010 23
Other methodologies
Members of the BELA team have run three
other meth...
TRU seminar, Kamloops, Canada, 9 September 2010 24
National indicators
Pick&Mix is mapped to the HEFCE
Measures of Succes...
TRU seminar, Kamloops, Canada, 9 September 2010 25
Comparative work
A databank of scores from 10 HEIs is
public in anonym...
TRU seminar, Kamloops, Canada, 9 September 2010 26
Carpets
3.5Quality Enhancement
2.7Decisions/Programmes
2.1Staff Recogni...
TRU seminar, Kamloops, Canada, 9 September 2010 27
Supplementary criteria - examples
 IT reliability
 Market research, c...
TRU seminar, Kamloops, Canada, 9 September 2010 28
Local criteria
Institutions can track their own “local
criteria”
But ...
TRU seminar, Kamloops, Canada, 9 September 2010 29
Slices (departments etc)
 As well as benchmarking the whole institutio...
TRU seminar, Kamloops, Canada, 9 September 2010 30
Evidence and Process
Iterative Self-Review
for public criterion systems
TRU seminar, Kamloops, Canada, 9 September 2010 31
The Iterative Self-Review Process
 For all the methodologies we deploy...
TRU seminar, Kamloops, Canada, 9 September 2010 32
Iterative Self-Review details
 Introductory meeting
 Initial collecti...
TRU seminar, Kamloops, Canada, 9 September 2010 33
How to handle evidence
Have a “file” for each criterion
Institutions ...
TRU seminar, Kamloops, Canada, 9 September 2010 34
Peer review
Peer review exists in the Iterated Self
Review model:
– Sp...
TRU seminar, Kamloops, Canada, 9 September 2010 35
Distance Learning
Benchmarking Club
A work package in the JISC Curricu...
TRU seminar, Kamloops, Canada, 9 September 2010 36
Re.ViCa
(Review of Virtual Campuses)
 Project supported by the Europea...
TRU seminar, Kamloops, Canada, 9 September 2010 37
Members
University of Leicester (UK)
University of Liverpool (UK)
Un...
TRU seminar, Kamloops, Canada, 9 September 2010 38
Process
Institutions were supposed to work in a
virtual cohort using t...
TRU seminar, Kamloops, Canada, 9 September 2010 39
4. Implications for
QA in e-learning
My thoughts
(as proposed to ENQA i...
TRU seminar, Kamloops, Canada, 9 September 2010 40
Too many concepts
Benchmarking
Standards?
Quality
Accreditation
/approv...
TRU seminar, Kamloops, Canada, 9 September 2010 41
My view - the pyramid
Critical Success
Factors (eg Re.ViCa) -
Benchma...
TRU seminar, Kamloops, Canada, 9 September 2010 42
Benchmarking frameworks
 It is implausible that there will be a global...
TRU seminar, Kamloops, Canada, 9 September 2010 43
Concordances
Mappings between systems are hard and
rarely useful (Bacs...
TRU seminar, Kamloops, Canada, 9 September 2010 44
Experience on methodologies
Methodologies do not survive without
regul...
TRU seminar, Kamloops, Canada, 9 September 2010 45
References
A key paper on the international aspects is
“BENCHMARKING E-...
Upcoming SlideShare
Loading in...5
×

Benchmarking-derived approaches to quality in e-learning

828

Published on

The slides from a presentation at Thompson Rivers University, British Columbia, Canada - given on 9 September 2010 in connection with the Distance Learning Benchmarking Club. The Club is a work item in the DUCKLING project at the University of Leicester, one of the projects in the JISC Curriculum Development Programme.

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
828
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • 9 September 2010
  • 9 September 2010
  • Transcript of "Benchmarking-derived approaches to quality in e-learning"

    1. 1. TRU seminar, Kamloops, Canada, 9 September 2010 1 UK approaches to quality in e-learning views from the UK benchmarking programmes, Re.ViCa project on critical success factors and the Distance Learning Benchmarking Club Professor Paul Bacsich Matic Media Ltd
    2. 2. TRU seminar, Kamloops, Canada, 9 September 2010 2 Topics 1. Introduction 2. The four phases of the UK HE Benchmarking Programme 3. More recent developments in UK HE benchmarking e-learning 4. Implications for schemes on Quality of e-Learning
    3. 3. TRU seminar, Kamloops, Canada, 9 September 2010 3 1. Introduction
    4. 4. TRU seminar, Kamloops, Canada, 9 September 2010 4 The requirement from ICDE/Athabasca/ Alberta Quality Council To produce an up to date set of guidelines for quality in e-learning, focussed on the needs of providers of distance e-learning, and taking into account developments in other provinces, (nearby) nations of the world, and international projects
    5. 5. TRU seminar, Kamloops, Canada, 9 September 2010 5 2. The four phases of the UK HE Benchmarking Programme an overview
    6. 6. TRU seminar, Kamloops, Canada, 9 September 2010 6 Benchmarking e-learning  At national level, started in UK and New Zealand – Soon spread to Australia – Not closely linked initially to quality agenda  At European level, developments include E-xcellence and UNIQUe – Some earlier work from OBHE, ESMU etc – but not in “public criterion” mode – Later, developments in other projects – Increasingly, links made to quality agenda
    7. 7. TRU seminar, Kamloops, Canada, 9 September 2010 7 Benchmarking e-learning (UK)  Foreseen in HEFCE e-learning strategy 2005  Higher Education Academy (HEA) oversaw it  Four phases – 82 institutions – 5 methodologies  Two consultant teams – BELA and OBHE  Justified entry to HEA Pathfinder and Enhancement National initiatives - and useful for JISC initiatives also (Curriculum Design etc)  Can be leveraged into update of learning and teaching strategy (e.g. Leicester U)
    8. 8. TRU seminar, Kamloops, Canada, 9 September 2010 8 Documentation – very good  HE Academy reports on benchmarking  Evaluator reports on each phase  Consultant team reports on each phase  Conference papers (EADTU/ICDE each year – and ALT-C etc)  Definitive book chapter (to appear)  HE Academy blog and wiki (web 2.0)  Specific HEI blogs and some public reports  http://elearning.heacademy.ac.uk/wiki/index.php/ Bibliography_of_benchmarking
    9. 9. TRU seminar, Kamloops, Canada, 9 September 2010 9 UK: benchmarking e-learning “Possibly more important is for us [HEFCE] to help individual institutions understand their own positions on e-learning, to set their aspirations and goals for embedding e-learning – and then to benchmark themselves and their progress against institutions with similar goals, and across the sector”
    10. 10. TRU seminar, Kamloops, Canada, 9 September 2010 10 Methodologies in UK HE  There were five methodologies used in UK but only two now have public criteria, are routinely updated and are available for single institutions (to use outside consortia):  Pick&Mix – Used under HEA auspices in 24 UK institutions – Including 4 diverse institutions in Wales – Now being used in a further UK HEI and one in Australia – About to be used by the 7-institution Distance Learning Benchmarking Club (UK, Sweden, Australia, Canada, New Zealand)  eMM – as used in New Zealand and Australia
    11. 11. TRU seminar, Kamloops, Canada, 9 September 2010 11 Pick&Mix overview  Focussed on e-learning, not general pedagogy  Draws on several sources and methodologies – UK and internationally (including US) and from college sector  Not linked to any particular style of e-learning (e.g. distance or on-campus or blended)  Oriented to institutions with notable activity in e-learning  Suitable for desk research as well as “in-depth” studies  Suitable for single- and multi-institution studies
    12. 12. TRU seminar, Kamloops, Canada, 9 September 2010 12 Pick&Mix history  Initial version developed in early 2005 in response to a request from Manchester Business School for an international competitor study  Since then, refined by literature search, discussion, feedback, presentations, workshops, concordance studies and four phases of use – fifth and sixth phases now  Forms the basis of the current wording of the Critical Success Factors scheme for the EU Re.ViCa project
    13. 13. TRU seminar, Kamloops, Canada, 9 September 2010 13 Pick&Mix Criteria and metrics
    14. 14. TRU seminar, Kamloops, Canada, 9 September 2010 14 Criteria  Criteria are “statements of practice” which are scored into a number of performance levels from bad/nil to excellent  It is wisest if these statements are in the public domain – to allow analysis & refinement  The number of criteria is crucial  Pick&Mix currently has a core of 20 – based on analysis from the literature (ABC, BS etc) and experience in many senior mgt scoring meetings
    15. 15. TRU seminar, Kamloops, Canada, 9 September 2010 15 Pick&Mix: 20 core criteria  Removed any not specific to e-learning – Including those in general quality schemes (QAA in UK)  Careful about any which are not provably success factors  Left out of the core were some criteria where there was not yet UK consensus  Institutions will wish to add some to monitor their KPIs and objectives. Recommended no more than 6. – Pick&Mix now has over 70 supplementary criteria to choose from – more can be constructed or taken from other schemes  These 20 have stood the test of four phases of benchmarking with only minor changes of wording – originally 18 - two were split to make 20
    16. 16. TRU seminar, Kamloops, Canada, 9 September 2010 16 Pick&Mix Scoring Use a 6-point scale (1-6) – 5 (cf Likert, MIT90s levels) plus 1 more for “excellence” Contextualised by “scoring commentary” There are always issues of judging progress especially “best practice” The 6 levels are mapped to 4 colours in a “traffic lights” system – red, amber, olive, green
    17. 17. TRU seminar, Kamloops, Canada, 9 September 2010 17 Pick&Mix System: summary Has taken account of “best of breed” schemes Output and student-oriented aspects Methodology-agnostic but uses underlying approaches where useful (e.g. Chickering & Gamson, Quality on the Line, MIT90s) Requires no long training course to understand
    18. 18. TRU seminar, Kamloops, Canada, 9 September 2010 18 Institutional competences  University of Leicester used Pick&Mix in the very first phase of the HEA programme – And two phases of re-benchmarking  Other universities with strong competence (with approved HEA Consultants) are University of Derby and University of Chester  Several other universities have done excellent work and produced public papers and reports (e.g. Northumbria, Worcester)
    19. 19. TRU seminar, Kamloops, Canada, 9 September 2010 19 Pick&Mix Three sample criteria
    20. 20. TRU seminar, Kamloops, Canada, 9 September 2010 20 P01 “Adoption” (Rogers) 1. Innovators only 2. Early adopters taking it up 3. Early adopters adopted; early majority taking it up 4. Early majority adopted; late majority taking it up 5. All taken up except laggards, who are now taking it up (or retiring or leaving) 6. First wave embedded, second wave under way (e.g. m-learning after e-learning)
    21. 21. TRU seminar, Kamloops, Canada, 9 September 2010 21 P10 “Training” 1. No systematic training for e-learning 2. Some systematic training, e.g. in some projects and departments 3. Uni-wide training programme but little monitoring of attendance or encouragement to go 4. Uni-wide training programme, monitored and incentivised 5. All staff trained in VLE use, training appropriate to job type – and retrained when needed 6. Staff increasingly keep themselves up to date in a “just in time, just for me” fashion except in situations of discontinuous change
    22. 22. TRU seminar, Kamloops, Canada, 9 September 2010 22 P05 “Accessibility” 1. VLE and e-learning material are not accessible 2. VLE and much e-learning material conform to minimum standards of accessibility 3. VLE and almost all e-learning material conform to minimum standards of accessibility 4. VLE and all e-learning material conform to at least minimum standards of accessibility, much to higher standards 5. VLE and e-learning material are accessible, and key components validated by external agencies 6. Strong evidence of conformance with letter & spirit of accessibility in all countries where students study
    23. 23. TRU seminar, Kamloops, Canada, 9 September 2010 23 Other methodologies Members of the BELA team have run three other methodologies: – MIT90s, eMM and ELTI for HE Academy And analysed most others: – Most US and European methodologies were analysed  QoL, E-xcellence, BENVIC, OBHE Insights from other methodologies are fed into Pick&Mix to improve it
    24. 24. TRU seminar, Kamloops, Canada, 9 September 2010 24 National indicators Pick&Mix is mapped to the HEFCE Measures of Success (England) Similar mappings were done for the Welsh Indicators of Success – draft and final and for the Becta Balanced Scorecard (for colleges)
    25. 25. TRU seminar, Kamloops, Canada, 9 September 2010 25 Comparative work A databank of scores from 10 HEIs is public in anonymous form Because each criterion is stable in concept, longitudinal comparisons (across time) are also possible – Old criteria are withdrawn if no longer relevant and new criteria introduced (e.g for Web 2.0 and work-based learning) – Several HEIs have done re-benchmarking
    26. 26. TRU seminar, Kamloops, Canada, 9 September 2010 26 Carpets 3.5Quality Enhancement 2.7Decisions/Programmes 2.1Staff Recognition 2.8Quality Assurance 3.3Tech Support to Staff 2.9Organisation 3.4Evaluation (e-learning) 2.7Planning Annually 1.4Costing 1.6Academic Workload 3.1Training 2.0Learning Material 2.9Pedagogy 3.4Decisions/Projects 3.9e-Learning Strategy 2.0Accessibility 2.5Usability 2.8Tools 5.1VLE stage 3.6Adoption AvIHGFEDCBACriterion name 3.5Quality Enhancement 2.7Decisions/Programmes 2.1Staff Recognition 2.8Quality Assurance 3.3Tech Support to Staff 2.9Organisation 3.4Evaluation (e-learning) 2.7Planning Annually 1.4Costing 1.6Academic Workload 3.1Training 2.0Learning Material 2.9Pedagogy 3.4Decisions/Projects 3.9e-Learning Strategy 2.0Accessibility 2.5Usability 2.8Tools 5.1VLE stage 3.6Adoption AvIHGFEDCBACriterion name
    27. 27. TRU seminar, Kamloops, Canada, 9 September 2010 27 Supplementary criteria - examples  IT reliability  Market research, competitor research  IPR  Research outputs from e-learning  Help Desk  Management of student expectations  Student satisfaction  Web 2.0 pedagogy
    28. 28. TRU seminar, Kamloops, Canada, 9 September 2010 28 Local criteria Institutions can track their own “local criteria” But this is rarely done – It is actually very hard to craft good criterion statements
    29. 29. TRU seminar, Kamloops, Canada, 9 September 2010 29 Slices (departments etc)  As well as benchmarking the whole institution, it is wise to look at a few “slices”:  Schools, Faculties,, Programmes…  Useful to give a context to scores  Do not do too many  Slices need not be organisational – Distance learning… – Thematic or dimensional slices like HR, costs…  Most other systems also now use this approach
    30. 30. TRU seminar, Kamloops, Canada, 9 September 2010 30 Evidence and Process Iterative Self-Review for public criterion systems
    31. 31. TRU seminar, Kamloops, Canada, 9 September 2010 31 The Iterative Self-Review Process  For all the methodologies we deployed, we use an Iterative Self-Review Process  The methodologies do NOT require it – it was what our UK institutions desired, for all the public criterion systems – strong resistance to documentary review  It encourages a more senior level of participation from the institution: the result is theirs, not the assessors  It allows them to get comfortable with the criteria as they apply to their institution  And move directly to implementation of change  But it selects against complex methodologies  And requires more effort from assessors
    32. 32. TRU seminar, Kamloops, Canada, 9 September 2010 32 Iterative Self-Review details  Introductory meeting  Initial collection of evidence  Selection of supplementary criteria  Mid-process meeting  Further collection of evidence  Scoring rehearsal meeting  Final tweaks on and chasing of evidence  Scoring meeting  Reflection meeting – to move to change
    33. 33. TRU seminar, Kamloops, Canada, 9 September 2010 33 How to handle evidence Have a “file” for each criterion Institutions normally group criteria according to their own L&T strategy or in terms of “owning” departments – We also supply some standard groupings, e.g. based on MIT90s, but few use these
    34. 34. TRU seminar, Kamloops, Canada, 9 September 2010 34 Peer review Peer review exists in the Iterated Self Review model: – Specialist assessors (normally two nowadays) have experience in the sector – Often, the benchmarking is done in a benchmarking cohort and the leaders of each HEI in the cohort form a peer group
    35. 35. TRU seminar, Kamloops, Canada, 9 September 2010 35 Distance Learning Benchmarking Club A work package in the JISC Curriculum Delivery project DUCKLING at the University of Leicester A number (7) of institutions in UK and beyond will be benchmarked this year – And again next year (Sept-Oct 2010) – The aim is to baseline and then measure incremental progress in e-learning
    36. 36. TRU seminar, Kamloops, Canada, 9 September 2010 36 Re.ViCa (Review of Virtual Campuses)  Project supported by the European Union under the Lifelong Learning Programme - Erasmus/Virtual Campus – With International Advisory Committee  Database of countries, agencies and Programmes (500)  Nine case studies  Set of 17 Critical Success Factors developed after wide international consultation – now embedded in Pick&Mix scheme  Organised post-secondary e-learning initiatives are found across the “G-100” (all except the Least Developed Countries)
    37. 37. TRU seminar, Kamloops, Canada, 9 September 2010 37 Members University of Leicester (UK) University of Liverpool (UK) University of Southern Queensland (Australia) Massey University (NZ) Thompson Rivers University (Canada) Lund University (Sweden) KTH (Sweden)
    38. 38. TRU seminar, Kamloops, Canada, 9 September 2010 38 Process Institutions were supposed to work in a virtual cohort using teleconferencing – in fact f2f meetings have been key Pick&Mix is used – with an adjusted set of Core Criteria to take account of: – Updated analysis of earlier benchmarking phases – Critical Success Factors for large dual-mode institutions (Re.ViCa) – The need for expeditious working
    39. 39. TRU seminar, Kamloops, Canada, 9 September 2010 39 4. Implications for QA in e-learning My thoughts (as proposed to ENQA in 2009)
    40. 40. TRU seminar, Kamloops, Canada, 9 September 2010 40 Too many concepts Benchmarking Standards? Quality Accreditation /approval /kitemarking Critical Success Factors E-learning is only a small part of the quality process – how can agencies and assessors handle five variants of the concept across many separate methodologies?
    41. 41. TRU seminar, Kamloops, Canada, 9 September 2010 41 My view - the pyramid Critical Success Factors (eg Re.ViCa) - Benchmarking ------ (eg Pick&Mix) Quality ---------------- (eg QA-QE SIG in UK) Detailed pedagogic guidelines ---------- Criteria are placed at different layers in the pyramid depending on their “level” Leadership level Senior managers
    42. 42. TRU seminar, Kamloops, Canada, 9 September 2010 42 Benchmarking frameworks  It is implausible that there will be a global scheme or even continent-wide schemes for benchmarking  But common vocabulary and principles can be enunciated – e.g. for public criterion systems: – Criteria should be public, understandable, concise and relatively stable – and not politicised or fudged – Criteria choice should be justified from field experience and the literature – Core and supplementary criteria should be differentiated for each jurisdiction – Core criteria should be under 40 in number – The number of scoring levels should be 4, 5 or 6
    43. 43. TRU seminar, Kamloops, Canada, 9 September 2010 43 Concordances Mappings between systems are hard and rarely useful (Bacsich and Marshall, passim) Concordances of systems are easier and helpful – e.g. to reduce the burden of benchmarking with a new methodology – Such approaches are used in the Distance Learning Benchmarking Club – for E-xcellence+/ESMU and ACODE
    44. 44. TRU seminar, Kamloops, Canada, 9 September 2010 44 Experience on methodologies Methodologies do not survive without regular updating by a design authority – this is difficult in a leaderless group context Forking of methodologies needs dealt with by folding updates back to the core system – otherwise survival is affected Complex methodologies do not survive well A public criterion system allows confidence, transparency, and grounding in institutions
    45. 45. TRU seminar, Kamloops, Canada, 9 September 2010 45 References A key paper on the international aspects is “BENCHMARKING E-LEARNING IN UK UNIVERSITIES: LESSONS FROM AND FOR THE INTERNATIONAL CONTEXT”, in Proceedings of the ICDE conference M-2009 at http://www.ou.nl/Docs/Campagnes/ICDE2009/Papers/Final_Paper_338Bacsich.pdf. A specific chapter on the UK HE benchmarking programme methodologies is: “Benchmarking e-learning in UK universities – the methodologies”, in Mayes, J.T., Morrison, D., Bullen, P., Mellar, H., and Oliver, M.(Eds.) Transformation in Higher Education through Technology-Enhanced Learning, York: Higher Education Academy, 2009 http://elearning.heacademy.ac.uk/wiki/index.php/Bibliography_of_benchmarking
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×