Quality, benchmarking and success factors in virtual schools


Published on

Presentation to a workshop joint with Walter Kugemann to the EFQUEL Academy just prior to the EFQUEL Innovation Forum

Published in: Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Quality, benchmarking and success factors in virtual schools

  1. 1. Quality in Virtual Schools and Colleges – a view from VISCED Paul Bacsich (paul.bacsich@sero.co.uk) VISCED – www.virtualschoolsandcolleges.info1 EFQUEL Academy
  2. 2. Introduction2 EFQUEL Academy
  3. 3. Topics• Historical overview• The Pick&Mix system• Supplementary criteria, local criteria and slices• Scoring, evidence and documentation3 EFQUEL Academy
  4. 4. Benchmarking and Quality in e- learning: My part of the story Overview and history4 EFQUEL Academy
  5. 5. Benchmarking e-learning (UK)• Foreseen in HEFCE e-learning strategy• Higher Education Academy oversaw it• Four phases – 82 institutions (2005-07)• Justified entry to Pathfinder and Enhancement programmes - and useful for JISC bids (Curriculum Design etc)• Was leveraged into refresh of learning and teaching strategy (Leicester etc)5 EFQUEL Academy
  6. 6. What is benchmarking? (HEFCE)“*for+ individual institutions [to] understand their own positions on e-learning, to set their aspirations and goals for embedding e- learning… to benchmark themselves and their progress against institutions with similar goals, and across the sector [world]”Can replace e-learning by “distance learning”, “OER”, “podcasting” ,etc6 EFQUEL Academy
  7. 7. Methodologies: Pick&Mix • First used for Manchester Business School 2005 • Has been used in 36 institutions, mainly new universities and university colleges – Including 4 very diverse institutions in Wales (2007-09) – And 5 in the Distance learning benchmarking Club (UK, Sweden, Canada) plus two more in UK (2009-12) – And basis for the Critical Success Factors schemes in two EU projects: Re.ViCa (2007-09: developments fed back into Pick&Mix) and now VISCED (2011-12) – Also an OER version of the scheme ready for use • Wording slightly revised for schools and colleges7 EFQUEL Academy
  8. 8. Pick&Mix Criteria and metrics8 EFQUEL Academy
  9. 9. Criteria• Criteria are “statements of practice” which are scored into a number of performance levels from “bad” to “best” to “excellent”• These statements are in the public domain – to allow analysis, refinement etc (cf.OER)• Pick&Mix has over 80 criteria… – But typically a set of 20-30 is chosen – this range is based on analysis from the literature (ABC, BS etc) and experience in many scoring meetings9 EFQUEL Academy
  10. 10. Pick&Mix Metrics• Use a 6-point scale (1-6) – ignore 6 if you wish – 5 (cf Likert, MIT90s levels) plus 1 more for “excellence”• Backed up by continuous metrics where possible• Also contextualised by narrative• The 6 levels are mapped to 4 colours in a “traffic lights” system (red, amber, olive, green)10 EFQUEL Academy
  11. 11. Pick&Mix Three sample criteria11 EFQUEL Academy
  12. 12. P01 “Adoption” (Rogers) – not used now1. Innovators only2. Early adopters taking it up3. Early adopters adopted; early majority taking it up4. Early majority adopted; late majority taking it up5. All taken up except laggards, who are now taking it up (or retiring or leaving)6. First wave embedded, second wave under way (e.g. m-learning after e-learning)12 EFQUEL Academy
  13. 13. P10 “Training”1. No systematic training for e-learning2. Some systematic training, e.g. in some projects and departments3. Institution-wide training programme but little monitoring of attendance or encouragement to go4. Institution-wide training programme, monitored and incentivised5. All staff trained in VLE use, training appropriate to job type – and retrained when needed6. Staff increasingly keep themselves up to date in a “just in time, just for me” fashion except in situations of discontinuous change13 EFQUEL Academy
  14. 14. P05 “Accessibility” – tough one1. e-learning material and services is not accessible2. Much e-learning material and most services conform to minimum standards of accessibility3. Almost all e-learning material and services conform to minimum standards of accessibility4. All e-learning material and services conform to at least minimum standards of accessibility, much to higher standards5. e-learning material and services are accessible, and key components validated by external agencies6. Strong evidence of conformance with letter & spirit of accessibility in all countries where students study 14 EFQUEL Academy
  15. 15. Pause for questions15 EFQUEL Academy
  16. 16. Extensions, local criteria and slices16 EFQUEL Academy
  17. 17. Extensions• Pick&Mix started with 20 core criteria – but now there are over 80 – and a different core (2.0)• There is room for more supplementary criteria – and further analyses are always being done to ensure topicality• Web 2.0 aspects are already covered• But it is mainly the institutions who have kept and will help to keep the system up to date• And some EU and other projects17 EFQUEL Academy
  18. 18. Supplementary criteria - examples• They include: – IT reliability – Market research, competitor research – IPR – Help Desk – Management of student expectations – Student satisfaction18 EFQUEL Academy
  19. 19. Slices• As well as benchmarking the whole institution it is wise in large institutions to look at a few “slices”:• Useful to give a context to scores• Slices need not be organisational or subject-based – Distance learning (is a natural one) – Thematic or dimensional like HR, costs…19 EFQUEL Academy
  20. 20. Scoring• Scores are usually in the range 1-5, point 6 is exceptional and cannot be planned for• Focus not only on best practice but also bad practice and real practice• Do not try to ensure that your institution scores 5 overall• 6 sometimes has aspects of 120 EFQUEL Academy
  21. 21. Use in Distance Learning(such as Virtual Schools) details EFQUEL Academy 21
  22. 22. Distance learning• A variant “mood” of the default Pick&Mix was developed to benchmark distance learning• This was used in the Distance Learning Benchmarking Club: – University of Leicester – Thompson Rivers University – Lund University – Royal Swedish Technical Institute (KTH)• and also at the University of Gotland• Most recently at University of Northampton EFQUEL Academy 22
  23. 23. Evidence and documentation• Normally institutions want a final report but but the aim is NOT to recreate the QAA-style SED or DATs – the report is an outcome of the benchmarking process not the start of it• Have a “file” for each criterion• Some narrative will be needed• Institutions normally group criteria according to their own L&T strategy or in terms of “owning” departments – We also supply some standard groupings based on MIT90s but few use these23 EFQUEL Academy
  24. 24. Documentary support• There are many reports available – many but not all are public, others are available to those participating in Pick&Mix• The HE Academy wiki “was” a good source of information and the Academy blog structure “was to be” maintained indefinitely…. Not now• We store all Pick&Mix key material separately from the HE Academy• There are now many papers and a bibliography is available24 EFQUEL Academy
  25. 25. Ongoing work• Evolving Pick&Mix into a benchmarking and quality scheme for virtual schools: – With specialist help from EFQUEL – And a research team at KU Leuven – And correlation with iNACOL scheme in US• Version of Pick&Mix ready to benchmark OER uptake (developed with Bieke Schreurs, OUNL)• Variant moods for accessibility and employability (Wales) EFQUEL Academy 25
  26. 26. Thank you for listening email for this work is paul.bacsich@sero.co.uk Benchmarking E-Learning in UK Universities: Lessons from and for the International Context , published in http://www.openpraxis.com/files/Bacsich%20et%20al..pdf http://www.virtualcampuses.eu/index.php/Bibliography_of_benchmarking and http://www.mendeley.com/groups/1075191/benchmarking-e-learning/26 EFQUEL Academy