‘ Eyes Wide Shut’: The Challenge of Evaluating E-Learning Alan Masson  Vilinda Ross University of Ulster SERA 2006
Session Overview <ul><li>CETL (NI) Overview </li></ul><ul><li>E-Learning Issues </li></ul><ul><li>Evaluation of E-Learning...
CETL Overview <ul><li>Aim : “promote, facilitate and reward the adoption of a “learner centred” reflective practice approa...
Promoting reflective practice <ul><li>Full “cycle” support </li></ul><ul><ul><li>Reflect </li></ul></ul><ul><ul><li>Inspir...
 
e-learning “evaluation” issues <ul><li>Expectations   </li></ul><ul><ul><li>(still) emergent and high profile – demand for...
Evaluation of  e-learning <ul><li>Scope & Activity? </li></ul><ul><ul><li>institutional structures, policies etc.  </li></...
VLE - a source of useful data?  <ul><li>WebCT (CE, Vista) </li></ul><ul><ul><li>Logging   (user centric) </li></ul></ul><u...
Extensive VLE Reporting Options
Evaluating VLE reports?
Data Health Risks <ul><li>Data requires context - Library book “impact” issue </li></ul><ul><li>Confidence in Data / Relev...
Data Usage – Realistic Scenarios <ul><li>Targeting subjective work (thresholds and trends) </li></ul><ul><li>Reinforcing s...
Making Sense of the available data resource <ul><li>Data at Face Value vs. Probing Data </li></ul><ul><li>Difficult to qua...
Benefits of approach so far <ul><li>Can better focus on learner experience </li></ul><ul><li>Complementary approach - allo...
Complementary tools <ul><li>Survey methodologies </li></ul><ul><ul><li>Teacher focused </li></ul></ul><ul><ul><li>Learner ...
Wrap <ul><li>Evaluation of E-Learning </li></ul><ul><li>Eyes Wide Open </li></ul>
Development of a transparent method of recording practice <ul><li>Hybrid Learning Model drawing on Verpoorten’s 8LEM model...
 
Example of flash cards used to facilitate modeling of practice
Early example: hybrid learning model of T&L practice
Upcoming SlideShare
Loading in …5
×

‘Eyes Wide Shut’: The Challenge of Evaluating E-Learning

1,498 views

Published on

‘Eyes Wide Shut’: The Challenge of Evaluating E-Learning

Published in: Economy & Finance, Education
  • Be the first to comment

  • Be the first to like this

‘Eyes Wide Shut’: The Challenge of Evaluating E-Learning

  1. 1. ‘ Eyes Wide Shut’: The Challenge of Evaluating E-Learning Alan Masson Vilinda Ross University of Ulster SERA 2006
  2. 2. Session Overview <ul><li>CETL (NI) Overview </li></ul><ul><li>E-Learning Issues </li></ul><ul><li>Evaluation of E-Learning </li></ul><ul><li>VLE Data </li></ul><ul><li>Data Health Risks </li></ul><ul><li>Data Usage – Realistic Scenarios </li></ul><ul><li>Making Sense of the Data </li></ul><ul><li>Wrap </li></ul>
  3. 3. CETL Overview <ul><li>Aim : “promote, facilitate and reward the adoption of a “learner centred” reflective practice approach to the development of teaching and learning, in particular wrt the use of e-learning technologies” </li></ul><ul><li>Posts : 1x Centre Director, 1x Academic Staff Developer, 1x Research Associate, 2 Tech Posts, 4 Content Developers, Learner Advocate </li></ul><ul><li>Cultural challenge : effecting changes in “teaching” practices - key to learning experience </li></ul>CETL (NI): Utilising Institutional E-Learning Services to Enhance the Learning Experience
  4. 4. Promoting reflective practice <ul><li>Full “cycle” support </li></ul><ul><ul><li>Reflect </li></ul></ul><ul><ul><li>Inspire </li></ul></ul><ul><ul><li>Develop </li></ul></ul><ul><ul><li>Evaluate </li></ul></ul><ul><li>Learner focus emphasised </li></ul>
  5. 6. e-learning “evaluation” issues <ul><li>Expectations </li></ul><ul><ul><li>(still) emergent and high profile – demand for evaluation </li></ul></ul><ul><ul><li>Quantitative “data” readily available </li></ul></ul><ul><li>Tensions between: </li></ul><ul><ul><li>reporting and evaluation </li></ul></ul><ul><ul><li>evaluation and reflection </li></ul></ul>
  6. 7. Evaluation of e-learning <ul><li>Scope & Activity? </li></ul><ul><ul><li>institutional structures, policies etc. </li></ul></ul><ul><ul><li>e-teaching </li></ul></ul><ul><ul><li>student learning </li></ul></ul><ul><li>CETL perspective </li></ul><ul><ul><li>Assist evaluation of practice </li></ul></ul><ul><ul><li>Examine macro impact on teaching AND learning </li></ul></ul>
  7. 8. VLE - a source of useful data? <ul><li>WebCT (CE, Vista) </li></ul><ul><ul><li>Logging (user centric) </li></ul></ul><ul><ul><li>Tool Usage (Communication, Assessment) (resource centric) </li></ul></ul><ul><ul><li>Reports - individual, module, cohort, subject, demographics…… </li></ul></ul><ul><li>Data Extraction </li></ul><ul><ul><li>(Log files, Review periods, Extensive data, Filter out variables, Statistical analysis ) </li></ul></ul><ul><li>User location (IP address), time… session profile </li></ul>
  8. 9. Extensive VLE Reporting Options
  9. 10. Evaluating VLE reports?
  10. 11. Data Health Risks <ul><li>Data requires context - Library book “impact” issue </li></ul><ul><li>Confidence in Data / Relevance </li></ul><ul><li>Eyes Wide Shut vs. Open </li></ul><ul><li>‘ Rule In’ (not trust anything except) - Eyes Open </li></ul><ul><li>‘ Rule Out’ (trust everything except) </li></ul><ul><li>- Eyes Shut </li></ul>
  11. 12. Data Usage – Realistic Scenarios <ul><li>Targeting subjective work (thresholds and trends) </li></ul><ul><li>Reinforcing studies (indicative) </li></ul><ul><li>CETL priority - cultural change (different agenda) </li></ul><ul><ul><li>Raise awareness </li></ul></ul><ul><ul><li>Promote reflection </li></ul></ul><ul><ul><li>Challenge practices </li></ul></ul><ul><ul><li>Provide new perspectives </li></ul></ul><ul><ul><li>Requires careful presentation </li></ul></ul>
  12. 13. Making Sense of the available data resource <ul><li>Data at Face Value vs. Probing Data </li></ul><ul><li>Difficult to quantify (needs context s ) </li></ul><ul><li>Making Inferences (meaningful trends, identifying potential sample groups) </li></ul><ul><li>Experience, Expectation (reinforcing theory) </li></ul>
  13. 14. Benefits of approach so far <ul><li>Can better focus on learner experience </li></ul><ul><li>Complementary approach - allows focus on practices and interactions </li></ul><ul><li>Detailed review of “data” - identifying number of added value uses </li></ul><ul><li>Language / context important to manage expectations </li></ul>
  14. 15. Complementary tools <ul><li>Survey methodologies </li></ul><ul><ul><li>Teacher focused </li></ul></ul><ul><ul><li>Learner focused </li></ul></ul><ul><li>Learner reflective journals </li></ul><ul><li>Focus groups </li></ul><ul><li>Exploring conversational approaches to evaluation </li></ul><ul><ul><li>Teacher / learner models of practice </li></ul></ul><ul><li>Resource intensive activities - getting best impact? </li></ul>
  15. 16. Wrap <ul><li>Evaluation of E-Learning </li></ul><ul><li>Eyes Wide Open </li></ul>
  16. 17. Development of a transparent method of recording practice <ul><li>Hybrid Learning Model drawing on Verpoorten’s 8LEM model and Bennett’s 30 learning verbs </li></ul>
  17. 19. Example of flash cards used to facilitate modeling of practice
  18. 20. Early example: hybrid learning model of T&L practice

×