‘ Eyes Wide Shut’: The Challenge of Evaluating E-Learning Alan Masson Vilinda Ross University of Ulster SERA 2006

Session Overview CETL (NI) Overview E-Learning Issues Evaluation of E-Learning VLE Data Data Health Risks Data Usage – Realistic Scenarios Making Sense of the Data Wrap

CETL (NI) Overview

E-Learning Issues

Evaluation of E-Learning

VLE Data

Data Health Risks

Data Usage – Realistic Scenarios

Making Sense of the Data

Wrap

CETL Overview Aim : “promote, facilitate and reward the adoption of a “learner centred” reflective practice approach to the development of teaching and learning, in particular wrt the use of e-learning technologies” Posts : 1x Centre Director, 1x Academic Staff Developer, 1x Research Associate, 2 Tech Posts, 4 Content Developers, Learner Advocate Cultural challenge : effecting changes in “teaching” practices - key to learning experience CETL (NI): Utilising Institutional E-Learning Services to Enhance the Learning Experience

Aim : “promote, facilitate and reward the adoption of a “learner centred” reflective practice approach to the development of teaching and learning, in particular wrt the use of e-learning technologies”

Posts : 1x Centre Director, 1x Academic Staff Developer, 1x Research Associate, 2 Tech Posts, 4 Content Developers, Learner Advocate

Cultural challenge : effecting changes in “teaching” practices - key to learning experience

Promoting reflective practice Full “cycle” support Reflect Inspire Develop Evaluate Learner focus emphasised

Full “cycle” support

Reflect

Inspire

Develop

Evaluate

Learner focus emphasised

e-learning “evaluation” issues Expectations (still) emergent and high profile – demand for evaluation Quantitative “data” readily available Tensions between: reporting and evaluation evaluation and reflection

Expectations

(still) emergent and high profile – demand for evaluation

Quantitative “data” readily available

Tensions between:

reporting and evaluation

evaluation and reflection

Evaluation of e-learning Scope & Activity? institutional structures, policies etc. e-teaching student learning CETL perspective Assist evaluation of practice Examine macro impact on teaching AND learning

Scope & Activity?

institutional structures, policies etc.

e-teaching

student learning

CETL perspective

Assist evaluation of practice

Examine macro impact on teaching AND learning

VLE - a source of useful data? WebCT (CE, Vista) Logging (user centric) Tool Usage (Communication, Assessment) (resource centric) Reports - individual, module, cohort, subject, demographics…… Data Extraction (Log files, Review periods, Extensive data, Filter out variables, Statistical analysis ) User location (IP address), time… session profile

WebCT (CE, Vista)

Logging (user centric)

Tool Usage (Communication, Assessment) (resource centric)

Reports - individual, module, cohort, subject, demographics……

Data Extraction

(Log files, Review periods, Extensive data, Filter out variables, Statistical analysis )

User location (IP address), time… session profile

Extensive VLE Reporting Options

Evaluating VLE reports?

Data Health Risks Data requires context - Library book “impact” issue Confidence in Data / Relevance Eyes Wide Shut vs. Open ‘ Rule In’ (not trust anything except) - Eyes Open ‘ Rule Out’ (trust everything except) - Eyes Shut

Data requires context - Library book “impact” issue

Confidence in Data / Relevance

Eyes Wide Shut vs. Open

‘ Rule In’ (not trust anything except) - Eyes Open

‘ Rule Out’ (trust everything except)

- Eyes Shut

Data Usage – Realistic Scenarios Targeting subjective work (thresholds and trends) Reinforcing studies (indicative) CETL priority - cultural change (different agenda) Raise awareness Promote reflection Challenge practices Provide new perspectives Requires careful presentation

Targeting subjective work (thresholds and trends)

Reinforcing studies (indicative)

CETL priority - cultural change (different agenda)

Raise awareness

Promote reflection

Challenge practices

Provide new perspectives

Requires careful presentation

Making Sense of the available data resource Data at Face Value vs. Probing Data Difficult to quantify (needs context s ) Making Inferences (meaningful trends, identifying potential sample groups) Experience, Expectation (reinforcing theory)

Data at Face Value vs. Probing Data

Difficult to quantify (needs context s )

Making Inferences (meaningful trends, identifying potential sample groups)

Experience, Expectation (reinforcing theory)

Benefits of approach so far Can better focus on learner experience Complementary approach - allows focus on practices and interactions Detailed review of “data” - identifying number of added value uses Language / context important to manage expectations

Can better focus on learner experience

Complementary approach - allows focus on practices and interactions

Detailed review of “data” - identifying number of added value uses

Language / context important to manage expectations

Complementary tools Survey methodologies Teacher focused Learner focused Learner reflective journals Focus groups Exploring conversational approaches to evaluation Teacher / learner models of practice Resource intensive activities - getting best impact?

Survey methodologies

Teacher focused

Learner focused

Learner reflective journals

Focus groups

Exploring conversational approaches to evaluation

Teacher / learner models of practice

Resource intensive activities - getting best impact?

Wrap Evaluation of E-Learning Eyes Wide Open

Evaluation of E-Learning

Eyes Wide Open

Development of a transparent method of recording practice Hybrid Learning Model drawing on Verpoorten’s 8LEM model and Bennett’s 30 learning verbs

Hybrid Learning Model drawing on Verpoorten’s 8LEM model and Bennett’s 30 learning verbs

Example of flash cards used to facilitate modeling of practice

Early example: hybrid learning model of T&L practice