Embedding evaluation in distance courses – using narrative methods 9 February 2000 Caroline Daly, Norbert Pachler, Jon Pickering, Jill Russell and Jon Wardle

Rationale for the research projects We need course development to be informed by e-learners’ experiences The ‘primacy of improving the student learning experience’ (HEA Strategic Plan 2005 – 2015 ) for overseas and home students The inadequacies of evaluation based on ‘satisfaction’ and ‘ exit’ models We perceive coherence in our learning designs where the students may not We need a sustainable way of understanding learners’ experiences in distance e-learning contexts We need to avoid the ‘sisyphus effect’

We need course development to be informed by e-learners’

experiences

The ‘primacy of improving the student learning experience’ (HEA

Strategic Plan 2005 – 2015 ) for overseas and home students

The inadequacies of evaluation based on ‘satisfaction’ and

‘ exit’ models

We perceive coherence in our learning designs where the students

may not

We need a sustainable way of understanding learners’

experiences in distance e-learning contexts

We need to avoid the ‘sisyphus effect’

E-learners’ experiences – four types of ‘newness’ which affect participation for all (CDE 1) The experience of managing work, life and learning with technology The experience of writing with fellow students in an online discussion in order to learn The experience of collaborative learning in an online tutor group The experience of social relationships with other e-learners

Narrative evaluation methods are based on… a ‘way of knowing’ or ‘mode of thought’ (Bruner, 1985) a way of filtering experience in order to make sense of it giving practitioners access to informal as well as formal experiences of students’ learning respondents in engaging with knowledge they have which is often tacit and unarticulated a qualitative way of understanding something for both the learner and the practitioner - an interpretation of reality the learner organises their experiences by ‘narrating’ the practitioner constructs the meanings to be coherent within a view of what has happened

a ‘way of knowing’ or ‘mode of thought’ (Bruner, 1985)

a way of filtering experience in order to make sense of it

giving practitioners access to informal as well as formal experiences of students’ learning

respondents in engaging with knowledge they have which is often tacit and unarticulated

a qualitative way of understanding something for both the learner and the practitioner - an interpretation of reality

the learner organises their experiences by ‘narrating’

the practitioner constructs the meanings to be coherent within a view of what has happened

Why narrative methods? A way of capturing the learner voice as a prime research or evaluation tool in qualitative approaches A way of developing situated accounts of respondents’ experiences To provide ongoing information about our practice as providers while it is happening To impact on learners’ abilities to adapt to new learning contexts (Levy, 2006) A way of better understanding the realities of ‘being a learner’ in complex contexts To inform effective education practitioner-development

Loosely structured “ without overspecifying the substance or the perspective of the talk” (McCracken,1988) Naïve interviewer stance Invitational, not interventional Long ‘ Confessional’ Narrative interviews

Loosely structured

“ without overspecifying the substance or the perspective of the talk” (McCracken,1988)

Naïve interviewer stance

Invitational, not interventional

Long

‘ Confessional’

What is going on in a narrative interview? Emplotment Chronology Opportunities for individual sense-making (e.g. metaphor-building in this example) Opportunities for deep levels of reflection by the respondent can elicit rich data The agenda becomes the respondent’s

Emplotment

Chronology

Opportunities for individual sense-making (e.g. metaphor-building in this example)

Opportunities for deep levels of reflection by the respondent can elicit rich data

The agenda becomes the respondent’s

Contemporary contexts for narrative evaluation ` Patterns of student participation are changing. A range of new contexts raise challenges for distance education and training, in terms of how we can evaluate the learner experience: E-learning Distance education Mass education Work-based learning Professional accreditation Retention, isolation, dilution, serial change, multiple pressures…

` Patterns of student participation are changing. A range of new contexts raise challenges for distance education and training, in terms of how we can evaluate the learner experience:

E-learning

Distance education

Mass education

Work-based learning

Professional accreditation

Retention, isolation, dilution, serial change, multiple pressures…

CDE 2 project 2006-7 Context Design 2 online masters degrees UCL & Bournemouth University Longitudinal – 9 months Participants (n=18) new students - 11 in primary healthcare (UCL) - 7 in creative media (BU) Quantitative data – pre-course questionnaire E-learning at a distance - online discussions - electronic text-based resources - wikis (BU) - blogs (BU) - podcasts (BU) Qualitative data – learner narratives - online discussion – course-based - narrative interviews by telephone - online commentaries - individual email correspondence - focus groups with tutors International - UCL cohort includes a high proportion of South African students - CEMP cohort is mostly UK based Analytical approach based on systematic reading and open-coding of range of narratives to establish the effectiveness of methods Main aims: to develop methods to collect evaluation information from distance learners to design an embedded approach to evaluation in fully online contexts

Main aims:

to develop methods to collect evaluation information from distance learners

to design an embedded approach to evaluation in fully online contexts

Narrative evaluation methods for distance learners Group narratives aimed at collective sense-making of experiences. The UCL ‘intensive introduction to e-learning’ as an embedded evaluation activity; Telephone interviews . Adapting individual, loosely-structured narrative interviews using ‘postmodern’ techniques (Gubrium, 2003) from face to face contexts; Virtual focus groups using online forums to elicit reflective group discussion on key issues; Individual email correspondence (post-interviews and post-online forums) to invite reflective review of earlier narratives.

Group narratives aimed at collective sense-making of experiences. The UCL ‘intensive introduction to e-learning’ as an embedded evaluation activity;

Telephone interviews . Adapting individual, loosely-structured narrative interviews using ‘postmodern’ techniques (Gubrium, 2003) from face to face contexts;

Virtual focus groups using online forums to elicit reflective group discussion on key issues;

Individual email correspondence (post-interviews and post-online forums) to invite reflective review of earlier narratives.

Narrative interviewing at a distance The power relations between tutor/evaluator and learner changed The learner knew more than the interviewer about the context of the interview The learner controlled the context of the interview The learner questioned the interviewer The interviewer exercised measured tentativeness The interview became a dialogue where prompts must be given, but control over the agenda is handed over to the learner as far as possible

The power relations between tutor/evaluator and learner

changed

The learner knew more than the interviewer about the context of the interview

The learner controlled the context of the interview

The learner questioned the interviewer

The interviewer exercised measured tentativeness

The interview became a dialogue where prompts must be given, but control over the agenda is handed over to the learner as far as possible

Tutors’ responses The content of narrative interviews did not ‘surprise’ the tutors much The narratives provided a coherent picture – told the ‘whole story’ of the learners’ experiences in a way not normally available The data did expose experiences not anticipated by tutors (e.g. the ‘feeling sick’ about the experience of posting to the forum; scale of the struggle to balance work/life/study; the fact that students participate in the middle of the working day e.g. ‘between patients’) Highlighted the need for more close engagement with students’ experiences Highlighted students’ desire to ‘own’ evaluation processes, e.g. to decide when and how it is convenient to discuss their experiences

The content of narrative interviews did not ‘surprise’ the tutors much

The narratives provided a coherent picture – told the ‘whole story’ of the learners’ experiences in a way not normally available

The data did expose experiences not anticipated by tutors (e.g. the ‘feeling sick’ about the experience of posting to the forum; scale of the struggle to balance work/life/study; the fact that students participate in the middle of the working day e.g. ‘between patients’)

Highlighted the need for more close engagement with students’ experiences

Highlighted students’ desire to ‘own’ evaluation processes, e.g. to decide when and how it is convenient to discuss their experiences

Dialogue – the future for meaningful evaluation? The value of narrative evaluation is much more than viewing it as an effective means of eliciting rich evaluation data. “ An alternative view of evaluation redefines it as a social practice in itself and so the emphasis becomes as much on the value of the substantive processes of evaluation as on the product (data) elicited by the evaluation” Russell (2008)

The value of narrative evaluation is much more than viewing it as an effective means of eliciting rich evaluation data.

“ An alternative view of evaluation redefines it as a social practice in itself and so the emphasis becomes as much on the value of the substantive processes of evaluation as on the product (data) elicited by the evaluation”

Russell (2008)

e- VALU -ation “… involves not only collecting descriptive information …but also using something called “values” to (a) determine what information should be collected and (b) draw explicitly evaluative inferences from the data… … research can tell us “what’s so” but only evaluation can tell us “so what”. Davidson (2005)

“… involves not only collecting descriptive information …but also using something called “values” to (a) determine what information should be collected and (b) draw explicitly evaluative inferences from the data…

… research can tell us “what’s so” but only evaluation can tell us “so what”.

Davidson (2005)

Project publications Daly, C. (2008) ‘Evaluation for new learning contexts – how can it be ‘fit for purpose’?’ Reflecting Education 4 (1) 127-138. http://www.reflectingeducation.net Daly, C., Pachler, N., Pickering, J. and Bezemer, J. (2007) Teachers as e-learners: exploring the experiences of teachers in an online professional master’s programme Journal of In-service Education 33 (4) 443-462. Daly, C., Pachler, N., Pickering, J. and Bezemer, J. (2006) ‘A study of e-learners’ experiences in the mixed-mode professional master’s programme, the Master of Teaching’ WLE Centre, Institute of Education, www.wlecenre.ac.uk

Daly, C. (2008) ‘Evaluation for new learning contexts – how can it be ‘fit for purpose’?’ Reflecting Education 4 (1) 127-138. http://www.reflectingeducation.net

Daly, C., Pachler, N., Pickering, J. and Bezemer, J. (2007) Teachers as e-learners: exploring the experiences of teachers in an online professional master’s programme Journal of In-service Education 33 (4) 443-462.

Daly, C., Pachler, N., Pickering, J. and Bezemer, J. (2006) ‘A study of e-learners’ experiences in the mixed-mode professional master’s programme, the Master of Teaching’ WLE Centre, Institute of Education, www.wlecenre.ac.uk