Research in Distance Education: impact on practice conference, 27 October 2010. Presentation in Assessment Strand by Dr Stylianos Hatzipanagos, Lecturer in Higher education/Head of e-learning, King’s College London. Teaching and Research Award Holder. More details at www.cde.london.ac.uk.

1. Feedback as dialogue and learning
technologies: can e-assessment be
formative?
Stylianos Hatzipanagos
King's College London
April 2010

2. The seminar will:
 Put forward a conceptual model of formative
assessment and examine how this can be made to
work purposefully within the specific constraints
of ODL;
 Explore the relationship between formative
assessment and learning technologies and the
opportunities for computer mediated
communication and dialogue in these
environments.

3. Closing the loop: two projects
 explored policy, teaching practices and tutor and
student views in three Open and Distance Learning
environments.
 investigated relationship between formative
assessment and learning technologies

4. Projects team
♣ Paul Black (King’s)
♣ Stylianos Hatzipanagos (King’s)
♣ Ana Lucena (IOE)
♣ Bob McCormick (OU)
♣ Steven Warburton (King’s)

5. Types of assessment
summative (SA)
 numerical results about student performance
 generally carried out at the end of a course/programme
formative (FA)
 a continuous/periodic appraisal of students’
performance
 may inform and improve competence and contribute to
further learning
(Sadler, 1989; Boston, 2002; Black et al., 2003; York, 2004)

6.  Formative or assessment for learning
(Albon, 2003; Wiliam et al., 2004; Nicol & MacFarlane-Dick, 2004;
Black, 2005)
 Duality “SA/FA” may not represent opposite poles of assessment
(Hargreaves, 2005; Dylan, 2006)
 FA is SA with feedback, which can be used by the learner (Taras
2005)
 positive implications for student learning
 allows students to play a more active role in management of own
learning (Nicol 1997)

7. Purposes of assessment types
(Hatzipanagos & Warburton 2009, adapted from Bull &
McKenna 2004)
Type of
assessment
Feedback to
student
Feedback to
tutor
Self testing Determining
level/progression
diagnostic 2 2 1
formative 1 2 1 2
summative 2 1
Primary purpose: 1, Secondary purpose: 2

8. Dimensions of FA
 Power (autonomy and ownership)
 Dialogue
 Peer/self assessment
 Feedback (attributes)
 Visibility
 Reflection
 Action
Black et al. (2003), Gibbs & Simpson (2004),
Giroux (1992), McConnell (2006).

9. Feedback Comments – what messages
are being conveyed?
! ‘be more
critical’
‘you’re argument needs to be more academic’
X ‘More use of analysis would have
helped’
‘requires further elaboration’ 
‘you need to think carefully about your structure’
‘You write in a journalistic style’
Medland 2009

10. FA and technologies
Learning technologies promote innovative assessment
practices and lead to deeper thinking about how tutors
conceptualise assessment in higher education (McCormick
2004).
assessment practices have been supported by technology
for many years. However…
….main focus on developing tools such as objective tests
rather than addressing fundamental issues, such as how
they can be used to support effective assessment
approaches (Nicol and Milligan, 2006).

11. Characteristics of FA technologies:
potential
Can be used as living record of student learning
Blur boundaries between formal and informal learning
spaces
Challenge dominant model of hierarchical, tutor
centred education
Embrace an ideology of openness, dialogue and
ownership.

12. Open & Distance Learning
ODL environments :
 necessity for FA practices.
 proactive in FA practices out of need to provide
systematic feedback to students.

13. e-literacy in pedagogical terms
Required set of competencies for teaching practitioners is more
complex and consists of:
 Updated knowledge of e-learning resources and their role in
enriching the learner experience.
 Expertise of range of tools and of the pedagogical affordances
that each provides
 Ability to evaluate technologies and engage in reflection on how
they impact on learning

14. Staff Perspective
‘...part of the feedback given to pupils in class is like so
many bottles thrown out to sea. No one can be sure
that the message they contain will one day find a
receiver’
(Perrenoud, 1998: 87)

15. Student Perspective
 Dearing Report (1997)
 National Student Survey (2004-
present)

16. Methodology
 Open-ended interviews with 17 tutors and 20 students to
explore perceptions of assessment.
 Informed content of online questionnaire that was
administered to students within the three ODL environments.
 In the online survey, students were asked to express their level
of agreement to a number of statements about a five-point
Likert scale.
 Qualitative data using also open-ended questions in the
questionnaire.

17. Institutional differences
 Huge diversity in practices in all three of the
environments.
 From the three environments, two were broadly similar,
and they were characterised by consistent elements of
good practice.
 One of the three environments had in place an
infrastructure to provide more systematic provision of
feedback. In addition, there was in place a framework,
which emphasised periodic assessment rather than end
of year assessments.

18. Institutional cultures and the nature of an institution
determine assessment practices:
King’s ODL and the External programmes broadly similar.
External programmes : there is an indication of huge diversity in practices.
Good practice in policy :
 policy documents that set the frameworks within which the tutors will operate
and increased use of e-assessment.
 Infrastructure to provide more systematic provision of feedback.
 Framework, which emphasises periodic assessment rather than end of year
assessments.
 Particularly at the OU tutors are continuously monitored about their assessment
practices.

19. Tutor engagement
Disjunction between beliefs, ambitions and pragmatic approaches to
the use of FA.
Practitioners’ attitudes to FA context-dependent and sometimes
discipline-oriented.
Classification of approaches in two substantial groups and one smaller:
The first two claimed that they were proponents of FA;
they either used FA in their practices (group 1)
or claimed that they did not, however they would consider it if
pragmatic constraints allowed it (group 2)
Third smaller group did not consider FA necessary for their context.

20. Significant number of tutors engaged in FA but not
extensively in what we consider FA.
Notion of FA varied e.g. often was equated to ‘continuous
assessment’.
Formats of assessment have changed because of the
possibilities new technologies can afford.

21. Student engagement
 Target audience of institutional groups were diverse in terms
of perceptions.
 Students’ attitudes to assessment were not discipline
dependent. It was the broad context (the ODL environment)
that determined attitudes.
 Facilitation of feedback mechanisms using computer
mediated communication was recognised as a significant
component of the assessment process.
 Majority of learners recognised the challenges in providing a
suitably formative environment in these settings.
 Difficulties in defining their personalised learning environment
and the affordances of the tools they used.

22. Q31. I am interested in the marks and not in the feedback.
0
50
100
150
200
250
300
350
400
3.8% 13.4% 41.8% 41.0%
Always Often Sometimes Never
Series1

23. Q24. I would learn more if I received feedback
0
50
100
150
200
250
300
350
400
28.9% 27.1% 44.0%
more frequently with longer comments shortly after submission of
my work
Series1

24. Q28. The feedback guides me on what I need to do to improve my work.
0
50
100
150
200
250
300
350
400
37.9% 43.5% 16.9% 1.6%
Always Often Sometimes Never
Series1

25. Q13. I have used a computer to receive feedback.
0
100
200
300
400
500
600
700
800
83.6% 13.2% 3.8%
Yes No Not sure
Series1

26. e-Assessment
 The projects considered whether current formative
assessment practices can cope with emerging
technologies.
 Learning technologies (a fundamental channel of student
support)
 Received favourably, though overall there was no sense of
ownership or always a clear perception of purpose
regarding the affordances of each technology.

27. e-assessment: range of technologies
Non-formative
 Objective tests (non adaptive and they ‘disagree’ with certain
disciplines)
 Model answers received or revealed after students submit an answer, as
non-personalised feedback
 Electronic submission of coursework
Formative
 Communication tools in VLEs
 Online tutorial environments
 Games that allow monitoring and intervention
 Audio to canvas opinions/understanding of concepts/issues (audio more
meaningful conceptually than video)
 Tools such as certainty based marking
 Videoconferencing
 Social software: Blogs, Wikis
 e-portfolios

28. Key processes
of formative
assessment
Traditional technologies Social software
email
discussion
board
quiz blog wiki
Power
(ownership and
autonomy)
H L L H
H
(distributed)
Dialogue
L to M H L
H (informal
tone)
M
Timeliness
L to M H variable H H
Visibility
L variable L H H
Action
L to M L to M M H H
Community
(peer support) L variable L
H (via
aggregation)
H
Reflection
variable M M H M

29. Relationship between social software & FA
…social dimensions of social software
allow for formative assessment practices
to be re-invented or at the very least
facilitated by participative and student
focussed interventions.
(Hatzipanagos & Warburton 2009)

30. Four Stages to embed FA practices in the assessment cycle
Stage Zero  Assessment mostly through exams and end of assessment term projects.
 No provision of feedback.
 Limited or no peer/self assessment opportunities.
Limited or no use of learning technologies to support assessment practices.
Stage One  Generalised feedback on student work but of limited customisation to the needs of the
individual learner.
 Examiner reports with model answers for monitoring/evaluating assessment practice.
 Use of learning technologies to support assessment through objective tests.
Stage Two All the above in Stage One plus:
 Periodic/continuous assessment for learners to rehearse arguments that they will use in end
of assessment period assessments.
 Feedback is monitored, to ensure that students will act upon negotiated targets and the
feedback loop is closed.
 Learner responses to feedback become an essential part of the assessment cycle.
 Use of learning technologies such as computer mediated communication to facilitate the
assessment cycle.
Stage Three All the above in Stage Two plus:
 Peer/self assessment.
 Student involvement in setting marking criteria.
 Use of learning technologies to peer review, and to construct knowledge collaboratively.
Hatzipanagos 2010

31. Conclusions
Towards an understanding of the significance of formative
assessment in ODL.
A conceptual model of formative assessment and how this
can be made to work purposefully within the specific
constraints of ODL environments.
FA practices can be problematic in courses with emphasis on
end of year assessments as the ‘closing the loop’
component of the assessment process very rarely takes
place.
FA can enrich e-learning approaches by making the feedback
central to all e-assessment activities.
e-assessment tools promote a dialogue in relation to
feedback, peer and self assessment activities which by
their nature place the student at the centre of the
educational process.

32. Some questions to explore
1. Are assessment practices in HE stifling innovation?
2. How formative SA practices must be to have an
impact on student learning?
3. Can self or peer assessment practices be summative
or only formative?
4. How important is context on assessment practices
and student performance?
5. How do we accredit learning that takes place in
informal spaces afforded by learning technologies

33. Further work
♣ Comparison of current technology supported FA
mechanisms across the ODL landscape.
♣ identify types of processes and learning technology
tools that might best support effective feedback
♣ encourage effective feedback approaches that both
empower and enhance the learning experience for
the distance learner.

34. If you are interested to find out more
…
email s.hatzipanagos@kcl.ac.uk