Presentation at Networked Learning 2012

1. The role of feedback in the design of
learning activities
or…how do we know that a good learning activity is 'good'
Dai Griffiths – CETIS / Institute for Educational Cybernetics, The University of Bolton
Griff Richards, Michelle Harrison – Thompson Rivers University

2. Introduction
The Insiders
 Producing courses
 Research into instructional design practice
The Visitor
 Technology Enhanced Learning researcher
 Learning Design

3. Context
TRU Open Learning
 Over 400 courses – print/web, now moving to
online (a shift in practice)
 Revision/New course development - large
teams, work with external SMEs
 Online paced cohort – design for open
contexts (new focus on OERs)
Instructional design team initiatives
 In a constantly changing environment how do
you more systematically improve, evaluate,
share and reflect on practice?

4. Initiatives
So far…
 Workshops
 Attempt at activity tagging – development of a
catalogue?
Pilot survey for student feedback (online questionnaire)
To do…
 Focus groups (Faculty/other stakeholders)
 Embed tools directly in courses (at the activity level)
 Survey questionnaire
 Analytics (some constraints)
 Development of activity catalogue (higher level) for
sharing

5. Idea: design patterns + analytics can
provide feedback and improve practice
But it was not possible to inspect the courses and
identify the patterns
So what feedback could help the design task?
We did interviews to establish
Factors which determine success in learning
activities
The feedback which designers would like

6. The interviews were very
interesting!
We drew out the themes with a Qualitative
Data Analysis tool
Obtained a list of factors determining success
of learning activities
These are candidates for gathering feedback
We also found
Approaches taken by designers in seeking
effective activities
Constraints on designers in doing this

7. Students Lecturers Designers Delivery team
Acessibility issues Appropriate group Activity Promptness/delay
formation instructions in the system
Cultural fit with Evident presence Amount of text and Reliability of
students of lecturer online balance of media technical systems
Perceived activity Lecturer 'buy in' to Degree of Scheduling of
usefulness the activity complexity courses
Student learning Level of formative Facilitator
process feedback workload
preferences
Technical barriers Preparation for the Activity structures
activity familiar to students
Time pressure Quality of Fit of pedagogy /
facilitiation context / student
Uneven Relationship to
participation levels learning objectives
Rubrics for
marking

8. How can institutions deal with 24
combinatorial states?
We can attenuate the variety, through well established
methods which position students as being identical:
Curricula
Cohorts
Assessments
We can amplify our response, for example
Through peer learning
Through team work
Or we can shut it out and hope it goes away.
Through institutional double speak about the importance of
the learning experience
By isolating strategic planning, design and delivery
This simplifies the institutions strategy (at least in the short
term!)
See Oleg Liber's application of Stafford Beer to education

9. What are the implications of the
interviews for the designers task?
Designers use professional and personal knowledge,
skills and intuition to produce good solutions to
impossible problems
These major themes (and more) need to be balanced
Institutional policy
The learners, their capabilities, preferences, and
available time
The type and level of the knowledge
The preferences and capabilities of the subject matter
experts
Fit with learning outcomes
The intricacies of copyright
The nature of the online environment v. face to face
equivalent

10. How could we amplify the designers
response?
Designers currently wrestle with the combinatorial
states
Individually
Intuitively, based on their experience of learning
and of the context
Unaware of the wider implications
The complexity and strategic importance of what
they do is not recognised
Make explicit the rules of thumb for
The problems faced
How we deal with that particular problem around
here
Start the design problem further along
See Pawson & Tilley's Realistic Evaluation for approach to rules

11. Off-load some of the effort to a document
See Hollan & Hutchins distributed cognition

12. “In this instructional design group this
is how we resolve that problem...”
a) There is a moral imperative for equality of
opportunity
b) Many of our learners are
Unused to group work
Unused to online collaboration
Get stressed when they are being assessed
c)So it is our rule of thumb to enable learners to
practice taking on roles in complex activities
by providing a non-assessed activity, before
using a complex activity in assessment

13. But of course, this is contested...
Alternatively:
a) There is a moral and economicimperative to serve the
learner
b) Our learners are focused on obtaining a qualification
They look for the assessment weighting of the activities
They act strategically and avoid any learning activity which
does not contribute to their grades
c)So it is our rule of thumb that all learning activities
(especially complex activities which take a lot of time)
will lead to assessment
The role of feedback and analytics is to enable us to
choose between rival interpretations

14. More feedback, or a dictat, is just
another thing to deal with
The rules of thumb should always be
An answer to a problem identified by the
people who have to apply them
Provisional hypotheses
Socially constructed
Contested
Feedback on learning activities should be
focused on confirming or falsifying the
hypotheses
Feedback on activities then becomes a research
based capacity raising exercise

15. In summary, a methodology for
feedback on learning activities...
Does not simply confirm successful delivery
Hypothesises what works where when and why
Examines hypotheses in an iterative process of action
research which informs practice
Is consultative and negotiated
Explicitly links feedback, analytics & design
Results in documents that
Take (a little) pressure off Instructional Designers by
providing design principles
Provide a basis for explaining the design task and
decisions to students and colleagues
So another title for you: “How a learning designer
learned to stop worrying and love learning analytics”

16. Thanks for your attention, and please
feel free to contact us
 Dai Griffiths: d.e.griffiths@bolton.ac.uk
 Michelle Harrison: mharrison@tru.ca
Dai Griffiths thanks Thomson Rivers Open Learning for their
support provided in his visiting scholarship in 2011. Without
it this research would not have taken place.