1. From Evaluation to Sensemaking: Emergent development of a
masters distance learning research methods module
Phil Wood, University of Leicester
ALDinHE 11th April, 2017

2. • Can sensemaking be positively applied to the process of programme
development?
• How might a variant of lesson study be used as a medium for sensemaking in
programme development?
Foci

3. Guiding Considerations:
• Many students don’t fill in distance learning evaluations
• Evaluations often too generalised
• Evaluations are inherently retrospective
• Evaluations often focus on activities, tutoring, resources, environments, but rarely
learning and student action.
• Analysis often gets reduced to simplistic reflections on numbers

4. Are there alternatives?
• Ellery (2006) – multidimensional evaluation on a campus-based course (evaluation
throughout the module, with both student and lecturer perspectives)
• Benson et al (2009) - Participatory evaluation, again multi-modal. Based on Jackson and
Kassam 1998
a process of self-assessment, collective knowledge production, and cooperative
action in which the stakeholders in a development intervention participate
substantively in the identification of the evaluation issues, the design of the
evaluation, the collection and analysis of the data, and the action taken as a
result of the evaluation findings.
(1998, 3)

5. ‘Sensemaking is the ability or attempt to make sense of an ambiguous situation. More
exactly, sensemaking is the process of creating situational awareness and
understanding in situations of high complexity or uncertainty in order to make
decisions. It is “a motivated, continuous effort to understand connections (which can
be among people, places, and events) in order to anticipate their trajectories and act
effectively” ‘
Klein et al 2006 in Snowden http://cognitive-edge.com/blog/what-is-sense-making/
Sensemaking vs Evaluation
• Approach which works with complexity and process
• Develops insights and narratives
• A move from quality assurance to quality growth

6. Course context
• MA International Education (DL)
• Small cohorts with very diverse backgrounds
• Based on developing personal/professional interests
• Professionals already in post

7. We wanted to develop an approach which would allow for:
• Making distance learning review more than a ‘performative’ activity
• Putting pedagogy (interpenetration of teaching, learning, curriculum and assessment and
their interaction with teachers and students) at the centre of the process
• Trialling and emergence of new approaches as a standard element of our work
• Working with complexity and process

8. 1. Tutor group
identifies an issue
as the ‘learning
challenge’
2. OPTIONAL:
tutors interview
students
concerning prior
learning and
conceptual
understanding
3. The tutor group
collaboratively
plan a week long
package to meet
the learning
challenge
4. Students
complete the
week-long work
package & tutors
interview them to
discuss learning
experiences
5. The tutor group
meets to consider
the evidence from
the interviews
6. Insights from
the evaluation are
considered in
helping plan the
next work package
of interest
Lesson Study Cycle
Based on
Wood and Cajkler 2016
Wood and Cajkler 2017

9. Basic outline of process
Weeks 1 & 2
Introduction &
research
questions
Weeks 3 & 4
Data collection
approaches &
sampling
Weeks 5 & 6
Ethics and data
collection
Weeks 7 & 8
Data analysis
Formative
assessment
Weeks 9 to 11
Methodologies
Philosophical
foundations
Week 12
Critical writing &
introduction to
main assignment

10. Potential of sensemaking
• Development of rich narratives
• Greater insights into the complexity of student learning ecologies
• Course development and reflection in ‘real time’
• Makes processes of student engagement and learning more explicit
• Greater understanding of contexts for student work

11. General Insights
• Structure of the course materials seen as a great strength
• All students applying the process to practical issues and questions in their own professional work
• Use of ipads to make learning more flexible
• Materials on research questions found to be very useful – general to specific
• Use of video to explain an exemplar assignment found to be very useful
• More materials needed to help analysis of observation data
• Activities included each week are seen as a real strength of the module – helps to frame thinking
• The learning on the module seen as being practical and applied – very positive reflections

12. Same module, different processes
Student B
• Some experience of social science research
• Makes use of ipad for reading, laptop for writing
• Hardly used wiki – found it was of little use
• Engages with module work through reading
• Developed research around international student
disability
• Seeing module as focused on practical application
• Focused on reading to make sense of the module
• First experience of writing a research report
• A practical approach as training for the
dissertation
• Had a very general idea of the features of critical
writing before cycle 3
• Module work has led to a more evidence-based
and critical approach to professional work
Student A
• First attempt at social sciences research
• Makes use of ipad for most work
• Valued wiki/research diary for peer feed forward
• Major focus on videos and some reading
• Developed research around a learning
development project
• Seeing module as focused on practical application
• Focused on resources and discussion with others
on the module to help develop and critique ideas
• First experience of qualitative research
• A very reflective approach
• Had a clear but general idea of the features of
critical writing before cycle 3
• Pilot study used in professional work

13. Some Initial Reflections
• Danger that TEF will lead to further slide towards reductive, data-driven development
• Sensemaking makes process and complexity explicit
• Offers chance to develop rich narratives which work with learning analytics and
evaluations – development of a mixed methods approach
• Opportunity to develop curriculum in real time
• Opens up further understanding of learning ecologies – but much still unknown
• More considered approach to course design and learning processes
• Making time for ‘deep work’ (Newport, 2016) to draw out insights/reflections and
emerging models of pedagogy.

14. References
Jackson, E.T., and Y. Kassam (1998) Knowledge shared: Participatory evaluation in development
cooperation. West Harford, CT: Kumarian Press
Newport, C. (2016) Deep Work, Rules for focused success in a distracted world. London, Little Brown.
Benson, R.; Samarawickrema, G. & O’Connell, M. (2009) ‘Participatory evaluation: implications for
improving electronic learning and teaching approaches.’ Assessment & Evaluation in Higher Education,
34:6, 709-720.
Ellery, K. (2006) ‘Multi‐dimensional evaluation for module improvement: a mathematics‐based case
study.’ Assessment & Evaluation in Higher Education, 31:1, 135-149.
Klein, G., Moon, B. and Hoffman, R.F. (2006a) ‘Making sense of sensemaking I: alternative perspectives.’
IEEE Intelligent Systems, 21:4, 70–73.
Wood, P. & Cajkler, W. (2016) ‘A participatory approach to Lesson Study in higher education.’
International Journal for Lesson and Learning Studies, 5:1, 4-18.
Wood, P. & Cajkler, W. (2017) ‘Lesson Study: a Collaborative Approach to Scholarship for Teaching and
Learning in Higher Education.’ Journal of Further and Higher Education.
http://dx.doi.org/10.1080/0309877X.2016.1261093