This document discusses the relationship between learning design and learning analytics. It argues that learning design provides context that helps frame analytics questions and identify appropriate analysis. A MOOC planner is presented that prompts designers to plan different activity types like delivered content, reflection, collaboration, and assessment. Analytics can then provide insight into how learners engaged with those different activities. The document also discusses clustering analysis of learner engagement patterns in MOOCs, like samplers, strong starters, mid-way dropouts, and keen completers. Engagement patterns can vary based on pedagogy and learning design.

1. Learning design & learning
analytics – building the links
Rebecca Ferguson
The Open University
What the Research Says: November 2014

2. Why learning design?
• Provides a set of tools and information to support a
learner-activity based approach
• Helps to show the costs and performance outcomes of
design decisions
• Puts the learning journey at the heart of the design
process
• Enables the sharing of best practice
• Helps partners choose and integrate a coherent range of
media, technologies and pedagogies
• Enables a consistent and structured approach to review
and analytics

3. Advantages of design for analytics
Helps to frame and focus analytics questions
What did they learn?… in relation to learning outcomes
Were they social?... when they were collaborating
Did they share links?... when encouraged to browse
Did they return to steps?... when encouraged to reflect
Helps to identify appropriate forms of analysis
The same step, but with a focus on
• Number of visits if content
• Length, quality, number of comments if conversational
• Dwell time and repeat visits if reflection

4. MOOC planner
Prompts designers
to block out different
types of learning
activity:
• Delivered
• Reflection
• Collaboration
• Conversation
• Networking
• Browsing
• Assessment

5. Planner and analytics
Delivered Content
(reading, watching, listening and observing)
Analytics: amount of content viewed, dwell time
Reflection
(thinking, considering and reflecting)
Analytics: returns to the same material, reflection
exercises completed, quality of reflection
Collaboration
(constructing, collaborating, defining and engaging)
Analytics: collaboration exercises completed, quality
of collaboration

6. MOOC map
• Guidance & support
• Content & experience
• Reflection &
demonstration
• Communication &
collaboration

7. Activity and learning design

8. Patterns of engagment: Coursera
Sampling: learners explored some course videos.
Auditing: learners watched most videos, but
completed assessments rarely, if at all
Disengaging: learners completed assessments at
the start of the course and then reduced their
engagement
Completing: learners completed most assessments
Kizilcec, R., Piech, C., and Schneider, E., 2013. Deconstructing disengagement:
analyzing learner subpopulations in massive open online courses. In LAK13

9. Cluster analysis

10. Patterns of engagement
On an eight-week MOOC
Samplers visit only briefly
[1, 0, 0, 0, 0, 0, 0, 0]
Strong starters do first assessment
[9, 1, 0, 0, 0, 0, 0, 0]
Returners come back in Week 2
[9, 9, 0, 0, 0, 0, 0, 0]
Mid-way Dropouts
[9, 9, 9, 4, 1, 1, 0, 0]
Nearly There drop out near the end
[11, 11, 9, 11, 9, 9, 8, 0]
Late Completers finish
[5, 5, 5, 5, 5, 5, 9, 9, 9]
Keen Completers do almost
everything [11, 11, 9, 9, 11, 11, 9, 9] Patterns vary with pedagogy
and learning design

11. Short MOOCs
Surgers concentrate their effort after the first week of a three-week
course. They do little in Week 1 other than submit their
assessment late, engage more in Week 2, but still submit their
assessment late (working on it in Week 3), and engage but do not
submit in Week 3. On average, they post one or two comments.
A typical engagement profile for this cluster is: [4, 6, 2]
Improvers fall behind in Week 1, submitting their first assessment
late. However, they engage more in Week 2 and by Week 3 they
are on schedule and submit their assessment on time. They view
the majority of steps on the course and typically post more than
one comment.
A typical engagement profile for this cluster is: [5, 6, 9]

12. Developing this work