Learning design and learning
analytics: building the links
EMMA Summer School, Ischia, July 2015

Rebecca Ferguson
• The Open University: largest in UK
• Informal learning: iTunes, YouTube…
• MOOCs on FutureLearn, OpenLearn
and elsewhere
• Making use of big data for
more than 40 years
• Learning analytics research / events
• LACE project
2
Lead on MOOC evaluation at The Open University, UK
http://www.laceproject.eu/

Workshop overview
09.45 Introduction
Linking learning analytics, learning design and MOOCs
10.05 Group work
Focusing on learning outcomes in MOOCs
10.25 Plenary
Discussing how these data might be used to support learning
10.40 Group work
Analytics, step by step
11.00 Plenary
What would MOOC learners
and educators need to know
to use these analytics?
11.15 Workshop end
3
You can view and download these slides at
http://www.slideshare.net/R3beccaF

What are learning analytics?
High-level figures
Brief overviews for internal and external
reports
Academic analytics
Figures on retention and success, for the
institution to assess performance
Learning analytics
Use of big data to provide actionable
intelligence for learners and educators
4

Educators use analytics to
• Monitor the learning process
• Explore student data
• Identify problems
• Discover patterns
• Find early indicators for success
• Find early indicators for poor marks or drop-out
• Assess usefulness of learning materials
• Increase awareness, reflect and self reflect
• Increase understanding of learning environments
• Intervene, supervise, advise and assist
• Improve teaching, resources and the environment
5
Dyckhoff, A L, Lukarov, V, Muslim, A, Chatti, M A, & Schroeder, U. (2013).
Supporting Action Research with Learning Analytics. Paper presented at LAK13.

Learners use analytics to
• Monitor their own activities and interactions
• Monitor the learning process
• Compare their activity with that of others
• Increase awareness, reflect and self reflect
• Improve discussion participation
• Improve learning behaviour
• Improve performance
• Become better learners
• Learn!
6
Dyckhoff, A L, Lukarov, V, Muslim, A, Chatti, M A, & Schroeder, U. (2013).
Supporting Action Research with Learning Analytics. Paper presented at LAK13.

Analytics example: UK schools
7
• Aligned with
clear aims
• Huge and
sustained effort
• Agreed proxies
for learning
• Clear and
standardised
visualisation
• Driving
behaviour at
every level
BUT
• Stressed, unhappy learners
• Analytics with little value for learners or educators
• Omission of key areas, such as collaboration

Analytics example: Course Signals
Developed at Purdue University
8
Arnold, K E, & Pistilli, M (2012). Course Signals at Purdue: Using Learning Analytics
To Increase Student Success. Paper presented at LAK12, Vancouver, Canada.

Analytics example: SNAPP
Network analysis
9

Analytics example: iSpot
Heading
10

Learning design in MOOCs
● Puts the learning journey at the heart of the design process
● Provides a set of tools and information to support a learner-
activity based approach
● Helps to show the costs and performance outcomes of
design decisions
● Enables the sharing of best practice
● Helps MOOC designers to choose and integrate a coherent
range of media, technologies and pedagogies
● Enables a consistent and structured approach
to review and analytics
11
Mor, Y, Ferguson, R, & Wasson, B. (2015). Editorial: learning design, teacher inquiry into student learning
and learning analytics: a call for action. British Journal of Educational Technology, 46(2), 221-229.

MOOC learning design tools
• MOOC design template
• MOOC planner
• MOOC map
• Journey planner
12

Design template analytics
13
Learning outcome How this is assessed
1. Be able to define an ecosystem.
2. Have joined the iSpot community
and obtained identifications for
animals, plants or fungi.
1. Multiple choice. Week 1, question
5
2. Self report.
Analytics
1. How many attempted that question? How many got it right
1st / 2nd / 3rd time? How many followed the link back to resources?
2. Access to iSpot data. Use of MOOC hashtag. Persistence over time.
Ethical implications of tracking off-site.
Short description of course and learning outcomes

MOOC planner
• Delivered
• Reflection
• Collaboration
• Conversation
• Networking
• Browsing
• Assessment
14
Blocking out types of learning activity
Conole, Gráinne. (2010). Learning design – making practice explicit.
Paper presented at ConnectEd, Sydney, Australia. http://cloudworks.ac.uk/cloud/view/4001

MOOC planner analytics
Delivered Content (reading, watching, listening and observing)
Analytics: amount of content viewed, dwell time
Reflection (thinking, considering and reflecting)
Analytics: returns to the same material, reflection exercises completed,
quality of reflection
Collaboration (constructing, collaborating, defining and engaging)
Analytics: collaboration exercises completed, quality of collaboration
Conversation (debating, arguing, questioning, discussing…)
Analytics: number and length of contributions, quality of discussion
Browsing (exploring, searching, finding and discovering)
Analytics: Number of click-throughs to external links, number of visits,
number of resources
Assessment (answering, presenting, demonstrating, critiquing…)
Analytics: Assessments completed, scores, dwell time on hints,
persistence in answering questions
15

MOOC map analytics
● How long did you expect learners
to spend on these key elements?
● How long did learners actually
spend on the key elements
● How many missed out these
elements?
● How many jumped ahead to
these elements?
● Which types of element are
consistently (un)popular?
● How many left the MOOC at
these points?
16
The MOOC map identifies key elements of the course
0
100
200
300
400
500
600
Assimilative
InformationHandling
Productive
Experiential
Adaptive
Communicative
Assessment
Organisation
Minutes

MOOC journey planner analytics
17
Relationships between tools, resources, activities & narrative
A framework for data collection

Analytics to solve problems
Analytics could filter discussions or group learners
18
You have been actively engaged in the
discussions, which is excellent, thank you,
but with more than 23,000 participants it
means that our responses and comments
risk getting lost.
This will be primary school material for
some of you and exactly the opposite for
others. It is just not possible to tailor the
material to each of you […]
Introduction to
Forensic Science:
University of Strathclyde

Start with the pedagogy
• How do people learn?
• How can we use data to facilitate that process in our MOOC?
• Which elements are learners struggling with?
• Which sections engage them the most?
• What prompts them to ask questions?
• Are they finding assessment challenging?
• What misconceptions have learners shown?
• Are there any accessibility issues?
• How can analytics be used to obtain desired learning outcomes?
19

Learning analytics and design
Learning design – helping to identify useful analytics
● What do learners need to know in order to
network, collaborate, browse or reflect?
● What do educators need to know to support them?
Learning design – helping to identify gaps in the data
● What data do we need to collect?
Learning design – helping to identify gaps in our toolkit
● Which design elements can we look at easily?
● Which ones still pose problems?
Learning design – helping to frame & focus analytics questions
● What did they learn?… in relation to learning outcomes
● Were they social?... when they were collaborating
● Did they share links?... when encouraged to browse
● Did they return to steps?... when encouraged to reflect
20
Making the links

Group activity
20 minutes
Visit FutureLearn and register if necessary
https://www.futurelearn.com/
Select a MOOC currently open for registration.
Look at the introductory material and the first
week – What are the expected
learning outcomes?
How will learners know they
have achieved these?
21

Plenary
15 minutes
What types of learning outcome are
specified?
In what ways are learners assessed or could
they be assessed?
What sorts of data and analytics could be
used to support learners?
What sorts of data and analytics
could be used to support
educators?
22

Group activity
20 minutes
Take the first week of your chosen MOOC
Classify each step in terms of learning activity:
(delivered content, reflection, collaboration,
conversation, browsing or assessment)
Note how long the step would be likely to take
Discuss which types of analytic would
be useful to learners
and to educators
23

Plenary
15 minutes
Share your findings
Which types of data and analytic
could be used to support these types of
learning activity?
What would the educators need to know?
What would the learners need to know?
24