Bang for Your eLearning Buck: Enhancing the Learning Experience by Leveraging Learner Analytics

Enhancing the Learning Experience by
Leveraging Learner Analytics
A guide to how collecting & assessing learning outcomes
helps you deliver a differentiated elearning experience.

Agenda
1. EdTech Solutions that Achieve Learning Objectives
○ A review of the must-have solution components
○ All-in-one vs. modular solutions
○ How to deliver & track learning events: xAPI/TinCan vs. IMS Caliper
2. Translating Learning Events into Learning Analytics
○ What analytics work best to determine course effectiveness?
○ How to interpret learning data: learner performance vs. course performance
○ How instructors can leverage data for course improvements
○ What’s next: Adaptive learning

1. EdTech Solutions that
Achieve Learning Objectives

WebReader, iOS, Android AppsLMS
Learning Management System
User
Mgmt
Authoring Solution / LCMS
Project ManagementImport
(XML, ePub, PDF)
Authoring
styling, content, activities, media, ...User
Mgmt
Content Distribution
API
LTI
Analytics
Teaching Content
Content Creation
Analytics
Read, Interact
& Learn
Learning Library
Content Producers / Authors Instructors Learners
Analytics
User
Mgmt
build, manage & distribute
teaching & learning content
manage, assemble & distribute
learning content
learn on learning content
Learning Event Server
Global Solution Architecture

User Management
● User invitation,
registration, login
● SSO, access control,
module permissions
● Project teams & project
permissions
➢ Users: Education Publishers, Authors, Business Learning Centers, Communication Dpt, ...
➢ Solution Role: Build, Manage & Distribute teaching, learning or communication content
Project Management
● New project creation
● Project import via various
formats (ePub, PDF, XML, …)
● Project list, sort, search, delete
● Project information & metadata
● Project settings & accessibility
Analytics
Insights for content creators based on end-user interactions to improve content
Project Authoring
● Project theme/layout: colors,
fonts, styles, layout CSS
● Project enrichment: media,
activities, widgets, …
● Print/digital or learner/instructor
specific content
Project distribution
● Project versioning
● Project team-review
● Distribution to learners:
LMS, LTI direct linking,
custom API, direct to
readers
The Authoring Solution

User Management
● User invitation,
registration, login, SSO
● Instructor management
● Classrooms / Groups
Content Library
● Content preview, bookmarks
● Content assembling from
learning objects
● External content import
Analytics
Insights for Instructors on content end-user interactions to improve learning
Content Authoring
● Simplified authoring
● Insertion of media, activities
● Distribution to group of
learners / users
● Assignation, expiration
date, specific settings
➢ Users: Teachers, Instructors, Communication Managers
➢ Solution Role: Manage & Distribute learning or communication content
The Learning Management Solution

User Management
● User registration,
login, SSO
● User profile, settings
Learning Library
● List, sort content, detailed info
● Content download for offline reading
● Search content
● Additional content purchase
● Social
Analytics
Summary of interactions with content, progress reports
Content Readers
● Content display with layout
● Media rendering
● Activities & assessments
● Search in content
● Accessibility
● User annotations
● Interaction events
➢ Users: Students, Learners, Employees, Customers, ...
➢ Solution Role: Discover & Interact with the distributed content
Android Apps
Chromebooks
MacOS
iOS apps
Windows
The Readers for End-Users

To build & deliver digital content, we see companies follow a similar development path:
To integrate with the legacy ecosystem already in place, specialized solutions need to:
● Be modular with open API available
● Support industry standards: IMS LTI, Caliper in Education/Publishing, Scorm / Tincan for Business Learning
● Integrate with market-leading solutions such as Moodle or Blackboard LMS
Start with an all-in-one
solution, generally a LMS
that can author
& deliver to the web.
Addition & integration of more
specialized solutions.
1
Expansion of digital distribution
=> Richer needs,
workflow industrialization
2
Authoring Solution
Mobile & Web
Distribution Solutions
Analytics Solution?
Integrated or Modular Solution?

Analytics are about building a quantitative analysis of the end-users’ behavior with your content.
● Recipients: end-user, instructors, content producers, …
● Types of analytics: aggregate figures, ratio, charts & graphs (dataviz)
Track meaningful end-user
interactions
with content (= events)
Flow events into different user
interfaces
Conclude
most meaningful analytics for
recipient based on events
Build interfaces that provide
easy access to desired
info / analytics
Getting Started with Analytics

xAPI (Project TinCan)
Initiative led by the &
Open-source, Apache v2 license
Successor of SCORM since 2013
API Goals:
● Allow applications to share data about human
performance on instructional content & context information
● Store information in a Learning Record Store (LRS)
➔ JSON format over HTTP + OAuth
➔ Service and device agnostic = versatile, transferable
from LMS to mobile devices, wearables, search
engine, gamification, blog, webinar, CRM, ...
24 conformant Learning Record Stores, 39 official adopters
➔ Versatile, but mostly adopted in business learning
Best practices for collecting end-user interaction events led to 2 different analytics standards:
Caliper v1.1
Initiative led by the , started 2015
Closed consortium which controls the Caliper IP
Goals:
● Approach to describing, collecting & exchanging learning
activity data
● Way of measuring learning activities & effectiveness
● Enables new services to measure, report & visualize
➔ JSON format over HTTPS/TLS with API key
➔ Focus around the LMS only => adopted or being
adopted by the main LMS vendors
Only 11 solutions certified by IMS, ex: Blackboard, Canvas,
VitalSource + others having “adopted” ex: Moodle, Elsevier
➔ Focused on learning, mostly in e-education
Analytics Standards

Caliper v1.1xAPI
Analytics Standards Visualized

xAPI (Project TinCan)
The Activity Provider delivers the learning experiences and
communicates the learning info to the Learning Record Store.
Learning experiences are described through statements:
● Actor: individual/group performing an action within an Activity.
● Verb: action performed during the learning experience.
The xAPI only specifies how to create meaningful Verbs.
● Object: something that the Actor interacted with,
ex: a book, an elearning course, a unit of instruction, an activity
● Result (optional): measured outcome of the statement
ex: score, success, completion, response, activity duration
● Context (optional): additional information ex: the instructor,
inclusion in a broader activity or team activity
● Attachment (optional): evidence of a learning experience
ex: a record of communication, an essay, a video, or a
certificate
● TimeStamp, Authority: device/service delivering the statement
Caliper v1.1
An activity within an Application is tracked through a Sensor
Endpoint that sends events to a Record Store via the Sensor
API.
A Metric Profile describes a type of learning activity with a
controlled & constrained vocabulary.
ex: Annotation, Assessment, Forum, Media, Reading, Outcome, …
● Actor
● Action
● Activity
● Optional: EventTime, target, generated, referrer, edApp, group,
membership, session, federatedSession (LTI), extensions
Analytics Standards Cont.

2. Translating Learning Events
into Learning Analytics

Standard Analytics*
1. Content Popularity > Volume & Audience
> Check time spent, # of pages read & # of unique users
> Compare between content subparts & similar content blocks
● Beware that time spent can be rigged; check how pages read are
counted (ie. scroll in the page, assess minimum time spent)
● Check the distribution scenarios; volume can come from content
popularity either among instructors or learners
2. Learner Engagement > Interactions with Content
Digital learning brings the unvalued benefit of engaging learners through
interactivities, making the learning more active and thus efficient.
> Check # of interactions & time spent with videos, audio, quizzes or
widgets within a content block
3. Learner Success > Scores
> Check learner score across different activities to assess their
understanding of a course
> Compare between content blocks to assess their relative difficulty
* Standard learning stats:
- Time spent
- Average result
- Last activity date
- Sessions, ...
What analytics best determine course effectiveness?

Advanced Analytics
1. Learner Cohort Analysis
> Check % of pages read & volume of interactions by learner over time
If % is plateauing at a low level vs other content, this might be due to not
having enough engaging content or having irrelevant courses.
2. Time-Efficient Learning
> How much time is needed on average for a learner to master a topic
having learned with your content?
Compare average score on the same assessment between readers having
interacted with different content pieces with similar learning objectives.
3. Learner Progression
> Identify correlations between learner result progression over time and
their interaction with specific content or piece of subcontent.
Example of Cohort Analysis
What analytics best determine course effectiveness?

Analyze Learner Performance Over Time, Across Courses
● Check how learner average performance progressed after a course:
○ Assess before & after the course
○ Compare course performance vs difficulty
● Check correlation between performance with time spent & volume of
engagement:
○ Check performance of most engaged learners vs the others
○ Take note that more engaged learners are often low-level learners
Analyze Performance Distribution Across Learners
● Check how course is performing among low-level, standard &
high-level learners:
○ Assess before & after the course
○ High-level students usually need less teaching content => course
performance is mostly reflected on the performance increase on
low-level & standard learners
How do you clearly distinguish the respective impact of
learning content, instructor methodology & learner work
on learner performance ?
Interpreting Learning Data: Learner Performance vs.
Course Performance

Learning Content Improvements
● Course Clarity & Efficiency
○ How are learners performing on the content?
Is the score distribution binary or more evenly distributed?
○ Are learners spending too much time on the content in
relation to the performance increase?
➢ Review the course layout and pedagogy
● Course Engagement
○ Is the level of learner engagement matching the level of
other content?
➢ Add interactive elements & activities to engage learner
● Course Level / Difficulty
○ Compare the average score across learners for the
different course sections and across courses vs
expectations
Teaching Improvements
Teaching efficiency is highly correlated with learner engagement / level of
activity and teaching personalization.
● Engage Learners
○ Ensure courses & assignments have a good mix of activities
○ Encourage learners to interact more with the content
● Personalize Teaching
○ Start with a quick test to break down the classroom into ability groups
○ Differentiate assignments for each ability group via digital learning
○ Identify the sticking points & root-causes for low-level learners:
■ Assess pre-requisites
■ Slow down learning pace / try a more progressive learning method,
(ex: step-by-step > application > more complex assessments)
■ Try other pedagogies & learning content types (videos, schema, …)
Which data & actions can improve your teaching ?
Leveraging Data for Course Improvements

What is Adaptive Learning?
Adaptive learning is about automatically personalizing the learning path of
each learner to optimize the efficiency of their learning experience for
self-defined learning objectives or by assignment from an instructor.
Analyzing learner grades helps you to continually determine:
● 🔽Is a learner struggling? → teach prerequisite knowledge
● = Is a learner progressing as expected? → no action required
● 🔼Is a learner already mastering the knowledge? → save time & push them
learning content for more advanced or other supplementary knowledge
What’s Next: Adaptive Learning
➔ It can also try to identify per learner the most efficient types of learning
content, learning pace, optimal hours of learning, …
➔ A learning profile is built for each learner over time based on all their
interactions with the content & assessment results.
➔ Memory anchoring features, is available, can ensure knowledge is correctly
memorized over time by regularly reviewing it (recurrence of testing is
adapted to each learner).

Prerequisites & Adaptive Learning Engine
1. Add a Semantic Layer to Your Content
○ Atomic learning objects (LOs) provide more granularity for
personalizing each learner’s learning path
○ LOs to be metattagged with role & target knowledge
ex: “LO 1 teach addition with 2 digits”
○ Build a knowledge tree that links the different knowledge of the
content together to detail which ones are prerequisites to others
2. Track Learner Behavior: Assessment scores, time spent, interactions
with media and interactivities, …
Based on the LO semantic & learner interactions, adaptive learning
engines leverage bayesian or neural network algorithms to dynamically
identify which next LO should be shown to each learner.
Limits & Perspective
● Requires additional work to add semantic layer to learning content
● Fully automated engines (ex: blackbox) work better for self-paced
learning, but are harder to incorporate with an instructor pedagogy
● Works best as an instructor aid with learning recommendations &
insights on learners
What’s Next: Adaptive Learning

Summary
○ The must-have components:
■ A global architecture
■ A combination of authoring, learning management and end-user solutions
○ All-in-one vs. modular: we recommend all-in-one, pending your unique needs
○ xAPI/TinCan vs. IMS Caliper: either, as long as you can build a quantitative analysis of user behavior
○ Standard analytics: Time spent, # pages read, # unique users, learner scores
○ Advanced analytics: Learner cohort analysis, time efficiency, learner progression
■ Both kinds of data help educators pivot course improvement and student engagement
○ Adaptive learning: the future of personalizing the elearning experience for all learners

25
EVP Global Sales and Marketing, Gutenberg
Technology
Linkedin page: sethpuri/
Twitter ID: @SethPuri
Email: spuri@gutenberg-technology.com
Website: gutenberg-technology.com
Editor, Aggregage
Linkedin page: elearning-learning/
Twitter ID: @eLearningPosts
Email: eve.lyons-berg@aggregage.com
Website: elearninglearning.com

IMS Caliper Adoption / Certification

{
"actor": {
"name": "Sally Glider",
"mbox": "mailto:sally@example.com"
},
"verb": {
"id": "http://adlnet.gov/expapi/verbs/experienced",
"display": {"en-US": "experienced"}
},
"object": {
"id": "http://example.com/activities/solo-hang-gliding",
"definition": {
"type": "http://adlnet.gov/expapi/activities/course",
"name": { "en-US": "Solo Hang Gliding" }
}
},
"context": {
"instructor": {
"name": "Irene Instructor",
"mbox": "mailto:irene@example.com"
},
"contextActivities":{
"parent": { "id": "http://example.com/activities/hang-gliding-class-a" },
"grouping": { "id": "http://example.com/activities/hang-gliding-school" }
}
}
}
xAPI Statement Example

Bang for Your eLearning Buck: Enhancing the Learning Experience by Leveraging Learner Analytics

Recommended

Recommended

More Related Content

Similar to Bang for Your eLearning Buck: Enhancing the Learning Experience by Leveraging Learner Analytics

Similar to Bang for Your eLearning Buck: Enhancing the Learning Experience by Leveraging Learner Analytics (20)

Recently uploaded

Recently uploaded (20)

Bang for Your eLearning Buck: Enhancing the Learning Experience by Leveraging Learner Analytics