The document discusses getting started with learning content analytics. It provides an overview of key concepts including why organizations measure learning activities, common data sources, and how to share insights. Examples are presented from clients like Western Union who use analytics to understand learner engagement, reactions, competence and perceptions. The session aims to help participants think more like data scientists and use analytics to improve learning programs and outcomes.

1. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
1
DevLearn 2023 – Session 509
Presented by: Cammy Bean, Joe Replin,
and special guest Rodrigo Bolanos
Getting Started with
Learning Content Analytics

2. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
2
Poll:
Where is your
organization
regarding learning
data & analytics?

3. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
3
of organizations say better
analytics and intelligence is a
priority for 2023
Fosway Digital Learning Realities
2022
93%

4. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
4
And yet, that’s not our
core skillset….
ATD Talent Development Capability Model – Capability Benchmark,
https://d22bbllmj4tvv8.cloudfront.net/2f/0f/e7a3766a43df84df51e8af7e689e/benchmarks-from-the-talent-development-
capability-model-2023-q2-with-segments.pdf
Retrieved 10/2/2023.

5. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
5
Cammy’s next book?

6. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
6
For today….
Let’s get ourselves upskilled and start thinking more like
data scientists.
Laying the
Foundation
Building Blocks
of a Data-
Driven
Organization
Thinking in
Strategic
Dimensions
A Framework
for Learning
Content
Analytics
Content
Analytics
Applied
Examples &
Tales from the
Real World

7. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
7
Cammy Bean
Senior Solutions
Consultant
Joe Replin
Senior Technical
Consultant
So, who are we?
What makes us think we can talk about learning content analytics
with any authority?
Rodrigo Bolanos
CEO, Kineo LATAM

8. The Building Blocks
of a Data-Driven
Organization
8
7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics

9. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
9
But first, WHY do we measure?
Inform
Monitor
Evaluate & Analyze
Manage

10. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
10
And another way to think about the
WHY of analytics….which also lends
itself to a useful maturity model….
Descriptive
Explains
WHAT
happened
Diagnostic
Delves into
WHY it
happened
Prescriptive
RECOMMENDS
action based on
the forecast
Predictive
Forecasts what
MIGHT happen

11. 11
Outcomes
IMPACT of learning
on business or HR
goals
Effectiveness
QUALITY – Reaction,
Learning, Application,
Impact, ROI, Manager
Support
Building Block #1: Knowing WHAT to measure….
Efficiency
QUANTITY of training
delivered

12. 12
The Learning Guild Research Report: The State of Data & Analytics in L&D: 2023”

13. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
13
Building Block #2: Knowing WHAT data sources
you have
Internal
Primary
SCORM | Kineo Analytics | xAPI | Surveys | Focus Groups |
Knowledge Checks & Assessments
Secondary
Interviews | Collaboration Tools | Performance & Skills
Reviews | Peer Feedback | Sales Results | KPIs | Company
Portfolio
External
Primary
Industry Trends | Accessibility & DEI Reviews | Localizations &
Translations
Secondary
Internet of Things | Social Media | Google Analytics |
Clickstream Data | Mobile Apps

14. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
14
Building Block #3:
Sharing meaningful
insights back to the
business
Numbers don’t tell a
story on their own.
Be the storyteller….

15. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
15
Learning
leader
Business
& policy owner
Learning
designer
Learner
Who’s your audience? What stories do they need to hear?
Show me whether we’re
creating the best learning
experiences for our employees
and whether we’re adding
value to the business.
Show me that people aren’t
spending unnecessary time on
training and that our people
have the skills we need them
to have.
Show me how
people are
interacting with the
experiences I design
and where I can
improve them.
Show me if we are mitigating
risks and can prove this to
compliance regulators.
Show me that you’re listening
to me and developing
programs that support my
development and give me the
best tools to do my job.

16. Telling an impactful story
7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
16
It’s about people, not numbers

17. Telling an impactful story
7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
17
Make it visual!

18. Take your audience
on a journey
Telling an impactful story
7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
18

19. Telling an impactful story
7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
19
Give it real life, relatable context

20. Telling an impactful story
7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
20
Describe how the data has
inspired change – share
what you’re going to DO
differently as a result

21. Your thoughts
21
What’s holding your
organization back from doing
more?
What are the biggest barriers?
7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics

22. What’s holding organizations back?
7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
22
Knowing
what data to
collect –
within data
regulations
Cost of
analytics
solutions
Internal
expertise in
analyzing
data
Presenting
data in a
compelling
way
Identifying the
potential
Limits to
SCORM

23. Learning Content
Analytics:
A Framework
23
7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics

24. Learning content analytics aren’t
just about who launched what and
who’s completed what – they’re
about understanding how learners
interact with the content and how
they experience the learning.
7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
24

25. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
25
Course
launch
Course
launch
Course
launch
Course
launch
Hmm...people really
struggle here. Maybe we
need to try something
different.
Isn’t that interesting? Maybe we need to
adjust that question. And look - barely
anyone is accessing this via mobile…
How’d it go? What did
we notice? What can
we do better next time?

26. Don’t just use data as another way
to tick the box – use it to make
better experiences for learners
7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
26

27. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
27
Traditional reporting (SCORM)
• Individual Learner details
• Completion & Objective Status
• Time & Score
Learning content analytics
Who did what? How and why?
• Require further/skills processing for
deeper insight
• Aggregated trends on learning
behavior
• Learner Engagement & Performance
throughout the experience
Wider insight via automated visualisation
Limited insight via manual processing
• Interactive dashboards on trends
• Changes in perceptions, confidence,
reactions over time
• Measuring against learning
objectives

28. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
28
Impact
Behavior
Learning
Reaction
Learning content analytics and evaluation
Learning content
analytics allow us to
measure learning
and reaction…
…and give us a foundation
for measuring impact
and behavior

29. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
29
Learning Content Analytics:
Thinking in strategic dimensions

30. It’s easy to “drown in data” if we
don’t sort it and categorize it in a
logical way…
…so Kineo uses four dimensions
to categorize data and insights.
….and sometimes we add a few
more….
7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
30

31. Engagement Reaction
Competence
(Applied Knowledge)
Perception
(Confidence)
Measures
Ability to apply
knowledge in realistic
scenarios and/or
exercises
Example
• The difference
between activities
at the start and end
of the learning
• Performance
against different
learning objectives
31
7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics

32. Engagement Reaction
Competence
(Applied Knowledge)
Perception
(Confidence)
Measures
Ability to apply
knowledge in realistic
scenarios and/or
exercises
Perceived confidence
around an area of
behavior or knowledge
Example
• The difference
between activities
at the start and end
of the learning
• Performance
against different
learning objectives
• Confidence before
and after the
learning
• General confidence
about specific skills
or knowledge
• How they rate
current mastery
level
32
7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics

33. Engagement Reaction
Competence
(Applied Knowledge)
Perception
(Confidence)
Measures
Ability to apply
knowledge in realistic
scenarios and/or
exercises
Perceived confidence
around an area of
behavior or knowledge
How someone
interacts with the
learning
Example
• The difference
between activities
at the start and end
of the learning
• Performance
against different
learning objectives
• Confidence before
and after the
learning
• General confidence
about specific skills
or knowledge
• How they rate
current mastery
level
• How someone
uses the learning
• Time spent /what
content visited:
• Duration
• Repeat visits
• Completion
rates
• Accessibility
• Usability
33
7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics

34. Engagement Reaction
Competence
(Applied Knowledge)
Perception
(Confidence)
Measures
Ability to apply
knowledge in realistic
scenarios and/or
exercises
Perceived confidence
around an area of
behavior or knowledge
How someone
interacts with the
learning
People’s opinion or
feelings about the
course and intent to
apply
Example
• The difference
between activities
at the start and end
of the learning
• Performance
against different
learning objectives
• Confidence before
and after the
learning
• General confidence
about specific skills
or knowledge
• How they rate
current mastery
level
• How someone
uses the learning
• Time spent /what
content visited:
• Duration
• Repeat visits
• Completion
rates
• Accessibility
• Usability
• How do they feel
about the course?
• Was it useful and
relevant?
• Did they enjoy it?
• Intent to behave
differently?
34
7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics

35. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
35
Meet Heather: She oversees a large mandatory training
portfolio at a global bank
Use Case: Maximize impact while minimizing seat time
• HR department wants
required training to be
delivered as concisely
as possible
BUSINESS NEED
• Know what to do and how
to behave in a range of
situations
• Confidence to apply on
the job
• But - a little fatigued from
years of mandatory
learning
LEARNER NEED
• Focus future training
on what learners need
most
• Reduce seat time
while making sure all
training gaps are filled
DESIRED OUTCOME

36. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
36
Meet Heather: She oversees a large mandatory training
portfolio at a global bank
Use Case: Maximize impact while minimizing seat time
Engagement
• Average time to
complete?
• Where spending
the most time?
• Where breezing
through?
Reaction
• Did learners
find the course
relevant and
enjoyable?
• Do they intend
to behave
differently?
Competence
• Pre & post
tests – what do
learners
already know?
• What do they
get wrong?
(Applied Knowledge)
Perception
• Confidence
levels going
into course and
after?
• Does
confidence
match
competence?
(Confidence)

37. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
37
Meet Mo: They are launching an optional course for a global
sales team
Use Case: Building learning programs to inspire culture
change
• Create a global sales
language to enable
consistent internal
and external
customer experiences
BUSINESS NEED
• Use the same terminology
and processes as peers
• Move on from some siloed
behaviours
• Need to squeeze training
in between face-to-face
meetings
LEARNER NEED
• A pilot phase to
inform a full roll-out
• Include feedback from
regional leads on how
well the learning is
being applied in real
life
DESIRED OUTCOME

38. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
38
Meet Mo: They are launching an optional course for a global
sales team
Use Case: Building learning programs to inspire culture chang
Engagement
• How long are
they spending
on the training?
• Is it in one
sitting or in
pieces?
• What devices
are they using?
Reaction
• Did learners find
the course
relevant and
enjoyable?
• Do they intend
to behave
differently?
Competence
• Using scenarios
or simulations,
can learners
apply the
preferred
behaviors?
(Applied Knowledge)
Perception
• How do they
feel about new
behaviors?
• How effective
do they believe
new behaviors
will be?
(Confidence)

39. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
39
Examples
(…or the part of the session
where we show how we’re
tackling the data question
for some of our clients)

40. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
40
Western Union – xAPI
to tell a better
content story

41. Western Union: Compliance, Gamification and
Analytics
7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
41
• Compliance and System Simulator
• 8,000 agents
• Main business drivers:
• Client Satisfaction
• Transaction speed
• Compliance
• Intentionally designed: Moments of
Risk
• Adapt Learning + xAPI + LRS

42. Design Characteristics
7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
42
• Gamification
• Three main business drivers converted
into points buckets. Compliance
activities = lifes
• Depending on answers the three
metrics will be impacted
• Conversational: emulating a
conversation "a la" chatbot
• Challenges based on moment of risk
during transactions

43. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
43

44. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
44

45. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
45

46. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
46

47. Results
7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
47
• Roll out to 8,000 agents
• Leaderboard for rewards
• Map of most incorrect
answers:
• Map of the activities with
more risk: valuable
information to focus
efforts
• How many times they
"lost their life"

48. Results
7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
48
• Roll out to 8,000 agents
• Leaderboard for rewards
• Map of most incorrect
answers:
• Map of the activities
with more risk:
valuable information
to focus efforts
• How many times
they "lost their life"

49. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
49
One organization’s
journey from scratch
to data maturity….

50. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
50
One organization’s three-year journey to
data & analytics maturity
(Hint: it really does take time to do this right!)
Developing our
strategy
Examining
results
Refine & Retest

51. Year 1 – Build your strategy & find your stories
7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
51
Frame the
Problem
Collect
Raw Data
Clean
the Data
Explore
the Data
Analyze
the Data
Tell Your
Hidden Stories
2
1 3 4 5 6

52. Year 1 – Frame the problem
(what to measure and why, how to collect the data)
7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
52
• The goal is to attempt to produce questions
that we feel data might help us find answers
to.
• Build your requirements to the purpose you
are using it for.
• You want to collect purposeful data, not fluff.
• Avoid talking about visualizations, graphs, &
what data to collect.
• Document!

53. Year 1 – Frame the problem
7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
53
• Does our expected course duration align with
actual seat time?
• How are learners performing on knowledge
checks & scenarios?
• Are the videos we are producing worth the
time & effort spent?
• How much value does our UI offer?
• 8% of learners reported connectivity issues
previously. How can we help the Learning
Support Team?

54. Year 1 – Collect raw data
7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
54
Tying questions to course events and assigning
xAPI verbs & context.
• Track active course time
• Track all video interactions
• Gather detailed question answer & scoring
data
• Track all UI interactions
• Implement connection test in course.
Measure the frequency and types of data loss
issues occur between course, LMS & LRS

55. 55
The big project
• 2-hour, organizational-wide compliance
• 4 organizations working collaboratively
• Video-driven episodic experience
• 200,000+ learners
• Cornerstone, Learning Locker, & Adapt
Framework
• WCAG 2.0 AA
• 5 additional languages
• By design, learners are afforded the
opportunity to test out of a final
assessment
• Videos are optional to watch
7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics

56. Year 2 – Analysis at-a-glance
7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
56

57. Year 2 – Explore, Analyze & Tell a Story
7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
57
• Don’t give learners such a hard time.
• Practice behavior change over strict
competency - allow learner to ask for help
• No change to video approach as they were
well-received
• Implement inactive learner notification
• Slightly raise visibility of course resources

58. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
58
Year 3 – Analysis again!

59. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
59
Kineo Analytics –
another way we’re
trying to help our
clients solve the data
challenge

60. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
60
The challenges of xAPI…
Despite all the hype,
the LRS (Learning
Record Store) and
xAPI are seeing really
slow growth.

61. Raise your hands!
61
Who’s using xAPI?
7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics

62. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
62
The Learning Guild Research Report: The State of Data & Analytics in L&D: 2023”

63. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
63
Kineo Analytics tracks data
across your digital learning
experiences, providing the
insights you need to prove
the success of your online
learning content and to
make smarter decisions.

64. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
64
A cloud-based
analytics service
Works with any LMS
with no need to invest
in an extra platform.
Included with our
Adapt builds
Get more value from
the content you
develop with us, as
standard.
Goes beyond
SCORM
Track metrics, trends
and patterns that
normally require an
analytics platform.
Advanced
dashboard
visualizations
Identify trends within
your data through
interactive
visualisations, with
preset views.
Kineo Analytics service overview

65. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
65
What’s happening behind the scenes?

66. 66
Kineo Analytics – Better
Data for Better Decisions
▪ Cloud service captures
anonymized data about
usage of Adapt learning
▪ Goes beyond SCORM
tracks across several data
dimensions for deep insight
▪ Works on any LMS without
custom set-up and at no
extra cost
Standard
SCORM
Kineo
Analytics
Individual Completion Status x
Individual Assessment Score x
Question Responses x
Average Duration x
Average Attempts x
Perception/Confidence Changes x
Pre versus Post Assessment Scores x
Percentage Incorrect Responses Per Question x
Number of Attempts Per Question x
7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics

67. 67
Kineo Analytics
Data on how audiences have used your
solutions:
• Number of users / launches / completions
• Average time to complete
• Pre and post assessment scores
• Impact on knowledge, perceptions and
confidence
• Learner experiences
• Different devices used to access learning
Fully
GDPR
compliant
7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics

68. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
68
Screen
resolution
and
orientation
Connection
(e.g., 4G,
5G)
Device,
operating
system and
browser
Time spent
in learning/
Active time
in learning
No. of
attempts
per
question
Pre and
post
assessment
scores and
comparison
Average
correct/
incorrect
responses
Average
ratings
Performanc
e against
learning
objectives
…and more

69. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
69
Wrapping up

70. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
70
What can you do with these insights?
Prove your
success
Improve
learning
experiences
Prioritize
learning
budgets
Tell stories
with data
Better
understand
your
learners

71. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
71
Resources

72. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
72
Check out Cammy’s book!

73. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
73
Cammy Bean
Senior Solutions Consultant
Cammy.bean@kineo.com
https://www.linkedin.com/in/
cammybean/
Joe Replin
Senior Technical Consultant
Joe.replin@kineo.com
https://www.linkedin.com/in/
joseph-replin/
Stay in touch!
Rodrigo Bolanos
CEO, Kineo LATAM
Rodrigo.bolanos@kineo.com
https://www.linkedin.com/in/
rodrigobolanos/
www.kineo.com
Find out more about Kineo Analytics
Get in touch: info@kineo.com
Keep up with us on socials:
https://www.linkedin.com/company/kineo

74. 7
November,
2023
DevLearn
2023:
Getting
started
with
content
analytics
74
Thank you!